diff --git a/source/Development/developing_change.rst b/source/Development/developing_change.rst index ba4646cb..31f6ab23 100644 --- a/source/Development/developing_change.rst +++ b/source/Development/developing_change.rst @@ -54,6 +54,7 @@ carefully: kgo diagnostics rose_stem + testdata testing .. important:: diff --git a/source/Development/testdata.rst b/source/Development/testdata.rst new file mode 100644 index 00000000..cc68c8b2 --- /dev/null +++ b/source/Development/testdata.rst @@ -0,0 +1,122 @@ +.. ----------------------------------------------------------------------------- + (c) Crown copyright Met Office. All rights reserved. + The file LICENCE, distributed with this code, contains details of the terms + under which the code may be used. + ----------------------------------------------------------------------------- + +.. _testdata: + +Adding Test Data +================ + +.. note:: + + This page is a placeholder for information about test data. It is not yet + complete and will be updated in due course. + + *These instructions apply to all Met Office managed systems and related + external platforms (such as JULES data on JASMIN), though other sites may + manage their test data differently.* + +.. important:: + + If you have questions about the process or concerns about the provenance of + the data you want to include, please engage with the IAO as early as + possible to prevent delays to your change later on. + +.. .. important:: **Attribution Metadata Policy** + +.. If the change requires a new or updated file in ``LFRIC_DATA_DIR`` then you +.. will need to work with the Information Asset Owner (IAO) to ensure that data +.. in ``LFRIC_DATA_DIR`` includes clear attribution and licence metadata. +.. Where possible, this should follow existing UM ``ANCILDIR`` conventions (`see +.. below `_), with ``.attribution`` and ``.license`` +.. files or equivalent NetCDF **global attributes** (at least, ``references``, +.. ``license``, ``source``, and ``history``). Attribution must reflect the +.. original data source and be provided by the data creators before deployment, +.. share, or distribution. + +.. It is treated as an **Information Asset / licensing requirement**, not just +.. a best practice. + + +For UM related datasets, please Email the `MIAO team `_ +to discuss the best way to share the data. + +.. _prerequisites-section: + +Prerequisites +------------- + +Before adding test data, you should have a good understanding of the change you +are making and the tests you will be adding. You should also have a good +understanding of the codebase and the testing framework you will be using. + +Licensing and Attribution +~~~~~~~~~~~~~~~~~~~~~~~~~ + +All files require a licence and a record of where they have come +from, both for legal and auditing purposes. In your request +please describe where and how the data was generated, and the +terms and conditions of its licence. + +Before any files can be deployed, they must be approved by an IAO +and this cannot be done without information about the licencing +terms. It is your responsibility as a developer to obtain this +approval. + +Where the data is in netCDF format, it should use the global +attributes described in the section below; data files in other +formats should follow the existing UM ``ANCILDIR`` conventions. +Attribution must reflect the original data source and be provided +by the data creators before deployment, share, or distribution. + +Metadata +~~~~~~~~ + +All file requirements should be recorded in or alongside the files being +deployed. + +If a source file has a licence that imposes requirements on derived +works, then any ancillary file (or an intermediate file used to generate an +ancillary) counts as a derived work for the purposes of recording metadata. + +In cases where a file has been generated from multiple sources, the licences +must be compatible with each other and it should be made clear where each +licence/attribution/acknowledgement has come from. + +NetCDF Files +^^^^^^^^^^^^ + +NetCDF files should have the relevant metadata included in the file itself. +The metadata should include the following information: + +* The licence should be in a ``license`` global attribute as per + `ESIP Attribute Convention for Data Discovery `_. + +* If there is a paper attribution requirement, the relevant paper(s) should be + cited in the ``references`` global attribute as per + `CF conventions `_. + +* If there is an organisation attribution requirement, it should be in the + ``institution`` global attribute (again, as per CF). + +* If there is any other attribution requirement (e.g. for an individual), it + should be in the ``acknowledgement`` global attribute (again, as per ACCD). + +* If there are restrictions on usage (e.g. "research only"), these should be in + a ``restrictions`` global attribute. + +Other Files +^^^^^^^^^^^ + +* Licence should be in an accompanying plain text file with the same name as the + data file, but with a ``.license`` suffix. + +* Attribution should be in an accompanying plain text file with the same name as + the data file, but with a ``.attribution`` suffix. + +* Restrictions on usage (e.g. "research only") should be in an accompanying + plain text file with the same name as the data file, but with a + ``.restrictions`` suffix. +