Add resourceTypeGeneral for dataset files

[philippconzett]

What is the problem that your suggestion solves?

Our research data repository uses DOIs at file level, meaning that each file within a dataset gets its own DOI. For example, the file https://doi.org/10.18710/WUJCYB/VLJNH1 is part of the https://doi.org/10.18710/WUJCYB dataset. However, in DataCite Commons and other discovery services relying on DataCite metadata, dataset files are handled at the same level as datasets. For example, ORCID records of depositors are flooded with dataset file records, so it's almost impossible to spot the major achievements, which in case of data deposits would be the datasets.

What solution might meet your needs?

To solve this problem, I suggest introducing a resourceTypeGeneral for dataset files to the DataCite metadata schema. This would allow metadata harvesters and indexers to filter information in a more targeted manner. For more information, see Dataverse GitHub issue #5086 and PID Forum thread.

Your name

Philipp Conzett

Your organization

UiT The Arctic University of Norway

What alternatives have you tried or considered?

Stop using DOIs at file-level. However, this alternative would not be aligned with good practice recommendation of granular PID strategies.

Is there anything else you would like to share?

This suggestion might relate to the current discussion of introducing a new Distribution property.

What group(s) would benefit from your suggestion?

• DataCite members registering DOIs and metadata and administering accounts
• Developers building integrations with DataCite services
• Harvesters who index DataCite DOIs and metadata
• Individuals or institutions analyzing DataCite DOIs and metadata
• DataCite staff
• Other

If other group(s), please describe.

Researchers relying on granular metadata

[KellyStathis]

Thanks @philippconzett for this suggestion! If you have any specific suggestions for the resourceTypeGeneral name ("File"? "Component"?), we'd be happy to consider that too as we explore this. We definitely want to find a solution that helps discovery services filter out file-level DOIs when desired.

[philippconzett]

Thanks, @KellyStathis! I'll ping the Dataverse and other communities. Also, I slightly edited the suggestion above, making sure we're talking about resourceTypeGeneral.

[pdurbin]

Hi! I saw Philipp's post at https://groups.google.com/g/dataverse-community/c/hKVje3PMw5E/m/Z9OJUsGbBgAJ

As I understand it, here are the options we have today, from https://datacite-metadata-schema.readthedocs.io/en/4.6/properties/resourcetype/

Let's say we add the new option Philipp wants, a generic option for dataset files. Would it make sense to use it for all dataset files? What if the file in question actually matches something that's already in the list, like Image, Software, Sound, or Text? Should we send the more specific value if we can?

[jggautier]

@pdurbin you wrote that the free text resourceType is not used. Are we sure that it's not used?

For the dataset at https://doi.org/10.7910/DVN/S8239A, depositors typed in "Qualitative" in the "Data Type" free text field:

And in that dataset's "Datacite" export, I see <resourceType resourceTypeGeneral="Dataset">Qualitative</resourceType>:

@KellyStathis when you asked "What does Dataverse currently use for the free text resourceType?", it sounded like you meant the value of resourceType, which can be free text, as opposed to the value of resourceTypeGeneral, which must come from DataCite's resourceTypeGeneral list.

[jggautier]

@philippconzett you wrote that:

I noticed that figshare uses different values for resourceTypeGeneral. For example, for a record (or what in Dataverse is called a dataset) that consists of a TIFF file, they use "Image" as the resourceTypeGeneral.

I wonder if this design conflicts with recent guidance from the GREI Metadata Recommendations from DataCite schema version 4.6. It highlights that data repositories should use "the value Dataset for resourceTypeGeneral". And I'd guess that the folks at Figshare are considering changing how Figshare uses resourceTypeGeneral to follow that guidance.

[bryanmgee]

I agree with @jggautier, I think whatever is entered in "Data Type" is entered into the resourceType field; I just checked a dataset that I published myself in the Texas Data Repository where I put 'Tabular data' in that field, and that's what shows up in the DataCite API record.

Whether to use 'Dataset' in resourceTypeGeneral for a singular file that has a DOI but that is also nested within a "true" dataset that has its own DOI and that is also labeled as 'Dataset' is tricky. I would prefer that individual files with their own DOI not be labeled 'Dataset' in resourceTypeGeneral because it makes large data analyses and retrieval complicated. For example, if someone queries 'Dataset' objects through the DataCite API, they will probably retrieve an order of magnitude more objects that are individual files (within datasets that have their own DOIs) that are labeled as 'Dataset' (e.g., every time I look for any published datasets affiliated with UT Austin, 95% of the records I get are individual file DOIs in the Texas Data Repository with their own DOIs that are labeled as 'Dataset' in resourceTypeGeneral, and I have to remove those in the end anyway). If there was a controlled vocabulary in resourceType or a subsidiary field that could be filtered, or files with DOIs had a different label like 'Image' or 'File' in resourceTypeGeneral, this would be less of a problem.

A very long time ago, Dryad used to mint DOIs for individual files by adding /* to the end of the dataset, where * was a sequential integer, and this is how they did the metadata (example dataset). It's internally consistent, but I don't know that anyone else uses that specific label or the field in the same way:

"types": {

            "ris": "DATA",
            "bibtex": "misc",
            "citeproc": "dataset",
            "schemaOrg": "Dataset",
            "resourceType": "DataFile",
            "resourceTypeGeneral": "Dataset"
}

FWIW, some of Figshare's problems with labels in resourceTypeGeneral are because of how millions of their deposits are automated in part or in full, and I don't recall them minting DOIs for individual files within a dataset (although their automated processes sometimes create deposits that are split one set of files into one-file-per-deposit/DOI, so it sort of emulates file-level DOIs).

[KellyStathis]

Thanks @pdurbin @jggautier and @bryanmgee for exploring this with us! Appreciate all of this feedback.

[pdurbin]

I agree with @jggautier, I think whatever is entered in "Data Type" is entered into the resourceType field

You two are right. I just hacked on this area of the Dataverse code for an unmerged pull request and added this table and note to the Dataverse guides:

Again, this is unmerged, a work in progress, but I hope it helps once it's merged. I looked in the Dataverse guides for any description of this "Data Type" (kindOfData) behavior with regard to DataCite, but couldn't find any, which is why I added something. 😅

[KellyStathis]

Circling back to this thread @philippconzett @jggautier: we've been discussing this in the Metadata Working Group and have explored a few different approaches, including adding a new resourceTypeGeneral for individual files/smaller components of a research object. We haven't reached consensus on adding "File" yet. A key issue is that there's overlap between "File" and the other resourceTypeGeneral values:

1. A dataset could consistent of one file - which resourceTypeGeneral should be used then, File or Dataset?
2. A individual file in a dataset could be an image file, or software file, or text file - should this be File or Image/Software/Text?

In general, we're trying to reduce this sort of overlap where a user would have to pick between two equally descriptive terms.

As far as recommendations, the Metadata WG thinks the relatedIdentifier metadata is critical here. Specifically, linking datasets to files with relatedIdentifiers using relationType="HasPart", and vice versa to link files to datasets with relatedIdentifiers using relationType="IsPartOf". I think Dataverse software already includes these relationships, which is great, as this can help harvesters in making sense of whole/part relationships among DOIs from the same repository.

Separately, DataCite's product team (myself and @codycooperross) are working on how we might better support harvesters in excluding parts from requests to get lists of DOIs—for example, through a REST API filter parameter that excludes DOIs with an "IsPartOf" link to another DOI within the same repository (or an incoming "HasPart" link from a DOI within the same repository). This is exploratory and we welcome your thoughts on this approach!

[jggautier]

Thanks @KellyStathis!

I think Dataverse software already includes these relationships, which is great, as this can help harvesters in making sense of whole/part relationships among DOIs from the same repository.

I think that's what a team at Elsevier wound up doing for their Data Monitor product, when they needed to filter out information about files and show to users only information about datasets from Harvard Dataverse. Although it took a while for them to figure out that they could use the relationType info to do it.

Separately, DataCite's product team (myself and @codycooperross) are working on how we might better support harvesters in excluding parts from requests to get lists of DOIs

Awesome! Would https://metadata.datacite.org be considered a harvester?

[KellyStathis]

Makes sense—I'm glad that approach worked for Data Monitor!

That's a good question—the metadata dashboard at https://metadata.datacite.org is more of an evaluation tool to review your DataCite metadata quality. For a Dataverse repository registering file-level DOIs, you could use the "Filter By Query" function to exclude file-level DOIs from the analysis. E.g. for Harvard Dataverse (gdcc.harvard-dv) enter NOT relatedIdentifiers.relationType:IsPartOf in the "Filter by Query" search: https://metadata.datacite.org/gdcc.harvard-dv?query=NOT+relatedIdentifiers.relationType%3AIsPartOf

[philippconzett]

Thanks, @KellyStathis! Using relatedIdentifier metadata the way you described makes sense. Hopefully, this could be implemented by services such as DataCite Commons and ORCID.

[pdurbin]

I just made a pull request for Dataverse to send resourceTypeGeneral=Other for files so this discussion is on my mind again! 😄 We see sending "Other" as an intermediate step until we all come to a consensus in this DataCite thread.

As to your questions:

1. A dataset could consistent of one file - which resourceTypeGeneral should be used then, File or Dataset?

This isn't possible in Dataverse. There can't be a file unless there's a dataset to contain it. Also, in Dataverse, a dataset (which is a container) can have no files. We sometimes refer to this as a metadata record.

2. A individual file in a dataset could be an image file, or software file, or text file - should this be File or Image/Software/Text?

Yes, this is exactly what I was trying to get at in my comment above. Once Dataverse starts sending resourceTypeGeneral=Other for all files (once the pull request above has been merged), a further refinement could be to send an appropriate value like "Image" or "Software" or "Text" when we can.

Can we add "File" to the list of acceptable values for resourceTypeGeneral? @KellyStathis in an earlier comment you suggested both "File" and "Component". I think I like "File" a bit more than "Component". Unless someone has a term they like better! 😅

If we had "File" as an option for resourceTypeGeneral, I would use it instead of "Other". And then I think I would probably want to send something more specific like "Image" or "Software" or "Text" if the file type is known, if that's what you all want! 😄

(I'm trying to keep the focus on resourceTypeGeneral, specifically.)

[vaidasmo]

To my mind, a file is not a separate resource. It is part of a dataset (and without other parts of the dataset it cannot function and is useless). It should, therefore, be described as part of the resource, which it belongs to.

[artinthetrees]

• "File" seems like a good addition to the list of resourceTypeGeneral values, to be applied to files that are part of a larger "dataset" resource

• @pdurbin et al., This may introduce undesired complexity and/or may have already been considered and rejected, but would there be any appetite for taking a more nested approach to specifying resource type, especially in the case of a file that is part of a larger "dataset" resource type? The motivation I am thinking of would be to help people to find/filter for what they are looking for, and also potentially to help guide documenters by making it clearer what values to provide for resourceTypeGeneral and sub-properties..

  • In this approach:
    • any file that is part of a larger dataset on Dataverse would be given a resourceTypeGeneral of "File" even if more information is known about the file
    • there might be a new sub property of resourceTypeGeneral called resourceTypeGeneralFileCategory, required or suggested if resourceTypeGeneral is File, to give a general idea of what kind of information this file contains with respect to the larger research process, with suggested values of: "publication","result","tabular-data","non-tabular-data","metadata","code"
    • there might also be additional new sub properties of resourceTypeGeneral as follows:
      • resourceTypeGeneralFileDataSubcategory, required or suggested if resourceTypeGeneralFileCategory is tabular-data or non-tabular data, with suggested values of: "raw","processed-intermediate","processed-final"
      • resourceTypeGeneralFileMetadataSubcategory, required or suggested if resourceTypeGeneralFileCategory is metadata, with suggested values of: "xxx-formatted-data-dictionary","other-formatted-data-dictionary","protocol","analysis-plan","other" (where xxx could be many accepted/standardized data dictionary formats - e.g. DDI)
      • resourceTypeGeneralFilePublicationSubcategory, required or suggested if resourceTypeGeneralFileCategory is publication, with suggested values of: "peer-review-manuscript","report","white-paper","presentation","poster"
      • etc. (e.g. there could be a subcategory property for non-tabular data with suggested values of: image, interview-transcript, etc.)

• In this approach, many of the current values allowed for resourceTypeGeneral would be folded into these file category or subcategory sub properties of resourceTypeGeneral

• Suggestion inspired by the approach to categorizing study file resources in the HEAL data packaging framework - see properties category, categorySubMetadata, categorySubData, categorySubResult, categorySubPublication in the resource json schema here

[pdurbin]

@artinthetrees sure, sound fancy and fine. In the short term, I'm hoping to be told that a new resourceTypeGeneral value of "File" is now available (or whatever equivalent value, since I'm not picky). But once "File" is available, I'm fine with always using "File" even if more information is known about the file. No objection.

[jggautier]

@artinthetrees your approach reminds me of this Resource Type vocabulary that this group called the Confederation of Open Access Repositories put together. They describe it at https://vocabularies.coar-repositories.org/resource_types and it's hierarchical like the nested approach you described.

@pedroprincipe and his colleagues at OpenAIRE have mentioned it to us at Dataverse a few times as something they've been interested in using for their guidelines for describing data, which take a lot of cues from DataCite's guidelines.

[artinthetrees]

@jggautier the Resource Type vocabulary from the Confederation of Open Access Repositories looks very interesting - are discussions with @pedroprincipe and colleagues at OpenAIRE ongoing?

[jggautier]

Heya @artinthetrees. I'm not sure about the state of discussions. I just remember @pedroprincipe mentioning it during a presentation he gave at one of the Dataverse Community Meetings and thought I'd bring it up here 🙂