Repository navigation
Update to latest version of DwC-DP - #47
Conversation
| templateMap.put("publication_date_time", DATE_TIME_FORMATTER.format(now)); | ||
| templateMap.put("number_of_source_systems", numberOfSourceSystem); | ||
| templateMap.put("package_id", UUID.randomUUID().toString()); | ||
| templateMap.put("export_download_link", |
There was a problem hiding this comment.
Is it typical to have the download link in the eml? that means we're committing to storing these files forever.
Do you check first if this dwc-dp/dwca exists? like what we have for the doi download? that could reduce storage load.
There was a problem hiding this comment.
ok i see this is just moving existing logic to an abstract class, point still stands but it's probably out of scope.
There was a problem hiding this comment.
Yes just took over the logic to a new shared DwC class as only DwC-A and DwC-DP need this logic. As we are not sure if the data has changed between the run of two exports we do not first check if it already exists. We do now keep the DwC-DP forever. Maybe at some point we get a policy that we only store them for a couple of years or something.
There was a problem hiding this comment.
formatting is weird here too
There was a problem hiding this comment.
Reformatted file
|
fyi you can "accept" these issues on sonar. Maybe you can even disable this rule for this repo? |
|
2 participants
Update to the latest version of the DwC-DP and add support for non-source-system jobs.
The latest version of DwC-DP: https://github.com/gbif/dwc-dp/tree/master/dwc-dp/table-schemas
As the schema's are not yet final and no central repository is hosting the schema's yet the recommendation was to include the whole schema description in the data-package.json. This resulted in a high number of lines changes.
Generated DwC-DP have been validated against the frictionless data validation tool: https://pypi.org/project/frictionless/
First touch with new formatter, tried to limit the impact to files I actually thouched.
Sonar
We don't comply with the Sonar rules where there should be no underscores in the names (that is all the issues it indicates). However, if we want to comply it means we need an additional mapping in the CsV header strategy to map all the Java names to the DwC-DP names, whcih is also ugly. I am inclined to accept the Sonar issues and keep the names as is. I will indicate the issues also with John as I don't like the mixing conventions.
Duplication happened because there is duplication of fields in the domain object which just represents the DwC-DP classes. There is no logic duplication (or not above 3.0% anyway).
AI help
AI helped identify differences and update the domain models.
AI helped update the tests to align with the latest changes.