diff --git a/.rubocop_todo.yml b/.rubocop_todo.yml index 2f11606..481f375 100644 --- a/.rubocop_todo.yml +++ b/.rubocop_todo.yml @@ -377,6 +377,7 @@ RSpec/MultipleExpectations: - 'spec/dcc/extract/file_spec.rb' - 'spec/dcc/i18n/text_lookup_spec.rb' - 'spec/dcc/inspect/report_spec.rb' + - 'spec/dcc/migrate_spec.rb' - 'spec/dcc/namespace_spec.rb' - 'spec/dcc/plugin_spec.rb' - 'spec/dcc/quantity_format/formatter_spec.rb' @@ -398,6 +399,7 @@ RSpec/ExampleLength: - 'spec/dcc/cli/formatters_spec.rb' - 'spec/dcc/convert/additional_spec.rb' - 'spec/dcc/extract/file_spec.rb' + - 'spec/dcc/migrate_spec.rb' - 'spec/dcc/type_spec.rb' - 'spec/dcc/validate/business_rules_spec.rb' - 'spec/dcc/validate_spec.rb' diff --git a/CHANGELOG.adoc b/CHANGELOG.adoc index 5bb5df7..fdbbfac 100644 --- a/CHANGELOG.adoc +++ b/CHANGELOG.adoc @@ -78,7 +78,10 @@ and this project adheres to https://semver.org/spec/v2.0.0.html[Semantic Version * No `require "dcc/..."` with internal paths. * No `send` / `instance_variable_set` / `instance_variable_get`. * No `respond_to?` for type checking. -* No Nokogiri outside `lib/dcc/validate/xsd.rb`. +* No Nokogiri outside `lib/dcc/validate/xsd.rb`. The v2-to-v3 migration later + added `lib/dcc/migrate/v2_to_v3.rb` as a second sanctioned file, because a + version transform has to rewrite documents the object model cannot + round-trip. To be removed once the parser round-trips without loss. * No doubles in specs. * `frozen_string_literal: true` everywhere. * OCP/MECE/DRY throughout. diff --git a/CONTRIBUTING.adoc b/CONTRIBUTING.adoc index 9b26507..f09bc31 100644 --- a/CONTRIBUTING.adoc +++ b/CONTRIBUTING.adoc @@ -34,7 +34,14 @@ constraints: * `send` to call private methods — redesign the API boundary. * `instance_variable_set` / `instance_variable_get` — use accessors. * `respond_to?` for type checking — use `is_a?` or model the type hierarchy. -* Nokogiri anywhere outside `lib/dcc/validate/xsd.rb`. +* Nokogiri anywhere outside `lib/dcc/validate/xsd.rb` and + `lib/dcc/migrate/v2_to_v3.rb`. ++ +The migration exception exists because a version transform rewrites element +names and moves subtrees on documents the object model cannot round-trip — +doing it through the model would discard the very constructs the transform +has to carry. Revisit and remove this exception when the parser round-trips +a DCC without loss, at which point the transform can run on the model. * Doubles in specs — use real model instances. * Hand-rolled `to_h`/`from_h` on model classes — use lutaml-model mappings. diff --git a/TODO.complete/28-migration.md b/TODO.complete/28-migration.md index 99229b1..24448b3 100644 --- a/TODO.complete/28-migration.md +++ b/TODO.complete/28-migration.md @@ -3,11 +3,14 @@ **Status:** PARTIAL ## Gaps -- `lib/dcc/migrate/` does not exist: `route.rb`, `v2_to_v3.rb` and - `v3_to_v3_4.rb` are all absent. -- `lib/dcc/migrate.rb` is a 41-line shim that serializes, re-parses under the - target context and rewrites `schemaVersion`. There are no per-field - transformations and no D-SI v1 → v2 transition. +- `lib/dcc/migrate/v3_to_v3_4.rb` is absent, and there is no migration to + v3.4.0-rc.2. That version is not bundled — `Schema::Version::DCC_ALL` stops + at `3.3.0` and `resolve_dcc("3.4.0-rc.2")` raises `UnknownVersionError`. +- Only `2.3.0 → 3.3.0` has a real transform. `Route.supported_pairs` also + lists `3.2.1 → 3.3.0`, which only rewrites `schemaVersion`; every other + pair raises. DCC 2.1.0 and 2.1.1 import `dsi/v1.0.1.xsd`, whose + targetNamespace is `https://intranet.ptb.de/...`, and nothing rewrites it, + so those versions are refused rather than migrated. ## Goal `Dcc.migrate(dcc, from:, to:)` upgrades or downgrades a DCC between schema versions, applying the necessary field renames and D-SI version transitions. diff --git a/docs/schema-versions.adoc b/docs/schema-versions.adoc index 49a4120..7f84769 100644 --- a/docs/schema-versions.adoc +++ b/docs/schema-versions.adoc @@ -51,15 +51,28 @@ Dcc::Schema::Version.resolve_dcc(:auto, xml: xml) # => "3.3.0" == Migration -Use `Dcc.migrate` to upgrade or downgrade between schema versions: +Use `Dcc.migrate` to upgrade a DCC to a newer schema version: [source,ruby] ---- -dcc = Dcc.parse(File.read("v3.xml")) -migrated = Dcc.migrate(dcc, from: "3.3.0", to: "3.2.1") -puts migrated.schema_version # => 3.2.1 +xml = File.read("v2.xml") +migrated = Dcc.migrate(xml, from: "2.3.0", to: "3.3.0") +puts migrated.schema_version # => 3.3.0 ---- -Cross-major migration (e.g. v2 → v3) serializes the source DCC and -re-parses it under the target version's context. Same-major just -rewrites the `schemaVersion` attribute in place. +Pass the source XML whenever you have it. `Dcc.migrate` also accepts a +parsed DCC, but a model has to be serialized back to XML before the +transform runs — so anything the parser could not round-trip is already +gone, and the migration can neither carry it nor report it. + +Migration is one-way. Downgrading raises `Dcc::UnsupportedMigrationError`, +because v3 adds elements and attributes no older schema accepts: + +[source,ruby] +---- +Dcc.migrate(xml, from: "3.3.0", to: "3.2.1") # raises +---- + +Cross-major migration (e.g. v2 → v3) applies the route's field rules to the +XML and parses the result under the target version's context. Same-major +just rewrites the `schemaVersion` attribute. diff --git a/lib/dcc.rb b/lib/dcc.rb index 7e73266..818c3b2 100644 --- a/lib/dcc.rb +++ b/lib/dcc.rb @@ -112,12 +112,25 @@ def plugin_path(name) end private :plugin_path - # Migrate a parsed DCC object from one schema version to another. - # @param dcc [Dcc::V2::DigitalCalibrationCertificate, Dcc::V3::DigitalCalibrationCertificate] + # Migrate a DCC from one schema version to another. + # + # Prefer passing the source XML. A parsed model has to be serialized back + # to XML before the transform runs, so anything the model could not + # round-trip is already gone and cannot be migrated or reported. + # + # @param input [String, IO, Dcc::V2::DigitalCalibrationCertificate, + # Dcc::V3::DigitalCalibrationCertificate] DCC source XML, or a parsed DCC. # @param from [String] source version, e.g. "2.3.0". # @param to [String] target version, e.g. "3.3.0". - def migrate(dcc, from:, to:) - Migrate.call(dcc, from: from, to: to) + # @param on_loss [#call, nil] callable receiving each message describing a + # construct the migration changed or discarded (a String per loss). + # When given, the messages go to the callable and nothing is sent to + # `Kernel.warn`; when nil (the default), each loss report is warned to + # stderr as before. + # @return [Dcc::V2::DigitalCalibrationCertificate, + # Dcc::V3::DigitalCalibrationCertificate] always a parsed DCC. + def migrate(input, from:, to:, on_loss: nil) + Migrate.call(input, from: from, to: to, on_loss: on_loss) end # Return the parser module for the given major version. diff --git a/lib/dcc/base/condition.rb b/lib/dcc/base/condition.rb index 00a088c..d2bcde1 100644 --- a/lib/dcc/base/condition.rb +++ b/lib/dcc/base/condition.rb @@ -3,7 +3,10 @@ module Dcc module Base # `dcc:conditionType` — single influence condition: name + description + - # optional state enum (beforeAdjustment, etc.) + data. + # optional status enum (beforeAdjustment, etc.) + data. + # + # v3 calls that child `status`; v2 calls it `state`, which + # `Dcc::V2::Condition` maps. module Condition def self.included(klass) klass.class_eval do @@ -24,7 +27,7 @@ def self.included(klass) map_attribute "refType", to: :ref_type map_element "name", to: :name map_element "description", to: :description - map_element "state", to: :status + map_element "status", to: :status map_element "data", to: :data end end diff --git a/lib/dcc/base/core_data.rb b/lib/dcc/base/core_data.rb index 14bc737..8fe673a 100644 --- a/lib/dcc/base/core_data.rb +++ b/lib/dcc/base/core_data.rb @@ -20,6 +20,7 @@ def self.included(klass) attribute :receipt_date, :date attribute :begin_performance_date, :date attribute :end_performance_date, :date + attribute :performance_location, :string attribute :previous_report, :previousReport xml do @@ -38,6 +39,7 @@ def self.included(klass) map_element "receiptDate", to: :receipt_date map_element "beginPerformanceDate", to: :begin_performance_date map_element "endPerformanceDate", to: :end_performance_date + map_element "performanceLocation", to: :performance_location map_element "previousReport", to: :previous_report end end diff --git a/lib/dcc/base/identification.rb b/lib/dcc/base/identification.rb index 79f937d..5fc43d2 100644 --- a/lib/dcc/base/identification.rb +++ b/lib/dcc/base/identification.rb @@ -3,14 +3,17 @@ module Dcc module Base # `dcc:identificationType` — single identifier (issuer, value, optional - # description). Issuer enum: manufacturer, calibrationLaboratory, customer, + # name). Issuer enum: manufacturer, calibrationLaboratory, customer, # owner, other. + # + # v3 calls the third child `name`; v2 calls it `description`, which + # `Dcc::V2::Identification` maps. module Identification def self.included(klass) klass.class_eval do attribute :issuer, :string attribute :value, :string - attribute :description, :text + attribute :name, :text xml do namespace ::Dcc::Namespace::Dcc @@ -18,7 +21,7 @@ def self.included(klass) ordered map_element "issuer", to: :issuer map_element "value", to: :value - map_element "description", to: :description + map_element "name", to: :name end end end diff --git a/lib/dcc/error.rb b/lib/dcc/error.rb index f897975..a1db037 100644 --- a/lib/dcc/error.rb +++ b/lib/dcc/error.rb @@ -37,6 +37,10 @@ class TransformError < Error; end # Raised when an unsupported schema version is requested. class UnknownVersionError < Error; end + # Raised when a migration direction is not supported — any downgrade, and + # any pair whose incompatibilities have not been verified. + class UnsupportedMigrationError < Error; end + # Raised when the builder DSL is used incorrectly. class BuilderError < Error; end diff --git a/lib/dcc/migrate.rb b/lib/dcc/migrate.rb index 120679c..04d448f 100644 --- a/lib/dcc/migrate.rb +++ b/lib/dcc/migrate.rb @@ -1,40 +1,144 @@ # frozen_string_literal: true -# `Dcc::Migrate` upgrades or downgrades a parsed DCC between schema versions, -# applying the necessary field renames and D-SI version transitions. +# `Dcc::Migrate` upgrades a DCC to a newer schema version. # -# Currently a thin shim: serializes the source DCC and re-parses under -# the target version's context. Future work will add per-field transformations. +# Migration is a document transform: the route's field rules are applied to +# the XML and the result is parsed under the target version. The input is +# never modified. +# +# Pass the source XML when you have it. Passing a parsed model instead costs +# whatever that model could not round-trip, because the model has to be +# serialised back to XML before the transform can see it — the transform +# cannot report a construct the parse already dropped. +# +# Migrations are one-way. A downgrade cannot preserve the source — v3 adds +# elements and attributes that no older schema accepts — so downgrading warns +# and then raises rather than returning a quietly invalid document. module Dcc module Migrate + autoload :Route, "dcc/migrate/route" + autoload :V2ToV3, "dcc/migrate/v2_to_v3" + class << self - # @param dcc [Lutaml::Model::Serializable] - # @param from [String] source version (e.g. "2.3.0"). - # @param to [String] target version (e.g. "3.3.0"). - # @return [Dcc::V2::DigitalCalibrationCertificate, Dcc::V3::DigitalCalibrationCertificate] - def call(dcc, from:, to:) - from_major = ::Dcc::Schema::Version.major(from) - to_major = ::Dcc::Schema::Version.major(to) - if from_major == to_major - # Same major: just rewrite the schemaVersion attribute. - return rewrite_schema_version(dcc, to) - end + # @param input [String, IO, Lutaml::Model::Serializable] DCC source XML, + # or a parsed DCC. Prefer the source XML: see the note on this module. + # @param from [String] source version, e.g. "2.3.0". + # @param to [String] target version, e.g. "3.3.0". + # @param on_loss [#call, nil] callable receiving each loss message (a + # String) the transform would otherwise send to `Kernel.warn`; when + # given, nothing is warned. Nil keeps the stderr report. + # @raise [Dcc::UnknownVersionError] if either version is not bundled. + # @raise [Dcc::UnsupportedMigrationError] on a downgrade, or on a pair + # whose incompatibilities have not been verified. + # @return [Dcc::V2::DigitalCalibrationCertificate, + # Dcc::V3::DigitalCalibrationCertificate] always a parsed DCC. + def call(input, from:, to:, on_loss: nil) + source = ::Dcc::Schema::Version.resolve_dcc(from) + target = ::Dcc::Schema::Version.resolve_dcc(to) + return same_version_result(input, target) if source == target - # Cross-major migration: serialize then re-parse under target. - xml = dcc.to_xml - target_parser = ::Dcc.parser_for(to_major) - target_parser.load_all! - migrated = target_parser.parse(xml) - rewrite_schema_version(migrated, to) + reject_downgrade(source, target) + reject_unregistered(source, target) + + transform = Route.for(source, target) + xml = source_xml(input) + migrated = if transform + transform.call(xml, to: target, + on_loss: on_loss) + else + xml + end + parse_as(migrated, target) end private - def rewrite_schema_version(dcc, version) - return dcc unless Dcc::TypeGuards.has_writer?(dcc, :schema_version) + # A same-version call must not round-trip a model through XML: that + # would apply the parse loss this transform exists to avoid. The model + # is still checked, so the no-op path cannot hand back a non-DCC object + # the other paths would have refused. + # + # The version is checked too. Every other route re-parses under the + # target, so its result carries the requested version whatever `from:` + # said. This route returns the input untouched, so a mis-declared + # `from:` would hand back a document of some other version and make both + # parameters meaningless. + def same_version_result(input, target) + if dcc_document?(input) + reject_version_mismatch(input, target) + return input + end + + parse_as(source_xml(input), target) + end + + def reject_version_mismatch(dcc, target) + actual = dcc.schema_version + return if actual.nil? || actual == target + + raise ::Dcc::Error, + "Dcc.migrate was asked for #{target} but the document " \ + "declares #{actual}." + end + + # The one place an input becomes XML, and the one place the DCC root is + # checked. + # + # The root check belongs here rather than in the transform: the + # schemaVersion-only and same-version routes never reach a transform, + # and any `Serializable` serialises happily, so both would otherwise + # turn a foreign document into an empty certificate. + def source_xml(input) + V2ToV3.assert_dcc_root(raw_xml(input)) + end + + # `Dcc.read_input` falls through to `to_s` for anything that is not + # IO-like, so the parsed-model case has to be answered first. + def raw_xml(input) + return input.to_xml if input.is_a?(::Lutaml::Model::Serializable) + + reject_untranslatable(input) + ::Dcc.read_input(input) + end + + # The two DCC root models. Checked by class rather than by serialising + # and re-reading, so the same-version no-op stays a no-op. + def dcc_document?(input) + input.is_a?(::Dcc::V2::DigitalCalibrationCertificate) || + input.is_a?(::Dcc::V3::DigitalCalibrationCertificate) + end + + def reject_untranslatable(input) + return if input.is_a?(::String) || ::Dcc.io_like?(input) + + raise ::Dcc::Error, + "Dcc.migrate expects DCC source XML or a parsed DCC document, " \ + "got #{input.class}." + end + + def reject_downgrade(source, target) + return if ::Gem::Version.new(target) > ::Gem::Version.new(source) + + message = "Cannot migrate #{source} to #{target}: downgrading " \ + "discards information the older schema cannot represent." + Kernel.warn(message) + raise ::Dcc::UnsupportedMigrationError, message + end + + def reject_unregistered(source, target) + return if Route.supported?(source, target) + + raise ::Dcc::UnsupportedMigrationError, + "No verified migration from #{source} to #{target}. " \ + "Supported: #{Route.supported_pairs.join(', ')}." + end - dcc.schema_version = version - dcc + def parse_as(xml, version) + parser = ::Dcc.parser_for(::Dcc::Schema::Version.major(version)) + parser.load_all! + parsed = parser.parse(xml) + parsed.schema_version = version + parsed end end end diff --git a/lib/dcc/migrate/route.rb b/lib/dcc/migrate/route.rb new file mode 100644 index 0000000..222b318 --- /dev/null +++ b/lib/dcc/migrate/route.rb @@ -0,0 +1,44 @@ +# frozen_string_literal: true + +module Dcc + module Migrate + # Registry of migrations whose full incompatibility set has been verified + # against the bundled XSDs. + # + # Unregistered pairs raise rather than run. DCC 2.1.0 and 2.1.1 sit in the + # `https://ptb.de/si/smartcom/d-si/v1_0_1` D-SI namespace, which no + # transform here addresses, so accepting them by major version alone would + # silently emit an unmigrated document. + module Route + TRANSFORMS = { %w[2.3.0 3.3.0] => :V2ToV3 }.freeze + + # v3.3.0 is a strict superset of v3.2.1 at instance level; only the + # `schemaVersion` pattern differs. + SCHEMA_VERSION_ONLY = [%w[3.2.1 3.3.0]].freeze + + class << self + # @param from [String] normalized source version, e.g. "2.3.0". + # @param to [String] normalized target version, e.g. "3.3.0". + # @return [Boolean] whether this migration is registered. + def supported?(from, to) + TRANSFORMS.key?([from, to]) || + SCHEMA_VERSION_ONLY.include?([from, to]) + end + + # @return [Module, nil] a module responding to + # `.call(xml, to:, on_loss:)`, or nil when only `schemaVersion` + # needs rewriting. + def for(from, to) + name = TRANSFORMS[[from, to]] + name && ::Dcc::Migrate.const_get(name) + end + + # @return [Array] every registered pair, for error messages. + def supported_pairs + (TRANSFORMS.keys + SCHEMA_VERSION_ONLY) + .map { |from, to| "#{from} to #{to}" } + end + end + end + end +end diff --git a/lib/dcc/migrate/v2_to_v3.rb b/lib/dcc/migrate/v2_to_v3.rb new file mode 100644 index 0000000..8f07542 --- /dev/null +++ b/lib/dcc/migrate/v2_to_v3.rb @@ -0,0 +1,445 @@ +# frozen_string_literal: true + +require "nokogiri" + +module Dcc + module Migrate + # Rewrites a DCC v2.x document into the v3.x wire format. + # + # v3 renamed several elements, renamed the `hashType` children on all three + # v2 elements carrying that type, made `coreData/performanceLocation` and + # `measurementResult/name` mandatory, moved the statement `refId` from a + # child element to an attribute, narrowed four `description` elements to + # single occurrence, tightened many string types to reject surrounding + # whitespace, and switched from D-SI v1 to D-SI v2. + # + # Two v2 constructs have no v3 home — `formula/@lang` and the second and + # later `calibrationLaboratory/contact`. Those, the element ids lost to a + # description merge, and every trimmed value are reported rather than + # dropped in silence. + # + # Every rule is conditional, so applying the transform twice is a no-op. + module V2ToV3 + NS = { + "dcc" => "https://ptb.de/dcc", + "si" => "https://ptb.de/si", + }.freeze + + ROOT_ELEMENT = "digitalCalibrationCertificate" + + # Only the canonical URI. `Dcc::Namespace::Dcc` also accepts the + # `https://ptb.de/dcc.xsd` alias, but every rule below binds `dcc:` to + # the canonical URI, so an alias document would match no xpath: it would + # come back unmigrated, reporting nothing, and fail the v3 schema. + # Refusing it says so instead of returning a quietly broken document. + ROOT_NAMESPACE = NS["dcc"] + + # `performanceLocationType` is an enum and v2 records no location, so + # the catch-all is the only truthful value. + DEFAULT_PERFORMANCE_LOCATION = "other" + DEFAULT_MEASUREMENT_RESULT_NAME = "Measurement result" + + # `byteData`, `descriptionData` and `document` are all typed + # `byteDataType`, so all three carry the `data` child v3 renamed. + BYTE_DATA_DATA_XPATH = [ + "//dcc:byteData/dcc:data", + "//dcc:descriptionData/dcc:data", + "//dcc:document/dcc:data", + ].join(" | ").freeze + + # The three elements v2.3.0 types `dcc:hashType`. v3 renamed that type's + # `reference`/`referenceID` children, so all three need the rename, not + # `previousReport` alone. `linkedReport` is itself a `hashType` child, so + # the `//` anchor covers nesting at any depth. + # + # A fourth `identifier` appears in the file but sits inside the + # commented-out `metrologicallyTraceableType` block (v2.3.0 + # dcc.xsd:325-341), so no valid v2 document can carry it. + # + # The anchor is what keeps this safe: v2 also declares `reference` under + # `equipmentClass` and `statement`, and v3 keeps BOTH of those named + # `reference`. An unanchored `//dcc:reference` would corrupt them. + HASH_TYPE_PARENTS = %w[previousReport certificate linkedReport].freeze + + HASH_TYPE_REFERENCE_XPATH = HASH_TYPE_PARENTS + .map { |parent| "//dcc:#{parent}/dcc:reference" }.join(" | ").freeze + HASH_TYPE_REFERENCE_ID_XPATH = HASH_TYPE_PARENTS + .map { |parent| "//dcc:#{parent}/dcc:referenceID" }.join(" | ").freeze + + RENAMES = { + "//dcc:identification/dcc:description" => "name", + "//dcc:influenceCondition/dcc:state" => "status", + HASH_TYPE_REFERENCE_XPATH => "referral", + HASH_TYPE_REFERENCE_ID_XPATH => "referralID", + BYTE_DATA_DATA_XPATH => "dataBase64", + "//dcc:formula/dcc:siunitx" => "latex", + }.freeze + + # v2 let these `description` elements repeat; v3 allows one. + MERGED_DESCRIPTION_PARENTS = %w[ + software item measuringEquipment usedMethod + ].freeze + + # Elements retyped to `notEmptyStringType`, which rejects leading and + # trailing whitespace as well as empty values. + TRIMMED_ELEMENTS = %w[ + release model uniqueIdentifier value classID reference norm + convention calibrationLaboratoryCode city postCode postOfficeBox + state street streetNo eMail phone fax referralID procedure + fileName mimeType + ].freeze + + # Children a v2.2.1 `si:list` accepts directly; anything else in a v1 + # list body has to move into a wrapper. + LIST_CONTAINER_CHILDREN = %w[ + label dateTime realList realListXMLList complexList + complexListXMLList list + ].freeze + + class << self + # Every route in `Dcc::Migrate` funnels raw input through here, not + # just this transform: this file is the only one sanctioned to parse + # XML directly, so the DCC-root check has to live with it. + # + # @param xml [String] candidate DCC XML. + # @return [String] the same XML, when the root is a DCC root. + # @raise [Dcc::ParseError] otherwise. + def assert_dcc_root(xml) + reject_rootless(parse(xml)) + xml + end + + # @param xml [String] DCC v2 XML. + # @param to [String] target v3 version for `schemaVersion`. + # @param on_loss [#call, nil] receives each loss message (a String) + # instead of the stderr report; nil keeps the `Kernel.warn` report. + # @return [String] DCC v3 XML. + # @raise [Dcc::ParseError] on malformed XML or a non-DCC root. + def call(xml, to:, on_loss: nil) + doc = parse(xml) + reject_rootless(doc) + + losses = [] + doc.root["schemaVersion"] = to + apply_rules(doc, losses) + report(losses, to, on_loss) + + doc.to_xml + end + + private + + # Strict, so a truncated or malformed document is refused rather than + # silently recovered into a partial one. Nokogiri's default recovery + # mode would drop everything after the first syntax error, and the + # migration would report no loss for the sections it never saw. + def parse(xml) + ::Nokogiri::XML(xml, &:strict) + rescue ::Nokogiri::XML::SyntaxError => e + raise ::Dcc::ParseError, + "Dcc::Migrate::V2ToV3 expects well-formed DCC XML: #{e.message}" + end + + # Checking the name and namespace, not merely that a root exists: the + # target parser builds an empty certificate out of a foreign root + # rather than refusing it, so a stray document would otherwise migrate + # to a valid-looking but empty v3 result. + def reject_rootless(doc) + root = doc.root + return if root && root.name == ROOT_ELEMENT && + root.namespace&.href == ROOT_NAMESPACE + + raise ::Dcc::ParseError, + "Dcc.migrate expects a root " \ + "in the #{ROOT_NAMESPACE} namespace, got " \ + "#{describe_root(root)}." + end + + def describe_root(root) + return "input with no root element" unless root + + "<#{root.name}> in #{root.namespace&.href.inspect}" + end + + def apply_rules(doc, losses) + rename_elements(doc) + move_ref_id_to_attribute(doc) + add_performance_location(doc) + add_measurement_result_names(doc) + merge_descriptions(doc, losses) + rehome_description_data(doc) + trim_not_empty_strings(doc, losses) + upgrade_dsi(doc) + collect_losses(doc, losses) + end + + def rename_elements(doc) + RENAMES.each do |xpath, name| + doc.xpath(xpath, NS).each { |node| node.name = name } + end + end + + def move_ref_id_to_attribute(doc) + doc.xpath("//dcc:statement/dcc:refId | //dcc:metaData/dcc:refId", NS) + .each do |node| + node.parent["refId"] = node.text.strip + node.remove + end + end + + def add_performance_location(doc) + doc.xpath("//dcc:coreData", NS).each do |core_data| + next if core_data.at_xpath("dcc:performanceLocation", NS) + + node = element(doc, "dcc:performanceLocation") + node.content = DEFAULT_PERFORMANCE_LOCATION + performance_anchor(core_data).add_next_sibling(node) + end + end + + # `endPerformanceDate` is mandatory in v2, so its absence means the + # input is not the v2 document this transform was handed. + def performance_anchor(core_data) + core_data.at_xpath("dcc:endPerformanceDate", NS) || + raise(::Dcc::ParseError, + "coreData has no endPerformanceDate, so the mandatory v3 " \ + "performanceLocation cannot be placed.") + end + + def add_measurement_result_names(doc) + doc.xpath("//dcc:measurementResult", NS).each do |result| + next if result.at_xpath("dcc:name", NS) + + name = result_name(doc) + anchor = result.children.first + anchor ? anchor.add_previous_sibling(name) : result.add_child(name) + end + end + + def result_name(doc) + name = element(doc, "dcc:name") + content = element(doc, "dcc:content") + content.content = DEFAULT_MEASUREMENT_RESULT_NAME + name.add_child(content) + name + end + + # v2 `textType` carries an optional `@id` and real documents use it. + # Merging removes the ids of descriptions 2..N, so every reference to + # one is repointed at the survivor before the element goes away. + def merge_descriptions(doc, losses) + remap = {} + MERGED_DESCRIPTION_PARENTS.each do |parent| + doc.xpath("//dcc:#{parent}", NS) + .each { |node| merge_parent_descriptions(node, remap, losses) } + end + repoint_references(doc, remap, losses) + end + + def merge_parent_descriptions(node, remap, losses) + first, *rest = node.xpath("dcc:description", NS).to_a + rest.each do |extra| + extra.element_children.each { |child| first.add_child(child) } + absorb_id(first, extra, remap, losses) + extra.remove + end + end + + # The survivor ends up holding the removed description's content, so a + # reference to the removed id still means the same thing. Adopt the id + # outright when the survivor has none. + def absorb_id(first, extra, remap, losses) + id = extra["id"] + return unless id + + first["id"] ||= id + return if first["id"] == id + + remap[id] = first["id"] + losses << "description/@id=#{id} merged into #{first['id']}" + end + + # Without this a `refId` would be left pointing at an id that no + # longer exists anywhere in the document. + def repoint_references(doc, remap, losses) + return if remap.empty? + + doc.xpath("//*[@refId]").each do |node| + before = node["refId"].split + after = before.map { |token| remap.fetch(token, token) } + next if after == before + + node["refId"] = after.join(" ") + losses << "#{node.name}/@refId repointed to #{after.join(' ')} " \ + "(merged descriptions)" + end + end + + # v3 dropped `descriptionData`; `richContentType/file` carries the + # same `byteDataType` payload. + def rehome_description_data(doc) + doc.xpath( + "//dcc:item/dcc:descriptionData | " \ + "//dcc:measuringEquipment/dcc:descriptionData", NS + ).each do |node| + parent = node.parent + node.name = "file" + description(doc, parent).add_child(node) + end + end + + def description(doc, parent) + parent.at_xpath("dcc:description", NS) || + element(doc, "dcc:description").tap do |node| + parent.at_xpath("dcc:name", NS).add_next_sibling(node) + end + end + + # `notEmptyStringType` rejects surrounding whitespace, so trimming is + # required for validity — but it changes a value the author wrote, + # which this transform reports rather than doing silently. + def trim_not_empty_strings(doc, losses) + TRIMMED_ELEMENTS.each do |name| + doc.xpath("//dcc:#{name}", NS) + .each { |node| trim_node(node, name, losses) } + end + end + + def trim_node(node, name, losses) + return unless node.element_children.empty? + + original = node.text + trimmed = original.strip + reject_empty(name, original) if trimmed.empty? + return if trimmed == original + + node.content = trimmed + losses << "#{name} whitespace trimmed to #{trimmed.inspect} " \ + "(v3 notEmptyStringType rejects it)" + end + + # v2 typed these `xs:string` and accepted an empty or whitespace-only + # value; v3's `notEmptyStringType` rejects it and there is nothing + # truthful to put in its place. + def reject_empty(name, original) + raise ::Dcc::UnsupportedMigrationError, + "Cannot migrate #{original}: v3 " \ + "types it notEmptyStringType, which rejects an empty or " \ + "whitespace-only value, and no substitute can be invented." + end + + def upgrade_dsi(doc) + wrap_uncertainty(doc, "expandedUnc", "expandedMU", + "uncertainty" => "valueExpandedMU") + wrap_uncertainty(doc, "coverageInterval", "coverageIntervalMU", + "standardUnc" => "valueStandardMU") + wrap_list_bodies(doc) + normalize_probabilities(doc) + end + + # Only reals OUTSIDE an `si:list`. `realInListType` misspells the + # modern wrapper as `measurementUncertaintyUnivaraite` and accepts the + # deprecated forms as siblings, so rewriting a list-local uncertainty + # to the correct spelling would turn a valid document invalid. Both + # deprecated forms stay legal in v2.2.1, so leaving them costs + # nothing. `dcc:list` is a different element and does not match. + def wrap_uncertainty(doc, legacy_name, modern_name, renames) + doc.xpath("//si:real[not(ancestor::si:list)]/si:#{legacy_name}", NS) + .each do |legacy| + univariate = element(doc, "si:measurementUncertaintyUnivariate") + univariate.add_child( + modern_uncertainty(doc, legacy, modern_name, renames), + ) + legacy.replace(univariate) + end + end + + def modern_uncertainty(doc, legacy, modern_name, renames) + element(doc, "si:#{modern_name}").tap do |modern| + legacy.element_children.each do |child| + child.name = renames.fetch(child.name, child.name) + modern.add_child(child) + end + end + end + + # v2.2.1 `si:list` is a pure container; a v1 body moves into one + # `si:realList` or `si:complexList`. + def wrap_list_bodies(doc) + doc.xpath("//si:list", NS).each do |list| + body = list.element_children.reject do |child| + LIST_CONTAINER_CHILDREN.include?(child.name) + end + next if body.empty? + + wrapper = element(doc, "si:#{list_wrapper_name(body)}") + body.each { |child| wrapper.add_child(child) } + list.add_child(wrapper) + end + end + + def list_wrapper_name(body) + if body.any? { |child| child.name == "complex" } + "complexList" + else + "realList" + end + end + + # D-SI v1 accepted `.95`; v2.2.1 requires a leading zero. + def normalize_probabilities(doc) + doc.xpath("//si:coverageProbability", NS).each do |node| + text = node.text.strip + node.content = "0#{text}" if text.start_with?(".") + end + end + + def collect_losses(doc, losses) + collect_formula_lang(doc, losses) + collect_extra_lab_contacts(doc, losses) + end + + def collect_formula_lang(doc, losses) + doc.xpath("//dcc:formula[@lang]", NS).each do |node| + losses << "formula/@lang=#{node['lang']} " \ + "(v3 formulae carry no language)" + node.remove_attribute("lang") + end + end + + def collect_extra_lab_contacts(doc, losses) + doc.xpath("//dcc:calibrationLaboratory", NS).each do |lab| + extra = lab.xpath("dcc:contact", NS).to_a.drop(1) + next if extra.empty? + + extra.each(&:remove) + losses << "#{extra.size} extra calibrationLaboratory/contact " \ + "(v3 allows one)" + end + end + + # With an `on_loss` callable the caller owns the loss list: every + # message goes to the callable, one call per loss, and nothing is + # warned. Without one, the report goes to stderr as before. + def report(losses, to, on_loss) + return if losses.empty? + + if on_loss + losses.each { |loss| on_loss.call(loss) } + return + end + + Kernel.warn( + "Dcc.migrate 2.3.0 to #{to} changed or discarded " \ + "#{losses.size} construct(s) v3 cannot represent as written: " \ + "#{losses.join('; ')}", + ) + end + + def element(doc, name) + ::Nokogiri::XML::Node.new(name, doc) + end + end + end + end +end diff --git a/lib/dcc/transform.rb b/lib/dcc/transform.rb index 7e5ccc3..57d745a 100644 --- a/lib/dcc/transform.rb +++ b/lib/dcc/transform.rb @@ -1,9 +1,10 @@ # frozen_string_literal: true # `Dcc::Transform` provides XSLT transformations. We cannot use Nokogiri -# for this per project rule (only validation may touch Nokogiri); instead -# we shell out to the system `xsltproc` CLI for XSLT 1.0, or to -# `java -jar saxon-he.jar` for XSLT 2.0/3.0 when SAXON_JAR is set. +# for this per project rule (only XSD validation and the v2-to-v3 migration +# may touch Nokogiri); instead we shell out to the system `xsltproc` CLI for +# XSLT 1.0, or to `java -jar saxon-he.jar` for XSLT 2.0/3.0 when SAXON_JAR +# is set. module Dcc module Transform autoload :Xslt, "dcc/transform/xslt" diff --git a/lib/dcc/transform/xslt.rb b/lib/dcc/transform/xslt.rb index 25f2142..d771174 100644 --- a/lib/dcc/transform/xslt.rb +++ b/lib/dcc/transform/xslt.rb @@ -6,9 +6,9 @@ module Dcc module Transform # `Dcc::Transform::Xslt` applies an XSLT stylesheet to a DCC XML document. # - # Nokogiri is OFFICIALLY banned outside validation, so we shell out to - # the system `xsltproc` (libxslt) for XSLT 1.0 transforms. For XSLT - # 2.0/3.0, set `ENV["SAXON_JAR"]` to point at saxon-he.jar. + # Nokogiri is banned outside XSD validation and the v2-to-v3 migration, + # so we shell out to the system `xsltproc` (libxslt) for XSLT 1.0 + # transforms. For XSLT 2.0/3.0, set `ENV["SAXON_JAR"]` to saxon-he.jar. module Xslt class << self # @param xml [String, Lutaml::Model::Serializable] diff --git a/lib/dcc/v2/condition.rb b/lib/dcc/v2/condition.rb index 425b382..0a5a6be 100644 --- a/lib/dcc/v2/condition.rb +++ b/lib/dcc/v2/condition.rb @@ -1,8 +1,29 @@ # frozen_string_literal: true module Dcc::V2 + # v2 calls the influence condition's state element `state`; v3 renamed it to + # `status`, which the shared base module carries. class Condition < CommonElements - include ::Dcc::Base::Condition + attribute :id, :string + attribute :ref_id, :string + attribute :ref_type, :string + attribute :name, :text + attribute :description, :text + attribute :status, :string + attribute :data, :data, collection: true + + xml do + namespace ::Dcc::Namespace::Dcc + element "influenceCondition" + ordered + map_attribute "id", to: :id + map_attribute "refId", to: :ref_id + map_attribute "refType", to: :ref_type + map_element "name", to: :name + map_element "description", to: :description + map_element "state", to: :status + map_element "data", to: :data + end end Configuration.register_model(Condition, id: :influenceCondition) end diff --git a/lib/dcc/v2/identification.rb b/lib/dcc/v2/identification.rb index acfa4c3..41f22fb 100644 --- a/lib/dcc/v2/identification.rb +++ b/lib/dcc/v2/identification.rb @@ -1,8 +1,21 @@ # frozen_string_literal: true module Dcc::V2 + # v2 calls the identification's third child `description`; v3 renamed it to + # `name`, which the shared base module carries. class Identification < CommonElements - include ::Dcc::Base::Identification + attribute :issuer, :string + attribute :value, :string + attribute :description, :text + + xml do + namespace ::Dcc::Namespace::Dcc + element "identification" + ordered + map_element "issuer", to: :issuer + map_element "value", to: :value + map_element "description", to: :description + end end Configuration.register_model(Identification, id: :identification) end diff --git a/spec/dcc/migrate_spec.rb b/spec/dcc/migrate_spec.rb index 7ff0799..1eeaad9 100644 --- a/spec/dcc/migrate_spec.rb +++ b/spec/dcc/migrate_spec.rb @@ -1,19 +1,816 @@ # frozen_string_literal: true require "spec_helper" +require "nokogiri" +require "stringio" RSpec.describe Dcc::Migrate do before { Dcc.load_all! } - it "rewrites schemaVersion within the same major" do - dcc = Dcc.parse(File.read(fixtures_path("dcclib", "valid.xml"))) - migrated = described_class.call(dcc, from: "3.3.0", to: "3.2.1") - expect(migrated.schema_version.to_s).to eq("3.2.1") + # The migration reports losses on stderr. Specs that assert on the migrated + # document rather than the report keep the suite output readable. + def silence_stderr + original = $stderr + $stderr = StringIO.new + yield + ensure + $stderr = original end - it "raises UnknownVersionError for unsupported target" do - dcc = Dcc.parse(File.read(fixtures_path("dcclib", "valid.xml"))) - expect { described_class.call(dcc, from: "3.3.0", to: "9.9.9") } - .to raise_error(Dcc::UnknownVersionError, /Unsupported DCC version/) + let(:namespaces) do + { "dcc" => "https://ptb.de/dcc", "si" => "https://ptb.de/si" } + end + + let(:v3_dcc) { Dcc.parse(File.read(fixtures_path("dcclib", "valid.xml"))) } + + describe "version handling" do + it "returns the same object untouched when from and to are equal" do + expect(described_class.call(v3_dcc, from: "3.3.0", to: "3.3.0")) + .to equal(v3_dcc) + end + + it "rejects a model whose version is not the one asked for" do + v2_dcc = Dcc.parse(File.read(fixtures_path("migrate", + "v2_full_coverage.xml"))) + + expect { described_class.call(v2_dcc, from: "3.3.0", to: "3.3.0") } + .to raise_error(Dcc::Error, /asked for 3\.3\.0.*declares 2\.3\.0/) + end + + it "rejects a target outside the bundled versions" do + expect { described_class.call(v3_dcc, from: "3.3.0", to: "3.4.0-rc.2") } + .to raise_error(Dcc::UnknownVersionError, /3\.4\.0-rc\.2/) + end + + it "rejects an unknown target major" do + expect { described_class.call(v3_dcc, from: "3.3.0", to: "9.9.9") } + .to raise_error(Dcc::UnknownVersionError) + end + + it "rejects a minor-version downgrade" do + expect { described_class.call(v3_dcc, from: "3.3.0", to: "3.2.1") } + .to raise_error(Dcc::UnsupportedMigrationError, /3\.3\.0.*3\.2\.1/) + end + + it "rejects a major-version downgrade" do + expect { described_class.call(v3_dcc, from: "3.3.0", to: "2.3.0") } + .to raise_error(Dcc::UnsupportedMigrationError, /information/i) + end + + it "warns before raising on a downgrade" do + expect do + described_class.call(v3_dcc, from: "3.3.0", to: "2.3.0") + rescue Dcc::UnsupportedMigrationError + nil + end.to output(/downgrad/i).to_stderr + end + + it "rewrites schemaVersion on a same-major upgrade" do + dcc = Dcc.parse(File.read(fixtures_path("dcclib", "valid.xml"))) + dcc.schema_version = "3.2.1" + migrated = described_class.call(dcc, from: "3.2.1", to: "3.3.0") + + expect(migrated).not_to equal(dcc) + expect(migrated.schema_version.to_s).to eq("3.3.0") + end + end + + describe Dcc::Migrate::Route do + it "routes 2.3.0 to 3.3.0 through the V2ToV3 transform" do + expect(described_class.for("2.3.0", "3.3.0")).to eq(Dcc::Migrate::V2ToV3) + end + + it "needs no field transform for 3.2.1 to 3.3.0" do + expect(described_class.for("3.2.1", "3.3.0")).to be_nil + end + + it "does not know 2.1.0 to 3.3.0" do + expect(described_class.supported?("2.1.0", "3.3.0")).to be(false) + end + + # 12 versions, 132 ordered pairs, no prereleases — the whole space. + it "registers only the pairs whose incompatibilities were verified" do + pairs = Dcc::Schema::Version::DCC_ALL.permutation(2).select do |from, to| + described_class.supported?(from, to) + end + + expect(pairs).to contain_exactly(%w[2.3.0 3.3.0], %w[3.2.1 3.3.0]) + end + end + + describe "unregistered pairs" do + it "raises rather than silently no-opping on 2.1.0 to 3.3.0" do + dcc = Dcc.parse(File.read(fixtures_path("dcc_examples", "example.xml"))) + expect { described_class.call(dcc, from: "2.1.0", to: "3.3.0") } + .to raise_error(Dcc::UnsupportedMigrationError, /2\.1\.0.*3\.3\.0/) + end + + # Every ordered pair that is not one of the two registered ones, in both + # directions — 130 of the 132. Downgrades warn before raising, so the + # whole loop runs inside a stderr capture to keep the output clean. + it "raises for all 130 unregistered ordered pairs" do + dcc = Dcc.parse(File.read(fixtures_path("dcclib", "valid.xml"))) + pairs = Dcc::Schema::Version::DCC_ALL.permutation(2).reject do |from, to| + Dcc::Migrate::Route.supported?(from, to) + end + expect(pairs.size).to eq(130) + + expect do + pairs.each do |from, to| + expect { described_class.call(dcc, from: from, to: to) } + .to raise_error(Dcc::UnsupportedMigrationError), + "expected #{from} -> #{to} to raise" + end + end.to output.to_stderr + end + end + + describe Dcc::Migrate::V2ToV3 do + [ + %w[dcc_examples example.xml], + %w[dcc_examples siliziumkugel.xml], + %w[dcc_xsd example.xml], + %w[dcc_xsd siliziumkugel.xml], + ].each do |dir, file| + it "produces XSD-valid v3 XML from #{dir}/#{file}" do + migrated = described_class.call( + File.read(fixtures_path(dir, file)), to: "3.3.0" + ) + result = Dcc::Validate::Xsd.call(migrated, version: "3.3.0") + + expect(result.errors).to be_empty + expect(result).to be_ok + end + end + + let(:source) { File.read(fixtures_path("dcc_examples", "example.xml")) } + let(:migrated) { described_class.call(source, to: "3.3.0") } + + it "writes the target schemaVersion" do + expect(migrated).to include('schemaVersion="3.3.0"') + end + + it "upgrades every D-SI v1 expandedUnc to expandedMU" do + expect(source.scan("]/).size) + .to eq(source.scan(/]/).size) + expect(migrated.scan(/]/).size) + .to eq(source.scan(/]/).size) + end + + # Checked structurally: a regex with `.*?` under /m runs past the closing + # tag and matches an unrelated description later in the document. + it "renames identification description to name" do + doc = Nokogiri::XML(migrated) + + expect(doc.xpath("//dcc:identification/dcc:description", namespaces)) + .to be_empty + expect(doc.xpath("//dcc:identification/dcc:name", namespaces).size) + .to eq(6) + end + + it "renames influenceCondition state to status" do + expect(source).to include("") + expect(migrated).not_to include("") + expect(migrated).to include("") + end + + it "moves the metaData refId element onto the attribute" do + expect(source).to match(%r{.*?MRA}m) + expect(migrated).to include('refId="MRA"') + expect(migrated).not_to include("") + end + + it "adds the mandatory performanceLocation" do + expect(migrated) + .to include("other") + end + + it "adds the mandatory measurementResult name" do + expect(source).not_to match(%r{\s*}m) + expect(migrated).to match(%r{\s*}m) + end + + it "merges the repeated usedMethod descriptions into one" do + counts = migrated.scan(%r{.*?}m) + .map { |block| block.scan("").size } + + expect(counts.max).to eq(1) + end + + it "is idempotent" do + expect(described_class.call(migrated, to: "3.3.0")).to eq(migrated) + end + end + + describe "the coverage fixture" do + let(:source) { File.read(fixtures_path("migrate", "v2_full_coverage.xml")) } + let(:migrated) { Dcc::Migrate::V2ToV3.call(source, to: "3.3.0") } + + it "is valid v2.3.0 to begin with" do + expect(Dcc::Validate::Xsd.call(source, version: "2.3.0").errors) + .to be_empty + end + + it "migrates to valid v3.3.0" do + expect(Dcc::Validate::Xsd.call(migrated, version: "3.3.0").errors) + .to be_empty + end + + it "renames byteData data to dataBase64 under every byteDataType parent" do + doc = Nokogiri::XML(migrated) + + expect(doc.xpath("//dcc:dataBase64", namespaces).size).to eq(2) + expect(doc.xpath("//dcc:file/dcc:data", namespaces)).to be_empty + end + + it "converts siunitx to latex" do + expect(migrated).to include("") + expect(migrated).not_to include("") + end + + it "rehomes descriptionData as description/file" do + doc = Nokogiri::XML(migrated) + + expect(doc.xpath("//dcc:descriptionData", namespaces)).to be_empty + expect(doc.xpath("//dcc:description/dcc:file", namespaces).size).to eq(2) + end + + it "strips whitespace from notEmptyString elements" do + expect(source).to match(%r{\s+\S}) + expect(migrated).to include("Model X100") + end + + it "upgrades a coverageInterval outside any si:list" do + expect(migrated).to include("") + expect(migrated).to include("") + expect(migrated).not_to include("") + end + + it "normalizes a leading-dot coverage probability" do + expect(source).to include(".95<") + expect(migrated).to include("0.95<") + end + + it "merges repeated descriptions under every narrowed parent" do + doc = Nokogiri::XML(migrated) + + %w[software item measuringEquipment usedMethod].each do |parent| + doc.xpath("//dcc:#{parent}", namespaces).each do |node| + expect(node.xpath("dcc:description", namespaces).size).to eq(1), + parent + end + end + end + + it "reports the description ids lost to the merge" do + expect { migrated }.to output(%r{description/@id=}).to_stderr + end + + it "reports the trimmed whitespace" do + expect { migrated }.to output(/whitespace trimmed/).to_stderr + end + + it "reports the constructs v3 cannot represent" do + expect { migrated } + .to output(%r{formula/@lang.*calibrationLaboratory}m).to_stderr + end + + it "is idempotent" do + expect(Dcc::Migrate::V2ToV3.call(migrated, to: "3.3.0")).to eq(migrated) + end + + context "when migrated through Dcc.migrate" do + let(:result) do + Dcc.migrate(Dcc.parse(source), from: "2.3.0", to: "3.3.0").to_xml + end + + it "still validates against the v3.3.0 XSD" do + expect(Dcc::Validate::Xsd.call(result, version: "3.3.0").errors) + .to be_empty + end + + it "keeps the upgraded non-list uncertainty" do + expect(result).to include("") + expect(result).to include("") + end + end + end + + # Transform level only. `Dcc::Si::Base::SiList` has no mapping for a v1 + # `si:list`'s direct `si:real` children, so this document cannot survive + # `Dcc.parse`. Asserting it through `Dcc.migrate` would fail for a reason + # that has nothing to do with migration. + describe "a v1-shaped si:list" do + let(:source) { File.read(fixtures_path("migrate", "v2_si_list.xml")) } + let(:migrated) { Dcc::Migrate::V2ToV3.call(source, to: "3.3.0") } + + it "is valid v2.3.0 to begin with" do + expect(Dcc::Validate::Xsd.call(source, version: "2.3.0").errors) + .to be_empty + end + + it "migrates to valid v3.3.0" do + expect(Dcc::Validate::Xsd.call(migrated, version: "3.3.0").errors) + .to be_empty + end + + it "renames the previousReport reference elements" do + expect(migrated).to include("", "") + expect(migrated).not_to include("", "") + end + + it "wraps the v1 list body in si:realList" do + doc = Nokogiri::XML(migrated) + + expect(doc.xpath("//si:list/si:realList/si:real", namespaces).size) + .to eq(2) + expect(doc.xpath("//si:list/si:real", namespaces)).to be_empty + end + + it "leaves list-local deprecated uncertainty alone" do + list = migrated[%r{.*?}m] + + expect(list).to include("", "") + expect(list).not_to include("") + end + + it "is idempotent" do + expect(Dcc::Migrate::V2ToV3.call(migrated, to: "3.3.0")).to eq(migrated) + end + end + + describe "version-specific element names" do + it "keeps the v3 identification name through a round trip" do + dcc = Dcc.parse(File.read(fixtures_path("dcclib", "valid.xml"))) + identification = + dcc.administrative_data.core_data.identifications.identification.first + + expect(identification.name.content.map(&:value).flatten) + .to include("Order no.") + expect(Nokogiri::XML(dcc.to_xml) + .xpath("//dcc:identification/dcc:name", namespaces)) + .not_to be_empty + end + + # example.xml carries its identifications under items and statements, not + # coreData, so this asserts on the serialized shape rather than a path. + it "keeps the v2 identification description through a round trip" do + dcc = Dcc.parse(File.read(fixtures_path("dcc_examples", "example.xml"))) + doc = Nokogiri::XML(dcc.to_xml) + + expect(doc.xpath("//dcc:identification/dcc:description", namespaces).size) + .to eq(6) + expect(doc.xpath("//dcc:identification/dcc:name", namespaces)).to be_empty + end + + it "keeps performanceLocation through a v3 round trip" do + dcc = Dcc.parse(File.read(fixtures_path("dcclib", "valid.xml"))) + + expect(dcc.administrative_data.core_data.performance_location) + .not_to be_nil + expect(dcc.to_xml).to include("") + end + end + + describe "documents the migration must not silently break" do + let(:source) { File.read(fixtures_path("migrate", "v2_full_coverage.xml")) } + + # A statement whose refId element points at the description the merge + # removes. v2 allows this; leaving it alone emits a refId that resolves + # to nothing. + let(:with_reference) do + statements = " \n " \ + "\n " \ + "ISO 17025\n " \ + "item-desc-2\n " \ + "\n " \ + "\n " \ + "" + source.sub(" ") { statements } + end + + let(:whitespace_only) do + source.sub(" Model X100 ") do + " " + end + end + + it "starts from valid v2 documents in both cases" do + expect(Dcc::Validate::Xsd.call(with_reference, version: "2.3.0").errors) + .to be_empty + expect(Dcc::Validate::Xsd.call(whitespace_only, version: "2.3.0").errors) + .to be_empty + end + + it "repoints a reference to a merged-away description" do + doc = Nokogiri::XML( + Dcc::Migrate::V2ToV3.call(with_reference, to: "3.3.0"), + ) + ids = doc.xpath("//*[@id]").map { |node| node["id"] } + refs = doc.xpath("//*[@refId]").flat_map { |node| node["refId"].split } + + expect(refs).to include("item-desc-1") + expect(refs - ids).to be_empty + end + + it "reports the repointing" do + expect { Dcc::Migrate::V2ToV3.call(with_reference, to: "3.3.0") } + .to output(/repointed to item-desc-1/).to_stderr + end + + it "still validates after repointing" do + migrated = Dcc::Migrate::V2ToV3.call(with_reference, to: "3.3.0") + + expect(Dcc::Validate::Xsd.call(migrated, version: "3.3.0").errors) + .to be_empty + end + + it "refuses a value that trims to empty" do + expect { Dcc::Migrate::V2ToV3.call(whitespace_only, to: "3.3.0") } + .to raise_error(Dcc::UnsupportedMigrationError, /notEmptyStringType/) + end + end + + # v2.3.0 types THREE live elements `dcc:hashType` — certificate, + # previousReport and linkedReport — and v3 renamed that type's + # reference/referenceID children on all of them. A fourth `identifier` + # appears in the file but sits inside the commented-out + # metrologicallyTraceableType block, so no valid v2 document carries it. + describe "hashType children on every parent carrying the type" do + let(:hash_type_source) do + doc = Nokogiri::XML( + File.read(fixtures_path("migrate", "v2_full_coverage.xml")), + ) + equipment = doc.at_xpath("//dcc:measuringEquipment", namespaces) + # v2 measuringEquipmentType orders these name, equipmentClass, + # description, descriptionData, certificate. + equipment.at_xpath("dcc:name", namespaces) + .add_next_sibling(Nokogiri::XML(<<~CLASS).root) + + DKD-R-6-1 + 3.2 + + CLASS + # `previousReport` is the last child of coreData. The fixture omits it, + # so without this the "every hashType parent" claim goes untested for + # one of the three. + doc.at_xpath("//dcc:coreData", namespaces) + .add_child(Nokogiri::XML(<<~PREV).root) + + Prior + PREV-1 + SHA256xyz + + PREV + # `statements` is the last child of administrativeData, and + # statementMetaDataType also carries a `reference` v3 keeps as-is. + doc.at_xpath("//dcc:administrativeData", namespaces) + .add_child(Nokogiri::XML(<<~STMT).root) + + ISO 17025 + + STMT + anchor = equipment.at_xpath("dcc:descriptionData", namespaces) || + equipment.at_xpath("dcc:description", namespaces) + anchor.add_next_sibling(Nokogiri::XML(<<~XML).root) + + Cert doc + CERT-1 + SHA256 + abc123 + + Linked + LINK-1 + SHA256 + def456 + + + XML + doc.to_xml + end + + let(:hash_type_migrated) do + silence_stderr { Dcc::Migrate::V2ToV3.call(hash_type_source, to: "3.3.0") } + end + + it "starts from a valid v2.3.0 document" do + expect(Dcc::Validate::Xsd.call(hash_type_source, version: "2.3.0").errors) + .to be_empty + end + + it "migrates a certificate to a valid v3 document" do + errors = Dcc::Validate::Xsd + .call(hash_type_migrated, version: "3.3.0").errors + + expect(errors).to be_empty + end + + it "renames reference and referenceID under certificate" do + certificate = Nokogiri::XML(hash_type_migrated) + .at_xpath("//dcc:certificate", namespaces) + + expect(certificate.at_xpath("dcc:referral", namespaces).text) + .to eq("Cert doc") + expect(certificate.at_xpath("dcc:referralID", namespaces).text) + .to eq("CERT-1") + end + + it "renames reference and referenceID under previousReport" do + previous = Nokogiri::XML(hash_type_migrated) + .at_xpath("//dcc:previousReport", namespaces) + + expect(previous.at_xpath("dcc:referralID", namespaces).text) + .to eq("PREV-1") + expect(previous.at_xpath("dcc:reference", namespaces)).to be_nil + end + + it "renames them inside a nested linkedReport" do + linked = Nokogiri::XML(hash_type_migrated) + .at_xpath("//dcc:certificate/dcc:linkedReport", namespaces) + + expect(linked.at_xpath("dcc:referralID", namespaces).text).to eq("LINK-1") + expect(linked.at_xpath("dcc:reference", namespaces)).to be_nil + end + + # Spelled out rather than iterating HASH_TYPE_PARENTS: a spec driven by + # the constant it is testing cannot catch that constant losing an entry. + it "leaves no v2 reference or referenceID under any hashType parent" do + doc = Nokogiri::XML(hash_type_migrated) + + %w[previousReport certificate linkedReport].each do |parent| + expect(doc.xpath("//dcc:#{parent}/dcc:reference", namespaces)) + .to be_empty + expect(doc.xpath("//dcc:#{parent}/dcc:referenceID", namespaces)) + .to be_empty + end + end + + it "covers exactly the three v2 elements typed dcc:hashType" do + expect(Dcc::Migrate::V2ToV3::HASH_TYPE_PARENTS) + .to contain_exactly("previousReport", "certificate", "linkedReport") + end + + # v3 keeps equipmentClass/reference and statement/reference named + # `reference`, so the rename must stay anchored on the hashType parents. + it "does not rename reference under equipmentClass or statement" do + doc = Nokogiri::XML(hash_type_migrated) + + %w[equipmentClass statement].each do |parent| + expect(doc.xpath("//dcc:#{parent}/dcc:reference", namespaces)) + .not_to(be_empty, "expected #{parent}/reference to survive") + expect(doc.xpath("//dcc:#{parent}/dcc:referral", namespaces)) + .to be_empty + end + end + end + + # `Dcc.migrate` transforms the source XML, so a construct the v2 model + # cannot round-trip still reaches the transform. descriptionData is the + # worked case: the v2 model drops it, and the transform rewrites it to + # description/file, which the v3 model does preserve. + describe "migrating source XML rather than a re-serialized model" do + let(:coverage_xml) do + File.read(fixtures_path("migrate", "v2_full_coverage.xml")) + end + + def attachment_census(xml) + doc = Nokogiri::XML(xml) + { + file: doc.xpath("//dcc:description/dcc:file", namespaces).size, + data: doc.xpath("//dcc:dataBase64", namespaces).size, + } + end + + it "carries descriptionData attachments through to the returned model" do + migrated = silence_stderr do + Dcc.migrate(coverage_xml, from: "2.3.0", to: "3.3.0") + end + + expect(attachment_census(migrated.to_xml)).to eq(file: 2, data: 2) + end + + it "loses them when handed a parsed model instead" do + migrated = silence_stderr do + Dcc.migrate(Dcc.parse(coverage_xml), from: "2.3.0", to: "3.3.0") + end + + expect(attachment_census(migrated.to_xml)).to eq(file: 0, data: 0) + end + + it "reports every loss on the source-XML path" do + expect { Dcc.migrate(coverage_xml, from: "2.3.0", to: "3.3.0") } + .to output(include("discarded 7 construct")).to_stderr + end + + it "always returns a parsed model, never the source string" do + migrated = silence_stderr do + Dcc.migrate(coverage_xml, from: "2.3.0", to: "2.3.0") + end + + expect(migrated).to be_a(Dcc::V2::DigitalCalibrationCertificate) + end + end + + # `on_loss:` hands the loss list to the caller. Without it every loss is + # reported through Kernel.warn, which a program cannot read back. + describe "on_loss:" do + let(:coverage_xml) do + File.read(fixtures_path("migrate", "v2_full_coverage.xml")) + end + + it "collects each loss message and warns nothing" do + losses = [] + + expect do + Dcc.migrate(coverage_xml, from: "2.3.0", to: "3.3.0", + on_loss: ->(message) { losses << message }) + end.not_to output.to_stderr + + expect(losses.size).to eq(7) + expect(losses).to all(be_a(String)) + expect(losses).to include(a_string_matching(%r{formula/@lang})) + .and include(a_string_matching(/whitespace trimmed/)) + end + + it "does not change the return value" do + migrated = Dcc.migrate(coverage_xml, from: "2.3.0", to: "3.3.0", + on_loss: ->(_message) {}) + + expect(migrated).to be_a(Dcc::V3::DigitalCalibrationCertificate) + expect(migrated.schema_version.to_s).to eq("3.3.0") + end + + it "still warns to stderr when omitted" do + expect { Dcc.migrate(coverage_xml, from: "2.3.0", to: "3.3.0") } + .to output(/changed or discarded/).to_stderr + end + + it "is never called when the migration loses nothing" do + dcc = Dcc.parse(File.read(fixtures_path("dcclib", "valid.xml"))) + dcc.schema_version = "3.2.1" + losses = [] + + Dcc.migrate(dcc, from: "3.2.1", to: "3.3.0", + on_loss: ->(message) { losses << message }) + + expect(losses).to be_empty + end + end + + describe "input validation" do + it "rejects nil" do + expect { Dcc.migrate(nil, from: "2.3.0", to: "3.3.0") } + .to raise_error(Dcc::Error, /expects DCC source XML or a parsed DCC/) + end + + it "rejects a type that is neither source XML nor a parsed DCC" do + expect { Dcc.migrate(42, from: "2.3.0", to: "3.3.0") } + .to raise_error(Dcc::Error, /got Integer/) + end + + it "accepts source XML as a string" do + xml = File.read(fixtures_path("migrate", "v2_full_coverage.xml")) + migrated = silence_stderr { Dcc.migrate(xml, from: "2.3.0", to: "3.3.0") } + + expect(migrated).to be_a(Dcc::V3::DigitalCalibrationCertificate) + expect(migrated.schema_version.to_s).to eq("3.3.0") + end + + it "accepts source XML as an IO" do + io = StringIO.new( + File.read(fixtures_path("migrate", "v2_full_coverage.xml")), + ) + migrated = silence_stderr { Dcc.migrate(io, from: "2.3.0", to: "3.3.0") } + + expect(migrated).to be_a(Dcc::V3::DigitalCalibrationCertificate) + end + + it "rejects XML with no root element" do + expect { Dcc::Migrate::V2ToV3.call("", to: "3.3.0") } + .to raise_error(Dcc::ParseError, /well-formed|no root element/) + end + + # Strict parsing: recovery mode would drop everything after the first + # syntax error and migrate the fragment, reporting no loss for the + # sections it never saw. + it "rejects truncated XML instead of migrating the fragment" do + xml = File.read(fixtures_path("migrate", "v2_full_coverage.xml")) + truncated = xml[0, xml.index("", from: "2.3.0", to: "3.3.0") } + .to raise_error(Dcc::ParseError, /digitalCalibrationCertificate/) + end + + # The schemaVersion-only and same-version routes never reach a transform, + # so the DCC-root check cannot live inside one. + it "rejects a foreign root on every route, not just the transform" do + [%w[2.3.0 3.3.0], %w[3.2.1 3.3.0], %w[3.3.0 3.3.0]].each do |from, to| + expect { Dcc.migrate("", from: from, to: to) } + .to raise_error(Dcc::ParseError, /digitalCalibrationCertificate/), + "expected #{from} -> #{to} to reject a foreign root" + end + end + + # Any Serializable serialises happily, so the root check has to cover the + # model path too or a non-DCC model migrates into an empty certificate. + it "rejects a parsed model that is not a DCC document" do + expect { Dcc.migrate(Dcc::V3::Formula.new, from: "3.2.1", to: "3.3.0") } + .to raise_error(Dcc::ParseError, /digitalCalibrationCertificate/) + end + + # The same-version no-op returns its input, so it needs the same check or + # it becomes the one path that hands back a non-DCC object. + it "rejects a non-DCC model on the same-version path too" do + expect { Dcc.migrate(Dcc::V3::Formula.new, from: "3.3.0", to: "3.3.0") } + .to raise_error(Dcc::ParseError, /digitalCalibrationCertificate/) + end + + it "rejects a DCC root in the wrong namespace" do + expect do + Dcc.migrate( + '', + from: "2.3.0", to: "3.3.0", + ) + end.to raise_error(Dcc::ParseError, /namespace/) + end + end + + describe "end to end through Dcc.migrate" do + let(:source) { File.read(fixtures_path("dcc_examples", "example.xml")) } + let(:result) do + Dcc.migrate(Dcc.parse(source), from: "2.3.0", to: "3.3.0").to_xml + end + + def count(xml, tag) + xml.scan(/<#{tag}[ >]/).size + end + + it "carries the uncertainty upgrade all the way to the output" do + expect(count(source, "si:expandedUnc")).to eq(10) + expect(count(result, "si:expandedUnc")).to eq(0) + expect(count(result, "si:expandedMU")).to eq(10) + expect(count(result, "si:valueExpandedMU")).to eq(10) + end + + it "loses no quantity, list or D-SI real" do + %w[dcc:quantity dcc:list si:real].each do |tag| + expect(count(result, tag)).to eq(count(source, tag)), + "#{tag}: #{count(source, tag)} in, " \ + "#{count(result, tag)} out" + end + end + + # example.xml is valid v2.3.0, but a plain parse-and-serialize round trip + # already fails v2.3.0 with 8 "quantity missing child" errors: the model + # types `noQuantity` as :string where the XSD says textType, so all 8 are + # dropped before migration runs. Requiring the migrated output to be clean + # would require the migration to repair a parse loss. What it must + # guarantee is that it adds nothing new — asserted here. Absolute XSD + # validity through the public API is covered by the coverage fixture. + it "introduces no XSD errors beyond the pre-existing parse loss" do + before = Dcc::Validate::Xsd + .call(Dcc.parse(source).to_xml, version: "2.3.0").errors + after = Dcc::Validate::Xsd.call(result, version: "3.3.0").errors + + expect(before.size).to eq(8) + expect(after.size).to eq(before.size) + expect(after.map(&:message)) + .to all(include("quantity", "Missing child element")) + end + + it "carries the renamed identification name and condition status" do + doc = Nokogiri::XML(result) + + expect(doc.xpath("//dcc:identification/dcc:name", namespaces).size) + .to eq(6) + expect(result).to include("") + expect(result).to include("") + end end end diff --git a/spec/fixtures/migrate/v2_full_coverage.xml b/spec/fixtures/migrate/v2_full_coverage.xml new file mode 100644 index 0000000..daf92e1 --- /dev/null +++ b/spec/fixtures/migrate/v2_full_coverage.xml @@ -0,0 +1,164 @@ + + + + + + + + Coverage Tool + v1.0 + + First software description. + + + Zweite Software-Beschreibung. + + + + + + DE + en + en + COV-0001 + 2024-01-01 + 2024-01-02 + + + + + Calibration item + + First item description. + + + Zweite Item-Beschreibung. + + + Item datasheet + Attached datasheet + item.txt + text/plain + SGVsbG8gaXRlbQ== + + + Instrument Maker GmbH + + Model X100 + + + manufacturer + SN-0001 + + Serial number + + + + + + + + + Primary laboratory contact + lab@example.org + Braunschweig + + + Secondary laboratory contact + lab2@example.org + Berlin + + + + + + + Responsible Signer + + + + + + Customer Ltd + customer@example.org + Hamburg + + + + + + + + Weighing method + + First method description. + + + Zweite Methodenbeschreibung. + + + + + + + Reference balance + + First equipment description. + + + Zweite Ausstattungsbeschreibung. + + + Equipment datasheet + Attached datasheet + equipment.txt + text/plain + SGVsbG8gZXF1aXA= + + + + + + + Mass result + + + \SI{1.234}{\kilogram} + + + Mass + + 1.234 + \kilogram + + 0.001 + 1.232 + 1.236 + .95 + + + + + + + + + diff --git a/spec/fixtures/migrate/v2_si_list.xml b/spec/fixtures/migrate/v2_si_list.xml new file mode 100644 index 0000000..5978078 --- /dev/null +++ b/spec/fixtures/migrate/v2_si_list.xml @@ -0,0 +1,118 @@ + + + + + + + + List Tool + v1.0 + + + + + DE + en + en + LIST-0001 + 2024-01-01 + 2024-01-02 + + + Previous calibration report + + PREV-0001 + SHA-256 + 3a7bd3e2360a3d29eea436fcfb7e44c735d117c4 + + + + + + Calibration item + Item under test. + + Instrument Maker GmbH + + + + manufacturer + SN-0002 + + + + + + + + Laboratory contact + lab@example.org + Braunschweig + + + + + + + Responsible Signer + + + + + + Customer Ltd + customer@example.org + Hamburg + + + + + + + + Temperature series + + + Temperature points + + \degreeCelsius + + 21.4 + + 0.2 + 2 + 0.95 + + + + 22.6 + + 0.1 + 22.4 + 22.8 + 0.95 + + + + + + + + + +