feat(provider-tck): emit a machine-readable conformance report

[aepfli]

Stacked on #1830 (feat/provider-tck). Part of open-feature/spec#424; the envelope this emits is defined by the schema in open-feature/spec#425.

This has been reworked. The earlier revision defined its own per-scenario results format: a list of scenarios, each with a four-value outcome and the Examples row it came from. That was the wrong call, and the review that said so was right. A results format has to be maintained, versioned and reimplemented in four languages, and everything it carried is already specified by Cucumber Messages. The schema in #425 has been reshaped accordingly and so has this.

Set PROVIDER_TCK_REPORT_DIR and each suite now writes two files:

File	What it is
<configuration>.json	the envelope — what was tested, and what the provider claims
<configuration>.ndjson	the results — a Cucumber Messages stream

The envelope's results.location names the stream and results.digest covers it. Unset means no report, and that is not an error.

Real output

Both flagd resolvers, on this branch, against the flagd testbed:

$ PROVIDER_TCK_REPORT_DIR=./reports mvn -pl providers/flagd test -Dtest='Flagd*TckTest'
...
Tests run: 58, Failures: 0, Errors: 0, Skipped: 2

$ ls reports/
flagd-in-process.json  flagd-in-process.ndjson  flagd-rpc.json  flagd-rpc.ndjson

{
  "schemaVersion" : "1",
  "provider" : { "name" : "flagd", "language" : "java", "configuration" : "flagd-rpc" },
  "sdk" : { "name" : "dev.openfeature:sdk", "version" : "1.22.0" },
  "tck" : {
    "implementation" : "java-sdk-contrib/tools/provider-tck",
    "version" : "0.0.1",
    "specRevision" : "dfa16586d91ca020ef1b3b82a7c972d833ff8f29"
  },
  "backend" : {
    "description" : "Docker Compose stack docker-compose.yaml, service backend",
    "controlApi" : "http"
  },
  "declaration" : {
    "declared" : [ "@lifecycle", "@events", "@stale", "@configuration-change", "@object", "@unavailable", "@targeting", "@caching" ]
  },
  "results" : {
    "format" : "cucumber-messages",
    "location" : "flagd-rpc.ndjson",
    "digest" : "sha256:c390f2bd578fe323d0c2e54fdfce1606df09b1258ed65d74d95f8b3436a9bbd5"
  },
  "knownDeviations" : [ {
    "capability" : "@strict-numeric-typing",
    "summary" : "Evaluating float-flag (0.5) through the integer API returns 0 with no error code, rather than TYPE_MISMATCH with the code default: the value is silently narrowed. Both resolvers behave identically, which places the defect in the shared provider layer rather than in either transport."
  } ]
}

Reading the outcomes out of the stream, which needs one thing understood — a scenario's outcome is the most severe result among its steps, hooks included, because testCaseFinished carries no status of its own:

$ jq -c 'select(.testStepFinished) | .testStepFinished
         | {c: .testCaseStartedId, s: .testStepResult.status}' reports/flagd-rpc.ndjson \
    | jq -s 'group_by(.c) | map({s: (map(.s) | if any(. == "FAILED") then "FAILED"
                                              elif any(. == "SKIPPED") then "SKIPPED"
                                              else "PASSED" end)})
             | group_by(.s) | map({(.[0].s): length}) | add'
{
  "PASSED": 28,
  "SKIPPED": 1
}

Identical for flagd-in-process. 29 pickles, 29 testCase, 29 testCaseStarted, 29 testCaseFinished, nothing started twice, nothing left unexecuted. The one skip in each is A float flag is not silently narrowed to an integer, and it is the aborted @Before hook that makes it a skip: its step result is SKIPPED and carries the gate's own message, Skipped: provider does not declare capability STRICT_NUMERIC_TYPING (tag @strict-numeric-typing). That is the rule Appendix F cares about, and it is truthful in the stream.

What the stream carries, and what it replaced

Scenario Outline row identity. pickle.astNodeIds is [scenario id, table row id], and the second entry resolves in the gherkinDocument message to the Examples row the scenario was compiled from. For the eleven rows of Requesting the wrong type returns the code default in errors.feature:

$ jq -c 'select(.pickle) | .pickle
         | select(.name == "Requesting the wrong type returns the code default")
         | {id, row: .astNodeIds[1]}' reports/flagd-rpc.ndjson
{"id":"6c8debd2-...","row":"ab8b4a4b-..."}   # -> ["string-flag","Boolean","false"]
{"id":"a7c76b0a-...","row":"63d6d6c8-..."}   # -> ["string-flag","Integer","1"]
{"id":"fecd333d-...","row":"bbd7f5ee-..."}   # -> ["string-flag","Float","0.1"]
{"id":"1a999e43-...","row":"0f899f28-..."}   # -> ["wrong-flag","Boolean","false"]
{"id":"32a4239b-...","row":"f88ca988-..."}   # -> ["boolean-flag","String","fallback"]
{"id":"f5b60fb7-...","row":"0597f6a3-..."}   # -> ["boolean-flag","Integer","1"]
{"id":"1bdf7aa2-...","row":"adee07be-..."}   # -> ["boolean-flag","Float","0.1"]
{"id":"392a66f7-...","row":"3409ce6f-..."}   # -> ["integer-flag","Boolean","false"]
{"id":"c9efc3ab-...","row":"adba2cac-..."}   # -> ["integer-flag","String","fallback"]
{"id":"94c6c1a0-...","row":"48bbee15-..."}   # -> ["float-flag","Boolean","false"]
{"id":"6acf3b06-...","row":"ef0378ff-..."}   # -> ["float-flag","String","fallback"]

Eleven distinct row ids for eleven scenarios sharing one name. ScenarioExamples and the report's example field are deleted: they re-parsed the feature source Cucumber publishes on TestSourceRead and matched a pickle's reported line number, via TestCase.getLocation(), back against the Examples tables — an approximation, reverse-engineered from CucumberQuery.getLocationBy, of exactly the mechanism the format already provides. That is the clearest argument for adopting a standard format rather than defining one, so it is worth saying rather than quietly dropping.

Also deleted: the per-scenario list, the per-capability rollup, the four-value Outcome enum, and tck.assetsTree — the stream carries the source of every feature that executed, which is strictly better than a tree hash asserting which revision it came from. specRevision stays, because it identifies the two artifacts the stream does not carry, flags/canonical-flags.json and openapi/control-api.yaml.

Tags, including per-Examples-block tags, are on pickle.tags with the AST node each came from. Gherkin allows a tag on an individual Examples block, so two rows of one outline can differ in whether the capability gate stops them; the self-test fixture has exactly that shape and asserts that only the tagged row is skipped.

Cucumber's own formatter, at a path the run chooses

The stream is produced by io.cucumber.core.plugin.MessageFormatter — the same class the built-in message:<path> plugin instantiates — so the bytes are what --plugin message:... would have written. ConformanceReportPlugin registers it against the same publisher and writes only the envelope.

The built-in plugin is not used directly for one reason: a @ConfigurationParameter value is a compile-time constant, so cucumber.plugin=message:<path> cannot have a path derived from PROVIDER_TCK_REPORT_DIR, and flagd's two suites in one module would write to the same file. Delegating gets the standard bytes without giving up per-suite naming or the zero-configuration adoption.

Two consequences worth naming. The stream is buffered in memory and written at the end, because the file name comes from the provider configuration, which the suite only reports once its runtime has started — after the first messages have been emitted. It is 330 KB here. And the envelope handler is registered after the formatter's, because Cucumber invokes handlers for one event type in registration order and the formatter closes its writer on the run-finished message; going second is what guarantees the digest covers a complete file.

What stays OpenFeature-specific, and why

The envelope is not a summary of the results. Every field in it answers a question no results format answers, because a Messages stream cannot say what it was a test of:

• provider — what the provider calls itself through its own metadata, not the suite name. The suite name reads well in a failure message (flagd-rpc), which makes it the configuration; one provider with two materially different modes produces two reports that are not interchangeable.
• sdk — read from the classpath rather than declared, because the TCK depends on an SDK version range so that adopting it can never force an upgrade. What a consumer actually ran against is only knowable at runtime.
• tck — which implementation asked the questions, and which revision of the artifacts.
• declaration — the capability set the provider claims. This is the load-bearing one, and it is an input to reading the results rather than a summary of them, which is why it cannot be derived from the stream. The stream says a scenario was skipped; only the declaration says whether that is because the provider declines the capability it needed. Given the declaration and a scenario's tags — both present — the reason for each skip follows, so it does not have to be transported per scenario, which is what let the whole per-scenario list go.

knownDeviations is the one thing neither the stream nor the declaration can express. Withholding a capability reads identically whether it describes a limitation or works around a bug, and the TCK cannot tell the two apart from the outside. So ProviderTckHarness.knownDeviations() lets the provider author say, and flagd says it: @strict-numeric-typing is withheld because flagd narrows a float to an integer with no error code, in both resolvers, which places the defect in the shared provider layer rather than in either transport.

Recorded as untracked, not omitted. There is no issue for this gap — it was found by this suite and has not been filed — and the schema makes issue optional for that case. Silence was the alternative, and it is worse: a consumer would read flagd declining @strict-numeric-typing exactly as it reads a provider with no streaming transport declining @configuration-change, and one of those is a decision while the other is a bug. KnownDeviation.tracked(...) takes the issue link once there is one.

Testing the property rather than the serialisation

ConformanceReportPluginTest no longer asserts over a report this code wrote. It runs a fixture suite through the real Cucumber engine on the JUnit Platform, with the real plugin registered, and reads the emitted stream back the way a consumer would. That change is the point: "a gated scenario is never reported as passed" is now a property of what Cucumber emits, and only a real run can demonstrate it. Asserting over hand-built messages would only have checked Cucumber's serialiser.

The fixture is shaped like the suite rather than minimal — a capability tag on the feature, one on a scenario, one on a single Examples block, and an outline whose rows share a name — because those are the shapes the properties depend on. Fourteen tests: every pickle executed exactly once, outcome counts, both gated scenarios SKIPPED and neither PASSED, the gate's reason present, Examples-block tags reaching the right row only, eleven distinct row ids resolving to the right cells, the executed source byte-identical to the file, the digest matching the stream, and the envelope carrying what the schema requires and nothing it forbids.

The capability gate moved to CapabilityGate.requireDeclared so that the gate producing the skip and the test proving the skip survives are looking at the same code. Inlined in the step definitions, the self-test could only have shown that some abort becomes a skip.

Verification

• mvn -pl tools/provider-tck verify — green: 14 tests, 0 Checkstyle violations, PMD clean, SpotBugs BugInstance size is 0, spotless clean.
• mvn -pl providers/flagd test -Dtest='Flagd*TckTest' with PROVIDER_TCK_REPORT_DIR set — 58 scenarios, 0 failures, 2 skipped. Four files written, one pair per resolver, no collision.
• Both envelopes valid against the reshaped schema from chore(deps): update actions/cache digest to 704facf #425 with a Draft 2020-12 validator (python jsonschema 4.10.3).
• results.digest matches sha256sum of the corresponding .ndjson for both.
• Both streams valid against the published Cucumber Messages schema, jsonschema/messages.schema.json from cucumber/messages@main — 744 messages each, zero invalid.
• Stream accounting per resolver: 29 pickles / 29 testCase / 29 testCaseStarted / 29 testCaseFinished, no pickle executed twice or never, 28 PASSED + 1 SKIPPED, and the one scenario carrying an undeclared capability tag is the SKIPPED one.
• Both jq recipes in the README were run against the real output and produce what they claim.

Run on JDK 21 with Docker; the flagd suites need a Docker daemon, so they are not part of the module's own test run.

How this interacts with the rest of the stack

• feat(provider-tck): in-process control path and in-memory/multi-provider self-tests #1837 — the conflict has grown, not shrunk, and it is worth being precise: 4 files / 6 hunks / 124 conflicted lines before these changes, 6 files / 8 hunks / 163 after. Deleting a lot of code did not help because the deletions are in files feat(provider-tck): in-process control path and in-memory/multi-provider self-tests #1837 does not touch — ConformanceReport, ConformanceReportPlugin, ScenarioExamples, Outcome — while the additions landed in three of the four files it does. ProviderTckHarness grew from 21 to 41 conflicted lines (knownDeviations() sits next to capabilities(), which feat(provider-tck): in-process control path and in-memory/multi-provider self-tests #1837 also moves) and TckRuntime from 76 to 90 (recordRun). Two new files conflict: steps/ProviderSteps (one line, the gate call) and AbstractFlagdTckTest (two lines). All of it is mechanical. pom.xml and README.md still merge cleanly despite both branches editing them. The controlApi() follow-up described before still applies: it should become a default method on BackendControl returning empty, overridden by HttpBackendControl with http and InProcessBackendControl with in-process.
• feat(provider-tck): source the spec artifacts from the open-feature/spec submodule #1838 — introduces the spec submodule. The one remaining pinned POM property should then be generated from git rev-parse HEAD in the same step that copies the artifacts. provider-tck.spec.assets-tree is gone, so there is one fewer thing to generate.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}