feat(provider-tck): emit a conformance report whose results are Cucumber Messages

[aepfli]

Stacked on #409 — base is feat/provider-tck, so the diff here is only the report emitter. Part of open-feature/spec#424; the schema is open-feature/spec#425.

This PR was reworked after review. It originally defined a per-scenario result format in the report schema. It no longer does: the results are carried in Cucumber Messages, the ndjson protocol, and the schema is an envelope that references them. The rest of this description describes what that means and what was deleted.

Setting TCK_REPORT_DIR makes each suite write two files: an envelope at <dir>/<name>.json and the results it points at at <dir>/<name>.ndjson.

$ TCK_REPORT_DIR=./reports uv run pytest tools/openfeature-tck/tests
provider-tck [controllable-in-memory]: report written to reports/controllable-in-memory.json with results in controllable-in-memory.ndjson (1 failed, 24 passed, 4 skipped)
provider-tck [in-memory]: report written to reports/in-memory.json with results in in-memory.ndjson (1 failed, 23 passed, 5 skipped)

96 passed, 9 skipped, 2 xfailed

The envelope:

{
  "schemaVersion": "1",
  "provider": { "name": "In-Memory Provider", "language": "python", "configuration": "in-memory" },
  "sdk": { "name": "openfeature-sdk", "version": "0.10.0" },
  "tck": {
    "implementation": "python-sdk-contrib/tools/openfeature-tck",
    "version": "0.1.0",
    "specRevision": "aa2ad24f5a14ae2b5756df0b6d23f493f39507e6"
  },
  "declaration": { "declared": ["@events", "@numeric-coercion", "@object"] },
  "results": {
    "format": "cucumber-messages",
    "formatVersion": "34.2.0",
    "location": "in-memory.ndjson",
    "digest": "sha256:bfa19f6f07aaac33effac4ad898e6fe05a875c22932428c502c0a0fcb20b342a"
  },
  "backend": {
    "description": "the Python SDK's InMemoryProvider, rebuilt per scenario",
    "controlApi": "in-process"
  },
  "knownDeviations": [
    {
      "issue": "https://github.com/open-feature/python-sdk/issues/619",
      "summary": "python-sdk: a boolean satisfies an Integer request. …"
    }
  ]
}

Two capability changes came with the submodule bump

The spec submodule moves from dfa16586 to dc4d7ae8, which renames a tag and states a rule the enum was only documenting. Both land here as separate commits.

@strict-numeric-typing is now @numeric-coercion, and the rule is corrected

The rule is lossless coercion is permitted; lossy coercion must fail with TYPE_MISMATCH — 10.0 requested as an integer succeeds, 0.5 does not. Appendix F previously said "does not coerce between integer and float", which forbids the case flagd's accepted numeric coercion ADR (open-feature/flagd#1996) requires to work. flagd's own testbed is gaining @numeric-coercion scenarios, so the tag takes flagd's name rather than leaving the reference implementation and the specification with two vocabularies for one property.

So Capability.STRICT_NUMERIC_TYPING becomes Capability.NUMERIC_COERCION, the marker and tag become numeric-coercion and @numeric-coercion, and the docstring states the rule that holds rather than the one the tag was named for. The pytest marker registration needed no change — it iterates the enum. The existing scenario is unchanged and still passes: it asks about 0.5, which is the lossy half.

Two gaps stay open, and no scenario is added here. The lossless half has no scenario in any language, because the canonical flag set holds no integral float to ask it of and adding one changes the flag set for every language at once — so a provider that wrongly rejects 10.0 as an integer still passes this suite. And the width of a language's integer accessor is not modelled at all: the ADR distinguishes a 64-bit accessor from a 32-bit one, which flagd's testbed tags @int32-bounded, and Appendix F has nothing equivalent. Appendix F records both as open rather than closing them, and the capability docstring and README say so too. @numeric-coercion passing is not the same as the coercion contract being tested.

A reserved capability can no longer be declared

@caching is reserved: it exists in the vocabulary and no scenario carries it. @targeting was in the same position when this was written and stopped being so at spec 26362f85, once scenarios started carrying it — which is the shape a reserved tag is meant to have: a placeholder that either grows scenarios or is removed, never a claim a report can carry indefinitely. The enum said so in a docstring and left it there, which reads as documentation rather than as the rule it is. The report schema now states it — see the declaration.declared description — because a capability nothing carries cannot be verified, cannot produce a skip, and tells a reader of a report only that something was claimed and nothing examined.

This was a live defect rather than a hypothetical one: a real Java report asserts both tags as declared, not because anyone decided to claim them but because that adoption declares "every capability except X" and picks up every reserved tag on the way past.

• The set is written down once, as RESERVED_CAPABILITIES, and read everywhere else — by Capability.reserved, by the declare-everything helper and by TckConfig's validation — so the list cannot drift from the rule.
• ALL_CAPABILITIES becomes DECLARABLE_CAPABILITIES: the vocabulary minus the reserved tags, named for what it is rather than for "all", because the declare-everything convenience is precisely the route a reserved tag takes into a report by accident. It is also TckConfig.capabilities' default, so a suite that does not narrow its capabilities no longer declares a tag nothing tests.
• Naming one explicitly raises from TckConfig, in capabilities or in not_applicable. TckConfig already refuses a capability claimed as both declared and not-applicable; this is the same class of error — a claim that cannot be true — caught in the same place, where the adopter's own code is still on the stack. A silent drop would make a rejected configuration look like an accepted one, and a warning is a line of CI output nobody reads while an untested capability goes on being asserted in a published report, which is how it got into one.
• Nothing filters the declaration at emission time, and report.py says why: by the time an envelope is built, a reserved tag cannot be in the TckConfig at all.

Why the results are not our format

Per-scenario outcomes, tags, Scenario Outline row identity and the executed feature source are all already specified by Cucumber Messages, which is maintained, cross-language, schema'd, and emitted natively by cucumber-jvm. Defining them again in the report schema created a second format to maintain and version, and two places for the same fact to disagree.

What was deleted

scenarios[]. Now TestCase / TestCaseStarted / TestStepStarted / TestStepFinished / TestCaseFinished, keyed by ids the protocol defines.

The four-value outcome enum. Cucumber's own seven statuses replace it. The not-declared / not-applicable split it existed for was never a property of the run: it follows from the envelope's declaration and the scenario's tags, both of which are present, so it is stated once instead of once per scenario. declaration.notApplicable keeps the distinction the enum was drawing, where it belongs — as an input to reading the results rather than a per-scenario fact.

example. This is the clearest case. It was added days ago so an outline row could be identified, and four implementations each reinvented it — one of them, this one, by reverse-engineering how its runner maps a pickle back to a table row. Messages has carried that identity all along: a pickle's astNodeIds are [scenario id, table row id], and the row id resolves in the GherkinDocument to exactly the cells the feature file wrote.

tck.assetsTree. The stream carries the executed feature Source verbatim, so a consumer can read the questions that were actually asked rather than trusting a hash to stand for them. That is strictly stronger than what the tree hash was carrying, and it is one fewer thing for the build to get right. The generated spec_revision.json is now just {"specRevision": "…"}.

What stays OpenFeature-specific, and why

The declaration is an input, not a summary. A skipped scenario in the payload says the question was not put to this provider; only the declaration says whether that is because the provider declines the capability. It cannot be derived from the results, which is exactly why it has to be stated.

No standard results format identifies the tested subject. Messages' Meta records the runtime, the OS and the CPU — what produced the answers, not what was being asked about. So provider, sdk, tck and backend stay in the envelope.

knownDeviations. A gap the provider acknowledges is a claim about the provider, not a result. The Python suite marks one scenario xfail(strict=True) against python-sdk#619; the payload still reports that scenario as FAILED, and the envelope carries the acknowledgement beside it. An adoption declares it with TckConfig.known_deviations. Recording it in the envelope is what lets a consumer tell a tracked gap from a surprise without the result itself being weakened, which is the trade the previous version got wrong by carrying the issue reference as prose inside a reason.

Dependencies added

pytest-bdd emits no Cucumber Messages. It ships cucumber_json.py, the legacy Cucumber JSON format, and nothing for the ndjson protocol — so messages.py assembles the stream. Two libraries, each doing the half it owns:

• cucumber-messages (34.2.0, new) — the official Python types, published from cucumber/messages itself. Zero dependencies, requires-python >=3.9, actively released. Used for the execution messages: Meta, TestCase, TestCaseStarted, TestStepFinished, TestStepResult, TestRunStarted/Finished, Hook.
• gherkin-official (>=29, promoted from a transitive dependency of pytest-bdd to a direct one) — the reference Gherkin parser. It produces the gherkinDocument and pickle payloads, which are Messages: emitting Messages ndjson is what that library exists for, so its output is used as it comes rather than round-tripped through another representation that could quietly drop a field it does not model.

uv.lock is regenerated, and uv sync --frozen verified against it. Only cucumber-messages is genuinely new to the lock.

The feature files are parsed a second time, by this package. pytest-bdd parses them with gherkin-official too but converts the result into dataclasses of its own that do not carry the AST node ids — and those ids are what a pickle refers to and what makes one outline row distinguishable from another. Four small files, once per session.

Truthful skips, and per-step results

Appendix F requires that a scenario skipped for an undeclared capability is reported as skipped with the reason and never as passed. pytest, unlike godog, already reports a skip honestly, and Cucumber's SKIPPED says the same thing, so this is the easy part in Python — but it is checked explicitly rather than assumed.

Step results come from pytest-bdd's step hooks (pytest_bdd_after_step, pytest_bdd_step_error, pytest_bdd_step_func_lookup_error) rather than from the scenario's verdict. Marking all eight steps of a scenario FAILED because the scenario failed would be saying something untrue about the seven that passed and the ones never reached. The failing row of the type-mismatch matrix comes out as:

test-case-7-setup     PASSED
test-case-7-0         PASSED
test-case-7-1         PASSED
test-case-7-2         PASSED
test-case-7-3         FAILED   flag 'boolean-flag' resolved to True (bool), expected 1 (int)
test-case-7-4         SKIPPED
test-case-7-5         SKIPPED
test-case-7-6         SKIPPED
test-case-7-teardown  PASSED

with the assertion carried as a TestStepResult.exception of type AssertionError, which is what that field is for.

Every test case also carries a before- and after-hook TestStep, with a Hook message declaring each. pytest runs a scenario in three phases and only the middle one executes Gherkin steps: the capability gate skips during setup, and a provider that fails to shut down fails during teardown. Neither has a pickle step to attach a result to, so without hooks a gated skip would have to borrow the first step's result and a teardown failure would be invisible behind a row of passed steps. Cucumber models exactly this.

That also closes a gap worth naming. A consumer derives a test case's outcome as the worst of its steps. A verdict no step accounts for — a strict xfail that passes, which pytest fails while every step passed — would be lost on the way out, so it is attached to the after-hook. There is a unit test for it.

Scenarios are enumerated at collection and resolved only at the end of the session, unchanged from before and still load-bearing: a scenario skipped by a marker never runs a fixture, so an emitter that learned of a scenario when its fixtures ran would leave it out of the stream entirely.

TckConfig briefly gained not_applicable={Capability.X: "why"} for a capability that cannot hold rather than one the provider declines. It is gone. Appendix F now makes that the implementation's call rather than each adopter's — a capability the language's SDK cannot express is refused at configuration time, uniformly, rather than being declared inapplicable by whoever remembers to — and the schema's declaration object is additionalProperties: false over declared alone, so there was nowhere for the field to go. Python has no such capability today, which is why the removal cost it nothing.

Three schema changes since this was opened

All three are in spec#425 and this branch emits against them.

• backend.controlApi is required. A provider with no backend still had its flag state manipulated somehow, and which of the two ways that was is the single most important thing a reader needs in order to know what the results are worth. The old shape made the whole backend block omissible, which made the one value most worth knowing — in-process, the case the enum exists for, and the one both of this package's own suites use — the one that could never appear.
• A known deviation's summary is required, and its issue link is not. A deviation with no prose is a line of tags that tells a reader nothing; a deviation with no filed issue is an honest state to be in, and requiring one encourages filing a placeholder.
• The declaration no longer claims to explain every skip. It explains the skips that come from an undeclared capability. It does not explain a skip the implementation refuses — @large-integers in Java, @numeric-coercion in JavaScript — which is a property of that language's SDK rather than of the provider, and a reader who conflated the two would read a language's limit as a provider's decision. Python refuses nothing today; the field it used to carry for this, not_applicable, is removed above.

Verification

Run on WSL Ubuntu, Python 3.10 (the repo's pinned version), against the schema as it stands on feat/provider-tck-report-schema. CI does not run on this PR: build.yml triggers only on pull requests targeting main, and this one targets feat/provider-tck, so everything below is local.

Re-run in full after the submodule bump to aa2ad24f: the test suite, ruff, mypy, envelope validation against the report schema, the digest, the message count, the outcome counts and the gated-skip listing. The Cucumber Messages payload schema validation and the uv build check below were done at dfa16586 and were not repeated; the bump changes feature-file text and a scenario name, not the shape of the stream.

Package test suite. 96 passed, 9 skipped, 2 xfailed. ruff check, ruff format --check and mypy (whose scope includes tests) are all clean.

The renamed tag is what actually ran. Every executed Source carries @numeric-coercion and none carries @strict-numeric-typing, and tck.specRevision in both envelopes is aa2ad24f5a14ae2b5756df0b6d23f493f39507e6 — the submodule pin.

No reserved capability reaches a declaration. No reserved tag appears in declaration.declared in either envelope, and test_a_reserved_capability_cannot_reach_the_declaration asserts it of the emitted envelope for a config that leaves capabilities at its default — the "declare everything" case that put one into a published report elsewhere. test_a_reserved_capability_cannot_be_declared covers the explicit route for every member of RESERVED_CAPABILITIES rather than for a list written down beside it.

Envelope validation. Both envelopes validate with jsonschema as Draft 2020-12 against conformance-report.schema.json, zero errors, and results.digest matches a SHA-256 recomputed over the ndjson bytes:

Suite	Envelope	Digest	Payload
in-memory	VALID	matches	661 messages
controllable-in-memory	VALID	matches	661 messages

Payload validation. Both streams validate clean — 661 of 661 messages, zero errors — against the published Cucumber Messages JSON schema at tag v34.2.0 (jsonschema/src/*.schema.json from cucumber/messages, resolved locally through a referencing registry so the relative $refs work). Message census, identical for both suites:

meta 1, testRunStarted 1, hook 2, source 4, gherkinDocument 4, pickle 29,
testCase 29, testCaseStarted 29, testStepStarted 266, testStepFinished 266,
testCaseFinished 29, testRunFinished 1

Every scenario accounted for. 29 TestCase messages against 29 scenarios from pytest's own --collect-only, in both suites, with (uri, scenario name, Examples row) distinct across all 29. test_every_collected_scenario_appears_exactly_once checks the total against --collect-only rather than a number written down beside it, so adding a scenario to the specification cannot leave it passing while the payload loses one.

Outcome counts, computed the way a consumer must — worst step status per test case:

Suite	PASSED	SKIPPED	FAILED
in-memory	23	5	1
controllable-in-memory	24	4	1

Per step, for in-memory: 220 PASSED, 45 SKIPPED, 1 FAILED. Of the 45, 42 belong to the five gated scenarios (their before-hooks included, which is where the reason is) and three are the steps of the failing scenario that were never reached.

Gated skips are truthful. Every scenario carrying a tag absent from declaration.declared is SKIPPED, at every step, in both suites — none is reported as passed. Checked against the reason the gate actually gave, so the derivation from the two documents agrees with reality rather than merely being possible:

SKIPPED  ['@configuration-change']       A configuration change is signalled and applied
         provider does not declare capability @configuration-change. Declared: @events @numeric-coercion @object
SKIPPED  ['@lifecycle']                  A provider that successfully initializes becomes ready
SKIPPED  ['@lifecycle', '@unavailable']  A provider that cannot reach its backend reports an error
SKIPPED  ['@lifecycle', '@unavailable']  A provider that cannot reach its backend still returns code defaults
SKIPPED  ['@stale']                      Losing the backend makes the provider stale, regaining it makes it ready again

(The first scenario's name changed with the submodule bump too, from "A provider reaching its backend becomes ready"; nothing in this package referred to it by name.)

And the distinction survives: the scenario the generated test suite skips with a plain marker is also SKIPPED, but carries no undeclared tag, so it is not attributable to a capability.

The payload disagrees with the runner, deliberately. pytest exits zero — the one scenario the SDK fails is xfailed — and the stream reports it FAILED. That is asserted, not just observed.

The eleven rows. Requesting the wrong type returns the code default produces eleven test cases sharing one scenario name, told apart by the second entry of the pickle's astNodeIds (the id of the TableRow it was compiled from), which resolves in the GherkinDocument to the cells the feature file wrote:

astNodeIds=[25, 9]   PASSED   {key: string-flag,  requested: Boolean, default: false}
astNodeIds=[25, 10]  PASSED   {key: string-flag,  requested: Integer, default: 1}
astNodeIds=[25, 11]  PASSED   {key: string-flag,  requested: Float,   default: 0.1}
astNodeIds=[25, 12]  PASSED   {key: wrong-flag,   requested: Boolean, default: false}
astNodeIds=[25, 15]  PASSED   {key: boolean-flag, requested: String,  default: fallback}
astNodeIds=[25, 16]  FAILED   {key: boolean-flag, requested: Integer, default: 1}
astNodeIds=[25, 17]  PASSED   {key: boolean-flag, requested: Float,   default: 0.1}
astNodeIds=[25, 20]  PASSED   {key: integer-flag, requested: Boolean, default: false}
astNodeIds=[25, 21]  PASSED   {key: integer-flag, requested: String,  default: fallback}
astNodeIds=[25, 22]  PASSED   {key: float-flag,   requested: Boolean, default: false}
astNodeIds=[25, 23]  PASSED   {key: float-flag,   requested: String,  default: fallback}

25 is the Scenario node all eleven share, which is right: the scenario is one scenario. The rows the tests recover from the stream this way are compared against the three Examples tables read out of the Gherkin by hand, rather than against the parser that produced them.

The same holds for rows the capability gate skipped, since row identity comes from the pickle rather than from the run: with @object undeclared, all four rows of Requesting a structured flag as a scalar returns the code default are present, SKIPPED, and individually identified.

Tags on an Examples block. Still covered. Gherkin lets an Examples block carry its own tags, so two rows of one outline can differ in which capability gates them, and those tags are on neither the scenario, the feature nor the rule. No canonical feature file does this yet, so the test writes its own: with one row of a three-row outline gated, all three appear, the gated one is SKIPPED and carries @object, and its two siblings pass.

The executed source is present and correct. Every Source message's data compares equal to the feature file on disk, and the set of Source uris equals the set of uris the test cases name.

Cross-platform. pathlib and list-argv subprocess throughout. Feature uris are normalised to forward slashes: pytest-bdd builds its relative filename with os.path.join, so on Windows it arrives backslash-separated, and the same string has to appear in the Source, the GherkinDocument and every Pickle or nothing ties them together — a report emitted on Windows would otherwise not be comparable with one emitted on Linux. There is a test for it. The emitter itself was developed and verified on Linux; the Windows path of that normalisation is covered by the unit test rather than by a Windows run.

Packaging. uv build produces a wheel carrying spec_revision.json alongside the four feature files, the canonical flag set and control-api.yaml, and declaring the two new requirements.

Failure to write still fails the run loudly, verified by pointing TCK_REPORT_DIR at a path under a regular file. A run that asked for a report and silently did not get one is how a publishing pipeline ends up serving a stale result forever. A scenario that ran and matches no pickle fails the session the same way and names the scenario, because that is the one failure mode this format exists to rule out.

Things worth raising, rather than glossing

The capability rollup is gone entirely, not moved. The previous version summarised a verdict per capability. Both defects raised against it — a failed capability with no reason, and a declared-but-untested capability reported green — disappear with it, but so does the summary. A consumer wanting "did @object pass" now computes it from the payload: the scenarios carrying that tag, and their statuses. That is more work for the consumer and one fewer thing to disagree with the results. Whether the schema should offer a derived summary is worth deciding deliberately rather than by inheritance.

There is still no overall verdict in the envelope. TestRunFinished.success in the payload is false when any scenario failed, which is more than the previous version had, but a consumer must open the payload to see it. Worth deciding whether the envelope wants a top-level verdict.

The stream is assembled by this package, not by the runner. If pytest-bdd ever emits Messages natively, messages.py should shrink to a shim. Two places where that assembly makes a judgement a native emitter would not have to: the join from a pytest node to a pickle goes through (uri, scenario name, Examples row), because pytest-bdd's parse and this one are separate; and test-case/step ids are synthesised (test-case-7-3) rather than being ids the runner already owns. Both are internally consistent and neither is observable across runs, but they are not the ids cucumber-js would emit.

Meta.protocolVersion is read from the installed cucumber-messages version, not pinned in code, so it cannot go stale after a dependency bump. It reports the library version rather than a protocol version negotiated with anything.

Not done here: CI does not upload the reports as artifacts. That belongs with whatever consumes them, and can follow.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

❌ Patch coverage is 76.24811% with 157 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.01%. Comparing base (01bb657) to head (b2f9e77).

Files with missing lines	Patch %	Lines
...e-tck/src/openfeature/contrib/tools/tck/emitter.py	69.60%	69 Missing ⚠️
...-tck/src/openfeature/contrib/tools/tck/messages.py	72.54%	56 Missing ⚠️
...re-tck/src/openfeature/contrib/tools/tck/report.py	87.66%	19 Missing ⚠️
tools/openfeature-tck/hatch_build_sync.py	47.36%	10 Missing ⚠️
...tck/src/openfeature/contrib/tools/tck/canonical.py	94.44%	2 Missing ⚠️
...nfeature/contrib/tools/tck/steps/provider_steps.py	87.50%	1 Missing ⚠️

Additional details and impacted files

@@                  Coverage Diff                  @@
##           feat/provider-tck     #413      +/-   ##
=====================================================
- Coverage              89.02%   86.01%   -3.01%     
=====================================================
  Files                     44       48       +4     
  Lines                   2160     2818     +658     
=====================================================
+ Hits                    1923     2424     +501     
- Misses                   237      394     +157     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.