diff --git a/tools/tck/README.md b/tools/tck/README.md index 242d8009b..79d7dafae 100644 --- a/tools/tck/README.md +++ b/tools/tck/README.md @@ -345,6 +345,176 @@ the exclusion **and** narrows Surefire's includes to it: **Both halves are needed.** Dropping alone runs the module's unit tests alongside the suites; narrowing alone leaves the exclusion in force and runs nothing. +## The canonical set cannot be reduced + +Extending the suite is safe by convention. Shrinking it is what a conformance suite has to prevent, +because a run that asks all but two of the canonical questions and reports success is +indistinguishable, in every artifact it produces, from one that asked every question there is. + +`CanonicalScenarioGuard` is an ordinary JUnit test that the suite selects, and it fails the build if +this run is set up to execute less than the canonical set: + +- a feature file added to `gherkin/`, or shadowing a canonical one — the selected scenarios no + longer match what this artifact ships, which it reads from its own JAR rather than through the + classpath +- `cucumber.filter.tags` or `cucumber.filter.name` — Cucumber applies these by skipping scenarios at + execution, so the run is filtered however the plan looks +- selectors or glue overridden in your `junit-platform.properties` + +It checks the setup rather than counting afterwards: both the discovered plan and the run's filter +configuration are settled before the first scenario, so the check needs no backend and takes no +measurable time. Where its result appears in the run depends on the order the JUnit Platform executes +the suite's two engines in, which is not specified. Extension scenarios are ignored: the check is +defined over `gherkin/` alone. + +Narrowing a run legitimately is what `capabilities()` is for — those scenarios are reported as +skipped with a reason, which a filtered scenario is not. To filter anyway while debugging, set +`-Dtck.partial=true` (or `TCK_PARTIAL`). The guard then reports itself as +**skipped** rather than passed, so the run states that its canonical set was not verified. + +What the guard does not establish is that the canonical files contain what they should — a +replacement placing its scenarios on the same lines would satisfy it. That is covered better +elsewhere: the results stream carries the `source` of every feature that executed, and +`tck.specRevision` says which revision it should match. + +## Conformance reports + +Set `TCK_REPORT_DIR` and each suite writes two files: an envelope conforming to the +[report schema][report-schema] in the specification, and the run's results as a +[Cucumber Messages][messages] stream. + +```console +$ TCK_REPORT_DIR=./reports mvn -Ptck -pl providers/flagd test +$ ls reports/ +flagd-in-process.json flagd-in-process.ndjson flagd-rpc.json flagd-rpc.ndjson +``` + +`-Dtck.report.dir=...` does the same thing and is often easier to pass through Maven. The +environment variable is the portable spelling — every language's TCK reads it, so one cross-language +CI job can set one thing. + +It is an environment variable rather than a method on `ProviderTckHarness` so that emitting a report +is a property of the run and not of the code: CI sets it, a developer running the suite locally does +not, and no adopter changes a line to publish one. Unset means no report, which is not an error. +Several suites in one JVM each write their own pair, so flagd's two resolvers do not collide. + +### The results are not a format this project defines + +The `.ndjson` is a Cucumber Messages stream, produced by Cucumber's own `MessageFormatter` — the +same class the built-in `message:` plugin instantiates, so the bytes are what +`--plugin message:...` would have written. It already carries everything a per-scenario report would +have had to invent: the outcome of every scenario, its tags including any set on an individual +`Examples` block, an exact Scenario Outline row identity, and the source of every feature that ran. + +The plugin exists rather than the built-in one because a `@ConfigurationParameter` value is a +compile-time constant, so the built-in plugin's path cannot be derived from the directory the run +asked for — and flagd's two suites would write to the same file. + +Reading it needs no special tooling, but it does need one thing understood: **a scenario's outcome +is the most severe result among its steps**, hooks included. `testCaseFinished` carries no status of +its own. That is what makes a capability-gated skip truthful, because the aborted `@Before` hook +contributes a `SKIPPED` result that outranks every step it stopped from running. + +```console +$ jq -c 'select(.testStepFinished) | .testStepFinished + | {c: .testCaseStartedId, s: .testStepResult.status}' reports/flagd-rpc.ndjson \ + | jq -s 'group_by(.c) | map({s: (map(.s) | if any(. == "FAILED") then "FAILED" + elif any(. == "SKIPPED") then "SKIPPED" + else "PASSED" end)}) + | group_by(.s) | map({(.[0].s): length}) | add' +{ + "PASSED": 28, + "SKIPPED": 1 +} +``` + +The [`cucumber-query`](https://github.com/cucumber/messages/tree/main/java) helpers do this properly +and in several languages; the above is only to show that the fact is in the file. + +### What identifies a scenario + +`pickle.astNodeIds`. For a scenario compiled from a Scenario Outline it is +`[scenario id, table row id]`, and the second entry resolves in the `gherkinDocument` message to the +`Examples` row the scenario was built from. Feature and name are not enough — the type-mismatch +matrix in `errors.feature` is eleven rows sharing one name — and this is exact rather than derived: + +```console +$ jq -c 'select(.pickle) | .pickle + | select(.name == "Requesting the wrong type returns the code default") + | {id, row: .astNodeIds[1]}' reports/flagd-rpc.ndjson | head -3 +{"id":"6c8debd2-...","row":"ab8b4a4b-..."} +{"id":"a7c76b0a-...","row":"63d6d6c8-..."} +{"id":"fecd333d-...","row":"bbd7f5ee-..."} +``` + +An earlier version of this module reverse-engineered the same fact by re-parsing the feature source +and matching a pickle's reported line number against the Examples tables. The stream states it +outright, which is the whole argument for a standard format over one we maintain. + +### What the envelope is for + +A Messages stream cannot say what it was a test *of*. The envelope carries the five things no +standard results format identifies: + +- **`provider`** — what the provider calls itself through its own metadata, not the suite name. The + suite name is chosen to read well in a failure message (`flagd-rpc`), which makes it the + *configuration*, and it is reported as such. One provider with two materially different modes + produces two reports that are not interchangeable. Derived from the suite class name + (`MyProviderInProcessTest` → `my-provider-in-process`) unless the suite overrides + `ProviderTckHarness.configuration()`, which flagd's two do — a class in a package already called + `.../flagd/tck/` is named `InProcessTest`, and `in-process` on a report does not say whose. +- **`sdk`** — read from the classpath rather than declared, because the TCK depends on an SDK version + *range* so that adopting it can never force an upgrade. What a consumer actually ran against is + only knowable at runtime. +- **`tck`** — which implementation asked the questions, and `specRevision`, the open-feature/spec + commit the packaged artifacts came from. Baked into the JAR at build time from this module's POM: + the artifacts travel in the JAR, the repository they came from does not. The executed Gherkin no + longer rests on that pin alone — the stream carries the `source` of every feature, so it can be + diffed against the revision — but the pin is what identifies the two artifacts the stream does not + carry, `flags/canonical-flags.json` and `openapi/control-api.yaml`. +- **`backend`** — what the provider was pointed at, and `controlApi`, which of the two control + contracts drove it. Always present, both of them: the same scenarios passing over the HTTP control + API and passing through in-process manipulation of a provider that *does* have a backend are not + the same claim, and this is the only field that separates them, so an omission would be an + unfalsifiable claim rather than no claim at all. See [Identifying the + run](#identifying-the-run). +- **`declaration`** — the capability set the provider claims. This is an **input** to reading the + results, not a summary of them, which is why it cannot be derived from the stream. The stream says + a scenario was skipped; only the declaration says whether that is because the provider declines the + capability it needed. Given the declaration and a scenario's tags — both present — the reason for + each skip follows, so it does not have to be transported per scenario. + + **One skip does not follow from it, and that one is why the reason travels anyway.** A capability + this SDK cannot express is absent from every Java declaration, and absent for a reason that says + nothing about the provider — `@large-integers` is not there because `Client.getIntegerDetails` is + 32 bits, not because anyone declined it. A reader who inferred *"the provider declined"* from that + absence would be reading a decision into something no Java provider was ever offered. The gate's + reason is carried on the hook result that produced the skip, and its wording is deliberately unlike + an undeclared capability's, so the two are distinguishable in the results themselves. See + [Declaring capabilities](#declaring-capabilities). + +`knownDeviations` is the one thing neither the stream nor the declaration can express: whether a +withheld capability is a limitation or a bug. See [Known deviations](#known-deviations). + +`results.digest` covers the `.ndjson`, so a consumer that fetched the two separately can tell that +what it has is what the envelope describes. + +### What the report is for + +This suite promises that a scenario skipped for an undeclared capability is reported as skipped with +the reason and *never* as passed — and a promise is not a check. The stream records every scenario +individually, so a consumer can verify the rule instead of trusting a runner's headline number. Go's +runner counts capability-gated skips in its **passed** tally, which is exactly the failure mode this +makes impossible to hide. + +Every scenario appears exactly once, whatever happened to it. A report that quietly omitted the +scenarios it did not run would satisfy every rule above and still mislead, because a reader would +have no way to know how many questions went unasked. `ConformanceReportPluginTest` runs a fixture +suite through the real Cucumber engine and asserts both properties over the emitted stream. + +[report-schema]: https://github.com/open-feature/spec/blob/main/specification/assets/provider-tck/report/conformance-report.schema.json +[messages]: https://github.com/cucumber/messages + ## Extending it A provider with features of its own — flagd's `fractional` targeting, a vendor's proprietary mode — diff --git a/tools/tck/pom.xml b/tools/tck/pom.xml index 99e5987db..cc2a5e02d 100644 --- a/tools/tck/pom.xml +++ b/tools/tck/pom.xml @@ -25,6 +25,39 @@ --> false + + ff68adb4c7617ad2d980988241e92603bc247926 3.27.7 4.3.0 2.22.1 @@ -109,6 +142,55 @@ cucumber-junit-platform-engine + + + io.cucumber + cucumber-core + + + + io.cucumber + messages + + + + + + io.cucumber + gherkin + compile + + + + + org.junit.jupiter + junit-jupiter-api + compile + + io.cucumber @@ -217,6 +299,23 @@ + + + + src/main/resources + false + + + src/main/resources-filtered + true + + +