[Tracking] A cross-language conformance suite for OpenFeature providers

[aepfli]

The problem

OpenFeature's central promise is vendor neutrality: swapping providers must not change application
behaviour. Nothing currently verifies that.

Every provider tests differently. Some have thorough suites, some have almost none, and no two agree
on what "implements the provider contract" means. The consequences are real and already visible:

• Providers disagree on whether a type mismatch returns the code default or throws.
• Providers disagree on whether an unreachable backend yields ERROR promptly, hangs, or throws out
  of provider registration.
• Providers disagree on which lifecycle events fire, and in what order, around a disconnect.
• Some providers silently coerce between numeric types instead of reporting TYPE_MISMATCH.

Every one of these is invisible until an application swaps providers in production. The spec
describes the correct behaviour in prose; there is no executable definition of it.

This is the tracking issue. The design lives in Appendix F
(#423) — the three artifacts, the control API,
the capability vocabulary, the extension rules and the remaining open questions are all specified
there, and that is the copy to read and to argue with. This issue carries status, per-language
links and what the suite has found.

What this is

A language-agnostic conformance suite for providers: Gherkin scenarios, a standardised backend
control API and a canonical flag set, living in this repository so that every language's TCK runs
the same definition of conformance rather than inventing its own. Each language packages those
artifacts and supplies the runner — step definitions, the container lifecycle, a control-API client
— in whatever form its testing culture expects. It verifies the provider contract only: typed
resolution, keeping integer and float distinct, error handling, lifecycle and events. Backend
evaluation logic, targeting correctness and the provider↔backend wire protocol are all out of scope.

Status

All four languages have the same shape — a suite PR off main, forking into a flagd adoption with
an OFREP adoption stacked on it, and a conformance report PR — on matching branch names
(feat/provider-tck, then -flagd, -ofrep, -report). Nothing has merged yet; everything is in
draft pending review of #423.

All four pin the spec at the same commit and read the three artifacts from that pin rather than from
a copy, so no language can drift from the definition of conformance without the pin moving.

Appendix F + the three artifacts	#423
Conformance report JSON schema	#425
Publishing conformance reports	#424
What a provider must do when a number does not fit the accessor	#430

Decisions settled across all four languages

Each of these began as four different answers, and several were only visible because there were
four implementations to compare:

• The package is tck in every language — dev.openfeature.contrib.tools:tck,
  github.com/open-feature/go-sdk-contrib/tools/tck, @openfeature/tck, openfeature-tck — all at
  0.1.0. It was provider-tck everywhere; renamed at
  @erka's request,
  since the artifact is not a provider and a flat tck leaves room for a suite that tests something
  other than one.
• The suite owns the container stack. An adopter supplies a Compose file, the ports the provider
  connects to, and a factory taking a discovered endpoint; everything else — starting the stack,
  discovering dynamically mapped ports, building the control client, waiting for the control API,
  teardown — belongs to the TCK. Three of four languages originally shipped only the control-API
  client, and each adoption then hand-rolled between 130 and 460 lines of the same wrapper. Now
  required by Appendix F.
• knownDeviations has one meaning. Two legitimate shapes, with declare-and-let-it-fail
  preferred over withhold-and-skip, summary required and issue optional. Previously Java and Go
  documented one shape, JS the other and called the first discouraged, and Python required an issue
  link with no untracked form — one field read three ways.
• controlApi is required and closed-typed, stated by the control rather than inferred by the
  harness. It was a defaulting method in Java, an optional field in JS, a duck-typed property in
  Python and absent from Go's interface. Nothing outside the control can tell which path a run used,
  and an absent value is not neutral: every run is one or the other.
• Every state-changing control endpoint must serve the new state before returning. /start
  already said so; /change and /reset did not, and /reset is the endpoint a suite calls before
  every scenario — so a window there is re-rolled per scenario rather than once per suite.
• /restart is optional. It was [REQUIRED] on the strength of a claim in its own description
  that the TCK used it for disconnect/reconnect. It does not: that scenario is an unbounded outage,
  served by /stop + /start, and nothing in the shipped Gherkin reaches /restart in any language.
• The resolution reason is a claim, not a house rule. The suite asserted an exact reason in
  thirteen places across three feature files, narrowing a SHOULD into a MUST for every adopter.
  Those are gone; the reasons now live in one gated reason.feature, and declaring
  @standard-reasons means "I use the standard vocabulary with the standard meanings". A provider
  whose backend reports vendor-specific reasons simply does not declare it and loses nothing — its
  values, variants and error codes are asserted on MUSTs regardless. This also lets Appendix F
  state what each reason means without asking the specification to close the set, since the mapping
  is the content of an opt-in claim.
• A capability the language's SDK cannot express is refused by the implementation, not left to
  adopters. Two exist: @large-integers where the integer accessor is 32-bit, @numeric-coercion
  where the language has one numeric type. Neither says anything about a provider. Before this,
  JavaScript restated that fact in five places and Java in five — every one a chance to put a claim
  in a report that no scenario could verify.
• An adoption suite is not a required CI gate while real gaps remain, because its honest output
  is red and making it block a merge forces someone to silence it. All four languages believed their
  adoption suites were excluded from CI; all four were running them, red and unwatched, each
  defeated by a different mechanism — a profile that cleared the exclusion, a build tag applied to
  every module, a default test task that simply collected them, and a run-all target.

Per-language tracking

Language	Issue	Suite	flagd	OFREP	Report
Java (reference)	#1829	#1830	#1847	#1840	#1841
Go	#938	#940	#941	#942	#944
Python	#410	#409	#411	#414	#413
JavaScript	#1607	#1606	#1608	#1612	#1610

Not started: .NET, PHP, Ruby, Kotlin/Swift (client-side). JavaScript is server-SDK only so far, so a
web-SDK provider cannot adopt it yet.

OFREP adoption is cheap in any language that has an OFREP provider: flagd already serves OFREP on
port 8016 and flagd-testbed's compose exposes it, so the existing testbed and launchpad control
API drive it with no new infrastructure.

Backend under test

The suite drives flagd-testbed. Four changes to it came out of this work:

Numeric-precision flags and scenarios	flagd-testbed#392
Type-mismatch matrix across the value types	flagd-testbed#393
/start must wait until flags are actually served	flagd-testbed#394
No /reset, and /start cannot restore the baseline	flagd-testbed#395

#394 and #395 together gate the most, and they are one defect seen from two ends. The launchpad
404s on POST /reset, so a suite that prefers /reset for scenario isolation falls back to
POST /start?config=default before every scenario — a full flagd restart each time. And /start
returns before the file source has loaded. So a race measured at ~40 ms and 35% of starts is
re-rolled per scenario instead of once per suite, which is why two runs of the same suite disagree
about which scenarios failed. Measured on one adoption at 5 to 41 failures out of 47 across five
runs, and equally bad against the hand-rolled container wrapper it replaced, so it is the backend
and not the harness. #394 makes /start honest; #395 removes the need to restart at all.

Findings from the language implementations

Bugs the suite surfaced, listed because the point of the exercise is whether it finds real things.

Finding	Where	Tracked	Status
/readyz returns 200 while the flag store is still empty, so a flag the configuration defines answers FLAG_NOT_FOUND	flagd	flagd#2047	open
flagd narrows a float flag to an integer, returning 0 with no error code — both resolvers	flagd	flagd#1996	open, ADR accepted
The RPC resolver never emits PROVIDER_STALE, while in-process does	flagd provider, Go	go-sdk-contrib#939	open
A boolean satisfies a Float request, because bool subclasses int	flagd-core, Python	python-sdk-contrib#417	open
A boolean satisfies an Integer request, same cause	python-sdk	python-sdk#619	merged, no release carries it
The in-memory provider cannot update its flag set or emit PROVIDER_CONFIGURATION_CHANGED, which Appendix A requires	go-sdk	go-sdk#530	open
Same gap, found independently	python-sdk	python-sdk#620	merged, no release carries it
A disabled flag returns a GENERAL error alongside the default, against requirement 2.2.6	go-sdk	go-sdk#552	merged, no release carries it
InMemoryFlag.state is declared and never read, so a DISABLED flag resolves as enabled	python-sdk	python-sdk#627	open (asked as a question)
MultiProvider never subscribes to its children, swallowing their events	java-sdk	java-sdk#1882	open
MultiProvider flattens every child error code to GENERAL	js-sdk	js-sdk#1452	open
The OFREP provider does not handle codeDefaultFlag responses, so a disabled flag surfaces FLAG_NOT_FOUND	java-sdk-contrib	java-sdk-contrib#1850	open (asked as a question)
variant and reason indexed unconditionally where the OFREP schema makes both optional	python-sdk-contrib	python-sdk-contrib#418	open (asked as a question)
flagd's two resolvers disagree on numeric coercion: in-process returns TYPE_MISMATCH, RPC returns 0 with no error code	flagd provider, Python	python-sdk-contrib#420	open (asked as a question)
shutdown() closes the gRPC channel without stopping the connectivity watcher, leaving Cannot invoke RPC: Channel closed! in a host application's log	flagd provider, Python	python-sdk-contrib#419	open (asked as a question)
The in-memory provider does update and emit correctly — the reference behaviour the other two should grow	js-sdk	n/a	positive result

Four of these are worth dwelling on.

One defect was surfaced by a scenario that passed. The Python flagd provider's shutdown
completes, and within the bound the scenario asserts — and leaves a gRPC traceback behind it, because
the channel closes while the connectivity watcher is still polling it. Nothing required is unmet, so
the results record a pass and the report has nowhere to put it. It was found only because the
@lifecycle capability had never been declared in that adoption, so those six scenarios had never
run at all; declaring it was a one-line change. Two lessons: an undeclared capability is a coverage
gap that compounds silently, and a conformance report is not the whole of what running the suite
tells you.

The flagd resolvers report an outage differently from each other, which is precisely the
vendor-neutrality claim the suite is meant to check.

The Python bool bugs could not have been found by any other language's suite — bool is a
subclass of int only in Python. That is the clearest argument yet for implementing this in more
than one language rather than treating one implementation as sufficient. Having been caught twice,
once per numeric accessor, it is also an argument for the suite asserting both directions.

Three findings are merged and carried by no release. python-sdk#619, python-sdk#620 and
go-sdk#552 are all fixed on main while the newest releases predate the fixes, so both suites still
resolve pre-fix SDKs, still ship a wrapper class, and still hold a strict xfail. Each needs a pin
bump and a marker removed rather than further investigation — worth tracking as a release question
rather than as three open bugs.

flagd#2047 needs no provider, SDK or TCK to
reproduce — plain flagd, one file source, a tight loop on /readyz. It surfaced because the suite
has to wait for readiness before it can evaluate anything, which turned "when is a backend ready?"
into a question someone had to answer precisely. flagd is not violating its documented contract; the
claim is that "one successful data sync" is the wrong semantics for a readiness probe, because
delivered is not applied.

Open questions

Appendix F carries the full set; these are the ones that most want other people's opinions.

1. Should reason be an open set at all? The suite no longer depends on the answer — it asks
   only providers that declare @standard-reasons — but the question stands on its own.
   Requirement 2.2.5
   permits "some other string", while the OpenTelemetry convention that consumes reason is closed
   at exactly the nine OpenFeature values, and Appendix D already instructs integrations to treat it
   as an enumeration and transform it. So a vendor-specific reason has nowhere to land downstream.
   Worth knowing before deciding: how many providers actually emit one today. Every conformance
   report now answers that for its provider, which turns the question into something measurable
   rather than a matter of opinion.
2. A report's declaration no longer explains every skip, and consumers should know it. A
   capability may be absent because the provider declined it or because the language's SDK cannot
   express it, and the two are indistinguishable from the declaration alone. The results payload
   carries the distinction per skip and Appendix F records which capabilities are affected per
   language, but anything comparing providers across languages has to read the skip reason rather
   than infer from the declaration.
3. Nothing validates an emitted report against the schema in CI, in any language — each checks
   fields by hand or validates out of band. The schema is what makes reports comparable across
   languages, so this is the weakest link in the design, and it wants one answer rather than four.
4. What a provider must do when a number does not fit the accessor —
   #430, which @numeric-coercion currently gates
   rather than answers.

Prior art in this ecosystem

• open-feature/test-harness — the flagd-oriented
  Gherkin suite and testbed this work generalises.
• open-feature/flagd-testbed — the launchpad
  control API the standardised one is derived from.
• dev.openfeature.contrib.tools:flagd-api-testkit in java-sdk-contrib — an existing "abstract
  suite + SPI factory, features packaged in the JAR" testkit whose adoption ergonomics the Java
  implementation copies.

[aepfli]

Three questions from running the suite for real

The Go flagd suite ran against Docker for the first time today — 29 scenarios, both resolvers. Three things came out of it that are questions for this appendix rather than for any one implementation.

---

1. Nothing verifies that an implementation reports skips as skips

The appendix is unambiguous:

Scenarios whose tag is not declared are reported as skipped, with the reason — never as passed.

a conformance suite that quietly goes green on scenarios it did not run is worse than no suite at all.

The Go implementation violates this, and it took a real run to notice. godog's summary:

29 scenarios (29 passed)
208 steps (191 passed, 17 skipped)

Two of those 29 were capability-gated skips (@stale, @strict-numeric-typing). The steps are correctly marked skipped, but the scenario tally counts them as passed. Only a separate log line written by the harness itself reveals the truth:

provider-tck [flagd-rpc]: 2 scenario(s) skipped because a capability was not declared.

This is a property of the runner, not of the harness design: pytest reports them as SKIPPED with the reason attached, and jest-cucumber maps them to test.skip. Go's runner treats "did not fail" as "passed".

So an implementation can satisfy the letter of the rule in its own logging while its headline number says something false — and the appendix gives no way to tell the two apart.

Question: should the appendix require an implementation to emit a machine-readable per-scenario outcome rather than relying on the runner's summary? That would make the rule checkable instead of aspirational, and it is the same artifact #424 needs for publishing conformance reports — so the two could share one definition rather than inventing it twice. At minimum, "the reported totals must not count a skipped scenario as a passed one" seems worth stating explicitly.

---

2. "Submodule" and "packaged into the artifact" can be mutually exclusive

The appendix says:

Each language's TCK then vendors the spec repo (git submodule or equivalent) and packages these files into its own distribution format, so that adopting a TCK never requires a consumer to check out a submodule of their own.

Those are two requirements, and in Go they appear to conflict. Go module zips are built from the VCS tree, where a submodule is only a gitlink — golang.org/x/mod/zip excludes nested-VCS content (cf. golang/go#26716). A published Go module that go:embeds straight from a submodule would ship an empty directory, and an adopter's go get would fail to compile — a hard failure discovered only after tagging.

Java has no such problem: the submodule is consumed at build time and the assets land in the JAR. Python is the same via a build hook. JavaScript can copy into the package at build. Go is the one where the obvious reading of "use a submodule" breaks the second requirement.

Question: is the normative requirement the outcome — an adopter never needs a submodule — with the mechanism left to each language? If so the appendix should say that plainly, because "vendors the spec repo (git submodule or equivalent)" currently reads as prescribing the mechanism, and at least one language cannot satisfy both readings at once.

(Related, and worth deciding alongside: whether the spec revision a TCK ran against must be recorded in the artifact. A submodule pin does that for free; a generated copy needs it written down. See #424.)

---

3. @strict-numeric-typing now has two confirmed failures in the same provider

Reinforcing open question 4 with observed rather than inferred evidence. flagd narrows a float to an integer in both Java and Go, in both resolvers of each:

flag "float-flag" resolved to 0 (int64), expected 1 (int64)

float-flag (0.5) evaluated as an integer returns 0 with no error code at all, rather than TYPE_MISMATCH with the code default. The application sees a plausible value and nothing to indicate anything went wrong. Both Go resolvers behave identically, which places the defect in the shared provider layer rather than in either transport — matching what the Java implementation found.

So the most widely deployed OSS provider fails this capability in at least two languages, and both adoptions have to withhold it to stay green.

Question: does that change the answer to open question 4? Modelling a specification violation as an optional capability was justified as pragmatic — "so a provider with the defect can adopt today". Two independent confirmations in the reference provider suggest the pragmatism is doing more work than expected, and that a distinct known-deviation concept (visible, attributed, and not confusable with "this provider chose not to implement an optional feature") may be worth the extra machinery after all.

The Python implementation already needed something like it for a different reason: xfail(strict=True) on a scenario that fails because of python-sdk#619, so the deviation stays visible and un-hides itself automatically when the SDK is fixed. That is a third shape for the same idea, invented locally because the vocabulary has no word for it.

[aepfli]

Coverage gaps, and one the suite is already exposed to

The Gherkin set in #423 is not final, and it is worth writing down what it does not yet cover while the appendix is still in draft. These are follow-ups rather than blockers — every current adoption leaves the relevant tags undeclared, so adding scenarios later turns nothing red.

Caching, and why it is not merely absent

@caching is reserved: the tag exists in the vocabulary and no scenario carries it. That understates the situation, because the suite is already running against a caching provider.

flagd's RPC resolver enables an LRU cache by default — defaultCache = cache.LRUValue in configuration.go — and on a hit it rewrites the resolution reason to CACHED, in all five typed paths of pkg/service/rpc/service.go. The conformance adoption does not turn it off.

So the suite asserts reason STATIC on every evaluation while running against a provider that will answer CACHED under the right conditions. It passes only because no scenario evaluates the same flag twice in a way that hits the cache. A scenario added later that does will fail against flagd RPC, and the failure will look like a provider defect rather than a test-design one.

There is also an accidental half-test already in there. The configuration-change scenario reads changing-flag, changes it, and reads again — which against flagd RPC only produces the right answer because the change event evicts the cache entry. A provider whose cache was not invalidated would serve the stale value and fail a scenario that never mentions caching. We are depending on invalidation working, untagged and unstated.

If @caching is written up later, the invalidation case is the one that earns its keep. A cache that is not invalidated is a correctness bug that ships silently: the application serves stale flags indefinitely and nothing errors. "The cache exists" is far less interesting than "the cache is correct".

Two questions to settle first, neither obvious:

• Is CACHED on a repeat evaluation required of a provider declaring @caching, or merely permitted? A cache may legitimately miss, and a scenario that assumes a hit is a flaky scenario.
• Should flagd's in-process resolver declare it? It holds the entire ruleset locally, so "cached" arguably means something different there than it does for RPC.

Worth noting for contrast: the Go OFREP provider has no provider-side cache at all — no cache, no ETag handling — which is consistent with stateless request/response. The OFREP protocol does define caching semantics for the bulk endpoint, but that is the client-side flavour and a different thing from a provider cache; I have not checked how the other languages' OFREP providers treat it.

The rest of the gaps

• Evaluation context passthrough is unverifiable. Scenarios can build an evaluation context but cannot assert one actually reached the backend. That needs an echo operation on the control API — something that reflects back what the backend received. Until then a provider that silently drops the context passes every scenario. @targeting is reserved for exactly these, and is the reason it appears in the vocabulary with nothing behind it.
• POST /restart is unused. No current scenario needs a bounded outage; the stale scenario uses an explicit disconnect and reconnect instead. The endpoint stays in the control API for a scenario that wants a self-healing outage, but nothing exercises it today.
• Hooks and flag metadata are not covered at all.

Why this belongs on the record now

Two reserved tags with no scenarios behind them (@targeting, @caching) were quietly being reported as passed by three of the four emitters until this week — a green result for a claim nothing had examined. That is fixed, and a capability no scenario exercises is now omitted from a report rather than claimed.

But it is a good illustration of the general hazard: an untested capability in the vocabulary is a place where a conformance claim can appear from nowhere. Either the scenarios get written, or the tags should be removed from the vocabulary until they do. Leaving a named capability permanently unexercised is the weakest of the three options.