Run automated conformance suite in CI

[russlan23]

Describe your change

Adds a Playwright contribution check for the FDC3 conformance framework. The check starts the conformance app and the FDC3 for Web reference Desktop Agent, launches the runner inside the Desktop Agent, selects the complete non-manual pack, and fails with the underlying conformance failure messages when any test fails.

The browser runner exposes machine-readable status and totals. The result gate requires a passing status, zero failures, at least one test, and completion of every reported test. It uses Mocha's published browser bundle. The destructured-method setup and cleanup fixes have already landed through #2143 and are no longer part of this PR's diff.

Manual intent-resolver and channel-selector tests remain outside this automated check.

The conformance pack now has its own Automated Conformance workflow, separate from ordinary tests and coverage. It supports manual dispatch and reusable workflow calls. Automatic runs are limited to the conformance, web-implementation and agent-proxy source paths plus the harness/workflow files. The actual conformance path is toolbox/fdc3-conformance/src. Reports and failure traces are retained for seven days; branch protection should not require this path-filtered check on unrelated PRs.

Current validation:

• September 10: the workflow split in b451bb2 passes YAML/trigger/permission checks, 12 positive and negative path cases, coverage-separation checks, formatting and git diff --check. A merge simulation against main at 6cf0001 succeeds. No runtime code or dependencies changed in this follow-up.
• September 10 full browser run (6.1 minutes): all three result-gate tests pass, while the full-pack test correctly fails on the same three reference-agent behavior failures after Add newInstance parameter to raiseIntent and raiseIntentForContext #1942: RaiseIntentExistingInstanceRequired, RaiseIntentFailExistingInstanceRequired, and IntentContextMetadataWithAppMetadata. Details are in the scope and test note. No assertion was weakened or skipped; maintainer direction on a separate behavioral follow-up is still pending.
• The earlier 115/115 full-pack result predates those upstream changes and is not a passing result for the current reference agent.

Related Issue

Resolves #2045.

Contributor License Agreement

• I acknowledge that a contributor license agreement is required and that I have one in place or will seek to put one in place ASAP.

Review Checklist

• Issue: The contribution is linked to Make FDC3 conformance test framework run under CI as a contribution test #2045.
• CHANGELOG: A CHANGELOG.md entry is included.
• API changes: No FDC3 API is changed.
• Context types: No Context type is added or modified.
• Intents: No Intent is added or modified.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: russlan23 / name: russlan23 (94990af)

[netlify]

✅ Deploy Preview for fdc3 ready!

Name	Link
🔨 Latest commit	b451bb2
🔍 Latest deploy log	https://app.netlify.com/projects/fdc3/deploys/6aa22efe817b7c00089d1f6d
😎 Deploy Preview	https://deploy-preview-2048.preview-fdc3.finos.org
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[russlan23]

/easycla

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.03%. Comparing base (cb00631) to head (b139a63).
⚠️ Report is 8 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2048   +/-   ##
=======================================
  Coverage   95.03%   95.03%           
=======================================
  Files          73       73           
  Lines        4251     4254    +3     
  Branches      596      602    +6     
=======================================
+ Hits         4040     4043    +3     
  Misses        211      211           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Roaders]

This looks promising as an initial way to get these tests running as part of the CI process but we must be sure that the build will fail if the tests fail. As the tests timeout at the moment but this build is still fine I am not sure this is the case.

[russlan23]

Addressed in 335fc22. I added an explicit result gate that throws unless the conformance run reports passed, has zero failures, executes at least one test, and completes every reported test. I also added negative coverage for failed or timed-out and incomplete runs.

Local verification: npm run test:ci --workspace fdc3-conformance — 4 passed in 5.9m. A conformance failure, incomplete run, or the existing outer Playwright timeout now returns a non-zero exit from the command used directly by the CI job.

[russlan23]

Addressed in 335fc22. I added an explicit result gate that throws unless the conformance run reports , has zero failures, executes at least one test, and completes every reported test. I also added negative coverage for failed/timed-out and incomplete runs.\n\nLocal verification:

fdc3-conformance@3.0.0-alpha.2 test:ci
playwright test

Running 4 tests using 1 worker

✓ 1 e2e/conformance.spec.ts:9:1 › runs all automated conformance tests against the reference Desktop Agent (5.8m)
✓ 2 e2e/conformanceResults.spec.ts:10:3 › conformance result gate › accepts a completed passing run (0ms)
✓ 3 e2e/conformanceResults.spec.ts:22:3 › conformance result gate › rejects a failed or timed-out conformance test (1ms)
✓ 4 e2e/conformanceResults.spec.ts:34:3 › conformance result gate › rejects a run that ends without completing every test (0ms)

Slow test file: e2e/conformance.spec.ts (5.8m)
Consider running tests from slow files in parallel. See: https://playwright.dev/docs/test-parallel
4 passed (5.8m) — 4 passed in 5.9m. A conformance failure, incomplete run, or the existing outer Playwright timeout now returns a non-zero exit from the command used directly by the CI job.

[russlan23]

Rebased this branch onto current main after #2143 merged, so the overlapping destructured-method setup and cleanup fix now drops out of this PR. Mocha 11 also requires its published browser bundle here; importing the Node entry caused Vite to externalize node:util and left the automated test selector empty. That compatibility fix is in d540915, and the conformance package build passes.

The full gate now executes and reports three failures on current main (cb00631d) after #1942:

• RaiseIntentExistingInstanceRequired: two instances remain after newInstance=false, expected one.
• RaiseIntentFailExistingInstanceRequired: the call does not surface TargetInstanceUnavailable as expected.
• IntentContextMetadataWithAppMetadata: the delivered traceId is regenerated instead of preserving intent-trace-456.

The three result-gate unit tests still pass. I have not mixed changes to #1942 behavior into this CI PR; I will keep those fixes separate if maintainers want a follow-up.

[kriswest]

I'll check this out when I have a minute - very keen to get this in. However, one change i would make is to separate it from the normal CI workflow so it can be run as its own task and be triggered as a CI check separately. The tests take an age to run (near 6 mins) and I wouldn't want to execute them on every PR. But I definitely DO want to be able to run them in AI workkflows, which I currently need to do manually and report results.

Perhaps we should only run on changes to packages/fdc3-conformance/src, toolbox/fdc3-for-web/fdc3-web-impl/src and packages/fdc3-agent-proxy/src ?

@Roaders WDYT?

[russlan23]

Moved the pack into .github/workflows/conformance.yml in b451bb2. It supports workflow_dispatch and workflow_call; ordinary coverage no longer installs Chromium or runs conformance.

The automatic paths use your three source directories, with toolbox/fdc3-conformance/src as the actual conformance path. I also included the harness/configuration and workflow files so changes to the check test themselves. Reports and failure traces are retained for seven days. The README notes that an unconditional required check would block unrelated PRs when path filters skip the workflow.

The YAML/trigger/permission checks and 12 path cases pass. I reran the complete browser pack: the three result-gate tests pass, and the pack still reports exactly RaiseIntentExistingInstanceRequired, RaiseIntentFailExistingInstanceRequired, and IntentContextMetadataWithAppMetadata. Those are unchanged from the earlier behavior report; no assertions were weakened or skipped in this workflow-only follow-up.