Skip to content

Run browser inference and GPU correctness checks in CI #521

Description

@sauravpanda

Suggested priority: P1 — protect the WASM-first runtime.

At a8cb07c, CI compiles WASM and runs npm pack --dry-run, but does not execute browser inference. The workspace suite passed 622 tests during review, with 43 ignored tests including 40 GPU tests. Compilation and synthetic CPU tests do not establish browser/GPU numerical correctness.

Scope and acceptance criteria

  • Add a browser CI job that loads the built npm package and a tiny pinned model fixture, then performs actual inference.
  • Cover worker initialization, CPU fallback, streaming Unicode, cancellation, reload/reset, and offline cache reuse.
  • Add scheduled/manual GPU runs that execute the ignored tests and fail explicitly if the required adapter is unavailable.
  • Compare CPU/GPU prefill and decode logits with documented tolerances; include mixed quantization and context-boundary cases.
  • Add pinned tokenizer IDs and small reference-generation fixtures from a trusted implementation, recording model/checksum, reference version, and generation settings.
  • Exercise supported browser/platform configurations with explicit pass/skip reporting.
  • Save runtime/browser/adapter metadata and failure logs as artifacts.
  • Measure load time, time to first token, and decode throughput on stable runners before enforcing performance thresholds.

Evidence: WASM CI job, ignored GPU integration tests.

Related: #448 covers model verification, #441 covers a Q4_K benchmark, and closed #450 added informational performance CI. Coordinate fixtures and coverage with those efforts.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    browserBrowser/WASM specifictestingTest coverage improvements

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions