Skip to content

Add a gated end-to-end LLM smoke test + deploy healthcheck #520

Description

@dkirby-ms

Summary

Every existing test uses mock transports or is gated behind unset flags (OPENAI_LLM_TEST), so no real LLM is ever exercised in CI, and there is no deploy-time check that narration works. Add an opt-in e2e smoke test against a real endpoint and a deploy healthcheck.

Scope / Tasks

  • Add an opt-in e2e test (gated by an env flag) that calls the real provisioned endpoint via the chosen transport and asserts a valid, contract-passing completion (reuse validateLLMOutput).
  • Add a post-deploy healthcheck (script or container readiness probe) that issues one narration request and verifies an LLM (not fallback) response, surfacing fallback as a warning.
  • Fix/rename the misleading azure-llm-transport.integration.test.ts to reflect what it actually tests (or repoint it at the real Azure transport from Issue 3).
  • Document how to run the gated smoke test locally and in CI.

Acceptance Criteria

  • A gated e2e test passes against the live endpoint when the flag is set; skipped cleanly otherwise.
  • Deploy pipeline reports LLM reachability (pass/fallback) for the environment.
  • The vestigial test name no longer misrepresents Azure coverage.

Dependencies

#510 (config flows), #511 (transport), #514 (a real player-visible path to exercise).

Size

M

Derived from the LLM narration gap analysis (Volo) and decomposition (Elminster).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ciContinuous integration and deliveryinfrallmLLM narration integrationsquad:khelbenAssigned to Khelben (CI/CD Dev)squad:minscAssigned to Minsc (Tester)testing

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions