Split deterministic v1 tests from live E2Es

[xeophon]

Overview

Separate deterministic v1 coverage from live, provider-dependent end-to-end evaluations while preserving both test surfaces.

Changes

• Keep deterministic v1 tests on the full Python 3.11, 3.12, and 3.13 matrix through the existing e2e marker boundary.
• Run live v1 evaluations in a dedicated Python 3.12 job with bounded test concurrency and a job timeout.
• Keep Prime and Modal runtime placements local-only, matching the existing CI boundary.
• Scope provider credentials to the live job and skip that job for fork pull requests where secrets are unavailable.
• Remove the standalone Prime CLI installation from the deterministic matrix.
• Document the deterministic and live pytest selectors alongside the v1 fixtures.

Impact

Deterministic regressions remain covered across every supported Python version, while provider, runtime, server, and dynamic-tool behavior stays visible in a dedicated CI result with controlled resource usage.

Note

Split deterministic v1 tests from live E2E tests in CI

• Deterministic v1 tests now run in the main matrix job via -m "not e2e" without PRIME credentials; the main job also excludes tests/v1 from its own test run to avoid duplication.
• A new v1-e2e job (Python 3.12 only) runs live E2E tests via -m "e2e and not prime and not modal" with PRIME credentials, and is skipped for external PRs and Dependabot.
• Updates tests/v1/conftest.py comments to reflect the new split run instructions.
• A top-level permissions: contents: read block is added to test.yml.

Changes since #2127 opened

• Changed pytest-xdist parallel worker configuration from fixed count to auto-detection [4f1bbc9]

^{Macroscope summarized 28314f0.}

---

Note

Low Risk
Changes are limited to GitHub Actions wiring and test-selection markers; no application runtime or auth logic is modified.

Overview
CI now runs v1 in two lanes: the main Verifiers matrix (Python 3.11–3.13) runs non-v1 tests with --ignore=tests/v1, then deterministic v1 via -m "not e2e" without Prime credentials or a separate prime install. Live model E2Es move to a new v1-e2e job on Python 3.12 only (-m "e2e and not prime and not modal"), with a 120-minute timeout and PRIME_* secrets scoped to that job.

The live job is skipped for fork PRs and Dependabot (when secrets aren’t available). Workflow permissions: contents: read is added. tests/v1/conftest.py comments are updated to document the deterministic vs live pytest selectors.

^{Reviewed by Cursor Bugbot for commit 4f1bbc9. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 2951c19. Configure here.}

[macroscopeapp]

Approvability

Verdict: Approved

This PR reorganizes CI test execution by splitting deterministic v1 tests from live E2E tests into separate jobs. It's a standard workflow optimization that doesn't affect production behavior. The unresolved comment about -n 2 appears stale - the current code uses -n auto.

^{You can customize Macroscope's approvability policy. Learn more.}