Repository navigation
ci: add Daily test budgets and seven-day rotation - #3254
Conversation
Merger | Merge ReadinessHead: Updated: 2026-11-03 16:14:53 UTC Status: MERGED Merger decision: MERGED (shadow assessment; no merge executed).
Next owner: None. Required next step: No further merge action. This agent did not execute a merge. Merge observed: wxyucs; commit Merge reliability: 0.00 / 1.00 (WITHHELD)
Evidence subtotal: 0.00; applicable ceiling: 0.00. Final score = min(subtotal, ceiling). Policy:
AI review: NOT_ASSESSED. PR is no longer open. Review findings: unknown; P0/P1 findings in that review: unknown. Findings are not a count of currently unresolved threads. CI summary: No check results reported. Check source: Next action: No further action; this report is final. CI check details
This is an uncalibrated evidence score, not a probability, correctness guarantee or approval. Zero means the assessment is withheld or lacks positive evidence, not a proven-bad patch. Automatic merging is disabled. New commits or base changes invalidate earlier evidence; never authorize a merge from this score alone. |
wxyucs
added
kind/improvement
ChatGPT Codex Connector
Merge Protections🟢 All 2 merge protections satisfied — ready to merge. Show 2 satisfied protections🟢 Require kind label
🟢 Require version label
|
mergify
Bot
added
area/docs
vsag-bot
added
the
area/tools
|
Jev-driven classification Applied: Option probabilities
Model: |
There was a problem hiding this comment.
🟡 Changes recommended
Critical defects in suppression paths, capability-aware test selection, skip detection, and workflow permissions break Linux test planning, reject expected exclusions, allow selected tests to be skipped unnoticed, and invalidate manual performance invocations.
4 open issues
What changed in this PR
Adds opt-in Daily regression planning to VSAG’s CI layering work in #3225, with a 360-minute cumulative test-execution budget and seven-day configuration rotation. Regression scheduling remains disabled; full runtime and budget acceptance are still pending.
Changes:
- Adds frozen planning context, configuration selection, execution records and strict result reconciliation.
- Adapts HGraph, Pyramid and SINDI tests for Daily selection while preserving default generator behavior.
- Integrates evidence collection across Daily lanes, rotates SIMD configurations and replaces example shell loops with manifest-based execution.
| File | Description |
|---|---|
tests/test_sindi.cpp |
Adapts finite generators for Daily selection. |
tests/test_pyramid.cpp |
Adapts finite generators for Daily selection. |
tests/test_index/test_index_build.cpp |
Adds build-phase timing events. |
tests/test_hgraph.cpp |
Enumerates metric/dimension groups and emits phase events. |
tests/test_daily_plan.cpp |
Tests generator defaults, selection and validation. |
tests/scripts/test_daily_workflow.py |
Adds workflow-wiring regression checks. |
tests/scripts/test_daily_plan.py |
Tests planning, rotation and result handling. |
tests/fixtures/framework/daily_test_plan.h |
Adds generator selection and phase-event helpers. |
scripts/testing/test_parallel_by_name.sh |
Routes explicit Daily runs to the planner. |
scripts/testing/test_parallel_bg.sh |
Routes Daily coverage runs to the planner. |
scripts/testing/daily_timing_history.json |
Initializes explicitly unmeasured timing history. |
scripts/testing/daily_run.py |
Inventories and executes selected configurations. |
scripts/testing/daily_report.py |
Reconciles results and reports budget usage. |
scripts/testing/daily_plan.py |
Freezes context and builds seven-day plans. |
scripts/testing/daily_parameters.py |
Enumerates finite generator choices from source. |
scripts/testing/daily_measure.py |
Measures non-Catch test commands. |
scripts/testing/daily_matrix.json |
Defines ownership and SIMD rotation. |
scripts/testing/daily_entry.py |
Bridges existing runners to Daily execution. |
scripts/testing/daily_budget_policy.json |
Defines budgets and configuration timeout. |
docs/agents/daily-regression.md |
Documents selection, replay and acceptance limits. |
docs/agents/ci-routing.md |
Links Daily regression guidance. |
docs/agents/build-and-test.md |
Links Daily regression guidance. |
.github/workflows/daily_test.yml |
Wires planning, lane evidence and final reporting. |
.github/actions/python-build-and-test-test-python-x86/action.yml |
Wraps Python tests with Daily measurement. |
.github/actions/python-build-and-test-test-python-aarch64/action.yml |
Wraps ARM Python tests with Daily measurement. |
.github/actions/check-compatibility-compatibility/action.yml |
Measures compatibility-test execution. |
.github/actions/asan-build-and-test-test-example-x86/action.yml |
Uses measured manifest-based example execution. |
.github/actions/asan-build-and-test-test-example-aarch64/action.yml |
Uses measured manifest-based full example execution. |
.github/actions/asan-build-and-test-test-asan-macos-arm64/action.yml |
Measures full macOS example execution. |
Review effort: Balanced
vsag-lgtm
left a comment
vsag-lgtm
left a comment
There was a problem hiding this comment.
Automated review @ 4af4eb40 — 4 inline suggestion(s).
4af4eb4 to
5079deb
Compare
Reviewer | Review completeAutomated pull request review completed. Model: No actionable inline findings were found. |
vsag-lgtm
left a comment
vsag-lgtm
left a comment
There was a problem hiding this comment.
All prior review threads are resolved and the incremental changes address the earlier comments. No new critical, warning, or suggestion findings in this push.
There was a problem hiding this comment.
🟡 Changes recommended
The summary can report correctness/completeness as passed while full lint is still running or has failed because full-lint-clang-tidy-check is missing from its needs dependencies.
1 open issue
4 resolved since last review
Review effort: Balanced
vsag-bot
left a comment
vsag-bot
left a comment
There was a problem hiding this comment.
Reviewer | Code review
Automated pull request review completed.
Model: gpt-6.1-sol; reasoning effort: high.
Review scope: high (2225 changed lines across 32 files).
No actionable inline findings were found.
Review coverage checks completed; this does not prove absence of defects.
Static review only; compilation and runtime tests are outside this review's scope.
File coverage: 32/32; observed passing test runs: 0.
Reviewed commit 5079deb.
5079deb to
1cfdefd
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Splitting the test into separate invocations resets the backend cursor, leaving all SQ8 configurations on block_memory_io and losing the previous mmap_io coverage.
1 open issue
1 resolved since last review
Review effort: Balanced
vsag-bot
left a comment
vsag-bot
left a comment
There was a problem hiding this comment.
Reviewer | Code review
Automated pull request review completed.
Model: gpt-6.1-sol; reasoning effort: high.
Review scope: high (2255 changed lines across 32 files).
No actionable inline findings were found.
Review coverage checks completed; this does not prove absence of defects.
Static review only; compilation and runtime tests are outside this review's scope.
File coverage: 32/32; observed passing test runs: 0.
Reviewed commit 1cfdefd.
1cfdefd to
093f6ba
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
The cross-platform execution and evidence-collection changes still lack complete fixed-source validation on intended runners, requiring maintainer review of macOS/ARM, Python, coverage and compatibility runtime evidence before merging or enabling regression scheduling.
No open issues 🎉
1 resolved since last review
Review effort: Balanced
vsag-bot
left a comment
vsag-bot
left a comment
There was a problem hiding this comment.
Reviewer | Code review
Automated pull request review completed.
Model: gpt-6.1-sol; reasoning effort: high.
Review scope: high (2442 changed lines across 32 files).
No actionable inline findings were found.
Review coverage checks completed; this does not prove absence of defects.
Static review only; compilation and runtime tests are outside this review's scope.
File coverage: 32/32; observed passing test runs: 0.
Reviewed commit 093f6ba.
vsag-lgtm
left a comment
vsag-lgtm
left a comment
There was a problem hiding this comment.
Automated review @ 093f6baa — 2 inline suggestion(s).
| state = shards.execute(command, directory, directory / "tmp", | ||
| directory / "enumeration.log", 120, cancelled) | ||
| if state["exit_code"] != 4 or not manifest.is_file(): | ||
| raise dp.PlanError("HGraph inventory failed: " + group[0]["name"]) |
There was a problem hiding this comment.
[warning] Hard-coded exit code 4 for the HGraph enumeration helper is brittle.
The enumeration command uses Catch2's SKIP("Daily configuration enumeration only") and then asserts that the process exits with 4. This value is Catch2's "all tests skipped" return code only in certain versions/conditions. If Catch2 behavior changes, if a future version assigns a different return code, or if the binary starts returning another non-zero code, the Daily planner will abort with a generic "HGraph inventory failed" error that gives no hint about the actual exit code. Please replace the magic number with a named constant (e.g. CATCH2_ALL_SKIPPED_EXIT_CODE) and include the received state["exit_code"] in the error message.
| os.chdir(args.cwd) | ||
| os.execvp(command[0], command) | ||
| ctx = dp.validate_context(dp.load(os.environ["VSAG_DAILY_CONTEXT"])) | ||
| lane = os.environ["VSAG_DAILY_LANE"] + "-" + args.name |
There was a problem hiding this comment.
[suggestion] Validate the required Daily environment variables before use.
After confirming VSAG_DAILY_CONTEXT is present, the code immediately reads VSAG_DAILY_LANE and VSAG_DAILY_ENVIRONMENT. If a caller sets only the context but forgets the lane/environment variables, this raises an uncaught KeyError with a stack trace rather than a clear PlanError. Add an explicit check that all three variables are set and raise PlanError("Missing required Daily environment variables ...") if any are absent.
Freeze a reproducible Daily context and enumerate explicit HGraph, Pyramid and SINDI configurations into daily core and seven rotation slots. Report cumulative test budgets with strict result reconciliation, preserve coverage and default test paths, and keep regression scheduling disabled. Related: antgroup#3225 Signed-off-by: Xiangyu Wang <wxy407827@antgroup.com> Assisted-by: Codex:GPT-6
093f6ba to
6aca2e7
Compare
vsag-lgtm
left a comment
vsag-lgtm
left a comment
There was a problem hiding this comment.
Automated review @ 6aca2e7c — no new findings; 2 prior agent threads remain unresolved.
There was a problem hiding this comment.
🔵 Needs a closer look
Maintainer review is needed because fixed-source Daily runtime acceptance remains unvalidated across ARM/macOS, Python, coverage, and compatibility lanes, with documented HGraph failures and long-tail timeouts requiring assessment.
No open issues 🎉
Review effort: Balanced
vsag-bot
left a comment
vsag-bot
left a comment
There was a problem hiding this comment.
Reviewer | Code review
Automated pull request review completed.
Model: gpt-6.1-sol; reasoning effort: high.
Review scope: high (2448 changed lines across 33 files).
No actionable inline findings were found.
Review coverage checks completed; this does not prove absence of defects.
Static review only; compilation and runtime tests are outside this review's scope.
File coverage: 33/33; observed passing test runs: 0.
Reviewed commit 6aca2e7.
What changed
Related: #3225 (the umbrella issue remains open).
This introduces an opt-in Daily regression planner with a 360-minute cumulative test-execution budget, daily core coverage, and seven disjoint extension slots. Parallel execution and repeated platforms/sanitizers are charged separately; build and queue time are outside this test budget. Excess budget warns, while assertion failures, timeouts, skipped selected cases and missing results fail.
sh/[[loop with the existing manifest-based runner; align Daily TSan setup with the maintained suppression/BLAS settings.Validation
Validated the proposed source based on main
922cce7e:make test_parallelwith GCC 13/system OpenBLAS passed: unit, HGraph and other functional groups all succeeded. Compilation used 32 jobs in an isolated development checkout; no host swap/cleanup steps were reproduced.The first release-mode manual weekday trial selected 557 configurations from 482 core + 501 weekly extensions. It exposed:
82.699996948 > 84.150001526failed.5 == 10failed.The initial trial also exposed an evidence-reader bug: Catch2 stores captured streams inside XML. That reader was corrected and regression-tested; original trial evidence is retained and is not relabeled as a successful full run. A stale cross-build inventory used during an initial ASan probe was rejected; fresh plans from the matching binary passed the five configurations above.
Review follow-up (5079deb)
Draft / remaining acceptance
The maintainer explicitly authorized this Draft PR with the exceptions listed here. Do not merge or enable regression scheduling yet.
TestGeneralcontracts and complete quantizer/tune groups. Further contract deduplication, richer phase accounting, environment calibration and measured budget acceptance remain open in [improve](ci): separate PR, main and daily validation layers #3225.Lint summary follow-up (1cfdefd)
The final Daily summary now waits for full lint and includes its result. Regression tests exercise successful, failed, cancelled and skipped lint results and verify that lint adds no test-budget time. All 33 Daily helper/workflow tests and actionlint passed. Regression scheduling remains disabled; this change does not claim full Daily acceptance.
Disk IO coverage follow-up (093f6ba)
0,1,2,2,0,1; a real SQ8+mmap Daily configuration passed 48,065 assertions.make test_parallel: unit/HGraph/other groups all passed, including the new regression.tests/test_index/test_index_update.cpp:53. It is reproducible in the earlier pre-fix ASan validation logs as well. Assertion success is not a sanitizer-clean claim; this unrelated fixture issue remains open, alongside the previously listed acceptance gaps.C API CI stability follow-up (6aca2e7)
vsag_c_api basic test: the initial self-search did not include its own ID. The container hook error wrapped this test failure; the subsequent leak report came from assertion unwinding before manual index destruction.aea0cf04. This C API fixture and the production algorithm were not changed by the Daily implementation.make test_parallelpassed all unit/HGraph/other groups. Clang 20 TSan passed 28 concurrent cases / 2,252,316 assertions. All 33 Daily helper/workflow tests, clang-format 20, docs and diff checks passed.Risk and rollback
Only explicit Daily context enables parameter selection. No Daily context preserves the original generator product. Revert this change to restore the previous manual Daily execution; keep regression scheduling disabled until a fixed-source manual acceptance succeeds.
Public guidance:
docs/agents/daily-regression.md. Detailed operational evidence is retained privately.