refactor(bolt): compile Jolt verifier through typed plans

[quangvdao]

Stacks on top of Markos' jolt-v2/equivalence stack, currently based at 1faea61a (perf(equivalence): tighten Bolt prover gate). This collapses Quang's side of the stack into one PR: the earlier quang/bolt-stack work and the follow-up quang/bolt-verifier-program-refactor branch are combined here.

This PR advances the Bolt-generated Jolt verifier from stage-shaped generated Rust toward a typed verifier-program pipeline. It preserves the current full-field Jolt semantics and proof serialization, but moves verifier facts into compiler-owned typed plan data and a reusable runtime instead of late string matching and stage-local helper code.

Diff shape at the current head: 96 files changed, +30,835 / -13,674.

Stack shape

Before this update:

Markos stack:      jolt-v2/generated-roles -> jolt-v2/equivalence
Quang PR:          quang/bolt-stack
Quang follow-up:   quang/bolt-verifier-program-refactor

After the collapse:

Markos stack:      jolt-v2/generated-roles -> jolt-v2/equivalence
Single Quang PR:   quang/bolt-stack

Major changes

1. Typed generated verifier surface

• Replaces stringly verifier surfaces with typed plan records for relation dispatch, field expressions, scalar/point operands, sumcheck claims, batch operands, opening claims, opening batches, point sources, eval families, and relation outputs.
• Keeps strings primarily as diagnostic/serialization names; execution contracts now use typed rows, typed relation IDs, and typed value references.
• Adds cleanup gates in crates/bolt/tests/verifier_cleanup.rs so converted surfaces stay converted: relation string sites, batch operand strings, claim input opening strings, point-concat input strings, field-expression operand constants, stage-local macros, indexed-eval prefix APIs, and handwritten expected-output helpers are all guarded.

2. bolt-verifier-runtime extraction

• Adds crates/bolt-verifier-runtime as the shared generic verifier runtime crate.
• Deletes generated crates/jolt-verifier/src/stages/common.rs and the old verifier_common.rs.template path.
• Moves generic mechanics into the runtime: plan structs, ValueStore, transcript helpers, sumcheck driver plumbing, opening equality checks, point/scalar/vector/eval-family storage, relation-output execution scaffolding, and structural plan errors.
• Keeps Jolt-specific verifier math in crates/jolt-verifier/src/stages/jolt_relations.rs and generated from verifier_jolt_relations.rs.template.

3. Top-level typed verifier program

• Adds a top-level verifier-program model in crates/jolt-verifier/src/verifier.rs: proof slots, checkpoints, targets, evaluation policy, typed program steps, and execution artifacts.
• Routes verifier entrypoints through typed program execution while preserving the public proof layout.
• Treats numbered stages as proof-layout and diagnostic scopes, not as the only model for verifier control flow.

4. CPU-to-Rust verifier planning boundary (S2.75)

• Adds compiler-side verifier planning modules such as rust_target_plan.rs, verifier_plan.rs, verifier_values.rs, verifier_sumcheck_rows.rs, verifier_opening_rows.rs, verifier_program_rows.rs, and related row/value modules.
• Centralizes Stage 2-7 planning through shared planning functions for program steps, transcript flow, verifier sumchecks, value and relation outputs, opening flow, relation-local inputs, and target validation.
• Moves verifier-mode sumcheck-flow, batch/driver consistency, relation-output validation, value-source conflict checks, and opening-flow checks onto VerifierStagePlan instead of repeated stage-local string sets and maps.
• Routes Stage 2 through the shared planning boundary as well; it now uses the shared CPU row types and shared verifier-plan validation path.

5. Output claims, relation outputs, and typed value sources

• Lifts Stage 3, Stage 4, Stage 5, Stage 6, and Stage 7 output-claim math into generated plan data.
• Introduces typed scalar, point, field-vector, eval-family, local-scalar, local-input, and relation-output references.
• Renames and reshapes the generated Rust surface around RelationOutputPlan, relation_outputs, and STAGE*_RELATION_OUTPUTS instead of older output-claim naming.
• Names terminal expected_output scalar values separately from input sumcheck claim_value data.
• Removes stale expected_stage67_* helper paths from generated/runtime code and gates handwritten_expected_output_functions: 0.
• Starts lowering concrete relation-output values, including Stage 5/6 read-RAF paths, Stage 6 bytecode output terms, hamming and increment output scalars, and shared relation-local inputs.

6. Eval-family cutover (S4)

• Adds explicit piop.sumcheck_eval_family, compute.sumcheck_eval_family, and cpu.sumcheck_eval_family rows.
• Parses Stage 5 instruction read-RAF and Stage 6 bytecode read-RAF family membership from typed MLIR/CPU rows rather than oracle prefixes, raw eval ordering, or generated symbol spelling.
• Emits NamedEvalFamilyPlan rows, seeds them into ValueStore as field-vector values, and reuses those vectors in relation-output plans.
• Removes the old eval-prefix reconstruction path from generated verifier/runtime execution and gates against reintroducing indexed_evals_by_prefix* APIs.

7. Stage 5/6/7 read-RAF and bytecode planning seams

• Adds stage5_instruction_read_raf_plan.rs and stage6_bytecode_read_raf_plan.rs as explicit compiler-side planning seams.
• Moves bytecode read-RAF rows and output terms into typed plan constants rather than hand-authoring the generated Rust block directly in the Stage 6 emitter.
• Keeps the bytecode evaluator quarantined on the Jolt side because the row encoding is protocol-specific.
• Shares Stage 6/7 token helpers in plan_tokens.rs and replaces kernel ABI match blocks with table-driven ABI checks.

8. Shared polynomial helpers

• Adds verifier-facing helper coverage in jolt-poly for indexed equality, less-than, identity, and related structured polynomial evaluation paths.
• Reuses LtPolynomial and indexed equality helpers from the verifier runtime instead of open-coded local formulas.

9. Prover and equivalence adapters

• Keeps prover/verifier proof serialization aligned while the verifier execution shape changes.
• Updates jolt-prover stages, especially Stage 6/8 plumbing, to stay aligned with the generated role outputs and opening-flow shape.
• Updates jolt-equivalence plan adapters, tamper tests, generated-stage adapters, and oracle wiring for typed verifier plans.
• Enables the Stage 3+ Bolt parity/tamper gate by removing the stale ignore from bolt_stage3_batched_real_muldiv_self_parity.

10. Refactor plan docs

• Rewrites crates/bolt/GOAL.md around the typed verifier-program objective, audit tiers, non-regression contracts, current S2.75-S5 status, and perf/readability/LOC gates.
• Adds crates/bolt/VERIFIER_PROGRAM_REFACTOR_PLAN.md as the main S2-S6 verifier-program plan.
• Adds crates/bolt/PROVER_PROGRAM_REFACTOR_PLAN.md to mirror the longer-term prover direction while keeping verifier work as the priority.

Latest recorded cleanup metrics

Latest recorded verifier_cleanup pass from the worklog after the Stage 2 verifier-plan cutover:

generated_surface_loc: 6055
tier_b_jolt_verifier_core_loc: 696
total_loc: 6751
relation_string_sites: 0
relation_indexed_eval_prefix_sites: 0
handwritten_expected_output_functions: 0

The earlier baseline was roughly 21.5k LOC of generated jolt-verifier, with Stage 6/7 alone at roughly 13.2k LOC. The current shape is much smaller and, more importantly, the remaining generated surface is mostly typed declarative data plus thin wrappers.

What this PR does not finish

This PR is still not the final verifier compiler architecture. The remaining risks are narrower and called out in VERIFIER_PROGRAM_REFACTOR_PLAN.md / GOAL.md:

• Stage 2 now uses the shared verifier-plan boundary, but its relation-output formula table is still handwritten Rust planning data. The next completion audit should decide whether that is acceptable for this PR or must be made more declarative before closure.
• Prover-mode Stage 2 validation remains stage-local for prover-only input-opening and kernel-binding details that are not part of the verifier runtime plan.
• Eval-family membership is now explicit plan data for the Stage 5/6 consumers here; future family-shaped relations should follow the same typed-row rule instead of rebuilding prefix logic.
• The bytecode row encoder remains Jolt-specific Tier B logic. S6 can make it typed table data later, but this PR intentionally stops short of generalizing bytecode semantics into the generic runtime.
• The final S2.75-S5 completion audit still needs to separate true remaining architecture gaps from completion gates: semantic/tamper behavior, host/host+zk e2e, and perf stability.

Verification

Recorded local verification across the final slices includes:

cargo fmt --check
cargo check -p bolt -q
cargo check -p bolt -p jolt-verifier -p jolt-prover -p jolt-equivalence -q
cargo clippy -p bolt -p jolt-verifier -p jolt-prover -p jolt-equivalence -q --all-targets -- -D warnings
cargo nextest run -p bolt --test commitment_ir --cargo-quiet stage2_rust_targets_extract_and_compile stage3_rust_targets_extract_and_compile stage4_rust_targets_extract_and_compile stage5_rust_targets_extract_and_compile stage6_rust_targets_extract_and_compile stage7_rust_targets_extract_and_compile
cargo nextest run -p bolt --lib --cargo-quiet relation_output verifier_plan
cargo nextest run -p bolt --test verifier_cleanup --no-capture
cargo nextest run -p jolt-equivalence --test bolt_commitment --cargo-quiet
cargo clippy -p jolt-equivalence -q --test bolt_commitment -- -D warnings
cargo nextest run -p jolt-core muldiv --cargo-quiet --features host
cargo nextest run -p jolt-core muldiv --cargo-quiet --features host,zk
git diff --check

The latest three-sample 2^20 SHA2-chain perf oracle also passed:

JOLT_BOLT_PERF_SAMPLES=3 cargo nextest run -p jolt-equivalence --test bolt_perf --release --cargo-quiet --run-ignored only --no-capture bolt_sha2_chain_2_20_core_vs_bolt_perf_oracle

Latest sampled summary:

verify_ms ratio: 1.089x
prove_ms mean ratio: 1.168x, 95% CI [0.876x, 1.460x]
proof_bytes ratio: 1.363x
peak_rss_mb ratio: 1.339x

CI should still be treated as the source of truth for the full matrix.

Review notes

• This is a full-cutover refactor, not a compatibility layer: old generated helper paths were deleted where their typed replacements landed.
• bolt-verifier-runtime should stay boring and generic. New protocol facts should be represented as MLIR/codegen planning output, not runtime helpers that infer Jolt meaning from names.
• Some generated declarative data grew while handwritten Rust shrank. That is intentional only when the generated data is typed, auditable, and compiler-owned.
• The highest-value next step is the completion audit for S2.75-S5, especially Stage 2's handwritten formula table, semantic/tamper coverage, host/zk e2e, and perf margin.

---

AI-authored PR description update posted by Cursor assistant (model: GPT-5) on behalf of the user (Quang Dao) with approval.

[github-actions]

Warning

This PR has more than 500 changed lines and does not include a spec.

Large features and architectural changes benefit from a spec-driven workflow.
See CONTRIBUTING.md for details on how to create a spec.

If this PR is a bug fix, refactor, or doesn't warrant a spec, feel free to ignore this message.