Skip to content

docs: ContextScorer cost model and go/no-go (phase 4) - #30

Merged
BunsDev merged 1 commit into
mainfrom
docs/context-scorer-cost-model
Sep 22, 2026
Merged

BunsDev merged 1 commit into
mainfrom
docs/context-scorer-cost-model

Conversation

@BunsDev

@BunsDev BunsDev commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

What changed

Adds docs/context-scoring-cost-model.md, the phase 4 cost model for per-chunk context scoring, and links it from docs/architecture.md (host seams / ContextScorer) and roadmap phase 4. It ticks only "Compare scoring/re-prefill costs with forfeited prefix-cache reuse". The egress review, shadow experiment and integration decision items stay open. No ContextScorer code is added.

Decision: no-go for ContextScorer integration code; conditional go for a shadow experiment on synthetic context only.

  • Break-even drop fraction (with s'=0 and the scored block placed last): d* = s·(1 − r) + ρ·(1 + ε). Here r = 0.1 is the cache-read multiplier (OpenAI and Anthropic docs) and ρ = p_J/P is Jev's price relative to the proposer's: $0.042/Mtok against $1.75/Mtok for gpt-5.3-codex, so ρ = 0.024.
  • A context block that stays stable across turns needs d ≥ 0.84 at 10 turns and 0.93 in the long-session limit. That is where scorer recall errors hurt most, so it is a no-go. A block that is reassembled every turn anyway needs d ≈ 0.03, which is cheap in money.
  • Jev latency is a serial cost: 206 ms p50, 474 ms p95, 975 ms max over 132 recorded receipts from four live runs (the often-quoted 257 ms is the mean of run 1 alone).
  • The only recorded token-usage workload (Codex CLI arena, ~29.6k input tokens, 57.6–91.1% cached) contains one 92-byte scorable file. Nothing there to score.

Every number cites a recorded artifact at a commit SHA (playground 2c6cac9, this repo 3b1704f) or an official doc URL checked 2026-09-22. Unknowns stay explicit parameters: context stability s, drop fraction d, chunk shape, turns per session, prefill time per token τ, the proposer model the arena used, Jev latency at 10–32k-token states, and scorer recall. The doc specifies the shadow experiment that would measure them.

Closes #3

Verdict impact

None. No decision-table row, question, fixture, threshold or model pin changes.

Verification

  • pnpm typecheck (exit 0)
  • pnpm test (exit 0; 84 pass, 0 fail)
  • Commits are signed
  • No provider call, credential, execution path, or new verdict added
  • If the decision table or a question changed: not applicable

🤖 Generated with Claude Code

Compare per-chunk Jev scoring plus re-prefill against forfeited
prefix-cache reuse, using recorded Jev latency (132 receipts), arena
token usage, and official TypeSafe, OpenAI and Anthropic pricing.
Decision: no-go for ContextScorer integration code; conditional go for
a shadow experiment on synthetic context only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
jev-harness Ready Ready Preview Sep 22, 2026 1:15pm UTC

Request Review

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The model incorrectly treats request count as a fixed overhead ratio and does not fully check Jev’s state-plus-question token limit.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 Medium severity · 1 Low severity

Open (2)
What changed in this PR

Adds the phase 4 ContextScorer cost model and records a no-go for integration code, while keeping a synthetic shadow experiment conditional.

Changes:

  • Adds pricing, latency, cache, break-even, egress, and experiment guidance.
  • Links the decision from architecture documentation.
  • Marks the cost-model roadmap item complete; no runtime or verdict changes.
File Description
docs/​context-scoring-cost-model.md New ContextScorer cost model and decision record
docs/​architecture.md Links the cost model and decision
docs/​roadmap.md Updates phase 4 progress

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +259 to +260
5. **The request fits Jev's limits.** `K` plus the longest question fits in
32k tokens per request (TS), or the scorer splits requests and pays the

### Break-even drop fraction `d*`

With `ε = 0.1`, meaning one fan-out request per turn:
@BunsDev
BunsDev merged commit f3a7a10 into main Sep 22, 2026
4 checks passed
@BunsDev
BunsDev deleted the docs/context-scorer-cost-model branch September 22, 2026 14:15

This branch was successfully deployed

1 active deployment
Preview — cbf403b7 Deployed Sep 22, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Phase 4: ContextScorer cost model before any code

2 participants