Repository navigation
Add an offline shadow experiment for context relevance #42
Description
We want to learn whether Jev can identify useful pieces of context before letting it change what an agent reads. Build a toy experiment that shows which pieces it would keep while leaving the original context untouched. Report uncertainty and cache assumptions clearly. This follows issue #3's cost model and does not enable production filtering.
Follow-up to completed issue #3 and PR #30.
The first contribution is an offline, example-local kit with synthetic chunks and scripted Noul evidence. It demonstrates the experiment mechanics; it does not measure Jev's relevance accuracy or real savings.
How it works
flowchart TD
A["Synthetic task and context chunks"] --> B["Build both Noul request layouts"]
A --> H["Original context stays intact with caller"]
B --> C["Scripted fake adapter — no provider call"]
C --> D{"Complete usable evidence?"}
D -->|Yes| E["Proposed keep/drop sets; uncertainty retained"]
D -->|No| F["Unavailable: retain all; propose no drops"]
E --> G["JSON or Markdown report"]
F --> G
L["Evaluation labels and cost observations"] -.->|Report only| G
Synthetic example
The displayed values are scripted, as recorded in the sample result.
Acceptance
- Compare one request containing all chunks, with a Noul question per chunk, against a separate request per chunk. Preserve identity and exclude evaluation labels from adapter inputs and fake evidence.
- Pin
jev-1.13.0; version the example's relevance questions separately; preserve explicit criteria and the fixed untrusted-data note. - Freeze illustrative cutoffs: below
0.2would drop;0.2through0.8is uncertain and retained; above0.8is relevant and retained. These are not calibrated policy. - Keep original context byte-for-byte unchanged. Missing, malformed, cancelled, or wrong-model evidence is unavailable; incomplete turns produce no complete drop recommendation.
- Report proposed sets, uncertainty, relevance recall, failures, and byte/token proxies. Cost arithmetic requires dated prices and explicit segment-level token/cache observations. Aggregate-only cache counts yield unknown savings.
- Label bundled results synthetic demonstration, prevent fake results from claiming to be live, and keep JSON and Markdown reports consistent.
- Add a Mermaid flow and a visual aid using the repository's colors.
No production ContextScorer, root export, provider client, dependency, or Arena change. Phase 4 live shadow measurement and the integration decision remain open.
Verdict impact: None. Preserve the four verdicts, proposal-review question semantics, model pin, host ownership of authorization, and secret guards.
Activity
fix: address context shadow review feedback
fix: harden context shadow inputs and close test gaps