Skip to content

Add an offline shadow experiment for context relevance #42

Closed

Description

@snowopsdev

We want to learn whether Jev can identify useful pieces of context before letting it change what an agent reads. Build a toy experiment that shows which pieces it would keep while leaving the original context untouched. Report uncertainty and cache assumptions clearly. This follows issue #3's cost model and does not enable production filtering.

Follow-up to completed issue #3 and PR #30.

The first contribution is an offline, example-local kit with synthetic chunks and scripted Noul evidence. It demonstrates the experiment mechanics; it does not measure Jev's relevance accuracy or real savings.

How it works

flowchart TD
    A["Synthetic task and context chunks"] --> B["Build both Noul request layouts"]
    A --> H["Original context stays intact with caller"]
    B --> C["Scripted fake adapter — no provider call"]
    C --> D{"Complete usable evidence?"}
    D -->|Yes| E["Proposed keep/drop sets; uncertainty retained"]
    D -->|No| F["Unavailable: retain all; propose no drops"]
    E --> G["JSON or Markdown report"]
    F --> G
    L["Evaluation labels and cost observations"] -.->|Report only| G
Loading

Synthetic example

Synthetic timeout example: configuration would be kept, the uncertain caller retained, and styling dropped. The original context keeps all three notes.

The displayed values are scripted, as recorded in the sample result.

Acceptance

  • Compare one request containing all chunks, with a Noul question per chunk, against a separate request per chunk. Preserve identity and exclude evaluation labels from adapter inputs and fake evidence.
  • Pin jev-1.13.0; version the example's relevance questions separately; preserve explicit criteria and the fixed untrusted-data note.
  • Freeze illustrative cutoffs: below 0.2 would drop; 0.2 through 0.8 is uncertain and retained; above 0.8 is relevant and retained. These are not calibrated policy.
  • Keep original context byte-for-byte unchanged. Missing, malformed, cancelled, or wrong-model evidence is unavailable; incomplete turns produce no complete drop recommendation.
  • Report proposed sets, uncertainty, relevance recall, failures, and byte/token proxies. Cost arithmetic requires dated prices and explicit segment-level token/cache observations. Aggregate-only cache counts yield unknown savings.
  • Label bundled results synthetic demonstration, prevent fake results from claiming to be live, and keep JSON and Markdown reports consistent.
  • Add a Mermaid flow and a visual aid using the repository's colors.

No production ContextScorer, root export, provider client, dependency, or Arena change. Phase 4 live shadow measurement and the integration decision remain open.

Verdict impact: None. Preserve the four verdicts, proposal-review question semantics, model pin, host ownership of authorization, and secret guards.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions