Validation accepts logically-invalid hedges at a high rate — deterministic pre-checks proposal

[fashionmascherine-svg]

Thanks for open-sourcing this — the pipeline design is clean, and PR #61 (prompt caching) came out of reading it closely.

While evaluating the detector on the science tag, we ran a downstream audit of the validated pairs and found that a large majority aren't actually valid covers — there's a plausible outcome where both legs lose. Sharing the data; happy to send a follow-up PR.

Setup: commit 65aea7a, science tag, 2026-07-15. LLM base URL redirected to DeepSeek (one-line env patch); IMPLICATIONS_MODEL and VALIDATION_MODEL = deepseek-chat. Numbers below are from a single full run of one small tag, so rates may differ elsewhere — but the deterministic findings (bin math, window mismatch, cost ≥ $1) are model-independent: those pairs are invalid regardless of which LLM validated them.

Measured (one science run, 201 portfolios)

• Numeric pairs (counts / thresholds / magnitude bins on the same or nested variable), checked deterministically (bin math, no LLM): 87/94 = 92.6% invalid.
• Non-numeric pairs surviving a window+cost filter, re-checked with an adversarial "find a both-lose scenario" prompt: 16/17 rejected.
• After all checks: 0/201 pairs are simultaneously logically valid, economical (cost < $1), and executable (book depth > 0 at the quoted price).

Reproducible examples (each with an explicit both-lose scenario)

1. Threshold ladder — TARGET ≥12,500 measles cases in 2026 (YES) / COVER ≥2,300 measles cases in 2026 (NO). Both lose if the count is in [2300, 12500), e.g. 5,000: target loses (<12,500), cover loses (≥2,300).
2. Inverted magnitude direction — TARGET 10.0+ earthquake (YES) / COVER 9.0+ earthquake (NO). A magnitude-9.5 quake: target loses (no 10.0+), cover loses (there was a 9.0+).
3. Different count variables — TARGET exactly 9 quakes ≥5.5 (YES) / COVER exactly 0 quakes ≥6.5 (NO). 8×≥5.5 and 0×≥6.5: target loses (not exactly 9), cover loses (it IS exactly 0).
4. Wrong-side implication — TARGET New Coronavirus Pandemic in 2026 (YES) / COVER New pandemic in 2026 (NO). A novel-influenza pandemic makes "new pandemic" resolve YES → the NO cover loses; the coronavirus-specific target resolves NO → target loses too. The implication coronavirus-pandemic → pandemic is real, but the cover is on the wrong side, so it doesn't cover.

Root causes

• The contrapositive cover is derived without verifying the numeric/temporal relationship holds for the specific thresholds/deadlines.
• No hard gate on total_cost < 1 (many pairs cost ≥ $1 = guaranteed loss).

Proposal (two parts)

A. Deterministic pre-checks (zero-LLM). A numeric coverage check (enumerate the outcome space, with nesting for magnitude ladders, incl. a direction check for the inverted case); resolution-window compatibility; and an economic gate total_cost < 1. In our run these also cut LLM validation cost: numeric pairs that fail bin-math never reach the (more expensive) LLM validation step — fewer calls. As a downstream gate between candidate pairs and validation, it dropped ~57% of candidate pairs on science before any LLM call.

B. Stronger validation. An adversarial framing ("construct a concrete both-lose scenario; valid only if none exists") plus a stronger judge model both help. On the same 94 numeric pairs, scored against the deterministic ground truth: claude-opus-4-8 = 94.7% (89/94, 0 false positives, 5 false negatives) vs the current default deepseek-chat (v4-flash) = 7.4% (7/94, 87 false positives). The part-A checks are model-independent and catch the rest for free.

We've implemented A as a standalone downstream filter and can open a PR wiring it in as a pre-validation gate if you're open to it.