Repository navigation
bench: report recall@10 with tie bounds - #83
Merged
Merged
Conversation
detect_extension_points breaks score ties on (module, lineno) — the alphabet. When a band of equally-scored candidates straddles rank 10, the alphabet, not the ranking, decides which labels count as hits. Measured across all seven targets: 17 of 34 labels are tie-decided, so the published 18/34 is one sample from the interval 6..23. score.py now reports tie bounds per target and in aggregate. Two pure functions, tie_bounds and recall_bounds, are covered by the framework-free self-check. Output-only: the ranking is untouched and the benchmark still reads 18/34. Two findings recorded in benchmarks/README.md: - JUnit and SQLAlchemy are NOT the same problem. All 18 candidates tied at 1.00 in JUnit live in org.junit.jupiter.api.extension and implement Extension — every one is a genuine extension point, and the 7 labels are a partial list of 18 correct answers. That band is a metric artefact, not a ranking failure. SQLAlchemy's 50-wide band really does mix real seams with internal machinery. - The unclamped signal sum is anti-correlated as a tiebreak (18/34 -> 15/34), as is ranking by signal count. The clamp is not hiding a usable lever. The regression canaries were never canaries: click's 4/4 has a lower bound of 2/4 and pluggy's 3/3 a lower bound of 0/3. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MODScan attack-surface diff —
|
MODScan extension-point diffNo breaking changes to extension points |
This was referenced Jul 24, 2026
Rinkia
added a commit
that referenced
this pull request
Jul 24, 2026
…king (#85) The detector breaks equal scores on (module, lineno) — alphabetically. A reader taking the top ten off a band of eighteen equally-scored candidates was reading the alphabet, and nothing in the output said so. PR #83 stopped the benchmark telling itself that story; this stops the product telling it to users. `modscan detect` now prints, above the table, how many candidates share the score at the cutoff and how many of them are shown: **18 candidates score exactly 1.00, and this list shows 12 of them.** The 6 left out are not ranked lower — ties are broken by module name, so the cut through this band is alphabetical, not evidence-based. When the band is fully visible the note says so instead of claiming truncation, and when the cutoff is unambiguous there is no note at all — a warning that fires on every run is one nobody reads. `--json` gains `tied_with` per point, since a machine consumer that re-ranks or truncates the payload needs the same fact and cannot read Markdown. The field is additive; `modscan diff` compares ids, category and kind, not the payload shape. Ties are measured on the full ranked list before `--limit` cuts it — computing them afterwards would describe the truncated list rather than the ranking. Presentation only: no weight, no signal and no ordering changed. Benchmark unchanged at 18/34, bounds 6..23. examples/detect-markdown.md is regenerated from its committed recipe. It was already stale — it predated the MS-MOD-* signal ids from #56 — and it turns out to be the exact failure this change fixes: twelve rows shown out of eighteen tied at 1.00. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
4 tasks done
Rinkia
added a commit
that referenced
this pull request
Jul 24, 2026
…86) Measuring ties in #83 exposed that every target here scores its labels inside a tied band. click's perfect 4/4 has a lower bound of 2/4; pluggy's 3/3 has a lower bound of 0/3. A benchmark of seven targets had, in effect, no regression guard — the two designated canaries could not falsify anything on their own. flask 3.1.2 was picked by measuring candidate distributions first, not by taste. Its ranking is graded rather than flat — 3 candidates at 1.00, 6 at 0.90, then exactly ONE at the rank-10 cutoff — so its recall@10 has no tie bounds at all: worst case equals best case. It is the only target of the eight whose number is pure evidence. flask 3.1.2 | 98 candidates | 6 labels | ranks 2, 3, 10, 38, 39, 98 | 3/6 Aggregate 18/34 -> 21/40, bounds 6..23 -> 9..26. All six new labels are tie-free, so the tie-decided share falls from 17/34 to 17/40. What it already says, recorded in benchmarks/README.md: - MethodView ranks 98 of 98, dead last. Arguably Flask's most-used class-based extension point, and every existing signal is blind to it. - DefaultJSONProvider ranks 1st while JSONProvider — the class the docstring explicitly says to subclass — ranks 10th. Role-suffix and base-role reward the concrete implementation over the contract it implements. - Dynamic imports take ranks 4-9 and push Flask and Blueprint below them: the pygments failure again, on a package with no plugin registry at all. Labels quote the docstrings shipped in the 3.1.2 wheel, per the rule about preferring docs pinned to the measured version. Two exclusions are recorded with their reasons: Flask itself (its shipped docstring never invites subclassing) and Blueprint (you construct one and register the instance — calling an API). Flask's largest extension surface, the before_request / errorhandler / template_filter decorators, cannot be labelled at all: they are methods on the app object and the parser only takes top-level defs as seams. That blind spot is now written down in the labels file instead of being an unwritten limitation. Labels only — no ranking code touched, as the gate requires. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
detect_extension_pointsbreaks score ties on(module, lineno)— alphabetically. When a band of equally-scored candidates straddles rank 10, the alphabet decides which labels count as hits.Measured across all seven targets:
Half the headline is not evidence, and the regression canaries were never canaries — click's perfect 4/4 has a lower bound of 2/4, pluggy's 3/3 a lower bound of 0/3.
What this changes
score.pyreports the bounds; the accept bar becomes the lower bound. Output only — the ranking is untouched and the benchmark still reads 18/34.Two findings recorded in benchmarks/README.md
JUnit and SQLAlchemy are not the same problem. All 18 candidates tied at 1.00 in JUnit live in
org.junit.jupiter.api.extensionand implementExtension— every one is a genuine extension point, and the 7 labels are a partial list of 18 correct answers. The ranking put 18/18 real seams in the top 18 and scored 4/7 for it. No tiebreak signal is warranted there; there is nothing to discriminate. SQLAlchemy's 50-wide band genuinely mixesDialect/CreateEnginePlugin/TypeDecoratorwithNoCursorDQLFetchStrategyand internal adapters — that one is a real ranking failure.Rejected: the unclamped signal sum as a tiebreak.
scoreismin(sum, 1.0), so the clamp discards how much evidence a point carried. Using the raw sum to break ties reorders only within a band — a free lever if it correlates. It is anti-correlated: 18/34 → 15/34, and ranking by signal count gives the same 15/34. Recorded so nobody re-derives it.Test plan
tests/test_benchmark_scoring.py— four new cases on the pure functions (band spanning, JUnit's shape, unambiguous bands, the lower-bound accept bar). No target packages needed.python benchmarks/score.pystill reports 18/34 — verified, this is an output-only change.ground_truth.jsonand no ranking code touched, so the labels gate is not engaged.🤖 Generated with Claude Code