Repository navigation
bench: add flask, the first target that can falsify a ranking change - #86
Merged
Merged
Conversation
Measuring ties in #83 exposed that every target here scores its labels inside a tied band. click's perfect 4/4 has a lower bound of 2/4; pluggy's 3/3 has a lower bound of 0/3. A benchmark of seven targets had, in effect, no regression guard — the two designated canaries could not falsify anything on their own. flask 3.1.2 was picked by measuring candidate distributions first, not by taste. Its ranking is graded rather than flat — 3 candidates at 1.00, 6 at 0.90, then exactly ONE at the rank-10 cutoff — so its recall@10 has no tie bounds at all: worst case equals best case. It is the only target of the eight whose number is pure evidence. flask 3.1.2 | 98 candidates | 6 labels | ranks 2, 3, 10, 38, 39, 98 | 3/6 Aggregate 18/34 -> 21/40, bounds 6..23 -> 9..26. All six new labels are tie-free, so the tie-decided share falls from 17/34 to 17/40. What it already says, recorded in benchmarks/README.md: - MethodView ranks 98 of 98, dead last. Arguably Flask's most-used class-based extension point, and every existing signal is blind to it. - DefaultJSONProvider ranks 1st while JSONProvider — the class the docstring explicitly says to subclass — ranks 10th. Role-suffix and base-role reward the concrete implementation over the contract it implements. - Dynamic imports take ranks 4-9 and push Flask and Blueprint below them: the pygments failure again, on a package with no plugin registry at all. Labels quote the docstrings shipped in the 3.1.2 wheel, per the rule about preferring docs pinned to the measured version. Two exclusions are recorded with their reasons: Flask itself (its shipped docstring never invites subclassing) and Blueprint (you construct one and register the instance — calling an API). Flask's largest extension surface, the before_request / errorhandler / template_filter decorators, cannot be labelled at all: they are methods on the app object and the parser only takes top-level defs as seams. That blind spot is now written down in the labels file instead of being an unwritten limitation. Labels only — no ranking code touched, as the gate requires. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MODScan extension-point diffNo breaking changes to extension points |
MODScan attack-surface diff —
|
4 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 2 of the agreed sequence, and a direct consequence of #83.
Why another target, and why this one
#83 measured that every target here scores its labels inside a tied band. click's perfect 4/4 has a lower bound of 2/4; pluggy's 3/3 has a lower bound of 0/3. The two designated regression canaries cannot falsify a change on their own — the benchmark had no real guard.
flask 3.1.2 was chosen by measuring candidate distributions before labelling anything, per the risk noted when this was planned. Its ranking is graded, not flat:
Exactly one candidate holds the cutoff score, so flask's recall@10 has no tie bounds — worst equals best. It is the only target of the eight whose number is pure evidence.
(
pytestwas measured and rejected as a target:pytest/is a re-export shim and yields 0 candidates. The real code is_pytest, at 620.)Baseline
Aggregate 18/34 → 21/40, bounds 6..23 → 9..26. The tie-decided share falls from 17/34 to 17/40 because all six new labels are tie-free.
What it already says about the ranking
MethodViewranks 98 of 98 — dead last. Arguably Flask's most-used class-based extension point. Not an ABC, no known role suffix, base (View) likewise, not advertised as a seam by the root__init__. Every existing signal is blind to it.DefaultJSONProvideris 1st;JSONProvider— the class whose docstring says "subclass this base class and implement at least dumps and loads" — is 10th.FlaskandBlueprintbeneath them. The pygments failure again, on a package with no plugin registry at all.Labelling
Six labels, each quoting the docstring shipped in the 3.1.2 wheel — per the rule preferring docs pinned to the measured version over a current web guide.
Two exclusions, recorded with reasons rather than left implicit:
Flask— its shipped docstring describes what the object does and never invites subclassing. The web guide is more encouraging; the pinned docs win.Blueprint— you construct one and register the instance, which the rule counts as calling an API.And one honest limitation now written into the labels file: Flask's largest extension surface —
before_request,errorhandler,template_filter,context_processor— cannot be labelled at all, because they are methods on the app object and the parser only takes top-level defs as seams. That blind spot is a reason to add this target, not a reason to avoid it.Test plan
tests/test_benchmark_labels.py— id-belongs-to-target invariant passes for all six new ids.python benchmarks/score.pyreproduces the table above.benchmark-labels-untouchedgate is satisfied. The only other files arebenchmarks/README.mdand the pinned-install line inscore.py's docstring.🤖 Generated with Claude Code