Skip to content

bench: add flask, the first target that can falsify a ranking change - #86

Merged
Rinkia merged 1 commit into
masterfrom
bench/flask-target
Jul 24, 2026
Merged

Rinkia merged 1 commit into
masterfrom
bench/flask-target

Conversation

@Rinkia

@Rinkia Rinkia commented Jul 24, 2026

Copy link
Copy Markdown
Owner

Step 2 of the agreed sequence, and a direct consequence of #83.

Why another target, and why this one

#83 measured that every target here scores its labels inside a tied band. click's perfect 4/4 has a lower bound of 2/4; pluggy's 3/3 has a lower bound of 0/3. The two designated regression canaries cannot falsify a change on their own — the benchmark had no real guard.

flask 3.1.2 was chosen by measuring candidate distributions before labelling anything, per the risk noted when this was planned. Its ranking is graded, not flat:

Score 1.00 0.90 0.85 0.80 0.60
Candidates 3 6 1 23 1

Exactly one candidate holds the cutoff score, so flask's recall@10 has no tie bounds — worst equals best. It is the only target of the eight whose number is pure evidence.

(pytest was measured and rejected as a target: pytest/ is a re-export shim and yields 0 candidates. The real code is _pytest, at 620.)

Baseline

Target Candidates Labels Rank of each label recall@10 Median
flask 3.1.2 98 6 2, 3, 10, 38, 39, 98 3/6 24

Aggregate 18/34 → 21/40, bounds 6..23 → 9..26. The tie-decided share falls from 17/34 to 17/40 because all six new labels are tie-free.

What it already says about the ranking

  • MethodView ranks 98 of 98 — dead last. Arguably Flask's most-used class-based extension point. Not an ABC, no known role suffix, base (View) likewise, not advertised as a seam by the root __init__. Every existing signal is blind to it.
  • The implementation outranks the contract. DefaultJSONProvider is 1st; JSONProvider — the class whose docstring says "subclass this base class and implement at least dumps and loads" — is 10th.
  • Dynamic imports hold ranks 4–9, pushing Flask and Blueprint beneath them. The pygments failure again, on a package with no plugin registry at all.

Labelling

Six labels, each quoting the docstring shipped in the 3.1.2 wheel — per the rule preferring docs pinned to the measured version over a current web guide.

Two exclusions, recorded with reasons rather than left implicit:

  • Flask — its shipped docstring describes what the object does and never invites subclassing. The web guide is more encouraging; the pinned docs win.
  • Blueprint — you construct one and register the instance, which the rule counts as calling an API.

And one honest limitation now written into the labels file: Flask's largest extension surface — before_request, errorhandler, template_filter, context_processor — cannot be labelled at all, because they are methods on the app object and the parser only takes top-level defs as seams. That blind spot is a reason to add this target, not a reason to avoid it.

Test plan

  • tests/test_benchmark_labels.py — id-belongs-to-target invariant passes for all six new ids.
  • Full framework-free suite green.
  • python benchmarks/score.py reproduces the table above.
  • Labels only — no ranking code touched, so the benchmark-labels-untouched gate is satisfied. The only other files are benchmarks/README.md and the pinned-install line in score.py's docstring.

🤖 Generated with Claude Code

Measuring ties in #83 exposed that every target here scores its labels inside a
tied band. click's perfect 4/4 has a lower bound of 2/4; pluggy's 3/3 has a lower
bound of 0/3. A benchmark of seven targets had, in effect, no regression guard —
the two designated canaries could not falsify anything on their own.

flask 3.1.2 was picked by measuring candidate distributions first, not by taste.
Its ranking is graded rather than flat — 3 candidates at 1.00, 6 at 0.90, then
exactly ONE at the rank-10 cutoff — so its recall@10 has no tie bounds at all:
worst case equals best case. It is the only target of the eight whose number is
pure evidence.

  flask 3.1.2 | 98 candidates | 6 labels | ranks 2, 3, 10, 38, 39, 98 | 3/6

Aggregate 18/34 -> 21/40, bounds 6..23 -> 9..26. All six new labels are tie-free,
so the tie-decided share falls from 17/34 to 17/40.

What it already says, recorded in benchmarks/README.md:

- MethodView ranks 98 of 98, dead last. Arguably Flask's most-used class-based
  extension point, and every existing signal is blind to it.
- DefaultJSONProvider ranks 1st while JSONProvider — the class the docstring
  explicitly says to subclass — ranks 10th. Role-suffix and base-role reward the
  concrete implementation over the contract it implements.
- Dynamic imports take ranks 4-9 and push Flask and Blueprint below them: the
  pygments failure again, on a package with no plugin registry at all.

Labels quote the docstrings shipped in the 3.1.2 wheel, per the rule about
preferring docs pinned to the measured version. Two exclusions are recorded with
their reasons: Flask itself (its shipped docstring never invites subclassing) and
Blueprint (you construct one and register the instance — calling an API).

Flask's largest extension surface, the before_request / errorhandler /
template_filter decorators, cannot be labelled at all: they are methods on the
app object and the parser only takes top-level defs as seams. That blind spot is
now written down in the labels file instead of being an unwritten limitation.

Labels only — no ranking code touched, as the gate requires.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

MODScan extension-point diff

No breaking changes to extension points

@github-actions

Copy link
Copy Markdown

MODScan attack-surface diff — modscan

No new execution sinks

New attack surface to review — NOT a vulnerability verdict. This compares enumerated execution sinks between two snapshots. It does not trace whether input reaches them (no taint analysis), match CVEs, or detect secrets. An empty diff means no new catalogued sinks — not that the change is safe.

The two snapshots enumerate the same execution sinks.

@Rinkia
Rinkia merged commit 2bf79b3 into master Jul 24, 2026
7 checks passed
@Rinkia
Rinkia deleted the bench/flask-target branch July 24, 2026 12:44
@Rinkia Rinkia mentioned this pull request Sep 8, 2026
4 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant