Skip to content

bench: add function labels, and say plainly why the aggregate jumped - #82

Merged
Rinkia merged 1 commit into
masterfrom
bench/function-labels
Jul 24, 2026
Merged

Rinkia merged 1 commit into
masterfrom
bench/function-labels

Conversation

@Rinkia

@Rinkia Rinkia commented Jul 24, 2026

Copy link
Copy Markdown
Owner

What

Until now every label was a class, so the benchmark could not judge any signal that only scores functions — the hook-name patterns, registration decorators, _W_NAME_HOOK. It could detect the damage such a change did and never its benefit. That is why #15 was unjudgeable.

Five function labels fix it — registration points a user decorates their own function with, each quoting the target's own docstring:

Label Rank Docstring
marshmallow.decorators:validates 1 "Register a validator method for field(s)."
marshmallow.decorators:post_dump 4 "Register a method to invoke after serializing an object."
marshmallow.decorators:pre_load 5 "Register a method to invoke before deserializing an object."
sqlalchemy.event.api:listens_for 361 "Decorate a function as a listener for the given target + identifier."
sqlalchemy.ext.compiler:compiles 1026 "Register a function as a compiler for a given ClauseElement type."

The aggregate went 15/29 → 18/34. That is not a ranking improvement.

Three of the five were already inside their target's top ten, so the number rose by construction.

Saying so prominently — in the commit, the PR and benchmarks/README.md — because letting a jump like that read as progress is precisely the abuse the gate on this directory exists to prevent. "Labels are never silently added to make a number move."

What actually changed

  • Expand hook/registration name patterns in the detector #15 is now judgeable — there are labels a function-name signal could move.
  • Two honest hard targets appeared. listens_for (361) and compiles (1026) are far outside the top ten. compiles is the pointed one: the compiler-registration mechanism that the rejected whole-program-resolution work kept circling is now a labelled, measurable miss instead of an anecdote.

Per target: marshmallow 1/3 → 4/6; SQLAlchemy gains two labels and catches neither, 1/5 → 1/7.

Honest limits, recorded

Five function labels out of thirty-four, three of them easy — so a function-name signal is judgeable but on thin evidence. The README now says that, and flags that the #15 sweep predates these labels and is worth re-running.

Partial lists as ever: pre_dump, post_load, validates_schema are documented identically and deliberately not all labelled.

Test plan

  • tests/test_benchmark_labels.py passes (ids belong to their target, justifications present).
  • Suite 26/26; benchmark reports 18/34.
  • Labels only — no ranking or front-end code touched, per the gate.

🤖 Generated with Claude Code

Until now every label was a class, so the benchmark could not judge any signal
that only scores functions - the hook-name patterns, the registration-decorator
patterns, _W_NAME_HOOK. It could detect the damage such a change did and never
its benefit, which is why #15 was unjudgeable.

Five function labels fix that. Each is a registration point a user decorates
their own function with, quoting the target's own docstring:

  marshmallow.decorators:validates    rank 1     "Register a validator method"
  marshmallow.decorators:post_dump    rank 4     "Register a method to invoke after serializing"
  marshmallow.decorators:pre_load     rank 5     "Register a method to invoke before deserializing"
  sqlalchemy.event.api:listens_for    rank 361   "Decorate a function as a listener"
  sqlalchemy.ext.compiler:compiles    rank 1026  "Register a function as a compiler"

The aggregate goes 15/29 -> 18/34, and that is NOT a ranking improvement. Three
of the five were already inside their target's top ten, so the number rose by
construction. Saying so in the README because letting a jump like that read as
progress is exactly the abuse the gate on this directory exists to prevent.

What changed is what the benchmark can see. #15 now has labels it could move.
And two honest hard targets appeared: listens_for at 361 and compiles at 1026,
both far outside the top ten. compiles is the pointed one - the
compiler-registration mechanism the rejected whole-program-resolution work kept
circling is now a labelled, measurable miss instead of an anecdote.

marshmallow moves 1/3 -> 4/6. SQLAlchemy gains two labels and catches neither,
1/5 -> 1/7.

Partial lists, as ever: pre_dump, post_load and validates_schema are documented
identically and deliberately not all labelled.

Labels only - no ranking or front-end code touched, per the gate. Suite 26/26.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

MODScan extension-point diff

No breaking changes to extension points

@github-actions

Copy link
Copy Markdown

MODScan attack-surface diff — modscan

No new execution sinks

New attack surface to review — NOT a vulnerability verdict. This compares enumerated execution sinks between two snapshots. It does not trace whether input reaches them (no taint analysis), match CVEs, or detect secrets. An empty diff means no new catalogued sinks — not that the change is safe.

The two snapshots enumerate the same execution sinks.

@Rinkia
Rinkia merged commit f881b16 into master Jul 24, 2026
7 checks passed
@Rinkia
Rinkia deleted the bench/function-labels branch July 24, 2026 09:03
@Rinkia Rinkia mentioned this pull request Sep 8, 2026
4 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant