Repository navigation
bench: add function labels, and say plainly why the aggregate jumped - #82
Merged
Merged
Conversation
Until now every label was a class, so the benchmark could not judge any signal that only scores functions - the hook-name patterns, the registration-decorator patterns, _W_NAME_HOOK. It could detect the damage such a change did and never its benefit, which is why #15 was unjudgeable. Five function labels fix that. Each is a registration point a user decorates their own function with, quoting the target's own docstring: marshmallow.decorators:validates rank 1 "Register a validator method" marshmallow.decorators:post_dump rank 4 "Register a method to invoke after serializing" marshmallow.decorators:pre_load rank 5 "Register a method to invoke before deserializing" sqlalchemy.event.api:listens_for rank 361 "Decorate a function as a listener" sqlalchemy.ext.compiler:compiles rank 1026 "Register a function as a compiler" The aggregate goes 15/29 -> 18/34, and that is NOT a ranking improvement. Three of the five were already inside their target's top ten, so the number rose by construction. Saying so in the README because letting a jump like that read as progress is exactly the abuse the gate on this directory exists to prevent. What changed is what the benchmark can see. #15 now has labels it could move. And two honest hard targets appeared: listens_for at 361 and compiles at 1026, both far outside the top ten. compiles is the pointed one - the compiler-registration mechanism the rejected whole-program-resolution work kept circling is now a labelled, measurable miss instead of an anecdote. marshmallow moves 1/3 -> 4/6. SQLAlchemy gains two labels and catches neither, 1/5 -> 1/7. Partial lists, as ever: pre_dump, post_load and validates_schema are documented identically and deliberately not all labelled. Labels only - no ranking or front-end code touched, per the gate. Suite 26/26. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MODScan extension-point diffNo breaking changes to extension points |
MODScan attack-surface diff —
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Until now every label was a class, so the benchmark could not judge any signal that only scores functions — the hook-name patterns, registration decorators,
_W_NAME_HOOK. It could detect the damage such a change did and never its benefit. That is why #15 was unjudgeable.Five function labels fix it — registration points a user decorates their own function with, each quoting the target's own docstring:
marshmallow.decorators:validatesmarshmallow.decorators:post_dumpmarshmallow.decorators:pre_loadsqlalchemy.event.api:listens_forsqlalchemy.ext.compiler:compilesThe aggregate went 15/29 → 18/34. That is not a ranking improvement.
Three of the five were already inside their target's top ten, so the number rose by construction.
Saying so prominently — in the commit, the PR and
benchmarks/README.md— because letting a jump like that read as progress is precisely the abuse the gate on this directory exists to prevent. "Labels are never silently added to make a number move."What actually changed
listens_for(361) andcompiles(1026) are far outside the top ten.compilesis the pointed one: the compiler-registration mechanism that the rejected whole-program-resolution work kept circling is now a labelled, measurable miss instead of an anecdote.Per target: marshmallow 1/3 → 4/6; SQLAlchemy gains two labels and catches neither, 1/5 → 1/7.
Honest limits, recorded
Five function labels out of thirty-four, three of them easy — so a function-name signal is judgeable but on thin evidence. The README now says that, and flags that the #15 sweep predates these labels and is worth re-running.
Partial lists as ever:
pre_dump,post_load,validates_schemaare documented identically and deliberately not all labelled.Test plan
tests/test_benchmark_labels.pypasses (ids belong to their target, justifications present).🤖 Generated with Claude Code