Repository navigation
bench: add commander, the first JavaScript target, and record its baseline - #74
Merged
Merged
Conversation
…eline Closes #63. The five existing targets are Python, so the ranking had never been measured on a second language. commander 14.0.2 asks that question, and the answer is recorded rather than tuned away: 0/2, both labelled seams at ranks 11 and 15, aggregate 9/20 -> 9/22. What sits above them is the finding. TypeScript interface declarations are scored as abstract classes - the front-end treats an interface as a pure contract to implement, which is reasonable in the abstract and wrong in practice. commander's top eight are ParseOptions, HelpContext, OutputConfiguration and friends, option bags from typings/index.d.ts, each scoring 0.70 with "meant to be subclassed", while the two classes the documentation actually tells you to subclass score 0.10. That is left alone here on purpose. Changing it is a ranking change, and a ranking change is judged against a baseline in its own pull request - fixing it in the same change that introduces the labels is precisely the circularity the gate on this directory exists to prevent. This commit therefore touches no front-end and no weights. Labels come from commander's own Readme: Help is named as subclassable ("or by subclassing Help using .createHelp()"), and createCommand is presented as an override point ("you may override it to customise the new subcommand"). Option and Argument are deliberately NOT labelled - the docs present them as things you construct explicitly, which is calling, not extending. The scorer gains a "language" field per target: JS/TS targets resolve under benchmarks/js/node_modules, take their version from the package's own package.json, and are skipped with a notice when the installed version is not the pinned one. Ids are target-qualified path form (commander/lib/help:Help), keeping the "an id belongs to its target" invariant the label self-check enforces. package.json and the lockfile are committed; node_modules is not. Suite 25/25. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MODScan attack-surface diff —
|
MODScan extension-point diffNo breaking changes to extension points |
4 tasks done
Rinkia
added a commit
that referenced
this pull request
Jul 23, 2026
Judged against the commander baseline recorded in #74. Two candidate fixes were swept before either was written, and they turned out identical: variant commander aggregate baseline 0/2 (11, 15) 9/22 B interface no longer abstract 2/2 (3, 7) 11/22 C skip *.d.ts 2/2 (3, 7) 11/22 D both 2/2 (3, 7) 11/22 That they are identical is itself the finding: both describe the same phenomenon, and combining them adds nothing. C is adopted, because B is right about commander for the wrong reason. A hand-written TypeScript interface genuinely is a contract to implement - the front-end's own fixture has RenderPlugin, and B would demote it. What was actually wrong was reading a declaration file at all. A .d.ts declares types for code defined elsewhere, so commander's Command and Help were each counted twice, and option bags existing only as declarations (ParseOptions, HelpContext, OutputConfiguration) became candidates and outranked the classes the documentation tells you to subclass. Skipping declaration files also matches the rule the labels already follow: a seam is recorded where it is defined, in the module holding the class statement. commander's candidates drop 25 -> 11 as the phantom copies disappear, its median rank goes 13 -> 5, and no Python target moves: click stays 4/4, pluggy 3/3. Aggregate 9/22 -> 11/22. This changes ranking behaviour, so per the gate on benchmarks/ it deliberately carries no label change. Suite 25/25. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Rinkia
added a commit
that referenced
this pull request
Jul 23, 2026
Bump to 0.1.7 and move the [Unreleased] entries under [0.1.7]. Ships the Java front-end, the security lens over MCP and for TypeScript/JavaScript, the process sinks the Bandit cross-check exposed, and two TypeScript front-end fixes. Also adds the two CHANGELOG entries that were missing: recognising CommonJS exports (#73) and no longer scanning .d.ts declaration files (#76). Both change what `--language typescript` returns, so both belong in a user-facing changelog. The cross-check harnesses (#70, #72) and benchmark targets (#74, #78) stay out: they are repository artifacts, not shipped in the wheel. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Closes #63. The five existing targets are Python, so the ranking had never been measured on a second language.
commander14.0.2 asks that question — and the answer is recorded, not tuned away.Aggregate 9/20 → 9/22. Both labelled seams land just outside the top ten.
The finding
TypeScript
interfacedeclarations are scored as abstract classes. The front-end treats an interface as "a pure contract to implement" — reasonable in the abstract, wrong in practice. commander's top eight areParseOptions,HelpContext,OutputConfiguration,AddHelpTextContextand friends — option bags fromtypings/index.d.ts— each at 0.70 with "meant to be subclassed", while the two classes the documentation actually tells you to subclass score 0.10.Deliberately not fixed here. Changing it is a ranking change, judged against a baseline in its own PR. Fixing it in the same change that introduces the labels is exactly the circularity the gate on this directory exists to prevent — so this PR touches no front-end and no weights, and the gate passes on its own terms.
Labels, from commander's own Readme
commander/lib/help:Help— "or by subclassingHelpusing.createHelp()". Subclassing named outright.commander/lib/command:Command— "you may override it to customise the new subcommand".OptionandArgumentare deliberately not labelled: the docs present them as things you construct explicitly, which is calling, not extending — and the labelling rule excludes "merely calls".Scorer changes
"language"field per target; JS/TS resolve underbenchmarks/js/node_modules, version read from the package's ownpackage.json, skipped with a notice on a version mismatch.commander/lib/help:Help), keeping the "an id belongs to its target" invariant the label self-check enforces — I hit that assertion and fixed the id form rather than weakening the test.package.json+ lockfile committed;node_modulesgitignored and confirmed not staged.Test plan
tests/test_benchmark_labels.pypasses (ids belong to their target, justifications present).benchmark-labels-untouchedgate: this PR changes labels but no ranking/front-end code.Next, separately: judge
interface → abstractagainst this baseline.🤖 Generated with Claude Code