Repository navigation
feat(plugins): toolscale sweep of tool count, similarity, query cue, and filler density - #324
Merged
Merged
Conversation
Add a toolscale plugin that measures tool-selection accuracy as the number of offered tools and the similarity of the target's nearest neighbours vary independently. Each item offers the target, three neighbours from one tier (near, related, distinct), and unrelated fillers up to N tools, and grades the first tool call's name. Fillers are shared across tiers and nested across N, and the target keeps its relative position, so every contrast is paired by target. Wrong picks are split into neighbour and filler picks, and accuracy among items with a call separates abstention from confusion. A 272-tool catalog (17 domains x 4 families x 4 variants) defines the tiers. A TF-IDF check confirms near > related > distinct for every tool, and lint rules keep every query unambiguous. The catalog was calibrated against a live model: one ambiguous query was rewritten and one variant the model refused even alone was replaced. --toolscale-catalog runs a private catalog whose items the report and stored results withhold. Category L is unchanged. Closes #319
Calibration showed two models making almost no wrong tool picks, with every miss in the blended accuracy being a clarifying question. Report accuracy when a tool was called apart from the call rate, keeping the blended accuracy only as the stored score so runs still sort. Add a cue axis: every catalog tool now has a query_implied that implies its qualifier without naming it, sent with the identical tool list as the literal query so the cue effect is paired. Lint rejects implied queries that name their own or a sibling's qualifier. A query check reports how much each cue overlaps its target versus its closest sibling. --toolscale-cues selects cues (default: literal,implied). Calibration on two models found neither cue produces measurable confusion up to 64 tools; docs record the results and point the next difficulty axes at description quality, crowded fillers, and larger N. One ambiguous implied query and the flight-cabin descriptions were fixed. The console grid now fits 80 columns.
Add --toolscale-fillers sparse,crowded. Crowded fillers draw whole 4-variant families instead of one tool per family, walking the same family order, so the target's neighbours, slot layout, and target position are unchanged and sparse minus crowded is paired by target. Crowded skips the control and N=4, which have no fillers, and adding it leaves the sparse items untouched. The report adds a density effect, a headline by filler density, density slices in every grid and effect table, and a filler-crowding measure (each filler's similarity to its closest other filler) that confirms crowded lists are denser: 0.605 against 0.174 on the built-in catalog. Calibration on two models found no density effect; docs record it.
Give each filler density its own maximum count. Sparse lists still stop at the number of unrelated families (65 built-in); crowded lists reach the variant count left after reserving the three largest outside families (248 built-in). Counts above the sparse maximum build crowded cells only. Long lists stretch the request timeout, and runs above 128 tools warn about provider tool caps. Fixes from an adversarial review: - an empty or partly empty grid (crowded-only at N=4) now fails with a clear error instead of an IndexError or silently dropped counts; - a control-only run scores the control, and the final tally counts the same items as the score; - the call rate leaves request errors out of its denominator; - the failed-items table escapes pipes and carriage returns in model and catalog text, and shows why a request failed; - private-catalog tool names stay out of grid errors and log lines; - the flat-nearest similarity check reads sparse lists, filler crowding is measured on each density's own largest list, and the console picks effects from the measured counts; - docs now state the real count limits, the timeout stretch, the call rate and score denominators, and the provider cap.
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Closes #319.
tool-eval-bench could not show how tool-selection accuracy changes as the number of offered tools grows, or separate that from how similar the tool descriptions are. Category L measures one point (a fixed 52-tool set), TC-70 tests near-duplicates at a fixed count, and published results usually confound count with similarity, because adding tools adds near-duplicates.
Change
A new
toolscaleplugin (--toolscale,--toolscale-only,plugin toolscale). Each item is one streamed request offering a target tool, 3 neighbours from one similarity tier, and unrelated fillers up to N tools. Only the first tool call's name is graded.Four independent, paired axes:
Fillers are shared across tiers and nested across N, neighbours keep their slots, and the target keeps its relative position, so every contrast is paired by target with a bootstrap interval.
The headline is two numbers, not one: accuracy when a tool was called (tool confusion) and call rate (whether the model acted at all). In calibration a mid-sized model scored 83% blended while never picking a wrong tool; every miss was a clarifying question. The blended accuracy is kept as the stored score so runs still sort.
--toolscale-catalog PATHruns a private catalog with the same schema; the report, stored results, logs, and errors withhold its tool names and queries.Category L and all existing scores are unchanged. The plugin adds eight
--toolscale*flags; the compatibility snapshots change only by those additions.Calibration
Two local models ran the grids, about 10,000 live requests in total:
Neither model showed meaningful tool confusion on any axis: not with near-duplicates present, not when the query only implied the target, not in crowded catalogs, and not at 248 tools (DeepSeek 537/537, Gemma 483/486 correct when called). What separated the models was the call rate. Calibration also fixed one ambiguous literal query, one ambiguous implied query, and replaced a production-deploy variant that the strong model refused even as the only tool offered.
docs/toolscale.mdrecords the results;docs/decisions.mdrecords why a larger catalog and real-world filler tools were deferred.Regression coverage
tests/test_toolscale.py(97 tests) covers catalog validation and lint (including implied-query rules), every grid invariant (tier membership, shared and nested fillers, slot and position pairing, crowded families, per-density count limits, empty-grid rejection), grading outcomes, statistics, aggregation on partial grids, the two-number headline and call-rate denominator, report rendering (including pipes and carriage returns in model output), private-catalog withholding in details, reports, errors, and logs, and CLI wiring for flat flags and thepluginsubcommand. An independent review also fuzzed the grid builder with 150 random catalogs against the same invariants.Updated expectations in
test_cli_dispatch_golden.py,test_json_output_contract.py, and thebench --helplength budget intest_cli_parser.pyreflect the new plugin and flags.Changelog and documentation
changelog.d/319.added.mddocs/toolscale.md(new),README.md,docs/cli-reference.md,docs/methodology.md,docs/decisions.md,docs/adding-a-plugin.mdContributor checklist
tests-not-needed.changelog.d/fragment was added or updated, or a maintainer appliedskip-changelog.Written with AI assistance; reviewed before opening.