Skip to content

Commit f2f2819

Browse files
Rinkiaclaude
andauthored
feat(detect): disclose score ties instead of presenting them as a ranking (#85)
The detector breaks equal scores on (module, lineno) — alphabetically. A reader taking the top ten off a band of eighteen equally-scored candidates was reading the alphabet, and nothing in the output said so. PR #83 stopped the benchmark telling itself that story; this stops the product telling it to users. `modscan detect` now prints, above the table, how many candidates share the score at the cutoff and how many of them are shown: **18 candidates score exactly 1.00, and this list shows 12 of them.** The 6 left out are not ranked lower — ties are broken by module name, so the cut through this band is alphabetical, not evidence-based. When the band is fully visible the note says so instead of claiming truncation, and when the cutoff is unambiguous there is no note at all — a warning that fires on every run is one nobody reads. `--json` gains `tied_with` per point, since a machine consumer that re-ranks or truncates the payload needs the same fact and cannot read Markdown. The field is additive; `modscan diff` compares ids, category and kind, not the payload shape. Ties are measured on the full ranked list before `--limit` cuts it — computing them afterwards would describe the truncated list rather than the ranking. Presentation only: no weight, no signal and no ordering changed. Benchmark unchanged at 18/34, bounds 6..23. examples/detect-markdown.md is regenerated from its committed recipe. It was already stale — it predated the MS-MOD-* signal ids from #56 — and it turns out to be the exact failure this change fixes: twelve rows shown out of eighteen tied at 1.00. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent a2a1bdc commit f2f2819

6 files changed

Lines changed: 207 additions & 23 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1515
`pyproject.toml`; a source checkout that is not installed reports
1616
`0.0.0+unknown` rather than lying.
1717

18+
- **`modscan detect` says when the ranking is a tie.** The detector breaks equal
19+
scores on module name, so a band of equally-scored candidates is ordered
20+
alphabetically — and a reader taking the top ten off a band of eighteen was
21+
reading the alphabet without being told. `detect` now prints how many
22+
candidates share the score at the cutoff and how many of them are shown, and
23+
`--json` carries `tied_with` per point for consumers that cannot read the
24+
prose. Nothing about the ranking changed; it just stopped presenting a tie as
25+
an ordering. The same disclosure was added to the benchmark's own numbers.
26+
1827
### Fixed
1928

2029
- **`@overload` stubs no longer report a symbol several times.** `typing.overload`

‎README.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -179,6 +179,20 @@ Point it at an installed package to see it work immediately:
179179
modscan detect "$(python -c 'import os,click;print(os.path.dirname(click.__file__))')" --limit 5
180180
```
181181

182+
**Read the tie note before you read the ranking.** Equal scores are broken by
183+
module name, so when many candidates share a score the order among them is
184+
alphabetical rather than evidence-based. `detect` says so when the list you asked
185+
for cuts through such a band:
186+
187+
> **18 candidates score exactly 1.00, and this list shows 12 of them.** The 6
188+
> left out are not ranked lower — ties are broken by module name, so the cut
189+
> through this band is alphabetical, not evidence-based.
190+
191+
`--json` carries the same fact as `tied_with` on each point. This is a real
192+
property of the ranking, not a disclaimer: see
193+
[`benchmarks/README.md`](benchmarks/README.md) for how much of the benchmark's
194+
own headline the alphabet turned out to own.
195+
182196
### Security lens (`modscan-audit`)
183197

184198
The same seams a plugin system exposes are also where untrusted code can enter. A

‎examples/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,10 @@ Honest reading, not cherry-picked:
8888
- The **Plugin registration points** section shows how markdown discovers plugins
8989
(`entry_points`, plus `import_module`/`find_spec` loader sites), reported apart
9090
from the implement-this ranking rather than flooding it.
91+
- The note under the header is the honest part: **eighteen candidates tie at
92+
1.00 and the run shows twelve**, so the six left out were cut by module name,
93+
not by evidence. Without that line the twelve rows read as a ranking. They are
94+
a band with an alphabetical cut through it.
9195

9296
Reproduce it (free):
9397

‎examples/detect-markdown.md‎

Lines changed: 14 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -2,20 +2,22 @@
22

33
12 candidate(s), ranked by moddability. No LLM was used — this is the static ranking only.
44

5+
**18 candidates score exactly 1.00, and this list shows 12 of them.** The 6 left out are not ranked lower — ties are broken by module name, so the cut through this band is alphabetical, not evidence-based. Widen `--limit` to see the whole band.
6+
57
| # | Extension point | Category | Score | Why |
68
|---|---|---|---|---|
7-
| 1 | `markdown.extensions.abbr:AbbrExtension` | subclass | 1.00 | public class; name ends in 'Extension' — role type; subclasses a 'Extension' role type |
8-
| 2 | `markdown.extensions.admonition:AdmonitionExtension` | subclass | 1.00 | public class; name ends in 'Extension' — role type; subclasses a 'Extension' role type |
9-
| 3 | `markdown.extensions.attr_list:AttrListExtension` | subclass | 1.00 | public class; name ends in 'Extension' — role type; subclasses a 'Extension' role type |
10-
| 4 | `markdown.extensions.codehilite:CodeHiliteExtension` | subclass | 1.00 | public class; name ends in 'Extension' — role type; subclasses a 'Extension' role type |
11-
| 5 | `markdown.extensions.def_list:DefListExtension` | subclass | 1.00 | public class; name ends in 'Extension' — role type; subclasses a 'Extension' role type |
12-
| 6 | `markdown.extensions.extra:ExtraExtension` | subclass | 1.00 | public class; name ends in 'Extension' — role type; subclasses a 'Extension' role type |
13-
| 7 | `markdown.extensions.fenced_code:FencedCodeExtension` | subclass | 1.00 | public class; name ends in 'Extension' — role type; subclasses a 'Extension' role type |
14-
| 8 | `markdown.extensions.footnotes:FootnoteExtension` | subclass | 1.00 | public class; name ends in 'Extension' — role type; subclasses a 'Extension' role type |
15-
| 9 | `markdown.extensions.legacy_attrs:LegacyAttrExtension` | subclass | 1.00 | public class; name ends in 'Extension' — role type; subclasses a 'Extension' role type |
16-
| 10 | `markdown.extensions.legacy_em:LegacyEmExtension` | subclass | 1.00 | public class; name ends in 'Extension' — role type; subclasses a 'Extension' role type |
17-
| 11 | `markdown.extensions.md_in_html:MarkdownInHtmlExtension` | subclass | 1.00 | public class; name ends in 'Extension' — role type; subclasses a 'Extension' role type |
18-
| 12 | `markdown.extensions.meta:MetaExtension` | subclass | 1.00 | public class; name ends in 'Extension' — role type; subclasses a 'Extension' role type |
9+
| 1 | `markdown.extensions.abbr:AbbrExtension` | subclass | 1.00 | [MS-MOD-PUBLIC] public class; [MS-MOD-CLASSROLE] name ends in 'Extension' — role type; [MS-MOD-BASEROLE] subclasses a 'Extension' role type |
10+
| 2 | `markdown.extensions.admonition:AdmonitionExtension` | subclass | 1.00 | [MS-MOD-PUBLIC] public class; [MS-MOD-CLASSROLE] name ends in 'Extension' — role type; [MS-MOD-BASEROLE] subclasses a 'Extension' role type |
11+
| 3 | `markdown.extensions.attr_list:AttrListExtension` | subclass | 1.00 | [MS-MOD-PUBLIC] public class; [MS-MOD-CLASSROLE] name ends in 'Extension' — role type; [MS-MOD-BASEROLE] subclasses a 'Extension' role type |
12+
| 4 | `markdown.extensions.codehilite:CodeHiliteExtension` | subclass | 1.00 | [MS-MOD-PUBLIC] public class; [MS-MOD-CLASSROLE] name ends in 'Extension' — role type; [MS-MOD-BASEROLE] subclasses a 'Extension' role type |
13+
| 5 | `markdown.extensions.def_list:DefListExtension` | subclass | 1.00 | [MS-MOD-PUBLIC] public class; [MS-MOD-CLASSROLE] name ends in 'Extension' — role type; [MS-MOD-BASEROLE] subclasses a 'Extension' role type |
14+
| 6 | `markdown.extensions.extra:ExtraExtension` | subclass | 1.00 | [MS-MOD-PUBLIC] public class; [MS-MOD-CLASSROLE] name ends in 'Extension' — role type; [MS-MOD-BASEROLE] subclasses a 'Extension' role type |
15+
| 7 | `markdown.extensions.fenced_code:FencedCodeExtension` | subclass | 1.00 | [MS-MOD-PUBLIC] public class; [MS-MOD-CLASSROLE] name ends in 'Extension' — role type; [MS-MOD-BASEROLE] subclasses a 'Extension' role type |
16+
| 8 | `markdown.extensions.footnotes:FootnoteExtension` | subclass | 1.00 | [MS-MOD-PUBLIC] public class; [MS-MOD-CLASSROLE] name ends in 'Extension' — role type; [MS-MOD-BASEROLE] subclasses a 'Extension' role type |
17+
| 9 | `markdown.extensions.legacy_attrs:LegacyAttrExtension` | subclass | 1.00 | [MS-MOD-PUBLIC] public class; [MS-MOD-CLASSROLE] name ends in 'Extension' — role type; [MS-MOD-BASEROLE] subclasses a 'Extension' role type |
18+
| 10 | `markdown.extensions.legacy_em:LegacyEmExtension` | subclass | 1.00 | [MS-MOD-PUBLIC] public class; [MS-MOD-CLASSROLE] name ends in 'Extension' — role type; [MS-MOD-BASEROLE] subclasses a 'Extension' role type |
19+
| 11 | `markdown.extensions.md_in_html:MarkdownInHtmlExtension` | subclass | 1.00 | [MS-MOD-PUBLIC] public class; [MS-MOD-CLASSROLE] name ends in 'Extension' — role type; [MS-MOD-BASEROLE] subclasses a 'Extension' role type |
20+
| 12 | `markdown.extensions.meta:MetaExtension` | subclass | 1.00 | [MS-MOD-PUBLIC] public class; [MS-MOD-CLASSROLE] name ends in 'Extension' — role type; [MS-MOD-BASEROLE] subclasses a 'Extension' role type |
1921

2022
## Plugin registration points
2123

‎modscan/cli.py‎

Lines changed: 82 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -233,15 +233,74 @@ def _split_detect_points(points: list) -> tuple[list, list]:
233233
return implement, registration
234234

235235

236+
def _tied_counts(points: list) -> list[int]:
237+
"""How many *other* points share each point's score, positionally.
238+
239+
The detector breaks score ties on (module, lineno) — alphabetically. So a
240+
point's printed position inside a tied band reflects its name, not the
241+
evidence for it, and a reader taking "the top ten" off a band of eighteen is
242+
reading the alphabet. Counting the band is what lets the output say so.
243+
"""
244+
per_score: dict[float, int] = {}
245+
for point in points:
246+
per_score[point.score] = per_score.get(point.score, 0) + 1
247+
return [per_score[p.score] - 1 for p in points]
248+
249+
250+
def _tie_note(points: list, shown: int) -> str | None:
251+
"""One line telling the reader which part of this ranking is the alphabet.
252+
253+
``points`` is the full ranked list, ``shown`` how many of it are displayed —
254+
the band has to be measured before ``--limit`` cuts it, or the note would
255+
describe the truncated list rather than the ranking.
256+
"""
257+
if not points or shown <= 0:
258+
return None
259+
cutoff = points[min(shown, len(points)) - 1].score
260+
band = sum(1 for p in points if p.score == cutoff)
261+
if band < 2:
262+
return None
263+
above = sum(1 for p in points if p.score > cutoff)
264+
in_view = min(shown, len(points)) - above
265+
if band > in_view:
266+
return (
267+
f"**{band} candidates score exactly {cutoff:.2f}, and this list shows "
268+
f"{in_view} of them.** The {band - in_view} left out are not ranked lower — "
269+
"ties are broken by module name, so the cut through this band is "
270+
"alphabetical, not evidence-based. Widen `--limit` to see the whole band."
271+
)
272+
return (
273+
f"**The last {band} entries score exactly {cutoff:.2f}.** Ties are broken by "
274+
"module name, so their order among themselves is alphabetical, not "
275+
"evidence-based."
276+
)
277+
278+
236279
def _point_module(seam, root: str, label: str | None) -> str:
237280
"""Module qualname for output, or a clean fallback for a root-package symbol
238281
(empty qualname): the label if set, else the scan root's basename — never the
239282
absolute path, which would leak a local or CI filesystem layout into ids."""
240283
return seam.module or label or os.path.basename(os.path.normpath(root))
241284

242285

286+
def _detect_json_entry(point, tied_with: int, root: str, label: str | None) -> dict:
287+
return {
288+
"id": f"{_point_module(point.seam, root, label)}:{point.seam.name}",
289+
"module": point.seam.module or None,
290+
"category": point.category,
291+
"score": round(point.score, 4),
292+
"tied_with": tied_with,
293+
"kind": point.seam.kind,
294+
"signals": list(point.signals),
295+
}
296+
297+
243298
def _render_detect_markdown(
244-
implement: list, registration: list, root: str, label: str | None = None
299+
implement: list,
300+
registration: list,
301+
root: str,
302+
label: str | None = None,
303+
tie_note: str | None = None,
245304
) -> str:
246305
# Root-package symbols have an empty qualname; _point_module keeps their ids
247306
# clean (label, else the root basename) instead of leaking the scan path.
@@ -252,6 +311,10 @@ def _render_detect_markdown(
252311
f"{len(implement)} candidate(s), ranked by moddability. No LLM was used — "
253312
"this is the static ranking only.",
254313
"",
314+
]
315+
if tie_note:
316+
lines += [tie_note, ""]
317+
lines += [
255318
"| # | Extension point | Category | Score | Why |",
256319
"|---|---|---|---|---|",
257320
]
@@ -321,31 +384,39 @@ def _main_detect(argv: list[str]) -> int:
321384
codebase = get_language_parser(args.language).parse_codebase(args.root)
322385
points = detect_extension_points(build_graph(codebase), min_score=args.min_score)
323386
implement, registration = _split_detect_points(points)
387+
# Ties are measured on the FULL ranked list: --limit is exactly the cut that
388+
# can fall inside a band, so a note computed after truncation would describe
389+
# the wrong thing.
390+
tied = _tied_counts(implement)
391+
shown = len(implement) if args.limit is None else min(args.limit, len(implement))
392+
tie_note = _tie_note(implement, shown)
324393
# --limit caps the ranked implement-this list; registration points are a
325394
# small deduplicated set, always shown in full.
326395
if args.limit is not None:
327396
implement = implement[: args.limit]
397+
tied = tied[: args.limit]
328398

329399
# Signal strings contain an em-dash; a Windows cp1252 console would raise
330400
# UnicodeEncodeError printing them. Emit UTF-8 regardless of console locale.
331401
if hasattr(sys.stdout, "reconfigure"):
332402
sys.stdout.reconfigure(encoding="utf-8")
333403

334404
if args.json:
405+
# tied_with is the prose note's machine-readable twin: a consumer that
406+
# ranks or truncates this payload needs to know a score is shared, and
407+
# cannot read the Markdown warning. Registration points are a
408+
# deduplicated set rather than a ranking, so they carry 0.
335409
payload = [
336-
{
337-
"id": f"{_point_module(p.seam, args.root, args.label)}:{p.seam.name}",
338-
"module": p.seam.module or None,
339-
"category": p.category,
340-
"score": round(p.score, 4),
341-
"kind": p.seam.kind,
342-
"signals": list(p.signals),
343-
}
344-
for p in implement + registration
410+
_detect_json_entry(p, n, args.root, args.label)
411+
for p, n in list(zip(implement, tied)) + [(r, 0) for r in registration]
345412
]
346413
print(json.dumps(payload, indent=2))
347414
else:
348-
print(_render_detect_markdown(implement, registration, args.root, args.label))
415+
print(
416+
_render_detect_markdown(
417+
implement, registration, args.root, args.label, tie_note
418+
)
419+
)
349420
return 0
350421

351422

‎tests/test_cli.py‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -341,6 +341,85 @@ def test_detect_separates_and_dedups_registration_points() -> None:
341341
_cleanup(pkg)
342342

343343

344+
class _FakePoint:
345+
"""Only the attributes the tie helpers read."""
346+
347+
def __init__(self, score: float) -> None:
348+
self.score = score
349+
350+
351+
def test_tied_counts_report_the_band_not_the_position() -> None:
352+
from modscan.cli import _tied_counts
353+
354+
points = [_FakePoint(s) for s in (1.0, 1.0, 1.0, 0.8, 0.1)]
355+
assert _tied_counts(points) == [2, 2, 2, 0, 0]
356+
357+
358+
def test_tie_note_warns_when_the_limit_cuts_through_a_band() -> None:
359+
"""JUnit's shape: 18 tied at 1.00, a reader takes the top 10 and sees 4 of 7."""
360+
from modscan.cli import _tie_note
361+
362+
points = [_FakePoint(1.0) for _ in range(18)] + [_FakePoint(0.7) for _ in range(5)]
363+
note = _tie_note(points, shown=10)
364+
assert note is not None
365+
assert "18 candidates score exactly 1.00" in note
366+
assert "shows 10 of them" in note
367+
assert "alphabetical" in note, "the reader must be told what decided the cut"
368+
369+
370+
def test_tie_note_reports_a_fully_shown_band_without_crying_truncation() -> None:
371+
from modscan.cli import _tie_note
372+
373+
points = [_FakePoint(1.0) for _ in range(3)]
374+
note = _tie_note(points, shown=3)
375+
assert note is not None
376+
assert "last 3 entries" in note
377+
assert "left out" not in note, "nothing was cut — do not claim it was"
378+
379+
380+
def test_tie_note_is_silent_when_the_cutoff_is_unambiguous() -> None:
381+
"""No note is better than a note nobody needs — this one must not become noise."""
382+
from modscan.cli import _tie_note
383+
384+
points = [_FakePoint(s) for s in (1.0, 0.9, 0.8)]
385+
assert _tie_note(points, shown=2) is None
386+
assert _tie_note([], shown=10) is None
387+
assert _tie_note(points, shown=0) is None
388+
389+
390+
def test_detect_json_carries_tied_with() -> None:
391+
"""The prose note's machine-readable twin: a JSON consumer cannot read Markdown."""
392+
import io
393+
import json as _json
394+
from contextlib import redirect_stdout
395+
396+
pkg = "clities"
397+
with tempfile.TemporaryDirectory() as root:
398+
d = os.path.join(root, pkg)
399+
os.makedirs(d)
400+
open(os.path.join(d, "__init__.py"), "w").close()
401+
# two classes that score identically, and one that does not
402+
with open(os.path.join(d, "api.py"), "w", encoding="utf-8") as fh:
403+
fh.write(
404+
"class AlphaHandler:\n pass\n"
405+
"class BetaHandler:\n pass\n"
406+
"class Plain:\n pass\n"
407+
)
408+
try:
409+
buf = io.StringIO()
410+
with redirect_stdout(buf):
411+
assert main(["detect", root, "--json"]) == 0
412+
payload = _json.loads(buf.getvalue())
413+
by_name = {e["id"].rsplit(":", 1)[-1]: e for e in payload}
414+
415+
assert by_name["AlphaHandler"]["score"] == by_name["BetaHandler"]["score"]
416+
assert by_name["AlphaHandler"]["tied_with"] == 1
417+
assert by_name["BetaHandler"]["tied_with"] == 1
418+
assert by_name["Plain"]["tied_with"] == 0, "a unique score is tied with nothing"
419+
finally:
420+
_cleanup(pkg)
421+
422+
344423
def test_version_flag_prints_and_exits_zero() -> None:
345424
"""`--version` reports the installed version and exits 0.
346425
@@ -385,4 +464,9 @@ def test_version_flag_prints_and_exits_zero() -> None:
385464
test_consecutive_runs_are_independent()
386465
test_no_validate_examples_skips_preflight()
387466
test_version_flag_prints_and_exits_zero()
467+
test_tied_counts_report_the_band_not_the_position()
468+
test_tie_note_warns_when_the_limit_cuts_through_a_band()
469+
test_tie_note_reports_a_fully_shown_band_without_crying_truncation()
470+
test_tie_note_is_silent_when_the_cutoff_is_unambiguous()
471+
test_detect_json_carries_tied_with()
388472
print("OK: cli self-check passed")

0 commit comments

Comments
 (0)