Skip to content

Let the evidence leave the screen - #35

Merged
peopleworks merged 1 commit into
rules-that-misfirefrom
evidence-report
Aug 4, 2026
Merged

peopleworks merged 1 commit into
rules-that-misfirefrom
evidence-report

Conversation

@peopleworks

@peopleworks peopleworks commented Aug 4, 2026 •

Copy link
Copy Markdown
Owner

Closes #33. Depends on #34 — it branches from it, so merge that one first.

Until now the thing this project exists to produce died when the tab closed. A teacher could be shown every named tell, every hidden character with its line and column, every citation a document contradicts in its own bibliography — and forward none of it. That is a hole in the argument rather than a missing feature: a percentage is not evidence, and evidence nobody can hand to anyone is not much better.

The report saves as Markdown for pasting into an email or an LMS, or as a self-contained HTML page that opens by double-click and prints to PDF. No stylesheet, no script, no external request: it has to open from a downloads folder years later on a machine that has never heard of this project, and one that fetched anything would leak which documents were read and when. Buttons in the web app, the desktop app and the folder triage; --report <file> in the CLI.

What the review changed, which is most of what matters here

Two independent adversarial reviews ran against this branch before it was proposed. Between them they stopped it twice. Every severe claim was verified by hand before anything was touched.

It quoted a false-positive rate measured on another language. Docs/CALIBRATION.md already carried the sentence condemning this, one line above its own table: "a rate that holds in English and fails in Spanish is not one number, and reporting it as one would hide exactly the failure that matters here." English bounds at 5.6% on this corpus and Spanish at 13.3%; neither supports the 5% target alone, and both show a dash on that page. The report printed the aggregate 4.1% — handing the author of a Spanish essay a bound three times better than anything measured for their language, under the heading "A score is not proof". The embedded snapshot now carries the strata and the caveat reads the one for the language actually analysed, saying plainly when that language supports no threshold and that the overall figure is not a substitute.

It asserted a contradiction it had not checked. "None of this needed the internet: the document contradicts itself" printed whenever there was anything at all to say about sources — including directly beneath the line saying no reference list was found and the cross-checks had not run. A student who simply omitted a bibliography got a page accusing them in the report's own voice. Now conditional on an actual disagreement.

It opened with a verdict it spent four lines disowning. "Reads mostly human" above "treat the score as saying nothing" is a page arguing with itself, and a reader keeps whichever half suits them. Below the threshold the build can support, the number stands alone and the page says a low score is not evidence a person wrote something — a detector that detects nothing also returns zero.

A heading claimed something false about half its own table. "Characters that writing does not produce" covers soft hyphens and unusual spaces, which Word inserts by itself and any copy-paste carries. It now says what is in the file and names the innocent explanations.

User content could break the page. A | in a filename opened an extra table cell and shifted every number one column right — in the table a teacher reads scores against student names. A newline in an extractor's error closed the list and let the remainder become report prose; a line starting ## arrived as a heading in the report's own voice. Both verified, both fixed, both now have a regression test.

"Nought out of ninety" was hardcoded two lines under the same figures read from the calibration snapshot — the precise failure PublishedCalibration exists to prevent, committed inside it. The tool wrote that snapshot relative to the working directory, so running it from anywhere but the repo root left the real one stale. PublishedCalibration.Current could throw from a property getter, taking down every report over a malformed resource.

What the report now does

Three things on every page. It prints the error rate for the language analysed, from the upper end of the interval, and names the rules known to misfire so a reader can weigh evidence that leans on one. It puts the checkable facts in the headline rather than below the score — the document this project keeps writing about scores near zero and has an invented bibliography, and burying that would be choosing the wrong thing to make salient. And it states that it holds the document and was never uploaded, since the share card is built to be posted in public and confusing the two would publish a student's work.

The folder report is not the single-document one repeated. Above the table, before any name: "This is a reading order, not a ranking. A higher score means look sooner, and nothing more. Nothing on this page establishes that anyone did anything." Every file is listed rather than the first forty — it is the document a teacher keeps, and dropping 160 of 200, every low scorer among them, would be worse than not writing it. Files are named by their path under the scanned folder, since one directory per student makes "essay.docx" ambiguous exactly when it matters.

Known and filed rather than hidden

  • The evidence report is English-only, in a bilingual product #36 — the report is English-only, in a project whose Spanish half is its reason to exist. The caveats are the part we insist must be read, and they are the part a Spanish-speaking committee cannot read. It should not be presented to Spanish-speaking teachers as finished until that lands.
  • The report renders through a Markdown round-trip, and it costs it three ways #37 — it renders through a Markdown round-trip, which costs it three ways: truncation drops evidence by position rather than weight, ToMarkdown passes raw HTML through, and emphasis markers do not nest. Plus the two-phase build nothing enforces, and custom rule packs invalidating the published rate.

244 tests in Core, 323 across the solution, desktop solution included. All green.

Closes the first half of #33. Until now the thing this project exists to
produce — the named tells, the hidden characters at their line and column,
the citations a document makes that its own bibliography contradicts — died
when the tab closed. A teacher could be shown all of it and forward none of
it. That is a hole in the argument, not a missing feature: a percentage is
not evidence, and evidence you cannot hand to anyone is not much better.

The report comes out as Markdown for pasting into an email or an LMS, or as
a self-contained HTML page that opens by double-click and prints to PDF. No
stylesheet, no script, no external request — it has to open from a downloads
folder years later on a machine that has never heard of this project, and
one that fetched anything would leak which documents were read and when.

Three things it always does. It prints this build's own false-positive rate,
quoting the upper bound of the interval rather than the observed rate, and
names the rules known to misfire so a reader can weigh evidence that leans on
one — the sample report counts `rhet.in-terms-of` as a signal and says on the
same page that it fires on 10% of human academic writing. It separates
checkable facts from the judgement, because a DOI on two different works is
not an opinion about prose. And it states that it holds the document and was
never uploaded, since the share card is built to be posted in public and
confusing the two would publish a student's work.

Those numbers come from a calibration snapshot embedded at build time and
written by the same tool run that regenerates Docs/CALIBRATION.md, so the
page cannot quote a threshold three versions old. A fork that has never
measured itself ships no snapshot and the report says so, rather than
inheriting a figure it did not earn.

CLI: `--report <file>`, written after everything else so a failing
--max-score gate still leaves the document behind for whoever has to look
at it.

The Markdown-to-HTML conversion handles exactly what this file emits and
nothing more. The input is produced twenty lines above the converter, so
anything it cannot parse is a bug here rather than in someone's document,
and a real parser would be a dependency this library refuses to take.

Refs #33

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@peopleworks
peopleworks merged commit 9d1c830 into rules-that-misfire Aug 4, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant