Unfalsifiable requirements pass review, consume repeated review rounds, and inflate apparent coverage. Reviewers currently prove that a property holds; nothing asks whether it could fail.
Evidence
REQ-053.4.1 (panopticon, PR #224) — "leaving the horizontal scrollbar line unobscured." The overlay is implemented inside DataTable.render_line(), which renders only widget content lines; Textual composites scrollbars separately as a child widget. No diff of the reviewed component can violate this requirement. It went through three review rounds — two fails, then an elaborate pass comparing "every composited cell" against table.horizontal_scrollbar.render_line(0). That final test compares the scrollbar to itself through a path the implementation cannot influence: self-supplied evidence, passed as rigorous.
Independently reproduced: mutating task_content_bottom so the indicator draws onto the scrollbar row still passes, even with instrumentation confirming render_line is called at that y.
REQ-053.5.1, same PR — the parametrization/letter-versus-property cousin. "MUST remain able to reach every real task row." Three honesty-review rounds sharpened the letter (added upward traversal, identity checks) and never touched the property; the passing test asserts table.cursor_row == expected_row, an internal counter the change cannot affect. The cursor was in fact invisible for 21 of 30 rows. The spec's own verb invited the tautology, and no reviewer asked what diff would break it.
REQ-052.1.1 (panopticon) — 129 tests, 123 of them parametrized one-per-source-path, every one building a single-file universe. Restricting the digest to islice(_packaged_files(), 1) left the suite byte-identical to baseline. Line coverage would have read 100%. The universal quantifier was never exercised.
phoneopticon repeats both: after three fail rounds, REQ-009.3.x settled on asserting exact prose equality of a README section — mechanically strong, unfalsifiable for the actual property.
Why lint cannot do this
REQ-001.2.4 ("concrete, evaluable criterion") is a SHOULD with no emitter anywhere in src/. Falsifiability is undecidable statically — it depends on the implementation's structure, not the requirement's text. Attempting a lint rule here would produce false confidence.
Proposed: a prompt obligation in renderInstructions
Two added steps in the review packet:
-
"Name the concrete implementation change that would violate this requirement. If no diff of the reviewed component can violate it, FAIL the requirement as vacuous rather than passing its tests." This directly targets the REQ-053.4.1 class — and note that failing the requirement rather than the test is the right verdict, because the defect is in the spec.
-
"State whether the evidence exercises any universal quantifier over a population larger than one." Targets the parametrization illusion, where N isolated single-element cases masquerade as coverage of "any."
Gating
Do not ship this template change without regression-testing it against the calibration corpus (see the corpus-runner issue) — REQ-003.8.2 forbids template changes that lose a known catch, and there is currently no way to check that.
Priority
Quality, top tier. Costs a small amount of reviewer time per requirement; recovers far more by preventing multi-round oscillation on vacuous requirements.
Unfalsifiable requirements pass review, consume repeated review rounds, and inflate apparent coverage. Reviewers currently prove that a property holds; nothing asks whether it could fail.
Evidence
REQ-053.4.1(panopticon, PR #224) — "leaving the horizontal scrollbar line unobscured." The overlay is implemented insideDataTable.render_line(), which renders only widget content lines; Textual composites scrollbars separately as a child widget. No diff of the reviewed component can violate this requirement. It went through three review rounds — two fails, then an elaborate pass comparing "every composited cell" againsttable.horizontal_scrollbar.render_line(0). That final test compares the scrollbar to itself through a path the implementation cannot influence: self-supplied evidence, passed as rigorous.Independently reproduced: mutating
task_content_bottomso the indicator draws onto the scrollbar row still passes, even with instrumentation confirmingrender_lineis called at thaty.REQ-053.5.1, same PR — the parametrization/letter-versus-property cousin. "MUST remain able to reach every real task row." Three honesty-review rounds sharpened the letter (added upward traversal, identity checks) and never touched the property; the passing test assertstable.cursor_row == expected_row, an internal counter the change cannot affect. The cursor was in fact invisible for 21 of 30 rows. The spec's own verb invited the tautology, and no reviewer asked what diff would break it.REQ-052.1.1(panopticon) — 129 tests, 123 of them parametrized one-per-source-path, every one building a single-file universe. Restricting the digest toislice(_packaged_files(), 1)left the suite byte-identical to baseline. Line coverage would have read 100%. The universal quantifier was never exercised.phoneopticon repeats both: after three fail rounds,
REQ-009.3.xsettled on asserting exact prose equality of a README section — mechanically strong, unfalsifiable for the actual property.Why lint cannot do this
REQ-001.2.4("concrete, evaluable criterion") is a SHOULD with no emitter anywhere insrc/. Falsifiability is undecidable statically — it depends on the implementation's structure, not the requirement's text. Attempting a lint rule here would produce false confidence.Proposed: a prompt obligation in
renderInstructionsTwo added steps in the review packet:
"Name the concrete implementation change that would violate this requirement. If no diff of the reviewed component can violate it, FAIL the requirement as vacuous rather than passing its tests." This directly targets the
REQ-053.4.1class — and note that failing the requirement rather than the test is the right verdict, because the defect is in the spec."State whether the evidence exercises any universal quantifier over a population larger than one." Targets the parametrization illusion, where N isolated single-element cases masquerade as coverage of "any."
Gating
Do not ship this template change without regression-testing it against the calibration corpus (see the corpus-runner issue) —
REQ-003.8.2forbids template changes that lose a known catch, and there is currently no way to check that.Priority
Quality, top tier. Costs a small amount of reviewer time per requirement; recovers far more by preventing multi-round oscillation on vacuous requirements.