A '#' inside a { ... } constant shadow-parses to end of line: O(literals x line) in the tau grammar

[castrod]

Symptom

With tau-lang's parser/tau.tgf (main as of 2026-09-01, parser pin 9e78949), one always line holding N guarded comparisons written with hex literals such as { #x0003e8 }:bv[24] parses in time roughly proportional to N × line length:

input	predict inserts	peak RSS	parse
16 clauses, 5.6 KB, 112 hex literals	22 M	0.73 GB	2.0 s
64 clauses, 22 KB, 448 hex literals	277 M	6.4 GB	31.7 s
same 22 KB with decimal literals	25 M	0.64 GB	1.9 s

Two # per literal double the cost; -x or zz in the same position cost nothing. This is what made tau-testnet's per-user policy clauses (one long always line, one hex threshold per guard) look quadratic in the number of guards.

Mechanism

ba_constant => '{' _ (capture | source) _ '}' [ typed ].
_           => __ | null.
__          => space | comment | __ space | __ comment.
comment     => '#' (printable | '\t')* ('\n' | '\r' | eof).

The optional whitespace before the constant body admits a comment starting at the literal's own #. Earley then has to carry that alternative to the end of the line for every such #: one item chain per origin, O(line) each. The left-recursive repetition expansion (#17) does not change this, because the chains are per origin, not per prefix.

Grammar-side fix (landed in tau-lang, branch feedback/clause-compiling): ba_constant => '{' space* (capture | source) space* '}' [ typed ]. The 22 KB input then parses in 1.9 s / 0.64 GB.

A general engine fix (design only)

"Run" nonterminals. A nonterminal whose productions are R => R c | null with c a character class (or a disjunction of classes and terminals) is a regular run. When an item P => α • R β is predicted at set s and FIRST(β) is a non-nullable terminal or class disjoint from c, only the maximal run [s, e) can continue into β, so the engine can add P => α R • β directly at e (with e memoised per (R, s) by a linear scan) and never create the O(e − s) interior items. The forest builder would materialise R over [s, e) lazily, only when the root derivation reaches it, so shadow runs that die at their follower cost O(1) per origin.

Until something like that exists, the grammar-hygiene rule is: keep comment productions out of the optional whitespace used inside opaque bodies.

Reproducer

tau-meta bench_clause_compile.py, or any always line built from (i1[t]:bv[24] > { #x0003e8 }:bv[24] && !(i18[t]:bv[8] = { #xa0 }:bv[8])) terms joined by ||; measure with tgf parser/tau.tgf parse -i <file> -m.