Skip to content

s2/lex: FRONT lexer, errors as data - #14

Merged
vic merged 1 commit into
mainfrom
s2-lex
Nov 9, 2026
Merged

vic merged 1 commit into
mainfrom
s2-lex

Conversation

@vic

@vic vic commented Nov 9, 2026

Copy link
Copy Markdown
Owner

FRONT rung 1: the lexer (s2/lex.srs, module Lex). Tokens and layout follow s1/lex.srs.

  • One self-tail-recursive scan produces identifiers (Upper.name is one), ints with their value, strings and chars, and one- and two-character ops.
  • Layout: a deeper line opens an indent; a shallower one closes every indent above it; a newline inside ( or [ is no token.
  • Tokens are typed cases. A tab, a bad dedent, an unterminated literal or a bad character gives Lex.Bad(line, why), never an exit.

Tests: a snippet with every layout case, a comment, a dotted name, a newline inside parens, a char literal and a two-character op; each error kind; four real sources lex with balanced layout and one Eof. Four mutants (no depth decrement, no dotted names, no dedent tokens, no newline suppression) each change the output. Local tests/run.sh: 184 ok, 0 FAIL.

Lex.lex scans the source in one self-tail-recursive function: identifiers (Upper.name is one), ints
with their value, strings and chars (raw text), one- and two-character ops, and layout: a deeper line
opens an indent, a shallower one closes every indent above it, a newline inside ( or [ is no token, a
blank or comment line is nothing. Tokens are typed cases; a tab, a bad dedent, an unterminated
literal or a bad character is Lex.Bad(line, why), never an exit.

tests/s2/lex: a snippet with an indent, a blank line, two dedents at once, a comment, a dotted name, a
newline inside ( ), a char literal and a two-character op; each error kind; and lib/prelude.srs,
tests/rules.srs, tests/file_ok.srs and s2/coll.srs lex with balanced layout and one TEof. Mutants: no
depth decrement, no dotted names, no dedent tokens, no newline suppression: each changes the output.
@vic
vic merged commit 392b681 into main Nov 9, 2026
3 checks passed
@vic
vic deleted the s2-lex branch November 9, 2026 04:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant