test: run html5lib-tests entity fixtures against the decoders

[fb55]

What this does

Vendors the WHATWG html5lib-tests suite as a git submodule at test/fixtures/html5lib-tests (pinned to upstream 224991e) and runs its tokenizer entity fixtures against this package's decoders in vitest (src/html5lib.spec.ts). The last few real bugs here were HTML-spec-conformance bugs (#2208 attribute-mode legacy entities, #2262 streaming over-consume); these fixtures pin spec behavior permanently and update themselves.

Each included fixture is asserted against both decode paths:

• decodeHTML / decodeHTMLAttribute (the direct trie walk), and
• a streaming EntityDecoder fed one character per chunk (the state machine htmlparser2 uses — a genuinely separate code path).

Counts (at html5lib-tests 224991e)

fixture file	total	included	excluded
entities.test	80	80	0
namedEntities.test	4210	4210	0
numericEntities.test	336	336	0
pendingSpecChanges.test	1	0	1
total	4627	4626	1

All 4626 included fixtures pass on both paths. No known-failures list is needed — no conformance divergences were found.

Filter rationale (mechanical, documented at the top of the spec)

The fixtures drive a full tokenizer; this library only decodes character references. Included are:

1. Data-state tests (no initialStates beyond Data state, no lastStartTag) whose output is Character tokens only and whose input contains no < (needs tag-context modelling) and no \r (needs input-stream preprocessing) → concatenated character data must equal decodeHTML(input).
2. Attribute-value template tests — fixtures emitting exactly one <h a=…> start tag with a single a attribute (the suite's fixed template for attribute entity tests) → expected attribute value must equal decodeHTMLAttribute on the extracted raw value.

The single excluded test (pendingSpecChanges.test) emits a Comment token. doubleEscaped is handled; errors arrays are ignored — this library doesn't emit tokenizer parse errors (documented limitation in the spec header).

A guard test asserts ≥4000 fixtures survive the filter, so a future submodule update can't silently filter everything away.

Missing-submodule behavior

• CI (CI env set): the spec fails hard with html5lib-tests fixtures are missing. Run \git submodule update --init` to fetch them.` — a conformance suite must not silently pass in CI.
• Locally (fresh clone without --recurse-submodules): graceful skip with a console.warn carrying the same instruction, so npm test still works out of the box.

Auto-updates & CI

• .github/dependabot.yml gains a gitsubmodule ecosystem entry (same daily schedule as the existing entries), so upstream fixture changes arrive as regular dependabot PRs — and a fixture update that breaks conformance fails that PR's CI, which is the point.
• .github/workflows/nodejs-test.yml: the test job's checkout gets submodules: true (only the test job needs it; lint/build don't). Note: this overlaps with the in-flight ci/node-versions branch which touches the same workflow — the diff here is kept to exactly the two submodules lines to minimize the conflict.

Packaging & licensing

• npm pack verified: neither the spec (excluded via !**/*.spec.ts) nor the submodule (test/ isn't in files) is published.
• html5lib-tests is MIT-licensed; as a git submodule it is neither vendored into this repo's sources nor published to npm, so there is no license contamination.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[coderabbitai]

Warning

Review limit reached

@fb55, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 51 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 631a0cad-225c-4969-bd7e-b6fb334d35ea

📥 Commits

Reviewing files that changed from the base of the PR and between 8700ff2 and dc474a2.

📒 Files selected for processing (5)

• .github/dependabot.yml
• .github/workflows/nodejs-test.yml
• .gitmodules
• src/html5lib.spec.ts
• test/fixtures/html5lib-tests

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch test/html5lib-entities-fixtures

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Pull request overview

This PR adds a conformance-oriented test harness that runs the upstream WHATWG html5lib-tests tokenizer entity fixtures against both decoding implementations in this repo (direct trie-walk APIs and the streaming EntityDecoder state machine), with CI configured to fetch the fixtures via a git submodule.

Changes:

• Add src/html5lib.spec.ts to load/token-filter html5lib tokenizer fixtures and assert results against decodeHTML / decodeHTMLAttribute and a char-by-char streaming decoder.
• Add html5lib-tests as a git submodule at test/fixtures/html5lib-tests.
• Update CI checkout to fetch submodules and add Dependabot updates for the submodule.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

File	Description
src/html5lib.spec.ts	New vitest spec that filters and runs html5lib tokenizer entity fixtures against both direct and streaming decoder paths.
.gitmodules	Registers test/fixtures/html5lib-tests as a git submodule pointing to the upstream repo.
.github/workflows/nodejs-test.yml	Ensures the test job checks out submodules so fixtures are present in CI.
.github/dependabot.yml	Adds gitsubmodule ecosystem updates so fixture updates arrive automatically.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.