Skip to content

perf(parser): make get_all_text() O(nodes) - #378

Merged
D4Vinci merged 3 commits into
D4Vinci:devfrom
yetval:perf/get-all-text-single-pass
Jul 23, 2026
Merged

perf(parser): make get_all_text() O(nodes)#378
D4Vinci merged 3 commits into
D4Vinci:devfrom
yetval:perf/get-all-text-single-pass

Conversation

@yetval

@yetval yetval commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Problem

get_all_text() decides whether each text node is visible by walking up through all of its ancestors to see if any of them is in ignore_tags. It does that walk once per text node, so the cost grows with both the number of text nodes and the depth of the tree (O(text_nodes x depth)). On deep pages this gets slow.

Fix

We already know which elements are ignored, so instead of walking ancestors every time, I collect the ignored tags and all of their descendants into a set once up front. After that, checking a node is just a single set lookup, which makes the whole thing O(nodes).

The method still uses .//text() under the hood, so comments, CDATA, and tail text are handled exactly the way they were before. Nothing about the API or the options changes, and every caller gets the speedup for free.

Result

Output is byte for byte identical to the old version across 2240 combinations of HTML shapes, options, and keep_comments settings. The existing suite passes (139 tests), and I added regression tests for the comment-tail and deep-nesting cases.

Timings on a deeply nested page:

Tree depth Before After Speedup
200 0.54 ms 0.10 ms 5.5x
800 6.87 ms 1.02 ms 6.7x
2000 43.2 ms 7.91 ms 5.5x

The old timings climb faster than the tree grows, while the new ones scale linearly.

@rkfshakti rkfshakti left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix is clean and correct. Replacing the O(n^2) ancestor walk with a direct set lookup is the right approach — ignored_elements already contains all descendants via element.iter(), so a single owner not in ignored_elements check is sufficient.

The regression tests cover the key edge cases: comment tail text, deep nesting, and ignored subtrees. The depth=500 test gives confidence this won't regress on real-world pages.

One minor observation: the if element not in ignored_elements guard in the loop is redundant since set.update is idempotent, but it's harmless and makes the intent clearer.

@D4Vinci
D4Vinci merged commit 1208199 into D4Vinci:dev Jul 23, 2026
5 checks passed
@D4Vinci

D4Vinci commented Jul 23, 2026

Copy link
Copy Markdown
Owner

Nice work as always @yetval

@D4Vinci D4Vinci mentioned this pull request Jul 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants