Catch out-of-range numeric character references (#591) - #592
Closed
ChrisJr404 wants to merge 1 commit into
Closed
Conversation
The loose parser resolved every numeric character reference with
chr().encode("utf-8"), so a reference whose value overflowed the Unicode
range (�), landed above the maximum code point (�),
or was a lone surrogate (�) raised OverflowError, ValueError, or
UnicodeEncodeError and aborted the whole parse.
Wrap the conversion so those references fall back to their literal text,
which is how the HTML processor already handles them. Fixes kurtmckee#591.
Owner
|
I suspect this was generated by AI. I'm closing this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fixes #591. The loose parser resolved numeric character references with
chr(c).encode("utf-8"), so a reference that overflows the Unicode range like�, sits above the maximum code point like�, or is a lone surrogate like�would raiseOverflowError,ValueError, orUnicodeEncodeErrorand take down the whole parse.I wrapped just the conversion in a try/except and fall back to keeping the reference as literal text when it can't be turned into a character. That mirrors what
_BaseHTMLProcessor.handle_charrefin html.py already does for the same input, so the two paths now agree instead of one of them crashing.Added a small test module covering the three exception types plus a valid reference to make sure normal ones still resolve, and a changelog fragment. The three examples from the issue are used directly as test cases.