fix: do not crash on a backslash line continuation after a reformatted docstring - #378
Open
ekanshul wants to merge 1 commit into
Open
fix: do not crash on a backslash line continuation after a reformatted docstring#378ekanshul wants to merge 1 commit into
ekanshul wants to merge 1 commit into
Conversation
When a docstring edit shifts the lines that follow it, _do_update_token_indices() decides whether a token stays on the previous token's row by comparing its (not yet shifted) start row with the (already shifted) end row of the previous token. A backslash continuation emits no NL token between the two physical lines, so for the first token of the continuation line that comparison is satisfied by coincidence whenever the shift is one row, the token is put on the previous token's row, and tokenize.untokenize() raises "start (r,c) precedes previous end (r,c)". Treat a token whose previous token's physical line ends with a backslash as starting a new row, which is what the tokenizer guarantees. Fixes PyCQA#377
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #377.
When a docstring edit shifts the lines after it,
_do_update_token_indices()decides whether a token stays on the previous token's row withwhere
tokens[i - 1]has already been shifted andtokens[i]has not. A parenthesised continuation has anNLtoken between the physical lines, which routes the next token to the "new row" branch; a backslash continuation has noNLtoken, so for the first token on the continuation line the comparison is satisfied by coincidence whenever the shift is one row, the token is placed on the previous token's row, andtokenize.untokenize()raisesValueError: start (r,c) precedes previous end (r,c).The fix treats a token as starting a new row when the previous token's physical line ends with a backslash, which is exactly what the tokenizer guarantees for a continuation. Two cases in
do_format_code.toml(class docstring and module docstring, each followed by a backslash continuation) fail onmasterwith theValueErrorand pass with the change.Running
docformatter --checkover the Python 3.14 standard library, this crash was hit in_sitebuiltins.py,pdb.py,tokenize.py,ctypes/util.py,email/feedparser.py,encodings/utf_16.py,encodings/utf_32.py,encodings/utf_8_sig.pyandhttp/cookies.py; with the fix all nine format, the output still parses, and the diff against the original is limited to docstrings.