ValueError: start precedes previous end, for a backslash line continuation after a reformatted docstring

[ekanshul]

docformatter 1.7.8 (and current master, d5c7b77) crashes with the tokenizer ValueError whenever a docstring edit shifts the lines of a file that contains a backslash line continuation somewhere after that docstring. Running docformatter --check over the Python 3.14 standard library hits it in _sitebuiltins.py, pdb.py, tokenize.py, ctypes/util.py, email/feedparser.py, encodings/utf_16.py, encodings/utf_32.py, encodings/utf_8_sig.py and http/cookies.py.

Reproducer

class A:
    """Doc."""
    x = 1 \
        + 2

$ docformatter --check t.py
Traceback (most recent call last):
  ...
  File ".../docformatter/format.py", line 919, in _do_format_code
    _code = tokenize.untokenize(self.new_tokens)
  File ".../tokenize.py", line 178, in add_whitespace
    raise ValueError("start ({},{}) precedes previous end ({},{})"
ValueError: start (4,8) precedes previous end (4,9)

A module docstring followed directly by x = 1 \ / + 2 fails the same way. Adding a blank line after the docstring (so nothing needs to be inserted) makes it pass, and so does writing the continuation with parentheses instead of a backslash. Same result on Python 3.11 and 3.14.

Cause

docformatter inserts the missing blank line after the docstring, which shifts every following token down one row. _do_update_token_indices() then recomputes positions and decides whether a token continues the previous token's row with

is_same_position = tokens[i].start[0] == tokens[i - 1].end[0]

where tokens[i - 1] has already been shifted and tokens[i] has not. For a parenthesised continuation an NL token sits between the two physical lines, so the comparison is never reached; a backslash continuation produces no NL token, so for the first token on the continuation line the un-shifted row happens to equal the shifted row of the previous token, the token is treated as being on the same row, and its start ((4,8)) ends up before the previous token's end ((4,9)).

I have a fix (treat a token as starting a new row when the previous token's physical line ends with \) with a regression test; PR follows.