Fix crash on standalone comments in comprehensions and lambdas

[Pedro-Muller29]

Description

Fixes #4296.

Three inputs from the issue crash on the first formatting pass:

[
    x for
    # comment
    x, y in ["AB",]
]

[
    [
        x for x
        # comment
        in [
            # comment
            "ABC"
        ]
    ]
]

{(lambda
    # comment
    x: [
        # comment
    ]
)}

All of them blow up with Cannot parse: ... bad input because Black dumps a string where a standalone comment sits inline next to the following token, and Python's tokenizer swallows everything to the next newline.

Root cause

Two distinct things.

For cases B and C, transform_line runs through its splitters and, when nothing succeeds, falls through to yield line. Line.__str__ concatenates leaves without inserting newlines, so a STANDALONE_COMMENT mid-stream renders glued to whatever follows. Line.append_safe enforces "comment on its own line" only at bracket depth 0 or inside an open for / lambda. For comments living inside arbitrary nested brackets, there's no guard.

For case A, standalone_comment_split does pull the comment between for and the tuple target onto its own line, but the resulting sub-line x, y in ["AB",] no longer carries the for keyword. Its bracket tracker walks fresh, the comma between x and y lands at depth 0, gets COMMA_PRIORITY, and delimiter_split breaks the tuple apart. Black also tacks a stray trailing comma onto the iterable.

Fix

• linegen.py: when no transformer succeeds and the line still carries standalone comments, fall back to _force_standalone_comment_split, which yields each comment on its own line. The alternative is invalid Python, so the unconditional split is safe.
• brackets.py: is_split_after_delimiter returns 0 for commas whose parent is exprlist and whose grandparent is comp_for / old_comp_for / for_stmt. Those commas are tuple-packing, not list delimiters.

The fixture covers the three issue cases plus async-for and a nested tuple target (for (a, b), c in ...) to lock both fixes in.

Flagging that case B comes out with loose indentation:

[
    [x for x
    # comment
    in [
    # comment
    "ABC"]]
]

The inner [...] doesn't re-explode, so the comment and "ABC"]] sit at the wrong depth. Let me know if there's a target shape you'd want, or if this is fine for the crash fix.

Checklist - did you ...

• Implement any code style changes under the --preview style, following the stability policy?
• Add an entry in CHANGES.md if necessary?
• Add / update tests if necessary?
• Add new / update outdated documentation?

[github-actions]

diff-shades results comparing this PR (514b235) to main (13e97b4):

--preview style: no changes

--stable style: no changes

---

What is this? | Workflow run | diff-shades documentation

[cobaltt7]

Thanks!

Flagging that case B comes out with loose indentation: ...
The inner [...] doesn't re-explode, so the comment and "ABC"]] sit at the wrong depth. Let me know if there's a target shape you'd want, or if this is fine for the crash fix.

Yeah, that's not ideal indentation. However, that's already handled with the hug_parens_with_braces_and_square_brackets style (even without this change, but the change is necessary for some of the other failures):

[[
    x
    for x
    # comment
    in [
        # comment
        "ABC"
    ]
]]

On the other hand, that style is currently restricted to --unstable due to #4036 and some other issues.

If possible, it'd be nice to improve that here as well, but if it's a lot of extra code it's fine to merge as-is and depend on hug_parens_with_braces_and_square_brackets.