fix(websocket): tighten frame-bound checks across libunit and protocol

[andypost]

Summary

Audit-driven WebSocket framing safety pass (PR-A from security-audit.md
/ PR #10). CVE-track — bundles seven findings across libunit, the C
protocol layer, the Java JNI surface, and the Python ASGI handler.

Findings addressed

• V10 [High] WS header hsize OOB read (src/nxt_unit.c, combined with the buf.free advance — one guard covers both)
• V10 [Medium] WS buf.free advance past buf.end (src/nxt_unit.c, same guard as above)
• V10 [Medium] nxt_unit_websocket_retain hsize unchecked (src/nxt_unit.c)
• V12 [High] 64-bit payload MSB not validated (src/nxt_h1proto_websocket.c, callsite-side per audit option (b))
• V12 [High] Frame-size decrement no-op (src/nxt_http_websocket.c)
• V9 [Medium] Java sendWsFrame missing bounds (src/java/nxt_jni_Request.c)
• V8 [Medium] ASGI WS pending_payload_len overflow (src/python/nxt_python_asgi_websocket.c)

Departures from the audit

• V12 RSV bits — skipped. The C router does not negotiate Sec-WebSocket-Extensions; it is forwarded opaquely to language modules, and Java/Node apps negotiate permessage-deflate themselves (src/java/nginx/unit/websocket/PerMessageDeflate.java, src/nodejs/unit-http/websocket_request.js). Unconditionally rejecting RSV bits in nxt_h1proto_websocket.c would regress every working compressed-WebSocket deployment. The audit's "gate the rejection on no negotiated extension" requires an extension hook the C router does not have. Re-file once such a hook exists.
• V9 exception class — IllegalStateException is thrown instead of the audit-suggested IllegalArgumentException, to reuse the existing global-ref pool in nxt_jni.c rather than wiring a fourth exception class through nxt_java_jni_init and its cleanup ladder. Semantically close enough for the bounds-violation case.
• V12 MSB validation — applied at the nxt_h1proto_websocket.c parser (audit option (b)) rather than inside the nxt_websocket_frame_payload_len() helper, to avoid sentinel-value coupling at the four other call sites that do not need the check.

Test plan

• ./configure --openssl && make -j$(nproc) unitd — sandbox preparing this PR did not allow build commands; CI must validate.
• ./configure python --config=python3-config --module=python3 && make python3 (covers V8 ASGI fix).
• ./configure java && make java (covers V9 JNI fix; requires JDK).
• pytest-3 --collect-only test/test_python_websockets.py test/test_java_websockets.py (collection only — full run needs root and a built daemon).
• Manual: send a 2-byte client frame with payload_len=127 to confirm V10 OOB-read guard fires (warn + clean release) instead of crashing.
• Manual: send a payload_len=127 frame with the high bit of the 8-byte extended length set; expect protocol-error close (Invalid extended payload length).

Upstream

Same fixes apply to freeunitorg/freeunit; will forward after merge.
Maintainer should consider disclosure timing — V10 hsize OOB and V12 frame_size loop bug are the closest to RCE-shaped findings in the audit.

Generated by Claude Code

[gemini-code-assist]

Code Review

This pull request strengthens WebSocket frame-bound validation across libunit, the H1 protocol, and Java/Python bindings to prevent out-of-bounds reads, ensure RFC compliance for extended lengths, and guard against integer overflows. A critical bug was identified in the Python ASGI handler where an overflow check fails to clean up the frame queue, potentially leading to memory leaks and inconsistent state.

[gemini-code-assist]

The overflow check in nxt_py_asgi_websocket_suspend_frame returns early without cleaning up the state. At this point, the pending frame structure p has already been inserted into the ws->pending_frames queue (line 710), and the frame itself is not released. This leaves the WebSocket object in an inconsistent state (the queue contains a frame not accounted for in pending_payload_len) and leaks the frame. The frame should be removed from the queue, p should be freed, and nxt_unit_websocket_done(frame) should be called before returning.

    if (nxt_slow_path(frame->payload_len
                      > UINT64_MAX - ws->pending_payload_len))
    {
        nxt_unit_req_alert(ws->req,
                           "pending_payload_len overflow on suspend.");

        nxt_queue_remove(&p->link);
        nxt_unit_free(frame->req->ctx, p);
        nxt_unit_websocket_done(frame);

        PyErr_SetString(PyExc_RuntimeError,
                        "pending_payload_len overflow on suspend.");
        return;
    }

[andypost]

Gemini's [high] finding on ASGI overflow cleanup addressed in 18716620: the overflow check now runs before nxt_queue_insert_tail, so the failure exit no longer leaves a frame in pending_frames that isn't accounted for in pending_payload_len. On overflow we also nxt_unit_free(ws->req->ctx, p) and nxt_unit_websocket_done(frame) to avoid leaking the suspended-frame slot and the frame itself. Force-pushed; build clean.

[andypost]

CI fix in 3bbbf2d4: %z → %zu at nxt_unit.c:1694 — clang -Werror (Wformat-invalid-specifier) caught the typo in the truncated-frame warning. Force-pushed; local clean rebuild.

[andypost]

Last CI fix in b353b24b: test_asgi_websockets_length_long was asserting close code 1009 (CLOSE_TOO_LARGE), but my V12 MSB-of-64-bit-length fix now rejects length = 2**64-1 (high bit set) earlier with 1002 (CLOSE_PROTOCOL_ERROR), which is what RFC 6455 §5.2 actually mandates for that malformed frame. Updated the test to expect 1002 with an inline comment explaining the ordering — the test was previously relying on the policy size-limit to catch what should be a protocol-layer rejection.