Skip to content

fix issue leading to slowdown/unbounded memory growth in binary json processing#1199

Open
tomqext wants to merge 2 commits intoRobotWebTools:ros1from
Extend-Robotics:ros1_fix_decode_binary_websocket_frames_before_json_parse
Open

fix issue leading to slowdown/unbounded memory growth in binary json processing#1199
tomqext wants to merge 2 commits intoRobotWebTools:ros1from
Extend-Robotics:ros1_fix_decode_binary_websocket_frames_before_json_parse

Conversation

@tomqext
Copy link
Copy Markdown

@tomqext tomqext commented Mar 25, 2026

Public API Changes
None

Description

When a WebSocket client sends JSON as binary frames (ros-sharp does this), onMessage in the autobahn handler only decodes text frames - binary frames get pushed to the IncomingQueue as raw bytes. In Protocol.incoming(), the buffer concatenation does self.buffer = self.buffer + str(message_string), which in Python 3 produces the repr rather than decoding:

>>> str(b'{"topic":"foo"}')
"b'{\"topic\":\"foo\"}'"

This isn't valid JSON (leading b'), so json.loads() fails on every message and falls through to the bracket-scanning fallback (designed for reassembling partial TCP fragments). That fallback iterates over every {/} pair trying json.loads() on each substring - O(n²) in the number of brackets. It does eventually find the JSON inside the garbage, so everything worked, just very slowly.

In our case we were publishing 52-transform messages at 50Hz from ros-sharp (~322 bracket pairs per message). The fallback took ~48ms per message vs ~2.6ms for the actual deserialization + publish. The consumer thread couldn't keep up, so the IncomingQueue (unbounded deque) grew without bound - memory climbed indefinitely and data fell further and further behind real-time as the session progressed.

The fix decodes bytes before buffering instead of calling str(). The bracket-scanning fallback is still there for TCP fragmentation, it just stops being the hot path for every binary WebSocket message.

Most rosbridge clients (roslibjs, roslibpy, C++ clients) send text frames so they never hit this - the str() call is a no-op on an already-decoded string. ros-sharp is the main client that sends binary frames which is why this went undetected

@tomqext tomqext changed the title fix issue leading to ~20x slowdown in binary json processing (leading to infinitely growing queues/memory usage) fix issue leading to slowdown/unbounded memory growth in binary json processing Mar 25, 2026
tomqext added a commit to Extend-Robotics/rosbridge_suite that referenced this pull request Mar 25, 2026
@tomqext tomqext closed this Mar 25, 2026
@tomqext tomqext deleted the ros1_fix_decode_binary_websocket_frames_before_json_parse branch March 25, 2026 12:10
@tomqext tomqext restored the ros1_fix_decode_binary_websocket_frames_before_json_parse branch March 25, 2026 12:11
@tomqext tomqext reopened this Mar 25, 2026
@bjsowa
Copy link
Copy Markdown
Member

bjsowa commented Mar 27, 2026

Is this only related to the ros1 branch?

@bjsowa bjsowa added the ros1 ros1 label Mar 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ros1 ros1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants