Skip to content

Add IOBuf<->std::iostream adapters for zero-copy I/O#51

Open
chenBright wants to merge 1 commit into
masterfrom
iobuf_iostream
Open

Add IOBuf<->std::iostream adapters for zero-copy I/O#51
chenBright wants to merge 1 commit into
masterfrom
iobuf_iostream

Conversation

@chenBright
Copy link
Copy Markdown
Owner

What problem does this PR solve?

Issue Number: resolve

Problem Summary:

IOBuf integrates with protobuf via IOBufAsZeroCopy{Input,Output}Stream, but
there is no direct bridge to the std::iostream hierarchy. As a result, any
library that consumes/produces std::istream& / std::ostream& — most notably
nlohmann::json — cannot read from or write to an IOBuf without an
intermediate std::string:

auto j = nlohmann::json::parse(req->request_attachment().to_string());

This is an obstacle to adopting nlohmann::json in bRPC.

What is changed and the side effects?

Changed:

Introduce four classes in src/butil/iobuf.{h,cpp} that bridge IOBuf and
the standard iostream hierarchy:

  • IOBufAsInputStreamBuf — read-only std::streambuf over an IOBuf,
    walking the backing blocks via the public backing_block() API. Overrides
    underflow, xsgetn(bulk memcpy across block boundaries to avoid the
    default per-byte sbumpc loop), and showmanyc (saturating sum of
    remaining block sizes).
  • IOBufInputStreamstd::istream view that wires the streambuf above.
  • IOBufAsOutputStreamBuf — append-only std::streambuf that delegates to
    the existing IOBufAsZeroCopyOutputStream for block allocation and the
    three-branch BackUp (including the shared-block degradation that protects
    IOBufs sharing blocks from being corrupted by an in-flight write). Overrides
    overflow, xsputn (block-sized memcpy instead of per-byte sputc),
    and sync (forwards to shrink()).
  • IOBufOutputStreamstd::ostream view that wires the streambuf above;
    accepts an optional block_size to allocate dedicated blocks instead of
    drawing from the per-thread TLS pool.

Usage with nlohmann::json:

  // Parse: zero string copy
  butil::IOBufInputStream in(request_body);
  auto j = nlohmann::json::parse(in);

  // Serialize: bytes flow straight into IOBuf blocks
  butil::IOBufOutputStream os(response_body);
  os << j;

Side effects:

  • Performance effects:

  • Breaking backward compatibility:


Check List:

@chenBright chenBright force-pushed the iobuf_iostream branch 2 times, most recently from 0a48a11 to 3028871 Compare May 19, 2026 12:16
Introduce four classes that bridge IOBuf and the standard iostream
hierarchy, enabling parsers and serializers that take std::istream& /
std::ostream& (e.g. nlohmann::json) to read from and write to IOBuf
directly without an intermediate string copy:

  - IOBufAsInputStreamBuf  : read-only streambuf over IOBuf blocks
  - IOBufInputStream       : std::istream view over IOBuf
  - IOBufAsOutputStreamBuf : append-only streambuf, reusing
                             IOBufAsZeroCopyOutputStream's TLS block pool
  - IOBufOutputStream      : std::ostream view over IOBuf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant