feat: chunked file transfer protocol - #369
Open
llogen wants to merge 1 commit into
Open
Conversation
llogen
marked this pull request as draft
July 27, 2026 06:33
Files move between client and agent in bounded, individually acknowledged chunks, so large transfers stream reliably within the run. Signed-off-by: llogen <christoph.lange@blindspot.software>
llogen
force-pushed
the
feat/chunked-file-transfer
branch
from
July 27, 2026 08:00
ea122c3 to
54cdde7
Compare
llogen
marked this pull request as ready for review
July 27, 2026 08:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Chunked file transfer
Files now move between client and agent in 1 MiB chunks, each individually acknowledged, instead of a single protobuf message carrying the whole payload.
The old design held the entire file in memory on both ends, and while that one message was in flight nothing else on the
Runstream moved — including module output and stdin. Chunks interleave with the rest, so memory stays bounded at one chunk per side and logs keep flowing during a transfer.Protocol. Four new messages in both
RunRequestandRunResponse:FileMetadata,FileTransferRequest(carries aDirection),FileChunk,FileTransferResponse. Field numbers 3 and 4 are reserved rather than reused, so an old peer fails to parse instead of silently misreading a chunk as a whole file.Module API.
SendFilegains asizeparameter and returns as soon as the transfer is registered — bytes stream afterwards. Modules must therefore notdefer srcFile.Close(): the session takes ownership and closes it on completion, but only on a nil return.flashandfileare updated.Flow control. An upload keeps one chunk outstanding, and the agent acknowledges only after the module has consumed it — real end-to-end backpressure, so the client cannot outrun the module.
Safety. The client only writes to paths named in the command arguments; anything else is refused and the agent drops the transfer. Transfers outlive the module's
Run, so the agent waits for them before tearing down the workers.CI. New
protobufjob runningbuf lint,buf format, and a generated-code drift check.buf breakingis deliberately omitted while the wire format is still moving.Agent and client must be upgraded together — this is a deliberate wire break.
Open question: bounding transfers
There's currently no timeout, size cap, or concurrency limit on transfers. Every teardown path is safe — cancellation, disconnect, refusal and protocol errors all release cleanly — so this is a resource-limit question, not a liveness one.
I'd rather not pick a constant, because the sensible ceiling looks hardware-dependent: a Pi driving slow SPI flash and a rack server have different limits, so it probably belongs in agent config. Worth agreeing on the shape first:
flash writelegitimately moves hundreds of megabytes.