Skip to content

feat: chunked file transfer protocol - #369

Open
llogen wants to merge 1 commit into
mainfrom
feat/chunked-file-transfer
Open

feat: chunked file transfer protocol#369
llogen wants to merge 1 commit into
mainfrom
feat/chunked-file-transfer

Conversation

@llogen

@llogen llogen commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Chunked file transfer

Files now move between client and agent in 1 MiB chunks, each individually acknowledged, instead of a single protobuf message carrying the whole payload.

The old design held the entire file in memory on both ends, and while that one message was in flight nothing else on the Run stream moved — including module output and stdin. Chunks interleave with the rest, so memory stays bounded at one chunk per side and logs keep flowing during a transfer.

Protocol. Four new messages in both RunRequest and RunResponse: FileMetadata, FileTransferRequest (carries a Direction), FileChunk, FileTransferResponse. Field numbers 3 and 4 are reserved rather than reused, so an old peer fails to parse instead of silently misreading a chunk as a whole file.

Module API. SendFile gains a size parameter and returns as soon as the transfer is registered — bytes stream afterwards. Modules must therefore not defer srcFile.Close(): the session takes ownership and closes it on completion, but only on a nil return. flash and file are updated.

Flow control. An upload keeps one chunk outstanding, and the agent acknowledges only after the module has consumed it — real end-to-end backpressure, so the client cannot outrun the module.

Safety. The client only writes to paths named in the command arguments; anything else is refused and the agent drops the transfer. Transfers outlive the module's Run, so the agent waits for them before tearing down the workers.

CI. New protobuf job running buf lint, buf format, and a generated-code drift check. buf breaking is deliberately omitted while the wire format is still moving.

Agent and client must be upgraded together — this is a deliberate wire break.


Open question: bounding transfers

There's currently no timeout, size cap, or concurrency limit on transfers. Every teardown path is safe — cancellation, disconnect, refusal and protocol errors all release cleanly — so this is a resource-limit question, not a liveness one.

I'd rather not pick a constant, because the sensible ceiling looks hardware-dependent: a Pi driving slow SPI flash and a rack server have different limits, so it probably belongs in agent config. Worth agreeing on the shape first:

  • Which limit actually helps — stall timeout, max size, or concurrency cap? They cover different failures.
  • Per agent, per device, or per command? flash write legitimately moves hundreds of megabytes.
  • Is a default needed at all, given agents sit on a trusted network next to their hardware?

@llogen
llogen marked this pull request as draft July 27, 2026 06:33
Files move between client and agent in bounded, individually
acknowledged chunks, so large transfers stream reliably within the
run.

Signed-off-by: llogen <christoph.lange@blindspot.software>
@llogen
llogen force-pushed the feat/chunked-file-transfer branch from ea122c3 to 54cdde7 Compare July 27, 2026 08:00
@llogen
llogen marked this pull request as ready for review July 27, 2026 08:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant