[javieratomic]

Why

A rank export receives each cache payload object as a complete companion wrapper in a synced temporary file in the core cache. store_slice then hashes the received payload, copies it into a new wrapper in cache staging, syncs the copy, and hashes the installed copy again: three full reads and one write and sync per object, on the receiving stream's thread. When the received file already is the wrapper store_slice would write, the copy, its write and its sync repeat work whose result is already on disk. In a DeepSeek V4.1 Flash TP3 export on three DGX Spark GB10 hosts with an empty core cache, rank zero receives 203.5 GB from the other two ranks this way.

What changed

Before, the receiver always called store_slice. Now it first calls the new Root::adopt_companion, which links the received file itself into cache staging when it is byte-for-byte that wrapper. That removes the copy read, the write and the sync; both hashes remain.

The link step is shared with directory publication, which already hard-linked source files that are canonical wrappers from its own code in export.rs; that code moves into container::link_companion, which:

• checks the length and leading bytes against companion_prefix;
• links the opened file through its descriptor, so that a replaced path cannot be linked instead;
• checks that the link names the same inode;
• hashes the linked file against the declared identity. A mismatched identity is an error.

For a file that is not the canonical wrapper, or a link the filesystem refuses, link_companion returns false, adopt_companion returns None, and the receiver stores the slice as before. Adoption checks and links the descriptor the receiver opened, the same one store_slice copies from. An already cached identity returns the verified cached object, as store_slice does. Publication is unchanged: the object is made immutable, linked under its identity and verified once more, and the receiver closes its writable descriptor before the file is published.

Core format, cache layout and object identities are unchanged. Directory publication keeps its behavior, error kinds and messages. Adoption needs Linux (linkat through /proc/self/fd); on other platforms received objects are copied as before.

Result

Expected: with an empty core cache, "Receive rank export payloads" no longer copies, writes and syncs each received object. Measured on an earlier tree with an earlier version of this adoption (below), that phase took 280.6 s instead of 457.8 s. This PR has not been measured on its current base.

Evidence

Tests on this branch:

• helios cargo test --release --features server,runtime-core --lib, with FLARE_TEST_TEMPLATE_DIR prepared by tests/frontend/prepare_templates.py: 1335 passed, 0 failed, 26 ignored.
• heliopause cargo test --release: 190 passed, 0 failed, 3 ignored.
• A helios test covers:
  • adoption by link (same inode);
  • a wrapper with an extra tensor and a wrapper of the right length with different header bytes, both declined;
  • a mismatched identity, which is rejected;
  • a second arrival of a cached identity, which returns the cached object;
  • that cache staging is empty after the declined, rejected and cached cases.
• A heliopause test checks that the receiver's install step adopts a canonical wrapper (same inode) and copies another framing into a new file.
• The existing directory publication tests cover the shared link step.

The 203.5 GB is the byte count reported by "Receive rank export payloads" in an export on d2b5c5f3 without this change, with an empty core cache; there that phase took 462.3 s.

Measurement on an earlier tree: full TP3 exports on three DGX Spark GB10 hosts with the same model and export command, an empty core cache (then named the package cache) and an empty forward compilation cache. The tree was ca1f960e plus an earlier version of #124 that published cache payloads concurrently, without the directory publication changes merged as e13ecb74. Its adoption step linked the received file by path instead of through the shared link_companion. Phase names are as reported then.

	without adoption	with adoption
Receive rank export payloads	457.8 s (8 streams), 477.2 s (--transfer-streams 16)	280.6 s (8 streams)
Export native packages	1677.4 s	1473.2 s

The package exported with adoption served with flare run. It answered a 262k-token prompt that asks for 16 values placed through the context with all 16 correct, with identical output with an empty and with a warm prefix cache.

Limitations:

• One export per configuration was measured, on a different tree and with a different link step.
• The payload is still read twice after it arrives (identity check and post-installation verification) instead of being hashed while it streams.
• Publish core cache payloads on a largest-first work queue and read ahead in slice digests #124 also changes hashing in this phase: its read-ahead applies to Slice::digest of payloads of at least 32 MiB while fewer than 16 digests are reading ahead, which both store_slice and adoption use for the identity check. The two changes were not measured together.

🤖 Generated with Claude Code

[javieratomic]

Rebased onto bbad44fe. The code is unchanged apart from the package-to-core rename. Directory publication's link step, which this PR moves into container::link_companion, is identical on the old and new base, and on both it already runs on up to 8 concurrent installation workers; link_companion keeps no shared state. The cache payload checks that af0907d9 made concurrent do not use the link step. The branch merges cleanly with the #124 head 50c664a3. On the rebased branch: helios cargo test --release --features server,runtime-core --lib: 1335 passed, 0 failed, 26 ignored; heliopause cargo test --release: 190 passed, 0 failed, 3 ignored. The received-bytes figure (203.5 GB, previously 203.7 GB) now comes from an export on d2b5c5f3 without this PR, in which "Receive rank export payloads" took 462.3 s. The adoption measurement is still the earlier one on a different tree; this PR has not been measured on its current base.