Share prefix KV cache between instances and hosts through a selene pool manager

.dockerignore

@@ -28,8 +28,20 @@ eos/**
 selene/**
 !selene/Cargo.toml
 !selene/Cargo.lock
+!selene/build.rs
+!selene/examples/
+!selene/examples/**
+!selene/proto/
+!selene/proto/**
 !selene/src/
 !selene/src/**
+!selene-manager/
+selene-manager/**
+!selene-manager/Cargo.toml
+!selene-manager/tests/
+!selene-manager/tests/**
+!selene-manager/src/
+!selene-manager/src/**
 !ignition/recipes/qwen38/assets/
 !ignition/recipes/qwen38/assets/**
 !ignition/recipes/step5/assets/

.github/workflows/native-serving.yml

@@ -34,7 +34,13 @@ jobs:
         run: cargo build --locked --release --manifest-path flare/Cargo.toml
       - name: Test host OS helpers
         run: cargo test --locked --release --manifest-path eos/Cargo.toml --target-dir flare/target
-      - name: Test host cache tiers
+      - name: Test Selene and its manager on Linux
+        if: runner.os == 'Linux'
+        run: |
+          cargo test --locked --release --manifest-path selene/Cargo.toml --features test-support --target-dir flare/target
+          cargo test --locked --release --manifest-path selene-manager/Cargo.toml --features test-support --target-dir flare/target
+      - name: Test host cache tiers on Windows
+        if: runner.os == 'Windows'
         run: cargo test --locked --release --manifest-path selene/Cargo.toml --target-dir flare/target
       - name: Test native runtime on Linux
         if: runner.os == 'Linux'
@@ -65,3 +71,16 @@ jobs:
         with:
           name: flare-windows-native
           path: flare/target/release/flare.exe
+
+  rdma:
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install RDMA build dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y build-essential cmake ninja-build clang libclang-dev pkg-config protobuf-compiler libnl-3-dev libnl-route-3-dev libibverbs-dev librdmacm-dev liburing-dev
+      - name: Compile experimental RDMA
+        run: |
+          cargo check --locked --manifest-path selene/Cargo.toml --tests --features rdma
+          cargo check --locked --manifest-path selene-manager/Cargo.toml --tests --features rdma

Dockerfile

@@ -15,6 +15,7 @@ COPY flare/ flare/
 COPY helios/ helios/
 COPY eos/ eos/
 COPY selene/ selene/
+COPY selene-manager/ selene-manager/
 COPY ignition/recipes/qwen38/assets/ ignition/recipes/qwen38/assets/
 COPY ignition/recipes/step5/assets/ ignition/recipes/step5/assets/
 COPY ignition/recipes/kimi3/assets/ ignition/recipes/kimi3/assets/

docs/api.md

@@ -308,6 +308,13 @@ instantaneous gauges.
 | `host_bytes_in` | Bytes copied from the host store into GPU pages and rows | `flare_host_cache_bytes_in_total` |
 | `learned_drafter` | Boolean: the model has a learned drafter | none |
 
+Host retention also exports `flare_host_cache_held_total{tier}` and
+`flare_host_cache_dropped_total{tier,reason}`, with `dram` and `nvme` tiers and
+`end_of_chain`, `oversize`, `io_error` drop reasons. The gauge
+`flare_host_cache_parked_records` counts complete DRAM copies awaiting disk
+writes. `flare_host_cache_store_skipped_total` still includes duplicates and
+retries; it is not a dropped-record counter.
+
 Dashboard DEPTH is the change in `verification_drafts` divided by the change in
 `verification_rounds` over the sampling window. It averages each request's planned
 draft-prefix length, including zero-draft rounds, rather than graph width or batch

docs/cli-design.md

@@ -951,6 +951,7 @@ reserve = "0"                     # GPU memory left for processes started later.
 
 [prefix_cache]
 memory = ""                      # Host retention budget; empty: disabled.
+drain_on_shutdown = "off"         # Pooled DRAM persistence: on, off, or a duration.
 disk_limit = ""                  # Empty: 10% of free disk space when disk retention is enabled.
 disk_transport = ""              # Empty: staged; also accepts direct.
 
@@ -1049,6 +1050,20 @@ disabled unless configured. The memory setting includes registered checkpoint
 slabs and ordinary retained CPU inputs; bounded transfer, preprocessing, and
 decoded-source windows are accounted for separately at startup.
 
+`--prefix-cache-pool ADDRESS` accepts `unix:PATH` for a local socket,
+`HOST:PORT` for plain TCP, or `tls:HOST:PORT` for TLS. Network
+addresses require a nonempty host and a port from 1 through 65535; IPv6
+addresses use brackets, for example `[::1]:7000` or `tls:[::1]:7000`.
+The manager's `--name` supplies the pool salt for scope and instance identities.
+`--prefix-cache-pool-ca FILE` supplies the CA for a TLS manager;
+without it, TLS verifies the manager against the host's trust store. A CA
+cannot be used with a Unix socket or plaintext address. A manager declaring
+a different encryption setting disables the pool for TCP clients. Unix-socket
+clients trust filesystem permissions and can use the same TLS manager. The CA
+requires the pool flag. These command-line options are forwarded to
+every rank and have no configuration-file key. See
+[the pool section](prefix-cache-offload.md#215-sharing-through-a-pool).
+
 ## 4. Derived settings
 
 | Setting | Derived from |

docs/container.md

@@ -99,6 +99,7 @@ filename selection, and offline behavior.
 Disk-backed embedding tables and disk prefix-cache offload use Linux `io_uring`.
 The container's seccomp policy must permit `io_uring_setup`, `io_uring_enter`,
 and `io_uring_register` for those modes. Docker's default policy can block them;
-use a policy allowing those calls. Prefix-cache disk directories also need a
+use a policy allowing those calls, such as
+`selene-manager/deploy/seccomp-io-uring.json` (Docker's default plus io_uring). Prefix-cache disk directories also need a
 writable bind mount owned by the serving user. `--ulimit memlock=-1:-1` permits
 the pinned host storage used for transfers.