Share prefix KV cache across instances with the Selene pool

Add an optional Selene pool so Flare instances can share prefix KV
checkpoints across processes and hosts. Without --prefix-cache-pool,
Flare keeps using Selene's in-process host tiers as before.

- selene::shared holds the protocol and the definitions both sides use:
  wire messages, framing, rank slices, identity, limits, the record format
  and disk engine, the prefix index, regions, TLS setup and RDMA verbs.
- The selene manager (binary selene) owns one pool's DRAM and NVMe store,
  keeps an index of what every attached instance holds, and brokers
  lookup, claim, copy and release with fencing and quarantine. Copies use
  shared-memory windows on one host, a TCP or TLS data channel between
  hosts, relay through the manager, and experimental write-only RDMA.
- The pool client performs lookups, restores, exports and puts. A
  single-process instance is a world of one rank; multi-host
  tensor-parallel instances use per-rank slices and one rank link.
- Host tiers never drop a record mid-chain. Records move VRAM to DRAM to
  local NVMe to the pool, demoted ahead of need and dropped only at the
  end of the chain. An opt-in --prefix-cache-drain-on-shutdown persists
  DRAM-only records on a clean stop.
- Flare exposes checkpoint export chains and eviction headroom, attaches
  the pool at startup within its host and device budgets, and degrades to
  local tiers when the manager is unavailable.

Design-Signed-Off-By: yatesdr; design-review=2d93

Author: yatesdr

.dockerignore

@@ -28,8 +28,20 @@ eos/**
 selene/**
 !selene/Cargo.toml
 !selene/Cargo.lock
+!selene/build.rs
+!selene/examples/
+!selene/examples/**
+!selene/proto/
+!selene/proto/**
 !selene/src/
 !selene/src/**
+!selene-manager/
+selene-manager/**
+!selene-manager/Cargo.toml
+!selene-manager/tests/
+!selene-manager/tests/**
+!selene-manager/src/
+!selene-manager/src/**
 !ignition/recipes/qwen38/assets/
 !ignition/recipes/qwen38/assets/**
 !ignition/recipes/step5/assets/

.github/workflows/native-serving.yml

@@ -34,7 +34,13 @@ jobs:
         run: cargo build --locked --release --manifest-path flare/Cargo.toml
       - name: Test host OS helpers
         run: cargo test --locked --release --manifest-path eos/Cargo.toml --target-dir flare/target
-      - name: Test host cache tiers
+      - name: Test Selene and its manager on Linux
+        if: runner.os == 'Linux'
+        run: |
+          cargo test --locked --release --manifest-path selene/Cargo.toml --features test-support --target-dir flare/target
+          cargo test --locked --release --manifest-path selene-manager/Cargo.toml --features test-support --target-dir flare/target
+      - name: Test host cache tiers on Windows
+        if: runner.os == 'Windows'
         run: cargo test --locked --release --manifest-path selene/Cargo.toml --target-dir flare/target
       - name: Test native runtime on Linux
         if: runner.os == 'Linux'
@@ -65,3 +71,16 @@ jobs:
         with:
           name: flare-windows-native
           path: flare/target/release/flare.exe
+
+  rdma:
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install RDMA build dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y build-essential cmake ninja-build clang libclang-dev pkg-config protobuf-compiler libnl-3-dev libnl-route-3-dev libibverbs-dev librdmacm-dev liburing-dev
+      - name: Compile experimental RDMA
+        run: |
+          cargo check --locked --manifest-path selene/Cargo.toml --tests --features rdma
+          cargo check --locked --manifest-path selene-manager/Cargo.toml --tests --features rdma

Dockerfile

@@ -15,6 +15,7 @@ COPY flare/ flare/
 COPY helios/ helios/
 COPY eos/ eos/
 COPY selene/ selene/
+COPY selene-manager/ selene-manager/
 COPY ignition/recipes/qwen38/assets/ ignition/recipes/qwen38/assets/
 COPY ignition/recipes/step5/assets/ ignition/recipes/step5/assets/
 COPY ignition/recipes/kimi3/assets/ ignition/recipes/kimi3/assets/

docs/api.md

@@ -308,6 +308,13 @@ instantaneous gauges.
 | `host_bytes_in` | Bytes copied from the host store into GPU pages and rows | `flare_host_cache_bytes_in_total` |
 | `learned_drafter` | Boolean: the model has a learned drafter | none |
 
+Host retention also exports `flare_host_cache_held_total{tier}` and
+`flare_host_cache_dropped_total{tier,reason}`, with `dram` and `nvme` tiers and
+`end_of_chain`, `oversize`, `io_error` drop reasons. The gauge
+`flare_host_cache_parked_records` counts complete DRAM copies awaiting disk
+writes. `flare_host_cache_store_skipped_total` still includes duplicates and
+retries; it is not a dropped-record counter.
+
 Dashboard DEPTH is the change in `verification_drafts` divided by the change in
 `verification_rounds` over the sampling window. It averages each request's planned
 draft-prefix length, including zero-draft rounds, rather than graph width or batch

docs/cli-design.md

@@ -943,6 +943,7 @@ reserve = "0"                     # GPU memory left for processes started later.
 
 [prefix_cache]
 memory = ""                      # Host retention budget; empty: disabled.
+drain_on_shutdown = "off"         # Pooled DRAM persistence: on, off, or a duration.
 disk_limit = ""                  # Empty: 10% of free disk space when disk retention is enabled.
 disk_transport = ""              # Empty: staged; also accepts direct.
 
@@ -1040,6 +1041,20 @@ disabled unless configured. The memory setting includes registered checkpoint
 slabs and ordinary retained CPU inputs; bounded transfer, preprocessing, and
 decoded-source windows are accounted for separately at startup.
 
+`--prefix-cache-pool ADDRESS` accepts `unix:PATH` for a local socket,
+`HOST:PORT` for plain TCP, or `tls:HOST:PORT` for TLS. Network
+addresses require a nonempty host and a port from 1 through 65535; IPv6
+addresses use brackets, for example `[::1]:7000` or `tls:[::1]:7000`.
+The manager's `--name` supplies the pool salt for scope and instance identities.
+`--prefix-cache-pool-ca FILE` supplies the CA for a TLS manager;
+without it, TLS verifies the manager against the host's trust store. A CA
+cannot be used with a Unix socket or plaintext address. A manager declaring
+a different encryption setting disables the pool for TCP clients. Unix-socket
+clients trust filesystem permissions and can use the same TLS manager. The CA
+requires the pool flag. These command-line options are forwarded to
+every rank and have no configuration-file key. See
+[the pool section](prefix-cache-offload.md#215-sharing-through-a-pool).
+
 ## 4. Derived settings
 
 | Setting | Derived from |