This matrix describes the built-in offline engine stack. A capability is
supported only within its stated composition; individual supported features
are not automatically supported together. Optional stacks advertise their own
runner capabilities. Timing/profile availability
is a separate performance-model question.
| Feature | Backend / execution | Topology and workload | Boundaries |
|---|---|---|---|
| Native token scheduling | vLLM, SGLang, TensorRT-LLM | Aggregated or P/D; synthetic and compatible traces | Engine-specific admission/cache behavior; no real network or host-runtime overhead, except vLLM's optional pass launch costs |
| Attention-DP | All three native backends | Aggregated and independently sized P/D pools | Rank-local scheduling and G1; one grouped pass completes at the maximum rank time |
| Native MTP | Supported engine/model path | Token or qualified agentic replay with explicit acceptance | Separate from ngram; canonical MTP supports scoped G2 below, and a hybrid G2 with disable_eagle_block_drop; no grouped-cache/G3 combination; sub-block state-cache matching needs num_speculative_states; agentic accuracy is functional-only |
| Registered speculative schemes | Offline vLLM/SGLang, subject to scheme/model constraints | Prediction, aggregated or P/D; token or qualified agentic replay | Existing linear 1–5-token executor; fixed KV capacity; default timing with supported OP cost; only MTP supports scoped G2 or a hybrid G2 with disable_eagle_block_drop; no trees, grouped-cache/G3 combination or generic-scheme recommendation; sub-block state-cache matching needs num_speculative_states |
| Ngram prompt lookup | Offline vLLM | Aggregated or P/D, op-level or explicit timing; HBM-only agentic replay | Fixed draft/acceptance assumption; no lookup CPU cost, adapters or deployment generation |
| Sessions | Native token replay | Synthetic or compatible trace sessions | Concurrency counts sessions through think time; see trace-format restrictions |
| Prefix reuse | Native backend cache | G1 with prefix caching enabled | Actual reuse differs from Router overlap; complete block and backend admission rules apply; SGLang P/D decode reuse is opt-in through decode_radix_cache, without speculation |
| Request cache-salt isolation | Native SGLang | Aggregated or P/D; native requests and inline replay | Different salts keep distinct KV identities and retained storage in shared memory pools; events preserve the namespace; built-in trace schemas remain unsalted |
| Grouped FPM cache | vLLM | Cold aggregated, PP1/CP1 | HBM only, no speculation, prefix caching off, no scalar block capacity, no P/D |
| G2 host offload | vLLM | Aggregated and P/D, any attention-DP size | Prefix caching, no Belady; canonical MTP with synchronous scheduling, where an aggregated worker needs a local pool and P/D workers take either scope, or MTP for a recurrent state cache with disable_eagle_block_drop; no other speculation; otherwise local or layout-compatible shared pool; a sliding-window state cache offloads under either offloading connector name and follows vLLM 0.32's OffloadingConnector on a local pool; an aggregated worker's local pool can follow vLLM 0.32's OffloadingConnector or eager SimpleCPUOffloadConnector (no G3 or KVCR), which a recurrent state cache requires |
| Local HiCache | SGLang | Aggregated prediction and native offline replay; local host pool per attention-DP rank | Prefix caching; write-through, selective write-through or write-back; no P/D, recommendation, Belady, external storage or peer KVCR; excluded from HBM-only serving qualification |
| Agentic G2 | vLLM | Static aggregated or P/D workers of any count and attention DP | Canonical MTP, MTP on a hybrid G2, or no speculation, per role; an aggregated worker with MTP needs a local pool; no G3; each role may omit G2 or use local/shared scope |
| G3 offload | vLLM | Aggregated, attention DP1; fixed or scaled pools | Requires G2; no speculation, AgentX, P/D, recommendation or hardware integration |
| KVCR peer fetch | vLLM | Aggregated with any attention-DP size, or P/D prefill ranks; fixed or scaled pools | Requires private G2 and hints from a placement policy or ReplaySpec requests (CLI traffic has none); no G3, decode-role KVCR or recommendation; idle under AgentX |
| Best-effort Belady | All three native backends | Fixed aggregated, DP1, complete open-loop input trace | Native descriptor; prefix caching required; rejects closed-loop/generated/agentic/delta inputs, scaling and offload |
| Agentic dependency graph | vLLM, SGLang | Offline aggregated or P/D; Weka, Agentic Mooncake and agentic Dynamo traces | One target model for P/D; speculation uses HBM, the canonical MTP/G2 composition or MTP on a hybrid G2; no agentic TensorRT-LLM; HBM or qualified vLLM G2 |
| Seeded snapshots and warmup | Same agentic boundary | Timestamp load, positive lanes, unsigned 64-bit seed | Request-boundary history, not a physical engine checkpoint; warmup preserves snapshot frontier |
| Continuous agentic profiles | Same agentic boundary | Snapshots plus profile duration; warmup optional | No simultaneous virtual-time cap; grace can finish with unsettled server work; live host-memory supervision required |
| AIPerf metric selection | Native offline Engine runner including AFD, predict and recommend CLI, SDK | TTFT, request latency, per-request ITL and user TPS; avg, p50, p75, p90, p95 and p99 from the native summary | Success-only request population; nearest-rank percentiles; optional strict constraints; no analytical EPD or failure-adjusted metrics. See definitions. |
| Replay telemetry | Rust Replayer, Engine Python runner, CLI | Settled virtual-time samples; opt-in all-phase request/lifecycle capture | Offline vLLM/SGLang aggregated or P/D; consumer stacks have separate adapters |
| Detailed native artifacts | RoundRobin Replayer | Fixed single aggregated worker with attention DP1 | No multiworker/scaling/P/D or shared G2 artifact claim; normal reports have broader scope |
| Router and Planner | Optional Dynamo stack | Consumer-supported topologies/workloads | Separate installation, configuration ABI and downstream qualification |
| AFD pipeline timing with continuous replay | SGLang | Aggregated AFD or P/D with AFD on either or both roles; synthetic, trace and prefix-cached traffic | No KV-relative load, encoder composition or native deployment renderer |
| Analytical EPD | Offline engine analytical overlay | Encoder + aggregate or P/D language workers; Qwen3-VL and Kimi K3 towers only | Fixed synthetic lengths/images/concurrency; default op-level timing; aggregate means only, no adapters/per-request capture |
| Native SGLang VL replay | SGLang; op-level timing for the vision tower | Aggregated worker or P/D prefill worker; fixed synthetic images; PP1, attention DP1 | Frontend stages from a host cost table per serving environment; no decode-side frontend, GPU image processor, video or output-side costs |
| Native encoder pools | SGLang --encoder-only servers ahead of a --language-only worker |
Encoder + aggregate or P/D SGLang language workers; Qwen3-VL towers only (Kimi K3 refused at the image geometry, Llama 4 at the encoder builder); fixed synthetic images; static pools | CPU preprocessing extrapolated from the host table; no adaptive local encoding, GPU-direct transfer or encoder prefix cache |
The authoritative gates are
RunnerCapabilities.require_compatible,
EngineReplayRunner,
public engine validation,
and the native engine and
Replay validators. Native validation remains
necessary when callers bypass public YAML.
Qualification includes fixed-timing lifecycle tests, Python/native report checks and CLI artifact checks. AgentX qualification does not establish full AgentX benchmark parity or measured model accuracy. The retained replay evidence keeps scoped GPU/simulator comparisons, revisions and caveats separate from feature acceptance. See engine, KV cache, and traffic-format compatibility for the detailed constraints.