Skip to content

Latest commit

 

History

History
60 lines (54 loc) · 8.67 KB

features.md

File metadata and controls

60 lines (54 loc) · 8.67 KB

Replay feature support

This matrix describes the built-in offline engine stack. A capability is supported only within its stated composition; individual supported features are not automatically supported together. Optional stacks advertise their own runner capabilities. Timing/profile availability is a separate performance-model question.

Feature Backend / execution Topology and workload Boundaries
Native token scheduling vLLM, SGLang, TensorRT-LLM Aggregated or P/D; synthetic and compatible traces Engine-specific admission/cache behavior; no real network or host-runtime overhead, except vLLM's optional pass launch costs
Attention-DP All three native backends Aggregated and independently sized P/D pools Rank-local scheduling and G1; one grouped pass completes at the maximum rank time
Native MTP Supported engine/model path Token or qualified agentic replay with explicit acceptance Separate from ngram; canonical MTP supports scoped G2 below, and a hybrid G2 with disable_eagle_block_drop; no grouped-cache/G3 combination; sub-block state-cache matching needs num_speculative_states; agentic accuracy is functional-only
Registered speculative schemes Offline vLLM/SGLang, subject to scheme/model constraints Prediction, aggregated or P/D; token or qualified agentic replay Existing linear 1–5-token executor; fixed KV capacity; default timing with supported OP cost; only MTP supports scoped G2 or a hybrid G2 with disable_eagle_block_drop; no trees, grouped-cache/G3 combination or generic-scheme recommendation; sub-block state-cache matching needs num_speculative_states
Ngram prompt lookup Offline vLLM Aggregated or P/D, op-level or explicit timing; HBM-only agentic replay Fixed draft/acceptance assumption; no lookup CPU cost, adapters or deployment generation
Sessions Native token replay Synthetic or compatible trace sessions Concurrency counts sessions through think time; see trace-format restrictions
Prefix reuse Native backend cache G1 with prefix caching enabled Actual reuse differs from Router overlap; complete block and backend admission rules apply; SGLang P/D decode reuse is opt-in through decode_radix_cache, without speculation
Request cache-salt isolation Native SGLang Aggregated or P/D; native requests and inline replay Different salts keep distinct KV identities and retained storage in shared memory pools; events preserve the namespace; built-in trace schemas remain unsalted
Grouped FPM cache vLLM Cold aggregated, PP1/CP1 HBM only, no speculation, prefix caching off, no scalar block capacity, no P/D
G2 host offload vLLM Aggregated and P/D, any attention-DP size Prefix caching, no Belady; canonical MTP with synchronous scheduling, where an aggregated worker needs a local pool and P/D workers take either scope, or MTP for a recurrent state cache with disable_eagle_block_drop; no other speculation; otherwise local or layout-compatible shared pool; a sliding-window state cache offloads under either offloading connector name and follows vLLM 0.32's OffloadingConnector on a local pool; an aggregated worker's local pool can follow vLLM 0.32's OffloadingConnector or eager SimpleCPUOffloadConnector (no G3 or KVCR), which a recurrent state cache requires
Local HiCache SGLang Aggregated prediction and native offline replay; local host pool per attention-DP rank Prefix caching; write-through, selective write-through or write-back; no P/D, recommendation, Belady, external storage or peer KVCR; excluded from HBM-only serving qualification
Agentic G2 vLLM Static aggregated or P/D workers of any count and attention DP Canonical MTP, MTP on a hybrid G2, or no speculation, per role; an aggregated worker with MTP needs a local pool; no G3; each role may omit G2 or use local/shared scope
G3 offload vLLM Aggregated, attention DP1; fixed or scaled pools Requires G2; no speculation, AgentX, P/D, recommendation or hardware integration
KVCR peer fetch vLLM Aggregated with any attention-DP size, or P/D prefill ranks; fixed or scaled pools Requires private G2 and hints from a placement policy or ReplaySpec requests (CLI traffic has none); no G3, decode-role KVCR or recommendation; idle under AgentX
Best-effort Belady All three native backends Fixed aggregated, DP1, complete open-loop input trace Native descriptor; prefix caching required; rejects closed-loop/generated/agentic/delta inputs, scaling and offload
Agentic dependency graph vLLM, SGLang Offline aggregated or P/D; Weka, Agentic Mooncake and agentic Dynamo traces One target model for P/D; speculation uses HBM, the canonical MTP/G2 composition or MTP on a hybrid G2; no agentic TensorRT-LLM; HBM or qualified vLLM G2
Seeded snapshots and warmup Same agentic boundary Timestamp load, positive lanes, unsigned 64-bit seed Request-boundary history, not a physical engine checkpoint; warmup preserves snapshot frontier
Continuous agentic profiles Same agentic boundary Snapshots plus profile duration; warmup optional No simultaneous virtual-time cap; grace can finish with unsettled server work; live host-memory supervision required
AIPerf metric selection Native offline Engine runner including AFD, predict and recommend CLI, SDK TTFT, request latency, per-request ITL and user TPS; avg, p50, p75, p90, p95 and p99 from the native summary Success-only request population; nearest-rank percentiles; optional strict constraints; no analytical EPD or failure-adjusted metrics. See definitions.
Replay telemetry Rust Replayer, Engine Python runner, CLI Settled virtual-time samples; opt-in all-phase request/lifecycle capture Offline vLLM/SGLang aggregated or P/D; consumer stacks have separate adapters
Detailed native artifacts RoundRobin Replayer Fixed single aggregated worker with attention DP1 No multiworker/scaling/P/D or shared G2 artifact claim; normal reports have broader scope
Router and Planner Optional Dynamo stack Consumer-supported topologies/workloads Separate installation, configuration ABI and downstream qualification
AFD pipeline timing with continuous replay SGLang Aggregated AFD or P/D with AFD on either or both roles; synthetic, trace and prefix-cached traffic No KV-relative load, encoder composition or native deployment renderer
Analytical EPD Offline engine analytical overlay Encoder + aggregate or P/D language workers; Qwen3-VL and Kimi K3 towers only Fixed synthetic lengths/images/concurrency; default op-level timing; aggregate means only, no adapters/per-request capture
Native SGLang VL replay SGLang; op-level timing for the vision tower Aggregated worker or P/D prefill worker; fixed synthetic images; PP1, attention DP1 Frontend stages from a host cost table per serving environment; no decode-side frontend, GPU image processor, video or output-side costs
Native encoder pools SGLang --encoder-only servers ahead of a --language-only worker Encoder + aggregate or P/D SGLang language workers; Qwen3-VL towers only (Kimi K3 refused at the image geometry, Llama 4 at the encoder builder); fixed synthetic images; static pools CPU preprocessing extrapolated from the host table; no adaptive local encoding, GPU-direct transfer or encoder prefix cache

Validation and evidence

The authoritative gates are RunnerCapabilities.require_compatible, EngineReplayRunner, public engine validation, and the native engine and Replay validators. Native validation remains necessary when callers bypass public YAML.

Qualification includes fixed-timing lifecycle tests, Python/native report checks and CLI artifact checks. AgentX qualification does not establish full AgentX benchmark parity or measured model accuracy. The retained replay evidence keeps scoped GPU/simulator comparisons, revisions and caveats separate from feature acceptance. See engine, KV cache, and traffic-format compatibility for the detailed constraints.