engine describes the deployment that replays the workload:
the model, hardware and backend, the worker pools and their parallelism,
scheduler limits, KV cache, P/D transfer and speculative decoding. Every
predict and recommend configuration has one engine block next to
traffic.
engine:
mode: aggregated # deployment shape
model: Qwen/Qwen3-32B-FP8
hardware: h200_sxm
backend: vllm
workers: # one entry per worker role
aggregated:
parallelism: {} # replicas, TP, PP, attention DP, MoE TP/EP, CP
scheduler: {} # batch limits and backend cadence
kv_cache: {} # G1 capacity, prefix caching, G2/G3 offload
timing: {} # forward-pass timing provider
# kv_transfer: {...} # P/D only
# speculation: {...} # optional ngram speculative decoding
# ais_perf_model: {...} # optional perf model of each worker typeThe smallest valid block sets model, hardware and an empty worker role:
engine:
model: Qwen/Qwen3-32B-FP8
hardware: h200_sxm
workers:
aggregated: {}With every other field omitted, it is equivalent to:
engine:
mode: aggregated
model: Qwen/Qwen3-32B-FP8
hardware: h200_sxm
backend: vllm
context_length: max
workers:
aggregated:
parallelism: {replicas: 1, tensor: 1, pipeline: 1, attention_data: 1, moe_tensor: 1, moe_expert: 1}
scheduler: {max_batched_tokens: 8192, max_sequences: 256}
kv_cache:
block_size: 64
prefix_caching: true
bytes_per_token: auto
capacity: {type: default, memory_fraction: 0.9}
timing: {type: default}
startup_seconds: 0In recommend, omitted mode, backend and parallelism fields become search
domains instead of the single values above; see
Sweeper search space.
| Block | Page |
|---|---|
engine.mode, model, hardware, backend, backend_version, context_length, workers roles, tokenization, detokenization |
This page |
engine.workers.<role>.{hardware, startup_seconds, parallelism, scheduler}, engine.enable_chunked_prefill |
Workers |
engine.workers.<role>.kv_cache |
KV cache |
engine.kv_transfer |
P/D KV transfer |
engine.speculation, engine.nextn, engine.nextn_accepted |
Speculative decoding |
engine.workers.<role>.afd, engine.workers.encoder |
Analytical AFD and EPD |
engine.workers.{aggregated,prefill}.{host_loop, frontend, host_profile, vision} |
Workers; design on SGLang VL host loop |
engine.ais_perf_model |
This page; the value format is on Performance-model configuration |
engine.workers.<role>.timing, quantization and kernel selectors, and the deprecated estimator fields (estimation_mode, database_mode and the others listed under Perf model) |
Performance-model configuration |
Engine prediction and recommendation accept auto, op_level,
fpm_interpolation, fpm_regression, and fpm_learned in their global and
worker timing selection schemas. Replay prices every pass of a regression or
learned role with one ForwardPassMetrics query over the worker's
attention-DP ranks, adding the per-request lengths a learned artifact reads,
and rejects such a role when its model lacks a workload store the role needs,
hosts the vision encoder or uses speculative decoding other than the MTP depth
a vLLM regression was fitted at; see
engine replay timing.
These roles are latency-only, so their replay reports no power.
Field tables on these pages use the following columns:
- Default is the value used when the field is omitted in
predict. - Recommend is
fixedwhenrecommendaccepts only a concrete value,predict onlywhenrecommendrejects the field, or the default search domain whenrecommendsearches it.
| Knob | Default | Recommend | Rules |
|---|---|---|---|
engine.mode |
aggregated |
{choices: [aggregated, disaggregated]} |
aggregated or disaggregated. AFD is a per-role worker type described in Analytical AFD and EPD. |
engine.model |
Required | fixed | Hugging Face ID or local model directory. |
engine.hardware |
Required | auto allowed |
System identifier, such as h200_sxm. auto is recommendation-only and resolves from optimization.hardware. |
engine.backend |
vllm |
{choices: [vllm, sglang]} |
vllm, sglang, or trtllm. Selects both the scheduler semantics and the performance data. |
engine.backend_version |
current |
fixed | Selects performance data only. It does not change scheduler behavior. Must be a queryable version for the hardware and backend: current, previous when configured, next when newer data is available, or a version one of those available aliases resolves to. See backend versions. |
engine.context_length |
max |
fixed | Positive prompt-plus-output token limit, or max. With a number, prompts at or above the limit are rejected and generation stops at the limit. max applies the model config's maximum on vLLM; SGLang and TensorRT-LLM then run without a limit. |
engine.workers |
Required | Per mode | Role mappings; see Worker roles. A role may be {} to use all defaults. |
engine.ais_perf_model |
Unset | fixed | One perf-model config, or a list with one config per worker type. Sets the estimator of each worker and replaces the deprecated estimator fields; see Perf model. Rejected with AFD, an encoder pool, and fixed or polynomial timing. |
engine.tokenization |
Unset (off) | predict only | Optional workers (positive integer, default 1), base_ms and per_input_token_us (finite and non-negative, default 0). Tokenizes each request on its worker before the engine; see Request tokenization. Aggregated and disaggregated token replay on --stack engine only; online replay, worker scaling, AFD, encoder pools and SGLang frontend stages (which already price the tokenizer) are rejected. |
engine.detokenization |
Unset (off) | predict only | Optional workers (positive integer, default 1), base_ms and per_output_us (finite and non-negative, default 0). Detokenizes each pass's outputs on the worker that emits tokens before the client receives them; see Response detokenization. Aggregated and disaggregated token replay on --stack engine only; online replay, worker scaling, AFD, encoder pools and traffic.load.agentic_profile are rejected. |
A serving engine tokenizes each request in its server process before the
scheduler sees it, and longer prompts take longer. engine.tokenization models
this on every worker that receives new requests: the aggregated workers, or the
prefill workers in a disaggregated deployment.
engine:
tokenization:
workers: 1
base_ms: 0.7
per_input_token_us: 1.6Each worker has its own tokenizer with workers servers. After placement, a
request waits for a free server on its worker, takes
base_ms + per_input_token_us × input tokens on it, and then reaches that
worker's engine. Servers are taken first come, first served.
- With
workers: 1, requests on a worker are tokenized one at a time in arrival order. A long prompt delays every request behind it, and requests reach the engine in arrival order. - With more workers, a short prompt can finish before a long one that arrived earlier and reach the engine first.
Without this block, requests reach each engine at their arrival time, and requests that arrive together keep their authored order; see same-timestamp admission.
Set workers to the number of requests your engine tokenizes concurrently per
worker. The corresponding engine options are:
| Engine | Option | Default |
|---|---|---|
| vLLM | --renderer-num-workers per API server, times --api-server-count |
1, times 1 without data parallelism |
| SGLang | --tokenizer-worker-num |
1 |
TensorRT-LLM (trtllm-serve) |
--num_input_processor_workers, where available |
8 |
Measure base_ms and per_input_token_us on your own deployment. One way is
to send prompts of different lengths one at a time and fit the time from
request arrival to engine admission against input length. The per-token cost
depends mostly on the model's tokenizer and the host CPU. The values above
are only an illustration.
- TTFT and end-to-end latency are measured from the arrival time, so they include tokenization, as a client would observe.
- With
--capture-per-request, each record hastokenized_ms, the time the request reached its worker's engine. Subtract it fromfirst_token_msfor the engine-side TTFT, or fromfirst_admit_msfor the engine queue time. - Under
load.type: concurrency, a request being tokenized holds its concurrency slot. - With
traffic.stop.max_virtual_time_seconds, a request still being tokenized at the cutoff is reported as incomplete. - When an agentic profile cancels a request that is still being tokenized, the request never reaches the engine. Its tokenizer time stays reserved, so requests behind it on that worker are not released earlier.
- Setting both costs to
0is the same as omitting the block.
A serving engine returns generated tokens through its server processes, which
detokenize each request's new tokens, format streamed chunks and write them to
the clients. The client therefore receives every token, the first one
included, some time after the engine produced it, and more so when many
requests are running. engine.detokenization models this on every worker
that emits tokens: the aggregated workers, or the decode workers in a
disaggregated deployment.
engine:
detokenization:
workers: 1
base_ms: 0.5
per_output_us: 40Each worker has workers detokenizer servers, shared by its attention-DP
ranks, and each request belongs to one of them. When the engine returns a
pass's outputs, at the pass end or, with vLLM's
launch, once the next pass is launched, each
server takes the outputs of that pass that belong to it as one job, after the
jobs it has taken before. A job takes base_ms + per_output_us × outputs, where an
output is one request's new tokens in the pass, and the client receives every
output of the job when it ends.
- A pass with many running requests therefore delays all of its tokens, and when passes end faster than a server finishes its jobs, later passes wait for it.
- With more workers, the outputs of a pass are split over the servers and each server handles fewer of them.
- A request is assigned to a server by a stable hash of its request ID. This stands in for how the serving engine spreads connections over its server processes.
- A job's outputs reach the client together. vLLM hands a pass's outputs to
the connections in chunks of
VLLM_V1_OUTPUT_PROC_CHUNK_SIZE(128 by default), so with more than 128 outputs per server in one pass, replay delivers the earlier chunks somewhat late.
Set workers to the number of server processes that detokenize outputs. The
corresponding engine option is:
| Engine | Option | Default |
|---|---|---|
| vLLM | --api-server-count |
1 without data parallelism; with internal data-parallel load balancing, the data-parallel size |
Measure base_ms and per_output_us on your own deployment: record when the
engine produces each pass's outputs and when the client receives each token,
at a few concurrencies, and fit the delays. They depend mostly on the serving
frontend and the host CPU. The values above are only an illustration.
- Token times, TTFT, TPOT, end-to-end latency and throughput use the time the
client receives each token. With
--capture-per-request,first_token_ms,last_token_msandterminal_time_msare client receive times. - A request finishes when the client receives the end of its response. A request that waits on it, such as the next request of a concurrency slot or a session's next turn, is released at that time.
- A rejected request carries no output, so it skips detokenization: the client learns of the rejection when the engine rejects the request, in both aggregated and disaggregated deployments, and the rejection does not delay the other outputs of its pass.
- The TTFT stage breakdown of an SGLang
host_loopworker addsmean_detokenization_ms, the time from the engine emitting the first token to the client receiving it, so the stages still sum to TTFT. Without detokenization this stage is zero and TTFT ends when the engine emits the first token. - With
traffic.stop.max_virtual_time_seconds, a request whose response the client has not received by the cutoff is reported as incomplete. - Replay telemetry samples count a completion when the engine finishes the request, before the client receives it.
- Setting both costs to
0is the same as omitting the block.
The accepted versions are the populated slots defined by
query_versions.yaml:
currentis the newest completed full collection for the backend, a systematic campaign over the whole case plan on the fleet;previousis the full collection thatcurrentreplaced;nextis the in-development version: the highest version newer thancurrentfor which any measurements have landed (single-op or single-model drops; whatever it lacks is served fromcurrentand older data). When nothing newer thancurrenthas data,nextis the same version ascurrent.
Versions outside these slots, including some support-matrix PASS rows, are
rejected with an error listing the accepted versions. Set
engine.backend_version: current or use a version in that list.
See the support-matrix version boundary.
engine.workers holds one mapping per worker role. The roles that must be
present follow engine.mode:
mode |
Required roles | Rejected roles |
|---|---|---|
aggregated |
aggregated |
prefill, decode, and engine.kv_transfer |
disaggregated |
prefill and decode |
aggregated |
| AFD worker type | The same aggregated or P/D roles | Each role independently chooses ordinary or AFD execution; see AFD. |
Each ordinary (type: monolithic) role has the same sub-blocks (parallelism, scheduler, kv_cache,
timing, plus hardware and startup_seconds), configured independently. In
recommend, a mode domain may declare all three roles; each candidate keeps
only the roles its mode uses. An optional encoder role adds an analytical or
native EPD encoder pool; see EPD.
All roles share model, backend and backend_version;
per-role overrides of these fields are rejected. Only disaggregated prefill
and decode workers may override hardware.
Workers may override the shared context_length; see
context limits for prediction and recommendation rules.
engine.ais_perf_model sets the perf model that times each worker's forward
passes. It replaces the estimator fields under engine and under
engine.workers.<role>.timing.
engine:
mode: disaggregated
model: Qwen/Qwen3-32B-FP8
hardware: h200_sxm
workers:
prefill: {}
decode: {}
ais_perf_model:
- worker_type: prefill
estimation_mode: fpm_interpolation
fallback_policy: deny
systems_paths: [./fpm-data, default]
- worker_type: decode
estimation_mode: op_levelWhen every worker uses the same settings, write one config without
worker_type:
engine:
ais_perf_model:
estimation_mode: op_level
fallback_policy: deny- Worker types. Aggregated mode uses
aggregated. Disaggregated mode usesprefillanddecode. Inrecommend, a mode domain uses the worker types of every listed mode, and an omitted mode uses all three. A list needs an entry for each of these worker types and none for another. - Estimator keys. An entry sets
estimation_mode,fallback_policy,estimator_config,systems_paths,database_mode,transfer_policyandfpm_profile. - Other keys. Keys such as
model,systemortpare only checked against the engine section. A different value is an error.recommendcan check only a few of them. Inpredict, a key that is not a perf-model field is an error when the configuration is validated. from. An entry can start from a template file (from: ./base.yaml) or from a state snapshot (from: {state_snapshot: {path: ...}}). A string as the whole value is an error; writefrom: <file>to reuse a file.- Relative paths. In a configuration file they start at that file's
folder. Given with
--set, they start at the working directory. Symbolic links are followed the way the system follows them; see Relative paths. - Repeated keys. A key written twice inside
engine.ais_perf_model, at any depth, is an error in a configuration file:engine.ais_perf_model in predict.yaml: the key 'estimation_mode' is written twice (lines 9 and 11); write it once. So isais_perf_modelwritten twice in theenginesection, and anenginesection written twice when one of them writesais_perf_model; the error then namesthe key 'ais_perf_model'orthe key 'engine'.predictandrecommendexit with code2before they touch the output directory. A value given with--setis not checked, and any other repeated key in the file keeps its last value.
In this release the key is another spelling of the estimator fields. Entries
that agree are the engine-level spelling, entries that differ the per-worker
spelling. The example above is the same as the three prefill keys under
engine.workers.prefill.timing and estimation_mode: op_level under
engine.workers.decode.timing. Saved configurations and the candidate YAML
of recommend show those fields, not ais_perf_model.
Limits in this release:
- An AFD worker (
engine.workers.<role>.type: afd), an encoder pool (engine.workers.encoder), andtiming.type: fixedorpolynomialon an active worker reject the key. They keep the estimator fields. - A dataset entry (
from: {dataset: ...}) is rejected inpredictandrecommend. - A snapshot entry in
predictstill needs every saved estimator control written out.
The full rules, with the checks for each command, are in Unified perf-model input.
These fields are deprecated. They work as before in this release, and the next
release removes them. Use engine.ais_perf_model.
| Deprecated field | In an entry of engine.ais_perf_model |
|---|---|
engine.estimation_mode, fallback_policy, estimator_config, systems_paths, database_mode, transfer_policy, fpm_profile |
The same key, in one config or in every entry. |
engine.workers.<role>.timing.estimation_mode, fallback_policy, estimator_config, systems_paths, database_mode, transfer_policy |
The same key, in the entry for that worker type. |
engine.systems_path, engine.workers.<role>.timing.forward_model, engine.workers.<role>.timing.fpm_parquet_path |
See the list of old fields. |
- A configuration file that writes one of them prints one deprecation warning that lists the fields. A field that only repeats its default does not count.
aisim perf-model migrate prediction.yamlshows the converted block and writes nothing. Add--output prediction.yamlto convert the file in place; comments and formatting of the file are not kept. See perf-model tools.- An old field next to
engine.ais_perf_modelis an error, unless it holds the value the entries produce. timing.type,timing.prefill_msandtiming.decode_msare not deprecated.
engine:
mode: disaggregated
model: Qwen/Qwen3-32B-FP8
hardware: h200_sxm
backend: sglang
workers:
prefill:
parallelism: {replicas: 2, tensor: 2}
decode:
parallelism: {replicas: 1, tensor: 4}
scheduler: {max_sequences: 512}
kv_transfer:
bandwidth_gb_per_second: 50This deployment uses 2 × 2 + 1 × 4 = 8 GPUs. Run it with:
aisim predict --stack engine --config prediction.yaml --output-dir ./pd-predictionReplay validates the engine block before running. Unsupported combinations,
such as a backend-specific field on another backend or G3 offload in P/D,
fail with an error naming the field. They are never silently ignored. For which
features combine with which backends and topologies, see
feature support.