This guide documents the legacy aiconfigurator cli compatibility commands that still ship
with AISim. For the unified aisim CLI, see the
AISim CLI User Guide and
AIC migration guide.
As mentioned in root Readme, CLI supports six modes: default, recommend, exp, generate, estimate, and support. We'll go through these modes one by one.
Quantization defaults are inferred from the Hugging Face model config (config.json plus optional hf_quant_config.json).
For low-precision models, use a quantized HF ID (for example, Qwen/Qwen3-32B-FP8) or a local model directory containing those files.
These flags are shared across modes (a few are sweep-only, as noted):
--log-level LEVEL: Set the minimum log level (CRITICAL,ERROR,WARNING,INFO,DEBUG). Priority:--log-level > AICONFIGURATOR_LOG_LEVEL > --debug > INFO. (all modes)--debug: Deprecated alias for--log-level DEBUG, kept for backward compatibility. Prefer--log-level. (all modes)--no-color: Disable ANSI colors in output. (all modes)--save-dir DIR: Directory to write results and generated deployment artifacts. (default,exp,generate,estimate)--top-n N: Number of top configurations to output — per experiment inexpmode, or per serving mode (agg/disagg) indefaultmode. Default:5. (default,exp,generate,estimate)--systems-paths: System search paths (comma-separated). Usedefaultfor the built-in systems path; the first match wins for an identical system/backend/version. (default,exp,generate,estimate)--deployment-target: Generated-artifact platform —dynamo-j2(default),dynamo-python,llm-d-helm,llm-d-kustomize,fpm, orslurm. See Deployment Target Selection. (default,exp,generate,estimate)--engine-step-backend: Engine-step latency backend. The compiled Rust engine is the only step executor;rustis the only accepted value (the deprecatedpythonno-op was removed after its one-release window); any other value raises an error. Accepted by the five modes below (notsupport) but inert ingenerate, which performs no latency estimation. (default,recommend,exp,generate,estimate)--forward-model: Forward-pass modeling mode —op_level(default; granular per-op modeling) orfpm(predicts from collected whole-model forward-pass data; requiresfpm_forward_perfdata for the exact model/system/backend/version and never extrapolates outside the collected domain). Evaluates on the compiled engine's native FPM operation.fpmpredictions are only as accurate as the match between the deployed engine configuration and the collected data — in particular the CUDA-graph capture surface: regime cliffs are encoded in the data, not modeled, so a deployment whose capture config differs from the collection will mispredict. V1 accepts only vLLM identities the standard deployment path can reproduce: automatic MoE/attention backend selection with EPLB disabled. Pinned backend or EPLB identities are rejected until structured generator support lands. (default,exp,generate,estimate)
Built-in FPM coverage currently includes:
| Model | System | Backend | Collected parallelism |
|---|---|---|---|
nvidia/GLM-5.2-NVFP4 |
b200_sxm |
vLLM 0.25.1 | DEP8, TEP8 |
MiniMaxAI/MiniMax-M2.7 |
h200_sxm |
vLLM 0.25.1 | DEP2/4, TEP2/4, TP2/4 |
FPM data is exact-version and finite-domain only. TP2 prefill is absent for MiniMax-M2.7 because the collected two-GPU deployment could not fit the model. Queries outside the collected cells or coordinate ranges fail explicitly.
The support mode accepts only --log-level, --debug, and --no-color from this list. Generator-artifact flags (--generator-config, --generator-set, --generator-help, --generator-help-backend, --generated-config-version, --generator-dynamo-version) are documented under Default mode.
When using default mode, several parameters have default values that affect
which configurations are considered feasible. These defaults are applied when
the corresponding flag is not specified:
| Parameter | Default | Flag | Effect |
|---|---|---|---|
| ISL (Input Sequence Length) | 4000 | --isl |
Assumed input prompt length |
| OSL (Output Sequence Length) | 1000 | --osl |
Assumed output generation length |
| TTFT (Time to First Token) | 2000 ms | --ttft |
Max acceptable time to first token |
| TPOT (Time per Output Token) | 30 ms | --tpot |
Max acceptable time per output token |
| Strict SLA | off | --strict-sla |
Pre-filter Pareto frontier to only SLA-compliant configs |
| Inclusive TPOT | off | --inclusive-tpot |
Report TPOT inclusive of TTFT |
| Backend | trtllm | --backend |
Inference backend used for estimation |
| Prefix Cache Length | 0 | --prefix |
Prefix cache length for KV reuse |
| Database Mode | SILICON | --database-mode |
Source of performance data |
| Free GPU Memory Fraction | 1.0 | --free-gpu-memory-fraction |
KV-cache memory budget used to filter batch sizes |
| Max Sequence Length | isl + osl | --max-seq-len |
KV blocks TRT-LLM pre-allocates per sequence |
| Chunked Prefill | off | --enable-chunked-prefill |
Finer-grained context-token sweep |
Important: The TTFT and TPOT defaults act as SLA filters — configurations that exceed these thresholds are excluded from results. If you see fewer results than expected, consider relaxing these values or setting them explicitly. When defaults are used, a warning is printed at the start of the run so you can verify what values are in effect. By default, only the top-N picking step filters on TPOT; pass
--strict-slato also pre-filter the Pareto frontier (see Strict SLA filtering).
This mode generates a working configuration without running the full parameter sweep. It's useful when you want a quick deployment config without SLA optimization.
aiconfigurator cli generate --model-path Qwen/Qwen3-32B-FP8 --total-gpus 8 --system h200_sxmThe generate mode calculates the smallest tensor parallel (TP) size that fits the model in memory using the formula: TP * VRAM_per_GPU > 1.5 * model_weight_size. This ensures the model fits with room for KV cache and activations.
Required arguments:
--model-path(alias--model): HuggingFace model path (e.g.,Qwen/Qwen3-32B-FP8) or local path containingconfig.json--total-gpus: Total GPUs for deployment--system: System name (h200_sxm,h100_sxm,h100_pcie,gb200,b200_sxm,a100_sxm,a100_pcie,l40s,l4,a30,gb300)
Optional arguments:
--backend: Backend name (trtllm,vllm,sglang). Default:trtllm--save-dir: Directory to save generated artifacts--systems-paths: Override system YAML/data search paths (comma-separated;defaultmaps to the built-in systems path). First match wins for identical system/backend/version.
Example output:
============================================================
Naive Configuration Generated Successfully
============================================================
Model: Qwen/Qwen3-32B-FP8
System: h200_sxm
Backend: trtllm (1.2.0rc5)
Total GPUs: 8 (using 8)
Parallelism: TP=1, PP=1
Replicas: 8 (each using 1 GPUs)
Max Batch Size: 128
Output: ./output/Qwen_Qwen3-32B-FP8_naive_tp1_pp1_123456
============================================================
Python API equivalent:
from aisim.legacy_cli import cli_generate
result = cli_generate(
model_path="Qwen/Qwen3-32B-FP8",
total_gpus=8,
system="h200_sxm",
backend="trtllm",
output_dir="./output",
)
print(result["parallelism"]) # {'tp': 1, 'pp': 1, 'gpus_per_worker': 1, 'replicas': 8, 'gpus_used': 8}Note: This is a naive configuration without memory validation or performance optimization. For production deployments, use
aiconfigurator cli defaultto run the full parameter sweep with SLA optimization.
This mode runs a single-point performance estimation to predict TTFT (time to first token), TPOT (time per output token), and power consumption for a given model, system, and configuration. Unlike default mode, no parameter sweep or SLA optimization is performed — you specify the exact configuration and get back the predicted metrics.
aiconfigurator cli estimate --model-path Qwen/Qwen3-32B --system h200_sxm --tp-size 2 --batch-size 64 --isl 2048 --osl 512Required arguments:
--model-path(alias--model): HuggingFace model path (e.g.,Qwen/Qwen3-32B) or local path containingconfig.json--system: System name (h200_sxm,h100_sxm,h100_pcie,b200_sxm,gb200,a100_sxm,a100_pcie,l40s,l4,a30,gb300)
Optional arguments (shared):
--estimate-mode:agg(default, IFB) ordisagg(separate prefill/decode workers), or one of the single-pass static breakdown modesstatic/static_ctx/static_gen--backend: Backend name (trtllm,vllm,sglang). Default:trtllm--backend-version: Backend database version — a queryable slot version or the aliasescurrent/previous/next(seesystems/query_versions.yaml). Default:current.--attention-backend: Attention kernel backend — one offa3,triton,trtllm_mha,flashinfer,fla, ordefault. It applies to every model graph with standard denseContextAttention/GenerationAttentionoperations and to supported DeepSeek MLA/WideEP paths. Support is backend-, performance-table-, and version-specific; unsupported named values fail closed. For modeling, unset ordefaultuses the mapped framework default when available and otherwise the safedefaultfallback. SGLang WideEP maps unset/defaulttoflashinferand also supportsfa3. The deployment generator emits supported named SGLang values, omits unset/default, and rejectsflafor SGLang 0.5.14.--database-mode: Database mode (SILICON,HYBRID,EMPIRICAL,SOL). Default:SILICON--isl: Input sequence length. Default:1024--osl: Output sequence length. Default:1024--batch-size(alias--bs): Batch size (max concurrent requests, used for agg/static). Default:128--ctx-tokens: Context tokens budget for IFB scheduling (agg only). Default: same as ISL--tp-size(alias--tp): Tensor parallelism size. Default:1--pp-size(alias--pp): Pipeline parallelism size. Default:1--attention-dp-size(alias--dp): Attention data parallelism size. Default:1--moe-tp-size(alias--etp): MoE tensor parallelism size (auto-inferred if omitted)--moe-ep-size(alias--ep): MoE expert parallelism size (auto-inferred if omitted)--gemm-quant-mode: GEMM quantization mode (auto-inferred from model config if omitted)--kvcache-quant-mode: KV cache quantization mode (auto-inferred if omitted)--fmha-quant-mode: FMHA quantization mode (auto-inferred if omitted). DeepSeek-V3 / Kimi-K2.5 context attention (MLA prefill) has no fp8 FMHA data — auto-inferred fp8 is downgraded tobfloat16for context-touching estimates (agg, prefill, static, static_ctx), while decode/generation keeps fp8. Explicitly passingfp8for a context estimate is rejected with a clear error.--moe-quant-mode: MoE quantization mode (auto-inferred if omitted)--comm-quant-mode: Communication quantization mode (auto-inferred; defaulthalf)--prefix: Prefix cache length (subset of ISL already cached per request). Default:0--nextn: MTP draft length, orautoto use the checkpoint'snum_nextn_predict_layers(absent/0 keeps MTP disabled). Default:0; MTP is never enabled implicitly — pass--nextnexplicitly to model it--nextn-accepted: Average accepted draft tokens per decode step (0 <= nextn_accepted <= nextn). Required when the draft depth is > 0 (including via--nextn auto); use a measured value from your deployment--stride: (static modes only) OSL-sweep stride used byrun_static; ignored foragg/disagg. Default:32--free-gpu-memory-fraction: Fraction of free GPU memory for KV cache. Default:0.9. Used to estimate max concurrent sequences and warn when batch size exceeds KV cache capacity--max-seq-len: TRT-LLM--max_seq_len(default:isl + osl). Controls KV blocks pre-allocated per sequence; set to match your deployment for an accurate KV-capacity warning--detail: Comma-separated breakdown sections to print after the summary box. Choices:summary,memory,time,energy,source,all. Example:--detail memory,time. Default: none (useallto print every section)
Shared flags such as
--save-dir,--top-n, and--systems-pathsare listed in Common Arguments.
Disagg-specific arguments (used when --estimate-mode disagg):
--decode-system: System name for decode workers. Defaults to--system--prefill-tp-size,--prefill-pp-size,--prefill-attention-dp-size: Prefill parallelism overrides (default to shared args)--prefill-moe-tp-size,--prefill-moe-ep-size: Prefill MoE parallelism overrides--prefill-batch-size: Prefill batch size (required for disagg)--prefill-num-workers: Number of prefill workers (required for disagg)--decode-tp-size,--decode-pp-size,--decode-attention-dp-size: Decode parallelism overrides (default to shared args)--decode-moe-tp-size,--decode-moe-ep-size: Decode MoE parallelism overrides--decode-batch-size: Decode batch size (required for disagg)--decode-num-workers: Number of decode workers (required for disagg)
Example output (agg):
============================================================
Performance Estimate (agg)
============================================================
Model: Qwen/Qwen3-32B
System: h200_sxm
Backend: trtllm (1.2.0rc5)
------------------------------------------------------------
ISL: 2048
OSL: 512
Batch Size: 64
Context Tokens: 2048
TP Size: 2
PP Size: 1
------------------------------------------------------------
TTFT: 487.990 ms
TPOT: 29.118 ms
Request Latency: 15367.492 ms
Power (per GPU): 0.0 W
------------------------------------------------------------
tokens/s: 2,153.38
tokens/s/gpu: 1,076.69
tokens/s/user: 34.34
seq/s: 4.214
Concurrency: 64
Memory (GPU): 54.55 GB
============================================================
Disagg example:
aiconfigurator cli estimate \
--model-path Qwen/Qwen3-32B --system h200_sxm \
--estimate-mode disagg --isl 2048 --osl 512 --tp-size 2 \
--prefill-batch-size 4 --prefill-num-workers 2 \
--decode-batch-size 64 --decode-num-workers 2Python API equivalent:
from aisim.legacy_cli.api import cli_estimate
# Aggregated estimation
result = cli_estimate(
"Qwen/Qwen3-32B", "h100_sxm",
batch_size=64, isl=2048, osl=512, tp_size=2,
)
print(f"TTFT: {result.ttft:.2f} ms, TPOT: {result.tpot:.2f} ms")
power = "unavailable" if result.power_w is None else f"{result.power_w:.1f} W"
print(f"Power: {power}")
print(f"Throughput: {result.tokens_per_second_per_gpu:,.2f} tokens/s/gpu")
# Disaggregated estimation
result = cli_estimate(
"Qwen/Qwen3-32B", "h100_sxm", mode="disagg",
isl=2048, osl=512, tp_size=2,
prefill_batch_size=4, prefill_num_workers=2,
decode_batch_size=64, decode_num_workers=2,
)For vision-language models with image inputs, --enable-epd overlays a fixed encode-worker pool on an agg or disagg estimate (other estimate modes reject it):
--encoder-tp: Encode-worker TP (required with--enable-epd).--encoder-batch-size: Encode batch size. Default:1(maximum: 8).--encoder-num-workers: Encode workers in the pool. Default:1.
agg and disagg model continuous (in-flight) batching. The three static modes instead run a single forward pass through the model — no IFB scheduling — and report the per-phase latency and memory layout for that one pass. They are the quickest way to see where time and memory go for a given shape and parallelism.
static— one full pass over both phases; reports TTFT (context), TPOT (one decode step), and request latency.static_ctx— context (prefill) phase only; reports TTFT.static_gen— generation (decode) phase only; reports TPOT.
--stride N (static modes only, default 32) sets the stride run_static uses to accelerate the OSL sweep; it is ignored for agg/disagg.
aiconfigurator cli estimate \
--model-path Qwen/Qwen3-32B --system h200_sxm \
--estimate-mode static --isl 4096 --osl 1024 --tp-size 4 --batch-size 32A static estimate is a single-pass breakdown, not a served-throughput number — use
agg/disaggfor SLA-driven throughput. If the configuration does not fit in memory, the static report still renders but prints an OOM warning.
--detail prints additional breakdown sections after the summary box, for any estimate mode (agg, disagg, or the static modes). Pass a comma-separated list of sections, or all:
| Section | Shows |
|---|---|
summary |
High-level latency / throughput recap. |
memory |
Per-component memory (weights, kvcache, activations, nccl, others) as a share of GPU capacity, the KV footprint per sequence, and a KV-bound max-batch upper bound. |
time |
Per-op latency bars in context → generation order, each op's share of the phase, and (in static modes) a Speed-of-Light (SOL) comparison plus the per-op data source. |
energy |
Per-op energy breakdown, when energy data is available for the system. |
source |
Per-op data-source attribution — including silicon (measured), empirical (interpolated / formula), estimated (modeled), and mixed — so you can distinguish measured and modeled values. |
all |
Every section above. |
--detail replaces the removed --print-per-ops-latency (the old flag still works as a deprecation alias).
For SGLang DeepEP communication (deepep_ht or deepep_ll), AIConfigurator first selects a curve collected at the exact requested EP size and node count when that topology is available. DeepEP-HT returns that curve as silicon data. DeepEP-LL uses it as a calibration anchor and applies Monte Carlo routing skew, without applying an NVLink/IB specification floor, so the resulting LL value is still estimated. If a multi-node request has no exact all-to-all row but the same MoE shape has a compatible node-1 row, AIConfigurator may use that row as a donor; LL then applies both routing skew and topology bandwidth guardrails. Legacy HT node-1 rows represent EP8, while legacy LL node-1 rows use the system's physical node width (EP4 on GB200/GB300 and EP8 on HGX). A compatible MoE expert-compute row is still required at the requested EP size.
A successfully executed node-1 substitution keeps the generic per-op source as estimated and records separate requested-versus-measurement coordinates. The CLI emits a warning by default, for example context/deepep_ht: requested EP32/node8; using EP8/node1 silicon data. Add --detail source to show the same executed provenance alongside the per-op source table. Exact hits and failed substitute lookups do not produce a fallback record; an exact DeepEP-LL hit is nevertheless estimated because Monte Carlo scaling still runs.
The Python cli_estimate API exposes the ordered, de-duplicated records through EstimateResult.moe_comm_fallbacks. Each MoECommFallback identifies the inference phase, communication backend, requested EP size and node count, and measurement EP size and node count.
Task.run_single_agg, Task.run_single_disagg, and aggregate/disaggregate sweep result rows retain the records in the hidden object column _moe_comm_fallbacks, including through rate matching, Pareto selection, and visible-column deduplication. Saved best_config_topn.csv and pareto.csv files omit this object column; for every selected configuration that used a substitution, the corresponding topN/moe_comm_fallbacks.json sidecar contains the records and the CLI logs a warning naming that file.
aiconfigurator cli estimate \
--model-path Qwen/Qwen3-32B --system h200_sxm \
--estimate-mode static --isl 4096 --osl 1024 --tp-size 4 --batch-size 32 \
--detail allAbbreviated output (static modes add the SOL columns and per-op source tags shown below; agg/disagg omit the SOL comparison):
Memory Layout (capacity 141.00 GiB)
weights 15.256 GiB ████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 10.8%
kvcache 10.000 GiB ███░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 7.1%
activations 10.000 GiB ███░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 7.1%
nccl 0.383 GiB ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 0.3%
others 3.500 GiB █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 2.5%
----------------------------------------------------------------------
total 39.139 GiB ███████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 27.8% (free 101.861 GiB)
kvcache/seq 0.3125 GiB (seq_len=5120)
max batch (KV-bound, same isl/osl) ≈ 357
note: ignores activation growth with batch; treat as an upper bound.
Latency Summary
metric latency SOL SOL%
ttft 3900.357 ms 2997.913 ms 76.9%
tpot 10.970 ms 5.572 ms 50.8%
request latency 15122.575 ms 8697.917 ms 57.5%
Context phase (total = 3900.357 ms, SOL = 2997.913 ms, SOL% = 76.9%)
op latency SOL SOL% share (%) source
...
context_qkv_gemm 279.049 ms 222.348 ms 79.7% █░░░░░░░░░░░ 7.2% [silicon]
context_attention 334.500 ms 142.303 ms 42.5% █░░░░░░░░░░░ 8.6% [mixed]
context_gate_ffn1_gemm 1415.684 ms 1111.741 ms 78.5% ████░░░░░░░░ 36.3% [silicon]
context_ffn2_gemm 667.966 ms 555.870 ms 83.2% ██░░░░░░░░░░ 17.1% [silicon]
...
Generation phase (total = 11222.218 ms, SOL = 5700.004 ms, SOL% = 50.8%)
op latency SOL SOL% share (%) source
...
generation_attention 4470.776 ms 2055.779 ms 46.0% █████░░░░░░░ 39.8% [silicon]
generation_gate_ffn1_gemm 2509.475 ms 1803.466 ms 71.9% ███░░░░░░░░░ 22.4% [silicon]
generation_ffn2_gemm 1345.669 ms 903.968 ms 67.2% █░░░░░░░░░░░ 12.0% [silicon]
...
Data Source Breakdown (per-op)
context silicon=8, empirical=5, mixed=1
generation silicon=9, empirical=5
This is an optional pre-flight check to verify whether collected SILICON data supports a
specific model and hardware combination for both aggregated and disaggregated serving
modes. You can skip this and run cli default directly. PASS rows count as support;
HYBRID_PASS rows are reported separately as empirical estimability and do not make the
default SILICON support check pass. For unlisted models, support is determined by a
majority vote of SILICON results for models sharing the same architecture.
aiconfigurator cli support --model-path Qwen/Qwen3-32B-FP8 --system h200_sxmRequired arguments:
--model-path(alias--model): HuggingFace model path (e.g.,Qwen/Qwen3-32B-FP8) or local path containingconfig.json--system: System name (h200_sxm,gb200,b200_sxm,h100_sxm,h100_pcie,a100_sxm,a100_pcie,l40s,l4,a30,gb300)
Optional arguments:
--backend: Filter by specific backend (trtllm,vllm,sglang). Defaults totrtllm.--backend-version: Filter by a specific backend version (slot versions /current/previous/nextaliases). For a single system/backend, defaults to its current slot, even when that version has no support-matrix rows. Systems without slot policy retain the latest model-relevant support-matrix version as the compatibility default. This is a certification lookup; successful prediction alone does not establish SILICON support.--systems-paths: Override system YAML/data search paths (comma-separated;defaultmaps to the built-in systems path). First match wins for identical system/backend/version.
Example output:
============================================================
AIC Support Check Results
============================================================
Model: Qwen/Qwen3-32B-FP8
System: h200_sxm
Backend: trtllm
Version: 0.18.0
------------------------------------------------------------
Aggregated Support: YES
Disaggregated Support: YES
============================================================
Python API equivalent:
from aisim.legacy_cli import cli_support
agg_supported, disagg_supported = cli_support(
model_path="Qwen/Qwen3-32B-FP8",
system="h200_sxm",
backend="trtllm"
)
print(f"Agg: {agg_supported}, Disagg: {disagg_supported}")This mode finds the minimum number of GPUs needed to meet a performance target. It is designed as a procurement sizing tool — the output is unconstrained, suitable for driving purchasing decisions.
Instead of specifying a GPU count (like default mode), you specify exactly one load target (request rate or concurrency) along with SLA constraints, and the system calculates the minimum GPUs required.
The recommender searches both tensor-parallel and pipeline-parallel configurations to find the most efficient layout. For models too large to fit on a single node, it automatically escalates to multi-node configurations.
aiconfigurator cli recommend --model-path Qwen/Qwen3-32B --system h200_sxm --backend trtllm \
--target-request-rate 50 --ttft 2000 --tpot 30 --isl 4000 --osl 1000or with a concurrency target:
aiconfigurator cli recommend --model-path Qwen/Qwen3-32B --system h200_sxm --backend sglang \
--target-concurrency 200 --ttft 2000 --tpot 30Required arguments:
--model-path(alias--model): HuggingFace model path or local path containingconfig.json--system: System name (GPU type)- Exactly one of the following (mutually exclusive):
--target-request-rate: Target system request rate in req/s--target-concurrency: Target number of concurrent users
Optional arguments:
--backend: Backend name (trtllm,vllm,sglang,auto). Default:trtllm--ttft,--tpot: SLA targets in ms (default: 2000ms, 30ms)--request-latency: End-to-end request latency target in ms--isl,--osl: Input/output sequence lengths (default: 4000, 1000)--nextn: MTP draft length, orautoto use the checkpoint'snum_nextn_predict_layersand then DSPARK architecture metadata. Omitted or0keeps speculation disabled unless a DSPARK model is paired with an explicit--nextn-acceptedmeasurement.--nextn-accepted: Required when the resolved draft depth is greater than 0; it must be a measured average in the range0 <= nextn_accepted <= nextn. AIC never infers this workload-dependent value.- All other arguments match
defaultmode (quantization, prefix caching, etc.)
The output includes total_gpus_needed and replicas_needed columns, showing both agg and disagg configurations ranked by fewest GPUs first.
Python API equivalent:
from aisim.legacy_cli import cli_recommend
result = cli_recommend(
model_path="Qwen/Qwen3-32B",
system="h200_sxm",
target_request_rate=50.0,
ttft=2000,
tpot=30,
)
for mode, df in result.best_configs.items():
parallelism = ["tp"] if mode == "agg" else ["(p)tp", "(d)tp"]
columns = ["total_gpus_needed", "replicas_needed", *parallelism, "tpot"]
print(f"{mode}: {df[columns].head()}")This mode is triggered by
aiconfigurator cli default --model-path Qwen/Qwen3-32B-FP8 --total-gpus 32 --system h200_sxm
or
aiconfigurator cli default --model-path Qwen/Qwen3-32B-FP8 --total-gpus 32 --system h200_sxm --ttft 1000 --tpot 10 --isl 3000 --osl 512 --prefix 0model_path, total_gpus, system are three required arguments to define the problem.
If you want to specify your problem with more details, we allow to define ttft, tpot, isl, osl and prefix.
Beyond --ttft, --tpot, --isl, --osl, and --prefix, default mode accepts:
--decode-system: System (GPU type) for disagg decode workers. Defaults to--system. Use it for heterogeneous prefill/decode (e.g. B200 prefill + H200 decode).--backend-version: Backend database version — a queryable slot version or the aliasescurrent/previous/next. Default:current.--free-gpu-memory-fraction: Fraction of free GPU memory TRT-LLM allocates for KV cache (default:1.0). Filters batch sizes that would exceed KV cache capacity.--max-seq-len: TRT-LLM--max_seq_len(default:isl + osl). Controls how many KV blocks are pre-allocated per sequence; set to match your deployment for accurate KV-capacity filtering.--enable-chunked-prefill: Enable chunked prefill for a finer-grained context-token sweep. When off (default), the context-token stride is aligned to ISL for faster sweeping.--enable-wideep: Deprecated and ignored for large-EP modeling (accepted with a one-time warning). On SGLang, it still narrows the defaultmoe_tpcandidates to[1]; explicit*_moe_tp_candidatesvalues take precedence. Large-EP (wideEP) is explored automatically — see the note below.--attention-backend: Attention kernel backend — one offa3,triton,trtllm_mha,flashinfer,fla, ordefault. It applies to every model graph with standard denseContextAttention/GenerationAttentionoperations and to supported DeepSeek MLA/WideEP paths. Support is backend-, performance-table-, and version-specific; unsupported named values fail closed. For modeling, unset ordefaultuses the mapped framework default when available and otherwise the safedefaultfallback. SGLang WideEP maps unset/defaulttoflashinferand also supportsfa3. The deployment generator emits supported named SGLang values, omits unset/default, and rejectsflafor SGLang 0.5.14.--moe-backend: Explicit SGLang MoE backend.megamoeis a real kernel selection (use it to model DeepSeek-V4 MegaMoE on Blackwell);deepep_moeis deprecated: it is ignored for modeling (large-EP is explored automatically from data coverage), but on SGLang it still narrows the defaultmoe_tpcandidates to[1]— explicit*_moe_tp_candidatesalways win.
Large-EP (wideEP) is explored automatically. For MoE models, multi-node EP-only parallelism joins the search whenever the performance database covers the model's MoE shape on the target system/backend — both the MoE all-to-all (dispatch/combine) data and the EP compute data must be present. No flag is needed: fused and large-EP configurations compete in the same search. To restrict or force EP sizes, set
*_moe_ep_candidatesin an exp YAML (Exp mode); when a model's shape has no coverage, a one-time INFO log names the collectors to run (see Advanced Tuning).
Vision-language inputs (multimodal models such as Qwen3-VL and Qwen3.5):
--image-height,--image-width: Image dimensions in pixels. Default:0(disabled — the request is modeled as text-only).--num-images: Number of images per request. Default:1.--video-height,--video-width: Video frame dimensions in pixels. Default:0(disabled).--video-frames: Number of sampled/preprocessed frames per video. Default:0(disabled).--num-videos: Number of videos per request. Default:0.--num-video-tokens: Explicit post-merge tokens per video; requires--video-frames. Default:0(derive from dimensions).--disable-encoder-dp: Model the vision encoder as TP-sharded instead of the default data-parallel. Available indefaultandestimatemodes.
The image and video input flags are also available in recommend mode.
Image and video workloads must currently be estimated separately. Qwen3-VL dense/MoE and Qwen3.5 video token accounting uses each model's temporal patch size before the spatial merge.
Kimi K2.5 resizes to its processor's patch and side limits, then pads to the patch/merge stride. The bundled checkpoints use 16,384 image patches, 4,096 patches per video frame, and 512 patches per side before padding. Local and downloaded checkpoints may override these limits through preprocessor_config.json (legacy media_proc_cfg or native Transformers settings) and video_preprocessor_config.json. Native image and video side limits must match, with each independently defaulting to 512 patches per side when omitted; legacy media_proc_cfg uses a shared side limit. Video estimates support one processor chunk of up to four sampled frames per video, including when --num-video-tokens is supplied; longer videos are rejected until temporal chunk modeling is available. Kimi's projector remains replicated under encoder tensor parallelism.
Video estimates are supported for functional estimation, but saving deployment artifacts for a video workload is rejected until the generator has a video benchmark schema; run without --save-dir.
Encoder disaggregation (EPD) — serve the vision encoder from a dedicated encode-worker pool rate-matched against the LM workers (agg becomes E+agg, disagg becomes E+P+D). Requires image inputs; the LM workers are modeled language-only. Result rows carry (e)workers/(e)tp/(e)bs columns.
--enable-epd: Sweep dedicated encode workers alongside the LM sweep.--encoder-tp: Encode-worker TP sizes to sweep. Default:1 2 4 8.--encoder-system: System (GPU type) for the encode pool. Defaults to the LM-side system; backend and version always follow the P/agg side.--encoder-latency-correction: Latency correction scale for encode workers. Default:1.0.
Encode batch sizes are swept over 1 2 4 8, capped at 8 (SGLang's SGLANG_ENCODER_MAX_BATCH_SIZE default); explicit candidates above the cap are rejected. Further knobs (encoder_batch_candidates, max_encoder_workers, rate_match_encoder_degradation) are Task fields for exp-mode YAML — see Advanced Tuning. EPD rows are excluded from generator artifacts (see Generator Overview).
The SLA, precision, and speculative-decoding flags (--strict-sla, --request-latency, --inclusive-tpot, --nextn, --nextn-accepted, --database-mode) have dedicated subsections below. Shared flags such as --save-dir, --top-n, and --systems-paths are described in Common Arguments.
You can specify which inference backend to use with the --backend flag:
# Use TensorRT-LLM (default)
aiconfigurator cli default --model-path Qwen/Qwen3-32B-FP8 --total-gpus 32 --system h200_sxm --backend trtllm
# Use vLLM (dense models only, currently being evaluated)
aiconfigurator cli default --model-path Qwen/Qwen3-32B-FP8 --total-gpus 32 --system h200_sxm --backend vllm
# Use SGLang (dense and MoE models, currently being evaluated)
aiconfigurator cli default --model-path Qwen/Qwen3-32B-FP8 --total-gpus 32 --system h200_sxm --backend sglangUse --backend auto to sweep across all supported backends and compare results side by side.
Both agg and disagg results are merged across backends and the globally optimal configuration
is selected. This is useful for finding the best backend without running separate commands.
The command will create two experiments for the given problem, one is agg and another one is disagg. Compare them to find the better one and estimates the perf gain.
The experimental Spica smart sweeper has moved to Dynamo's standalone
AISim distribution.
The AIC --thorough-sweep and --thorough-config flags have been removed. Install
it from a matching Dynamo checkout with python -m pip install ./aisim, then run Spica
through python -m aisim.spica. Runnable configurations and tools live under
examples/aisim/spica.
You can override where system YAMLs and performance data are loaded from using --systems-paths.
aiconfigurator cli default \
--model-path Qwen/Qwen3-32B \
--total-gpus 32 \
--system h200_sxm \
--systems-paths "default,/opt/aic/systems,/data/aic/systems"- Paths are searched in order.
- Use
defaultto include the built-in systems path. - If the same system/backend/version exists in multiple paths, the first match is used.
The command will print out the result to your terminal with the basic info of the comparison, the pareto curve (the best point is tagged as x),
the worker setup for your reference. Let's split them into sections.
Let's run aiconfigurator cli default --model-path Qwen/Qwen3-32B-FP8 --total-gpus 32 --system h200_sxm --ttft 1000 --tpot 10 --isl 3000 --osl 512 --prefix 0
Note that the result might differ based on different versions of your aiconfigurator.
- Basic info of the comparison
Input Configuration & SLA Target:
Model: Qwen/Qwen3-32B-FP8 (is_moe: False)
Total GPUs: 32
Best Experiment Chosen: disagg at 913.82 tokens/s/gpu (1.43x better)
----------------------------------------------------------------------------
Overall Best Configuration:
- Best Throughput: 29,242.24 tokens/s
- Per-GPU Throughput: 913.82 tokens/s/gpu
- Per-User Throughput: 123.92 tokens/s/user
- TTFT: 202.65ms
- TPOT: 8.07ms
This shows that for model Qwen/Qwen3-32B-FP8 to deploy on 32 H200, if you require your TTFT to be less than 1000ms and TPOT to be less than 10ms, and your problem is isl=3000 osl=512, then disagg will be 1.43x of agg. The target result is shown as Overall Best Configuration.
Python API equivalent:
from aisim.legacy_cli import cli_default
result = cli_default(
model_path="Qwen/Qwen3-32B-FP8",
total_gpus=32,
system="h200_sxm",
ttft=1000,
tpot=10,
isl=3000,
osl=512
)
# Access the DataFrames
print(result.best_configs["disagg"])- Pareto frontier
Pareto Frontier:
Qwen/Qwen3-32B-FP8 Pareto Frontier: tokens/s/gpu vs tokens/s/user
┌──────────────────────────────────────────────────────────────────────────┐
2250┤ •• disagg │
│ ff agg │
│ xx disagg best │
│ │
1875┤ ff │
│ fff │
│ ff │
│ fff•• │
1500┤ f ••• │
│ ff •••••••• │
│ ffff • │
│ f ••••••• │
1125┤ ff • │
│ ff •••• │
│ ffff ••••x │
│ fff •••• │
750┤ fff • │
│ ffffff • │
│ ffffff •• │
│ fffffff •••••• │
375┤ ff • │
│ fffffff••••••••• │
│ ffffffffff │
│ │
0┤ │
└┬─────────────────┬──────────────────┬─────────────────┬─────────────────┬┘
0 60 120 180 240
tokens/s/gpu tokens/s/user
Pareto frontier shows the trade-off betwen generation speed tokens/s/user and throughput tokens/s/gpu. The best points is tagged as x. As you want the TPOT to be less than 10ms, which means the generation speed is faster than 1000/10ms = 100 tokens/s/user, then by reading the pareto froniter, you will get the point tagged as x. You can see that, if you want different TPOT, you will have different result. Sometimes, agg will be better than disagg (higher throughput at same tokens/s/user)
- Worker setup
Deployment Details:
(p) stands for prefill, (d) stands for decode, bs stands for batch size, a replica stands for the smallest scalable unit xPyD of the disagg system
Some math: total gpus used = replicas * gpus/replica
gpus/replica = (p)gpus/worker * (p)workers + (d)gpus/worker * (d)workers; for Agg, gpus/replica = gpus/worker
gpus/worker = tp * pp * dp = etp * ep * pp for MoE models; tp * pp for dense models (underlined numbers are the actual values in math)
disagg Top Configurations: (Sorted by tokens/s/gpu)
+------+--------------+---------------+--------+-------------+------------------+----------+----------------+------------+----------------+-------------+-------+------------+----------------+-------------+-------+
| Rank | tokens/s/gpu | tokens/s/user | TTFT | concurrency | total_gpus(used) | replicas | gpus/replica | (p)workers | (p)gpus/worker | (p)parallel | (p)bs | (d)workers | (d)gpus/worker | (d)parallel | (d)bs |
+------+--------------+---------------+--------+-------------+------------------+----------+----------------+------------+----------------+-------------+-------+------------+----------------+-------------+-------+
| 1 | 913.82 | 123.92 | 202.65 | 256(=64x4) | 32 (32=4x8) | 4 | 8 (=4x1+1x4) | 4 | 1 (=1x1) | tp1pp1 | 1 | 1 | 4 (=4x1) | tp4pp1 | 64 |
| 2 | 873.07 | 126.28 | 202.65 | 240(=60x4) | 32 (32=4x8) | 4 | 8 (=4x1+1x4) | 4 | 1 (=1x1) | tp1pp1 | 1 | 1 | 4 (=4x1) | tp4pp1 | 60 |
| 3 | 852.77 | 133.94 | 202.65 | 240(=240x1) | 32 (32=1x32) | 1 | 32 (=12x1+5x4) | 12 | 1 (=1x1) | tp1pp1 | 1 | 5 | 4 (=4x1) | tp4pp1 | 48 |
| 4 | 568.51 | 148.82 | 202.65 | 144(=72x2) | 32 (32=2x16) | 2 | 16 (=4x1+3x4) | 4 | 1 (=1x1) | tp1pp1 | 1 | 3 | 4 (=4x1) | tp4pp1 | 24 |
| 5 | 434.77 | 145.12 | 123.20 | 104(=104x1) | 32 (24=1x24) | 1 | 24 (=4x2+4x4) | 4 | 2 (=2x1) | tp2pp1 | 1 | 4 | 4 (=4x1) | tp4pp1 | 26 |
+------+--------------+---------------+--------+-------------+------------------+----------+----------------+------------+----------------+-------------+-------+------------+----------------+-------------+-------+
agg Top Configurations: (Sorted by tokens/s/gpu)
+------+--------------+---------------+--------+-------------+------------------+----------+--------------+-------------+----------+----+
| Rank | tokens/s/gpu | tokens/s/user | TTFT | concurrency | total_gpus(used) | replicas | gpus/replica | gpus/worker | parallel | bs |
+------+--------------+---------------+--------+-------------+------------------+----------+--------------+-------------+----------+----+
| 1 | 638.28 | 100.97 | 187.72 | 224(=28x8) | 32 (32=8x4) | 8 | 4 | 4 (=4x1) | tp4pp1 | 28 |
| 2 | 612.50 | 101.49 | 274.98 | 224(=14x16) | 32 (32=16x2) | 16 | 2 | 2 (=2x1) | tp2pp1 | 14 |
| 3 | 594.71 | 108.14 | 149.46 | 192(=24x8) | 32 (32=8x4) | 8 | 4 | 4 (=4x1) | tp4pp1 | 24 |
| 4 | 592.60 | 111.22 | 199.08 | 192(=24x8) | 32 (32=8x4) | 8 | 4 | 4 (=4x1) | tp4pp1 | 24 |
| 5 | 544.17 | 119.83 | 149.25 | 160(=20x8) | 32 (32=8x4) | 8 | 4 | 4 (=4x1) | tp4pp1 | 20 |
+------+--------------+---------------+--------+-------------+------------------+----------+--------------+-------------+----------+----+
If you want to reproduce the result we esimated, you need to follow the suggestions here. Take the disagg top1 result as an example.
We're expecting to achieve 913.82 tokens/s/gpu and 123.92 tokens/s/user with this config.
We have 1 definition replica, it means the number of copies of your xPyD disagg system. Say, here, we have 4 replicas, each replica contains 8 GPUs.
Each replica has a system of 4 prefill workers and 1 decode workers. Each prefill worker is using tp1pp1 which is 1 GPU per worker; while each decoder worker is using tp4pp1 which is 4 GPU per workers. These workers compose a 4P1D replica with 8 GPUs. As you want to deploy on 32 GPUs, then you will have 4 replicas.
bs is required to be set in framework as it limits the largest batch_size of the worker which is crucial to control the TPOT of the deployment.
concurrency = concurrency * replicas Use it to benchmark your deployment on total GPUs. If you only want to benchmark 1 replica, divide it by replicas
As this is still a little bit challenging to get the right configs for your deployment, we can further specify --save-dir DIR to output all the results here as well as generate the configs for frameworks automatically. Here is the output folder structure:
results/Qwen_Qwen3-32B-FP8_h200_sxm_trtllm_isl4000_osl1000_ttft1000_tpot20_904495
├── agg
│ ├── best_config_topn.csv
│ ├── config.yaml
│ ├── pareto.csv
│ ├── top1
│ │ ├── agg_config.yaml
│ │ ├── bench_run.sh # aiperf benchmark sweep script (bare-metal)
│ │ ├── k8s_bench.yaml # aiperf benchmark sweep Job (Kubernetes)
│ │ ├── k8s_deploy.yaml
│ │ ├── run_0.sh
│ │ └── generator_config.yaml
│ ...
├── disagg
│ ├── best_config_topn.csv
│ ├── config.yaml
│ ├── pareto.csv
│ ├── top1
│ │ ├── bench_run.sh # aiperf benchmark sweep script (bare-metal)
│ │ ├── decode_config.yaml
│ │ ├── k8s_bench.yaml # aiperf benchmark sweep Job (Kubernetes)
│ │ ├── k8s_deploy.yaml
│ │ ├── run_0.sh
│ │ ├── prefill_config.yaml
│ │ └── generator_config.yaml
│ ...
└── pareto_frontier.png
By default, we output the top 5 configs we have found. You can get the configs and scripts to deploy under each experiment's folder. The generated files depend on your --deployment-target:
- Dynamo (default):
k8s_deploy.yamlfor Kubernetes deployment, plus engine configs (agg_config.yaml,prefill_config.yaml,decode_config.yaml) and run scripts (run_0.sh) - llm-d:
llm-d-values.yamlfor Helm deployment with the llm-d-modelservice chart - FPM V1: exactly
k8s_deploy.yaml(a reusable keepalive Pod, LeaderWorkerSet, or Grove PodCliqueSet),fpm_env.sh(rank discovery plus the per-cell collection facts), andrun.sh(the launch-only vLLM command) - Slurm:
deploy.sbatch,benchmark.sbatch,submit.sh,environment.sh,deployment.json,slurm_runtime.py, andbench_run.sh, plus TRT-LLM engine configs when applicable. See Generate a Slurm deployment.
For benchmarking, see the Benchmark Artifacts section below. Refer to deployment guide for Dynamo deployments or the generator usage guide for llm-d deployments.
--save-dir DIR allows you to specify more information such as generating the config for a different version of the backend, say estimating the performance using trtllm 1.0.0rc3 but generate config for 1.0.0rc6. This is allowed and feasible. By passing --generated-config-version 1.0.0rc6 can give you the right result.
Use --deployment-target to choose which orchestration platform to deploy to:
dynamo-j2(default): Generates typed Dynamo Kubernetes manifestsdynamo-python: Generates Dynamo Kubernetes manifests using Dynamo's Python config modifiers (requiresdynamopackage)llm-d-helm: Generates Helm values for the llm-d-modelservice chartllm-d-kustomize: Generates Kustomize overlays for llm-d modelserver guidesfpm: Generates a reusable Kubernetes resource workload and a complete FPM launch script,run.shslurm: Generates single-node Dynamo service and AIPerf benchmark jobs for Slurm/Pyxis. See Generate a Slurm deployment.
The backend (--backend trtllm/vllm/sglang) and deployment target are generally orthogonal choices. Note that TRT-LLM only supports Dynamo platforms. FPM V1 is the exception: it supports only a vLLM single aggregated-worker topology with exactly one worker replica; that worker may span multiple nodes. Router/planner configurations and invalid FPM topologies fail closed.
Use --deployment-target fpm when a collector or agent should reserve Kubernetes resources and execute generated FPM launches in them. A generator config can add tokenized vLLM arguments, concrete environment variables, and resource-workload overlays:
WorkerConfig:
agg_workers: 1
Workers:
agg:
extra_cli_args:
- --scheduler-cls
- InstrumentedScheduler
- --benchmark-mode
- prefill
- --dump-config-to
- "/results/resolved-config-node{node_rank}.json"
K8sConfig:
# Optional; the multinode default is lws.
fpm_orchestrator: grove
fpm_shared_memory_size: 200Gi
fpm_resource_labels:
fpm.nvidia.com/run-id: glm52-example
fpm.nvidia.com/stage: probe
# Optional; set only when the target cluster uses KAI Scheduler.
kai.scheduler/queue: dynamo
worker_extra_pod_spec:
# Optional; Grove does not select a scheduler by default.
schedulerName: kai-scheduler
mainContainer:
resources:
requests:
memory: 448Gi
ephemeral-storage: 30Gi
extra_env:
- name: FPM_RUN_ID
value: glm52-example
- name: DYN_FPM_BENCHMARK_OUTPUT_PATH
value: /results/glm52-example.jsonWorkers.agg.extra_cli_args must be a list[str], and the final command must include --benchmark-mode with one of agg, prefill, or decode. This runtime option selects the FPM collection phase; it does not change the required single aggregated-worker deployment topology. K8sConfig.extra_env accepts concrete {name, value} entries only; valueFrom, envFrom, and Secret-derived values are not supported. The normal rule, mapping, and versioned-template pipeline still builds the base command before the extra arguments are appended. If both --benchmark-output-path and DYN_FPM_BENCHMARK_OUTPUT_PATH are supplied, they must be identical. If neither is supplied, the generator adds /results/benchmark.json to both the command and environment.
When the FPM collector drives these artifacts, its runtime also accepts this K8sConfig.extra_env entry:
| Environment entry name | Default | Accepted value and behavior |
|---|---|---|
FPM_FINALIZATION_TIMEOUT_SECONDS |
120 seconds |
A literal string containing an integer from 1 through 3600, such as {name: FPM_FINALIZATION_TIMEOUT_SECONDS, value: "600"}. Invalid values fail before engine startup. |
The collector stages this setting in collector-runtime-env.sh for both Kubernetes and Slurm so the parent collection process receives it before launching run.sh. The deadline starts after all local rank timing files are complete and bounds waiting for the matching merged artifact and any advertised worker-probe sidecar. It cannot extend the overall collection deadline. On expiry, collection fails and retains raw timings for recovery. Running the generated engine script alone does not enforce this collector-owned deadline.
K8sConfig.fpm_shared_memory_size controls the generated /dev/shm emptyDir limit, while K8sConfig.fpm_resource_labels adds labels to the workload and its Pods. Container memory, ephemeral-storage, and other requests or limits can be supplied under K8sConfig.worker_extra_pod_spec.mainContainer.resources; the Generator-resolved per-node GPU count cannot be changed there. Grove does not inject a scheduler or queue: clusters that use KAI Scheduler can set worker_extra_pod_spec.schedulerName and the kai.scheduler/queue resource label explicitly, while other clusters can supply their own scheduler settings through the same generic fields. Matching user-provided results or dshm volume-and-mount pairs are preserved instead of replaced.
The generated artifact directory contains only:
artifacts/
├── k8s_deploy.yaml
├── fpm_env.sh
└── run.sh
For a single-node topology, k8s_deploy.yaml is a keepalive Pod. For a multinode topology, it contains a keepalive LeaderWorkerSet by default or a PodCliqueSet when K8sConfig.fpm_orchestrator is grove. Its size and per-node GPU limit come from the resolved topology. GB200/MNNVL output also includes its ComputeDomain in the same YAML file. All forms contain the requested image, per-node GPU limit, preserved custom resources, volumes, and mounts, but no engine arguments or engine/FPM environment variables. By default they mount Pod-local emptyDir storage at /results. fpm_env.sh owns rank/leader discovery and exports the per-cell FPM_* collection facts; run.sh sources it, adds the engine environment exports, and execs the complete resolved python3 -m dynamo.vllm ... command.
Workers.agg.gpus_per_worker is the total GPU count for the one worker replica. When that topology exceeds NodeConfig.num_gpus_per_node, the generator emits the selected multinode workload and divides the GPU limit across its Pods. The total must divide evenly across the resolved node count, and TP * PP * DP must match it. Multinode DP must divide evenly across nodes and uses the mp data-parallel backend. Select lws for a cluster with LeaderWorkerSet, or grove for a cluster with Grove. The Collector exposes the same choice as --fpm-orchestrator.
For one node, apply the Pod, wait for it to become ready, and stream the script into it:
kubectl apply -f artifacts/k8s_deploy.yaml
kubectl wait --for=condition=Ready pod/<pod> --timeout=10m
kubectl exec -i <pod> -- bash -s < artifacts/run.shFor multiple nodes, the collector stages inputs first and then starts its collection runtime concurrently on every workload Pod; the runtime sources fpm_env.sh — which derives rank and leader address from the controller-injected LWS or Grove values — before invoking run.sh, which adds the rank, master, headless, or local-DP arguments required by that node. Multinode values passed to --dump-config-to must contain {node_rank}; the default is /results/resolved-config-node{node_rank}.json. This placeholder applies only to --dump-config-to, not to environment values or arbitrary CLI arguments. Callers must not pass Generator-owned orchestration options such as --nnodes, --node-rank, --headless, or --data-parallel-size-local in extra_cli_args. On multinode runs, leave DYN_FPM_WORKER_ID unset so the script derives <FPM_RUN_ID>-node<N>, unless the collector supplies a distinct value to each process.
With DP greater than one, DP rank 0 uses the configured benchmark output path and later DP ranks use _dp<N> before the extension. Each node waits for all results in its local rank range. Under the current FPM schema-v2 contract, a result counts as complete only when it has status: complete, valid: true, complete zero-skipped coverage, the requested benchmark mode and point phase, and nested FPM samples for the expected DP rank. For agg, result points may be prefill or decode. A terminal invalid result stops the script. The Collector still owns staging run.sh, fpm_env.sh, its collection runtime (fpm_exec.sh), and the runtime preflight on every Pod, plus the completion gate, strict result validation, result download/aggregation/evidence, exit coordination, and cleanup.
Every execution starts a new engine and reloads the model. run.sh refuses to overwrite any expected benchmark output, so each run must use new paths. With the default /results emptyDir, results remain only for the lifetime of the Pod. FPM V1 does not keep the engine or GPU-resident model alive between executions. Selecting any other deployment target preserves the existing generator output and behavior.
The current vLLM versioned template tops out at 0.20.1; newer vLLM versions (e.g. 0.24.0 from the Dynamo 1.3.0 entry) fall back to it, and all flags emitted by that template have been validated against the 0.24.0 runtime image. You may pass 0.24.0-only flags through extra_cli_args.
Generator Dynamo version (applies to Dynamo deployments only)
- Use
--generator-dynamo-version 0.7.1to select the Dynamo release. This affects both the generated backend config version and the default K8s image tag. - If
--generator-dynamo-versionis not provided, the default is the first entry inbackend_version_matrix.yaml(currently1.3.0). - If
--generated-config-versionis provided, it overrides the generated backend version, but the default K8s image tag still follows the selected Dynamo version mapping.
Use --generator-config path/to/file.yaml to provide ServiceConfig/K8sConfig/DynConfig/WorkerConfig/Workers.<role> sections, or add inline overrides via --generator-set KEY=VALUE. Examples:
--generator-set ServiceConfig.model_path=Qwen/Qwen3-32B-FP8--generator-set K8sConfig.k8s_namespace=dynamo \
You can switch the generator rule set via --generator-set rule=benchmark. This selects a rule plugin folder under src/aisim/generator/rule_plugin/.
- Default (production): if
ruleis not provided, the generator uses the default production rules. These are tuned for deployment (e.g., adjusted max batch size and CUDA graph batch sizes). - Benchmark:
--generator-set rule=benchmarkenables rules designed to align generated configs with AIC sdk results, including:- wider CUDA graph batch size coverage to match simulated results
- stricter max batch size that follows the simulated batch size
You can also define your own rule sets by adding a new folder under src/aisim/generator/rule_plugin/ and selecting it with --generator-set rule=<folder_name>.
Run aiconfigurator cli default --generator-help to print information that is sourced directly from src/aisim/generator/config/deployment_config.yaml and backend_config_mapping.yaml.
The --generator-help command supports three section options:
--generator-helpor--generator-help all(default): Shows both the full deployment schema and the backend parameter mappings--generator-help deploy: Shows the complete content ofgenerator/config/deployment_config.yamlin YAML format, including all sections such asServiceConfig.*,K8sConfig.*,WorkerConfig.*, etc.--generator-help backend: Shows only the backend parameter mappings table fromgenerator/config/backend_config_mapping.yaml, which maps unified parameter keys (e.g.,kv_cache_free_gpu_memory_fraction,kv_cache_dtype) to backend-specific parameter names for trtllm, vllm, and sglang
You can filter the backend-mapping output to a specific backend using --generator-help --generator-help-backend BACKEND, where BACKEND can be trtllm, vllm, or sglang. For example:
aiconfigurator cli default --generator-help backend --generator-help-backend sglang: Shows only sglang-specific parameter mappingsaiconfigurator cli default --generator-help backend --generator-help-backend trtllm: Shows only trtllm-specific parameter mappings
The command exits after printing the help information, so you do not need to provide the required default mode arguments (like --model-path, --backend, etc.) when using this flag.
--request-latency <ms> gives you a single end-to-end SLA on TTFT + TPOT × (OSL − 1). When the flag is set, default mode automatically enumerates TTFT/TPOT pairs that satisfy that budget (respecting any explicit --ttft, if provided) and only keeps configurations whose estimated request latency stays within the bound. Because the CLI derives TPOT from the request latency target, any --tpot argument is ignored in this mode.
- The detailed tables printed for both agg and disagg add a
request_latencycolumn, and the global Pareto plot flips to “request latency vs tokens/s/gpu” whenever every experiment is operating under this constraint. - You can still set
--ttftto reserve more headroom for prefill. Leaving it unset lets the enumerator try multiple TTFT splits automatically.
Example: search for 16x H200 configs that meet a 12s end-to-end budget while capping TTFT at 4s.
aiconfigurator cli default \
--model-path Qwen/Qwen3-32B-FP8 \
--total-gpus 16 \
--system h200_sxm \
--backend trtllm \
--request-latency 12000 \
--isl 4000 \
--osl 500 \
--ttft 4000The summary will highlight the fastest configuration whose estimated request latency is ≤ 12,000 ms and will show the derived TTFT/TPOT pair that satisfied the constraint. The example output,
********************************************************************************
* AIConfigurator Final Results *
********************************************************************************
----------------------------------------------------------------------------
Input Configuration & SLA Target:
Model: Qwen/Qwen3-32B-FP8 (is_moe: False)
Total GPUs: 16
Best Experiment Chosen: disagg at 932.91 tokens/s/gpu (disagg 1.09x better)
----------------------------------------------------------------------------
Overall Best Configuration:
- Best Throughput: 14,926.50 tokens/s
- Per-GPU Throughput: 932.91 tokens/s/gpu
- Per-User Throughput: 57.49 tokens/s/user
- TTFT: 542.58ms
- TPOT: 17.39ms
- Request Latency: 9222.18ms
----------------------------------------------------------------------------
Pareto Frontier:
Qwen/Qwen3-32B-FP8 Pareto Frontier: tokens/s/gpu_cluster vs request_latency
┌────────────────────────────────────────────────────────────────────────┐
1150.0┤ •• agg │
│ ff disagg │
│ xx disagg best │
│ │
958.3┤ │
│ ffffffffffffffx │
│ f • │
│ f ••• │
766.7┤ f ••••• │
│ f ••••• │
│ ff ••••• │
│ ff ••••• │
575.0┤ ff •••••• │
│ ff ••• │
│ ff •• │
│ •• │
383.3┤ •••• │
│ ••• │
│ •••••• │
│ ••• │
191.7┤ │
│ │
│ │
│ │
0.0┤ │
└┬─────────────────┬─────────────────┬────────────────┬─────────────────┬┘
0 3220 6440 9660 12880
tokens/s/gpu_cluster request_latency
----------------------------------------------------------------------------
Deployment Details:
(p) stands for prefill, (d) stands for decode, bs stands for batch size, a replica stands for the smallest scalable unit xPyD of the disagg system
Some math: total gpus used = replicas * gpus/replica
gpus/replica = (p)gpus/worker * (p)workers + (d)gpus/worker * (d)workers; for Agg, gpus/replica = gpus/worker
gpus/worker = tp * pp * dp = etp * ep * pp for MoE models; tp * pp for dense models (underlined numbers are the actual values in math)
agg Top Configurations: (Sorted by tokens/s/gpu)
+------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+--------------+-------------+----------+----+
| Rank | tokens/s/gpu | tokens/s/user | TTFT | request_latency | concurrency | total_gpus (used) | replicas | gpus/replica | gpus/worker | parallel | bs |
+------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+--------------+-------------+----------+----+
| 1 | 852.23 | 46.35 | 937.94 | 11704.26 | 320 (=40x8) | 16 (16=8x2) | 8 | 2 | 2 (=2x1x1) | tp2pp1 | 40 |
| 2 | 748.51 | 49.46 | 711.67 | 10799.77 | 256 (=64x4) | 16 (16=4x4) | 4 | 4 | 4 (=4x1x1) | tp4pp1 | 64 |
| 3 | 742.79 | 50.12 | 735.24 | 10691.50 | 256 (=16x16) | 16 (16=16x1) | 16 | 1 | 1 (=1x1x1) | tp1pp1 | 16 |
| 4 | 550.53 | 47.56 | 568.11 | 11060.92 | 192 (=96x2) | 16 (16=2x8) | 2 | 8 | 8 (=8x1x1) | tp8pp1 | 96 |
+------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+--------------+-------------+----------+----+
disagg Top Configurations: (Sorted by tokens/s/gpu)
+------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+----------------+------------+----------------+-------------+-------+------------+----------------+-------------+-------+
| Rank | tokens/s/gpu | tokens/s/user | TTFT | request_latency | concurrency | total_gpus (used) | replicas | gpus/replica | (p)workers | (p)gpus/worker | (p)parallel | (p)bs | (d)workers | (d)gpus/worker | (d)parallel | (d)bs |
+------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+----------------+------------+----------------+-------------+-------+------------+----------------+-------------+-------+
| 1 | 932.91 | 57.49 | 542.58 | 9222.18 | 384 (=384x1) | 16 (16=1x16) | 1 | 16 (=10x1+3x2) | 10 | 1 (=1x1) | tp1pp1 | 1 | 3 | 2 (=2x1) | tp2pp1 | 128 |
| 2 | 932.91 | 49.29 | 542.58 | 10666.29 | 384 (=192x2) | 16 (16=2x8) | 2 | 8 (=5x1+3x1) | 5 | 1 (=1x1) | tp1pp1 | 1 | 3 | 1 (=1x1) | tp1pp1 | 64 |
| 3 | 818.83 | 43.33 | 326.26 | 11842.68 | 328 (=328x1) | 16 (16=1x16) | 1 | 16 (=6x2+1x4) | 6 | 2 (=2x1) | tp2pp1 | 1 | 1 | 4 (=4x1) | tp4pp1 | 328 |
| 4 | 746.33 | 43.72 | 542.58 | 11955.71 | 496 (=496x1) | 16 (16=1x16) | 1 | 16 (=8x1+1x8) | 8 | 1 (=1x1) | tp1pp1 | 1 | 1 | 8 (=8x1) | tp8pp1 | 496 |
+------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+----------------+------------+----------------+-------------+-------+------------+----------------+-------------+-------+
********************************************************************************
2025-12-01 23:36:41,892 - aisim.legacy_cli.main - INFO - All experiments completed in 1.92 seconds
AIC's TPOT metric is the inter-token latency during the decode phase — it does not include TTFT. Pass --inclusive-tpot to report TPOT as (ttft + tpot × (osl − 1)) / osl, which spreads the TTFT cost across all output tokens. This matches the end-to-end per-token latency reported by GuideLLM and other benchmarking tools, making predicted values directly comparable to benchmark measurements.
The flag only affects terminal output and saved CSV — SLA filtering always uses inter-token latency.
By default, the Pareto frontier includes all configurations regardless of whether they meet the --tpot (or --request-latency) constraint — only the final top-N picking step filters on TPOT. This means pareto.csv and the Pareto plot may show configurations that violate your SLA targets.
Pass --strict-sla to pre-filter the Pareto frontier so that only SLA-compliant configurations are included. When this flag is active:
- Configurations exceeding
--tpot(or--request-latency) are removed before the Pareto frontier is computed. - The resulting
pareto.csv, Pareto plot, andbest_config_topnonly contain configs that meet the SLA. - TTFT filtering is already enforced at sweep time by all backends, so
--strict-slaonly adds TPOT / request-latency pre-filtering.
aiconfigurator cli default \
--model-path Qwen/Qwen3-32B-FP8 \
--total-gpus 32 \
--system h200_sxm \
--tpot 15 \
--strict-slaNote: With
--strict-sla, if no configuration meets the SLA targets, the Pareto frontier and best configs will be empty. Without the flag, the Pareto frontier preserves the full search space and you can still see which configs came closest to meeting the target.
The Python API equivalent accepts a strict_sla keyword argument:
from aisim.legacy_cli import cli_default
result = cli_default(
model_path="Qwen/Qwen3-32B-FP8",
total_gpus=32,
system="h200_sxm",
tpot=15,
strict_sla=True,
)The --database-mode argument controls how performance is estimated:
| Mode | Description |
|---|---|
SILICON |
(Default) Uses actual collected silicon data. Most accurate when data is available for your configuration. |
HYBRID |
Uses silicon data when available, falls back to SOL+empirical factor when data is missing. Best for exploring configurations that may not have complete silicon data. |
EMPIRICAL |
Uses Speed-of-Light (SOL) + empirical correction factors for all estimations. Useful for rough estimates without relying on collected data. |
SOL |
Provides theoretical Speed-of-Light time only. Useful for understanding theoretical limits. |
PCIe systems such as h100_pcie, a100_pcie, l4, and a30 are estimate-only unless you provide measured data with --systems-paths. They work with generate and with non-SILICON modes (SOL, EMPIRICAL, HYBRID); SILICON mode still requires a collected performance database.
Example using hybrid mode:
aiconfigurator cli default --model-path Qwen/Qwen3-32B-FP8 --total-gpus 32 --system h200_sxm --database-mode HYBRIDFor exp mode, you can specify database_mode in your YAML file:
exp_hybrid:
serving_mode: "agg"
model_path: "Qwen/Qwen3-32B-FP8"
system_name: "h200_sxm"
total_gpus: 8
database_mode: "HYBRID"Hybrid mode is a quick solution to support new models without modeling the operation and collecting the data. However, please be careful, only SILICON mode's result is reproducible. Other modes are for research purpose
These flags enable MTP (Multi-Token Prediction) speculative decoding in the
configuration search. MTP is never enabled implicitly — omitting both flags
keeps it off even for models that ship MTP layers. In recommend mode only, an
explicit --nextn-accepted measurement can pair with an omitted depth to select
a DSPARK architecture's fixed block size:
--nextn N— MTP draft length (compute cost side: extra MTP-layer forward plus the wider verify batch; no fixed upper bound). Default: 0 (disabled).--nextn auto— take the draft depth from the checkpoint'snum_nextn_predict_layers. If that value is absent or 0 for a DSPARK model, use its fixed architectural block size instead. Only the depth is resolved — the acceptance value below is still required because it is a property of your backend and workload, not of the model.--nextn-accepted A— Average accepted draft tokens per decode step (0 <= nextn_accepted <= nextn); each step yields1 + nextn_acceptedoutput tokens. Required whenever the draft depth is > 0 (explicit or viaauto) — there is no built-in acceptance assumption. Use a measured value from your deployment (e.g. the engine's reported average acceptance length minus 1).
nextn is part of the aic-core operation and iteration-cost model.
nextn_accepted is a workload assumption applied by the SDK predictor/sweep
after core timing, so acceptance values can be swept without recompiling or
rerunning the aic-core engine.
Example:
aiconfigurator cli default \
--model Qwen/Qwen3-32B-FP8 --total-gpus 8 --system h200_sxm \
--nextn 2 --nextn-accepted 1.2
# Depth from the checkpoint, acceptance from your measurements:
aiconfigurator cli default \
--model deepseek-ai/DeepSeek-V3 --total-gpus 8 --system h200_sxm \
--nextn auto --nextn-accepted 0.7Python API equivalent:
from aisim.legacy_cli import cli_exp
# Run experiments from a YAML file
result = cli_exp(yaml_path="example.yaml")
# Or run experiments from a dictionary
config = {
"my_exp": {
"serving_mode": "agg",
"model_path": "Qwen/Qwen3-32B-FP8",
"total_gpus": 8,
"system_name": "h200_sxm"
}
}
result = cli_exp(config=config)See src/aisim/legacy_cli/exps/database_mode_comparison.yaml for an example comparing different database modes.
For Dynamo and llm-d deployment targets, each topN directory includes two benchmark helpers alongside the deployment artifacts when --save-dir is used. Slurm emits bench_run.sh and benchmark.sbatch, as described in Generate a Slurm deployment. The FPM target does not emit benchmark helpers.
-
bench_run.sh-- A shell script for bare-metal benchmarking. It loops over a concurrency array and callsaiperf profilefor each level. Before running it, make sure the deployed service is reachable at the endpoint printed in the script, and thataiperfis installed (pip install aiperf). Usage:cd results/.../disagg/top1/ bash bench_run.sh -
k8s_bench.yaml-- A Kubernetes Job manifest that runs the sameaiperfconcurrency sweep inside the cluster. Apply it after the service is up:kubectl apply -f results/.../disagg/top1/k8s_bench.yaml
For Slurm jobs or direct shell benchmarks, see
Offline benchmark tokenizers.
The Slurm container mounts and environment settings described there do not
configure k8s_bench.yaml; a Kubernetes benchmark needs its cache volume mounts
and offline environment settings in the benchmark Job itself.
A locally mounted serving model does not guarantee that the selected AIPerf
version can load its tokenizer by filesystem path.
Concurrency sweep. The Dynamo and llm-d benchmark artifacts iterate over a base concurrency list [1, 2, 8, 16, 32, 64, 128]. When an estimated concurrency is available from the AIConfigurator run, three additional points are added: the estimate itself and its +/-5% neighbors. This targets the operating point AIConfigurator found optimal. Slurm uses SlurmConfig.benchmark_concurrency and SlurmConfig.benchmark_rounds.
Templated values. The scripts are pre-filled with the model name, tokenizer, ISL/OSL, endpoint URL, and streaming mode from the run that generated them -- no manual editing is needed for the common case.
If you want to customize your experiment apart from simple command which only compares disagg and agg of a same model, you can use exp mode. The command is,
aiconfigurator cli exp --yaml-path example.yamlYAML format: Experiment YAML uses the flat
Taskschema — every key maps 1:1 to aTaskfield, with nomode:selector and noconfig:/worker_config:nesting. Seeexample.yamlfor the annotated template.The legacy V1 nested format (
mode/config/worker_config/replica_config/profiles) is deprecated and only a limited compatibility shim remains: V1 YAML still loads, but it is auto-converted to V2 with aDeprecationWarning, and any field with no V2 equivalent is rejected (not silently dropped). Seeexample_v1_deprecated.yamlfor the old shape. Write all new configs in the flat V2 format below.
An example YAML file looks like this; see the annotated experiment template. Let's split the yaml file into several sections.
- exps
exps:
- agg_full
- disagg_fullexps section selects which experiments to run, in order. If omitted, all top-level experiments are run.
- A certain exp definition
disagg_full:
serving_mode: disagg # required
total_gpus: 32 # required
# Workload + SLA (shared across roles)
isl: 4000 # input sequence length (default 4000)
osl: 1000 # output sequence length (default 1000)
prefix: 0 # prefix cache length (default 0)
ttft: 1000.0 # target TTFT in ms (default 1000.0)
tpot: 40.0 # target TPOT in ms (default 40.0)
# Speculative decoding: never enabled implicitly; nextn_accepted is required
# when the draft depth is > 0. nextn: auto takes the depth from the
# checkpoint's num_nextn_predict_layers (the acceptance value is still yours).
nextn: 1
nextn_accepted: 0.85
# MoE kernel backend (shared; e.g. "megamoe" for DeepSeek-V4 on Blackwell SGLang).
# Large-EP (wideEP) is explored automatically when perf data covers the model
# shape; restrict with *_moe_ep_candidates.
moe_backend: null
# --- Prefill role ---
prefill_model_path: deepseek-ai/DeepSeek-V3 # required
prefill_system_name: h200_sxm # required
prefill_backend_name: trtllm # trtllm (default) | vllm | sglang
# Quant override (default: inferred from the HF model config)
prefill_gemm_quant_mode: fp8_block # fp8 | fp8_block | bfloat16
prefill_moe_quant_mode: fp8_block # fp8 | fp8_block | w4afp8 | bfloat16
prefill_kvcache_quant_mode: bfloat16 # fp8 | int8 | bfloat16
prefill_fmha_quant_mode: bfloat16 # fp8 | bfloat16
prefill_comm_quant_mode: half
# Search space (tp=attention, pp=layers, dp=attention DP, moe_tp/moe_ep=MoE)
prefill_num_gpu_candidates: [4, 8]
prefill_tp_candidates: [1, 2, 4, 8]
prefill_pp_candidates: [1]
prefill_dp_candidates: [1]
prefill_moe_tp_candidates: [1]
prefill_moe_ep_candidates: [1, 2, 4, 8]
# --- Decode role (model_path must equal the prefill model) ---
decode_model_path: deepseek-ai/DeepSeek-V3 # required
decode_system_name: h200_sxm # required
decode_backend_name: trtllm
decode_gemm_quant_mode: fp8_block
decode_moe_quant_mode: fp8_block
decode_kvcache_quant_mode: bfloat16
decode_fmha_quant_mode: bfloat16
decode_comm_quant_mode: half
decode_num_gpu_candidates: [4, 8]
decode_tp_candidates: [1, 2, 4, 8]
decode_pp_candidates: [1]
decode_dp_candidates: [1, 2, 4, 8]
decode_moe_tp_candidates: [1]
decode_moe_ep_candidates: [1, 2, 4, 8]
# --- Disagg orchestration: replica shaping + perf correction ---
num_gpu_per_replica: [8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128]
max_gpu_per_replica: 128 # caps num_gpu_per_replica (0 = no limit)
max_prefill_workers: 32 # max prefill workers per replica (x in xPyD)
max_decode_workers: 32 # max decode workers per replica (y in xPyD)
prefill_latency_correction: 1.1
decode_latency_correction: 1.08
prefill_max_batch_size: 1
decode_max_batch_size: 512This is long; the basics:
serving_mode:aggordisaggfor this experiment.total_gpus: total GPU budget for the deployment.- For
disagg, the worker spec is per-role: setprefill_*/decode_*formodel_path,system_name,backend_name, the*_quant_modefields, and the*_candidatessearch lists.decode_model_pathmust equalprefill_model_path(hetero-disagg means different systems, not models). - For
agg, the same fields are top-level (model_path,system_name,gemm_quant_mode,agg_tp_candidates, ...) — seeagg_fullin the template. backend_name:trtllm(default),vllm, orsglang.backend_version,isl,osl,ttft,tpot: same meaning as indefaultmode (shared, top-level).- Large-EP (wideEP) has no key: it is explored automatically whenever the performance database covers the model's MoE shape on the role's system/backend (MoE all-to-all dispatch/combine plus EP compute data). Restrict or force EP sizes with
*_moe_ep_candidates. The deprecated keys (enable_wideep,prefill_enable_wideep,decode_enable_wideep,moe_backend: deepep_moe) are still accepted with a one-time warning and have no modeling effect. One search-default residue remains: on SGLang, a config that spellsenable_wideep/moe_backend: deepep_moestill narrows the defaultmoe_tpcandidates to[1](a resolved-config compatibility behavior) — an explicit*_moe_tp_candidateslist always wins. nextn/nextn_accepted: MTP speculative decoding (never auto-enabled;nextn_acceptedis required when the resolvednextn > 0).- The replica/correction knobs (
num_gpu_per_replica,max_*_workers,*_latency_correction, ...) are covered in Advanced Tuning. Typically the only thing you need to touch is the quantization.
Quantization override order: explicit *_quant_mode fields take precedence; any mode left unset is filled from the model's HF quantization metadata.
You can drop everything optional and keep just the required fields plus the few knobs you care about. Here's a minimal disagg for a large-EP (wideEP) study — no flag needed, the multi-node EP ladder joins the search automatically because the gb200 database covers this model's MoE shape:
disagg_simplified:
serving_mode: disagg
total_gpus: 512
nextn: 2
nextn_accepted: 1.1
prefill_model_path: nvidia/DeepSeek-V3.1-NVFP4
prefill_system_name: gb200
decode_model_path: nvidia/DeepSeek-V3.1-NVFP4
decode_system_name: gb200Everything omitted falls back to defaults / HF inference. With large-EP candidates in play the replica budget widens automatically (max_gpu_per_replica defaults to 512). To pin a role to large EP sizes only, add e.g. decode_moe_ep_candidates: [16, 32, 64].
Let's go through some pre-defined experiments for reference.
- homegeneous vs. heterogenous The example yaml
exps:
- exp_h200_h200
- exp_b200_h200
exp_h200_h200:
serving_mode: disagg
total_gpus: 16
isl: 4000
osl: 500
ttft: 300.0
tpot: 50.0
prefill_model_path: Qwen/Qwen3-32B-FP8
prefill_system_name: h200_sxm # prefill on H200
prefill_backend_name: trtllm # vllm | sglang also work
decode_model_path: Qwen/Qwen3-32B-FP8
decode_system_name: h200_sxm # decode on H200
decode_backend_name: trtllm
exp_b200_h200:
serving_mode: disagg
total_gpus: 16
isl: 4000
osl: 500
ttft: 300.0
tpot: 50.0
prefill_model_path: Qwen/Qwen3-32B-FP8
prefill_system_name: b200_sxm # prefill on B200
prefill_backend_name: trtllm
decode_model_path: Qwen/Qwen3-32B-FP8
decode_system_name: h200_sxm # decode on H200
decode_backend_name: trtllmWe defined two experiments. exp_h200_h200 uses H200 for both prefill and decode. exp_b200_h200 uses B200 for prefill and H200 for decode — hetero-disagg is expressed purely by giving the two roles different *_system_name values (the model must be the same).
Note: You can also compare different backends by setting different backend_name values (trtllm, vllm, sglang) in your experiments.
- use a specific quantization The example yaml
exps:
- exp_agg
- exp_disagg
exp_agg:
serving_mode: agg
model_path: Qwen/Qwen3-32B-FP8
system_name: h200_sxm
total_gpus: 16
backend_name: trtllm
isl: 4000
osl: 500
ttft: 600.0
tpot: 16
# per-tensor FP8 on every component
gemm_quant_mode: fp8
moe_quant_mode: fp8
kvcache_quant_mode: fp8
fmha_quant_mode: fp8
comm_quant_mode: half
exp_disagg:
serving_mode: disagg
total_gpus: 16
isl: 4000
osl: 500
ttft: 600.0
tpot: 16
prefill_model_path: Qwen/Qwen3-32B-FP8
prefill_system_name: h200_sxm
prefill_backend_name: trtllm
prefill_gemm_quant_mode: fp8
prefill_moe_quant_mode: fp8
prefill_kvcache_quant_mode: fp8
prefill_fmha_quant_mode: fp8
prefill_comm_quant_mode: half
decode_model_path: Qwen/Qwen3-32B-FP8
decode_system_name: h200_sxm
decode_backend_name: trtllm
decode_gemm_quant_mode: fp8
decode_moe_quant_mode: fp8
decode_kvcache_quant_mode: fp8
decode_fmha_quant_mode: fp8
decode_comm_quant_mode: halfHere we override the quantization of Qwen/Qwen3-32B-FP8: the default is blockwise FP8 for GEMM, and we set per-tensor FP8 explicitly via the *_quant_mode fields. (The deprecated V1 way was profiles: ["fp8"], which expanded to exactly these fields.)
You can refer to src/aisim/legacy_cli/exps to find more reference yaml files.
Use exp mode for flexible experiments, default mode for convenient agg vs disagg comparison with SLA optimization, and generate mode for quick config generation without sweeping. All modes support generating configs for frameworks automatically by --save-dir DIR.
This section walks through the typical workflow from checking hardware/model support all the way to benchmarking a deployed service. Each step feeds into the next.
Scenario: Deploy Qwen3-32B-FP8 on 8x H200 GPUs with SLA targets of TTFT <= 600 ms and TPOT <= 50 ms.
You can optionally verify that your model/system combination is supported before running a sweep. This step is not required — you can skip it and run cli default directly.
aiconfigurator cli support --model Qwen/Qwen3-32B-FP8 --system h200_sxmIf the output shows Aggregated Support: YES and/or Disaggregated Support: YES, proceed. Otherwise, try a different backend (--backend vllm or --backend sglang) or system.
Run the parameter sweep to compare aggregated vs. disaggregated serving and find the best config under your SLA:
aiconfigurator cli default \
--model Qwen/Qwen3-32B-FP8 \
--total-gpus 8 \
--system h200_sxm \
--ttft 600 --tpot 50 \
--isl 4000 --osl 500 \
--save-dir results \
--generator-set ServiceConfig.head_node_ip=0.0.0.0 \
--generator-set ServiceConfig.model_path=/workspace/models/Qwen3-32B-FP8--save-dir generates deployment-ready artifacts (engine configs, run scripts, K8s manifests, and benchmark helpers) under results/.
If cli support shows your model/system combo is unsupported, or cli default fails to find a valid configuration, generate gives you the smallest TP that fits the model in memory. Otherwise, you can use the cli default results directly and skip this step.
aiconfigurator cli generate \
--model Qwen/Qwen3-32B-FP8 \
--total-gpus 8 \
--system h200_sxm \
--save-dir results_naiveUse the generated artifacts to launch the service. For bare-metal (single-node):
mkdir -p /workspace/engine_configs
cp results/.../disagg/top1/*_config.yaml /workspace/engine_configs/
cd results/.../disagg/top1/
bash run_0.shFor Kubernetes (Dynamo):
kubectl apply -f results/.../disagg/top1/k8s_deploy.yamlFor llm-d (Helm):
helm install my-model llm-d/llm-d-modelservice \
--values results/.../disagg/top1/llm-d-values.yamlSee the Deployment Guide for multi-node and K8s details.
After the service is healthy, run the generated benchmark sweep to validate performance at the predicted concurrency:
# Bare-metal
bash results/.../disagg/top1/bench_run.sh
# Or Kubernetes
kubectl apply -f results/.../disagg/top1/k8s_bench.yamlCompare the measured TTFT, TPOT, and tokens/s/gpu against the AIConfigurator estimates printed in Step 2. See Benchmark Artifacts for details on the generated scripts.
Use --deployment-target slurm with --generator-config containing a
SlurmConfig section. Supply account, partition, container_image and any
container_mounts; resource/time limits and benchmark concurrency can also be
set there. For example, --generator-set SlurmConfig.partition=batch overrides
the partition. Backend and Dynamo version options retain their normal meaning.
For generation without an SLA search, use aiconfigurator cli generate --model-path MODEL --system SYSTEM --backend vllm --total-gpus N --deployment-target slurm --generator-config slurm.yaml --save-dir ./results.
This compatibility CLI is included in AISim. Put rule: benchmark in the
input YAML when using the benchmark sizing rules.
The target produces deploy.sbatch for persistent serving and benchmark.sbatch
for a complete deploy/health-check/AIPerf/cleanup run. On the cluster, run
bash submit.sh benchmark --test-only, then bash submit.sh benchmark (or
bash submit.sh serve). Submissions save a job receipt and prevent duplicate runs
from the same bundle directory. V1 generates single-node NVIDIA agg/P-D topologies
for vLLM, SGLang and TRT-LLM. Automated tests cover emitted artifacts for all three
backends and CPU supervisor behavior using simulated launches and local HTTP
services. GPU execution and Slurm/Pyxis integration require validation with the
chosen container and cluster. The image must contain Dynamo, etcd/NATS and, for
benchmarks, AIPerf. Readiness retries temporary HTTP 404/503 and transport errors
within the startup deadline; other HTTP errors fail immediately. See the
Slurm target section
for the complete input example, outputs and current scope.
YAML format: Experiment YAML uses the flat
Taskschema — every key maps 1:1 to aTaskfield, with nomode:selector and noconfig:/worker_config:nesting. The examples below are in this format; seeexample.yamlfor the full template.The legacy V1 nested format is deprecated and kept only behind a limited compatibility shim: it auto-converts to V2 with a
DeprecationWarning, and any field with no V2 equivalent is rejected rather than silently dropped. Seeexample_v1_deprecated.yamlfor the old shape.
attention_backend and moe_backend use the public AttentionBackend and
MoEBackend string enums. YAML uses their lowercase values:
attention_backend:flashinferorfa3;nullkeeps the effectiveflashinferdefault.moe_backend:megamoeor the deprecated compatibility valuedeepep_moe;nullleaves backend selection unspecified.
These are performance-model selectors. They are separate from runtime-server
backend names such as TRT-LLM CUTLASS or WIDEEP.
In aiconfigurator, the inference framework and serving modeling is relatively complicated compared with the most simplified CLI entrypoint. For example, behind the command,
aiconfigurator cli default --model-path Qwen/Qwen3-32B-FP8 --total-gpus 512 --system h200_sxmWe hide a lot of default settings of the execution. Such as the quantization of each component, the matrix multiply, attention, moe, etc. We also hide the parallel config for how we search possible combinations.
The optional params of cli contains the definition of ISL, OSL, TTFT and TPOT while we don't cover these params mentioned above. In CLI, We auto populate all these stuff for default mode and allow users to modify in exp mode.
aiconfigurator cli exp --yaml-path example.yamlThe example.yaml is defined here. Let's take a look at example.yaml
## agg_full: aggregated, full control. Use as a template.
agg_full:
serving_mode: agg # required
model_path: deepseek-ai/DeepSeek-V3 # required
system_name: h200_sxm # required
total_gpus: 8 # required
backend_name: trtllm # trtllm (default) | vllm | sglang
isl: 4000
osl: 1000
ttft: 1000.0
tpot: 40.0
# Large-EP (wideEP) is explored automatically when perf data covers the model
# shape; restrict with *_moe_ep_candidates (see "large-EP exploration" below).
moe_backend: null
# Speculative decoding (MTP): opt-in only; nextn_accepted is required
# when nextn > 0 and must lie in [0, nextn].
nextn: 1
nextn_accepted: 0.85
# Quantization of each component (default: inferred from HF config)
gemm_quant_mode: fp8_block # fp8 | fp8_block | bfloat16
moe_quant_mode: fp8_block # fp8 | fp8_block | w4afp8 | bfloat16
kvcache_quant_mode: bfloat16 # fp8 | int8 | bfloat16
fmha_quant_mode: bfloat16 # fp8 | bfloat16
comm_quant_mode: half
# Parallelism search space
agg_num_gpu_candidates: [4, 8]
agg_tp_candidates: [1, 2, 4, 8]
agg_pp_candidates: [1]
agg_dp_candidates: [1, 2, 4, 8]
agg_moe_tp_candidates: [1]
agg_moe_ep_candidates: [1, 2, 4, 8]
## disagg_full: disaggregated, full control. Use as a template.
disagg_full:
serving_mode: disagg # required
total_gpus: 32 # required
isl: 4000
osl: 1000
ttft: 1000.0
tpot: 40.0
# MTP is opt-in; nextn_accepted required when nextn > 0, in [0, nextn].
nextn: 1
nextn_accepted: 0.85
# MoE kernel backend (shared). Large-EP (wideEP) is explored automatically
# when perf data covers the model shape; restrict with *_moe_ep_candidates.
moe_backend: null
# --- Prefill worker ---
prefill_model_path: deepseek-ai/DeepSeek-V3
prefill_system_name: h200_sxm
prefill_backend_name: trtllm
prefill_gemm_quant_mode: fp8_block
prefill_moe_quant_mode: fp8_block
prefill_kvcache_quant_mode: bfloat16
prefill_fmha_quant_mode: bfloat16
prefill_comm_quant_mode: half
prefill_num_gpu_candidates: [4, 8]
prefill_tp_candidates: [1, 2, 4, 8]
prefill_pp_candidates: [1]
prefill_dp_candidates: [1] # attention DP off here; raise to enable
prefill_moe_tp_candidates: [1]
prefill_moe_ep_candidates: [1, 2, 4, 8]
# --- Decode worker (model_path must equal the prefill model) ---
decode_model_path: deepseek-ai/DeepSeek-V3
decode_system_name: h200_sxm
decode_backend_name: trtllm
decode_gemm_quant_mode: fp8_block
decode_moe_quant_mode: fp8_block
decode_kvcache_quant_mode: bfloat16
decode_fmha_quant_mode: bfloat16
decode_comm_quant_mode: half
decode_num_gpu_candidates: [4, 8]
decode_tp_candidates: [1, 2, 4, 8]
decode_pp_candidates: [1]
decode_dp_candidates: [1, 2, 4, 8]
decode_moe_tp_candidates: [1]
decode_moe_ep_candidates: [1, 2, 4, 8]
# --- Replica shaping + perf correction (disagg only) ---
num_gpu_per_replica: [8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128]
max_gpu_per_replica: 128
max_prefill_workers: 32
max_decode_workers: 32
prefill_latency_correction: 1.1
decode_latency_correction: 1.08
prefill_max_batch_size: 1
decode_max_batch_size: 512We keep only the full agg and disagg versions here. Note:
- The worker spec is top-level for agg (
gemm_quant_mode,agg_tp_candidates, ...) and per-role for disagg (prefill_*/decode_*); the two roles look very similar. - Disagg additionally has the replica-shaping fields (
num_gpu_per_replica,max_*_workers) and the correction fields (*_latency_correction,*_max_batch_size). For prerequisites, read Common Arguments before tuning the search space.
Let's focus on search system config section. Let's take disagg config as an example,
A replica is defined the minimal scalable unit composed of xPyD, i.e., x prefill workers and y decode workers.
In the replica config, we use a list num_gpu_per_replica to define how many gpus we can have in a replica. This parameter helps
limit the max num gpu in a replica which avoids unreasonable results such as a single replica contains 2048 gpus. Even the theoretical perf
is good, it's not practical. It also helps align the replica to a multiplier of 8, which aligns with num gpu in a typical server.
max_gpu_per_replica is then capping the num_gpu_per_replica list if it's specified.
max_prefill_workers and max_decode_workers limit the x and y of xPyD. This helps reduce the search space. In some extreme experiments,
such as ISL:OSL is 8000:2, this will limit the disagg perf but in most cases, leave it to 32 makes sense.
Once we have the xPyD config, let's look into the config of p or d worker. We have two types of setting, quantization and parallelism.
We allow users to specify different quant methods for different components even the framework doesn't support it for users to study perf impact. Choose the one you want. Options are listed as comment. fp8 stands for fp8 per-tensor quant. fp8 block is for blockwise quant. bfloat16 is bf16.
Quantization defaults are inferred from the Hugging Face model config (config.json plus optional hf_quant_config.json).
Setting any *_quant_mode field explicitly overrides those defaults.
This is the most complicated part of the search space definition. Each dimension is a per-role list: agg_* for agg, prefill_* / decode_* for disagg.
First, *_num_gpu_candidates defines how many GPUs in a worker; the searched result will do exact match.
Then, we define options for different components, tp for attention module, pp for transformer layer. Specifically for MoE, dp for attention data parallel,
moe_tp for moe tensor parallel and moe_ep for moe expert parallel.
Here's the pseudo code about how we enumerate valid configs based on the various list definitions,
for config in space[tp x pp x dp x moe_tp x moe_ep]:
if config.tp * config.dp == config.moe_tp * config.moe_ep: # valid config, ensure the attention module has same gpus as ffn moe module
if config.tp * config.dp * config.pp in num_gpus: # valid num_gpus
yield configAll the valid combinations will print a line of log for each like this: Enumerated Disagg decode parallel config: tp=1, pp=1, dp=1, moe_tp=1, moe_ep=1
We will then find a best one among these enumrations.
Large-EP — multi-node, EP-only MoE parallelism, formerly gated behind the enable_wideep flag — no longer has a switch. For every MoE model, aiconfigurator probes whether the role's performance database covers the model's MoE shape (the MoE all-to-all dispatch/combine data plus the EP compute data, under the resolved moe_quant_mode). When it does, the default search lists are widened with a multi-node EP ladder (moe_ep up to 64) and the large-EP tuples compete with the fused configs inside the same search; the MoE communication backend is resolved per tuple from data coverage.
When the shape is not covered, large-EP exploration stays off for that model/system and a one-time INFO log tells you exactly what to collect:
large-EP exploration is OFF for deepseek-ai/DeepSeek-V3 on h200_sxm/trtllm: no MoE all-to-all + EP-compute coverage for this model shape (hidden=7168, topk=8, experts=256) under moe_quant_mode=fp8_block. Run the moe_a2a and moe_ep collectors for this model/system to enable it; the fused (small-EP) path is unaffected.
Run the named collectors (see collector/) for the model/system to enable it — no config change is needed afterwards.
To restrict or force EP sizes, set *_moe_ep_candidates explicitly: decode_moe_ep_candidates: [16, 32, 64] pins decode to large-EP tuples only, while [1, 2, 4, 8] keeps the search single-node. The deprecated keys (enable_wideep, prefill_enable_wideep, decode_enable_wideep, moe_backend: deepep_moe) are still accepted with a one-time warning and have no modeling effect; the one search-default residue is on SGLang, where spelling them still narrows the default moe_tp candidates to [1] — an explicit *_moe_tp_candidates list always wins.
The recommend-mode CLI spellings --enable-wideep and --moe-backend deepep_moe follow the same deprecated-and-ignored behavior.
The final tuning config is for some correction and deployment purpose.
prefill_latency_correction / decode_latency_correction scale the predicted prefill/decode worker perf. If you find the predicted latency too optimistic, set a factor to make it more realistic: latency_corrected = latency_predicted * latency_correction. This adjusts the generated configs for better alignment with real deployment.
prefill/decode_max_batch_size, in practical, you don't have to make decode batch size too large, 512 is a very high value. It's for local rank rather than the global batch size.
And for prefill, for typical ISL larger than 1000, it's almost saturating the compute flops, doing batching will not give you too much perf gain but makes the TTFT x times.
It's same for agg. You can treat agg as a prefill or decode worker.
For vision-language workloads (image inputs set), enable_epd: true serves the vision encoder from a dedicated
encode-worker pool instead of colocated with the LM workers — agg becomes E+agg, disagg becomes E+P+D. The encode
pool is rate-matched against the LM pools; result rows carry (e)workers/(e)tp/(e)bs columns. Optional knobs
(all require enable_epd; keys are Task field names, not CLI flags):
encoder_tp_candidates: encode-worker TP sizes to sweep (default[1, 2, 4, 8])encoder_batch_candidates: encode batch sizes to sweep (default[1, 2, 4, 8]; capped at 8, SGLang'sSGLANG_ENCODER_MAX_BATCH_SIZEdefault)max_encoder_workers: encode-worker cap per rate-matched cell (default 32)encoder_latency_correction: encode-latency correction scale (default 1.0)encoder_system_name: system (GPU type) for a heterogeneous encode pool; backend and version follow the P/agg siderate_match_encoder_degradation: encode-pool rate-matching degradation (default 0.9, alongside the prefill/decode factors)
See the vl_epd_agg experiment in python/aisim/src/aisim/legacy_cli/example.yaml for a complete template.
In order to save search time, you need to reduce the search space by choosing fewer parallel options. Say for *_num_gpu_candidates here, it's DeepSeek V3 with 671B model
parameters. With fp8_block, the rough estimation of the model weights is 671GB. You can not hold it on 4/2/1 gpus, you can modify it to [8] only.
Of source, in most cases, we would like to have the default set work. Ideally, users don't have to modify them. But for specific perf studies, you can try it.