Data collection is a standalone process for collecting the database for aiconfigurator. By default, you don't have to collect the data by yourself. Small versions of database will not introduce huge perf difference. Say, you can use 1.0.0rc3 data of trtllm on h200_sxm and deploy the generated configs with Dynamo + trtllm 1.0.0rc4 worker.
If you want to go through the process, you can try belowing commands. However, you need to prepare the env by yourself such as installing a specific trtllm version. This process is not well verified, you need to debug sometimes.
For a framework-version or GPU-platform upgrade, follow the
Collector Upgrade Playbook.
The repo-tracked aic-auto-collect
skill applies that workflow during long, resumable collection runs.
SGLang 0.5.14 Hopper/Blackwell follow-up work must also consult the
platform alignment ledger, a local campaign record kept outside the repo
(docs/perf_database/sglang-0.5.14-hopper-blackwell-ledger.md in the
campaign workspace).
Normal collection retains the shared grid and model geometry coverage. Add
--include-long-context to collect the additional correlated points declared
on selected model rows; it defaults to off. A model selection or
--model-cases-full is required. Existing long-context points in the normal
shared module grid remain enabled.
python collect.py --backend sglang --model-path zai-org/GLM-5.2 --ops dsa_generation_module --include-long-context
python collect.py --backend sglang --model-cases-full --sm 103 --plan-only --include-long-contextThe optional points reuse the native geometry, actual projection precision and
existing framework dispatch. Model context limits validate declarations; they
do not automatically request every length or batch. Workers receive explicit
points and retain grouped setup. No environment switch changes their rows.
--plan-only reports the selected mode and hashes its actual case list. Use
separate checkpoint/output roots for before/after cost experiments; changing a
group's point list changes its existing task ID and can repeat its older points
on resume, while preventing newly added points from being skipped.
See the declaration fields and coverage rationale. Collection-cost estimates count setup changes, point preparation, warmup/capture and timed forwards; scheduler wait and GPU allocation overhead are excluded.
Kimi-K3 adds no optional MLA long-context points yet: pinned vLLM 0.30.0
MultiHeadLatentAttention
uses NoPE and fused cache operations that the existing generic DeepSeek module
does not reproduce. A native adapter needs cache registration/binding, absorbed
weight processing, forward/metadata and FP8 policy support, plus a consumer
review of its output-gate timing boundary and silicon qualification. See the
deferred extension.
Normal coverage and independent KDA width additions remain enabled; the missing
native long-context route is an unmeasured gap, not zero collection cost.
SGLang 0.5.14 standalone MLA rows distinguish kernel compute from KV storage.
On B200/B300, trtllm_mla quantizes Q/K/V internally when the KV cache is FP8,
so those rows use mla_dtype=fp8, kv_cache_dtype=fp8. On H200, the absorbed
576-dimensional FA3 path uses BF16 compute, including when KV storage is FP8.
BF16 inputs to the collector do not determine the kernel's compute precision.
Compute-precision labels are restricted to the audited trtllm_mla, fa3,
and triton backends; an unknown backend raises instead of receiving a BF16 label.
Legacy Blackwell rows labeled bfloat16/fp8 need fresh measurements with the
correct label; do not copy their timings into a second precision key.
Before collecting the data, make sure you own the whole node and no interfierence happens. Next, please enable persistent-mode and lock frequency of the node. Make sure the cooling system of the node is working well.
sudo nvidia-smi -pm 1sudo nvidia-smi -ac yyy,xxxxxx, yyy frequency can be queried by nvidia-smi -q -i 0, refer to the Max Clocks part, xxx is SM frequency, yyy is Memory frequency. A script to set frequency:
#!/bin/bash
# Run nvidia-smi query and extract SM and Memory frequencies from Max Clocks
sm_freq=$(nvidia-smi -q -i 0 | grep -A 4 "Max Clocks" | grep "SM " | grep -o "[0-9]\+ MHz" | grep -o "[0-9]\+")
mem_freq=$(nvidia-smi -q -i 0 | grep -A 4 "Max Clocks" | grep "Memory " | grep -o "[0-9]\+ MHz" | grep -o "[0-9]\+")
# Check if frequencies were successfully extracted
if [ -z "$sm_freq" ] || [ -z "$mem_freq" ]; then
echo "Error: Could not extract SM or Memory frequency from Max Clocks."
exit 1
fi
# Generate the command
echo "sudo nvidia-smi -ac $mem_freq,$sm_freq"
Prepare a clean env with the target framework and nccl lib installed.
export PATH=$PATH:${NCCL_TEST_BIN_PATH}/
network/collect_comm.sh #all_reduce data will be collected using default trtllm backend
network/collect_comm.sh --all_reduce_backend vllm #all_reduce data will be collected using vllm backend
network/collect_comm.sh --all_reduce_backend vllm --device xpu #all_reduce data will be collected using vllm backend on XPUToday we only collect intra-node comm. This script will collect custom allreduce data for trtllm within a node.
It will also collect nccl allreduce, all_gather, all2all, reduce_scatter using nccl.
The standalone scripts stage nccl_perf.txt, oneccl_perf.txt, and
custom_allreduce_perf.txt. Collector finalization converts accepted output
to parquet under
python/aisim/src/aisim_core/systems/data/<system>/comm/<backend>/<version>/.
TRT-LLM MoE workers refuse a case before module allocation when its local resident expert weights alone exceed live device capacity. This lower bound accounts for gated/non-gated matrices and TP/EP sharding; passing it does not establish room for scales, inputs, outputs, graph buffers or native workspace. Those later allocation failures remain recorded failures rather than hidden shape exclusions. Plans remain independent of device memory.
The model-centric case layer (introduced as "collector v2") is how collection
coverage is declared, and it is unchanged under Collector V3 — V3 adds
op-centric management on top (per-family runtime pins, provenance sidecars,
declared reuse), it does not replace how cases are declared or healed. A healing run collects
cases for a specific model/GPU pair instead of running every op bucket and
hoping the support matrix improves. Use --model-cases-full when you want
the case YAML to define a full model-centric run. Omitting all model-case
flags runs the backend registry directly without a model-centric case plan.
# Heal one model on one GPU type.
python3 collect.py --backend sglang \
--model-path sgl-project/DeepSeek-V4-Flash-FP8 \
--gpu b200_sxm
# Inspect the resolved model plan without collecting.
python3 collect.py --backend sglang \
--model-path sgl-project/DeepSeek-V4-Flash-FP8 \
--gpu b200_sxm \
--plan-only
# Select the architecture case file directly.
python3 collect.py --backend trtllm \
--model-architecture Qwen3MoeForCausalLM \
--gpu b200_sxm \
--plan-only
# Full model-centric run: aggregate base ops plus every model case YAML file.
python3 collect.py --backend trtllm --model-cases-full
# Raw backend registry run: no model-centric case plan.
python3 collect.py --backend trtllm
# Ephemeral subset filter (healing): substring match, repeatable.
python3 collect.py --backend trtllm --ops moe --case-filter "tp=4"Case files (see cases/README.md for the full authoring guide):
cases/base_ops/<op>.yaml — shared sweep recipes (interpolation grid)
cases/models/<architecture>_cases.yaml — op activation + model shape values
model_cases.py — resolves which ops a model plan collects
skips.py — the one list of cases a collection leaves out
The plan is one equation: cases = dedup(base grid ∪ model shapes), minus what
skips.py skips. For an op with an op plan (OP_PLANS in op_plans.py),
--plan-only with --sm or --gpu also prints case counts per dtype and what
was left out under which entry; --plan-table DIR writes one CSV row per case.
TRT-LLM MLA precision and GPT-OSS TP profile collection evidence is recorded in the September 17 campaign.
Use the dedicated python3 -m collector.fpm_forward entry point for vLLM
whole-forward campaigns. The compatibility route
python3 collector/collect.py --ops fpm_forward remains supported and must be
selected alone. Both routes require a resolved model, --gpu, and
--fpm-max-gpus. The campaign derives a latency-blind prefill/decode
design from the model's AIC attention cases, asks Generator for three rendered
artifacts per cell — a keepalive Pod, LeaderWorkerSet, or Grove PodCliqueSet
manifest plus fpm_env.sh and a thin engine-launch run.sh — stages them with
its own in-pod runtime fpm_exec.sh (etcd lifecycle, result gate and checker,
completion barrier, exit-code policy), and publishes the formal table only
after every frozen cell passes.
For multinode runs, select the installed controller with
--fpm-orchestrator lws|grove. Start with --plan-only or a one-cell
--smoke --limit 1 run.
python3 -m collector.fpm_forward \
--model-path nvidia/GLM-5.2-NVFP4 \
--gpu b200_sxm \
--fpm-max-gpus 8 \
--plan-onlyFor profile-based FPM collection, the profile's backend_version is the dataset
identity label. It may be a custom string such as my-vllm-patch-3 and need not
match the installed vLLM package version. The collector retains the actual
container version separately in collector-provenance.json under
runtime.backend_version; runtime capability, benchmark-schema, topology and
precision checks still apply. An optional Dynamo version is provenance only and
does not select FPM templates or resolve the backend version. See the
onboarding version contract
for guided setup, omitted-version detection inside the target image, and
publication identity.
The low-level entry point also accepts --fpm-backend-version LABEL for dataset
identity and --fpm-runtime-backend-version VERSION for an independently observed
package-version pin. Guided onboarding supplies that pin after its image
preflight. The low-level collector does not launch this separate CPU preflight:
without an observed pin it can collect timings and record actual package
provenance, but it cannot select audited version-specific memory/execution hooks.
Supply the observed pin or accepted runtime instrumentation when those hooks are
required. The op-level collector's runtime manifest and version-routing rules
below are unchanged.
Generation-time admission may omit a topology only when a concrete AIC size-vs-capacity estimate proves its configured token envelope cannot fit. Missing performance data, bootstrap architectures, and structural estimator errors remain runnable so the target runtime supplies the observation.
Model identity is resolved from a local checkpoint/config, the packaged AIC
config cache, or Hugging Face. A real model architecture that is not registered
by AIC uses a config-derived bootstrap template; an unresolved or empty config
fails before planning. When the checkpoint exists only inside the runtime Pod,
pass its real local metadata explicitly with
--fpm-model-config /path/to/config.json (a directory containing config.json
is also accepted). The resolved config snapshot and SHA-256 are frozen into the
collection plan.
The collector detects a cache layout that config.json cannot describe, such as
linear-attention or recurrent state, with the same check as config intake. The
onboarding guide
lists the markers. No FPM attention template represents such a layout. Without
--fpm-model-profile or --fpm-backend-version, the collector refuses it, and
registered model cases do not change this. With one of these options, the
collector gives the layout the native_runtime capability. Precision then comes
from the checkpoint, and the runtime measures every cell.
Generator and Collector share the accepted Dynamo-native benchmark result
schema versions (2, and 3 with a power window) through the rendered
fpm_env.sh; the staged fpm_exec.sh validates that contract in-pod without
a Collector-side text patch. --fpm-measure-power (default on; never run by
--smoke) requests the per-point power window and publishes power_w; see the
onboarding guide
for its requirements and publication rules. Formal
fpm_forward_perf.parquet output uses metadata schema 7 for latency-only pairs
and schema 8 for power-window pairs, and records
collector_attempt_id, runtime_run_id, runtime_grid_digest, and
kv_seed_regime on every row. The seed regime is real_kv, fake_fallback,
an approved skip:<reason>, legacy, or n/a; non-benign warm-up skips
remain fake_fallback rather than acceptable evidence. For a cache layout that
config.json cannot describe, a cached prefill needs real prefix seeding.
Planning refuses DYN_BENCH_PREFILL_REAL_SEED set to off, 0 or false, and
publication refuses a cached prefill without the native real_prefix regime.
Decode rows keep the classification above. Republishing the same
attempt is idempotent. If a cell already exists under a different attempt or
runtime identity, first publisher wins: that entire cell is skipped while
non-overlapping cells in the same publication still land.
The default publication gate requires every frozen cell to pass.
--fpm-publish-partial explicitly publishes passed cells, records the exact
missing_cells in the checkpoint, and still returns a nonzero incomplete-run
result. The database checkpoint also records
skipped_first_publisher_wins. A validated completed schema-v6 database is
terminal on resume even if raw cell artifacts have since been pruned.
run-manifest.json schema v2 is invocation-scoped: its attempts include the
attempt ID, timestamps, final status, and partial phase timings for failed
attempts; a no-op resume writes no historical attempts.
There is no declarative expected-failure layer. A case that cannot run on the
current framework version fails fast and is recorded in errors_<module>.json
and collection_summary_<backend>.json with a classification and a
(model, dtype) group label. The summary aggregates failures by group: a whole
group failing is a fix-me signal (collector bug, unverified combo, framework
gap), not something to tolerate or auto-skip. Missing points are tolerated
downstream by interpolation and fallback. Escalation rules — when a failure
deserves a skips.py entry or a registry unverified marker — live in
.claude/rules/collector/failure_handling.md.
--gpu <type> resolves the SM version from
src/aisim_core/systems/<gpu>.yaml; use --sm <version> on an
unregistered GPU. Without either, the local device capability is detected
automatically.
To add a new architecture, create one cases/models/<architecture>_cases.yaml
file. To add a new model in an existing architecture, add the model path to
that architecture's model_paths list. Add shared op sweeps to the matching
cases/base_ops/<op>.yaml file. Add a new op collector only when the existing
ops cannot generate the needed data points.
Each backend (trtllm, vllm, sglang) has a registry (registry.py) that maps ops to collector modules, and a version resolver (version_resolver.py) that picks the right module at runtime. Individual collector files declare their compatibility via __compat__. The current collector framework versions and runtime images are declared in framework_manifest.yaml.
framework_manifest.yaml — current collector framework versions and images (schema v2)
framework_manifest.py — manifest loader/validator/resolver
op_catalog.py — table→family identity map (op_backend_catalog.yaml)
model_cases.py — model-centric model/SM case-plan resolver
registry.py — declares which module handles which version range
version_resolver.py — routes runtime version → module (packaging.version)
collect.py/collect_ops — validates __compat__ and fails incompatible ops
__compat__ — per-file metadata declaring supported framework versions
cases/ — model-centric case manifests and SM exceptions
wideep/ — WideEP collector namespace for special images/runtimes
wideep/*/registry.py — WideEP-only ops appended when the v2 plan requests them
network/ — collective communication collectors and Slurm network jobs
framework_manifest.yaml is schema_version: 2: each framework entry has a
default runtime (version + images) plus an optional families: map of
per-family overrides. Resolution for a given (framework, op) walks
table → catalog family → families[family] or default — exactly one runtime
per op, always. families: overrides require the op catalog
(op_backend_catalog.yaml); without it, family-scoped pins fail closed rather
than silently falling back to default.
WideEP runtimes (wideep_sglang, wideep_trtllm) are flattened into peer
framework entries rather than nested under a wideep: key, so the same
per-op resolution logic applies uniformly. Each WideEP entry adds
base_framework (the stock framework it forks from), data_backend (which
table namespace its output is written under), and collector_dir.
collect.py requires the exact public version for a run and rejects
mixed-pin selections across ops.
Image digests are policy, not decoration: the rule is syntactic — any image
reference containing / must be pinned with @sha256:<64 hex>; only bare
internal image names (no /, e.g. deepseek-v4-blackwell) are exempt. In
practice that means every public-registry reference is digest-pinned.
When bumping a version pin, re-fetch and update the digest for every image
variant touched — a stale digest silently keeps pulling the old image. Always
pin the manifest-list (index) digest, never a platform-child digest —
docker buildx imagetools inspect <image> prints the index digest; a child
digest pins one architecture and breaks the other platforms.
Use collector.framework_manifest.resolve_op_runtime(framework, op) to
resolve a single op's runtime, and
collector.framework_manifest.validate_resolution() to check the whole
manifest against every backend registry at once — it returns a list of error
strings (empty = valid) and is the fail-closed CI gate asserting every
registered op resolves to exactly one pinned runtime.
- No version fork:
collect_{op}.py(no suffix) - Version forks:
collect_{op}_v1.py,collect_{op}_v2.py, ... (sequential) - The number is just an ordering key; the actual version range lives in
__compat__
Every collector file must declare __compat__ after the license header:
# SPDX-FileCopyrightText: ...
# SPDX-License-Identifier: Apache-2.0
__compat__ = "trtllm>=1.1.0"The format is <framework><constraints> using PEP 440-style operators:
"trtllm>=1.1.0"— open-ended (latest version)"trtllm>=0.21.0,<1.1.0"— bounded range"sglang>=0.5.5"— sglang example
Pre-release ordering is respected: 1.1.0rc2 < 1.1.0 < 1.1.0.post1.
Each entry in registry.py is an OpEntry dataclass (defined in collector/registry_types.py).
Exactly one of module (unversioned) or versions (versioned) must be provided — this is
validated at construction time.
from collector.registry_types import OpEntry, VersionRoute
# Unversioned (no fork):
OpEntry(op="gemm", module="collector.trtllm.collect_gemm", get_func="...", run_func="...")
# Versioned (has forks) — VersionRoutes in descending min_version order:
OpEntry(op="myop", get_func="...", run_func="...", versions=(
VersionRoute("X.Y.Z", "collector.<backend>.collect_myop_v2"),
VersionRoute("0.0.0", "collector.<backend>.collect_myop_v1"),
))The resolver picks the first VersionRoute where min_version <= runtime_version.
- Create
collector/<backend>/collect_myop.py - Add
__compat__ = "<backend>>=X.Y.Z"after the license header - Export
get_myop_test_cases()andrun_myop()(orrun_myop_torch()) - Add an entry to
collector/<backend>/registry.py:OpEntry(op="myop", module="collector.<backend>.collect_myop", get_func="get_myop_test_cases", run_func="run_myop")
- Run
pytest tests/unit/collector/ -m unitto verify registry integrity
When upstream framework X.Y.Z changes an API that a collector depends on:
- Rename the original file to
collect_{op}_v1.py(if not already versioned) - Create
collect_{op}_v2.pywith the new API calls - Add
__compat__ = "<backend>>=X.Y.Z"to the new file - Add
__compat__upper bound to the old file if it doesn't have one (e.g. change">=1.1.0"to">=1.1.0,<X.Y.Z") - Convert the registry entry from unversioned to versioned (or prepend a new
VersionRoute):# Before (unversioned): OpEntry(op="gemm", module="collector.trtllm.collect_gemm", ...) # After (versioned — all forks carry explicit _vN suffix): OpEntry(op="gemm", versions=( VersionRoute("X.Y.Z", "collector.<backend>.collect_gemm_v2"), VersionRoute("0.0.0", "collector.<backend>.collect_gemm_v1"), ), ...)
- Run tests to validate
At collection time:
collect.pyreads the backend registrybuild_collections()resolves each op to the correct module for the detected framework version- If the resolved module declares
__compat__, it is validated against the runtime version — mismatches fail that op explicitly and are recorded in the error summary - Unsupported versions (no matching entry) are skipped with a warning
pytest tests/unit/collector/ -m unitCovers: version parsing (PEP 440 with rc/post), __compat__ constraint evaluation, module routing, and structural validation of all three registries.
Use --smoke to quickly verify the collector runs end-to-end. It randomly samples 4 test cases per op instead of running the full suite.
# Smoke test for sglang
python3 collect.py --backend sglang --smoke
# Smoke a specific op
python3 collect.py --backend trtllm --ops moe --smokeThe collector supports GPU power monitoring during kernel execution using NVML. This feature is optional and disabled by default.
# Basic power monitoring
python3 collect.py --backend trtllm --measure_power
# With custom minimum duration (default: 1.0s)
python3 collect.py --backend trtllm --measure_power --power_test_duration_sec 2.0--measure_power: Enable NVML-based power monitoring (samples at 100ms intervals)--power_test_duration_sec: Minimum test duration for accurate power readings (default: 1.0s)
When power monitoring is enabled, performance CSV files will include additional columns:
power: Average power consumption during kernel execution (Watts)power_limit: GPU power management limit (Watts)
Example output:
framework,version,device,op_name,kernel_source,gemm_dtype,m,n,k,latency,power,power_limit
TRTLLM,1.2.0,NVIDIA H200 SXM,gemm,torch_flow,bfloat16,1024,4096,4096,0.234,523.4,700.0Power monitoring requires:
pynvmlPython package:pip install pynvml- NVML support (NVIDIA drivers)
If unavailable, a warning is logged and execution continues without power data.
- Power monitoring adds minimal overhead (<1%)
- Kernel iterations are automatically adjusted to meet minimum duration for accurate measurements
- Backward compatible: without
--measure_power, CSVs remain unchanged
benchmark_with_power times kernels under CUDA-graph replay. A failed capture
raises (allow_graph_fail=False, the default) and the case fails classified.
- No collector that writes a perf table may pass
allow_graph_fail=True. The eager fallback times the Python launch loop and ships that floor as the kernel latency, and no perf table records the helper'sused_cuda_graphflag —.claude/rules/collector/layer_permissions.mdallows a measurement-method degradation only when it is recorded in the output row.tests/unit/collector/test_no_eager_graph_fallback_sites.pypins this for every file undercollector/. The last fallback sites (trtllm mamba2 and moe, wideep moe_compute, network trtllm alltoall) were closed; the sglang trtllm_mla prefill (B1) and the trtllm GDN lanes the day before. - A collector that sweeps several cells inside one task counts failed cells
and raises a strict-completeness error at the end (GDN, mamba2), so the
other cells still ship rows and the failure lands in
errors_<module>.json. - Multi-rank collectors agree on the capture outcome before any rank replays
(
collect_trtllm_alltoall._agree_graph_capturable); a one-sided capture failure would otherwise hang the peers' first replay. allow_graph_fail=Trueremains available for ad-hoc diagnosis only; the result then carriesused_cuda_graph=Falseand must not be persisted.
from helper import benchmark_with_power
def my_kernel():
# Your kernel code here
moe.forward(hidden_states, logits)
with benchmark_with_power(
device=device,
kernel_func=my_kernel,
num_warmups=3,
num_runs=6,
repeat_n=1,
) as results:
latency = results["latency_ms"]
power_stats = results["power_stats"]To collect performance data for mxfp4 kernels used in GPTOSS models, depending on the version of TensorRT-LLM, you might need to manually install triton-kernels. If your version is >= 1.3.0rc2, nothing needs to be done and you can run the collection process directly. For any version before 1.3.0rc2. Follow the below instructions to install triton-kernels and expose them to TensorRT-LLM
- Build and install Triton (tested with the commit below):
git clone https://github.com/triton-lang/triton.git
cd triton
# Specific commit verified with TensorRT-LLM
git checkout f3067cd3bd0c29065fa4ecdb724b6f29cbabea5f
pip install -r python/requirements.txt # build-time dependencies
pip install wheel build
python3 setup.py bdist_wheel # if this step fails due cuda related error, you might need to set CUDA_HOME following the optional step
pip install ./dist/*.whl- (Optional) You may need to set CUDA_HOME env variable to successfully build triton
export CUDA_HOME=/usr/local/cuda
export CPLUS_INCLUDE_PATH=$CUDA_HOME/include:$CPLUS_INCLUDE_PATH
export C_INCLUDE_PATH=$CUDA_HOME/include:$C_INCLUDE_PATH
python3 setup.py bdist_wheel- Expose the Triton kernels to TensorRT-LLM The kernels are not packaged in the wheel, so set the environment variable TRITON_ROOT to your Triton clone:
export TRITON_ROOT=/local/user/triton
# TensorRT-LLM expects the kernels at:
# $TRITON_ROOT/python/triton_kernelspython3 collect.py --backend trtllmFor trtllm, the whole collecting process takes about 30 gpu-hours. On 8-gpu, it takes 3-4 hours. Please note that the whole process will report a lot of missing datapoints with errors. But it's okay. Our system is kindof robust to fair amount of missing data. Once everything is done, you might see mutliple xxx.txt files under the same folder. Refer to src/aisim_core/systems/ folder to prepare the database including how many files are needed accordingly.
Checkpoint files are always written so an interrupted run can be resumed later:
# Normal run (writes checkpoint to .collector_checkpoint/)
python3 collect.py --backend trtllm
# Resume an interrupted run (skips already-attempted tasks)
python3 collect.py --backend trtllm --resume
# Custom checkpoint directory
python3 collect.py --backend trtllm --resume --checkpoint-dir /path/to/checkpointsA task is marked done once it is attempted (success or failure).
Only tasks that never finished are re-queued on --resume;
--resume-retry-failed additionally re-runs previously failed tasks.
Running without --resume always starts fresh (overwrites old checkpoint).
Use the pinned stock image, lmsysorg/sglang:v0.5.14 (or its -cu130
variant), for normal SGLang collection.
python3 collect.py --backend sglangThis collects the stock SGLang ops, including:
- GEMM operations (FP8, BF16, INT8, INT4)
- MLA (Multi-head Latent Attention) for context and generation
- MLA BMM (Batch Matrix Multiplication) operations
- MoE (Mixture of Experts) operations
- Normal attention operations
Large-EP local expert compute is available as the separately routed moe_ep
operator under wideep/sglang/collect_deepep_moe.py; run it explicitly in its
pinned SGLang 0.5.10 image. WideEP MLA is not registered because its legacy
wrapper now reaches the stock 0.5.14-only module implementation.
For models with DeepEP MoE, inter-node communication data requires a separate
multi-node setup. It is collected by the standalone
wideep/sglang/collect_moe_a2a.py collector, launched one Slurm job per
world size by network/slurm/submit_moe_a2a.sh — see section 4 of
network/slurm/README.md. The older manual log-scraping pipeline under
wideep/sglang/deepep/ is deprecated (its README carries the retirement
story); do not add new data through it.
Large-EP MoE uses stock moe_perf for modeled local expert compute and
moe_a2a_perf for measured dispatch/combine communication. To support model X:
-
Declare the shapes. Add the model to
cases/models/<Architecture>_cases.yaml(new architecture: one new file; new model in an existing architecture: append tomodel_paths) and mark itsmodel_case_values.moerows withwideep: true. Correlated MoE dimensions (hidden/inter size, topk, expert count) stay together in one row; never cross another model's values. Thewideepflag activates standalone A2A shape population. -
Ensure stock
moe_perfcoverage. Local compute is modeled as a uniformly balanced stream over EP-local experts, using TP=1 stock MoE coordinates. Cross-version stock-table reuse is allowed. -
Collect
moe_a2aacross nodes. Fromnetwork/slurm/:bash submit_moe_a2a.sh # sglang DeepEP HT+LL, one job per world size bash submit_trtllm_alltoall.sh # trtllm NVLink alltoall (single/multi-GPU NVL)
The comm tables have no model column — a new model whose (hidden_size, topk, num_experts) tuple is already covered needs no new runs; a new physical shape lands in the sweep automatically once declared in step 1.
-
Publish with sidecars. Finalized communication parquet goes into
.../<system>/comm/<backend>/<version>/moe_a2a_perf.parquettogether with itscollection_meta.yamlentry — never a parquet without its provenance. The per-worldmoe_a2aoutputs need the cross-job merge procedure innetwork/slurm/README.mdsection 4.3.
moe_a2a_perf records latency in microseconds; load_moe_a2a_data converts
leaves to milliseconds. Stock moe_perf retains its existing timing contract.
Stock moe_perf also supports optional Boolean default_eligible selection metadata. The Core API contract defines automatic and exact-source selection. Finalization validates non-null Boolean flags and the named source required by false, excludes the flag from measurement identity, and preserves existing annotations when merging a legacy collection that omits the column. Collectors do not infer eligibility from kernel labels or row order.
Legacy-overwrite caveats. A new-schema row replaces a legacy-adapted leaf only at the same key, and the legacy adapters derive their node/EP geometry rather than reading it:
- Legacy sglang DeepEP comm tables carry no
ep_size; the adapter assumesep_size = node_num * 8(8-GPU HGX fleets). Newmoe_a2aworlds therefore overwrite those leaves only when collected at 8 GPUs/node — a 4-GPU/node (GB200) world with the same node count keys differently and coexists with the legacy data instead of replacing it. - The legacy trtllm alltoall table (GB200 NVL4) carries no
num_nodescolumn; the adapter derivesnode_num = max(1, moe_ep_size // 4). New rows overwrite only at 4 GPUs/node; a row with an explicitnum_nodescolumn is honored as written.
log_perf freezes the CSV header from the first row, so optional
columns are all-or-nothing per file. In particular, power columns exist in a
staging CSV only if the very first logged row had them: flipping
--measure_power across a --resume of the same staging file appends rows
whose fields no longer match the frozen header — misaligned columns or empty
cells that fail at load time. Collect any one file with a single power
setting end to end. This property is also why the moe_a2a collector ships
no power column at all (owner ruling during PR review): its low-latency
timing covers one round-trip rather than a per-phase region, and meaningful
per-phase power would need winning-config re-runs — a measurement-method
design for hardware, not a column to fake. The loaders treat absent (or
null) power as 0.0.
Rebuild and install the new aiconfigurator. Please make sure you have your new system definition file prepared. It's src/aisim_core/systems/xxx.yaml
Today, we have limited method to validate the database. You can try tools/sanity_check to validate the database a little bit. But it highly depends on your understanding of the GPU system and kernel optimization.
Symptom: Collection stalls after a few test cases with no error messages.
Cause: fcntl.flock() doesn't work reliably on NFS. Workers deadlock when writing to shared output files.
Solution: Use /tmp/ for output files, then copy results after collection.
Browse the Legacy AIC Support Matrix for CLI compatibility coverage, or inspect the per-system CSV files. For strict-native estimator coverage, use the FPE Support Matrix.
For uniform-projection checkpoints, run_attention_torch(..., ordinary_mla=True)
collects the complete context attention module into
mla_context_module_perf.txt. It invokes SGLang's real prepare_qkv_latent
and creates fresh AttentionInputs on every warmup and timed iteration, so
Q/KV down-projection is included. Prefill is timed eagerly. The loaded projection
weights must match the requested single gemm_type; mixed BF16/NVFP4 modules
are rejected instead of mislabeled.
The CLI flag --ordinary-mla requires --mode context --attn-type mla.
Other combinations fail during argument parsing, before collection starts.
If any dispatch fails, the CLI reports its traceback, finishes the remaining
dispatches, and exits with status 1 so partial collection is not reported as success.
Pass --target-tp-size 4 (or 8) to derive the per-rank head count from the
model. It must be positive and divide the model's native head count. If
--num-heads is also supplied, it must equal native heads divided by TP.
Without an explicit TP size, ordinary MLA derives TP separately for each
per-rank head count. Python callers pass the actual per-rank head count and
target_tp_size. Set
--chunked-prefill-size 16384 on the CLI, or chunked_prefill_size=16384
on run_mla_module() or run_mla_module_worker(), to reproduce a serving
chunk limit of 16,384 tokens. The worker forwards the limit into its subprocess.
Explicit limits must be positive for one-shot module collection.
Omitting the option keeps SGLang's default. The existing
wide-EP and DSA paths keep their separate behavior. The ordinary context table
uses the existing consumer schema; this does not add mixed-precision modeling.
TRT-LLM MoE collection requires either a loaded tactic cache or at least one successful autotuning attempt before measuring a tuning-enabled case. Ordinary failures may try smaller eligible token counts; if every attempt fails, the case raises before timing or publishing rows. CUDA accelerator errors propagate immediately. The successful collection path and historical profile values are unchanged by this failure guard.
These are provisional planning estimates, in additional collection GPU-hours
above the pinned #676 baseline. They price the extra model-owned points enabled
by --include-long-context; the existing baseline anchors remain enabled when
the flag is off. They are not measured full-sweep totals or confidence bounds.
The table reflects the frozen pilot inventory; the later MiniMax MXFP8 alias
coverage correction adds projection-lane points that remain unpriced here.
The estimates use each sampled SKU's own framework measurements and the frozen added-case inventory. Rounded planning centers across the three frameworks are:
| GPU SKU | vLLM GPUh | SGLang GPUh | TRT-LLM GPUh | Long-context total GPUh |
|---|---|---|---|---|
| L40S | 2.0 | 2.6 | 3.4 | ~8 |
| H100 SXM | 4.4 | 2.7 | 1.9 | ~9 |
| H200 SXM | 3.9 | 10.2 | 1.7 | ~16 |
| B200 SXM | 2.6 | 18.3 | 5.5 | ~26 |
| B300 SXM | 2.2 | 18.7 | 1.3 | ~22 |
| GB200 | 3.5 | 9.5 | 4.5 | ~18 |
| GB300 | 5.6 | 13.9 | 10.6 | ~30 |
| A100 SXM80 | Unmeasured | Unmeasured | Unmeasured | Unmeasured |
The seven sampled SKUs total approximately 130 GPU-hours for the modeled long-context work. Rows include explicit analogous-operation provisions where native measurements are missing; 1,628 additional history cells still lack a same-framework analogue. The table does not establish that every planned shape fits memory or has a working native kernel.
Preparation scales with cache/input payload and execution with attention work, including the sliding-window cap. Precision, geometry and dispatch extrapolations remain assumptions. B300 uses the intended GLM NVFP4 collection recipe and its own measurements; these timings do not establish canonical BF16 startup or capacity equivalence. Other SKUs do not borrow B300 rates.
Inclusive point clocks contain input/history preparation, warmup, compilation, graph capture and forwards. Forward latency is not added again. Existing worker initialization is paid once when its group is extended; larger buffers and shape-dependent setup can still add cost. Scheduler wait, GPU allocation launch and idle reservation are excluded. Operator tests use one physical GPU even when their geometry represents a TP rank; actual collective ranks are counted for NCCL.
The independent MoE token, KDA width and NCCL payload changes add a separate approximately 55 GPU-hours of planning proxies across the seven sampled SKUs. They are outside the history flag and are not part of the 130-hour history total. Together, the modeled expansion subtotal is approximately 185 GPU-hours.
A provisional eight-SKU working budget of 300 GPU-hours adds selected reserves of 50 GPU-hours for unmeasured A100 work and 65 GPU-hours for remaining cases, setup changes and retries. A larger engineering allowance is 750 GPU-hours total, not 750 additional hours. These reserves are budget choices, not measured rates, statistical intervals or guarantees of successful coverage. No RTX PRO estimate is included in this eight-SKU budget.
The supporting planning calculations, unrounded per-framework values, case
counts, native/proxy distinctions, source hashes and scratch calculations are
retained as local campaign artifacts outside this PR, together with the six
collection-cost audit scripts and their tests. These audit-only scripts are not
shipped under tools/perf_database/. The separate strict audit
keeps unqualified full totals unknown; these planning figures do not change its
validation results. Deferred native Kimi coverage is not priced as zero or counted
as equivalent-coverage savings.