Skip to content

Latest commit

 

History

History
824 lines (675 loc) · 41 KB

README.md

File metadata and controls

824 lines (675 loc) · 41 KB

Introduction

Data collection is a standalone process for collecting the database for aiconfigurator. By default, you don't have to collect the data by yourself. Small versions of database will not introduce huge perf difference. Say, you can use 1.0.0rc3 data of trtllm on h200_sxm and deploy the generated configs with Dynamo + trtllm 1.0.0rc4 worker.

If you want to go through the process, you can try belowing commands. However, you need to prepare the env by yourself such as installing a specific trtllm version. This process is not well verified, you need to debug sometimes.

For a framework-version or GPU-platform upgrade, follow the Collector Upgrade Playbook. The repo-tracked aic-auto-collect skill applies that workflow during long, resumable collection runs. SGLang 0.5.14 Hopper/Blackwell follow-up work must also consult the platform alignment ledger, a local campaign record kept outside the repo (docs/perf_database/sglang-0.5.14-hopper-blackwell-ledger.md in the campaign workspace).

Optional model-owned long-context coverage

Normal collection retains the shared grid and model geometry coverage. Add --include-long-context to collect the additional correlated points declared on selected model rows; it defaults to off. A model selection or --model-cases-full is required. Existing long-context points in the normal shared module grid remain enabled.

python collect.py --backend sglang --model-path zai-org/GLM-5.2 --ops dsa_generation_module --include-long-context
python collect.py --backend sglang --model-cases-full --sm 103 --plan-only --include-long-context

The optional points reuse the native geometry, actual projection precision and existing framework dispatch. Model context limits validate declarations; they do not automatically request every length or batch. Workers receive explicit points and retain grouped setup. No environment switch changes their rows. --plan-only reports the selected mode and hashes its actual case list. Use separate checkpoint/output roots for before/after cost experiments; changing a group's point list changes its existing task ID and can repeat its older points on resume, while preventing newly added points from being skipped.

See the declaration fields and coverage rationale. Collection-cost estimates count setup changes, point preparation, warmup/capture and timed forwards; scheduler wait and GPU allocation overhead are excluded.

Kimi-K3 adds no optional MLA long-context points yet: pinned vLLM 0.30.0 MultiHeadLatentAttention uses NoPE and fused cache operations that the existing generic DeepSeek module does not reproduce. A native adapter needs cache registration/binding, absorbed weight processing, forward/metadata and FP8 policy support, plus a consumer review of its output-gate timing boundary and silicon qualification. See the deferred extension. Normal coverage and independent KDA width additions remain enabled; the missing native long-context route is an unmeasured gap, not zero collection cost.

Preparation

SGLang 0.5.14 standalone MLA rows distinguish kernel compute from KV storage. On B200/B300, trtllm_mla quantizes Q/K/V internally when the KV cache is FP8, so those rows use mla_dtype=fp8, kv_cache_dtype=fp8. On H200, the absorbed 576-dimensional FA3 path uses BF16 compute, including when KV storage is FP8. BF16 inputs to the collector do not determine the kernel's compute precision. Compute-precision labels are restricted to the audited trtllm_mla, fa3, and triton backends; an unknown backend raises instead of receiving a BF16 label. Legacy Blackwell rows labeled bfloat16/fp8 need fresh measurements with the correct label; do not copy their timings into a second precision key.

Before collecting the data, make sure you own the whole node and no interfierence happens. Next, please enable persistent-mode and lock frequency of the node. Make sure the cooling system of the node is working well.

sudo nvidia-smi -pm 1
sudo nvidia-smi -ac yyy,xxx

xxx, yyy frequency can be queried by nvidia-smi -q -i 0, refer to the Max Clocks part, xxx is SM frequency, yyy is Memory frequency. A script to set frequency:

#!/bin/bash

# Run nvidia-smi query and extract SM and Memory frequencies from Max Clocks
sm_freq=$(nvidia-smi -q -i 0 | grep -A 4 "Max Clocks" | grep "SM " | grep -o "[0-9]\+ MHz" | grep -o "[0-9]\+")
mem_freq=$(nvidia-smi -q -i 0 | grep -A 4 "Max Clocks" | grep "Memory " | grep -o "[0-9]\+ MHz" | grep -o "[0-9]\+")

# Check if frequencies were successfully extracted
if [ -z "$sm_freq" ] || [ -z "$mem_freq" ]; then
    echo "Error: Could not extract SM or Memory frequency from Max Clocks."
    exit 1
fi

# Generate the command
echo "sudo nvidia-smi -ac $mem_freq,$sm_freq"

Prepare a clean env with the target framework and nccl lib installed.

Collect comm data

export PATH=$PATH:${NCCL_TEST_BIN_PATH}/
network/collect_comm.sh #all_reduce data will be collected using default trtllm backend
network/collect_comm.sh --all_reduce_backend vllm #all_reduce data will be collected using vllm backend
network/collect_comm.sh --all_reduce_backend vllm --device xpu #all_reduce data will be collected using vllm backend on XPU

Today we only collect intra-node comm. This script will collect custom allreduce data for trtllm within a node. It will also collect nccl allreduce, all_gather, all2all, reduce_scatter using nccl. The standalone scripts stage nccl_perf.txt, oneccl_perf.txt, and custom_allreduce_perf.txt. Collector finalization converts accepted output to parquet under python/aisim/src/aisim_core/systems/data/<system>/comm/<backend>/<version>/.

Model-centric cases and healing runs

TRT-LLM MoE workers refuse a case before module allocation when its local resident expert weights alone exceed live device capacity. This lower bound accounts for gated/non-gated matrices and TP/EP sharding; passing it does not establish room for scales, inputs, outputs, graph buffers or native workspace. Those later allocation failures remain recorded failures rather than hidden shape exclusions. Plans remain independent of device memory.

The model-centric case layer (introduced as "collector v2") is how collection coverage is declared, and it is unchanged under Collector V3 — V3 adds op-centric management on top (per-family runtime pins, provenance sidecars, declared reuse), it does not replace how cases are declared or healed. A healing run collects cases for a specific model/GPU pair instead of running every op bucket and hoping the support matrix improves. Use --model-cases-full when you want the case YAML to define a full model-centric run. Omitting all model-case flags runs the backend registry directly without a model-centric case plan.

# Heal one model on one GPU type.
python3 collect.py --backend sglang \
  --model-path sgl-project/DeepSeek-V4-Flash-FP8 \
  --gpu b200_sxm

# Inspect the resolved model plan without collecting.
python3 collect.py --backend sglang \
  --model-path sgl-project/DeepSeek-V4-Flash-FP8 \
  --gpu b200_sxm \
  --plan-only

# Select the architecture case file directly.
python3 collect.py --backend trtllm \
  --model-architecture Qwen3MoeForCausalLM \
  --gpu b200_sxm \
  --plan-only

# Full model-centric run: aggregate base ops plus every model case YAML file.
python3 collect.py --backend trtllm --model-cases-full

# Raw backend registry run: no model-centric case plan.
python3 collect.py --backend trtllm

# Ephemeral subset filter (healing): substring match, repeatable.
python3 collect.py --backend trtllm --ops moe --case-filter "tp=4"

Case files (see cases/README.md for the full authoring guide):

cases/base_ops/<op>.yaml               — shared sweep recipes (interpolation grid)
cases/models/<architecture>_cases.yaml — op activation + model shape values
model_cases.py                         — resolves which ops a model plan collects
skips.py                               — the one list of cases a collection leaves out

The plan is one equation: cases = dedup(base grid ∪ model shapes), minus what skips.py skips. For an op with an op plan (OP_PLANS in op_plans.py), --plan-only with --sm or --gpu also prints case counts per dtype and what was left out under which entry; --plan-table DIR writes one CSV row per case.

TRT-LLM MLA precision and GPT-OSS TP profile collection evidence is recorded in the September 17 campaign.

Whole-forward FPM campaign

Use the dedicated python3 -m collector.fpm_forward entry point for vLLM whole-forward campaigns. The compatibility route python3 collector/collect.py --ops fpm_forward remains supported and must be selected alone. Both routes require a resolved model, --gpu, and --fpm-max-gpus. The campaign derives a latency-blind prefill/decode design from the model's AIC attention cases, asks Generator for three rendered artifacts per cell — a keepalive Pod, LeaderWorkerSet, or Grove PodCliqueSet manifest plus fpm_env.sh and a thin engine-launch run.sh — stages them with its own in-pod runtime fpm_exec.sh (etcd lifecycle, result gate and checker, completion barrier, exit-code policy), and publishes the formal table only after every frozen cell passes. For multinode runs, select the installed controller with --fpm-orchestrator lws|grove. Start with --plan-only or a one-cell --smoke --limit 1 run.

python3 -m collector.fpm_forward \
  --model-path nvidia/GLM-5.2-NVFP4 \
  --gpu b200_sxm \
  --fpm-max-gpus 8 \
  --plan-only

For profile-based FPM collection, the profile's backend_version is the dataset identity label. It may be a custom string such as my-vllm-patch-3 and need not match the installed vLLM package version. The collector retains the actual container version separately in collector-provenance.json under runtime.backend_version; runtime capability, benchmark-schema, topology and precision checks still apply. An optional Dynamo version is provenance only and does not select FPM templates or resolve the backend version. See the onboarding version contract for guided setup, omitted-version detection inside the target image, and publication identity.

The low-level entry point also accepts --fpm-backend-version LABEL for dataset identity and --fpm-runtime-backend-version VERSION for an independently observed package-version pin. Guided onboarding supplies that pin after its image preflight. The low-level collector does not launch this separate CPU preflight: without an observed pin it can collect timings and record actual package provenance, but it cannot select audited version-specific memory/execution hooks. Supply the observed pin or accepted runtime instrumentation when those hooks are required. The op-level collector's runtime manifest and version-routing rules below are unchanged.

Generation-time admission may omit a topology only when a concrete AIC size-vs-capacity estimate proves its configured token envelope cannot fit. Missing performance data, bootstrap architectures, and structural estimator errors remain runnable so the target runtime supplies the observation.

Model identity is resolved from a local checkpoint/config, the packaged AIC config cache, or Hugging Face. A real model architecture that is not registered by AIC uses a config-derived bootstrap template; an unresolved or empty config fails before planning. When the checkpoint exists only inside the runtime Pod, pass its real local metadata explicitly with --fpm-model-config /path/to/config.json (a directory containing config.json is also accepted). The resolved config snapshot and SHA-256 are frozen into the collection plan.

The collector detects a cache layout that config.json cannot describe, such as linear-attention or recurrent state, with the same check as config intake. The onboarding guide lists the markers. No FPM attention template represents such a layout. Without --fpm-model-profile or --fpm-backend-version, the collector refuses it, and registered model cases do not change this. With one of these options, the collector gives the layout the native_runtime capability. Precision then comes from the checkpoint, and the runtime measures every cell.

Generator and Collector share the accepted Dynamo-native benchmark result schema versions (2, and 3 with a power window) through the rendered fpm_env.sh; the staged fpm_exec.sh validates that contract in-pod without a Collector-side text patch. --fpm-measure-power (default on; never run by --smoke) requests the per-point power window and publishes power_w; see the onboarding guide for its requirements and publication rules. Formal fpm_forward_perf.parquet output uses metadata schema 7 for latency-only pairs and schema 8 for power-window pairs, and records collector_attempt_id, runtime_run_id, runtime_grid_digest, and kv_seed_regime on every row. The seed regime is real_kv, fake_fallback, an approved skip:<reason>, legacy, or n/a; non-benign warm-up skips remain fake_fallback rather than acceptable evidence. For a cache layout that config.json cannot describe, a cached prefill needs real prefix seeding. Planning refuses DYN_BENCH_PREFILL_REAL_SEED set to off, 0 or false, and publication refuses a cached prefill without the native real_prefix regime. Decode rows keep the classification above. Republishing the same attempt is idempotent. If a cell already exists under a different attempt or runtime identity, first publisher wins: that entire cell is skipped while non-overlapping cells in the same publication still land.

The default publication gate requires every frozen cell to pass. --fpm-publish-partial explicitly publishes passed cells, records the exact missing_cells in the checkpoint, and still returns a nonzero incomplete-run result. The database checkpoint also records skipped_first_publisher_wins. A validated completed schema-v6 database is terminal on resume even if raw cell artifacts have since been pruned. run-manifest.json schema v2 is invocation-scoped: its attempts include the attempt ID, timestamps, final status, and partial phase timings for failed attempts; a no-op resume writes no historical attempts.

Failure philosophy: observe, don't predict

There is no declarative expected-failure layer. A case that cannot run on the current framework version fails fast and is recorded in errors_<module>.json and collection_summary_<backend>.json with a classification and a (model, dtype) group label. The summary aggregates failures by group: a whole group failing is a fix-me signal (collector bug, unverified combo, framework gap), not something to tolerate or auto-skip. Missing points are tolerated downstream by interpolation and fallback. Escalation rules — when a failure deserves a skips.py entry or a registry unverified marker — live in .claude/rules/collector/failure_handling.md.

--gpu <type> resolves the SM version from src/aisim_core/systems/<gpu>.yaml; use --sm <version> on an unregistered GPU. Without either, the local device capability is detected automatically.

To add a new architecture, create one cases/models/<architecture>_cases.yaml file. To add a new model in an existing architecture, add the model path to that architecture's model_paths list. Add shared op sweeps to the matching cases/base_ops/<op>.yaml file. Add a new op collector only when the existing ops cannot generate the needed data points.

Version Management

Overview

Each backend (trtllm, vllm, sglang) has a registry (registry.py) that maps ops to collector modules, and a version resolver (version_resolver.py) that picks the right module at runtime. Individual collector files declare their compatibility via __compat__. The current collector framework versions and runtime images are declared in framework_manifest.yaml.

framework_manifest.yaml — current collector framework versions and images (schema v2)
framework_manifest.py   — manifest loader/validator/resolver
op_catalog.py         — table→family identity map (op_backend_catalog.yaml)
model_cases.py       — model-centric model/SM case-plan resolver
registry.py          — declares which module handles which version range
version_resolver.py  — routes runtime version → module (packaging.version)
collect.py/collect_ops — validates __compat__ and fails incompatible ops
__compat__           — per-file metadata declaring supported framework versions
cases/               — model-centric case manifests and SM exceptions
wideep/              — WideEP collector namespace for special images/runtimes
wideep/*/registry.py — WideEP-only ops appended when the v2 plan requests them
network/             — collective communication collectors and Slurm network jobs

framework_manifest.yaml is schema_version: 2: each framework entry has a default runtime (version + images) plus an optional families: map of per-family overrides. Resolution for a given (framework, op) walks table → catalog family → families[family] or default — exactly one runtime per op, always. families: overrides require the op catalog (op_backend_catalog.yaml); without it, family-scoped pins fail closed rather than silently falling back to default.

WideEP runtimes (wideep_sglang, wideep_trtllm) are flattened into peer framework entries rather than nested under a wideep: key, so the same per-op resolution logic applies uniformly. Each WideEP entry adds base_framework (the stock framework it forks from), data_backend (which table namespace its output is written under), and collector_dir. collect.py requires the exact public version for a run and rejects mixed-pin selections across ops.

Image digests are policy, not decoration: the rule is syntactic — any image reference containing / must be pinned with @sha256:<64 hex>; only bare internal image names (no /, e.g. deepseek-v4-blackwell) are exempt. In practice that means every public-registry reference is digest-pinned. When bumping a version pin, re-fetch and update the digest for every image variant touched — a stale digest silently keeps pulling the old image. Always pin the manifest-list (index) digest, never a platform-child digest — docker buildx imagetools inspect <image> prints the index digest; a child digest pins one architecture and breaks the other platforms.

Use collector.framework_manifest.resolve_op_runtime(framework, op) to resolve a single op's runtime, and collector.framework_manifest.validate_resolution() to check the whole manifest against every backend registry at once — it returns a list of error strings (empty = valid) and is the fail-closed CI gate asserting every registered op resolves to exactly one pinned runtime.

File Naming Convention

  • No version fork: collect_{op}.py (no suffix)
  • Version forks: collect_{op}_v1.py, collect_{op}_v2.py, ... (sequential)
  • The number is just an ordering key; the actual version range lives in __compat__

__compat__ Format

Every collector file must declare __compat__ after the license header:

# SPDX-FileCopyrightText: ...
# SPDX-License-Identifier: Apache-2.0

__compat__ = "trtllm>=1.1.0"

The format is <framework><constraints> using PEP 440-style operators:

  • "trtllm>=1.1.0" — open-ended (latest version)
  • "trtllm>=0.21.0,<1.1.0" — bounded range
  • "sglang>=0.5.5" — sglang example

Pre-release ordering is respected: 1.1.0rc2 < 1.1.0 < 1.1.0.post1.

Registry Format

Each entry in registry.py is an OpEntry dataclass (defined in collector/registry_types.py). Exactly one of module (unversioned) or versions (versioned) must be provided — this is validated at construction time.

from collector.registry_types import OpEntry, VersionRoute

# Unversioned (no fork):
OpEntry(op="gemm", module="collector.trtllm.collect_gemm", get_func="...", run_func="...")

# Versioned (has forks) — VersionRoutes in descending min_version order:
OpEntry(op="myop", get_func="...", run_func="...", versions=(
    VersionRoute("X.Y.Z", "collector.<backend>.collect_myop_v2"),
    VersionRoute("0.0.0", "collector.<backend>.collect_myop_v1"),
))

The resolver picks the first VersionRoute where min_version <= runtime_version.

Adding a New Op

  1. Create collector/<backend>/collect_myop.py
  2. Add __compat__ = "<backend>>=X.Y.Z" after the license header
  3. Export get_myop_test_cases() and run_myop() (or run_myop_torch())
  4. Add an entry to collector/<backend>/registry.py:
    OpEntry(op="myop", module="collector.<backend>.collect_myop", get_func="get_myop_test_cases", run_func="run_myop")
  5. Run pytest tests/unit/collector/ -m unit to verify registry integrity

Handling a Framework API Change

When upstream framework X.Y.Z changes an API that a collector depends on:

  1. Rename the original file to collect_{op}_v1.py (if not already versioned)
  2. Create collect_{op}_v2.py with the new API calls
  3. Add __compat__ = "<backend>>=X.Y.Z" to the new file
  4. Add __compat__ upper bound to the old file if it doesn't have one (e.g. change ">=1.1.0" to ">=1.1.0,<X.Y.Z")
  5. Convert the registry entry from unversioned to versioned (or prepend a new VersionRoute):
    # Before (unversioned):
    OpEntry(op="gemm", module="collector.trtllm.collect_gemm", ...)
    
    # After (versioned — all forks carry explicit _vN suffix):
    OpEntry(op="gemm", versions=(
        VersionRoute("X.Y.Z", "collector.<backend>.collect_gemm_v2"),
        VersionRoute("0.0.0", "collector.<backend>.collect_gemm_v1"),
    ), ...)
  6. Run tests to validate

Runtime Behavior

At collection time:

  1. collect.py reads the backend registry
  2. build_collections() resolves each op to the correct module for the detected framework version
  3. If the resolved module declares __compat__, it is validated against the runtime version — mismatches fail that op explicitly and are recorded in the error summary
  4. Unsupported versions (no matching entry) are skipped with a warning

Tests

pytest tests/unit/collector/ -m unit

Covers: version parsing (PEP 440 with rc/post), __compat__ constraint evaluation, module routing, and structural validation of all three registries.

Collect gemm/attention/moe data/etc.

Smoke Test

Use --smoke to quickly verify the collector runs end-to-end. It randomly samples 4 test cases per op instead of running the full suite.

# Smoke test for sglang
python3 collect.py --backend sglang --smoke

# Smoke a specific op
python3 collect.py --backend trtllm --ops moe --smoke

Power Monitoring (Optional)

The collector supports GPU power monitoring during kernel execution using NVML. This feature is optional and disabled by default.

Enable Power Monitoring

# Basic power monitoring
python3 collect.py --backend trtllm --measure_power

# With custom minimum duration (default: 1.0s)
python3 collect.py --backend trtllm --measure_power --power_test_duration_sec 2.0

Options

  • --measure_power: Enable NVML-based power monitoring (samples at 100ms intervals)
  • --power_test_duration_sec: Minimum test duration for accurate power readings (default: 1.0s)

Output

When power monitoring is enabled, performance CSV files will include additional columns:

  • power: Average power consumption during kernel execution (Watts)
  • power_limit: GPU power management limit (Watts)

Example output:

framework,version,device,op_name,kernel_source,gemm_dtype,m,n,k,latency,power,power_limit
TRTLLM,1.2.0,NVIDIA H200 SXM,gemm,torch_flow,bfloat16,1024,4096,4096,0.234,523.4,700.0

Requirements

Power monitoring requires:

  • pynvml Python package: pip install pynvml
  • NVML support (NVIDIA drivers)

If unavailable, a warning is logged and execution continues without power data.

Notes

  • Power monitoring adds minimal overhead (<1%)
  • Kernel iterations are automatically adjusted to meet minimum duration for accurate measurements
  • Backward compatible: without --measure_power, CSVs remain unchanged

CUDA Graph Timing Policy

benchmark_with_power times kernels under CUDA-graph replay. A failed capture raises (allow_graph_fail=False, the default) and the case fails classified.

  • No collector that writes a perf table may pass allow_graph_fail=True. The eager fallback times the Python launch loop and ships that floor as the kernel latency, and no perf table records the helper's used_cuda_graph flag — .claude/rules/collector/layer_permissions.md allows a measurement-method degradation only when it is recorded in the output row. tests/unit/collector/test_no_eager_graph_fallback_sites.py pins this for every file under collector/. The last fallback sites (trtllm mamba2 and moe, wideep moe_compute, network trtllm alltoall) were closed; the sglang trtllm_mla prefill (B1) and the trtllm GDN lanes the day before.
  • A collector that sweeps several cells inside one task counts failed cells and raises a strict-completeness error at the end (GDN, mamba2), so the other cells still ship rows and the failure lands in errors_<module>.json.
  • Multi-rank collectors agree on the capture outcome before any rank replays (collect_trtllm_alltoall._agree_graph_capturable); a one-sided capture failure would otherwise hang the peers' first replay.
  • allow_graph_fail=True remains available for ad-hoc diagnosis only; the result then carries used_cuda_graph=False and must not be persisted.

Usage Example

from helper import benchmark_with_power

def my_kernel():
    # Your kernel code here
    moe.forward(hidden_states, logits)

with benchmark_with_power(
    device=device,
    kernel_func=my_kernel,
    num_warmups=3,
    num_runs=6,
    repeat_n=1,
) as results:
    latency = results["latency_ms"]
    power_stats = results["power_stats"]

For TensorRT-LLM

Optional( Read if collecting for mxfp4 kernels)

To collect performance data for mxfp4 kernels used in GPTOSS models, depending on the version of TensorRT-LLM, you might need to manually install triton-kernels. If your version is >= 1.3.0rc2, nothing needs to be done and you can run the collection process directly. For any version before 1.3.0rc2. Follow the below instructions to install triton-kernels and expose them to TensorRT-LLM

  1. Build and install Triton (tested with the commit below):
git clone https://github.com/triton-lang/triton.git
cd triton
# Specific commit verified with TensorRT-LLM
git checkout f3067cd3bd0c29065fa4ecdb724b6f29cbabea5f
pip install -r python/requirements.txt # build-time dependencies
pip install wheel build
python3 setup.py bdist_wheel # if this step fails due cuda related error, you might need to set CUDA_HOME following the optional step
pip install ./dist/*.whl
  1. (Optional) You may need to set CUDA_HOME env variable to successfully build triton
export CUDA_HOME=/usr/local/cuda
export CPLUS_INCLUDE_PATH=$CUDA_HOME/include:$CPLUS_INCLUDE_PATH
export C_INCLUDE_PATH=$CUDA_HOME/include:$C_INCLUDE_PATH
python3 setup.py bdist_wheel
  1. Expose the Triton kernels to TensorRT-LLM The kernels are not packaged in the wheel, so set the environment variable TRITON_ROOT to your Triton clone:
export TRITON_ROOT=/local/user/triton
# TensorRT-LLM expects the kernels at:
#   $TRITON_ROOT/python/triton_kernels

Run collection for all available operations

python3 collect.py --backend trtllm

For trtllm, the whole collecting process takes about 30 gpu-hours. On 8-gpu, it takes 3-4 hours. Please note that the whole process will report a lot of missing datapoints with errors. But it's okay. Our system is kindof robust to fair amount of missing data. Once everything is done, you might see mutliple xxx.txt files under the same folder. Refer to src/aisim_core/systems/ folder to prepare the database including how many files are needed accordingly.

Resume Collection (Checkpoint)

Checkpoint files are always written so an interrupted run can be resumed later:

# Normal run (writes checkpoint to .collector_checkpoint/)
python3 collect.py --backend trtllm

# Resume an interrupted run (skips already-attempted tasks)
python3 collect.py --backend trtllm --resume

# Custom checkpoint directory
python3 collect.py --backend trtllm --resume --checkpoint-dir /path/to/checkpoints

A task is marked done once it is attempted (success or failure). Only tasks that never finished are re-queued on --resume; --resume-retry-failed additionally re-runs previously failed tasks. Running without --resume always starts fresh (overwrites old checkpoint).

For SGLang

Use the pinned stock image, lmsysorg/sglang:v0.5.14 (or its -cu130 variant), for normal SGLang collection.

python3 collect.py --backend sglang

This collects the stock SGLang ops, including:

  • GEMM operations (FP8, BF16, INT8, INT4)
  • MLA (Multi-head Latent Attention) for context and generation
  • MLA BMM (Batch Matrix Multiplication) operations
  • MoE (Mixture of Experts) operations
  • Normal attention operations

Large-EP local expert compute is available as the separately routed moe_ep operator under wideep/sglang/collect_deepep_moe.py; run it explicitly in its pinned SGLang 0.5.10 image. WideEP MLA is not registered because its legacy wrapper now reaches the stock 0.5.14-only module implementation.

DeepEP multi-node collector

For models with DeepEP MoE, inter-node communication data requires a separate multi-node setup. It is collected by the standalone wideep/sglang/collect_moe_a2a.py collector, launched one Slurm job per world size by network/slurm/submit_moe_a2a.sh — see section 4 of network/slurm/README.md. The older manual log-scraping pipeline under wideep/sglang/deepep/ is deprecated (its README carries the retirement story); do not add new data through it.

Supporting a new large-EP (WideEP) model

Large-EP MoE uses stock moe_perf for modeled local expert compute and moe_a2a_perf for measured dispatch/combine communication. To support model X:

  1. Declare the shapes. Add the model to cases/models/<Architecture>_cases.yaml (new architecture: one new file; new model in an existing architecture: append to model_paths) and mark its model_case_values.moe rows with wideep: true. Correlated MoE dimensions (hidden/inter size, topk, expert count) stay together in one row; never cross another model's values. The wideep flag activates standalone A2A shape population.

  2. Ensure stock moe_perf coverage. Local compute is modeled as a uniformly balanced stream over EP-local experts, using TP=1 stock MoE coordinates. Cross-version stock-table reuse is allowed.

  3. Collect moe_a2a across nodes. From network/slurm/:

    bash submit_moe_a2a.sh                      # sglang DeepEP HT+LL, one job per world size
    bash submit_trtllm_alltoall.sh              # trtllm NVLink alltoall (single/multi-GPU NVL)

    The comm tables have no model column — a new model whose (hidden_size, topk, num_experts) tuple is already covered needs no new runs; a new physical shape lands in the sweep automatically once declared in step 1.

  4. Publish with sidecars. Finalized communication parquet goes into .../<system>/comm/<backend>/<version>/moe_a2a_perf.parquet together with its collection_meta.yaml entry — never a parquet without its provenance. The per-world moe_a2a outputs need the cross-job merge procedure in network/slurm/README.md section 4.3.

MoE table units and caveats

moe_a2a_perf records latency in microseconds; load_moe_a2a_data converts leaves to milliseconds. Stock moe_perf retains its existing timing contract.

Stock moe_perf also supports optional Boolean default_eligible selection metadata. The Core API contract defines automatic and exact-source selection. Finalization validates non-null Boolean flags and the named source required by false, excludes the flag from measurement identity, and preserves existing annotations when merging a legacy collection that omits the column. Collectors do not infer eligibility from kernel labels or row order.

Legacy-overwrite caveats. A new-schema row replaces a legacy-adapted leaf only at the same key, and the legacy adapters derive their node/EP geometry rather than reading it:

  • Legacy sglang DeepEP comm tables carry no ep_size; the adapter assumes ep_size = node_num * 8 (8-GPU HGX fleets). New moe_a2a worlds therefore overwrite those leaves only when collected at 8 GPUs/node — a 4-GPU/node (GB200) world with the same node count keys differently and coexists with the legacy data instead of replacing it.
  • The legacy trtllm alltoall table (GB200 NVL4) carries no num_nodes column; the adapter derives node_num = max(1, moe_ep_size // 4). New rows overwrite only at 4 GPUs/node; a row with an explicit num_nodes column is honored as written.

log_perf freezes the CSV header from the first row, so optional columns are all-or-nothing per file. In particular, power columns exist in a staging CSV only if the very first logged row had them: flipping --measure_power across a --resume of the same staging file appends rows whose fields no longer match the frozen header — misaligned columns or empty cells that fail at load time. Collect any one file with a single power setting end to end. This property is also why the moe_a2a collector ships no power column at all (owner ruling during PR review): its low-latency timing covers one round-trip rather than a per-phase region, and meaningful per-phase power would need winning-config re-runs — a measurement-method design for hardware, not a column to fake. The loaders treat absent (or null) power as 0.0.

Test

Rebuild and install the new aiconfigurator. Please make sure you have your new system definition file prepared. It's src/aisim_core/systems/xxx.yaml

Validate the correctness

Today, we have limited method to validate the database. You can try tools/sanity_check to validate the database a little bit. But it highly depends on your understanding of the GPU system and kernel optimization.

Known Issues

NFS File Locking (Worker Deadlock)

Symptom: Collection stalls after a few test cases with no error messages.

Cause: fcntl.flock() doesn't work reliably on NFS. Workers deadlock when writing to shared output files.

Solution: Use /tmp/ for output files, then copy results after collection.

Support Matrix

Browse the Legacy AIC Support Matrix for CLI compatibility coverage, or inspect the per-system CSV files. For strict-native estimator coverage, use the FPE Support Matrix.

SGLang ordinary MLA context modules

For uniform-projection checkpoints, run_attention_torch(..., ordinary_mla=True) collects the complete context attention module into mla_context_module_perf.txt. It invokes SGLang's real prepare_qkv_latent and creates fresh AttentionInputs on every warmup and timed iteration, so Q/KV down-projection is included. Prefill is timed eagerly. The loaded projection weights must match the requested single gemm_type; mixed BF16/NVFP4 modules are rejected instead of mislabeled.

The CLI flag --ordinary-mla requires --mode context --attn-type mla. Other combinations fail during argument parsing, before collection starts. If any dispatch fails, the CLI reports its traceback, finishes the remaining dispatches, and exits with status 1 so partial collection is not reported as success.

Pass --target-tp-size 4 (or 8) to derive the per-rank head count from the model. It must be positive and divide the model's native head count. If --num-heads is also supplied, it must equal native heads divided by TP. Without an explicit TP size, ordinary MLA derives TP separately for each per-rank head count. Python callers pass the actual per-rank head count and target_tp_size. Set --chunked-prefill-size 16384 on the CLI, or chunked_prefill_size=16384 on run_mla_module() or run_mla_module_worker(), to reproduce a serving chunk limit of 16,384 tokens. The worker forwards the limit into its subprocess. Explicit limits must be positive for one-shot module collection. Omitting the option keeps SGLang's default. The existing wide-EP and DSA paths keep their separate behavior. The ordinary context table uses the existing consumer schema; this does not add mixed-precision modeling.

TRT-LLM MoE autotuning failures

TRT-LLM MoE collection requires either a loaded tactic cache or at least one successful autotuning attempt before measuring a tuning-enabled case. Ordinary failures may try smaller eligible token counts; if every attempt fails, the case raises before timing or publishing rows. CUDA accelerator errors propagate immediately. The successful collection path and historical profile values are unchanged by this failure guard.

Appendix: estimated extra collection time for long-context shapes

These are provisional planning estimates, in additional collection GPU-hours above the pinned #676 baseline. They price the extra model-owned points enabled by --include-long-context; the existing baseline anchors remain enabled when the flag is off. They are not measured full-sweep totals or confidence bounds. The table reflects the frozen pilot inventory; the later MiniMax MXFP8 alias coverage correction adds projection-lane points that remain unpriced here.

Estimated long-context increment

The estimates use each sampled SKU's own framework measurements and the frozen added-case inventory. Rounded planning centers across the three frameworks are:

GPU SKU vLLM GPUh SGLang GPUh TRT-LLM GPUh Long-context total GPUh
L40S 2.0 2.6 3.4 ~8
H100 SXM 4.4 2.7 1.9 ~9
H200 SXM 3.9 10.2 1.7 ~16
B200 SXM 2.6 18.3 5.5 ~26
B300 SXM 2.2 18.7 1.3 ~22
GB200 3.5 9.5 4.5 ~18
GB300 5.6 13.9 10.6 ~30
A100 SXM80 Unmeasured Unmeasured Unmeasured Unmeasured

The seven sampled SKUs total approximately 130 GPU-hours for the modeled long-context work. Rows include explicit analogous-operation provisions where native measurements are missing; 1,628 additional history cells still lack a same-framework analogue. The table does not establish that every planned shape fits memory or has a working native kernel.

Preparation scales with cache/input payload and execution with attention work, including the sliding-window cap. Precision, geometry and dispatch extrapolations remain assumptions. B300 uses the intended GLM NVFP4 collection recipe and its own measurements; these timings do not establish canonical BF16 startup or capacity equivalence. Other SKUs do not borrow B300 rates.

Inclusive point clocks contain input/history preparation, warmup, compilation, graph capture and forwards. Forward latency is not added again. Existing worker initialization is paid once when its group is extended; larger buffers and shape-dependent setup can still add cost. Scheduler wait, GPU allocation launch and idle reservation are excluded. Operator tests use one physical GPU even when their geometry represents a TP rank; actual collective ranks are counted for NCCL.

Broader expansion budget

The independent MoE token, KDA width and NCCL payload changes add a separate approximately 55 GPU-hours of planning proxies across the seven sampled SKUs. They are outside the history flag and are not part of the 130-hour history total. Together, the modeled expansion subtotal is approximately 185 GPU-hours.

A provisional eight-SKU working budget of 300 GPU-hours adds selected reserves of 50 GPU-hours for unmeasured A100 work and 65 GPU-hours for remaining cases, setup changes and retries. A larger engineering allowance is 750 GPU-hours total, not 750 additional hours. These reserves are budget choices, not measured rates, statistical intervals or guarantees of successful coverage. No RTX PRO estimate is included in this eight-SKU budget.

The supporting planning calculations, unrounded per-framework values, case counts, native/proxy distinctions, source hashes and scratch calculations are retained as local campaign artifacts outside this PR, together with the six collection-cost audit scripts and their tests. These audit-only scripts are not shipped under tools/perf_database/. The separate strict audit keeps unqualified full totals unknown; these planning figures do not change its validation results. Deferred native Kimi coverage is not priced as zero or counted as equivalent-coverage savings.