| title | Sweeper Optimization Goals |
|---|---|
| subtitle | Objective metrics, SLA constraints, and Pareto-front scoring |
Warning
Experimental. Sweeper's API, configuration schema, search results, and deployment output may change without a standard deprecation period. Validate shortlisted configurations on the target hardware before using them for production capacity or SLA decisions.
An OptimizationGoal (the goal: block of a SmartSearchConfig) declares what
"better" means plus the SLA constraint. It is pinned, never searched. It picks one
OptimizationTarget and — for pareto — the list of scalar objectives whose frontier to
trace. Optional providers receive the complete goal in SweepContext; any feature-specific
mapping belongs to the provider, not the Sweeper core.
The whole goal: block is optional: it defaults to a throughput goal with no SLA
(OptimizationGoal() → target = throughput). OptimizationGoal itself is
extra="forbid", so unknown goal keys are rejected.
The goal drives two things in score.py: the objective read from each replay
trace_report (objective_value), and step 3 of scoring — rank (scalar) or
pareto_front (multi-objective).
Every OptimizationTarget and the exact report metric it reads (score.objective_value):
target |
direction | report metric | needs SLA? |
|---|---|---|---|
throughput |
maximize | output_throughput_tok_s |
no |
throughput_per_gpu |
maximize | output_throughput_tok_s / avg_gpu (tok/s/gpu) |
no |
throughput_per_user |
maximize | mean_output_token_throughput_per_user (tok/s/user, one 1000 / ITL sample per request) |
no |
ttft |
minimize | mean_ttft_ms |
no |
e2e_latency |
minimize | mean_e2e_latency_ms |
no |
goodput |
maximize | goodput_output_throughput_tok_s |
yes |
goodput_per_gpu |
maximize | goodput_output_throughput_tok_s / avg_gpu (tok/s/gpu) |
yes |
min_gpus |
minimize | concrete candidate used_gpus (provisioned GPUs) |
yes |
pareto |
per-objective | a vector — one value per pareto_objectives entry |
iff an objective needs it |
ttft, e2e_latency, and min_gpus are minimized targets: OptimizationTarget.maximize returns
False for all three (and raises for pareto, which has no single direction). score_report negates
minimized targets so higher is always better internally; for a Pareto goal the raw
(unsigned) value is kept and _dominates applies each objective's own direction.
Missing-key defaults differ by direction: a maximized target reads 0.0 when its key is
absent, but ttft and e2e_latency default to +inf when their metric is missing or has
no qualifying samples. Such a latency report scores worst-possible (-inf after negation)
rather than best.
The *_per_user metric is already a rate, so it gets no GPU/time normalization. It is
the InferenceX x-axis (tok/s/user). Each completed request with at least two output tokens
and a positive ITL contributes one sample, 1000 / ITL, where ITL is the request's
first-to-last token interval divided by its output tokens minus one. This is
output_token_throughput_per_user.avg.
Compatibility: earlier releases averaged 1000 / gap over every individual token gap, and
the native itl distribution held every token gap. Both weighted long responses more and did
not match AIPerf. Native itl and output_token_throughput_per_user now hold one sample per
request, and user TPS percentiles are exact instead of sketch estimates.
The two *_per_gpu targets divide a tok/s rate by the time-averaged provisioned GPU
count (score._avg_gpu):
avg_gpu = gpu_hours / e2e_hours # e2e_hours = duration_ms / 3_600_000
This is the integral of provisioned GPUs over the run divided by its duration:
- static deployment —
avg_gpucollapses to the fixed GPU count (gpu_hours = gpu_count * e2e_hours). - runtime-scaled run —
avg_gpuaverages provisioned GPUs over startup + serve + drain.
Dividing the rate by gpu_hours directly would be wrong: the rate already has time
divided out. _avg_gpu returns 0.0 when gpu_hours or duration_ms is <= 0
(missing report keys default to 0.0, so the guard covers both missing and non-positive
values), and the *_per_gpu targets then return 0.0 (divide-by-zero guard).
The goodput targets need an SLA because their metric
(goodput_output_throughput_tok_s) counts only SLA-satisfying requests (the replay
bridge's per-request goodput SLA). _SLA_TARGETS = {goodput, goodput_per_gpu}.
OptimizationGoal._validate_goal computes the effective objective set — {target} for
a scalar goal, or the resolved pareto_objectives for a Pareto goal — and requires an SLA
iff that set intersects _SLA_TARGETS. So an SLA is mandatory when:
targetisgoodputorgoodput_per_gpu, ortargetisparetoand its objectives include one of those.
A satisfying per-request goodput SLA has at least one configured bound. TTFT and ITL are
independently optional; an unset field is unbounded. e2e_ms remains mutually exclusive
with either token-latency field.
By default SLA is not gated during aggregate feasibility (is_feasible checks only
the GPU budget) — it lives inside the goodput metric, so an unconditional aggregate
latency gate would double-count it. strict_sla: true is the explicit legacy-compatible
opt-in: it filters aggregate mean metrics before scalar ranking or Pareto dominance.
SLATarget shape (ms, each > 0, extra="forbid"):
| field | meaning |
|---|---|
ttft_ms |
independently optional time-to-first-token bound |
itl_ms |
independently optional inter-token-latency bound |
e2e_ms |
end-to-end bound — standalone alternative |
Strict aggregate comparisons are inclusive (value <= bound). A configured bound with
a missing/non-finite report metric or no qualifying latency samples rejects the candidate.
Use an AIPerf metric.statistic as a target or constraint. The supported metrics
are time_to_first_token, request_latency, inter_token_latency, and
output_token_throughput_per_user. Each accepts avg, p50, p75, p90, p95, and p99. Latency targets minimize milliseconds; user
throughput targets maximize tokens/s/user. The SDK also accepts these selectors in
pareto_objectives.
A selector reads the native replay summary. predict, recommend and the Sweeper
report the same fields, so no extra configuration is required:
| AIPerf metric | Native summary fields | Sample count |
|---|---|---|
time_to_first_token |
{mean,median,p75,p90,p95,p99}_ttft_ms |
num_ttft_samples |
request_latency |
…_e2e_latency_ms |
num_e2e_latency_samples |
inter_token_latency |
…_itl_ms |
num_tpot_samples |
output_token_throughput_per_user |
…_output_token_throughput_per_user |
num_tpot_samples |
avg reads mean_* and p50 reads median_*. Optional aggregate constraints live in
optimization.constraints.metrics.
optimization:
target: throughput_per_gpu
strict_sla: true
constraints:
metrics:
- metric: output_token_throughput_per_user.p50
operator: gte
threshold: 50
- metric: time_to_first_token.p99
operator: lte
threshold: 2000Set target: request_latency.p99 to minimize request latency p99 instead.
The SDK uses OptimizationGoal.metric_constraints for the same list. predict reports
the statistics but has no constraints.
Each entry requires metric, operator (lte or gte), and a finite,
nonnegative threshold. Equality passes. A selector with no qualifying samples
fails strict constraints and cannot qualify an objective, even though the
native summary reports zero for an empty distribution. User TPS uses the ITL
count, so if every ITL is zero it reads 0 tokens/s, where AIPerf reports no value.
strict_sla: true enforces these constraints before optimizer feedback and
ranking. With strict_sla: false, the metrics remain in candidate results, but
the constraints do not reject candidates. The target remains unchanged:
throughput counts all completed output, while goodput counts output from
requests that satisfy evaluation.sla. Neither target enables strict rejection
implicitly. The existing min_gpus target retains its mandatory SLA gate.
The native definitions follow ordinary metrics in AIPerf v0.13.0:
- Each successful request contributes one sample per available metric.
Failed, canceled, rejected and unfinished requests contribute no samples.
AIPerf's separate failure-adjusted
adj_*metrics are outside this selector set. - TTFT spans arrival to the first token; request latency ends at the last token. A single-token response contributes both, but no ITL or user TPS.
- ITL divides the first-to-last token interval by output tokens minus one. Replay
treats the first chunk as one token, which is AIPerf's default without
--per-chunk-usage. - User TPS is
1000 / ITLfor each request with positive ITL. A zero ITL remains an ITL sample but contributes no user TPS. Aggregation happens after inversion. - Counts and means match AIPerf. Percentiles use the nearest rank, the sample at
index
round((n - 1) * p / 100). AIPerf interpolates linearly, so percentiles can differ for small samples or sparse tails.
These selectors require native offline request replay. FPM, op-level timing, and native AFD use that path. Analytical EPD rejects them explicitly. Matching these definitions does not model an unconfigured transport or frontend cost. Compare calibrated replay against serving measurements before admission.
evaluation.sla and the goodput targets keep their native per-request filters.
They are not AIPerf goodput:
- Native goodput counts the output tokens/s of qualifying requests. AIPerf goodput counts qualifying requests/s.
- The native ITL filter divides by the actual output length minus one and admits a single-token response. AIPerf has no ITL value for such a response.
- The
aiperfconstraints apply to aggregate statistics, not to each request.
goal.target: min_gpus selects the smallest qualifying configuration found by the sweep.
It always enforces the configured aggregate-mean latency bounds, even when strict_sla is
false. Open-loop traffic also needs a positive goal.min_goodput_rps: a synthetic request or
session rate, or trace timestamps without agentic lanes. Under open-loop arrivals, latency bounds
alone can pass a deployment that falls behind, for example when only ITL is bounded; the floor
rejects it. The measured goodput_request_throughput_rps must meet that floor. This metric counts
only requests satisfying the per-request SLA, divided by the complete replay duration, including
startup and drain. Choose the measurement duration and offered traffic explicitly; offering
exactly the required rate can fall below the floor on a short run. The floor must not exceed the
offered request rate; a session rate offers its turns per session as requests. Trace timestamps
set their own rate, which validation does not read, so a floor above it rejects every candidate.
Missing rate evidence fails closed; raw throughput or offered rate is never substituted.
Closed-loop traffic (fixed concurrency or agentic lanes) needs no rate floor, but may specify one.
The search visits backend/topology pairs in ascending GPU count, whichever algorithm is
configured, and stops suggesting pairs larger than the best qualifying count found so far (see
search space). The optimizer observes -used_gpus for qualifying candidates
and infeasibility for latency or load failures. Final selection applies the same constraints to the full candidate pool before
top-N truncation, then sorts by GPU count, higher SLA-compliant output throughput, lower mean
E2E latency, and a deterministic configuration key. GPU count is provisioned topology size,
not time-averaged usage. A finite trial budget establishes the smallest qualifying configuration
found, not a global optimum. No qualifying candidate produces an empty selection.
This target supports static engine pools, either engine-only or behind the Dynamo Router
(--stack dynamo with a router section, Sweeper adapter dynamo.router), with synthetic
request, synthetic session, or trace traffic at one fixed absolute load. Sessions and traces can
reuse prompt prefixes, so prefix caching and KV-aware routing can change which candidates qualify.
The Router adds no GPUs, so the counted GPUs remain the candidate's provisioned used_gpus. The
optimizer observes the same -used_gpus for every qualifying candidate with a given GPU count, so
Router selections steer the search only through feasibility; pin the Router policy or keep its
domains small enough for the trial budget to cover the smallest pools. The Dynamo Planner
(dynamo.planner) is rejected because autoscaling makes the provisioned GPU count time-varying.
Other adapters, candidate-relative KV load, and searched load domains are also rejected, because
every candidate must serve the same load. Analytical EPD supports concurrency plus aggregate
latency only, without a rate floor.
min_gpus cannot be a Pareto objective, and min_goodput_rps is only accepted with this target.
In the CLI, use optimization.target: min_gpus and
optimization.constraints.min_goodput_rps; SLA remains under evaluation.sla. See the
minimum-GPU migration mapping.
pareto is the one multi-objective target. Instead of a scalar score it optimizes the
tradeoff between the scalar targets in pareto_objectives.
-
objectives —
OptimizationGoal.pareto_objectives(list[OptimizationTarget] | None).resolved_pareto_objectivesreturns it, or the default pair_DEFAULT_PARETO_OBJECTIVES = (throughput_per_gpu, throughput_per_user)only when it is unset (None) — the InferenceX tok/s/gpu (y) vs tok/s/user (x) frontier. An explicit list is kept as-is so the validator can reject it:>= 2entries, distinct, noparetoamong them, andpareto_objectivesis only legal under aparetotarget. -
optimizer selection — the historical SDK path (
sweep.max_trials: null) creates round-based Vizier studies. Its default algorithm isDEFAULT, subject to the algorithm override. Whensweep.max_trialsis set,sweep.algorithmselects the seeded Bayesian or random sampler. The unified CLI always supplies this total-trial budget fromoptimizer.max_trials;optimizer.algorithmselectsbayesianorrandom. Do not assume every CLI study uses Vizier's embeddedDEFAULTdesigner. For Pareto, samplers receive every objective and its direction. -
front —
score.pareto_frontreturns the non-dominated subset after optional strict aggregate SLA filtering._dominates(a, b)is true iffais at least as good asbon every objective (in that objective's ownmaximizedirection) and strictly better on at least one. The front is sorted by the last objective ascending — the x-axis — so the list traces the frontier left-to-right (e.g. low→high per-user throughput).Sweeper.runreturns this front for a Pareto goal, andrank(best score, ties → fewer GPUs) for every scalar goal exceptmin_gpus, which uses the constrained GPU-count ordering above. -
swept load dimension —
workload.concurrencyis always one fixed in-flight cap. A Pareto workload may instead setkv_load_ratio: [min, max], which Vizier models as a continuous parameter. Each ratio is converted to an absolute concurrency from that candidate's decode/agg KV capacity, so the model compares equivalent load pressure across different replica and parallel configurations. If a synthetic Pareto workload omits all load fields, the range defaults to[0.0, 1.0]; see traffic.md.
Per-objective raw values are stored on Candidate.objectives (keyed by
OptimizationTarget value, e.g. {"throughput_per_gpu": .., "throughput_per_user": ..})
by make_candidate; Candidate.score carries the first objective's value as a headline
number only (not used for Pareto ranking). objectives is None for a scalar goal.
This page describes SmartSearchConfig. In the unified CLI, put target and
strict_sla under optimization, SLA bounds under evaluation.sla, and search
controls under optimizer. The CLI's Pareto objectives are fixed to
throughput_per_gpu and throughput_per_user; the SDK allows an explicit
goal.pareto_objectives list. Do not copy a complete SDK YAML into the CLI.
For a CLI recommendation that minimizes TTFT, start with the bounded recommendation example and replace its goal with:
optimization:
target: ttft
constraints: {max_candidate_gpus: 4}Its score is negative mean TTFT, so a larger score means lower latency. Use
metrics.mean_ttft_ms when displaying the latency itself. The
prediction interpretation guide
explains latency populations and why strict aggregate SLA is not a p99 gate.