| title | Sweeper Results |
|---|---|
| subtitle | A lossless candidate ledger with scalar and Pareto views |
Warning
Experimental. Schema version 1.1 adds explicit resource-limited candidate outcomes; 1.0 remains readable.
Within a schema version, fields keep their meaning and units; incompatible changes require a new
schema_version and an explicit converter.
SweepResult is the canonical result envelope for both exhaustive and optimizer-guided execution.
The complete candidate ledger is the source of truth. Scalar top-N and Pareto fronts are views that
refer to ledger rows by stable candidate_id; they do not discard the other evaluated candidates.
The unified aisim recommend command writes this envelope as recommendation.json beside
the selected recommendations/*.yaml prediction inputs.
result = sweeper.run(config, top_n=5, candidate_retention="all")
result.to_json() # lossless interchange
result.to_csv() # documented analysis view
result.counts.failed # machine-readable outcome counts
result.selected_candidates # top-N or Pareto candidatesSweeper.run(config) returns SweepResult, the single public execution result. Callers that need
only the scalar top-N or Pareto selection use result.selected_candidates; rejected candidates,
run provenance, and counts remain available on the same result envelope.
Strict aggregate SLA filtering happens before scalar ranking or Pareto dominance. Rejected candidates
remain in the ledger with status infeasible and reason category sla_constraint.
With min_gpus, measured goodput below the requested rate floor is infeasible with
load_constraint; missing rate evidence is failed with runner_contract. Candidate metrics
retain request_throughput_rps, goodput_request_throughput_rps, and goodput_completed_requests
when supplied by the runner. GPU-count selection happens before the top-N view is truncated.
| Field | Meaning |
|---|---|
schema_version |
Result contract version. New results emit 1.1; reading 1.0 explicitly upgrades the envelope and defaults resource-limited counts to zero. |
candidate_retention |
all, feasible, or views; counts always describe the complete run. |
counts |
Outcome and cache counts for the complete run. |
candidates |
Retained candidate records in a reproducible order: each batch's pre-replay rejections, then its replays in suggestion order, independent of which parallel evaluation finishes first. |
views.top_n |
Best-first scalar candidate IDs, limited by top_n. |
views.pareto_front |
Non-dominated candidate IDs in frontier order. |
provenance |
Search strategy, run ID/time, implementation, input fingerprint, and validated input config. |
views.top_n and views.pareto_front are mutually exclusive. The unified CLI applies final adapter
canonicalization and concrete-config deduplication before writing the active view, so each selected
candidate ID maps one-to-one to one numbered prediction YAML. An empty result has empty views and an
empty candidate list, while its counts and run provenance remain present.
Every materialized or capability-gated candidate attempt has one record when retention is all.
| Field | Meaning |
|---|---|
candidate_id |
Stable within the result, formatted candidate-NNNNNN. |
status |
feasible, infeasible, unsupported, timed_out, failed, or resource_limited. |
config |
Concrete backend, topology, engine knobs, load, and adapter configuration available at the terminal status. |
prediction_config |
Concrete public aisim predict configuration when produced by the unified CLI. |
used_gpus |
Provisioned GPU count, or null if materialization failed before it was known. Unit: GPUs. |
score |
Scalar score normalized so larger is better; for a Pareto row it is the first objective's raw value. |
metrics |
Normalized replay metrics in natural units for every attempt that completed replay, including attempts later classified infeasible. Empty only when no valid replay report exists. |
objectives |
Natural-unit Pareto objective values, otherwise null. |
reason_category |
Stable category for a non-feasible candidate. |
reason |
Human-readable detail; consumers branch on reason_category, not this text. |
provenance |
Model, hardware, backend/version, topology, workload, objective/SLA, data, power, and per-operation evidence. |
Metric names and units are explicit: throughput is *_tok_s, latency is *_ms, energy is *_j,
power is *_w, duration is duration_ms, and gpu_hours is GPU-hours. score is not assumed to
have a unit; use the named metric or objectives for display and comparisons.
power_w and power_coverage follow the
modeled-power contract: active-forward-pass power per GPU,
energy-over-active-latency aggregation, and AIC's existing coverage rule.
Coverage is the share of modeled active time with operation-energy evidence.
For every candidate with a valid replay report, both keys must be present in
metrics and power provenance. Exactly 90% is sufficient to publish numeric
power_w; below 90%, power_coverage remains numeric while power_w is null.
A runner without typed operation-energy evidence must return null for both
values. Zero coverage is reserved for an energy-aware path with no covered
active latency. Null values must never be treated as zero watts or zero coverage.
Failed attempts without a valid replay report retain the empty metrics
envelope described above; they do not contain a power summary.
Valid replay reports preserve both nullable keys through runner normalization,
candidate metrics, power provenance, and SweepResult.to_json(). Publication
validation rejects numeric watts below the coverage gate. Null values never
become zero or enter objective arithmetic.
This contract applies to AISim ReplayReport and SweepResult outputs.
Raw DataFrames from the compatibility aiconfigurator sweep/picking APIs retain
their legacy schema and sentinels; the AIC mapping describes conversion targets,
not an automatic converter. They do not satisfy this power contract as-is. A
converter must establish coverage and preserve unavailable values before emitting
a conforming result; it cannot infer coverage from a legacy wattage column alone.
evaluated is the number of candidate attempts that reached materialization or replay and equals
feasible + infeasible + timed_out + failed. unsupported is separate because capability gating
rejects it before evaluation. resource_limited is also separate: the host could not complete
an evaluation, so it is not a modeled constraint failure. cache_hits counts repeated optimizer suggestions served from the
run-local completed-result cache or coalesced with an identical suggestion in the same ask batch.
Each such suggestion consumes a trial budget slot and is counted explicitly as a cache hit, but does
not create a duplicate ledger row.
| Status | When used |
|---|---|
feasible |
Replay succeeded, required metrics were present, and budget gates passed. |
infeasible |
A modeled constraint such as GPU budget, KV capacity, or strict aggregate SLA rejected the candidate. |
unsupported |
The runner does not support the backend/topology pair. |
timed_out |
Replay exceeded max_eval_seconds. It remains infeasible to the optimizer, but is distinct in results. |
failed |
Materialization, runner execution, or the runner/result contract failed. |
resource_limited |
Host memory admission or bounded runtime recovery could not complete this candidate. |
Stable reason categories are gpu_budget, kv_capacity, sla_constraint, load_constraint, backend_topology, runtime_timeout,
candidate_materialization, replay_runtime, runner_contract, invalid_metrics, no_samples,
parallel_projection, adapter_constraint, resource_limit, and unknown.
Run provenance stores the complete validated SmartSearchConfig and a SHA-256 fingerprint of its
canonical JSON. It also distinguishes optimizer_guided from exhaustive execution.
Candidate provenance repeats the context required to interpret a row:
- model and hardware identifiers;
- backend and resolved performance-model version;
- concrete agg/disagg topology;
- the concrete materialized replay workload (including effective concurrency), objective, and SLA payloads;
- exact per-role performance-model identity from
BackendDeploymentSpec.performance_model_metadataplus performance-data source records from runner metadata; - power/energy measurements from report metrics or runner metadata;
- operation-level source and version records; and
- the complete validated runner metadata payload.
Runner authors may populate ReplayReport.metadata.performance_data, .operations, and .power.
Unknown JSON metadata is preserved under runner_metadata, so provenance is not lost when a newer
runner supplies evidence an older consumer does not yet promote into typed fields. Candidate
provenance is derived from the concrete ReplaySpec, not the run-level search domain or timing-model
heuristics. Pre-materialization rejections have no replay specification and therefore retain only the
concrete fields known at their rejection point; the full input domain remains in run provenance.
SweepResult.to_json() is the lossless interchange format. It rejects non-finite numbers, sorts
mapping keys, includes empty-run provenance, and round-trips through SweepResult.from_json().
SweepResult.to_csv() is a one-row-per-retained-candidate analysis view with these columns:
schema_version, candidate_id, status, reason_category, reason, used_gpus, score,
config_json, prediction_config_json, metrics_json, objectives_json, provenance_json,
is_top_n, and is_pareto.
Nested fields are canonical JSON cells rather than lossy dotted columns. CSV is not the interchange format: an empty CSV has only its header and therefore cannot carry run provenance or counts.
Scalar result: two feasible rows may be retained while views.top_n contains only the best row.
{
"counts": {"evaluated": 2, "feasible": 2, "infeasible": 0, "unsupported": 0,
"timed_out": 0, "failed": 0, "cache_hits": 0},
"views": {"top_n": ["candidate-000002"], "pareto_front": []}
}Pareto result: dominated feasible rows remain in the ledger but only non-dominated IDs are in the view.
{"views": {"top_n": [], "pareto_front": ["candidate-000004", "candidate-000001"]}}Empty result: no candidate was feasible, but failure evidence is still machine-readable.
{
"counts": {"evaluated": 3, "feasible": 0, "infeasible": 0, "unsupported": 1,
"timed_out": 1, "failed": 2, "cache_hits": 0},
"views": {"top_n": [], "pareto_front": []}
}Mixed-mode result: agg and disagg rows share one ledger. Each candidate's
provenance.topology.deployment_mode identifies its branch; scalar ranking or Pareto dominance can
operate across branches without separate result schemas.
Failure row:
{
"candidate_id": "candidate-000003",
"status": "failed",
"reason_category": "runner_contract",
"reason": "goodput objective requires goodput_output_throughput_tok_s"
}Exact repeated suggestions reuse a result within the current run; the cache
is not shared between runs. The runner ABI owns
ReplaySpec and normalized report obligations. A ranked candidate is not a
serving manifest; use deployment generation to
render supported candidates, and choose one point explicitly for a Pareto result.
resource_limited candidates have reason category resource_limit and no
simulated metrics or score. counts.resource_limited is separate from
counts.evaluated: a host admission refusal is not evidence about model
feasibility. A recommendation with resource-limited candidates covers only
completed evaluations. The CLI preserves completed results and exits with
status 3 to make this partial coverage visible. See local execution resources.
<output-dir>/
├── recommendation.json
├── recommendation.csv
├── resource-plan.json # on refusal before the sweep starts
├── resource-runtime.json
├── execution-events.jsonl
└── recommendations/
├── 0001.yaml
├── 0002.yaml
└── ...
recommendation.jsonis the canonical lossless result (schema 1.1, with explicit upgrade of 1.0 input). Its candidate ledger retains feasible, infeasible, unsupported, timed-out, failed, andresource_limitedrows according to the declared retention policy; its counts describe the complete run.views.top_norviews.pareto_frontlists the candidate IDs corresponding to numbered YAML files in order.- Resource-limited rows have no simulated score or metrics.
counts.resource_limitedis separate fromcounts.evaluated; selected configurations cover completed evaluations. The CSV is a tabular view of the result. See local resources for the diagnostic files. - Each numbered YAML is a concrete prediction config. It excludes
optimization,optimizer, andpreset, contains no domains orautovalues, and can be passed directly toaisim predict.
For scalar optimization, file numbering follows best-to-worst rank. For Pareto optimization, it follows the deterministic display order of the complete nondominated front; that order does not imply a scalar ranking.
If a completed search has no feasible candidate, the CLI still writes recommendation.json with empty views, zero
selected YAML files, complete counts and retained failure records. It exits with status 1 when
there are no resource-limited candidates. Any resource-limited candidate makes the exit status 3,
even when fitting candidates and selected YAML files remain available. Other failed trials remain
in the ledger and permit status 0 when at least one selected configuration remains.
If the supervisor stops the entire execution, the event log may contain completed candidates
without a finalized recommendation.json; it is partial evidence, not a completed sweep.