Skip to content

Latest commit

 

History

History
146 lines (123 loc) · 7.74 KB

runner-abi.md

File metadata and controls

146 lines (123 loc) · 7.74 KB

Runner ABI

Source contracts: sweeper/replay.py, stack discovery, and the built-in runner.

Required interface

class RunnerFactory(Protocol):
    def capabilities(self) -> RunnerCapabilities: ...
    def create(self, worker_id: int) -> Runner: ...

class Runner(Protocol):
    def run(
        self,
        spec: ReplaySpec,
        *,
        output_requirements: ReplayOutputRequirements | None = None,
    ) -> ReplayReport: ...
    def close(self) -> None: ...

A factory must be serializable for spawned workers. Each worker creates one runner and reuses it across candidate replays; worker-local state must not leak one candidate's requests or caches into another. close() releases resources. The engine factory lazily imports the compiled runtime in its worker.

Optional factories are selected by entry-point name in aisim.runner_factories (--stack). Resolution validates capabilities and create; unavailable, duplicate or unloadable optional stacks raise StackNotFoundError, DuplicateStackError or StackLoadError. A factory's coarse capabilities are checked before runner creation; execution must still validate finer feature combinations.

Replay data contract

REPLAY_SPEC_API_VERSION is 1. ReplaySpec contains:

Field Meaning
backend_deployment Concrete backend/version, topology, role engine arguments, worker counts, parallel shape and exact estimator identities
workload Concrete source/load/stop fields; strict data crossing the process boundary
goal Evaluation/optimization requirements including optional SLA
concurrency Closed-loop source cap or None for open-loop arrivals
adapters Adapter-owned AdapterReplaySpec values
api_version Must equal the runner's replay-spec version
execution_mode offline by default; online only when advertised

runtime_hooks flattens adapter hook tuples in adapter insertion order. Each RuntimeHookSpec declares (provider, kind, api_version, config). Compatibility matches all three identity fields, with exact integer version checks. Config/payload values must be strict JSON: finite numbers, strings, booleans, null, lists and string-keyed dicts. canonical_json provides stable ordering for serialization/cache keys; it does not license arbitrary Python objects inside adapter payloads.

Capabilities

RunnerCapabilities declares backend/topology pairs, execution modes, trace formats and hook identities. Pair/trace declarations may use "*" where the contract allows it. More specific flags cover disaggregated attention-DP, AgentX backends/topologies/lanes/snapshots/warmup/profiles, offload/speculation, analytical EPD and its deferred overlay, cached-prefix tokens, state/grouped cache and model controls. Generic flag defaults are not qualification evidence for an optional stack. supported_worker_types defaults to ("monolithic",); a runner must explicitly include "afd" to accept AFD roles. The built-in engine advertises both types. Deployment modes are agg|disagg, independent of role implementation. Ordinary and native AFD deployments retain ReplaySpec API 1. Role-local AFD topology and qualification artifacts use their own schema 2; this does not require downstream runners to implement AFD or upgrade the common ReplaySpec protocol. The default monolithic-only capability rejects AFD roles before execution.

Old afd/afd+pd deployment modes and development API-2 saved inputs must be regenerated. API-version checks remain exact; API-2 inputs are not silently converted to API 1.

supports_generic_speculation defaults to False; the built-in engine factory sets it to True within the registered-scheme replay limits. Unsupported stacks reject generic scheme inputs before runner creation or resource preparation. supports_mtp_expected_acceptance only declares handling of MTP/NextN acceptance inputs; it does not establish support for EAGLE or other schemes' draft costs. A stack must explicitly declare generic speculation support and retain those costs through its timing provider.

supports_mtp_host_offload defaults to False and declares the combined MTP + G2 contract at any attention DP: aggregated workers with a rank-local pool, and prefill and decode workers with a local or a shared pool. The built-in engine advertises it; existing external runners, including Dynamo, must opt in only after their runtime and dependency pins support the same composition. Generic speculation or G2 support alone does not enable it.

require_compatible(spec) checks versions and coarse combinations before execution. AgentX G2 additionally requires offline vLLM and static workers. G3 cannot be combined with AgentX G2. Speculation with G2 is limited to canonical MTP with prefix caching and synchronous scheduling, and to MTP on a hybrid G2 with disable_eagle_block_drop. The feature matrix records the built-in stack's actual boundary, including limitations stricter than a capability flag.

If every backend/topology pair fails Sweeper preflight, RunnerIncompatibleError (subclass of NoViableParallelConfig) is raised before trials, with no serialized SweepResult. Mixed incompatibility and model/memory/data failures retain NoViableParallelConfig with diagnostics. The CLI reports configuration errors as exit status 2. Candidate-specific build, replay, budget and timeout failures are recorded as infeasible trials by Sweeper.

Output and resource lifecycle

ReplayReport contains metrics and JSON metadata. Non-power metrics must be finite numeric values; null is allowed only for designated power fields. Report the normalized metric semantics used by scoring, and retain provenance and limitations rather than substituting zero for unavailable measurements.

ReplayOutputRequirements defaults to summary output and can request raw, per-request, telemetry, memory or timing diagnostics. A runner must reject unsupported capture combinations. The Engine JSON runner supports telemetry for offline vLLM/SGLang aggregated and separate prefill/decode replay without encoder pools. capture_telemetry retains all-phase detail under metadata["native_report"]["telemetry"]; it does not change scoring metrics. See the telemetry contract.

A factory that sets supports_deferred_encoder_overlay accepts defer_encoder_overlay=True. For an analytical EPD spec, the runner then returns the normalized language report, which carries the metrics apply_encoder_overlay reads, without applying the overlay, and rejects the flag together with memory diagnostics. Sweeper fails a report that already carries the overlay as a runner contract violation. It reuses a returned language report for every encoder pool that shares it and applies the overlay per pool; a failed, timed-out or refused replay is not reused.

An optional estimate_host_resources(workload, *, concurrency=None) factory method participates in resource admission. It must return ResourceEstimate API version 1 with nonnegative byte counts, a positive known request count, and an optional positive peak estimate at least as large as its lower bound. Unknown peak is not zero memory: guarded execution admits at most one continuously supervised candidate when budgets permit. Low-level run() callers own their supervision; the runner ABI itself does not spawn a resource monitor or publish CLI artifacts.