Add per-stage load telemetry to the model server (RUN-544)

[kamh04]

🚀 What

Breaks the opaque model.load() time into named, machine-readable stages, emitted once per model load as a structured log line (model load telemetry: {...}) and per-stage OTel spans — with zero customer code changes:

• server_boot_ms — process start → load start (interpreter + server imports), recovered from /proc/self/stat
• Stage timings: setup, extensions_load (incl. TRT-LLM), import_model_module (customer imports — where import torch lands), model_init, setup_environment, model_load
• torch compile attribution: torch_compile_ms (Dynamo ledger diffed around load()), torch_compile_phases_ms (top-5 phases), torch_unique_graphs, torch_graph_breaks, torch_fx_cache_hits/_misses (is persistent compile caching actually working?)
• gpu_mem_allocated_gb / gpu_mem_delta_gb — weights→GPU footprint
• unattributed_ms — total minus the sum of stages, the honesty bucket

Context: reconstruction of the fleet's worst container_ready cold starts (Linear: RUN-544, "Profile container_ready portion of GPU Cold Start") showed 62–91% of the worst cases is deterministic recompute (compile/warmup) — visible today only via fragile per-framework log mining. This gives every truss model a first-party breakdown.

💻 How

A LoadTelemetry context manager (load_telemetry.py) wraps the stages of ModelWrapper._load_impl and snapshots torch state before/after the customer's load(). Design invariants, each pinned by a test:

• Telemetry is write-only: internal failures degrade to missing fields + a warning, never a failed or slower load.
• torch is never imported by the server: it is read from sys.modules, so it is only observed when the customer's code already imported it. Non-torch models pay nothing.
• torch.compile() is lazy (compilation happens at first forward), so compile time is read from Dynamo's compilation_time_metrics ledger, snapshotted before load() and diffed after — feature-detected because the API is semi-private and has moved across torch versions.
• GPU memory is only read when CUDA is already initialized, because memory_allocated() would otherwise trigger CUDA init.

Cost: microseconds of clock reads + one <1 KB log line per cold start; predict path untouched; spans go through the existing (default no-op) tracer.

🔬 Testing

• 12 unit tests (test_load_telemetry.py): stage/unattributed arithmetic, compile-delta attribution incl. the pre-seeded-exclusion boundary, version-fallback key ladder, phase-list bounding, counter deltas, CUDA-init guard, /proc parsing (incl. parens-in-comm), finalize-never-raises.
• Server-level integration test (test_truss_server.py::test_load_emits_load_telemetry_line): real ModelWrapper.load() against the fixture truss → exactly one parseable line, real stage timings, torch fields via sys.modules without torch installed, GPU fields absent (not fabricated) without CUDA.
• Real-torch end-to-end: built the serving image locally from this branch (TrussHandle.build_serving_docker_image) with a probe model whose load() runs torch.compile over 3 input shapes. Observed line matched all predictions: torch_unique_graphs: 3, torch_fx_cache_misses: 3, torch_compile_ms: 3839.6 with real Dynamo phase keys, load_total_ms exactly equal to the legacy "Completed model.load() in N ms" value, unattributed_ms: 0.0.
• Full truss/tests/templates/server suite green (49 tests); ruff clean.

Notes for review

• The telemetry line is on the customer-visible log stream (deliberate — customers see their own load breakdown), landing right after "Completed model.load() execution in N ms".
• Open question: the telemetry JSON is nested inside the log record's message, so Loki-side extraction needs regexp rather than | json. Happy to flatten fields to the top level of the record in this PR if preferred.
• Deployed-image verification note: truss push builds with the released context builder, so this change reaches deployments only via a release — verified via the local image build instead; a staged probe model exists for post-release rollout verification.

🤖 Generated with Claude Code

[linear]

RUN-544

[CLAassistant]

All committers have signed the CLA.