feat(loops): add truss loops metrics command - #2468
Open
rcano-baseten wants to merge 3 commits into
Open
Conversation
Surfaces compute utilization (GPU/CPU/memory) and Knative queue-proxy request metrics (rate, concurrent requests, latency p50/p95/p99, rate by status class) for a Loops trainer deployment. Pass --deployment-id or --base-model to identify the deployment; the latter resolves to the caller's active deployment for that model and errors on zero or multiple matches. Backend buddy PR adds POST /v1/loops/deployments/<id>/metrics. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
…me_stats) Matches the backend rename from `request_rate`/`request_latencies` to `inference_volume`/`response_time_stats` so trainer metrics nomenclature stays consistent with the oracle/inference surface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Backend PR clarified that response_time_stats values are milliseconds (matches the queue-proxy histogram bucket unit and the oracle response_time_stats convention). Updating the CLI labels from "(s)" to "(ms)" so the displayed header matches the value scale. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds `truss loops metrics --deployment-id ` (and `--base-model ` as a convenience selector) so users can see how their Loops trainer deployments are performing without leaving the CLI. Prints a service-level table (request rate, concurrent requests, latency p50/p95/p99, rate by 2xx/4xx/5xx) and a per-node compute table (CPU cores, CPU mem, GPU util/mem per rank, ephemeral storage utilization) using `rich`.
Pairs with backend PR https://github.com/basetenlabs/baseten/pull/20094 which adds `POST /v1/loops/deployments//metrics` and the parallel GraphQL query.
Test plan
🤖 Generated with Claude Code