Fix wait_for_active treating SCALED_TO_ZERO as a deployment failure

[FredLiu876]

🚀 What

truss.api.push(...).wait_for_active() raises ValueError: Deployment failed with status: SCALED_TO_ZERO while the push succeeds and the deployment goes on to serve normally.

ERROR: deploy_failed - Deployment failed with status: SCALED_TO_ZERO

Reported against a basetenlabs/action-truss-push workflow promoting to the production environment. The action failed ~13s in; the model served traffic 10 minutes later.

It is not a race

The obvious reading is that a deployment blips through SCALED_TO_ZERO on the way up and a 1s poller unluckily catches it. That is not what happens. Actual status history for the reported deployment:

Time (UTC)	Status
03:45:45.482	deployment created
03:45:46.073	BUILDING
03:45:56.994	DEPLOYING
03:45:58.406	SCALED_TO_ZERO — client raises here, T+12.9s
03:46:09.774	WAKING_UP
03:56:16.971	ACTIVE, first time ever

ACTIVE was 10m18s in the future when the client gave up. It was never observable. The SCALED_TO_ZERO window was 11.4s and the client dies on the first read of it, so window length only affects how often this reproduces, not whether it does. Two other deployments on the same model show 10.0s and 50.8s windows.

Why a new deployment reports SCALED_TO_ZERO having never had a replica

A rolling promotion with max_unavailable_percent > 0 creates the candidate deployment with initial_scale = 0. The state machine takes it PENDING → ready-with-zero-replicas directly, without ever starting a pod; there is a backend integration test asserting that as correct behavior. The environment's min_replica is applied only when the promotion completes, so in between the new deployment runs on its own default autoscaling and is scaled down for inactivity ~13s after creation.

Two consequences:

• The reported deployment had min_replica: 1. This is the promotion path, not the autoscaling setting, so raising min_replica is not a workaround.
• The backend considers the state healthy and terminal. SCALED_TO_ZERO is in both RUNNING_ORACLE_STATUSES and PROMOTABLE_STATUSES, and the deployment is marked live. The truss client is the only component reading it as a failure.

Separately, SCALED_TO_ZERO is also the resting state of any min_replica: 0 model, so wait_for_active can never succeed for one unless traffic happens to wake it.

💻 How

truss/api/definitions.py:39-43 is fail-closed with no explicit failure list:

if deployment_status == ACTIVE_STATUS:
    return True

if deployment_status not in DEPLOYING_STATUSES:
    raise ValueError(f"Deployment failed with status: {deployment_status}")

against truss/remote/baseten/core.py:29-30:

DEPLOYING_STATUSES = ["BUILDING", "DEPLOYING", "LOADING_MODEL", "UPDATING"]
ACTIVE_STATUS = "ACTIVE"

ACTIVE is success, those four keep polling, and literally everything else is a terminal failure. SCALED_TO_ZERO trips it, and so does WAKING_UP 11s later. The deeper problem is the polarity: any status the backend adds is a hard failure by default, so this breaks again on the next status the state machine learns to emit. truss/cli/cli.py:1008 / :1022 (truss push --wait) has the identical bug against the same constants.

Two new sets in truss/remote/baseten/core.py, as frozenset. DEPLOYING_STATUSES and ACTIVE_STATUS are left intact — truss watch, truss chains and the chains deployment client still import them.

• READY_STATUSES = {ACTIVE, SCALED_TO_ZERO} → return success
• TERMINAL_FAILURE_STATUSES → raise
• anything else → keep polling until the caller's timeout

Applied to both wait_for_active and truss push --wait.

SCALED_TO_ZERO is treated as ready, not as in-progress. The in-progress variant was already tried in #2162, which added WAKING_UP and SCALED_TO_ZERO to DEPLOYING_STATUSES; it closed unmerged. It would have made every scale-to-zero model hang until the timeout instead of failing fast — a worse failure, not a fix. Hence two sets rather than extending the existing one.

The failure list is drawn from statuses that actually exist in STATUS_TO_DISPLAYABLE (truss/remote/baseten/utils/status.py) rather than invented strings: BUILD_FAILED, BUILD_STOPPED, DEPLOY_FAILED, FAILED, INACTIVE, UNHEALTHY. That partitions all 14 public REST v1 statuses, with six left to keep polling: BUILDING, DEPLOYING, LOADING_MODEL, UPDATING, WAKING_UP, DEACTIVATING.

DEACTIVATING is deliberately in the polling bucket rather than the failure bucket. It is transient and converges to INACTIVE, which does raise, so this defers the error by one poll instead of dropping it.

Also folded in

MODEL_RUNNING_STATES (truss/remote/baseten/utils/status.py:20) had the same gap on the same non-chains path: it includes WAKING_UP but omitted SCALED_TO_ZERO, so truss push --tail and truss logs --tail stopped tailing the moment a promoting deployment scaled to zero — ~13s in, before any model-load output. Its only consumer is should_poll_again() at truss/cli/logs/model_log_watcher.py:45, and ACTIVE was already in the list, so this adds no new non-terminating case.

truss/cli/cli.py:1014 had deployment_status in ("LOADING_MODEL") — a string, not a tuple, so a substring test rather than a membership test. Harmless today because no REST v1 status is a substring of "LOADING_MODEL", but it is three lines from this diff and it is a real typo.

Deliberately out of scope

• Chains. chains_commands.py:428 is still fail-closed. feat(chains): Add SCALED_TO_ZERO as a ready status so chain deploys don't fail if a chainlet falls asleep #2490 closed unmerged; feat(chains): Add --no-sleep to chains watch #2500 mitigated it with keepalive threads rather than fixing the classification. Worth a separate PR.
• MODEL_READY is not in the ready set. This path reads public REST v1, where the ready value is ACTIVE and MODEL_READY does not exist on the wire. It appears only in the raw GraphQL status that the chains client reads untranslated, which is why action-truss-push's CHAIN_READY_STATUSES includes it and this does not.

Known limitation

In the promotion case above this returns success at ~13s, before the model has loaded. That matches how the platform defines a completed deploy, but a caller that immediately sends a request hits a cold start.

Distinguishing the two cases needs GET /v1/models/{id}/environments/{env}, which exposes in_progress_promotion / candidate_deployment / current_deployment: keep polling while a promotion is in flight, treat SCALED_TO_ZERO as ready otherwise. That is a larger change and only helps the environment-targeted path, so it is not in this PR. Happy to take it here instead if preferred.

Prior art

This fix already exists in other wait loops; the model SDK path is the one that never got it.

• truss/cli/utils/common.py:313-318 — truss watch already allows DEPLOYING_STATUSES + ["SCALED_TO_ZERO", "WAKING_UP", "UPDATING"], though as keep-polling rather than ready.
• basetenlabs/action-truss-push src/main.py:33 — CHAIN_READY_STATUSES = {"ACTIVE", "SCALED_TO_ZERO", "MODEL_READY"} plus an explicit CHAIN_FAILED_STATUSES, added in 2a75cf7 ("Fix chain wait: accept SCALED_TO_ZERO as ready"). Chains got both halves — ready-set and explicit failure list. The model path in that same action just calls deployment.wait_for_active() (src/main.py:90-93) and inherits the bug from here.

Rollout

action-truss-push installs truss unpinned (pip install -q truss requests pyyaml, action.yml:93), so affected users pick this up on the next release with no change on their side.

🔬 Testing

truss/tests/api/test_model_deployment.py (new) covers wait_for_active: SCALED_TO_ZERO returns True, WAKING_UP keeps polling then succeeds, every TERMINAL_FAILURE_STATUSES entry still raises, an unrecognized status keeps polling instead of raising, and the timeout still fires.

truss/tests/cli/test_model_log_watcher.py (new) covers should_poll_again: True across the six original running states and SCALED_TO_ZERO, False for stopped states, plus a SCALED_TO_ZERO → INACTIVE transition confirming the tail still terminates.

truss/tests/cli/test_cli.py — four added cases: truss push --wait succeeds on SCALED_TO_ZERO and keeps polling on an unknown status; --watch enters watch mode early on LOADING_MODEL; and a regression test for the tuple fix, verified to fail against the pre-fix line.

Existing DEPLOY_FAILED tests pass unchanged. TERMINAL_FAILURE_STATUSES is sorted() where it feeds @pytest.mark.parametrize, since frozenset iteration order varies with per-process string hash randomization and the repo shards with pytest-split.

Full suite (pytest -m 'not integration'): 1607 passed. Ruff, ruff-format and mypy clean via pre-commit.

🚢 Release requirements

• Pre-release: None.
• Post-release: None. action-truss-push picks the fix up automatically on the next truss release.
• External communication: Reported by a customer via a support channel; they are waiting on a release date and will be told when it ships.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[cretz]

From PR description:

The push succeeds, the deployment goes live, and ~20s later the action fails with:

So this polls once a second, does it not appear ACTIVE at some point before SCALED_TO_ZERO? Or is there a case where min_replicas means it never creates a replica at all? Just trying to understand if there's a real situation where a new model never reaches ACTIVE before scaled to zero.

[FredLiu876]

See this report:

#2579 (comment)

It seems like there's a bug with the state machine

[cretz]

It seems like there's a bug with the state machine

If this is considered a bug, can it be addressed server side?

[FredLiu876]

No, it never reports ACTIVE first. I pulled the actual status history for the deployment in the report:

Time (UTC)	Status
03:45:45.482	created
03:45:46.073	BUILDING
03:45:56.994	DEPLOYING
03:45:58.406	SCALED_TO_ZERO — client raises here, T+12.9s
03:46:09.774	WAKING_UP
03:56:16.971	ACTIVE, first time ever

ACTIVE was 10m18s in the future when the client gave up, so there was nothing to race. The observable SCALED_TO_ZERO window was 11.4s