feat: CON-1146: Add more graceful error handling

README.md

@@ -18,6 +18,7 @@ Current vLLM version: [0.29.0](https://github.com/vllm-project/vllm/releases/tag
 - [Setting up the Serverless Worker](#setting-up-the-serverless-worker)
   - [Option 1: Deploy Any Model Using Pre-Built Docker Image [Recommended]](#option-1-deploy-any-model-using-pre-built-docker-image-recommended)
     - [Configuration](#configuration)
+    - [When vLLM will not start](#when-vllm-will-not-start)
   - [Option 2: Build Docker Image with Model Inside](#option-2-build-docker-image-with-model-inside)
     - [Prerequisites](#prerequisites)
     - [Arguments](#arguments)
@@ -102,6 +103,24 @@ Mount the file anywhere into the container and point the `VLLM_CONFIG_FILE` env
 
 For the complete list of all available environment variables, examples, and detailed descriptions: **[Configuration](docs/configuration.md)**
 
+### When vLLM will not start
+
+Some startup failures a restart cannot fix: the model does not fit the GPU, `MAX_MODEL_LEN` is larger than the KV cache the GPU can hold, a flag or value vLLM rejects, a gated model without `HF_TOKEN`. Exiting on those crash-loops the worker: every attempt pays for the download and the model load, and the console shows only silent restarts with no error anywhere.
+
+The worker recognises these and stays up instead, answering every job with the cause and the fix:
+
+```json
+{
+  "error": {
+    "message": "Qwen/Qwen3-30B-A3B ran out of GPU memory during startup. This GPU has 19.57 GiB. Lower MAX_MODEL_LEN or MAX_NUM_SEQS, set ENFORCE_EAGER=true to skip CUDA graph capture, use a quantized checkpoint, or redeploy on a larger GPU (or more GPUs with TENSOR_PARALLEL_SIZE). ...",
+    "type": "startup_error",
+    "code": null
+  }
+}
+```
+
+The same message is logged by the worker, and the full vLLM traceback stays in the worker logs above it. Anything unrecognised still exits non-zero, because a failed download or a bad host is worth another attempt.
+
 ## Option 2: Build Docker Image with Model Inside
 
 To build an image with the model baked in, you must specify the following docker arguments when building the image.

docs/conventions.md

@@ -27,7 +27,9 @@ Container start:
     ├─ (optional) reads /local_model_args.json for baked-in models
     ├─ args_builder.build_vllm_args()  → env vars → CLI flags
     ├─ spawns `vllm serve --host 127.0.0.1 --port $VLLM_PORT ...`
-    ├─ polls GET /health until ready (fail fast if vLLM exits or times out)
+    ├─ polls GET /health until ready (exit non-zero if vLLM dies or times out,
+    │   unless startup_errors.classify() recognises the failure as one a restart
+    │   cannot fix — then stay up and answer every job with the cause)
     └─ starts runpod.serverless loop with handler.handler
 
 Per job:
@@ -45,6 +47,9 @@ boundary or the CLI; that is what makes vLLM upgrades a one-line `VLLM_VERSION`
 - `src/args_builder.py`: pure-Python env var → CLI flag translation (allowlist of
   `vllm serve` flags + legacy aliases + `VLLM_EXTRA_ARGS` passthrough). No third-party
   imports, so it is fully unit-testable on CPU runners.
+- `src/startup_errors.py`: regexes over the tail of vLLM's output that turn a known
+  fatal startup failure (CUDA OOM, KV cache too small for `MAX_MODEL_LEN`, rejected
+  flag, gated/missing model, out of disk) into one actionable sentence. Pure Python.
 - `src/handler.py`: the RunPod serverless handler; an aiohttp proxy that accepts three
   input shapes:
   1. `{"openai_route": ..., "openai_input": ...}` — RunPod's `/openai/*` passthrough
@@ -93,6 +98,7 @@ VLLM_EXTRA_ARGS  >  env aliases (MODEL_NAME, ...)  >  env flag scan
 src/
 ├── main.py            # Entrypoint: vLLM subprocess + RunPod loop lifecycle
 ├── args_builder.py    # env vars → `vllm serve` CLI flags (pure Python)
+├── startup_errors.py  # fatal startup failure → actionable message (pure Python)
 ├── handler.py         # RunPod handler: aiohttp proxy to the vLLM server
 └── download_model.py  # Build-time model download (Option 2)
 ```
@@ -132,8 +138,17 @@ workers at container start.
 
 ### 3. **Error Handling**
 
-- Startup failures (bad flag, missing model, OOM during load) → vLLM exits before
-  `/health` → `main.py` fails the worker fast instead of accepting jobs.
+- Startup failures → vLLM exits before `/health`. `main.py` keeps the last ~400 lines
+  of vLLM output and runs `startup_errors.classify()` over them:
+  - Recognised as unfixable by a restart (CUDA OOM, KV cache too small for
+    `MAX_MODEL_LEN`, `MAX_MODEL_LEN` above the model's limit, argparse rejection,
+    gated/missing HF repo, unsupported architecture, out of disk) → the worker stays
+    up and every job returns `{"error": {"type": "startup_error", "message": ...}}`
+    naming the cause and the fix. Crash-looping would pay for the download and load
+    on every attempt and surface nothing but silent restarts in the console.
+  - Anything else (flaky download, bad host) → exit non-zero so the platform retries.
+  Add a pattern only when a restart provably cannot help; a false positive here turns
+  a transient failure into a worker that never recovers.
 - Request failures → the proxy yields a RunPod job error containing the vLLM HTTP
   status and body (vLLM's own OpenAI-compatible error payloads flow through unchanged).
 - If the vLLM process dies while serving, the next job gets an immediate
@@ -178,6 +193,9 @@ workers at container start.
 - `tests/test_args_builder.py` covers the env var → CLI translation (mapping, aliases,
   bool handling, JSON passthrough, `VLLM_EXTRA_ARGS` precedence). Pure Python, no GPU,
   no vllm import — runs on any CPU runner with just `pytest`.
+- `tests/test_startup_errors.py` pins which vLLM failure messages are answered and
+  which are left for a restart; `tests/test_handler.py` checks the `startup_error`
+  short-circuit in the handler.
 
 ### 2. **Local Smoke Testing**
 

src/handler.py

@@ -2,7 +2,9 @@
 
 `main.py` starts `vllm serve` on 127.0.0.1:VLLM_PORT and only then starts the
 RunPod serverless loop, so by the time a job arrives the vLLM HTTP server is up
-and fully backwards/forwards compatible — we never import vLLM.
+and fully backwards/forwards compatible — we never import vLLM. The one
+exception: when vLLM could not start for a reason a restart cannot fix,
+`main.py` sets `startup_error` and every job is answered with that message.
 
 Accepted job input shapes (all under job["input"]):
 
@@ -38,6 +40,11 @@
 # instead of hanging on a dead server.
 vllm_process = None
 
+# Set by main.py when vLLM failed to start for a reason a restart cannot fix
+# (see startup_errors.py). Every job then answers with the cause instead of
+# dialling a server that is not there.
+startup_error: Optional[str] = None
+
 _default_model_cache: Optional[str] = None
 
 
@@ -103,11 +110,15 @@ def _normalize_job_input(job_input: dict) -> Tuple[str, str, Optional[dict]]:
     return DEFAULT_COMPLETION_ROUTE, "POST", body
 
 
-def _error(message: str) -> dict:
-    return {"error": {"message": message, "type": "worker_error", "code": None}}
+def _error(message: str, error_type: str = "worker_error") -> dict:
+    return {"error": {"message": message, "type": error_type, "code": None}}
 
 
 async def handler(job: dict) -> AsyncGenerator[Any, None]:
+    if startup_error:
+        yield _error(startup_error, error_type="startup_error")
+        return
+
     job_input = job.get("input") or {}
 
     try:

src/main.py

@@ -4,18 +4,27 @@
 (see args_builder.py), launch `vllm serve` on the loopback interface, poll
 /health until the server (and model) is ready, and only then start the RunPod
 serverless job loop so no job is pulled before the backend can serve it.
+
+If vLLM dies during startup for a reason a restart cannot fix (CUDA OOM, a
+MAX_MODEL_LEN the GPU cannot hold, a bad flag, a gated model), the worker stays
+up and answers every job with the cause instead of crash-looping; see
+startup_errors.py. Unrecognised failures still exit non-zero so the platform
+retries them.
 """
 
+import collections
 import json
 import logging
 import os
 import signal
 import subprocess
 import sys
+import threading
 import time
 import urllib.error
 import urllib.request
 
+import startup_errors
 from args_builder import build_vllm_args
 from download_model import LOCAL_MODEL_ARGS_PATH
 
@@ -25,9 +34,16 @@
 VLLM_PORT = os.getenv("VLLM_PORT", "8000")
 STARTUP_TIMEOUT = int(os.getenv("VLLM_STARTUP_TIMEOUT", "1200"))  # seconds
 HEALTH_POLL_INTERVAL = 2  # seconds
+# How long to wait for vLLM to exit after SIGTERM before SIGKILL.
+SHUTDOWN_GRACE = 30  # seconds
 
 vllm_process: subprocess.Popen | None = None
 
+# Enough of vLLM's output to recognise why it died. The traceback that matters
+# is always the last thing it prints.
+recent_output: collections.deque[str] = collections.deque(maxlen=400)
+output_pump: threading.Thread | None = None
+
 
 def apply_local_model_args() -> None:
     """Load args baked into the image by download_model.py (Option 2 builds).
@@ -52,13 +68,51 @@ def apply_local_model_args() -> None:
     os.environ["HF_HUB_OFFLINE"] = "1"
 
 
+def pump_output(proc: subprocess.Popen) -> None:
+    """Echo vLLM's output to our stdout while keeping the tail for diagnosis."""
+    for line in proc.stdout:  # type: ignore[union-attr]
+        sys.stdout.write(line)
+        sys.stdout.flush()
+        recent_output.append(line)
+
+
 def start_vllm() -> subprocess.Popen:
+    global output_pump
+
     argv = ["vllm", "serve", "--host", VLLM_HOST, "--port", VLLM_PORT]
     argv += build_vllm_args()
 
     logging.info("Starting vLLM: %s", " ".join(argv))
-    # Child gets our stdout/stderr so vLLM logs land in the worker logs.
-    return subprocess.Popen(argv)
+    # vLLM's stdout+stderr flow through a pipe so we can both forward them to the
+    # worker logs and keep the tail to classify a startup failure. Unbuffered so
+    # the child's log lines arrive as they are written, not when its buffer fills.
+    env = {**os.environ, "PYTHONUNBUFFERED": "1"}
+    proc = subprocess.Popen(
+        argv,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.STDOUT,
+        text=True,
+        errors="replace",
+        env=env,
+    )
+    output_pump = threading.Thread(target=pump_output, args=(proc,), daemon=True)
+    output_pump.start()
+    return proc
+
+
+def stop_vllm(proc: subprocess.Popen) -> None:
+    """Make sure a failed vLLM is gone (and its GPU memory released)."""
+    if proc.poll() is None:
+        proc.send_signal(signal.SIGTERM)
+        try:
+            proc.wait(timeout=SHUTDOWN_GRACE)
+        except subprocess.TimeoutExpired:
+            proc.kill()
+            proc.wait()
+    # Let the pump drain whatever the pipe still holds so the traceback tail
+    # is in recent_output before we look at it.
+    if output_pump is not None:
+        output_pump.join(timeout=10)
 
 
 def wait_for_vllm(proc: subprocess.Popen) -> None:
@@ -101,18 +155,29 @@ def main() -> None:
         signal.signal(sig, _forward_signal)
 
     vllm_process = start_vllm()
+    startup_error = None
     try:
         wait_for_vllm(vllm_process)
     except RuntimeError as e:
         logging.error("%s", e)
-        sys.exit(1)
+        stop_vllm(vllm_process)
+        startup_error = startup_errors.classify("".join(recent_output), model=os.getenv("MODEL_NAME"))
+        if startup_error is None:
+            # Nothing recognisable, so let the platform restart us: a failed
+            # download or a bad host is worth another attempt.
+            sys.exit(1)
+        # A restart cannot fix this one. Exiting would crash-loop the worker,
+        # paying for the download and the load on every attempt and showing the
+        # user a traceback instead of a cause, so stay up and answer jobs with it.
+        logging.error("vLLM cannot start on this configuration; answering jobs with the cause: %s", startup_error)
 
     # Import here (not at module import time) so the RunPod SDK and handler
-    # start only after the backend is confirmed healthy.
+    # start only after the backend is confirmed healthy (or confirmed dead).
     import handler as proxy_handler
     import runpod
 
     proxy_handler.vllm_process = vllm_process
+    proxy_handler.startup_error = startup_error
 
     max_concurrency = int(os.getenv("MAX_CONCURRENCY", "30"))
     runpod.serverless.start(

src/startup_errors.py

@@ -0,0 +1,118 @@
+"""Turn a failed `vllm serve` start into a sentence the user can act on.
+
+vLLM reports its problems as Python tracebacks a hundred lines long, and the
+worker only sees them as stdout. When a known one goes by we would rather answer
+the next job with the cause and the fix than let the container exit and be
+restarted into the same wall: a crash-loop pays for the download and the model
+load on every attempt and the console shows nothing but silent restarts.
+
+Only failures that a restart cannot fix belong here. Anything unrecognised is
+left alone (``classify`` returns None) so the platform retries it, which is the
+right move for a flaky download or a host that lost its GPU.
+"""
+
+import re
+from typing import Optional
+
+# --- GPU memory -------------------------------------------------------------
+# torch prints both the request and the card's capacity; the capacity is the
+# useful half, since the request is whatever happened to be next.
+_OOM = re.compile(r"torch\.OutOfMemoryError|CUDA out of memory", re.I)
+_OOM_CAPACITY = re.compile(r"total capacity of ([\d.]+) GiB", re.I)
+# vllm/v1/core/kv_cache_utils.py: the weights fit but nothing is left for the
+# KV cache, or not enough is left to hold a single request of max_model_len.
+_NO_KV_MEMORY = re.compile(r"No available memory for the cache blocks", re.I)
+_KV_TOO_SMALL = re.compile(
+    r"larger than the available KV cache memory"
+    r"|larger than the maximum number of tokens that can be stored in KV cache",
+    re.I,
+)
+_KV_ESTIMATED_LEN = re.compile(r"estimated maximum model length is (\d+)", re.I)
+
+# --- Configuration ----------------------------------------------------------
+# vllm/config/model.py: MAX_MODEL_LEN above what the model's config allows.
+_MAX_LEN_EXCEEDS_MODEL = re.compile(
+    r"User-specified max_model_len \((\d+)\) is greater than the derived max_model_len", re.I
+)
+# argparse: exit code 2 before the engine even starts.
+_BAD_ARGS = re.compile(r"(?:vllm serve|vllm): error: (.+)")
+
+# --- Model access -----------------------------------------------------------
+_GATED = re.compile(r"GatedRepoError|401 Client Error", re.I)
+_NOT_FOUND = re.compile(r"RepositoryNotFoundError|404 Client Error", re.I)
+_UNSUPPORTED_ARCH = re.compile(r"Model architectures \[.*?\] (?:are not supported|failed to be inspected)", re.I)
+
+# --- Disk -------------------------------------------------------------------
+_NO_SPACE = re.compile(r"No space left on device|ENOSPC|errno 28", re.I)
+
+
+def classify(output: str, model: Optional[str] = None) -> Optional[str]:
+    """One actionable message for a known fatal failure, or None to let it retry."""
+    named = model or "The model"
+
+    # Memory problems first: an OOM traceback is often surrounded by secondary
+    # errors (engine core died, connection reset) that would match nothing.
+    if _OOM.search(output) or _NO_KV_MEMORY.search(output) or _KV_TOO_SMALL.search(output):
+        capacity = _OOM_CAPACITY.search(output)
+        card = f" This GPU has {capacity.group(1)} GiB." if capacity else ""
+        estimated = _KV_ESTIMATED_LEN.search(output)
+        hint = (
+            f" vLLM estimates this GPU can serve a context of about {estimated.group(1)} tokens."
+            if estimated
+            else ""
+        )
+        return (
+            f"{named} ran out of GPU memory during startup.{card}{hint} "
+            f"Lower MAX_MODEL_LEN or MAX_NUM_SEQS, set ENFORCE_EAGER=true to skip "
+            f"CUDA graph capture, use a quantized checkpoint, or redeploy on a "
+            f"larger GPU (or more GPUs with TENSOR_PARALLEL_SIZE). If the model "
+            f"loaded but the KV cache did not fit, raising GPU_MEMORY_UTILIZATION "
+            f"a little (default 0.9) can also help."
+        )
+
+    match = _MAX_LEN_EXCEEDS_MODEL.search(output)
+    if match:
+        return (
+            f"MAX_MODEL_LEN={match.group(1)} is larger than the context length "
+            f"{named} declares in its config.json. Lower MAX_MODEL_LEN (or unset it "
+            f"to use the model's own limit). To override the limit anyway, set "
+            f"VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 and expect degraded output past it."
+        )
+
+    match = _BAD_ARGS.search(output)
+    if match:
+        return (
+            f"vLLM rejected its command line: {match.group(1).strip()} "
+            f"Check the environment variables that map to vLLM flags and "
+            f"VLLM_EXTRA_ARGS; run `vllm serve --help` in the image for the "
+            f"accepted flags and values."
+        )
+
+    if _GATED.search(output):
+        return (
+            f"{named} is gated or private on Hugging Face and the worker was not "
+            f"allowed to download it. Set HF_TOKEN to a token whose account has "
+            f"accepted the model's license, or pick a public model."
+        )
+
+    if _NOT_FOUND.search(output):
+        return (
+            f"{named} was not found on Hugging Face. Check MODEL_NAME for typos "
+            f"(it must be the full `org/repo` id) and MODEL_REVISION if set. Private "
+            f"repositories also return not-found until HF_TOKEN grants access."
+        )
+
+    if _UNSUPPORTED_ARCH.search(output):
+        return (
+            f"{named} uses an architecture this vLLM version cannot serve. Check "
+            f"the vLLM supported-models list, or try a newer worker release."
+        )
+
+    if _NO_SPACE.search(output):
+        return (
+            f"{named} ran out of disk while downloading. Increase the endpoint's "
+            f"container disk to comfortably exceed the size of the repository, or "
+            f"attach a network volume so the weights are cached there instead."
+        )
+
+    return None