This runbook provides site reliability engineers and operators with procedures for monitoring, diagnosing, and resolving incidents across the ADA platform.
When JSON_LOGGING=true (default in production), the application emits RFC 8259 compliant single-line JSON logs to standard output:
{
"timestamp": "2026-10-04T12:00:00Z",
"level": "INFO",
"service": "ada-backend",
"environment": "production",
"logger": "ada.access",
"message": "method=POST path=/api/v1/eta/calculate status=200 latency=0.0450s request_id=6f8d39e2-12a4-4c8d-9ef2-1b1d44c9b910",
"request_id": "6f8d39e2-12a4-4c8d-9ef2-1b1d44c9b910"
}The logger automatically intercepts and sanitizes:
- Authorization headers (
Bearer [REDACTED]) - API keys, OAuth tokens, and passwords in log arguments
- Precise destination addresses in telemetry metric labels
| Endpoint | Method | Purpose | Response |
|---|---|---|---|
/health/live |
GET |
Process liveness probe | 200 OK {"status": "alive"} |
/health/ready |
GET |
Dependency readiness probe | 200 OK {"status": "ready"} or 503 Unavailable |
/health/deps |
GET |
Diagnostic latency breakdown | 200 OK with latency per service (Postgres, Redis, Maps, AI) |
/metrics |
GET |
Prometheus telemetry scrape | Standard Prometheus format or JSON (?format=json) |
| Metric Name | Type | Warning Threshold | Critical Action |
|---|---|---|---|
ada_db_pool_checked_out |
Gauge | > 80% of pool size for 2m |
Investigate long-running queries or connection leak. |
ada_db_pool_overflow |
Gauge | > 10 for 1m |
Increase DB_POOL_SIZE or tune query concurrency. |
ada_uptime_seconds |
Counter | Decreasing | Unexpected container restarts / OOM kills. |
celery_task_failures |
Counter | > 5% failure rate |
Check third-party provider status (Google Maps / VAPID). |
http_requests_5xx |
Counter | > 1% over 5m |
Check application error logs filtering by level: ERROR. |
Symptom: API requests fail with TimeoutError: QueuePool limit of size 15 overflow 25 reached, connection timed out.
Immediate Steps:
- Check active database connections:
SELECT pid, state, age(clock_timestamp(), query_start), query FROM pg_stat_activity WHERE state != 'idle' ORDER BY query_start ASC;
- Terminate rogue / blocked queries:
SELECT pg_terminate_backend(<PID>);
- Adjust pool configuration in
.envif legitimate workload has scaled:DB_POOL_SIZE=30 DB_MAX_OVERFLOW=50 DB_POOL_TIMEOUT=45
- Restart API containers:
docker compose -f docker-compose.prod.yml restart api
Symptom: Alarms are not recalculating; notifications are delayed; /health/ready reports 503.
Immediate Steps:
- Check Redis process status:
docker compose -f docker-compose.prod.yml exec redis redis-cli ping - Inspect Celery queue backlog:
docker compose -f docker-compose.prod.yml exec celery_worker celery -A app.worker.celery_app.celery inspect active docker compose -f docker-compose.prod.yml exec celery_worker celery -A app.worker.celery_app.celery inspect reserved
- Restart Redis and Worker processes:
docker compose -f docker-compose.prod.yml restart redis celery_worker
- Verify the
reconcile_missed_alarmsperiodic task marks past elapsed tasks cleanly rather than firing late notifications.
Symptom: A user reports an alarm did not sound or calculate their departure time.
Diagnostic Steps:
- Locate the alarm ID and user ID in the database:
SELECT id, title, target_arrival_time, calculated_leave_time, calculated_wake_time, status, schedule_version FROM alarms WHERE user_id = '<USER_UUID>' ORDER BY created_at DESC LIMIT 5;
- Trace schedule changes for the alarm:
SELECT previous_wake_time, new_wake_time, change_reason, created_at FROM alarm_schedule_changes WHERE alarm_id = '<ALARM_UUID>' ORDER BY created_at DESC;
- Check notification delivery attempts:
SELECT id, notification_type, channel, status, retry_count, error_message, sent_at FROM notifications WHERE alarm_id = '<ALARM_UUID>';
- If browser push delivery failed with
410 GoneorInvalidSubscription:- The user uninstalled the PWA or revoked browser notification permissions.
- The subscription has been marked inactive. Instruct the user to re-enable notifications in Settings.
Symptom: Real-time traffic recalculations fail with OVER_QUERY_LIMIT or 429 Too Many Requests.
Remediation:
- The system automatically falls back from Google Maps to Mapbox if
ENABLE_PROVIDER_FALLBACK=true. - Inspect fallback logs:
docker compose -f docker-compose.prod.yml logs api | grep "Provider fallback"
- If primary key quota is exhausted, switch primary provider in
.env:ROUTING_PROVIDER=mapbox ROUTING_FALLBACK_PROVIDER=google
- Apply changes without downtime:
docker compose -f docker-compose.prod.yml up -d --no-deps api celery_worker