fix(grok): fail over OAuth credential errors safely

[heathermhuang]

Summary

This is the request-path half of the Grok OAuth incident fix in #4189, extended with the request-path defects confirmed in #4215 and #4213.

The verified #4189 failure mode was that Grok token acquisition returned ordinary errors, while handler account switching only recognized the upstream failover signal. A revoked OAuth credential could therefore be disabled but still terminate the request as a generic 502 instead of selecting another healthy account.

This draft:

• adds a typed, backward-compatible credential failover contract with stage, scope, stable reason, and next-account semantics;
• centralizes Grok credential acquisition and failure mapping across Responses, Messages/chat bridge, raw Chat Completions, WebSocket fallback, images, and media;
• separates account-permanent, account-transient, provider/configuration, and parent-cancellation outcomes;
• uses credential/proxy compare-and-swap mutations plus atomic scheduler outbox publication so concurrent reauthorization or proxy repair wins;
• waits for authoritative refreshed state on lock contention and never reuses the selected expired token;
• bounds credential failover latency and returns a fixed, redacted 503 when the candidate pool is exhausted;
• records credential failures as account_auth with inference upstream_status=0, including recovered provider-health rows;
• pins Grok OAuth text and media traffic to the CLI proxy while preserving the separate third-party API-key routing and URL-policy path;
• preserves existing 402/403/429 inference handling and fixes the pre-cancel release callback race exposed by the cancellation tests.

Additional issue coverage

#4215 — fixed in this draft

• A first Grok OAuth 429 permits exactly one different-account follow-up. Any failure from that follow-up stops failover, including mixed 429/5xx sequences and OAuth-to-API-key pool transitions.
• The bound is applied consistently to Responses, Messages/chat bridge, raw Chat Completions, WebSocket fallback, images, and the dedicated media loop; tests include a third eligible account to prove the pool is not swept.
• Repeated OAuth 429 cooldowns adapt from 2 minutes to 10 minutes, 30 minutes, then a 1-hour cap while preserving authoritative later reset boundaries. API-key cooldown behavior remains unchanged.
• A successful 2xx, including a headerless active probe, clears only the exact observed (rate_limited_at, reset_at) Grok OAuth generation. The repository CAS rejects newer generations, shorter clear/re-arm generations, and rows retyped to API-key while the request was in flight.

#4213 — fixed in this draft

Current main already accepts public HTTPS custom Grok API-key hosts. This draft completes that support across live Responses/chat/media, account tests, and quota probes by applying the configured global URL policy consistently:

• insecure HTTP is accepted only when the global policy allows it;
• enabled allowlists and private-host policy are enforced;
• rejected URLs return a stable redacted error;
• OAuth accounts remain pinned to cli-chat-proxy.grok.com and never inherit the API-key URL policy.

#4199 — evidence gap, not claimed fixed

The report contains a redacted generic 502 but still lacks the auth type, failing hop, version, payload shape, and same-key/model/body direct-vs-Sub2API reproduction needed to prove a Sub2API defect. This draft includes adjacent API-key routing, auth-header, URL-policy, and 429 non-regression coverage, but #4199 should remain open pending reproducible evidence.

PR #4221 remains independent and unchanged; it covers proactive OAuth pool health rather than request-path behavior.

Safety boundaries

• No OAuth request bypasses cli-chat-proxy.grok.com.
• Parent cancellation/deadline performs no account mutation and selects no further account.
• Shared provider/configuration failures do not mass-disable accounts.
• Public errors and Ops fields are redacted; no credential or private endpoint is included here.
• This PR does not merge or deploy anything.

Verification

All commands below passed locally after rebasing onto current origin/main (74e78c3de0784747fe079d448f088a61a01ff29f):

• focused service, handler, media, mixed-status, mixed-auth, URL-policy, cancellation, redaction, and API-key non-regression tests;
• focused repository integration test for exact-generation clear, shorter clear/re-arm rejection, and OAuth-to-API-key retype rejection;
• make test-unit;
• make test-integration;
• go test -race -tags=unit ./internal/service -run 'TestShouldStopOpenAIOAuth429Failover|TestUpdateGrokUsageFromResponse|TestForwardGrokResponsesNonStreamingUsesCacheIdentityAndCachedUsage|TestGrokQuotaServiceProbeUsageStoresNoHeadersState|TestGrokRateLimitResetAtForAccount' -count=1;
• go test -race -tags=unit ./internal/handler -run 'TestResponsesGrok429FailoverIsBounded|TestResponsesGrok429FailoverHandlesMixedStatuses|TestGrokMedia429FailoverIsBounded|TestGrokOAuthCredentialFailoverAcrossHTTPHandlers' -count=1;
• go test -race -tags=integration ./internal/repository -run 'TestAccountRepoSuite/TestClearRateLimitIfObservedProtectsRearmed429Generation' -count=1;
• go vet ./internal/pkg/xai ./internal/service ./internal/handler ./internal/repository;
• git diff --check origin/main...HEAD.

Independent backend, SRE/concurrency, API-test, request-path, and final full-diff reviews found no remaining actionable issue.

Part of #4189. Fixes #4215. Fixes #4213.

[heathermhuang]

Follow-up review for the newly reported Grok issues is now included in this draft.

• Grok free 账号触发限流之后，一分钟之后又恢复了，导致频繁被使用触发限流 #4215: request-local OAuth 429 failover now permits exactly one different-account follow-up across text, WebSocket, images, and dedicated media handlers. Mixed 429/5xx and OAuth/API-key pool sequences are covered, and adaptive cooldown recovery uses an exact Grok OAuth generation/identity CAS.
• Grok 第三方api 在docke版本下 http 还有 Invalid Grok base URL: invalid base url: invalid url scheme: http 等问题 #4213: Grok API-key Responses/chat/media/account-test paths now consistently apply the configured global URL policy, while OAuth remains pinned to the CLI gateway.
• Grok 链接过程报502错误 #4199: not claimed fixed. The available report still does not establish a reproducible Sub2API defect; adjacent API-key paths have non-regression coverage only.
• PR fix(grok): refresh OAuth pools proactively #4221 remains independent and unchanged.

The branch was rebased onto current origin/main (74e78c3d) and force-updated with an exact lease. Post-rebase focused tests, full unit/integration suites, targeted race tests, vet, and diff checks all passed locally. All PR checks are now green; nothing has been merged or deployed.

[heathermhuang]

@Wei-Shaw This is ready for maintainer review. Revalidated head 4c764d3: CI and Security Scan green; exact-head tests plus combined full backend unit/integration, vet, frontend typecheck/lint/build, and 1,018 frontend tests passed. A private combined deployment passed separate Grok OAuth and API-key text plus image canaries with expected usage attribution. Recent #4254 and #4263 overlap only bounded slices and do not replace this PRs credential-failure, CAS, URL-policy, media, and cross-handler scope. Please review when convenient.