fix(x402): inject agent upstream auth

[bussyjd]

Summary

Fixes the obol sell agent paid path where the x402 verifier successfully verifies payment, forwards the request to the seller's per-agent Hermes upstream, and then receives 401 Invalid API key because Hermes requires API_SERVER_KEY but the verifier had no way to inject it.

The fix keeps the current security boundary: Hermes remains bearer-token protected, buyers never see the key, and the verifier injects Authorization: Bearer <API_SERVER_KEY> only after x402 verification has accepted the request.

This PR also pins the embedded verifier deployment to a branch-built multi-arch image containing this source change:

ghcr.io/obolnetwork/x402-verifier:46e63fd@sha256:a8cd7946884c9a702b5cfcfad28d1f5eac1037899303eb4e0157e3ffab7a572c

Architecture

Before this change, agent offers produced a valid paid route, but the route source only knew how to read LiteLLM upstream credentials:

sequenceDiagram
    participant Buyer
    participant Verifier as x402-verifier
    participant SO as ServiceOffer route source
    participant Hermes as per-agent Hermes

    SO->>SO: Watch ServiceOffers
    SO->>SO: Watch litellm-secrets only
    SO->>Verifier: RouteRule{UpstreamURL: Hermes, UpstreamAuth: empty}
    Buyer->>Verifier: POST /services/agent/... + X-PAYMENT
    Verifier->>Verifier: Verify x402 payment
    Verifier->>Hermes: Forward request without Authorization
    Hermes-->>Verifier: 401 Invalid API key
    Verifier-->>Buyer: 401, no settlement

Loading

After this change, the verifier watches the per-agent API secret and uses the existing RouteRule.UpstreamAuth injection mechanism:

sequenceDiagram
    participant Buyer
    participant Verifier as x402-verifier
    participant SO as ServiceOffer route source
    participant Secret as hermes-api-server Secret
    participant Hermes as per-agent Hermes

    SO->>Secret: Watch hermes-api-server in agent namespaces
    SO->>SO: Decode API_SERVER_KEY
    SO->>Verifier: RouteRule{UpstreamURL: Hermes, UpstreamAuth: Bearer key}
    Buyer->>Verifier: POST /services/agent/... + X-PAYMENT
    Verifier->>Verifier: Verify x402 payment
    Verifier->>Hermes: Forward request with Authorization
    Hermes-->>Verifier: 200 response
    Verifier->>Verifier: Settle after upstream success
    Verifier-->>Buyer: 200 + X-PAYMENT-RESPONSE

Loading

Changes

• Split the verifier secret watch into scoped informers for litellm-secrets and hermes-api-server.
• Preserve LiteLLM auth behavior unchanged.
• For type=agent ServiceOffers, resolve UpstreamAuth from spec.agent.ref.namespace so cross-namespace references use the agent namespace, not the offer namespace.
• Extend the x402-verifier ClusterRole to read/watch only the additional hermes-api-server Secret name.
• Pin the embedded verifier image to the branch image that includes this fix.
• Add regression coverage for normal and cross-namespace agent auth routing, RBAC, and the production image pin.

Validation

go test ./internal/x402 -run 'TestRoutesFromStore|TestRouteRuleFromOffer_Agent' -count=1
go test ./internal/embed -run 'TestX402VerifierRBAC_CanReadAgentAPISecrets|TestX402VerifierImage_CarriesAgentAuthFix|TestEmbeddedImages_NamedImagesAreDigestPinned' -count=1

Image publication:

gh workflow run docker-publish-x402.yml --ref fix/agent-seller-upstream-auth
docker buildx imagetools inspect ghcr.io/obolnetwork/x402-verifier:46e63fd --format '{{ .Manifest.Digest }}'
# sha256:a8cd7946884c9a702b5cfcfad28d1f5eac1037899303eb4e0157e3ffab7a572c

Remaining smoke gate

Full flow smoke is running from a combined branch with forced local dev images, but it is currently blocked on the local sudo prompt. The LLM-gated dual-stack flows are also blocked because silvermesh.v1337.lan:8081 is reachable on the LAN but refusing TCP connections.

[OisinKyne]

I hope those service offers really can't leak those secrets.... maybe lets talk about it on sync