[Bug] Rate limiter records the per-delegate-key entry before a later layer can deny, and the "roll back" the comment promises does not exist #1002

Surface

Relayer / hosted API

Network

Mainnet (relayer.memory.walrus.xyz)

Package version

Relayer at dev f248bd37 (services/server, crate version 0.1.0)

What happened?

rate_limit_middleware runs three layers in sequence. Layer 1 atomically checks and records the per-delegate-key window entry. Layer 2 (per-account burst) can then return Denied — and the comment at that branch states the layer-1 entry is rolled back. No rollback exists: there is no ZREM or any other removal call between the two Lua invocations. Every request denied at the account layer therefore permanently consumes per-delegate-key budget for the rest of the 60 s window, and the 429 the caller eventually receives is labelled layer: "delegate_key" even though the delegate-key budget was only consumed because of the account-layer denial.

Steps to reproduce

// services/server/src/rate_limit.rs:505-537 — layer 2 denial
Ok(WindowCheckResult::Denied) => {
    // Roll back the delegate-key window entry just recorded above
    // (best-effort; a Lua multi-key script would be fully atomic across keys)
    return rate_limit_response("account_burst", config.max_requests_per_minute, "min", 60);
}

grep -n "ZREM\|rollback\|roll_back" services/server/src/rate_limit.rs
# -> one hit only: ZREMRANGEBYSCORE inside the sliding-window Lua (normal pruning)
#    plus a test asserting that Lua string contains it. No rollback call exists.

Runtime demonstration (unit test in the file's existing mod tests, next to concurrent_same_millisecond_requests_respect_limit):

1. Set a delegate-key budget of N and an account budget of 1.
2. Send two requests in the same window: the first is allowed by both layers; the second is denied at layer 2.
3. Assert ZCARD rate:dk:<public_key> — it still counts both, i.e. the denied request spent delegate-key budget.

Novelty

• charge_explicit_weight appears to prune current rate-limit history by using window_start = now #609 (rate-limit history pruning) and remember() rate limit (30 weighted-requests/min) is completely undocumented #686 (limit undocumented) are the nearest titles; neither covers a denial consuming a sibling layer's budget, and neither is about the missing rollback.
• security(server): unbounded namespace causes oversized inserts to fail and get retried indefinitely instead of being rejected upfront #493 (unbounded namespace retried indefinitely) is a different counter entirely.
• No issue mentions rate:dk: accounting or a rollback between layers.

SUGGESTED FIX

Move all three layers into a single Lua script keyed on the three windows so a denial rolls back the entries the same invocation already recorded, or (cheaper) explicitly ZREM the layer-1 member when a later layer denies — and while there, make the returned layer field name the layer that actually denied.

Expected

Either the layer-1 entry is removed when a later layer denies (as the comment claims), or the budgets are evaluated in one atomic multi-key script so a denied request consumes nothing in a sibling layer.

Actual

The denial consumes delegate-key budget, so a client that momentarily trips the account cap finds its tighter, fail-closed per-key budget pre-drained: an unrelated retry later in the same minute is rejected with layer: "delegate_key", pointing the user at the wrong remediation.

Logs or error text

429 {"error":"Rate limit exceeded","layer":"delegate_key",
     "limit":"60 weighted-requests/min","retry_after_seconds":60}

(the layer label is what misattributes the denial; the delegated budget was drained by an account-layer denial)

Checks

• I searched existing issues and this is not a duplicate.
• This report contains no private keys, mnemonics, or other secrets.