Skip to content

feat(relayer): persist congestion requeue telemetry on remember_jobs - #1351

Open
harrymove-ctrl wants to merge 1 commit into
devfrom
feat/remember-jobs-congestion-requeue-visibility
Open

harrymove-ctrl wants to merge 1 commit into
devfrom
feat/remember-jobs-congestion-requeue-visibility

Conversation

@harrymove-ctrl

Copy link
Copy Markdown
Collaborator

Summary

Persists upload congestion requeue telemetry (congestion_requeues, next_attempt_at, and upload_wallet_index) directly on the remember_jobs row instead of keeping it only inside the serialized Apalis payload and log stream.

  1. Migration (028_remember_jobs_congestion_requeues.sql):
    • Adds congestion_requeues INTEGER NOT NULL DEFAULT 0 and next_attempt_at TIMESTAMPTZ to remember_jobs using metadata-only fast defaults.
  2. Worker Congestion Telemetry (services/server/src/jobs.rs):
    • Adds record_congestion_requeue_on_remember_job and calls it across all 4 congestion requeue paths (schedule_metadata_congestion_retry, PreparedUpload::Deferred, durable upload congestion backoff, and atomic upload congestion backoff).
    • Preserves a pinned journal upload_wallet_index when a durable upload checkpoint (upload_wallet_address, upload_register_transaction, or upload_resume_step) is already recorded, and updates upload_wallet_index to the rotated wallet when the journal is uncommitted.
    • Clears next_attempt_at when an attempt starts running or transitions to uploaded, done, or failed, while retaining congestion_requeues and upload_wallet_index on the job row for operational visibility.
    • Resets congestion_requeues = 0 and next_attempt_at = NULL in unpaid_analyze_retry_reset_sql when an unpaid failed analyze job is reset to pending.
  3. Tests (services/server/src/jobs.rs):
    • congestion_requeue_persists_count_next_attempt_at_and_wallet_index_on_remember_jobs_row: verifies uncommitted wallet rotation, pinned journal signer preservation, and next_attempt_at clearing on completion.
    • burst_inflow_41_per_minute_against_5_wallets_completes_via_congestion_requeues_with_zero_failures: verifies a 41 req/min burst against 5 wallets (~16 uploads/min capacity, threshold = 20, consecutive = 4 saturation monitor) completes all 41 jobs with zero congestion failures and records congestion_requeues and upload_wallet_index on every overflow row.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants