Skip to content

feat: add resumable 90-day backfill with cumulative query budget - #19

Merged
yash-rai-93 merged 1 commit into
mainfrom
feat/resumable-backfill
Oct 24, 2026
Merged

yash-rai-93 merged 1 commit into
mainfrom
feat/resumable-backfill

Conversation

@yash-rai-93

@yash-rai-93 yash-rai-93 commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Adds backfill90-dry-run and backfill90-refresh to the existing main-only WIF workflow. The controller processes a pinned 500-repository, 90-day range in 13 contiguous batches, defaulting to three batches per invocation. Reruns skip completed batches.

Before a paid query, it persists a full-cap budget reservation and fixed BigQuery job ID using a generation-guarded GCS checkpoint. Interruptions reattach to that job; failed uploads and cursor writes preserve the pending reservation. Completed attempts replace reservations with actual billed bytes. Explicit failed-job replacement retains the old cap in cumulative accounting. Immutable batch manifests publish after validation and uploads. Already published manifests can repair interrupted cursor updates without re-querying events.

The successful seven-day pilot is documented separately. This change does not publish the 90-day dataset to the dashboard or deploy Kafka/Spark/Airflow.

Validation

All four CI jobs passed: Python/dbt/lint, workflow syntax, frontend/browser checks and backend container smoke tests. All 90 Python tests passed, including 20 new backfill tests. Local lint and formatting checks also pass.

CI: https://github.com/yash-rai-93/argo-data-platform/actions/runs/40104342021

Required next steps

After merge, change the existing ETL account from Storage Object Creator to Storage Object User on the existing bucket only. Run backfill90-dry-run with refresh_end=2026-10-23 and total_max_bytes_billed=10000000000, then review all estimates before selecting a per-query cap. The total budget and end date remain pinned across resumptions.

See docs/BACKFILL_90_DAYS.md for exact steps, cost accounting, recovery limitations and retention configuration. Live 90-day execution remains pending.

@vercel

vercel Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
argo-data-platform Ready Ready Preview Oct 24, 2026 2:51pm UTC

@yash-rai-93
yash-rai-93 marked this pull request as ready for review October 24, 2026 14:53
@yash-rai-93
yash-rai-93 merged commit 811f1db into main Oct 24, 2026
6 checks passed
@yash-rai-93
yash-rai-93 deleted the feat/resumable-backfill branch October 24, 2026 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant