Repository navigation
feat: add resumable 90-day backfill with cumulative query budget - #19
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds backfill90-dry-run and backfill90-refresh to the existing main-only WIF workflow. The controller processes a pinned 500-repository, 90-day range in 13 contiguous batches, defaulting to three batches per invocation. Reruns skip completed batches.
Before a paid query, it persists a full-cap budget reservation and fixed BigQuery job ID using a generation-guarded GCS checkpoint. Interruptions reattach to that job; failed uploads and cursor writes preserve the pending reservation. Completed attempts replace reservations with actual billed bytes. Explicit failed-job replacement retains the old cap in cumulative accounting. Immutable batch manifests publish after validation and uploads. Already published manifests can repair interrupted cursor updates without re-querying events.
The successful seven-day pilot is documented separately. This change does not publish the 90-day dataset to the dashboard or deploy Kafka/Spark/Airflow.
Validation
All four CI jobs passed: Python/dbt/lint, workflow syntax, frontend/browser checks and backend container smoke tests. All 90 Python tests passed, including 20 new backfill tests. Local lint and formatting checks also pass.
CI: https://github.com/yash-rai-93/argo-data-platform/actions/runs/40104342021
Required next steps
After merge, change the existing ETL account from Storage Object Creator to Storage Object User on the existing bucket only. Run backfill90-dry-run with refresh_end=2026-10-23 and total_max_bytes_billed=10000000000, then review all estimates before selecting a per-query cap. The total budget and end date remain pinned across resumptions.
See docs/BACKFILL_90_DAYS.md for exact steps, cost accounting, recovery limitations and retention configuration. Live 90-day execution remains pending.