Repository navigation
feat: prepare 500-repository GCS pilot and expansion runbook - #18
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the first executable milestone for expanding ARGO to 500 repositories: a versioned scope retaining all 24 existing projects, a credential-free 90-day batch planner, and an isolated seven-day BigQuery-to-GCS pilot.
The existing manual workflow gains pilot-dry-run and pilot-refresh operations under the same main-branch WIF trust. Apply re-estimates the query, enforces an explicit cap of at most 20 GB, checks GCS write access before the paid query, validates and deduplicates records on disk, then writes an immutable completion manifest last. Partial attempts never become checkpoints. This does not publish or change the live dashboard.
Validation
All four CI jobs passed: Python/dbt/lint, workflow syntax, frontend/browser checks and backend container smoke tests. Python: 70 tests passed, including 16 new pilot tests. Local scope planning confirms 500 unique repository IDs and 13 continuous batches for 90 days. Lint and formatting pass.
CI: https://github.com/yash-rai-93/argo-data-platform/actions/runs/40101083609
Cloud prerequisites and next milestone
Verify remaining GCP credits/expiry; configure a private US multi-region bucket, bucket-scoped Storage Object Creator for the existing ETL service account, and the GCP_DATA_BUCKET Actions variable. After merge, run pilot-dry-run with an explicit exclusive end date; inspect its estimate before running pilot-refresh with an explicit bytes cap.
Live estimates, paid ingestion and actual IAM remain unverified until the workflow is merged and configured. No Kafka, Spark or Airflow deployment is claimed in this milestone. The 90-day planner is implemented; resumable multi-batch ingestion and production promotion are subsequent milestones.
See docs/STREAMING_EXPANSION.md for setup, exact run steps, source/selection limitations, failure recovery and the remaining implementation milestones.