Skip to content

feat: prepare 500-repository GCS pilot and expansion runbook - #18

Merged
yash-rai-93 merged 1 commit into
mainfrom
feat/data-foundation
Oct 24, 2026
Merged

yash-rai-93 merged 1 commit into
mainfrom
feat/data-foundation

Conversation

@yash-rai-93

@yash-rai-93 yash-rai-93 commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Adds the first executable milestone for expanding ARGO to 500 repositories: a versioned scope retaining all 24 existing projects, a credential-free 90-day batch planner, and an isolated seven-day BigQuery-to-GCS pilot.

The existing manual workflow gains pilot-dry-run and pilot-refresh operations under the same main-branch WIF trust. Apply re-estimates the query, enforces an explicit cap of at most 20 GB, checks GCS write access before the paid query, validates and deduplicates records on disk, then writes an immutable completion manifest last. Partial attempts never become checkpoints. This does not publish or change the live dashboard.

Validation

All four CI jobs passed: Python/dbt/lint, workflow syntax, frontend/browser checks and backend container smoke tests. Python: 70 tests passed, including 16 new pilot tests. Local scope planning confirms 500 unique repository IDs and 13 continuous batches for 90 days. Lint and formatting pass.

CI: https://github.com/yash-rai-93/argo-data-platform/actions/runs/40101083609

Cloud prerequisites and next milestone

Verify remaining GCP credits/expiry; configure a private US multi-region bucket, bucket-scoped Storage Object Creator for the existing ETL service account, and the GCP_DATA_BUCKET Actions variable. After merge, run pilot-dry-run with an explicit exclusive end date; inspect its estimate before running pilot-refresh with an explicit bytes cap.

Live estimates, paid ingestion and actual IAM remain unverified until the workflow is merged and configured. No Kafka, Spark or Airflow deployment is claimed in this milestone. The 90-day planner is implemented; resumable multi-batch ingestion and production promotion are subsequent milestones.

See docs/STREAMING_EXPANSION.md for setup, exact run steps, source/selection limitations, failure recovery and the remaining implementation milestones.

@vercel

vercel Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
argo-data-platform Ready Ready Preview Oct 24, 2026 2:07pm UTC

@yash-rai-93
yash-rai-93 marked this pull request as ready for review October 24, 2026 14:09
@yash-rai-93
yash-rai-93 merged commit 0401acb into main Oct 24, 2026
6 checks passed
@yash-rai-93
yash-rai-93 deleted the feat/data-foundation branch October 24, 2026 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant