[MAINTENANCE] Discover stale BigQuery test datasets via datasets.list instead of INFORMATION_SCHEMA - #12017
Closed
joshua-stauffer wants to merge 1 commit into
Closed
[MAINTENANCE] Discover stale BigQuery test datasets via datasets.list instead of INFORMATION_SCHEMA#12017joshua-stauffer wants to merge 1 commit into
joshua-stauffer wants to merge 1 commit into
Conversation
…MATION_SCHEMA INFORMATION_SCHEMA.SCHEMATA is a project-level view, so querying it requires metadata read access across the whole project even though the sweep only needs to see datasets it created itself. It's also region-scoped, so a dataset created in a different location is invisible to the query and never gets cleaned up. The datasets.list API has neither limitation: it returns exactly the datasets the caller can see, works with a credential scoped to just the CI dataset namespace, and isn't tied to a single region. Age filtering now happens client-side against Dataset.created (fetched via get_dataset, since list_datasets doesn't include creation time), and the age threshold is an injectable parameter so it can be exercised in tests without waiting an hour. Deletion uses delete_dataset(..., delete_contents=True) as the equivalent of the previous DROP SCHEMA ... CASCADE, and tolerates a dataset disappearing between listing and deletion.
✅ Deploy Preview for niobium-lead-7998 canceled.
|
Collaborator
Author
|
wrong approach - datasets.list is too slow for this purpose |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rewrites the BigQuery test-dataset cleanup script to discover stale datasets via the
datasets.listAPI (throughgoogle-cloud-bigquery'sClient) instead of queryingINFORMATION_SCHEMA.SCHEMATAover a SQLAlchemy connection.Behavior is otherwise unchanged:
gx_ci_test_[a-f0-9]{10},py3[0-9]{1,2}_i[a-f0-9]{32}).delete_dataset(..., delete_contents=True)in place ofDROP SCHEMA ... CASCADE).Also adds a per-race guard: if a dataset disappears between being listed and being deleted, the sweep logs and continues instead of aborting.
Why
Two problems with querying
INFORMATION_SCHEMA.SCHEMATAdirectly:INFORMATION_SCHEMA.SCHEMATAis a project-level view, so reading it requires permission to see dataset metadata across the entire project, even though this job only ever needs to see the datasets it itself creates. Thedatasets.listAPI only returns datasets the caller can already see, so a credential scoped to just a naming-prefix namespace works correctly with it, but gets403 Access DeniedagainstINFORMATION_SCHEMA.SCHEMATA.INFORMATION_SCHEMAqueries are scoped to the region they run in, so a dataset created in a different location is invisible to the query and never gets swept.datasets.listhas no such scoping.Since
datasets.listresults don't include a dataset's creation time, the sweep now does aget_datasetper name-matching candidate to readDataset.createdbefore applying the age filter.User impact
None — this is a CI-only maintenance script with no effect on the
great_expectationspackage or its public API.How to review
find_stale_dataset_ids: pattern match ondataset_id, then age-filter onDataset.created, withNotFoundtolerated at both the per-dataset lookup and the delete step.tests/scripts/test_cleanup_big_query.pycovers the selection logic (pattern matching, age boundaries including a zero-threshold edge case, and the two race windows) against a mocked client.list_datasets/get_dataset/delete_dataseton its own namespace, confirmed a non-matching permanent dataset is never selected regardless of age, confirmed a freshly created dataset survives the default threshold and is picked up under a zero threshold, and confirmed cascade delete removes a dataset's contents.GE_TEST_BIGQUERY_DATASETis no longer read by the script (there's no default dataset to connect through anymore); it's harmless for the workflow to keep setting it as an unused env var.