Add Google Places ETL pipelines by TristanGramsch · Pull Request #19 · codeforboston/boston-circular-economy

TristanGramsch · 2026-05-06T19:14:44Z

Add ETL support for Google Places repair shops and donation centers.
Implement queriers and normalizers backed by live Google Places search responses.
Add tests using the checked-in sample payloads and verify live API responses locally.

Verification

uv run pytest pipelines/google_places/test_pipeline.py
Live Google Places fetch and normalization succeeded for both pipelines.

TristanGramsch · 2026-05-06T19:16:20Z

Sorry about the ugly format of the body.

cnapolit · 2026-05-12T14:23:05Z

+
+class GooglePlacesRepairQuerier(GooglePlacesQuerier):
+    def __init__(self, api_key: str | None = None):
+        super().__init__(api_key=api_key, text_query="shoe repair Boston MA", data_source="google_places_repair")


Should this be hard-coded to shoe repair?

MattClarke131

Overall this is great work! My comments are mostly just rearranging code to follow pattern, but this is 95% there. The tests inspires confidence ^^

Thanks for your volunteer effort! It's great to see data work taking off 😊

MattClarke131 · 2026-05-18T04:39:55Z

+class GooglePlacesRepairNormalizer(BaseNormalizer):
+    def normalize(self, raw_locations: list[RawLocation]) -> list[NormalizedLocation]:
+        return [
+            NormalizedLocation(**normalize_google_place(
+                raw.payload,
+                data_source=raw.data_source,
+                data_source_id=raw.data_source_id,
+                item_category=ItemCategory.SHOES,
+                activity=Activity.REPAIR_PAID,
+            ))
+            for raw in raw_locations
+        ]
+
+
+class GooglePlacesDonationNormalizer(BaseNormalizer):
+    def normalize(self, raw_locations: list[RawLocation]) -> list[NormalizedLocation]:
+        return [
+            NormalizedLocation(**normalize_google_place(
+                raw.payload,
+                data_source=raw.data_source,
+                data_source_id=raw.data_source_id,
+                item_category=ItemCategory.CLOTHING,
+                activity=Activity.DONATION_DROP,
+            ))
+            for raw in raw_locations
+        ]


📝 Changes requested: Let's remove these and create a single GooglePlacesNormalizer.

I would follow the pattern you did with GooglePlacesQuerier here: You have a class that takes init parameters and passes those through from the querier to the normalizer.

When it comes time to run these jobs, we might have something like the following.

queries = [ GooglePlacesQuery( text_query="shoe repair Boston MA", data_source="google_places_repair", item_category=ItemCategory.SHOES, activity=Activity.REPAIR_PAID ), GooglePlacesQuery( text_query="donation centers Boston MA", data_source="google_places_donations", item_category=ItemCategory.CLOTHING, activity=Activity.DONATION_DROP ), ] for query in queries: querier = GooglePlacesQuerier(text_query=query.text_query, data_source=query.data_source) normalizer = GooglePlacesNormalizer(item_category=query.item_category, activity=query.activity) run(querier, normalizer)

(But this part is out of scope for this ticket. We are just trying to scaffold the pipelines here.)

MattClarke131 · 2026-05-18T04:44:30Z

+
+
+class GooglePlacesQuerier(BaseQuerier):
+    def __init__(self, *, api_key: str | None = None, text_query: str, data_source: str):


Suggested change

def __init__(self, *, api_key: str | None = None, text_query: str, data_source: str):

def __init__(self, *, text_query: str, data_source: str, api_key: str | None = None):

optional parameters should go at the end

MattClarke131 · 2026-05-18T04:45:46Z

+        if not self.api_key:
+            raise RuntimeError("GOOGLE_API_KEY is required")


Nice error handling

MattClarke131 · 2026-05-18T04:49:01Z

+class GooglePlacesRepairQuerier(GooglePlacesQuerier):
+    def __init__(self, api_key: str | None = None):
+        super().__init__(api_key=api_key, text_query="shoe repair Boston MA", data_source="google_places_repair")
+
+
+class GooglePlacesDonationQuerier(GooglePlacesQuerier):
+    def __init__(self, api_key: str | None = None):
+        super().__init__(api_key=api_key, text_query="donation centers Boston MA", data_source="google_places_donations")


📝 Changes Requested: Let's remove these.

They were helpful for me to understand your intentions here, but these individual queries belong in their own dedicated space. We will deal with those later.

See my comment on the normalizer for more context.

MattClarke131 · 2026-05-18T04:52:37Z

+def test_repair_normalizer_maps_google_places_sample():
+    normalized = GooglePlacesRepairNormalizer().normalize(
+        _raw_locations("google_places_repair", "repair", "shoe-repair-00.json")
+    )
+
+    assert normalized
+    assert normalized[0].data_source == "google_places_repair"
+    assert normalized[0].services[0].activity == Activity.REPAIR_PAID
+    assert normalized[0].services[0].item_category == ItemCategory.SHOES
+    assert normalized[0].name == "David's Instant Shoe Repair"
+    assert normalized[0].address.street == "281 Franklin St, Boston, MA 02110, USA"
+
+
+def test_donation_normalizer_maps_google_places_sample():
+    normalized = GooglePlacesDonationNormalizer().normalize(
+        _raw_locations("google_places_donations", "donation", "donations-00.json")
+    )
+
+    assert normalized
+    assert normalized[0].data_source == "google_places_donations"
+    assert normalized[0].services[0].activity == Activity.DONATION_DROP
+    assert normalized[0].services[0].item_category == ItemCategory.CLOTHING
+    assert normalized[0].name == "Morgan Memorial Goodwill Industries"


🤙 You had a sharp catch here: You're testing the normalizer, but not the querier. It is tricky to test the querier, because it depends on google's api.

📝 We should change this test file to test_normalizer.py instead to reflect that

(And I will have to update the example query to reflect that as well)

MattClarke131 · 2026-05-18T04:54:54Z

+)
+
+
+SAMPLES = Path(__file__).resolve().parents[3] / "data-explorations" / "google-places" / "samples"


📝 Changes requested: Place test data inside this folder.

The data-explorations folder should be largely isolated, and expected to be removed soon.

MattClarke131 · 2026-05-18T04:55:09Z

+from pipelines.google_places.normalizer import (
+    GooglePlacesDonationNormalizer,
+    GooglePlacesRepairNormalizer,
+)


You'll need to update this statement to reflect the changes above

MattClarke131 · 2026-05-18T04:58:38Z

+def normalize_google_place(
+    raw: dict[str, Any],
+    *,
+    data_source: str,
+    data_source_id: str,
+    item_category: ItemCategory,
+    activity: Activity,
+) -> dict[str, Any]:
+    display_name = raw.get("displayName") or {}
+    formatted_address = raw.get("formattedAddress")
+    location = raw.get("location") or {}
+    return {
+        "data_source_id": data_source_id,
+        "data_source": data_source,
+        "name": display_name.get("text") or data_source_id,
+        "lat": location.get("latitude", 0.0),
+        "lon": location.get("longitude", 0.0),
+        "address": Address(
+            street=formatted_address,
+            postcode=extract_postcode(formatted_address),
+        ),
+        "contact": Contact(),
+        "services": [Service(activity=activity, item_category=item_category)],
+        "availability": Availability(),
+    }


📝 This method should be the GooglePlacesNormalizer class.
I would also move extract_postcode to that class as well

TristanGramsch requested a review from a team May 6, 2026 19:14

Add Google Places ETL pipelines

d170cd9

TristanGramsch force-pushed the google-places-etl branch from 4be3181 to d170cd9 Compare May 6, 2026 19:15

cnapolit reviewed May 12, 2026

View reviewed changes

MattClarke131 requested changes May 18, 2026

View reviewed changes

Navigation Menu

Uh oh!

Uh oh!

Add Google Places ETL pipelines - #19

Add Google Places ETL pipelines#19
TristanGramsch wants to merge 1 commit into
mainfrom
google-places-etl

TristanGramsch commented May 6, 2026 •

edited

Loading

Uh oh!

TristanGramsch commented May 6, 2026 •

edited

Loading

Uh oh!

cnapolit May 12, 2026

Uh oh!

MattClarke131 left a comment

Uh oh!

MattClarke131 May 18, 2026

Uh oh!

MattClarke131 May 18, 2026

Uh oh!

MattClarke131 May 18, 2026

Uh oh!

Uh oh!

MattClarke131 May 18, 2026

Uh oh!

MattClarke131 May 18, 2026

Uh oh!

MattClarke131 May 18, 2026

Uh oh!

MattClarke131 May 18, 2026

Uh oh!

MattClarke131 May 18, 2026

Uh oh!

MattClarke131 May 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants



		class GooglePlacesQuerier(BaseQuerier):
		def __init__(self, *, api_key: str \| None = None, text_query: str, data_source: str):

	def __init__(self, *, api_key: str \| None = None, text_query: str, data_source: str):
	def __init__(self, *, text_query: str, data_source: str, api_key: str \| None = None):

		if not self.api_key:
		raise RuntimeError("GOOGLE_API_KEY is required")

		)


		SAMPLES = Path(__file__).resolve().parents[3] / "data-explorations" / "google-places" / "samples"

Uh oh!

Uh oh!

Conversation

TristanGramsch commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Verification

Uh oh!

TristanGramsch commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MattClarke131 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

TristanGramsch commented May 6, 2026 •

edited

Loading

TristanGramsch commented May 6, 2026 •

edited

Loading