Adjusting models/intermediate/ngpvan/int_ngpvan__01__contact_attempts.sql join

[jeffjrodriguez]

HIA mentioned that a downstream table is not getting some campaign_names populated and the source of that is this table. Campaign_id appears to be globally unique based on a check in data marts.

[graemelorimer]

I don't think this has solved the issue. I ran this query and it looks like there are still a ton of rows missing campaign names, even though they have campaign_id's.

SELECT committee_name,committee_id, campaign_name, campaign_id FROM tmc-dev-394022.dbt_jrodriguez_dev_intermediate.int_ngpvan__01__contact_attempts where campaign_name is null and campaign_id is not null

One of HIA's campaigns was in that list for me. In that dev intermediate table, it says campaign_id 53691 belongs to the committee 113141: "Houston in Action - AAPI Voter Engagement Table (TMC)". In stg_ngpvan__campaigns, that campaign belongs to committee 79056.

[jeffjrodriguez]

@graemelorimer I see what happened I ran the dbt build command in vs code but that uses the GH repo tmc-dbt which is using the main branch from dbt-ngpvan that doesn't have the change.

I can't figure out how to test it in that ngpvan repo, but I did get the code that runs and edited it with the change here and when I run the query
SELECT committee_name,committee_id, campaign_name, campaign_id FROM tmc-dev-394022.dbt_jrodriguez_dev_intermediate.int_ngpvan__01__contact_attempts_test where campaign_name is null and campaign_id is not null

No rows show up.

I'm not following the committee_id part of your last comment could you share a screenshot that might help me understand better!

[graemelorimer]

@graemelorimer I see what happened I ran the dbt build command in vs code but that uses the GH repo tmc-dbt which is using the main branch from dbt-ngpvan that doesn't have the change.

I can't figure out how to test it in that ngpvan repo, but I did get the code that runs and edited it with the change here and when I run the query SELECT committee_name,committee_id, campaign_name, campaign_id FROM tmc-dev-394022.dbt_jrodriguez_dev_intermediate.int_ngpvan__01__contact_attempts_test where campaign_name is null and campaign_id is not null

No rows show up.

I'm not following the committee_id part of your last comment could you share a screenshot that might help me understand better!

@jeffjrodriguez

I think that's likely just a coincidence of the subset of records that are in your upstream dev tables. I ran that same query but replaced all the tmc-dev-349022.dbt_jrodiguez_dev_staging references to production staging tables in the warehouse to create tmc-dev-394022.dbt_glorimer_dev_intermediate.int_ngpvan__01__contact_attempts_test with the full set of production records. When I run

SELECT count(*)
FROM tmc-dev-394022.dbt_glorimer_dev_intermediate.int_ngpvan__01__contact_attempts_test
where contact_campaign_id is not null and campaign_name is null

I see quite a few records with no name.

The committee id part is best represented by this query

with id_subset as (SELECT committee_id, campaign_id FROM tmc-dev-394022.dbt_glorimer_dev_intermediate.int_ngpvan__01__contact_attempts_test
),

campaigns as (select committee_id, campaign_id from tmc-data-warehouse.dbt_staging.stg_ngpvan__campaigns)

, joined as (select campaigns.committee_id, id_subset.committee_id as other_committee
from id_subset inner join campaigns using(campaign_id)
)

select count(*) from joined where committee_id!=other_committee

If you run that, you'll see that the committee id's between the staging campaigns table don't line up with the committee id from this newly generated table.

I don't think we're going to solve this problem today, and we definitely shouldn't try to merge a fix to this upstream table the Friday before break.

[jeffjrodriguez]

@graemelorimer Good catch. I do see that the code edit seems to have fixed the campaign_names for the HIA compaign names that are showing up blank.

When I check the campaign_id's there's a bunch of 0's and from spot checking, it seems like all the campaign_id's don't exist in the warehouse campaign table, so this does sound like a sync-in issue as well.

I think next steps are that I message the member know that we couldn't fix the bug yet and asking for a timeline after the break. I am going to be OOO so who should own the ticket while I'm out?

[graemelorimer]

@graemelorimer Good catch. I do see that the code edit seems to have fixed the campaign_names for the HIA compaign names that are showing up blank.

When I check the campaign_id's there's a bunch of 0's and from spot checking, it seems like all the campaign_id's don't exist in the warehouse campaign table, so this does sound like a sync-in issue as well.

I think next steps are that I message the member know that we couldn't fix the bug yet and asking for a timeline after the break. I am going to be OOO so who should own the ticket while I'm out?

@jeffjrodriguez I can watch the ticket while you're out!