Skip to content
Merged
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 27 additions & 4 deletions .github/workflows/test-warehouse.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,8 @@ jobs:
test:
runs-on: ubuntu-latest
concurrency:
# This is what eventually defines the schema name in the data platform.
# Serialises runs for the same warehouse × dbt-version × branch.
# The schema name is derived from a hash of this group (see "Write dbt profiles").
group: tests_${{ inputs.warehouse-type }}_dbt_${{ inputs.dbt-version }}_${{ github.head_ref || github.ref_name }}
cancel-in-progress: true
steps:
Expand Down Expand Up @@ -125,13 +126,35 @@ jobs:
env:
CI_WAREHOUSE_SECRETS: ${{ secrets.CI_WAREHOUSE_SECRETS || '' }}
run: |
DBT_VERSION=$(echo "${{ inputs.dbt-version }}" | sed 's/\.//g')
UNDERSCORED_REF_NAME=$(echo "${{ inputs.warehouse-type }}_dbt_${DBT_VERSION}_${BRANCH_NAME}" | awk '{print tolower($0)}' | head -c 40 | sed "s/[-\/]/_/g")
# Build a short, collision-free schema identifier from the branch
# name and an 8-char hash of the full concurrency group.
#
# Previous approach used `head -c 40` on the full ref string, which
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Explaining the history is not interesting. Replace with a short explanation about how it is computed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — simplified the comment to just explain the format and budget, removed the history explanation. See c58c8c0.

# left only 1 character for branch differentiation on long warehouse
# names like databricks_catalog, causing cross-branch schema
# collisions when CI jobs ran concurrently.
#
# New approach: <truncated_branch>_<YYMMDD>_<8-char SHA-256 of concurrency group>
# The hash guarantees uniqueness across concurrent jobs (different
# warehouse types, dbt versions, or branches). The truncated branch
# name keeps schemas human-readable in the data platform. The date
# stamp (YYMMDD) makes it easy to identify and clean up old schemas.
#
# Schema name budget (PostgreSQL = most restrictive at 63 chars):
# dbt_pkg_ (8) + ref (≤40) + _elementary (11) + _gw7 (4) = 63
# ref = branch (≤24) + _ (1) + date (6) + _ (1) + hash (8) = 40
CONCURRENCY_GROUP="tests_${{ inputs.warehouse-type }}_dbt_${{ inputs.dbt-version }}_${BRANCH_NAME}"
SHORT_HASH=$(echo -n "$CONCURRENCY_GROUP" | sha256sum | head -c 8)
SAFE_BRANCH=$(echo "${BRANCH_NAME}" | awk '{print tolower($0)}' | sed "s/[^a-z0-9]/_/g" | head -c 24)
DATE_STAMP=$(date +%y%m%d)
SCHEMA_SUFFIX="${SAFE_BRANCH}_${DATE_STAMP}_${SHORT_HASH}"

echo "Schema suffix: $SCHEMA_SUFFIX (branch='${BRANCH_NAME}', date=${DATE_STAMP}, hash of concurrency group)"

python "${{ github.workspace }}/dbt-data-reliability/integration_tests/profiles/generate_profiles.py" \
--template "${{ github.workspace }}/dbt-data-reliability/integration_tests/profiles/profiles.yml.j2" \
--output ~/.dbt/profiles.yml \
--schema-name "dbt_pkg_$UNDERSCORED_REF_NAME"
--schema-name "dbt_pkg_$SCHEMA_SUFFIX"

- name: Install dependencies
working-directory: ${{ env.TESTS_DIR }}
Expand Down
Loading