Skip to content

materialize-redshift: load bounds and DISTSTYLE ALL for temp load table#4460

Open
mdibaiee wants to merge 2 commits into
mainfrom
mahdi/redshift-load-perf
Open

materialize-redshift: load bounds and DISTSTYLE ALL for temp load table#4460
mdibaiee wants to merge 2 commits into
mainfrom
mahdi/redshift-load-perf

Conversation

@mdibaiee
Copy link
Copy Markdown
Member

@mdibaiee mdibaiee commented May 13, 2026

Description:

  • We have noticed redshift load queries can be the most time-consuming part of pipelines for target tables that are large. We seem to scan all of the target table unfortunately (there are no "indices"). Here I am adding Key Bounds as well as a recommended "DISTSTYLE ALL" to distribute the load table across compute slices so they are all local during the lookup.
  • I would like to try this out on a customer task to see how it fares.

Workflow steps:

(How does one use this feature, and how has it changed)

Documentation links affected:

(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)

Notes for reviewers:

(anything that might help someone review this PR)

@mdibaiee mdibaiee requested a review from a team May 13, 2026 14:10
Copy link
Copy Markdown
Contributor

@danielnelson danielnelson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Would it be worth applying and testing the merge bounds change separately from changing the diststyle in order to be sure they are both giving improvements?

Comment thread materialize-redshift/sqlgen.go Outdated
Comment on lines +177 to +178
// at least on Redshift Serverless). The literal 'foo\bar' parses to
// "foobar" — the backslash and following char are consumed. So we
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the backslash and following character were consumed, then wouldn't foo\bar parse to fooar?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jacobmarble yes! you are right

@mdibaiee mdibaiee force-pushed the mahdi/redshift-load-perf branch from ce7aa8b to d0c077d Compare May 13, 2026 16:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants