Skip to content

postgres_cdc: refresh IAM auth token for each new connection - #4710

Open
ReguiguiMohamed wants to merge 1 commit into
redpanda-data:mainfrom
ReguiguiMohamed:fix/postgres-cdc-heartbeat-iam-token-refresh
Open

postgres_cdc: refresh IAM auth token for each new connection#4710
ReguiguiMohamed wants to merge 1 commit into
redpanda-data:mainfrom
ReguiguiMohamed:fix/postgres-cdc-heartbeat-iam-token-refresh

Conversation

@ReguiguiMohamed

Copy link
Copy Markdown

Fixes #4668.

openPgConnectionFromConfig copies cfg.DBConfig.Password into a fresh pgconn.Config and hands that to stdlib.OpenDB. The value is a snapshot, so every physical connection the returned *sql.DB opens later still authenticates with the token that existed when the pool was built. With aws.enabled: true the IAM token lasts around 15 minutes, after which the heartbeat write fails on every tick for the rest of the pipeline's life, while the replication stream stays healthy.

Config.RefreshAuthToken already exists for this, and is what Connect calls for the main connection. This passes it to pgx's BeforeConnect hook, which runs before each new physical connection and hands over the config that is about to be used, so the password gets rebuilt at the point a connection actually needs one. The heartbeat, monitor, snapshotter and server-version check all open through this helper, so this covers all four.

The refresh writes to the shared Config.DBConfig and the hook reads it straight back, and pools open connections from separate goroutines, so a mutex guards that pair.

Testing

connection_test.go exercises the hook without a database. Nothing listens on the port in the DSN, and BeforeConnect runs before the dial, so a refused connection still runs it. The tests check that the token is rebuilt on every connection attempt, that a failing refresh surfaces to the caller, and that a config without RefreshAuthToken is left alone.

Against the current code the first two fail, reporting 0 refreshes and the startup password still in place.

I left CHANGELOG.md alone since release notes look to be generated from PR titles at release time. Glad to add an entry if you'd prefer one.

@CLAassistant

CLAassistant commented Aug 18, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

openPgConnectionFromConfig copied the password into a fresh pgconn.Config
when the pool was built, so the heartbeat, monitor, snapshotter and
server-version check all kept authenticating with the token captured at
startup. Once that token expired the heartbeat write failed on every tick
for the rest of the pipeline's life, while the replication stream stayed
healthy.

Rebuild the password through the existing Config.RefreshAuthToken from a
BeforeConnect hook, which pgx runs before it opens each physical
connection.

Fixes redpanda-data#4668
@ReguiguiMohamed
ReguiguiMohamed force-pushed the fix/postgres-cdc-heartbeat-iam-token-refresh branch from dd280ce to 7ffd850 Compare August 18, 2026 17:15
@josephwoodward

Copy link
Copy Markdown
Contributor

Thanks for looking into this, I'll take a look shortly

@ReguiguiMohamed

Copy link
Copy Markdown
Author

Thanks for looking into this, I'll take a look shortly

hey @josephwoodward i'm truly honoured to contribute, will be here awaiting any fixes or updates to my PR in realtime or monitor and add anything

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

postgres_cdc: heartbeat connection never refreshes IAM auth token, fails every tick after ~15min

3 participants