Skip to content

oracledb_cdc: remove primary key ordering from unfiltered snapshot - #4696

Merged
josephwoodward merged 5 commits into
mainfrom
jw/oracledb_snapshot_ordering
Aug 21, 2026
Merged

oracledb_cdc: remove primary key ordering from unfiltered snapshot#4696
josephwoodward merged 5 commits into
mainfrom
jw/oracledb_snapshot_ordering

Conversation

@josephwoodward

@josephwoodward josephwoodward commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Problem

By default, OracleDB uses heaped tables which aren't sorted by primary key, meaning snapshotting ordered by primary key can incur high disk I/O as every single row lookup via ROWID forces Oracle to jump to a different physical data block on disk.

Solution

As this is a full table scan, and not batched/chunked into ranges we can remove ordering which improves overall read performance.

Proof of Work

AWS Benchmark

Benchmarks went from ~9.8MB/s (post cache-warm) to a consistent 98MB/s:

image

Local Benchmark:

Before: Avg. ~40MB/s

image

After: Avg. ~51MB/s

image

Comment thread internal/impl/oracledb/replication/snapshot.go
Comment thread internal/impl/oracledb/replication/snapshot.go Outdated
@josephwoodward
josephwoodward force-pushed the jw/oracledb_snapshot_ordering branch from 3abe4c3 to 30c28b0 Compare August 20, 2026 09:46
Comment thread internal/impl/oracledb/input_oracledb_cdc.go Outdated
Comment thread internal/impl/oracledb/replication/snapshot_test.go
Comment thread docs/modules/components/pages/inputs/oracledb_cdc.adoc Outdated
@josephwoodward
josephwoodward force-pushed the jw/oracledb_snapshot_ordering branch from 30c28b0 to 72a3870 Compare August 20, 2026 10:00
Comment thread internal/impl/oracledb/input_oracledb_cdc.go Outdated
@josephwoodward
josephwoodward force-pushed the jw/oracledb_snapshot_ordering branch from 72a3870 to 36aee7d Compare August 20, 2026 11:20
@josephwoodward
josephwoodward marked this pull request as ready for review August 20, 2026 11:25
Comment thread internal/impl/oracledb/replication/snapshot.go
Default(1)).
Field(service.NewIntField(ociFieldSnapshotMaxBatchSize).
Description("The maximum number of rows to be streamed in a single batch when taking a snapshot.").
Description("The maximum number of rows fetched per query when taking a snapshot of a table with a `" + ociFieldSnapshotFilters + "` entry configured. Tables without one are streamed through a single unordered cursor, where this value only paces how often a cancellation is checked.").

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

snapshot_max_batch_size now carries two unrelated meanings depending on whether the table happens to have a snapshot_filters entry: for filtered tables it is the page size of the keyset-pagination query, and for every other table (the default, since snapshot_filters is optional) it degrades to "how often ctx.Err() is polled" — see snapshot.go#L362-L381.

This is a material UX regression under CONTRIBUTING §1.1.3 ("UX should be intuitive and require minimal explanation") and §3.1.5 (consistency with the rest of the fleet, where snapshot_max_batch_size uniformly means rows-per-fetch). The description text itself is the evidence — it has to explain two divergent behaviours for one knob.

Concretely: a user who raised snapshot_max_batch_size to tune snapshot throughput — exactly what docs/benchmark-results/oracledb-cdc.md documents with snapshot_max_batch_size: 160000 — now gets no throughput effect at all on the default path, silently.

Suggested fix: keep the knob meaning "rows fetched per round trip" on both paths by wiring it into the driver's prefetch/array-fetch size for the full-scan cursor, and use a separate fixed internal constant for the cancellation-poll interval. If the two really must differ, a dedicated field for the scan path would be clearer than overloading this one.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we want snapshot_max_batch_size to be the prefetch_rows value, it would be better to have that as a dedicated config and deprecate snapshot_max_batch_size if needed.

Comment on lines +334 to +337
// snapshotTableFullScan performs a single, full unordered scan so Oracle's optimizer
// picks a full table scan (sequential multiblock reads) over random disk I/O when ordered.
func (s *Snapshot) snapshotTableFullScan(ctx context.Context, tx *sql.Tx, table UserTable, maxBatchSize int, tableName string) (numRowsProcessed int, err error) {
q := fmt.Sprintf(`SELECT * FROM "%s"."%s"`, table.Schema, table.Name)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is justified purely on throughput grounds — the doc comment here and the inline comment at snapshot.go#L232-L234 both claim the unordered scan is faster than the previous ORDER BY keyset pagination — but no benchmark evidence ships with it.

CONTRIBUTING §1.3.4/§1.3.5 requires both local and real-endpoint benchmarking at various throughput levels, with results recorded under docs/benchmark-results/. docs/benchmark-results/oracledb-cdc.md already has a "Snapshot Results" section, and it is left untouched by this PR — so the recorded numbers now describe a code path that no longer runs for unfiltered tables, and were captured under a snapshot_max_batch_size: 160000 setting whose meaning this PR changes.

Suggested fix: re-run the snapshot benchmarks against the new scan path and update docs/benchmark-results/oracledb-cdc.md, so the performance claim in this comment is backed by recorded numbers rather than an assertion about Oracle's optimizer.

@josephwoodward josephwoodward Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Benchmarks included in proof of work. Will look to update these as a separate PR.

@josephwoodward
josephwoodward merged commit 48a4b64 into main Aug 21, 2026
10 checks passed
@josephwoodward
josephwoodward deleted the jw/oracledb_snapshot_ordering branch August 21, 2026 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants