Snapshot restore silently leaves the target schema incomplete: partitioned indexes never attached, extensions not restored

[siriusfreak]

What happened

On a snapshot into an empty target, the schema restore logs restore: N errors ignored and continues — but two classes of those "ignored" errors leave the target's schema permanently incomplete, with no later signal that anything is missing.

Version: v1.3.1, postgres → postgres, source PG14, target PG16.

1. Partitioned indexes are never attached

ALTER INDEX ... ATTACH PARTITION is executed in the first restore pass, before the parent index exists, so it fails. The later restoring schema indices and constraints pass creates the parent index but does not retry the attach, so the parent index stays indisvalid = false forever.

Minimal reproduction:

-- source
CREATE TABLE jfbus_events (
  event_id bigint, topic_id int, partition_id int,
  PRIMARY KEY (event_id, topic_id, partition_id)
) PARTITION BY LIST (partition_id);
CREATE TABLE jfbus_events_0 PARTITION OF jfbus_events FOR VALUES IN (0);
CREATE TABLE jfbus_events_1 PARTITION OF jfbus_events FOR VALUES IN (1);
CREATE INDEX jfbus_events_topic_partition_event_id
  ON jfbus_events (topic_id, partition_id, event_id);
INSERT INTO jfbus_events SELECT g, 1, g % 2 FROM generate_series(1,5000) g;

Run a snapshot into an empty target with PGSTREAM_POSTGRES_SNAPSHOT_TABLES="public.*". The restore logs:

restore: 5 errors ignored
  relation does not exist: ERROR: relation "public.jfbus_events_pkey" does not exist:
    STATEMENT: ALTER INDEX public.jfbus_events_pkey ATTACH PARTITION public.jfbus_events_0_pkey;
  relation does not exist: ERROR: relation "public.jfbus_events_topic_partition_event_id" does not exist:
    STATEMENT: ALTER INDEX public.jfbus_events_topic_partition_event_id ATTACH PARTITION ...
restoring schema indices and constraints
logical replication started

All 5000 rows land, and then on the target:

SELECT indexrelid::regclass, indrelid::regclass FROM pg_index WHERE NOT indisvalid;

 jfbus_events_pkey                      | jfbus_events
 jfbus_events_topic_partition_event_id  | jfbus_events

A non-partitioned table in the same run is fine.

Why this matters beyond the index itself. An invalid partitioned index is not usable as a constraint, so the target ends up with no enforceable primary key for that table. Downstream that means no replica identity for UPDATE/DELETE, and no conflict target for on_conflict_action: update — so subsequent replication of that table silently drops rows (with strict_mode: false, the failed query is dropped and the checkpoint advanced).

Expected: either the attach is ordered after the parent index is created, or it is retried in the restoring schema indices and constraints pass.

2. Extensions are not restored

Same run, separate symptom. With pg_trgm installed on the source and an index that uses it:

-- source
CREATE EXTENSION pg_trgm;
CREATE INDEX trg_gin ON trg USING gin (lower(name) gin_trgm_ops);

the restore reports:

restore: 2 errors ignored
  relation does not exist: ERROR: operator class "public.gin_trgm_ops"
  does not exist for access method "gin":
    STATEMENT: CREATE INDEX trg_gin ON trg USING gin (lower(name) gin_trgm_ops);

and afterwards the target has SELECT extname FROM pg_extension → only plpgsql, with the index missing. The data loads fine, so the only trace is the ignored-error line.

Expected: either extensions are included in the restored schema, or their absence is reported as something stronger than an ignored error, since every dependent index is silently skipped.

Suggestion

Both are cases where restore: N errors ignored hides a target schema that does not match the source. Even without changing the ordering, a post-restore summary — "these objects were not created" — would turn a silent divergence into something an operator can act on.

[dosubot]

@siriusfreak, excellent analysis — both issues are rooted in separate design gaps, and I can point to the exact code paths involved.

Issue 1: ALTER INDEX … ATTACH PARTITION ordering

The parseDump function in the snapshot generator classifies lines into segments (filtered schema dump, indicesAndConstraints, views, etc.) based on prefix matching [1]. Crucially, ALTER INDEX … ATTACH PARTITION does not match any of the special cases in parseDump — it starts with ALTER INDEX, not ALTER TABLE, CREATE INDEX, or any other recognized prefix — so it falls through to the default case and lands in the filtered dump segment, which is restored before the indicesAndConstraints segment [2]. At that point the parent index hasn't been created yet, so the attach fails.

The restoreIndicesAndConstraints function simply forwards the dump blob to restoreDumpWithOptions with no reordering or retry logic [3]. The "does not exist" error is classified as ErrRelationDoesNotExist and silently placed in the ignored errors bucket [4], so the failure is never surfaced.

Issue 2: Extensions not restored

When you use PGSTREAM_POSTGRES_SNAPSHOT_TABLES="public.*" without PGSTREAM_POSTGRES_SNAPSHOT_INCLUDE_GLOBAL_DB_OBJECTS=true, the option generator hits the hasWildcardSchema && !includeGlobalDBObjects branch [5]. This discovers all user schemas and passes them as --schema filters to pg_dump. Since extensions aren't schema-scoped, pg_dump omits them entirely. The resulting "operator class does not exist" error contains "does not exist", so it's again bucketed as ErrRelationDoesNotExist and silently ignored [4].

Immediate workaround: set PGSTREAM_POSTGRES_SNAPSHOT_INCLUDE_GLOBAL_DB_OBJECTS=true. This switches to the exclude-schema approach [6], which causes pg_dump to emit CREATE EXTENSION statements. Note this includes all non-excluded schemas, so verify that's acceptable for your setup.

Issue 3: Silent divergence

The restoreDumpWithOptions function logs a warning with the count and list of ignored errors but makes no distinction between "this object already existed (benign)" and "this object was needed but couldn't be created (schema incomplete)" [7]. Treating ErrRelationDoesNotExist as universally ignorable is the assumption that breaks down for both cases you described.

These look like two distinct bugs: one for ALTER INDEX … ATTACH PARTITION not being classified into the correct restore phase, and one for the ErrRelationDoesNotExist classification being too broad. Your suggestion about a post-restore summary that distinguishes benign from schema-breaking ignored errors would address the observability gap across both.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[siriusfreak]

Correction to section 2 of this report: the extension behaviour is not a bug, it is include_global_db_objects, and I had not set it. Apologies for the noise.

With PGSTREAM_POSTGRES_SNAPSHOT_INCLUDE_GLOBAL_DB_OBJECTS=true on the same reproduction, pg_trgm is created on the target and the gin_trgm_ops index is restored, with zero ignored errors. The mechanism is clear in pg_options_generator.go — a non-wildcard schema scope sets opts.Schemas, which makes pg_dump --schema=... omit database-level objects.

The one suggestion I would keep from that section is about the reporting rather than the behaviour: when restore: N errors ignored includes a missing operator class, every index that depends on it is skipped and the load still reports success. A summary of what was not created would make that visible without changing any defaults.

Section 1 stands — the partitioned index ATTACH PARTITION ordering is unaffected by that setting, and I have reproduced it again with include_global_db_objects both on and off. A fix is on the way as a PR.

[siriusfreak]

Thanks — that matches what I measured, and our comments on include_global_db_objects crossed in flight. Confirming it from the running side: with PGSTREAM_POSTGRES_SNAPSHOT_INCLUDE_GLOBAL_DB_OBJECTS=true the extension is created and the gin_trgm_ops index restores cleanly, so that half is configuration and not a defect. I have struck it from the report above.

Two things I would add, one narrowing and one widening.

Narrowing. With #2 removed there is one bug here, not two: ALTER INDEX … ATTACH PARTITION is classified into the wrong restore phase. It is worth stating why it is more than a missing index, because that is what made it expensive for us rather than cosmetic:

• the parent index stays indisvalid = false, so it is unusable as an index and as a constraint;
• for a PRIMARY KEY that means the partitioned table has no enforceable key on the target;
• which means no replica identity for UPDATE/DELETE, and no conflict target for INSERT — buildOnConflictQuery returns "" when there are no primary key columns;
• so with strict_mode: false every subsequent change to that table is dropped and the checkpoint advanced.

A load that reports success therefore produces a table that silently stops replicating. On our schema that was jfbus_events, and we only found it by reading the ignored-error text.

Widening. Your third point is the one I would most like a maintainer's opinion on, because it is the reason both symptoms were invisible. Treating ErrRelationDoesNotExist as universally ignorable conflates two very different things: "this object already existed" (benign, expected on a re-run) and "this object could not be created because something it depends on is missing" (the target schema is now incomplete). The second class is exactly the one an operator needs to see, and today it is one line of restore: N errors ignored next to the benign ones.

I am opening a PR for the classification fix, scoped narrowly to routing ALTER INDEX … ATTACH PARTITION into indicesAndConstraints with a test. I have deliberately not touched the error bucketing in the same change — happy to follow up with it if you would take it, but it changes behaviour for every restore and seems like your call rather than mine.

[kvch]

Thanks for the thorough report, and for splitting the fix out so cleanly.

Status of the three parts, so this issue reflects what is left:

1. Partitioned indexes: Fixed and on main by fix: restore ALTER INDEX ... ATTACH PARTITION with indices and constraints #1094
2. Extensions: Agreed, this is include_global_db_objects and not a defect. Nothing to do here.
3. Ignored-error classification: Accepted, and this is what the issue stays open for.

One note: the conflation is not "already existed" vs "dependency missing". already exists is already its own type (ErrRelationAlreadyExists), and it is not what got bucketed alongside your attach failures. The benign source of does not exist is the cleanup pass: DROP statements against a target where the object legitimately is not there yet. (#742) So the split is by what produced the error.

On your widening question: we do not want to promote the whole ErrRelationDoesNotExist bucket to critical. For a good share of build-pass does not exist errors, divergence from the source is what was asked for. include_global_db_objects=false omits extensions so every dependent index fails. That last one is exactly the case you withdrew; making the class critical would turn a documented default into a hard restore failure.

"Target matches source" is not the invariant. The narrower one is:

the target contains everything the dump just handed to pg_restore asked for.

The attach case violates it. The parent index was in the dump, created in a later segment, and the reference still failed. The filtered cases do not: the referent was never in the dump at all. That is a discriminator we can implement, since parseDump already scans every line and recognizes the CREATE/ADD CONSTRAINT forms, and the failing statement is already attached to the error. It has to be built over the whole dump rather than the segment being restored. Pre-#1094 the failing attach was in filtered while the index it needed was in indicesAndConstraints, so a per-segment view would miss this.

3 outcomes:

• referent is in the dump and a retry succeeds → an ordering problem, logged and carried on. Worth having on its own: it covers the class generically, where fix: restore ALTER INDEX ... ATTACH PARTITION with indices and constraints #1094 is a special case for one statement prefix and the next such statement would need its own fix.
• referent is in the dump and the retry fails -> we produced a dump we cannot restore. Critical.
• referent is not in the dump → the filter or include_global_db_objects excluded it. We can ignore it.

Anything we cannot classify falls to "report", never to "fail". If name extraction misses a message form, or an object is created by something we do not parse, the verdict lands on "not in the dump".

Two limitations:

• Implicit objects. gin_trgm_ops has no CREATE OPERATOR CLASS line; it comes from CREATE EXTENSION pg_trgm. With include_global_db_objects=false it correctly lands in "not in the dump", but with it on, a failed CREATE EXTENSION
  makes the operator-class error a knock-on that a plain lookup would misread. Tracking whether the referent's own creating statement errored earlier in the run covers it.
• Ordering. The data load runs between the filtered and indicesAndConstraints passes, so a critical verdict on an index or constraint aborts after the data is in. Still better than diverging silently, but it is a loud failure rather t
  han prevention — which is the other reason the retry step earns its place.

Suggested split, if you would still like to take it: the summary of what was not created first, on its own. Today each pass logs its own restore: N errors ignored and nothing aggregates across passes, so an operator has to read raw error text to find out anything is missing. That change moves no error between buckets and nothing that works today starts failing. The classification and retry then go on top of it.

Happy to review both. Thanks again for the detail in the original report.

[siriusfreak]

Taking the split you suggested. The first part — the summary of what was not created — is #1223.

It aggregates what every restore pass ignored into one report at the end of CreateSnapshot, with the statement that was not applied and why it was let through. No error changes bucket, the per-pass restore: N errors ignored lines stay, and nothing that succeeds today starts failing.

The one enabling change is that the echoed statement is now carried on the error (ErrRestoreStatement) instead of only being concatenated into the message, so the summary does not have to parse the SQL back out. The message is byte for byte unchanged and errors.As still reaches the underlying type, which keeps addError classifying exactly as before.

The classification and the retry go on top, as a second PR. Your three outcomes and the two limitations are the design I will work to — in particular building the referent lookup over the whole dump rather than the segment being restored, and tracking whether the referent's own creating statement failed earlier in the run, so an implicit object like gin_trgm_ops after a failed CREATE EXTENSION does not read as a plain miss.