Skip to content

Avoid redundant DB write per object during Gramps XML import - #2494

Open
dsblank wants to merge 1 commit into
gramps-project:masterfrom
dsblank:perf/importxml-inaugurate-double-commit
Open

Avoid redundant DB write per object during Gramps XML import#2494
dsblank wants to merge 1 commit into
gramps-project:masterfrom
dsblank:perf/importxml-inaugurate-double-commit

Conversation

@dsblank

@dsblank dsblank commented Aug 9, 2026

Copy link
Copy Markdown
Member

Summary

GrampsParser.inaugurate() in gramps/plugins/importer/importxml.py was
writing every primary object to the database twice while importing a
Gramps XML file:

  1. A near-empty placeholder commit when the object's start_<tag> handler
    first assigned it a handle.
  2. A second, full commit when stop_<tag> finished parsing the object and
    called db.commit_<object>().

The first write only matters when a handle is genuinely forward-referenced
elsewhere in the file (e.g. hlink/parentin pointing at an object not yet
parsed) — the eventual get_raw_<object>_data() lookup used to backfill state
needs a row to exist. For the common case of an object first encountered via
its own tag, that placeholder write is discarded moments later by the real
commit.

This change skips the placeholder commit unless inaugurate() is invoked
from a reference site (i.e. prim_obj is a class, not an instance), deferring
the single actual database write to the object's own stop_<tag> commit.

Testing

  • Imported example/gramps/example.gramps (2157 people / 762 families / 3432
    events / ...) and a synthetic 32,001-person / 16,000-family file (with
    parentin hlink forward references to families) before and after the
    change; dumped every object's serialized state and diffed — byte-for-byte
    identical
    in both cases.
  • Benchmarked import of the 32k/16k file, median of 3 runs:
    • before: 11.61s
    • after: 8.25s (~29% faster)
  • Benchmarked the smaller example file, median of 9 runs:
    • before: 1.83s
    • after: 1.41s (~23% faster)
  • black --check and mypy pass on the changed file.

Test plan

  • Verified import produces identical database contents before/after
  • Benchmarked import time improvement on two dataset sizes
  • black --check gramps/plugins/importer/importxml.py
  • mypy gramps/plugins/importer/importxml.py

🤖 Generated with Claude Code

GrampsParser.inaugurate() wrote every primary object to the database
twice while importing a Gramps XML file: once as a near-empty
placeholder when its start_<tag> handler was reached, and again with
the fully populated object when its stop_<tag> handler committed it.
The placeholder write is only needed when a handle is genuinely
forward-referenced elsewhere in the file before its own definition is
parsed; it is now only performed in that case, cutting import time by
roughly a quarter on large files.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dsblank

dsblank commented Aug 9, 2026

Copy link
Copy Markdown
Member Author

Combined benchmark with #2493

Ran a controlled comparison on a synthetic 16k-person Gramps XML file (32,001
people / 16,000 families, with parentin hlink forward references), using
three isolated worktrees all built from the same master base commit
(a32b463b16) so the numbers are directly comparable. Median of 5 runs each:

Variant Median time vs. master vs. previous row
1. master (baseline) 12.64s
2. master + #2493 (cache get_schema()) 7.45s 41% faster 41% faster
3. master + #2493 + this PR 5.33s 58% faster 28% faster

Combined, the two PRs make this import ~2.4x faster than master.

Correctness: dumped and diffed every object's full serialized state across
all three variants — byte-for-byte identical database contents in every case,
confirming both optimizations are pure speedups with no behavior change.

@dsblank

dsblank commented Aug 9, 2026

Copy link
Copy Markdown
Member Author

Larger-scale benchmark: 100k people

Ran a single import each of a 100k-person Gramps XML file (100,000 people /
46,315 families / 331,067 events, gzip-compressed .gramps), comparing
master against master + #2493 + this PR, in isolated worktrees off the
same base commit:

Variant Time
master (baseline) 168.95s (2m49s)
master + #2493 + this PR 116.37s (1m56s)

31% faster (~1.45x speedup) at this scale, consistent with the 16k-person
numbers posted above. Both runs produced identical object counts
(100,000 people / 46,315 families / 331,067 events).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant