Avoid redundant DB write per object during Gramps XML import - #2494
Avoid redundant DB write per object during Gramps XML import#2494dsblank wants to merge 1 commit into
Conversation
GrampsParser.inaugurate() wrote every primary object to the database twice while importing a Gramps XML file: once as a near-empty placeholder when its start_<tag> handler was reached, and again with the fully populated object when its stop_<tag> handler committed it. The placeholder write is only needed when a handle is genuinely forward-referenced elsewhere in the file before its own definition is parsed; it is now only performed in that case, cutting import time by roughly a quarter on large files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Combined benchmark with #2493Ran a controlled comparison on a synthetic 16k-person Gramps XML file (32,001
Combined, the two PRs make this import ~2.4x faster than master. Correctness: dumped and diffed every object's full serialized state across |
Larger-scale benchmark: 100k peopleRan a single import each of a 100k-person Gramps XML file (100,000 people /
31% faster (~1.45x speedup) at this scale, consistent with the 16k-person |
Summary
GrampsParser.inaugurate()ingramps/plugins/importer/importxml.pywaswriting every primary object to the database twice while importing a
Gramps XML file:
start_<tag>handlerfirst assigned it a handle.
stop_<tag>finished parsing the object andcalled
db.commit_<object>().The first write only matters when a handle is genuinely forward-referenced
elsewhere in the file (e.g.
hlink/parentinpointing at an object not yetparsed) — the eventual
get_raw_<object>_data()lookup used to backfill stateneeds a row to exist. For the common case of an object first encountered via
its own tag, that placeholder write is discarded moments later by the real
commit.
This change skips the placeholder commit unless
inaugurate()is invokedfrom a reference site (i.e.
prim_objis a class, not an instance), deferringthe single actual database write to the object's own
stop_<tag>commit.Testing
example/gramps/example.gramps(2157 people / 762 families / 3432events / ...) and a synthetic 32,001-person / 16,000-family file (with
parentin hlinkforward references to families) before and after thechange; dumped every object's serialized state and diffed — byte-for-byte
identical in both cases.
black --checkandmypypass on the changed file.Test plan
black --check gramps/plugins/importer/importxml.pymypy gramps/plugins/importer/importxml.py🤖 Generated with Claude Code