Reduce server load when polling the transaction history

[DavidMStraub]

@dsblank since you mentioned you rely on polling, I invested some work to make polling of the history endpoint less expensive for the server.

---

Claude:

GET /api/transactions/history/ had no ETag, no Cache-Control and no caching,
so every poll did full work. It also loaded every change of a connection to
serve the few transactions on one page.

Changes

Change loading (undodb.py). Transaction._to_dict() no longer walks
connection.changes. _get_changes() fetches the changes of a page in one
query, built from each transaction's (connection_id, first..last) range, and
partitions the rows in Python. The legacy binary columns are always deferred,
old_json/new_json only when not requested. The connection is loaded with
contains_eager instead of lazily per row. Since page defaults to 0
(unpaginated), transactions are chunked at 500 per query.

ETag and 304 (history.py). get_transactions_state() returns
(max_id, count) from one aggregate query. The validator is
sha256(tree_id, max_id, count, args) and is checked against If-None-Match
before any change-log work, so a steady-state poll costs one aggregate query.
Responses carry ETag, Cache-Control: no-cache and X-Total-Count;
304s carry the headers without a body.

User names (cache.py). get_user_dict() moved out of history.py into
request_cache with a 60 s timeout, and is now called only after the 304
check. Users live in the auth database, so there is no change timestamp to key
on; a cached mapping missing one of the IDs being rendered is refetched.

return_304_if_unchanged() in resources/util.py was refactored to share the
new etag_unchanged() predicate.

Notes

• The validator does not cover user names: renaming a user becomes visible once
  the next transaction is written.
• get_transaction() now returns None instead of raising AttributeError
  for a missing ID. Two except AttributeError blocks became dead and were
  removed; TransactionHistoryResource gained the 404 it was missing (it
  previously returned 500).
• Unchanged and still worth a look: a transaction with first is None (an
  empty DbTxn) reports every change of its connection.

Tests

43 passing in test_history.py, test_undodb.py and test_transactions.py,
7 of them new: ETag round trip with invalidation on a new transaction, the
validator depending on query args, change scoping across transactions sharing a
connection, the chunk boundary, get_transactions_state, get_transaction
including the missing-ID case, and old/new gating.

[Copilot]

Pull request overview

This PR reduces server load for clients that poll the transaction history endpoint by introducing conditional requests (ETag/304) and by restructuring how undo DB transactions and their associated changes are loaded.

Changes:

• Refactors undo DB history queries to fetch only the relevant change rows per transaction page (with chunking and deferred columns) and adds an aggregate “state” query for fast revalidation.
• Adds ETag-based revalidation to GET /api/transactions/history/ (304 responses with headers, Cache-Control: no-cache, X-Total-Count) and fixes missing-ID behavior to return 404 instead of 500.
• Moves user-id→user-name mapping into a short-lived request cache and adds tests covering ETag behavior, chunking, and change scoping.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/test_undodb.py	Adds unit tests for DbUndoSQLWeb history queries (change scoping, chunking, state aggregation, old/new gating, missing transaction).
tests/test_endpoints/test_history.py	Adds endpoint tests verifying ETag round-trip (304) and that ETag varies with query args.
gramps_webapi/undodb.py	Implements chunked change fetching and adds get_transactions_state(); updates transaction fetch paths to eager-load connections and avoid walking connection.changes.
gramps_webapi/api/resources/util.py	Factors out etag_unchanged() and reuses it in return_304_if_unchanged().
gramps_webapi/api/resources/history.py	Adds ETag/304 flow to the history list endpoint, uses cached user mapping post-304 check, and fixes missing transaction handling (404).
gramps_webapi/api/cache.py	Introduces cached get_user_dict() keyed by tree/include-treeless, with short timeout and “refetch if missing requested IDs” behavior.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (3)

gramps_webapi/undodb.py:73

• CHANGES_QUERY_CHUNK_SIZE = 500 is likely to exceed SQLite’s default bound-parameter limit (commonly 999) because _get_changes_chunk() builds an OR of per-transaction ranges with ~3 bind params each. With 500 transactions that’s ~1500 params and can raise OperationalError: too many SQL variables when fetching unpaginated history on SQLite.

# transactions per change query, to keep the statement size bounded
CHANGES_QUERY_CHUNK_SIZE = 500

gramps_webapi/undodb.py:633

• before/after are checked via truthiness, so a valid value like 0.0 (Unix epoch) won’t apply the filter. These should be is not None checks to match the type (float | None) and avoid surprising query behavior.

        if before:
            query = query.filter(Transaction.timestamp < before * 1e9)
        if after:
            query = query.filter(Transaction.timestamp > after * 1e9)

gramps_webapi/api/resources/util.py:1857

• If-None-Match can legally contain a comma-separated list of validators. The current implementation passes the whole header value into normalize_etag(), which will never match in the multi-ETag case and prevents 304 responses (increasing load) when clients send lists.

def etag_unchanged(etag: str) -> bool:
    """Check whether the if none match header agrees with the current etag."""
    old_etag = request.headers.get("If-None-Match")
    return bool(old_etag) and normalize_etag(old_etag) == etag

[DavidMStraub]

Polling the transaction history: instruction for frontends 🤖

GET /api/transactions/history/?page=1&pagesize=20&sort=-id&after_id=<last seen>
If-None-Match: <etag from last 200>

• Always send If-None-Match — unchanged history returns 304, no body.
• Always set page=1 — it defaults to 0, meaning the entire history.
• Use after_id, not after — the timestamp cursor redelivers rows.
• Never send old=1/new=1 — full object JSON per change.
• Keep the URL byte-identical — a changed query string cannot 304.
• X-Total-Count is on the 304 too.
• Back off when idle, reset on a 200.