Raise work_mem to 64MB for readonlyuser on treetracker to cut temp-file spill

[arnoldcastro5000]

Summary

Give the readonlyuser role a larger work_mem (64MB, up from the cluster default) on the treetracker database only. This lets the map-tile read queries do more of their sorting and grouping in memory instead of spilling to temporary files on disk, which is a large, safe, and fully reversible reduction in disk load. It is a first step, not the complete fix (see Follow-ups).

Background (what is going wrong)

The tile server draws the map by running heavy read queries against a read replica of the treetracker database. Many of these queries sort or group large amounts of spatial data. PostgreSQL gives each such operation a memory budget called work_mem. When an operation needs more than work_mem, PostgreSQL does not fail; instead it writes the overflow to temporary files on disk and reads them back. That disk traffic is slow and it competes with every other query on the replica.

Two 12-hour monitoring runs show this clearly:

• The read workload writes a very large volume of temporary files (about 55 to 65GB of temp writes per 12-hour window, and roughly the same again in temp reads, because temp read and write are close to 1:1).
• Memory use on the replica sits low (around 35 percent) at the same time, which is the classic signature of a work_mem that is set too small: the database trades spare RAM for disk spill.
• The offenders are a small set of query families: the spatial cluster queries (labelled case1) and a recursive query over the organization tree (a recursive CTE).

The replica has 16GB of RAM. The current work_mem is 13MB (the DigitalOcean default on a 16GB node), so even moderate sorts spill. This change raises it to 64MB, about 5x.

The change

ALTER ROLE readonlyuser IN DATABASE treetracker SET work_mem = '64MB';

• Scope: only the readonlyuser role, and only inside the treetracker database. No other role, database, or the primary writer is affected. This is deliberate: work_mem is allocated per operation per connection, so a database-wide or cluster-wide change would multiply memory use across unrelated workloads and risk an out-of-memory (OOM) condition, especially on the smaller 8GB primary.
• Why 64MB and not higher: the value is bounded by peak concurrency, not by the average. The read role peaks at about 30 to 33 concurrent heavy queries (confirmed across both monitoring runs). At 64MB, the worst-case in-memory footprint stays safe on the 16GB replica. A larger value such as 256MB would remove more spill but risks OOM at that concurrency, so it is left as a later step once the indexes below reduce the pile-up.

Expected effect

The captured query plans show the spill comes from Sort nodes running in a single process, so work_mem applies at 1x here (hash_mem_multiplier and parallel workers do not apply to these sorts). At 64MB this gives a meaningful but partial reduction in temp-file writes, largest for the mid-size queries that just cross the current 13MB limit. The case1 spatial queries, whose per-call working set is about 1.33GB, are far above any safe work_mem and improve only marginally; they need the spatial index, not more memory (see Follow-ups). The exact percentage should be measured on the validation fork with EXPLAIN (ANALYZE, BUFFERS); an earlier estimate of about 46 percent was withdrawn because it assumed a hash and parallel model that the plans do not support. In short: this PR is a low-risk win, but it is a mitigation, not the cure.

How it is applied (and why this is not Terraform)

The cyrilgdn/postgresql provider (v1.22.0) used in database-grants/ has no resource for a per-role, per-database configuration parameter (the ALTER ROLE ... IN DATABASE ... SET form), so this change cannot be expressed as a Terraform resource. It is committed as an operator-applied SQL runbook under database-grants/tuning/prod/, and applied by a database administrator using the doadmin account. The CI service account is intentionally not permitted to change production data.

Apply steps:

1. A DBA runs database-grants/tuning/prod/work_mem-readonlyuser.sql against the production cluster as doadmin.
2. Recycle the pgpool connection pool (restart the pgpool deployment). The new value only takes effect for new backend sessions at login time; pgpool holds long-lived pooled connections, so they must be recycled to pick up 64MB.

Rollback

Fully reversible, no restart, no data change:

ALTER ROLE readonlyuser IN DATABASE treetracker RESET work_mem;

Then recycle pgpool again.

Verification

• Stored default is present:

  SELECT r.rolname, d.datname, s.setconfig
  FROM pg_db_role_setting s
  JOIN pg_roles r ON r.oid = s.setrole
  JOIN pg_database d ON d.oid = s.setdatabase
  WHERE r.rolname = 'readonlyuser' AND d.datname = 'treetracker';

• Effective value in a fresh readonlyuser session after the pgpool recycle: SHOW work_mem; returns 64MB.
• Load effect: after 24 hours, temp-file writes for this workload (from pg_stat_statements and the monitor report) should drop noticeably for the mid-size queries. If there is no drop at all, re-check that pgpool actually recycled.

Risk and blast radius

• Scoped to one role in one database, so the change cannot affect writes, other services, or the primary.
• Reversible in seconds with RESET.
• The only failure mode is memory pressure on the replica if concurrency is far higher than observed; 64MB was sized against the measured peak of 30 to 33. Because the spillers are single-process Sorts (not parallel hashes), worst-case work_mem use is roughly 4 to 6GB on top of shared_buffers (about 3.2GB), well under 16GB, so headroom is comfortable. Monitoring the replica memory after the pgpool recycle is the safety check.

Follow-ups (not in this PR)

• Spatial index for the case1 family (the real fix for the largest spillers).
• Indexes or rewrite for the recursive organization_children CTE.
• Fix the wallet.token.capture_id::text cast that defeats an existing index.
• Revisit raising work_mem toward 256MB once the indexes ease the concurrent pile-up.

Note on Terraform activation

The database-grants Terraform is applied manually (there is no CI apply), and whether prod state is currently reconciled has not been confirmed from the remote state. This does not affect the change: the provider does not model work_mem, so a terraform apply will neither create nor revert it, and there is no drift on this parameter. If the role turns out to be Terraform-managed, no change to this PR is needed beyond noting that the role identity and grants stay in Terraform while this GUC stays here by provider limitation.

Checklist

• SQL reviewed by a second person
• DBA has prod doadmin access ready
• pgpool recycle scheduled with the apply
• Post-change temp-write drop confirmed within 24h
• Rollback command captured in the change record

[arnoldcastro5000]

tested on dev. configured on prod today.