Skip to content

fix(bd): reap stale .beads/issues.jsonl on managed scopes#2060

Draft
scarson wants to merge 1 commit into
gastownhall:mainfrom
scarson:fix/mail-send-bd-create-autoimport
Draft

fix(bd): reap stale .beads/issues.jsonl on managed scopes#2060
scarson wants to merge 1 commit into
gastownhall:mainfrom
scarson:fix/mail-send-bd-create-autoimport

Conversation

@scarson
Copy link
Copy Markdown
Contributor

@scarson scarson commented May 13, 2026

Closes #2059. Companion to #1968 (which fixed export.auto going forward) and #1930 (empty .beads/dolt/ case).

Summary

  • #1968 made export.auto: false canonical so bd doesn't regenerate .beads/issues.jsonl on every write. That fix is correct but only suppresses future exports.
  • On long-lived managed cities, a pre-existing 15 MB / 34k-row issues.jsonl from before fix(bd): force export.auto: false in managed config (#1965) #1968 still triggers bd's auto-import-on-write path on every subsequent bd write. The bd subprocess stalls at the 2m timeout re-importing all 34k rows into Dolt before the write completes.
  • Symptom: gc mail send / gc sling / gc session new all time out at 2m on these cities. The user-visible workaround is manual rm .beads/issues.jsonl per scope, which is exactly what fix(bd): force export.auto: false in managed config (#1965) #1968 was supposed to eliminate.
  • This PR adds the stale-file cleanup as a two-layer defense:
    1. bdStoreFor{City,Rig} (cmd/gc/bd_env.go) — best-effort reap when the scope is gc-managed (steady-state, every store construction).
    2. ensureCanonicalScopeConfigState (cmd/gc/beads_provider_lifecycle.go) — reap when canonicalization actually mutates config (transition-moment cleanup for newly-normalized scopes).
  • Plus: tighten bdRuntimeEnv to set BD_EXPORT_AUTO=false on every gc-spawned bd subprocess. Belt to the canonical config's suspenders — covers fresh inits whose config hasn't been canonicalized yet, and short-circuits the export → next-write-import cycle even when an out-of-band caller left a stale JSONL on disk.

Why this is safe

  • The reaper only fires when the scope is provably gc-managed:
  • Rigs that deliberately keep JSONL-based sharing — those that set gc.endpoint_origin: explicit per fix(bd): force export.auto: false in managed config (#1965) #1968's escape hatch — are explicitly not treated as gc-managed, so their issues.jsonl is left alone.
  • Unmanaged scopes (no .beads/config.yaml, or config without any gc-managed signal) are also left alone — the reaper is conservative by default.
  • All operations are best-effort: os.Stat / os.Remove errors are swallowed because the env-var BD_EXPORT_AUTO=false in bdRuntimeEnv is a second line of defense for gc-initiated calls. A concurrent reader of the JSONL (e.g., a third-party bd-aware tool) won't fail the caller's operation.

Verified locally

samtown (34k beads, 15 MB stale JSONL, dolt server healthy):

command before after
gc mail send mayor "..." "..." from rig 120s (timeout) 3.0s
gc mail count 2.5s 1.7s
gc mail read <id> -- 2.2s
gc bd list 0.3s 0.3s
gc bd --rig <r> list 0.3s 0.3s

After the patched binary handles a write, .beads/issues.jsonl does not reappear — confirming the env-var + canonical-config defenses together hold the line.

Testing

  • Targeted unit tests (TestBdRuntimeEnvDisablesAutoExport, TestScopeIsGCManaged*, TestReapStaleBdExportJSONL*, TestReadExportAuto*, TestEnsureCanonicalConfigForcesAutoExportOff*) — all pass.
  • go vet ./... — clean.
  • gofmt -l on touched files — clean.
  • golangci-lint run --new-from-rev=main cmd/gc/... internal/beads/contract/... — 0 issues.
  • Live end-to-end probe on samtown (numbers above).
  • make check — there are pre-existing macOS-specific test flakes on main (TestResolveDoltConnectionTargetManagedCity_EnvOverride fails to bind 127.0.0.2; TestControllerStateMutationRollsBackWhenRefreshFails hangs at 5m) that I reproduce on clean 1c5b6073 before any of my changes. CI on Linux should run cleanly — flagging this for the reviewer rather than papering over it.
  • make check-docs — no docs/nav/links touched.
  • make test-integration — would appreciate the reviewer's read on whether the integration suite covers the canonicalize-then-write path I added a reaper to.

Checklist

Why this is a draft

CONTRIBUTING.md says to discuss in an issue first; #2059 is the issue. Marking as draft pending maintainer triage on:

  1. Whether the two-layer defense is the right approach (vs. e.g. a one-shot migration in gc doctor --fix).
  2. Whether the gc-managed signal set (export.auto: false OR managed endpoint origins, exclude explicit) matches the project's mental model.
  3. Whether a make check-docs / make test-integration confirmation is required before un-drafting.

Happy to iterate on any of the above.

PR gastownhall#1968 (closes gastownhall#1965) made `export.auto: false` canonical in
.beads/config.yaml so bd doesn't regenerate the JSONL export on every
write. That fix is correct, but it only suppresses *future* exports —
it doesn't remove a pre-existing issues.jsonl left on disk from before
the normalization. On long-lived managed cities (one observed at 15 MB /
34k issues), every subsequent `bd create` still hits bd's
auto-import-on-write path, sees the stale JSONL, decides the dolt scope
needs a refresh, and stalls for the full 2-minute subprocess timeout
re-importing all 34k rows before appending the new record.

Symptom from a samtown user: `gc mail send mayor "..." "..."` from any
rig agent times out at 2m. `gc mail count` also slow (~3s on a healthy
city; would be sub-second). Reproduces deterministically when the
stale file is present, even though the config was already canonical.

Two layers of defense:

1. **`cmd/gc/bd_env.go`** — when `bdStoreForCity` / `bdStoreForRig`
   constructs a store, best-effort remove `.beads/issues.jsonl` if the
   scope is gc-managed (config has `export.auto: false` or
   `gc.endpoint_origin` is one of the canonical managed origins).
   This catches the steady-state case where canonicalization already
   happened in an earlier `gc` run. Explicit-opt-out rigs
   (`gc.endpoint_origin: explicit`) are skipped — those rigs deliberately
   keep JSONL-based sharing.

2. **`cmd/gc/beads_provider_lifecycle.go`** — when
   `ensureCanonicalScopeConfigState` actually mutates config.yaml (i.e.
   the first run after the PR-gastownhall#1968 contract was added), also remove
   any stale JSONL. This catches the transition moment for newly-
   canonicalized scopes.

`internal/beads/contract/files.go` gets a new `ReadExportAuto` helper
so the bd_env layer can read the export.auto state without depending
on internal config parsing.

Also tightens `bdRuntimeEnv` to set `BD_EXPORT_AUTO=false` explicitly
on every gc-spawned bd subprocess. This is a per-invocation belt to the
config's suspenders — it covers fresh inits whose canonical config has
not yet been written, and short-circuits the export → next-write-import
cycle even if an out-of-band caller has left a stale JSONL on disk.

Verified locally on samtown (34k beads, 15 MB stale JSONL):
- `gc mail send mayor "..." "..."` from rig: 120s → 3s
- `gc mail count`: 1.7s (was 2-3s before)
- `gc mail read <id>`: 2.2s
- `gc bd list`, `gc bd --rig <r> list`: ~0.3s (unchanged)

7 new tests cover scope-managed detection (3 cases: explicit
export.auto:false, managed origin, explicit opt-out), reaper behavior
(3 cases: removes on managed, leaves on explicit opt-out, leaves on
unmanaged), and `BD_EXPORT_AUTO=false` env var. The `ReadExportAuto`
contract helper gets parametric coverage plus a missing-file case.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: pre-existing .beads/issues.jsonl on managed scope still stalls bd writes for 2m after #1968

1 participant