01a0cd69 - Retry an underfunded ping the same UTC day

.env.example

@@ -6,6 +6,8 @@ RECIPIENTS_FILE=./recipients.json
 STATE_DIR=./state
 BIND_ADDR=0.0.0.0:3000
 SPEND_LIVE=false
+# Unset RETRY_CATCHUP_MS means 900000 ms; 0 disables the same-day insufficient_balance retry.
+# RETRY_CATCHUP_MS=900000
 SPEND_LIGHTNING_ADDRESS=
 SPEND_DASHBOARD_PASSWORD=
 TELEGRAM_BOT_TOKEN=

README.md

@@ -8,7 +8,7 @@ Ping-triggered Lightning gift payouts plus a tiny HTTP dashboard. This process *
 
 Recipient amounts are **USD**. Each daily payout (`POST /ping` with `kind` omitted or `"daily"`, or CLI) reads the live roster `STATE_DIR/recipients.json`, fetches Coinbase BTC-USD spot, and pays `round(usd / btcUsd * 1e8)` sats. `kind: "moderator"` pays the amount listed for that address on the moderator roster (see Moderator below). `kind: "welcome"` pays **1 USD** once per Lightning Address (see Welcome below). Missing or unusable spot, or a conversion under 1 sat, is fail-closed (exit `3`). The optional `amountSats` field in a seed JSON is a snapshot only — the process does not read it.
 
-The long-running server (`bun src/server.ts`) serves the dashboard, `POST /ping`, and `/healthz`. It does **not** pay the roster at UTC midnight, on boot catch-up, or on a 15-minute retry. There is no in-process midnight scheduler. A UTC day still starts at 00:00 UTC (existing JSONL day files). A daily payout (`kind` omitted or `"daily"`) happens when the 21.gifts API `POST`s `/ping` for a Lightning Address that is on the live roster — or unlisted with `GET /invoices/eligible` grant `status` `admitted` or `trial` (1 USD, new-member handbook cap) — has no `paid` or persisted `failed` row for today, and today's JSONL has no `uncertain` row for that address, any other live recipient, or `*halt*`. `kind: "moderator"` follows the Moderator section. `kind: "welcome"` follows the Welcome section. Replies never reach this process; the API pings only for top-level posts. `SPEND_LIVE=true` pays; otherwise the queued run is dry-run.
+The long-running server (`bun src/server.ts`) serves the dashboard, `POST /ping`, and `/healthz`. UTC midnight still does not pay the roster. Boot still does not walk the roster. CLI behaviour is unchanged. There is still no in-process midnight scheduler. A UTC day still starts at 00:00 UTC (existing JSONL day files). A daily payout (`kind` omitted or `"daily"`) happens when the 21.gifts API `POST`s `/ping` for a Lightning Address that is on the live roster — or unlisted with `GET /invoices/eligible` grant `status` `admitted` or `trial` (1 USD, new-member handbook cap) — has no `paid` or persisted `failed` row for today, and today's JSONL has no `uncertain` row for that address, any other live recipient, or `*halt*`. `kind: "moderator"` follows the Moderator section. `kind: "welcome"` follows the Welcome section. Replies never reach this process; the API pings only for top-level posts. `SPEND_LIVE=true` pays; otherwise the queued run is dry-run. A live ping (daily, moderator, or welcome) that returns `insufficient_balance` is written to `STATE_DIR/YYYY-MM-DD.retry.jsonl` and retried the same UTC day only. The default interval is 15 minutes (`RETRY_CATCHUP_MS`, milliseconds; unset means 900000; `0` disables), with one pass when the process starts the timer. The retry pays only those owed addresses, through the same one-at-a-time gate, and does not send Telegram again for the same preflight reason. A later paid run still notifies. When a payment switch is off, the retry does not pay that bucket and does not forget the address. No day-file `failed` or `uncertain` row is written for this reason, so a new ping the same day is still allowed.
 
 `GET /` is the only UI page. It always shows the Spend block:
 
@@ -48,7 +48,7 @@ Body: JSON `{ "address": string, "kind": "moderator", "groupMessageId"?: string
 
 State is a separate JSONL: `STATE_DIR/YYYY-MM-DD.moderator.jsonl` and `YYYY-MM-DD.moderator.finished`. Match the moderator JSONL case-insensitively; the first persisted address is the payout key. Once per UTC day per address on that file (`paid`, persisted `failed`, or own-address `uncertain` → `200` skipped with that reason). A `*halt*` row or another address's `uncertain` in the moderator JSONL does not skip this ping (the daily file still never skips a moderator ping). The moderator file does not skip a daily ping.
 
-Amount is the listed `amountUsd` with comment `21gifts moderator`. Same in-process payout gate as daily (one run at a time). Scheduler / catch-up stay no-ops; this is ping-triggered only. Otherwise `202` `{ "status": "accepted" }` without waiting for Lightning.
+Amount is the listed `amountUsd` with comment `21gifts moderator`. Same in-process payout gate as daily (one run at a time). The midnight scheduler stays a no-op; an insufficient-balance retry can still pay that moderator address later the same UTC day. Otherwise `202` `{ "status": "accepted" }` without waiting for Lightning.
 
 ### Welcome
 
@@ -118,10 +118,10 @@ Deploy workflows require GitHub Actions secrets `DOCKER_USERNAME`, `DOCKER_PASSW
 - Invoice-create network/5xx **before any pay** (no invoice id) is `invoice_unreachable`: skipped, not persisted, no `*halt*`, no `.finished`, exit `3`. Later recipients in the same run are still attempted. A later `POST /ping` for that address can retry
 - `uncertain` covers two cases that both halt the rest of a **daily** live run (`--live` or `SPEND_LIVE=true`), append a `*halt*` JSONL row, `markFinished`, exit `4`, and are **not** retried the same UTC day: (1) invoice-create parse failures (`malformed invoice response` / `malformed paymentHash`) without a pay; (2) after an invoice id / pay attempt (amount mismatch, lndhub.pay error, missing/mismatched preimage, proof failure). Moderator and welcome live runs do not append `*halt*` and do not stop later addresses in the same run
 - Persisted `failed` (invoice 4xx) is not paid again the same UTC day, does not count in the balance preflight sum, and for daily, when every live recipient is paid/uncertain/failed, `.finished` is set. For moderator, `.moderator.finished` is set when the synthetic stipend list of that run is settled
-- Only Lightning Addresses that currently have a passkey on 21.gifts and a live non-profile forum post are paid. Daily also requires a photo or video on that post (`GET /invoices/posted` `hasMedia: true`). Welcome uses the same passkey + `hasMedia` checks (no `postedAt` UTC-day match). The moderator $5 stipend is unchanged: living-room `hasPosted` plus `postedAt` on this UTC day, and **no** `hasMedia` requirement. CLI and other non-ping roster runs also require `{ eligible: true }` from `GET /invoices/eligible` (live `GET /invoices/passkey` and `GET /invoices/posted` each run; eligible only when `checkFundingEligible` is not false). Listed daily pings, all moderator pings, and welcome pings still do not call `GET /invoices/eligible` — the gifts API already gated `eligibleToday` before pinging, and `checkFundingEligible` stays false on ping payout (welcome never calls eligible even if that option is omitted or true). Unlisted daily pings call `GET /invoices/eligible` only to read grant `status` (`admitted`/`trial` → 1 USD; other statuses → `not_listed`; lookup fail → skip `eligible_unreachable`, HTTP 200, not process exit 3). Welcome pings do not look up the grant. CLI / roster runs unchanged: still require `{ eligible: true }` when `checkFundingEligible` is not false; lookup fail on those runs remains abort exit `3` `eligible_unreachable`. Addresses without a passkey (or without an account) are skipped as `no_passkey` with no JSONL row so a later ping can retry the same UTC day; addresses with a passkey but no live forum post are skipped as `no_post` the same way; daily and welcome addresses with a passkey and a live forum post but without photo/video (`hasMedia` not true) skip as `no_media` the same way (no JSONL, retry later; missing or non-true `hasMedia` is that skip, not `posted_unreachable`); addresses the api reports as not eligible today are skipped as `not_eligible` the same way. Invoice-create `403` `Forum post required` is the same `no_post` skip (not persisted `failed`). A failed passkey lookup aborts the whole run with exit `3` (`passkey_unreachable`) and pays no one; a failed posted lookup does the same as `posted_unreachable`; a failed eligible lookup on a CLI/roster run does the same as `eligible_unreachable`
+- Only Lightning Addresses that currently have a passkey on 21.gifts and a live non-profile forum post are paid. Daily also requires a photo or video on that post (`GET /invoices/posted` `hasMedia: true`). Welcome uses the same passkey + `hasMedia` checks (no `postedAt` UTC-day match). The moderator $5 stipend is unchanged: living-room `hasPosted` plus `postedAt` on this UTC day, and **no** `hasMedia` requirement. CLI and other non-ping roster runs also require `{ eligible: true }` from `GET /invoices/eligible` (live `GET /invoices/passkey` and `GET /invoices/posted` each run; eligible only when `checkFundingEligible` is not false). Listed daily pings, all moderator pings, and welcome pings still do not call `GET /invoices/eligible` — the gifts API already gated `eligibleToday` before pinging, and `checkFundingEligible` stays false on ping payout (welcome never calls eligible even if that option is omitted or true). The same-day `insufficient_balance` retry also keeps `checkFundingEligible` false and does not call `GET /invoices/eligible` again. Unlisted daily pings call `GET /invoices/eligible` only to read grant `status` (`admitted`/`trial` → 1 USD; other statuses → `not_listed`; lookup fail → skip `eligible_unreachable`, HTTP 200, not process exit 3). Welcome pings do not look up the grant. CLI / roster runs unchanged: still require `{ eligible: true }` when `checkFundingEligible` is not false; lookup fail on those runs remains abort exit `3` `eligible_unreachable`. Addresses without a passkey (or without an account) are skipped as `no_passkey` with no JSONL row so a later ping can retry the same UTC day; addresses with a passkey but no live forum post are skipped as `no_post` the same way; daily and welcome addresses with a passkey and a live forum post but without photo/video (`hasMedia` not true) skip as `no_media` the same way (no JSONL, retry later; missing or non-true `hasMedia` is that skip, not `posted_unreachable`); addresses the api reports as not eligible today are skipped as `not_eligible` the same way. Invoice-create `403` `Forum post required` is the same `no_post` skip (not persisted `failed`). A failed passkey lookup aborts the whole run with exit `3` (`passkey_unreachable`) and pays no one; a failed posted lookup does the same as `posted_unreachable`; a failed eligible lookup on a CLI/roster run does the same as `eligible_unreachable`
 - If a live process crashes, the next run steals the leftover lock once the owner pid is gone, or when this process reused the pid but the lock timestamp predates this incarnation. A lock whose pid belongs to a different live process is never stolen. Steal is serialized by a virgin `O_EXCL` `{day}.taking` file; an existing taking file is never replaced. Remove a lock or taking file by hand only after checking that no spend is running and inspecting `STATE_DIR/YYYY-MM-DD.jsonl`, `STATE_DIR/YYYY-MM-DD.moderator.jsonl`, and `STATE_DIR/welcome.jsonl`
 - Unreadable JSONL (truncated/corrupt line) aborts with exit `4` (`corrupt_state`) so a damaged `paid`/`uncertain` row cannot be ignored
-- State: `STATE_DIR/YYYY-MM-DD.jsonl`, `STATE_DIR/YYYY-MM-DD.finished`, `STATE_DIR/YYYY-MM-DD.moderator.jsonl`, `STATE_DIR/YYYY-MM-DD.moderator.finished`, `STATE_DIR/welcome.jsonl`, `STATE_DIR/welcome.finished`, `STATE_DIR/YYYY-MM-DD.lock` while a run is in progress, and `STATE_DIR/YYYY-MM-DD.taking` (`O_EXCL`, owner pid) briefly while a leftover lock is stolen
+- State: `STATE_DIR/YYYY-MM-DD.jsonl`, `STATE_DIR/YYYY-MM-DD.finished`, `STATE_DIR/YYYY-MM-DD.moderator.jsonl`, `STATE_DIR/YYYY-MM-DD.moderator.finished`, `STATE_DIR/welcome.jsonl`, `STATE_DIR/welcome.finished`, `STATE_DIR/YYYY-MM-DD.retry.jsonl` (owed-address queue for same-UTC-day `insufficient_balance` retries; not a payout log), `STATE_DIR/YYYY-MM-DD.lock` while a run is in progress, and `STATE_DIR/YYYY-MM-DD.taking` (`O_EXCL`, owner pid) briefly while a leftover lock is stolen
 
 Exit codes: `0` ok, `1` drain timeout after SIGTERM/SIGINT (55s cap), `2` config, `3` preflight/balance/lock/spot/invoice-create unreachable, `4` failed, uncertain, or halted.
 

src/__tests__/retry-queue.test.ts

@@ -0,0 +1,184 @@
+import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { describe, expect, it } from 'vitest';
+import { appendRetryOwed, loadRetryOwed, retryQueuePath } from '../retry-queue';
+
+const DAY = '2026-08-25';
+const MESSAGE_ID = 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa';
+const OTHER_MESSAGE_ID = 'cccccccc-cccc-cccc-cccc-cccccccccccc';
+const GROUP_MESSAGE_ID = 'bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb';
+
+function tmp(): string {
+  return mkdtempSync(join(tmpdir(), 'spend-retry-q-'));
+}
+
+describe('retryQueuePath', () => {
+  it(`ends with ${DAY}.retry.jsonl`, () => {
+    expect(retryQueuePath('/state', DAY).endsWith(`${DAY}.retry.jsonl`)).toBe(true);
+  });
+});
+
+describe('appendRetryOwed / loadRetryOwed', () => {
+  it('roundtrips a daily row with messageId and amountUsd and a moderator row without amountUsd', () => {
+    const dir = tmp();
+    try {
+      appendRetryOwed(dir, DAY, {
+        address: ' alice@walletofsatoshi.com ',
+        bucket: 'daily',
+        messageId: MESSAGE_ID,
+        amountUsd: 1,
+      });
+      appendRetryOwed(dir, DAY, {
+        address: 'bob@walletofsatoshi.com',
+        bucket: 'moderator',
+        groupMessageId: GROUP_MESSAGE_ID,
+        amountUsd: 7.5,
+      });
+      expect(loadRetryOwed(dir, DAY)).toEqual([
+        {
+          address: 'alice@walletofsatoshi.com',
+          bucket: 'daily',
+          messageId: MESSAGE_ID,
+          amountUsd: 1,
+        },
+        {
+          address: 'bob@walletofsatoshi.com',
+          bucket: 'moderator',
+          groupMessageId: GROUP_MESSAGE_ID,
+        },
+      ]);
+      const lines = readFileSync(retryQueuePath(dir, DAY), 'utf8')
+        .trim()
+        .split('\n')
+        .map((line) => JSON.parse(line) as Record<string, unknown>);
+      expect(lines[1]).not.toHaveProperty('amountUsd');
+    } finally {
+      rmSync(dir, { recursive: true, force: true });
+    }
+  });
+
+  it('does not replace the first messageId or write a second line for the same identity', () => {
+    const dir = tmp();
+    try {
+      appendRetryOwed(dir, DAY, {
+        address: 'alice@walletofsatoshi.com',
+        bucket: 'daily',
+        messageId: MESSAGE_ID,
+        amountUsd: 1,
+      });
+      appendRetryOwed(dir, DAY, {
+        address: 'ALICE@walletofsatoshi.com',
+        bucket: 'daily',
+        messageId: OTHER_MESSAGE_ID,
+        amountUsd: 2,
+      });
+      expect(loadRetryOwed(dir, DAY)).toEqual([
+        {
+          address: 'alice@walletofsatoshi.com',
+          bucket: 'daily',
+          messageId: MESSAGE_ID,
+          amountUsd: 1,
+        },
+      ]);
+      const lines = readFileSync(retryQueuePath(dir, DAY), 'utf8').trim().split('\n');
+      expect(lines).toHaveLength(1);
+      expect(lines[0]).toContain(MESSAGE_ID);
+      expect(lines[0]).not.toContain(OTHER_MESSAGE_ID);
+    } finally {
+      rmSync(dir, { recursive: true, force: true });
+    }
+  });
+
+  it('skips a corrupt JSON line and keeps a following valid line', () => {
+    const dir = tmp();
+    try {
+      writeFileSync(
+        retryQueuePath(dir, DAY),
+        `not-json\n${JSON.stringify({
+          address: 'alice@walletofsatoshi.com',
+          bucket: 'daily',
+          messageId: MESSAGE_ID,
+        })}\n`,
+      );
+      expect(loadRetryOwed(dir, DAY)).toEqual([
+        {
+          address: 'alice@walletofsatoshi.com',
+          bucket: 'daily',
+          messageId: MESSAGE_ID,
+        },
+      ]);
+    } finally {
+      rmSync(dir, { recursive: true, force: true });
+    }
+  });
+
+  it('returns [] when the file is missing', () => {
+    const dir = tmp();
+    try {
+      expect(loadRetryOwed(dir, DAY)).toEqual([]);
+    } finally {
+      rmSync(dir, { recursive: true, force: true });
+    }
+  });
+
+  it('skips a bad address, bad bucket, bad UUID, and non-positive or non-number amountUsd', () => {
+    const dir = tmp();
+    try {
+      const valid = {
+        address: 'ok@walletofsatoshi.com',
+        bucket: 'daily',
+        messageId: MESSAGE_ID,
+      };
+      writeFileSync(
+        retryQueuePath(dir, DAY),
+        [
+          JSON.stringify({ address: 'not-an-address', bucket: 'daily' }),
+          JSON.stringify({ address: 'alice@walletofsatoshi.com', bucket: 'other' }),
+          JSON.stringify({
+            address: 'alice@walletofsatoshi.com',
+            bucket: 'daily',
+            messageId: 'not-a-uuid',
+          }),
+          JSON.stringify({ address: 'alice@walletofsatoshi.com', bucket: 'daily', amountUsd: 0 }),
+          JSON.stringify({ address: 'alice@walletofsatoshi.com', bucket: 'daily', amountUsd: -1 }),
+          JSON.stringify({ address: 'alice@walletofsatoshi.com', bucket: 'daily', amountUsd: '1' }),
+          JSON.stringify(valid),
+        ].join('\n') + '\n',
+      );
+      expect(loadRetryOwed(dir, DAY)).toEqual([valid]);
+    } finally {
+      rmSync(dir, { recursive: true, force: true });
+    }
+  });
+
+  it('keeps daily and moderator rows with the same address as two identities', () => {
+    const dir = tmp();
+    try {
+      appendRetryOwed(dir, DAY, {
+        address: 'alice@walletofsatoshi.com',
+        bucket: 'daily',
+        messageId: MESSAGE_ID,
+      });
+      appendRetryOwed(dir, DAY, {
+        address: 'alice@walletofsatoshi.com',
+        bucket: 'moderator',
+        groupMessageId: GROUP_MESSAGE_ID,
+      });
+      expect(loadRetryOwed(dir, DAY)).toEqual([
+        {
+          address: 'alice@walletofsatoshi.com',
+          bucket: 'daily',
+          messageId: MESSAGE_ID,
+        },
+        {
+          address: 'alice@walletofsatoshi.com',
+          bucket: 'moderator',
+          groupMessageId: GROUP_MESSAGE_ID,
+        },
+      ]);
+    } finally {
+      rmSync(dir, { recursive: true, force: true });
+    }
+  });
+});