diff --git a/README.md b/README.md index c0905d0..aa299ae 100644 --- a/README.md +++ b/README.md @@ -24,32 +24,36 @@ Reading the news is either too time-consuming or too algorithmically poisoned. M ## Architecture ``` -Vercel Cron — /api/daily-digest (05:00 UTC daily) +GitHub Actions — daily-digest.yml (06:00 UTC daily) → Perplexity Sonar API × N regions (parallel) - → format into digest JSON + → Claude ranks headlines per region (+ cross-region global digest) → store in Supabase → Firebase Cloud Messaging ping (null-notify_at devices) -Vercel Cron — /api/notify (every 30 min) - → FCM ping to devices whose notify_at matches the current window +GitHub Actions — notify.yml (every 30 min) + → FCM ping to devices whose notify_at falls in (last_run_at, now] — a + catch-up window so no scheduled device is dropped when the cron fires late ``` | Component | Role | | ------------------- | ------------------------------------------------------------------------------------------ | | Perplexity Sonar | News source — one call per region; handles sourcing, recency, deduplication, summarization | -| Supabase | Authoritative store for digests and device tokens | +| Claude (Anthropic) | Ranks headlines by importance per region, and selects cross-region global top stories | +| Supabase | Authoritative store for digests, device tokens, and cron run reports | | Firebase (FCM) | Lightweight "digest ready" ping; full content fetched from Supabase on tap | | React Native (Expo) | Android-first app; local AsyncStorage cache + Supabase for recovery | +| GitHub Actions | Runs the two cron jobs (digest, notify) and CI | --- ## Subsystems -| Directory | README | -| ----------- | ---------------------------------------------------------------------- | -| `cron/` | [cron/README.md](cron/README.md) — pipeline, config, how to run | -| `app/` | [app/README.md](app/README.md) — Expo setup, feature phases, data flow | -| `supabase/` | [supabase/README.md](supabase/README.md) — schema, RLS, V2 additions | +| Directory | README | +| ----------- | ----------------------------------------------------------------------------- | +| `cron/` | [cron/README.md](cron/README.md) — pipeline, config, run-report observability | +| `app/` | [app/README.md](app/README.md) — Expo setup, navigation, data flow | +| `supabase/` | [supabase/README.md](supabase/README.md) — schema, RLS, definer RPCs | +| `shared/` | [shared/CLAUDE.md](shared/CLAUDE.md) — region constants, config, shared types | --- @@ -59,11 +63,14 @@ Vercel Cron — /api/notify (every 30 min) | ---------------- | ------------------------------------------------------------------------------------------------------- | | Platform | Android first | | News source | Perplexity Sonar API — one call per region, parallel | +| Ranking | Claude reorders headlines per region by importance, and selects a cross-region global digest | +| Scheduling | GitHub Actions cron (best-effort); notify uses a `(last_run_at, now]` catch-up window for late fires | | Push strategy | Cron writes to Supabase, then FCM ping; app fetches on tap | | Digest storage | One row per region per date, shared across all users | | Device identity | UUID generated on first launch; resets on reinstall | -| Auth | Supabase Auth — email/password; `user_preferences` keyed on `auth.uid()` (Phase 4, done) | -| Device writes | Publishable key + open INSERT/UPDATE RLS (MVP); V2: Edge Function with secret key | +| Auth | Supabase Auth — email/password; `user_preferences` keyed on `auth.uid()` | +| Device writes | RLS-locked `devices` table; all client access via SECURITY DEFINER RPCs (no open anon writes) | +| Observability | Each pipeline run persists a report row to `pipeline_runs` (cost, tokens, per-region success/failure) | | Source diversity | Prompt-driven (preferred outlets hint per region); domain allow-lists rejected — caused empty results | | URL quality | Pattern filter on model output; HTTP HEAD validation rejected — news sites return 403 for section pages | | Bias control | Prompt-driven neutrality; `user_location.country` steers regional sourcing | diff --git a/app/README.md b/app/README.md index cf63f81..a2987c4 100644 --- a/app/README.md +++ b/app/README.md @@ -26,75 +26,33 @@ npm install Copy `.env.example` to `.env` inside `app/` and fill in: -| Variable | Description | -| ------------------------------- | ------------------------------- | -| `EXPO_PUBLIC_SUPABASE_URL` | Supabase project URL | -| `EXPO_PUBLIC_SUPABASE_ANON_KEY` | Supabase anon (publishable) key | +| Variable | Description | +| -------------------------- | ------------------------------- | +| `EXPO_PUBLIC_SUPABASE_URL` | Supabase project URL | +| `EXPO_PUBLIC_SUPABASE_KEY` | Supabase publishable (anon) key | -FCM config lives in `app/android/app/google-services.json`, not env vars. - ---- - -## Android dev setup on WSL2 (one-time) - -The Android SDK lives on Windows; WSL2 needs to reach it. - -**1. Add to `~/.bashrc`:** - -```bash -export ANDROID_HOME=/mnt/c/Users/HP/AppData/Local/Android/Sdk -export JAVA_HOME="/mnt/c/Program Files/Android/Android Studio/jbr" -export PATH=$PATH:$ANDROID_HOME/platform-tools:$ANDROID_HOME/tools/bin:$ANDROID_HOME/emulator:$JAVA_HOME/bin -``` - -**2. Create an `adb` wrapper** so WSL2 uses the Windows ADB server (which can reach your phone): - -```bash -mkdir -p ~/.local/bin -cat > ~/.local/bin/adb << 'EOF' -#!/bin/bash -exec /mnt/c/Users/HP/AppData/Local/Android/Sdk/platform-tools/adb.exe "$@" -EOF -chmod +x ~/.local/bin/adb -``` - -**3. Pair your phone (one-time per device):** - -- Phone: Settings → Developer Options → Wireless debugging → **Pair device with pairing code** -- Note the pairing port and 6-digit code -- In Windows CMD (not WSL2): - -```cmd -adb pair 192.168.64.4: -``` - -Enter the code. You'll see `Successfully paired`. +FCM config lives in `app/google-services.json` (referenced from `app.json` as `./google-services.json`), not env vars. The file is gitignored — never commit it. --- ## Connecting your phone (every session) -Your phone's connection port changes each session. Run the helper script: +Wireless debugging port changes each session. Use the helper script: ```bash ./scripts/adb-connect.sh ``` -Or manually — check the port in Settings → Developer Options → Wireless debugging, then in Windows CMD: - -```cmd -adb connect 192.168.64.4: -``` - -Verify WSL2 sees it: +Or manually — check the port in Settings → Developer Options → Wireless debugging, then: ```bash -adb devices # should list 192.168.64.4: device +adb connect : +adb devices # should list : device ``` --- -## Running (added by app/foundation slice) +## Running This app uses native modules (`@react-native-firebase/messaging`), so **Expo Go will not work** — a custom dev client is required. @@ -112,16 +70,24 @@ npx expo start --dev-client # subsequent runs — scan QR in the installed d ```bash npm run build # tsc --noEmit (typecheck) npm run lint # ESLint on src/ -npm test # Jest (available after app/foundation slice) +npm test # Jest ``` --- +## Navigation + +No React Navigation. The root `App.tsx` conditionally renders screens based on the Zustand `nav` slice (`splash` / `digest` / `settings` / `login`), gated by an `appState` machine (`booting` → `prefs-loading` → `ready`, plus `maintenance` and `update-required` stub screens). Nav state (screen + day index) persists to MMKV with a TTL; the open article is transient. + +--- + ## Data flow **Digests:** cache-first (MMKV) with a configurable stale window for today. Notification tap forces a full remote fetch. Past dates are immutable — never re-fetched. -**Auth:** `useSupabaseAuth` manages session via Supabase. MMKV persists the session across restarts. Password-reset deep links (`pulse://reset-password`) use PKCE or implicit token flow depending on what Supabase sends. +**Auth:** `useSupabaseAuth` manages session via Supabase. MMKV persists the session across restarts. + +**Deep links:** handled in `useDeepLinkRecovery` via `expo-linking` — both cold-start (`Linking.getInitialURL()`) and warm (`addEventListener('url', …)`). Two schemes are recognized: `pulse://reset-password` (password recovery → routes to the reset screen) and `pulse://confirm` (sign-up confirmation → establishes the session, which routes into the app). Each is exchanged via PKCE (`exchangeCodeForSession`) or implicit token flow (`setSession`) depending on what Supabase sends. Duplicate URLs are de-duped. **Preferences:** keyed on `session.user.id`. Local writes are immediate. Supabase push is batched — flushed on settings close and on app background. diff --git a/cron/README.md b/cron/README.md index 0faaa31..38d6680 100644 --- a/cron/README.md +++ b/cron/README.md @@ -1,6 +1,6 @@ # Pulse — Cron Pipeline -Daily digest pipeline: fetches regional headlines from Perplexity, stores results in Supabase, and fires FCM push notifications to registered devices. +Daily digest pipeline: fetches regional headlines from Perplexity, ranks them with Claude, stores results in Supabase, and fires FCM push notifications to registered devices. Runs as GitHub Actions cron jobs (`.github/workflows/daily-digest.yml`, `.github/workflows/notify.yml`). --- @@ -47,6 +47,7 @@ npm run e2e:countryRanking -- US GB DE # per-region rank on selected region | Variable | Description | | ----------------------- | ----------------------------------------------------- | | `PERPLEXITY_API_KEY` | Perplexity Sonar API key | +| `ANTHROPIC_API_KEY` | Claude API key (headline ranking) | | `SUPABASE_URL` | Supabase project URL | | `SUPABASE_SECRET_KEY` | Service-role key (bypasses RLS) | | `FIREBASE_PROJECT_ID` | Firebase project ID | @@ -70,17 +71,33 @@ loadPulseConfig() → createSource() → runFetchPipeline() → resolveRegions() → fetchDigest() × N [Promise.allSettled, staggered] - → Perplexity retry loop + → Perplexity retry loop [429/5xx, exponential backoff] → parseHeadlines() → URL filter + slug dedup + topic dedup → rankHeadlines() [Claude — per-region reorder] + → deduplicateAcrossDigests() [null out images shared across regions] → persistDigests() [upsert to Supabase] - → rankGlobalHeadlines() [Claude — cross-region top stories] + → rankGlobalHeadlines() [Claude — cross-region top stories, if enabled] → persistGlobalDigest() - → sendNotifications() [FCM multicast] - → writeRunLog() + → dispatchFcm() [FCM multicast] + → buildRunReport() → formatRunReportSummary() (log line) → persistRunReport() [→ pipeline_runs] + → writeRunLog() [JSONL time-series, only if log.qualityLog] ``` -`api/daily-digest.ts` follows the same path but only FCM-pings devices with `notify_at = NULL`. `api/notify.ts` handles devices with a specific scheduled time. +The two cron jobs share `runFetchPipeline`: + +- `jobs/daily-digest.ts` runs the full pipeline above, then FCM-pings devices with `notify_at = NULL` ("notify me when the digest is ready"). +- `jobs/notify.ts` runs every ~30 min and notifies devices whose `notify_at` falls in `(last_run_at, now]` — a catch-up window so no scheduled device is dropped when the cron fires irregularly. + +--- + +## Run-report observability + +Every `daily-digest` run assembles a versioned `RunReport` (`schema: pulse.run.v1`) — the durable, queryable companion to the human-readable log lines. It aggregates region success/failure, headline counts, fetch + ranking token cost, notify outcome, and duration. + +- A one-line summary (`formatRunReportSummary`) is logged via Winston. +- The full report is best-effort persisted to `public.pipeline_runs` (`persistRunReport`): columns `run_at`, `status` (`ok`/`partial`), `total_cost_usd`, `total_tokens`, `regions_succeeded`, `regions_failed`, `duration_ms`, plus the whole `report` as JSON. Persistence failures are logged and swallowed — observability never breaks the digest pipeline. + +Bump `schema` on any breaking shape change so downstream queries can branch on the version. --- diff --git a/cron/e2e/fetch.ts b/cron/e2e/fetch.ts index b584c8b..003c324 100644 --- a/cron/e2e/fetch.ts +++ b/cron/e2e/fetch.ts @@ -49,7 +49,7 @@ async function main() { printTotals(retryStats, resolvedRegions.length, fetchUsage, rankingUsage, config.api.fetch.count); if (config.api.ranking.global.enabled) { - const globalHeadlines = await rankGlobalHeadlines(digests, config); + const { headlines: globalHeadlines } = await rankGlobalHeadlines(digests, config); printGlobalHeadlines(globalHeadlines); } diff --git a/cron/e2e/full.ts b/cron/e2e/full.ts index 3dfa942..f50398b 100644 --- a/cron/e2e/full.ts +++ b/cron/e2e/full.ts @@ -34,7 +34,7 @@ async function main() { await persistDigests(digests, config); if (config.api.ranking.global.enabled) { - const globalHeadlines = await rankGlobalHeadlines(digests, config); + const { headlines: globalHeadlines } = await rankGlobalHeadlines(digests, config); if (globalHeadlines.length > 0) { await persistGlobalDigest(globalHeadlines); } diff --git a/cron/e2e/globalRanking.ts b/cron/e2e/globalRanking.ts index da0b96b..ce9e34b 100644 --- a/cron/e2e/globalRanking.ts +++ b/cron/e2e/globalRanking.ts @@ -40,7 +40,7 @@ async function main() { digests.forEach((d) => log.info(` ${d.region.padEnd(20)} ${d.headlines.length} headlines`)); log.info('\nRunning global ranking…'); - const globalHeadlines = await rankGlobalHeadlines(digests, config); + const { headlines: globalHeadlines } = await rankGlobalHeadlines(digests, config); printGlobalHeadlines(globalHeadlines); } diff --git a/cron/jobs/daily-digest.ts b/cron/jobs/daily-digest.ts index e0fd59f..ed8c910 100644 --- a/cron/jobs/daily-digest.ts +++ b/cron/jobs/daily-digest.ts @@ -1,13 +1,20 @@ import { loadPulseConfig, createSource } from '../src/config'; import { persistDigests, persistGlobalDigest, buildClient, dispatchFcm } from '../src/notify'; import { rankGlobalHeadlines } from '../src/rankHeadlines'; -import { getLogger } from '../src/logging'; +import { getLogger, formatError } from '../src/logging'; +import type { DigestUsage } from '../src/types'; import { buildRunLog, deduplicateAcrossDigests, runFetchPipeline, writeRunLog, } from '../src/pipeline'; +import { + buildRunReport, + formatRunReportSummary, + persistRunReport, + type NotifyOutcome, +} from '../src/runReport'; /** * GitHub Actions cron job (.github/workflows/daily-digest.yml) — fetch all @@ -25,6 +32,8 @@ async function main(): Promise { const source = createSource(config); try { + log.info(`Starting daily digest — ${config.api.regions.length} region(s) configured`); + const { resolvedRegions, digests: fetchedDigests, @@ -34,7 +43,7 @@ async function main(): Promise { const digests = deduplicateAcrossDigests(fetchedDigests); errors.forEach((error) => - log.error(`Region fetch failed: ${error.region}: ${String(error.reason)}`), + log.error(`Region fetch failed: ${error.region}: ${formatError(error.reason)}`), ); if (digests.length === 0) { @@ -43,8 +52,10 @@ async function main(): Promise { await persistDigests(digests, config); + let globalRankingUsage: DigestUsage | null = null; if (config.api.ranking.global.enabled) { - const globalHeadlines = await rankGlobalHeadlines(digests, config); + const { headlines: globalHeadlines, usage } = await rankGlobalHeadlines(digests, config); + globalRankingUsage = usage; if (globalHeadlines.length > 0) { await persistGlobalDigest(globalHeadlines); @@ -58,28 +69,40 @@ async function main(): Promise { .select('fcm_token') .is('notify_at', null); + let notify: NotifyOutcome = { devicesTargeted: 0, sent: 0, failed: 0 }; if (error) { log.warn(`Failed to read null-notify_at devices: ${error.message}`); } else if (devices?.length) { const tokens = devices.map((d) => d.fcm_token as string); const regions = digests.map((d) => d.region).join(','); - await dispatchFcm(tokens, regions); + const result = await dispatchFcm(tokens, regions); + notify = { + devicesTargeted: result.total, + sent: result.sent, + failed: result.total - result.sent, + }; } - const totalTokens = digests.reduce((sum, d) => sum + (d.usage?.totalTokens ?? 0), 0); - - log.info(`Done — ${digests.length}/${resolvedRegions.length} regions, ${totalTokens} tokens`); + const report = buildRunReport({ + config, + resolvedRegions, + digests, + errors, + startTime, + globalRankingUsage, + notify, + }); + log.info(formatRunReportSummary(report)); + await persistRunReport(report); if (config.log.qualityLog) { const runLog = buildRunLog(config, resolvedRegions, digests, errors, startTime); - const logPath = writeRunLog(runLog, resolvedRegions); - log.info(`Quality log → ${logPath}`); } } catch (err) { - log.error(`Unhandled error: ${String(err)}`); + log.error(`Unhandled error: ${formatError(err)}`); process.exit(1); } diff --git a/cron/jobs/notify.ts b/cron/jobs/notify.ts index 98ace4b..5e87499 100644 --- a/cron/jobs/notify.ts +++ b/cron/jobs/notify.ts @@ -1,6 +1,6 @@ import { loadPulseConfig } from '../src/config'; import { sendDueNotifications } from '../src/notify'; -import { getLogger } from '../src/logging'; +import { getLogger, formatError } from '../src/logging'; /** * GitHub Actions cron job (.github/workflows/notify.yml) — notify devices whose @@ -17,10 +17,11 @@ async function main(): Promise { const log = getLogger('notify-cron'); try { + log.info('Starting due-notification run'); const { sent, total } = await sendDueNotifications(); log.info(`Sent ${sent}/${total} notifications`); } catch (err) { - log.error(`Unhandled error: ${String(err)}`); + log.error(`Unhandled error: ${formatError(err)}`); process.exit(1); } diff --git a/cron/package.json b/cron/package.json index 404e2c6..f0a2634 100644 --- a/cron/package.json +++ b/cron/package.json @@ -4,7 +4,7 @@ "version": "0.1.0", "scripts": { "build": "tsc --noEmit", - "lint": "eslint --ext .ts src api jobs", + "lint": "eslint --ext .ts src jobs e2e", "test": "jest", "test:watch": "jest --watch", "test:coverage": "jest --coverage", diff --git a/cron/src/lib/parseHeadlines.ts b/cron/src/lib/parseHeadlines.ts index fa4e370..cc3c0cd 100644 --- a/cron/src/lib/parseHeadlines.ts +++ b/cron/src/lib/parseHeadlines.ts @@ -112,7 +112,7 @@ export async function parseHeadlines( if (seenInBatch.has(headline.url)) return false; const slug = urlSlug(headline.url); if (slug && usedSlugs.has(slug)) { - log.info(`Slug duplicate filtered: ${headline.url}`); + log.debug(`Slug duplicate filtered: ${headline.url}`); return false; } seenInBatch.add(headline.url); diff --git a/cron/src/logging.ts b/cron/src/logging.ts index 40871db..e74f06b 100644 --- a/cron/src/logging.ts +++ b/cron/src/logging.ts @@ -7,9 +7,13 @@ const customFormat = printf((info) => { return `${info['timestamp']} ${info.level} (${info['component']}) ${info.message}`; }); +// ANSI colors aid readability in a terminal but become escape-code noise in +// non-TTY sinks such as GitHub Actions logs — colorize only on a real TTY. +const formats = [timestamp(), ...(process.stdout.isTTY ? [colorize()] : []), customFormat]; + const logger = winston.createLogger({ level: 'info', - format: combine(timestamp(), colorize(), customFormat), + format: combine(...formats), transports: [new winston.transports.Console()], }); @@ -20,3 +24,12 @@ export function configureLogger(config: PulseConfig): void { export function getLogger(component: string): winston.Logger { return logger.child({ component }); } + +/** + * Render an unknown thrown value for a log line, preserving the stack trace for + * Error instances. The job entrypoints previously logged `String(err)`, which + * collapsed errors to "[object Object]"/message-only and dropped the stack. + */ +export function formatError(err: unknown): string { + return err instanceof Error ? (err.stack ?? err.message) : String(err); +} diff --git a/cron/src/rankHeadlines.ts b/cron/src/rankHeadlines.ts index f26c98a..4dd3871 100644 --- a/cron/src/rankHeadlines.ts +++ b/cron/src/rankHeadlines.ts @@ -72,6 +72,23 @@ export interface RankingResult { usage: DigestUsage | null; } +export interface GlobalRankingResult { + headlines: GlobalHeadline[]; + /** Combined usage across every Claude pass (round 1 chunks + round 2). Null when no call was made. */ + usage: DigestUsage | null; +} + +/** Sum a list of usage records into one. Returns null when the list is empty. */ +function sumUsages(usages: DigestUsage[]): DigestUsage | null { + if (usages.length === 0) return null; + return usages.reduce((acc, u) => ({ + promptTokens: acc.promptTokens + u.promptTokens, + completionTokens: acc.completionTokens + u.completionTokens, + totalTokens: acc.totalTokens + u.totalTokens, + costUsd: acc.costUsd + u.costUsd, + })); +} + /** * Reorders headlines by country importance using Claude. * Falls back to original order if the ranking call fails or returns an invalid result. @@ -172,7 +189,7 @@ async function globalRankingPass( count: number, model: string, maxTokens: number, -): Promise { +): Promise<{ headlines: GlobalHeadline[]; usage: DigestUsage }> { const log = getLogger('rankHeadlines'); const actualCount = Math.min(count, candidates.length); @@ -187,6 +204,12 @@ async function globalRankingPass( const { input_tokens, output_tokens } = response.usage; const costUsd = input_tokens * COST_PER_INPUT_TOKEN + output_tokens * COST_PER_OUTPUT_TOKEN; + const usage: DigestUsage = { + promptTokens: input_tokens, + completionTokens: output_tokens, + totalTokens: input_tokens + output_tokens, + costUsd, + }; log.info( `Global pass — ${candidates.length} candidates → ${actualCount} — ${input_tokens}+${output_tokens} tokens | $${costUsd.toFixed(6)}`, ); @@ -211,7 +234,7 @@ async function globalRankingPass( seen.add(idx); } - return indices.map((i) => { + const headlines = indices.map((i) => { const c = candidates[i - 1]!; return { title: c.title, @@ -223,6 +246,7 @@ async function globalRankingPass( imageUrl: c.imageUrl, }; }); + return { headlines, usage }; } /** @@ -234,9 +258,9 @@ async function globalRankingPass( export async function rankGlobalHeadlines( digests: RegionDigest[], config: PulseConfig, -): Promise { +): Promise { const { enabled, count, model, maxTokens, chunkSize } = config.api.ranking.global; - if (!enabled) return []; + if (!enabled) return { headlines: [], usage: null }; const log = getLogger('rankHeadlines'); const candidates: Candidate[] = digests.flatMap((d) => @@ -250,21 +274,21 @@ export async function rankGlobalHeadlines( imageUrl: h.imageUrl, })), ); - if (candidates.length === 0) return []; + if (candidates.length === 0) return { headlines: [], usage: null }; const client = getClient(); if (!client) { log.warn('ANTHROPIC_API_KEY not set — skipping global ranking'); - return []; + return { headlines: [], usage: null }; } const actualCount = Math.min(count, candidates.length); try { if (candidates.length <= chunkSize) { - const result = await globalRankingPass(client, candidates, actualCount, model, maxTokens); - log.info(`Global ranked: ${result.map((h) => h.region).join(' > ')}`); - return result; + const pass = await globalRankingPass(client, candidates, actualCount, model, maxTokens); + log.info(`Global ranked: ${pass.headlines.map((h) => h.region).join(' > ')}`); + return { headlines: pass.headlines, usage: pass.usage }; } const chunks: Candidate[][] = []; @@ -279,21 +303,23 @@ export async function rankGlobalHeadlines( `Global ranking round 1: ${candidates.length} candidates → ${chunks.length} chunks, keeping ${survivorsPerChunk} each`, ); - const survivorGroups = await Promise.all( + const survivorPasses = await Promise.all( chunks.map((ch) => globalRankingPass(client, ch, Math.min(survivorsPerChunk, ch.length), model, maxTokens), ), ); - const survivors = survivorGroups.flat(); + const survivors = survivorPasses.flatMap((p) => p.headlines); log.info(`Global ranking round 2: ${survivors.length} survivors → ${actualCount} final`); - const result = await globalRankingPass(client, survivors, actualCount, model, maxTokens); - log.info(`Global ranked: ${result.map((h) => h.region).join(' > ')}`); - return result; + const finalPass = await globalRankingPass(client, survivors, actualCount, model, maxTokens); + log.info(`Global ranked: ${finalPass.headlines.map((h) => h.region).join(' > ')}`); + + const usage = sumUsages([...survivorPasses.map((p) => p.usage), finalPass.usage]); + return { headlines: finalPass.headlines, usage }; } catch (err) { log.warn( `Global ranking failed (${err instanceof Error ? err.message : String(err)}) — skipping global section`, ); - return []; + return { headlines: [], usage: null }; } } diff --git a/cron/src/runReport.ts b/cron/src/runReport.ts new file mode 100644 index 0000000..f73e444 --- /dev/null +++ b/cron/src/runReport.ts @@ -0,0 +1,163 @@ +import type { PulseConfig } from '@shared/config'; +import type { DigestUsage, RegionDigest, DigestQuality } from './types'; +import type { RegionConfig } from './regions'; +import type { RunConfig } from './qualityLog'; +import { buildRunConfig } from './pipeline'; +import { buildClient } from './notify'; +import { formatError, getLogger } from './logging'; + +export interface UsageTotals { + promptTokens: number; + completionTokens: number; + totalTokens: number; + costUsd: number; +} + +export interface NotifyOutcome { + devicesTargeted: number; + sent: number; + failed: number; +} + +/** + * One complete, versioned record of a daily-digest pipeline run — the durable, + * queryable companion to the human-readable log lines. Persisted to + * `public.pipeline_runs` (see supabase/schema.sql). Bump `schema` on any + * breaking shape change so downstream queries can branch on the version. + */ +export interface RunReport { + schema: 'pulse.run.v1'; + runAt: string; + status: 'ok' | 'partial'; + durationMs: number; + runConfig: RunConfig; + regions: { requested: string[]; succeeded: string[]; failed: string[] }; + cost: { + fetch: UsageTotals; + ranking: UsageTotals; + total: { totalTokens: number; costUsd: number }; + }; + headlines: { requested: number; fetched: number }; + notify: NotifyOutcome; + quality: DigestQuality[]; +} + +export interface BuildRunReportArgs { + config: PulseConfig; + resolvedRegions: RegionConfig[]; + digests: RegionDigest[]; + errors: Array<{ region: string; reason: unknown }>; + startTime: number; + /** Combined usage from the global-ranking step; null when it did not run. */ + globalRankingUsage: DigestUsage | null; + notify: NotifyOutcome; +} + +const ZERO: UsageTotals = { promptTokens: 0, completionTokens: 0, totalTokens: 0, costUsd: 0 }; + +/** Sum a list of usage records (skipping null/undefined) into a single total. */ +function sumUsage(usages: Array): UsageTotals { + return usages.reduce( + (acc, u) => + u + ? { + promptTokens: acc.promptTokens + u.promptTokens, + completionTokens: acc.completionTokens + u.completionTokens, + totalTokens: acc.totalTokens + u.totalTokens, + costUsd: acc.costUsd + u.costUsd, + } + : acc, + { ...ZERO }, + ); +} + +/** + * Assemble the complete run report. Pure — aggregates fetch cost, per-region + * ranking cost and the global-ranking cost, partitions regions, and folds in + * notify and headline counts. + */ +export function buildRunReport(args: BuildRunReportArgs): RunReport { + const { config, resolvedRegions, digests, errors, startTime, globalRankingUsage, notify } = args; + + const fetch = sumUsage(digests.map((d) => d.usage)); + const ranking = sumUsage([...digests.map((d) => d.rankingUsage), globalRankingUsage]); + + return { + schema: 'pulse.run.v1', + runAt: new Date().toISOString(), + status: errors.length > 0 ? 'partial' : 'ok', + durationMs: Date.now() - startTime, + runConfig: buildRunConfig(config), + regions: { + requested: resolvedRegions.map((r) => r.region), + succeeded: digests.map((d) => d.region), + failed: errors.map((e) => e.region), + }, + cost: { + fetch, + ranking, + total: { + totalTokens: fetch.totalTokens + ranking.totalTokens, + costUsd: fetch.costUsd + ranking.costUsd, + }, + }, + headlines: { + requested: resolvedRegions.length * config.api.fetch.count, + fetched: digests.reduce((sum, d) => sum + d.headlines.length, 0), + }, + notify, + quality: digests.flatMap((d) => (d.quality ? [d.quality] : [])), + }; +} + +/** Compact token count for log lines: 45231 → "45.2k". */ +function formatTokens(n: number): string { + return n >= 1000 ? `${(n / 1000).toFixed(1)}k` : String(n); +} + +/** + * One dense, human-readable summary line — meaningful at a glance, in the + * spirit of the existing usageSummary. The structured RunReport carries the + * queryable detail; this is the on-screen companion. + */ +export function formatRunReportSummary(report: RunReport): string { + const { regions, headlines, cost, notify, status, durationMs } = report; + return ( + `Run [${status}] — ${regions.succeeded.length}/${regions.requested.length} regions, ` + + `${headlines.fetched} headlines | ` + + `fetch $${cost.fetch.costUsd.toFixed(4)} (${formatTokens(cost.fetch.totalTokens)} tok) | ` + + `ranking $${cost.ranking.costUsd.toFixed(4)} (${formatTokens(cost.ranking.totalTokens)} tok) | ` + + `total $${cost.total.costUsd.toFixed(4)} | ` + + `notify ${notify.sent}/${notify.devicesTargeted} | ` + + `${(durationMs / 1000).toFixed(1)}s` + ); +} + +/** + * Persist one run report to `public.pipeline_runs` via the service-role client. + * Observability must never break the product pipeline, so any failure here is + * logged and swallowed — the digest run still counts as a success. + */ +export async function persistRunReport(report: RunReport): Promise { + const log = getLogger('run-report'); + try { + const db = buildClient(); + const { error } = await db.from('pipeline_runs').insert({ + run_at: report.runAt, + status: report.status, + total_cost_usd: report.cost.total.costUsd, + total_tokens: report.cost.total.totalTokens, + regions_succeeded: report.regions.succeeded.length, + regions_failed: report.regions.failed.length, + duration_ms: report.durationMs, + report, + }); + if (error) { + log.warn(`Failed to persist run report: ${error.message}`); + return; + } + log.info('Run report persisted to pipeline_runs'); + } catch (err) { + log.warn(`Failed to persist run report: ${formatError(err)}`); + } +} diff --git a/cron/src/tests/rankHeadlines.test.ts b/cron/src/tests/rankHeadlines.test.ts index 78cd08c..8d64db7 100644 --- a/cron/src/tests/rankHeadlines.test.ts +++ b/cron/src/tests/rankHeadlines.test.ts @@ -208,7 +208,8 @@ describe('rankGlobalHeadlines', () => { const digests = makeDigests(['Hungary', 'United States'], 3); const result = await rankGlobalHeadlines(digests, config); - expect(result).toEqual([]); + expect(result.headlines).toEqual([]); + expect(result.usage).toBeNull(); expect(mockCreate).not.toHaveBeenCalled(); }); @@ -217,7 +218,8 @@ describe('rankGlobalHeadlines', () => { const digests = makeDigests(['Hungary'], 0); const result = await rankGlobalHeadlines(digests, config); - expect(result).toEqual([]); + expect(result.headlines).toEqual([]); + expect(result.usage).toBeNull(); expect(mockCreate).not.toHaveBeenCalled(); }); @@ -228,7 +230,8 @@ describe('rankGlobalHeadlines', () => { const digests = makeDigests(['Hungary', 'United States'], 2); const result = await rankGlobalHeadlines(digests, config); - expect(result).toEqual([]); + expect(result.headlines).toEqual([]); + expect(result.usage).toBeNull(); }); it('selects globally important headlines in a single pass when candidates ≤ chunkSize', async () => { @@ -240,12 +243,19 @@ describe('rankGlobalHeadlines', () => { mockCreate.mockResolvedValueOnce(makeGlobalResponse([1, 3])); const result = await rankGlobalHeadlines(digests, config); - expect(result).toHaveLength(2); - expect(result[0]!.region).toBe('Hungary'); // candidate 1 is Hungary headline 1 - expect(result[1]!.region).toBe('United States'); // candidate 3 is US headline 1 + expect(result.headlines).toHaveLength(2); + expect(result.headlines[0]!.region).toBe('Hungary'); // candidate 1 is Hungary headline 1 + expect(result.headlines[1]!.region).toBe('United States'); // candidate 3 is US headline 1 // imageUrl must thread through to the global headline so it persists in the global digest. - expect(result[0]!.imageUrl).toBe('https://img.example.com/Hungary/1.jpg'); - expect(result[1]!.imageUrl).toBe('https://img.example.com/United States/1.jpg'); + expect(result.headlines[0]!.imageUrl).toBe('https://img.example.com/Hungary/1.jpg'); + expect(result.headlines[1]!.imageUrl).toBe('https://img.example.com/United States/1.jpg'); + // usage from the single pass is surfaced for cost reporting. + expect(result.usage).toEqual({ + promptTokens: 200, + completionTokens: 30, + totalTokens: 230, + costUsd: expect.any(Number), + }); }); it('runs two rounds when candidates exceed chunkSize', async () => { @@ -263,8 +273,15 @@ describe('rankGlobalHeadlines', () => { mockCreate.mockResolvedValueOnce(makeGlobalResponse([1, 2])); const result = await rankGlobalHeadlines(digests, config); - expect(result).toHaveLength(2); + expect(result.headlines).toHaveLength(2); expect(mockCreate).toHaveBeenCalledTimes(3); // 2 chunk passes + 1 final pass + // usage is summed across all three passes (200+30 tokens each). + expect(result.usage).toEqual({ + promptTokens: 600, + completionTokens: 90, + totalTokens: 690, + costUsd: expect.any(Number), + }); }); it('returns empty array and does not throw when the SDK throws', async () => { @@ -273,6 +290,7 @@ describe('rankGlobalHeadlines', () => { mockCreate.mockRejectedValueOnce(new Error('rate limit')); const result = await rankGlobalHeadlines(digests, config); - expect(result).toEqual([]); + expect(result.headlines).toEqual([]); + expect(result.usage).toBeNull(); }); }); diff --git a/cron/src/tests/runReport.test.ts b/cron/src/tests/runReport.test.ts new file mode 100644 index 0000000..d159431 --- /dev/null +++ b/cron/src/tests/runReport.test.ts @@ -0,0 +1,209 @@ +import { defaultConfig } from '../config'; +import type { RegionDigest, DigestUsage, DigestQuality } from '../types'; +import type { RegionConfig } from '../regions'; +import { + buildRunReport, + formatRunReportSummary, + persistRunReport, + type BuildRunReportArgs, +} from '../runReport'; + +// buildClient is mocked so persistRunReport never touches firebase-admin / Supabase. +const mockInsert = jest.fn(); +jest.mock('../notify', () => ({ + buildClient: () => ({ from: () => ({ insert: mockInsert }) }), +})); + +jest.mock('../logging', () => ({ + getLogger: () => ({ info: jest.fn(), warn: jest.fn(), debug: jest.fn(), error: jest.fn() }), + formatError: (e: unknown) => String(e), +})); + +// ── Fixtures ────────────────────────────────────────────────────────────────── + +const fetchUsage: DigestUsage = { + promptTokens: 100, + completionTokens: 50, + totalTokens: 150, + costUsd: 0.001, +}; +const rankUsage: DigestUsage = { + promptTokens: 20, + completionTokens: 10, + totalTokens: 30, + costUsd: 0.0002, +}; +const globalUsage: DigestUsage = { + promptTokens: 200, + completionTokens: 30, + totalTokens: 230, + costUsd: 0.005, +}; + +const regions: RegionConfig[] = [ + { region: 'Hungary', country: 'HU', sources: [] }, + { region: 'United States', country: 'US', sources: [] }, +]; + +function makeDigest(overrides: Partial = {}): RegionDigest { + return { + region: 'Hungary', + headlines: [], + attempts: 1, + usage: fetchUsage, + rankingUsage: rankUsage, + ...overrides, + }; +} + +function makeArgs(overrides: Partial = {}): BuildRunReportArgs { + return { + config: defaultConfig, + resolvedRegions: regions, + digests: [ + makeDigest({ region: 'Hungary', headlines: [{ title: 'a', summary: '', url: '' }] }), + makeDigest({ + region: 'United States', + headlines: [ + { title: 'b', summary: '', url: '' }, + { title: 'c', summary: '', url: '' }, + ], + }), + ], + errors: [], + startTime: Date.now() - 1000, + globalRankingUsage: globalUsage, + notify: { devicesTargeted: 10, sent: 8, failed: 2 }, + ...overrides, + }; +} + +// ── buildRunReport ────────────────────────────────────────────────────────── + +describe('buildRunReport', () => { + it('aggregates fetch cost across digests', () => { + const report = buildRunReport(makeArgs()); + expect(report.cost.fetch.totalTokens).toBe(300); // 150 × 2 + expect(report.cost.fetch.costUsd).toBeCloseTo(0.002); + }); + + it('aggregates ranking cost from per-region ranking AND the global pass', () => { + const report = buildRunReport(makeArgs()); + // per-region: 30 × 2 = 60 tokens, plus global 230 = 290 + expect(report.cost.ranking.totalTokens).toBe(290); + expect(report.cost.ranking.costUsd).toBeCloseTo(0.0002 * 2 + 0.005); + }); + + it('treats a null global-ranking usage as zero', () => { + const report = buildRunReport(makeArgs({ globalRankingUsage: null })); + expect(report.cost.ranking.totalTokens).toBe(60); // only the two per-region passes + }); + + it('computes total as fetch + ranking', () => { + const report = buildRunReport(makeArgs()); + expect(report.cost.total.totalTokens).toBe(590); + expect(report.cost.total.costUsd).toBeCloseTo(0.0074); + }); + + it('reports status ok when there are no region errors', () => { + expect(buildRunReport(makeArgs()).status).toBe('ok'); + }); + + it('reports status partial when some regions failed', () => { + const report = buildRunReport( + makeArgs({ errors: [{ region: 'France', reason: new Error('boom') }] }), + ); + expect(report.status).toBe('partial'); + expect(report.regions.failed).toEqual(['France']); + }); + + it('partitions regions into requested / succeeded', () => { + const report = buildRunReport(makeArgs()); + expect(report.regions.requested).toEqual(['Hungary', 'United States']); + expect(report.regions.succeeded).toEqual(['Hungary', 'United States']); + }); + + it('counts headlines requested and fetched', () => { + const report = buildRunReport(makeArgs()); + expect(report.headlines.requested).toBe(regions.length * defaultConfig.api.fetch.count); + expect(report.headlines.fetched).toBe(3); // 1 + 2 + }); + + it('passes through the notify outcome', () => { + const report = buildRunReport(makeArgs()); + expect(report.notify).toEqual({ devicesTargeted: 10, sent: 8, failed: 2 }); + }); + + it('includes only digests that carry quality signals', () => { + const quality = { region: 'Hungary' } as DigestQuality; + const report = buildRunReport( + makeArgs({ + digests: [makeDigest({ quality }), makeDigest({ region: 'United States' })], + }), + ); + expect(report.quality).toHaveLength(1); + }); + + it('stamps schema version and an ISO runAt', () => { + const report = buildRunReport(makeArgs()); + expect(report.schema).toBe('pulse.run.v1'); + expect(report.runAt).toMatch(/^\d{4}-\d{2}-\d{2}T/); + expect(report.durationMs).toBeGreaterThanOrEqual(0); + }); +}); + +// ── formatRunReportSummary ──────────────────────────────────────────────────── + +describe('formatRunReportSummary', () => { + it('renders the key figures in one line', () => { + const summary = formatRunReportSummary(buildRunReport(makeArgs())); + expect(summary).toMatch(/^Run \[ok\] — 2\/2 regions, 3 headlines/); + expect(summary).toContain('fetch $0.0020 (300 tok)'); + expect(summary).toContain('ranking $0.0054 (290 tok)'); + expect(summary).toContain('total $0.0074'); + expect(summary).toContain('notify 8/10'); + }); + + it('abbreviates large token counts with k', () => { + const big: DigestUsage = { ...fetchUsage, totalTokens: 45231 }; + const report = buildRunReport(makeArgs({ digests: [makeDigest({ usage: big })] })); + expect(formatRunReportSummary(report)).toContain('(45.2k tok)'); + }); +}); + +// ── persistRunReport ────────────────────────────────────────────────────────── + +describe('persistRunReport', () => { + beforeEach(() => mockInsert.mockReset()); + + it('inserts a row with promoted scalar columns plus the full report', async () => { + mockInsert.mockResolvedValueOnce({ error: null }); + const report = buildRunReport(makeArgs()); + + await persistRunReport(report); + + expect(mockInsert).toHaveBeenCalledTimes(1); + expect(mockInsert).toHaveBeenCalledWith( + expect.objectContaining({ + run_at: report.runAt, + status: 'ok', + total_cost_usd: report.cost.total.costUsd, + total_tokens: report.cost.total.totalTokens, + regions_succeeded: 2, + regions_failed: 0, + duration_ms: report.durationMs, + report, + }), + ); + }); + + it('swallows a Supabase insert error (never throws)', async () => { + mockInsert.mockResolvedValueOnce({ error: { message: 'permission denied' } }); + await expect(persistRunReport(buildRunReport(makeArgs()))).resolves.toBeUndefined(); + }); + + it('swallows a thrown client error (never throws)', async () => { + mockInsert.mockRejectedValueOnce(new Error('network down')); + await expect(persistRunReport(buildRunReport(makeArgs()))).resolves.toBeUndefined(); + }); +}); diff --git a/docs/superpowers/plans/2026-06-12-usage-tracking.md b/docs/superpowers/plans/2026-06-12-usage-tracking.md deleted file mode 100644 index 8a2e3af..0000000 --- a/docs/superpowers/plans/2026-06-12-usage-tracking.md +++ /dev/null @@ -1,206 +0,0 @@ -# Usage Tracking — Handover - -**Date:** 2026-06-12 -**Status:** PR #45 closed (wrong architecture). Vercel server setup required before reimplementation. -**Branch to delete:** `feat/usage-tracking` (carry-over from closed PR; do not merge) - ---- - -## What PR #45 Did (Do Not Repeat) - -PR #45 tracked events by writing directly from the phone to Supabase using the publishable key. -It was closed because: - -- No server-side rate limiting or validation — any client could write arbitrary events -- Every article tap fired a network call — excessive, battery-draining, noisy data -- Publishable key in the DB write path is the wrong trust boundary for user analytics - -### Code that was written in PR #45 (reference only) - -| File | What it did | -| -------------------------------------- | ---------------------------------------------------------------------------------------- | -| `app/src/analytics/track.ts` | Fire-and-forget `trackEvent()` calling `supabase.from('usage_events').insert()` directly | -| `app/src/screens/ArticleScreen.tsx` | `trackEvent('article_open')` on mount; `trackEvent('article_read')` on openArticle tap | -| `app/src/components/DigestPage.tsx` | `trackEvent('digest_viewed')` when digest loads and page is active | -| `cron/src/notify.ts` | Eviction of `usage_events` older than `db.evictUsageDays` days alongside digest eviction | -| `shared/src/config.ts` | Added `evictUsageDays: number` to `DbConfig` | -| `shared/pulse.config.json` | `"evictUsageDays": 90` in the `db` section | -| `cron/src/config.ts` | Default `evictUsageDays: 90`; validation rule `>= 1` | -| `supabase/schema.sql` | `usage_events` table + index + RLS (see schema section below) | -| `cron/src/tests/notify.test.ts` | Added `evictUsageDays: 90` to `makeConfig()` | -| `cron/src/tests/rankHeadlines.test.ts` | Added `evictUsageDays: 90` to `makeConfig()` | - -### Schema from PR #45 (keep this — it is still correct) - -```sql -CREATE TABLE IF NOT EXISTS usage_events ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - user_id UUID NOT NULL REFERENCES auth.users(id) ON DELETE CASCADE, - event_type TEXT NOT NULL, - metadata JSONB NOT NULL DEFAULT '{}', - occurred_at TIMESTAMPTZ NOT NULL DEFAULT now() -); - -CREATE INDEX IF NOT EXISTS usage_events_user_occurred - ON usage_events (user_id, occurred_at DESC); - -ALTER TABLE usage_events ENABLE ROW LEVEL SECURITY; - --- Service-role key (cron) bypasses RLS; app will write via server endpoint --- so no INSERT policy needed for the publishable key. -CREATE POLICY "users read own events" - ON usage_events FOR SELECT USING (auth.uid() = user_id); -``` - -> Note: The INSERT policy from PR #45 (`WITH CHECK (auth.uid() = user_id)`) should **not** be -> recreated. The app will POST to a Vercel endpoint; the endpoint writes with the service-role key. -> No INSERT policy for the publishable key is needed. - ---- - -## Correct Architecture - -``` -App session - └─ in-memory buffer + MMKV persistence (@pulse/analytics_queue) - └─ flush trigger (background / launch / logout / 50-event cap) - └─ POST /api/events (Vercel, JWT-verified) - └─ batch INSERT → usage_events (service-role key) -``` - ---- - -## Prerequisites (Manual Steps Before Coding) - -1. **Deploy to Vercel** - - Connect the `GJanos/pulse-news` GitHub repo in the Vercel dashboard - - Set env vars: `SUPABASE_URL`, `SUPABASE_SECRET_KEY`, `FIREBASE_PROJECT_ID`, - `FIREBASE_CLIENT_EMAIL`, `FIREBASE_PRIVATE_KEY` - - Verify `/api/account` is reachable at `https://.vercel.app/api/account` - - Vercel auto-deploys on push to `develop` - -2. **Apply the DB migration** (schema above, minus the INSERT policy) - ---- - -## Implementation Plan - -### Phase A — Server endpoint: `cron/api/events.ts` - -Pattern: identical to `cron/api/account.ts` (JWT verification via `db.auth.getUser(token)`). - -``` -POST /api/events -Authorization: Bearer -Content-Type: application/json - -{ - "events": [ - { "type": "article_read", "metadata": { "url": "...", "region": "GB" }, "occurred_at": "..." }, - { "type": "digest_viewed", "metadata": { "dates": ["2026-06-12"] }, "occurred_at": "..." } - ] -} -``` - -Server responsibilities: - -- Verify JWT → resolve `user_id` -- Validate each event `type` against allowlist: `['article_open', 'article_read', 'digest_viewed']` -- Strip any unexpected `metadata` keys (whitelist: `url`, `region`, `dates`, `count`) -- Batch-insert with service-role key -- Return `{ ok: true, saved: N }` -- Max payload: 100 events per request (guard against abuse) - -### Phase B — App buffer: `app/src/analytics/buffer.ts` - -```typescript -// Pseudocode — not production code -const QUEUE_KEY = '@pulse/analytics_queue'; -const MAX_BUFFER = 50; - -// trackEvent() — call from UI, no network -export function trackEvent(type, metadata) { - const event = { type, metadata, occurred_at: new Date().toISOString() }; - pushToBuffer(event); // in-memory array - persistBufferToMmkv(); // MMKV write for crash recovery - if (buffer.length >= MAX_BUFFER) void flushBuffer(); // safety cap -} - -// flushBuffer() — call on app background / launch / logout -export async function flushBuffer() { - const events = readAndClearMmkv(); - if (!events.length) return; - const token = getSession()?.access_token; - if (!token) return; // unauthenticated — drop (user has no events row) - await fetch(`${VERCEL_URL}/api/events`, { - method: 'POST', - headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' }, - body: JSON.stringify({ events }), - }); - // On failure: re-enqueue events back into MMKV (or accept loss after N retries) -} -``` - -### Phase C — Flush triggers - -| Trigger | How | -| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | -| App → background | `AppState.addEventListener('change', state => { if (state === 'background') flushBuffer() })` in `useAppInit` or a new `useAnalyticsFlush` hook | -| App launch | Call `flushBuffer()` early in `useAppInit`, before rendering, to recover crashed-session events | -| Logout | Call `flushBuffer()` inside `deleteAccount()` and `signOut()` before clearing storage | -| Buffer hits 50 | Handled inside `trackEvent()` — auto-flushes | - -### Phase D — Revised event schema (less frequent) - -Rather than one event per tap, collect during the session and flush once: - -| Event | When emitted into buffer | Metadata | -| --------------- | --------------------------------------------- | ------------------------------------------------------ | -| `article_open` | On ArticleScreen mount | `{ url, region }` — individual, low cardinality | -| `article_read` | On "Read full article" tap | `{ url, region }` — individual, intentional action | -| `digest_viewed` | On DigestPage becoming active + digest loaded | `{ date }` — deduplicated per date per session via ref | - -> `article_open` and `article_read` stay as individual events because they are low-frequency -> meaningful actions. `digest_viewed` fires at most once per date per session. -> None are high-frequency enough to be a problem when batched. - -### Phase E — Config + eviction (carry over from PR #45, already correct) - -- `db.evictUsageDays: 90` in `DbConfig` and `pulse.config.json` -- Cron evicts `usage_events` older than `evictUsageDays` inside `persistDigests` when `db.evict` is enabled -- These config changes from PR #45 must be re-applied on the new branch - ---- - -## File Checklist for New Branch - -``` -New files: - app/src/analytics/buffer.ts ← buffer + flushBuffer() - cron/api/events.ts ← POST /api/events handler - -Modified files: - shared/src/config.ts ← add evictUsageDays to DbConfig - shared/pulse.config.json ← "evictUsageDays": 90 - cron/src/config.ts ← default + validation - cron/src/notify.ts ← evict usage_events in persistDigests - cron/src/tests/notify.test.ts ← add evictUsageDays to makeConfig() - cron/src/tests/rankHeadlines.test.ts← add evictUsageDays to makeConfig() - supabase/schema.sql ← usage_events table (no INSERT RLS policy) - app/src/screens/ArticleScreen.tsx ← trackEvent calls (buffer, not direct) - app/src/components/DigestPage.tsx ← trackEvent('digest_viewed') (buffer) - app/src/hooks/useAppInit.ts ← flushBuffer() on launch + AppState listener - app/src/hooks/useSupabaseAuth.ts ← flushBuffer() before signOut/deleteAccount - todo.md ← mark Phase 4 done -``` - ---- - -## Open Questions Before Starting - -1. **Vercel project URL** — needs to be hardcoded or env-var'd in the app - (`EXPO_PUBLIC_API_URL` or baked into `app/src/config.ts`) -2. **Flush on failure** — re-enqueue failed events or silently drop after 2 retries? - Recommendation: drop after 2 retries (analytics loss is acceptable; retry loops are not) -3. **Unauthenticated sessions** — events from users who haven't signed in are dropped at flush time - (no `user_id` to attach). Acceptable for V1. diff --git a/docs/superpowers/plans/2026-06-15-devices-rls-lockdown.md b/docs/superpowers/plans/2026-06-15-devices-rls-lockdown.md deleted file mode 100644 index 9858413..0000000 --- a/docs/superpowers/plans/2026-06-15-devices-rls-lockdown.md +++ /dev/null @@ -1,580 +0,0 @@ -# Devices Table RLS Lockdown Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Replace the three open `USING(true)` RLS policies on `devices` with a deny-all table whose only client write surface is `SECURITY DEFINER` RPCs, and delete the now-dead `/api/account` endpoint. - -**Architecture:** Keep RLS enabled on `devices` but drop every policy, so direct anon/authenticated access is denied. Add two definer RPCs — `update_notify_time` and `link_device_to_user` (which derives `user_id` from `auth.uid()`) — alongside the existing `register_device`. The React Native app switches its two direct table UPDATEs to these RPCs. - -**Tech Stack:** Postgres/Supabase (RLS + plpgsql RPCs), TypeScript, React Native (Expo), Jest + ts-jest. Branch: `feat/devices-rls-lockdown` (already created). - -**Spec:** `docs/superpowers/specs/2026-06-15-devices-rls-lockdown-design.md` - ---- - -## File Structure - -| File | Responsibility | Change | -| --------------------------------------------- | --------------------------- | ---------------------------------------------------------------------- | -| `supabase/schema.sql` | Schema source-of-truth | Drop 3 policies; add `update_notify_time` + `link_device_to_user` RPCs | -| (live Supabase project) | Running database | Apply the same migration | -| `app/src/notifications/devices.ts` | App device write helpers | `updateNotifyTime` + `linkDeviceToUser` → RPC calls | -| `app/src/hooks/useDeviceRegistration.ts` | Registration lifecycle hook | Update `linkDeviceToUser` call site (drop `userId` arg) | -| `app/src/tests/notifications/devices.test.ts` | App write-helper tests | Rewrite mocks to RPC-based | -| `cron/api/account.ts` | Dead serverless endpoint | Delete | -| `cron/src/tests/account.test.ts` | Its test | Delete | - ---- - -## Task 1: Database — definer RPCs + drop open policies - -**Files:** - -- Modify: `supabase/schema.sql:29-39` (replace the 3 policies) and after `:74` (add 2 RPCs) -- Apply: live Supabase project (via `mcp__supabase.apply_migration`) - -- [ ] **Step 1: Replace the open policies in `supabase/schema.sql`** - -Replace lines 29–39 (the three `CREATE POLICY` blocks and their comment, **keep** line 27 `ALTER TABLE devices ENABLE ROW LEVEL SECURITY;`) with: - -```sql --- All client writes go through SECURITY DEFINER RPCs (register_device, --- update_notify_time, link_device_to_user); the cron reads via the service-role --- key, which bypasses RLS. No open policy is defined, so RLS denies every direct --- anon/authenticated SELECT/INSERT/UPDATE — closing publishable-key token --- harvesting and blind tampering. DROP IF EXISTS keeps re-applying this file --- idempotent against a database that still carries the old open policies. -DROP POLICY IF EXISTS "device self-select" ON devices; -DROP POLICY IF EXISTS "device self-register" ON devices; -DROP POLICY IF EXISTS "device self-update" ON devices; -``` - -- [ ] **Step 2: Add the two RPCs after the `register_device` grant** - -Immediately after `supabase/schema.sql:74` (`GRANT EXECUTE ON FUNCTION register_device(uuid, text, uuid) TO anon, authenticated, service_role;`), insert: - -```sql - --- update_notify_time — the only path for the app to change a device's notify_at. --- Definer so it works with the publishable key against the now-policyless table. --- Keyed by the per-install device UUID (an unguessable capability). Returns FOUND --- so the app can warn when the row does not exist yet (registration not finished). -CREATE OR REPLACE FUNCTION update_notify_time(p_id uuid, p_notify_at time) -RETURNS boolean -LANGUAGE plpgsql -SECURITY DEFINER -SET search_path = public -AS $$ -BEGIN - UPDATE devices - SET notify_at = p_notify_at, updated_at = now() - WHERE id = p_id; - RETURN FOUND; -END; -$$; - -REVOKE EXECUTE ON FUNCTION update_notify_time(uuid, time) FROM PUBLIC; -GRANT EXECUTE ON FUNCTION update_notify_time(uuid, time) TO anon, authenticated, service_role; - --- link_device_to_user — stamps the auth link onto a device row. user_id is NOT a --- parameter: it is derived from the caller's JWT via auth.uid(), so a device can --- only ever be linked to the authenticated caller's own identity. Granted to --- authenticated only (the app calls this exclusively post-login). Returns FOUND so --- the app's retry can wait for the row to appear. -CREATE OR REPLACE FUNCTION link_device_to_user(p_id uuid) -RETURNS boolean -LANGUAGE plpgsql -SECURITY DEFINER -SET search_path = public -AS $$ -DECLARE - v_uid uuid := auth.uid(); -BEGIN - IF v_uid IS NULL THEN - RAISE EXCEPTION 'link_device_to_user requires an authenticated user'; - END IF; - UPDATE devices - SET user_id = v_uid, updated_at = now() - WHERE id = p_id; - RETURN FOUND; -END; -$$; - -REVOKE EXECUTE ON FUNCTION link_device_to_user(uuid) FROM PUBLIC; -GRANT EXECUTE ON FUNCTION link_device_to_user(uuid) TO authenticated, service_role; -``` - -- [ ] **Step 3: Apply the migration to the live project** - -Use the Supabase MCP tool (project is the live Pulse project). Call `mcp__supabase.apply_migration` with: - -- `name`: `devices_rls_lockdown` -- `query`: the exact SQL from Step 1 (the three `DROP POLICY` lines) **followed by** the two `CREATE OR REPLACE FUNCTION` + `REVOKE`/`GRANT` blocks from Step 2. - -Expected: success, no error. - -- [ ] **Step 4: Verify the lockdown with role-scoped SQL** - -Run via `mcp__supabase.execute_sql`, one statement group at a time: - -```sql --- (a) anon cannot read the table (harvest closed) -SET ROLE anon; -SELECT count(*) AS visible_rows FROM devices; -RESET ROLE; -``` - -Expected: `visible_rows = 0` (RLS hides all rows from anon). - -```sql --- (b) anon CAN register + set notify_at via the RPCs -SET ROLE anon; -SELECT register_device('00000000-0000-4000-8000-0000000000c3', 'rls-test-token', NULL); -SELECT update_notify_time('00000000-0000-4000-8000-0000000000c3', '09:00') AS updated; -RESET ROLE; -``` - -Expected: `register_device` returns void (no error); `update_notify_time` returns `updated = true`. - -```sql --- (c) anon CANNOT call link (no auth.uid) and CANNOT even execute it (not granted) -SET ROLE anon; -SELECT link_device_to_user('00000000-0000-4000-8000-0000000000c3'); -RESET ROLE; -``` - -Expected: an error — either `permission denied for function link_device_to_user` (not granted to anon) or the `requires an authenticated user` exception. Either proves anon cannot link. - -```sql --- (d) cleanup the test row -DELETE FROM devices WHERE id = '00000000-0000-4000-8000-0000000000c3'; -``` - -Expected: 1 row deleted. - -- [ ] **Step 5: Commit** - -```bash -git add supabase/schema.sql -git commit -m "feat(db): lock down devices RLS behind definer RPCs" -``` - ---- - -## Task 2: App — `updateNotifyTime` via RPC - -**Files:** - -- Modify: `app/src/notifications/devices.ts:95-118` (`updateNotifyTime`) -- Test: `app/src/tests/notifications/devices.test.ts` (rewrite the mock harness + `updateNotifyTime` block; the `upsertDevice` block continues to pass) - -- [ ] **Step 1: Rewrite the test harness + `upsertDevice`/`updateNotifyTime` blocks** - -Replace the top of `app/src/tests/notifications/devices.test.ts` (lines 1–80, i.e. imports through the end of the `upsertDevice` describe) with the RPC-based harness below, then replace the `updateNotifyTime` describe (lines 153–204) with the new block in Step-2-of-this-file. Full new harness + `upsertDevice`: - -```ts -import { storage } from '../../storage/mmkv'; -import { getSupabase } from '../../supabase/client'; -import { TOKEN_KEY } from '../../notifications/keys'; -import { upsertDevice, linkDeviceToUser, updateNotifyTime } from '../../notifications/devices'; - -jest.mock('../../supabase/client', () => ({ getSupabase: jest.fn() })); - -const mockGetSupabase = getSupabase as jest.MockedFunction; - -interface FakeClient { - rpc: jest.Mock; - from: jest.Mock; -} - -// Every write now flows through client.rpc(name, args). `from` exists only so -// tests can assert it is NEVER called (no direct table access remains). -function makeClient(): FakeClient { - return { - rpc: jest.fn().mockResolvedValue({ data: true, error: null }), - from: jest.fn(), - }; -} - -beforeEach(() => { - jest.clearAllMocks(); - storage.clearAll(); -}); - -describe('upsertDevice', () => { - it('registers via the register_device RPC with p_id + p_token', async () => { - const client = makeClient(); - mockGetSupabase.mockReturnValue(client as never); - await upsertDevice({ deviceId: 'dev-1', fcmToken: 'tok-1' }); - expect(client.rpc).toHaveBeenCalledWith('register_device', { - p_id: 'dev-1', - p_token: 'tok-1', - }); - expect(client.from).not.toHaveBeenCalled(); - }); - - it('omits p_user_id — login links the user separately', async () => { - const client = makeClient(); - mockGetSupabase.mockReturnValue(client as never); - await upsertDevice({ deviceId: 'dev-1', fcmToken: 'tok-1' }); - const args = client.rpc.mock.calls[0]![1] as Record; - expect('p_user_id' in args).toBe(false); - }); - - it('never writes notify_at — registration does not own it', async () => { - const client = makeClient(); - mockGetSupabase.mockReturnValue(client as never); - await upsertDevice({ deviceId: 'dev-1', fcmToken: 'tok-1' }); - const args = client.rpc.mock.calls[0]![1] as Record; - expect('notify_at' in args).toBe(false); - expect('p_notify_at' in args).toBe(false); - }); - - it('returns false when Supabase is unconfigured', async () => { - mockGetSupabase.mockReturnValue(null); - await expect(upsertDevice({ deviceId: 'dev-1', fcmToken: 'tok-1' })).resolves.toBe(false); - }); - - it('returns false and does not throw when the RPC returns an error', async () => { - const client = makeClient(); - client.rpc.mockResolvedValue({ data: null, error: { message: 'boom' } }); - mockGetSupabase.mockReturnValue(client as never); - await expect(upsertDevice({ deviceId: 'dev-1', fcmToken: 'tok-1' })).resolves.toBe(false); - }); -}); - -describe('updateNotifyTime', () => { - it('skips when no FCM token is cached (device not yet registered)', async () => { - const client = makeClient(); - mockGetSupabase.mockReturnValue(client as never); - await updateNotifyTime('dev-1', '09:00'); - expect(client.rpc).not.toHaveBeenCalled(); - }); - - it('calls update_notify_time with p_id + p_notify_at and never touches the table', async () => { - storage.set(TOKEN_KEY, 'tok-cached'); - const client = makeClient(); - mockGetSupabase.mockReturnValue(client as never); - await updateNotifyTime('dev-1', '09:00'); - expect(client.rpc).toHaveBeenCalledWith('update_notify_time', { - p_id: 'dev-1', - p_notify_at: '09:00', - }); - expect(client.from).not.toHaveBeenCalled(); - }); - - it('forwards a null notify_at (clearing the time)', async () => { - storage.set(TOKEN_KEY, 'tok-cached'); - const client = makeClient(); - mockGetSupabase.mockReturnValue(client as never); - await updateNotifyTime('dev-1', null); - const args = client.rpc.mock.calls[0]![1] as Record; - expect('p_notify_at' in args).toBe(true); - expect(args.p_notify_at).toBeNull(); - }); - - it('resolves when the RPC reports no row updated (data false)', async () => { - storage.set(TOKEN_KEY, 'tok-cached'); - const client = makeClient(); - client.rpc.mockResolvedValue({ data: false, error: null }); - mockGetSupabase.mockReturnValue(client as never); - await expect(updateNotifyTime('dev-1', '09:00')).resolves.toBeUndefined(); - }); - - it('resolves without throwing when the RPC errors', async () => { - storage.set(TOKEN_KEY, 'tok-cached'); - const client = makeClient(); - client.rpc.mockResolvedValue({ data: null, error: { message: 'boom' } }); - mockGetSupabase.mockReturnValue(client as never); - await expect(updateNotifyTime('dev-1', '09:00')).resolves.toBeUndefined(); - }); -}); -``` - -> Note: the `linkDeviceToUser` describe block (old lines 82–151) is replaced in Task 3. After this step that block still references the old two-arg signature and will fail to compile — that is expected; Task 3 fixes it. To keep this task independently green, **also** apply Task 3's test edit before running. If executing strictly one task at a time, run the targeted command below which only runs the rewritten blocks. - -- [ ] **Step 2: Run the updateNotifyTime tests to verify they fail** - -Run: `cd app && npx jest src/tests/notifications/devices.test.ts -t "updateNotifyTime"` -Expected: FAIL — `updateNotifyTime` still calls `.from('devices')`, so `client.rpc` was not called / `client.from` was. - -- [ ] **Step 3: Rewrite `updateNotifyTime` in `app/src/notifications/devices.ts`** - -Replace the body of `updateNotifyTime` (lines 95–118) with: - -```ts -export async function updateNotifyTime(deviceId: string, notifyAt: string | null): Promise { - const cachedToken = storage.getString(TOKEN_KEY) ?? null; - if (!cachedToken) { - log.debug('updateNotifyTime: no cached token — device not yet registered, skipping'); - return; - } - const supabase = getSupabase(); - if (!supabase) return; - log.info(`updating notify_at → ${notifyAt ?? 'null'} for device ${deviceId.slice(0, 8)}…`); - const { data, error } = await supabase.rpc('update_notify_time', { - p_id: deviceId, - p_notify_at: notifyAt, - }); - if (error) { - log.warn(`updateNotifyTime failed: ${error.message}`); - } else if (data !== true) { - log.warn( - `updateNotifyTime: no device row for ${deviceId.slice(0, 8)}… — notify_at not saved (device may not have registered yet)`, - ); - } else { - log.debug(`notify_at updated for device ${deviceId.slice(0, 8)}…`); - } -} -``` - -- [ ] **Step 4: Run the updateNotifyTime tests to verify they pass** - -Run: `cd app && npx jest src/tests/notifications/devices.test.ts -t "updateNotifyTime"` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add app/src/notifications/devices.ts app/src/tests/notifications/devices.test.ts -git commit -m "feat(app): updateNotifyTime writes via update_notify_time RPC" -``` - ---- - -## Task 3: App — `linkDeviceToUser` via RPC + call site - -**Files:** - -- Modify: `app/src/notifications/devices.ts:49-81` (`linkDeviceToUser`) -- Modify: `app/src/hooks/useDeviceRegistration.ts:97` (call site) -- Test: `app/src/tests/notifications/devices.test.ts` (replace the `linkDeviceToUser` describe block) - -- [ ] **Step 1: Replace the `linkDeviceToUser` describe block in the test file** - -Replace the old `linkDeviceToUser` describe (lines 82–151) with: - -```ts -describe('linkDeviceToUser', () => { - it('links via link_device_to_user with p_id and never touches the table', async () => { - const client = makeClient(); - mockGetSupabase.mockReturnValue(client as never); - await linkDeviceToUser('dev-1'); - expect(client.rpc).toHaveBeenCalledWith('link_device_to_user', { p_id: 'dev-1' }); - expect(client.from).not.toHaveBeenCalled(); - }); - - it('does not pass a user_id — it is derived server-side from the JWT', async () => { - const client = makeClient(); - mockGetSupabase.mockReturnValue(client as never); - await linkDeviceToUser('dev-1'); - const args = client.rpc.mock.calls[0]![1] as Record; - expect('user_id' in args).toBe(false); - expect('p_user_id' in args).toBe(false); - }); - - it('no-ops when Supabase is unconfigured', async () => { - mockGetSupabase.mockReturnValue(null); - await expect(linkDeviceToUser('dev-1')).resolves.toBeUndefined(); - }); - - it('does not retry on a Supabase error — exits immediately', async () => { - const client = makeClient(); - client.rpc.mockResolvedValue({ data: null, error: { message: 'denied' } }); - mockGetSupabase.mockReturnValue(client as never); - await linkDeviceToUser('dev-1'); - expect(client.rpc).toHaveBeenCalledTimes(1); - }); - - it('succeeds on the first attempt when data is true', async () => { - const client = makeClient(); - mockGetSupabase.mockReturnValue(client as never); - await linkDeviceToUser('dev-1'); - expect(client.rpc).toHaveBeenCalledTimes(1); - }); - - it('retries on a false (0-row) result, succeeding on the last attempt', async () => { - jest.useFakeTimers(); - const client = makeClient(); - client.rpc - .mockResolvedValueOnce({ data: false, error: null }) - .mockResolvedValueOnce({ data: false, error: null }) - .mockResolvedValue({ data: true, error: null }); - mockGetSupabase.mockReturnValue(client as never); - const promise = linkDeviceToUser('dev-1'); - await jest.advanceTimersByTimeAsync(2000); - await jest.advanceTimersByTimeAsync(2000); - await promise; - expect(client.rpc).toHaveBeenCalledTimes(3); - jest.useRealTimers(); - }); - - it('stops after 3 attempts when the row never appears (always false)', async () => { - jest.useFakeTimers(); - const client = makeClient(); - client.rpc.mockResolvedValue({ data: false, error: null }); - mockGetSupabase.mockReturnValue(client as never); - const promise = linkDeviceToUser('dev-1'); - await jest.advanceTimersByTimeAsync(2000); - await jest.advanceTimersByTimeAsync(2000); - await promise; - expect(client.rpc).toHaveBeenCalledTimes(3); - jest.useRealTimers(); - }); -}); -``` - -- [ ] **Step 2: Run the linkDeviceToUser tests to verify they fail** - -Run: `cd app && npx jest src/tests/notifications/devices.test.ts -t "linkDeviceToUser"` -Expected: FAIL — current `linkDeviceToUser` takes two args and calls `.from('devices')`, so `client.rpc` is not called with `link_device_to_user`. - -- [ ] **Step 3: Rewrite `linkDeviceToUser` in `app/src/notifications/devices.ts`** - -Replace the body of `linkDeviceToUser` (lines 49–81) with: - -```ts -/** - * Associate this device with the signed-in user via the link_device_to_user RPC. - * The RPC derives user_id from auth.uid() (the caller's JWT), so no user id is - * passed and a device can only be linked to the caller's own identity. Retries up - * to 3 times on a false result (the device row may not exist yet if registration - * is still in-flight). Gives up immediately on a DB error. No-op when unconfigured. - */ -export async function linkDeviceToUser(deviceId: string): Promise { - const supabase = getSupabase(); - if (!supabase) return; - - const MAX_ATTEMPTS = 3; - const RETRY_DELAY_MS = 2000; - - for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) { - const { data, error } = await supabase.rpc('link_device_to_user', { p_id: deviceId }); - if (error) { - log.warn(`linkDeviceToUser failed: ${error.message}`); - return; - } - if (data === true) { - log.info(`device ${deviceId.slice(0, 8)}… linked`); - return; - } - if (attempt < MAX_ATTEMPTS) { - log.debug( - `linkDeviceToUser: 0-row (attempt ${attempt}/${MAX_ATTEMPTS}) — device row not yet present, retrying in ${RETRY_DELAY_MS}ms`, - ); - await new Promise((resolve) => setTimeout(resolve, RETRY_DELAY_MS)); - } - } - log.warn( - `linkDeviceToUser: no device row for ${deviceId.slice(0, 8)}… after ${MAX_ATTEMPTS} attempts — not linked (device may not have registered yet)`, - ); -} -``` - -- [ ] **Step 4: Update the call site in `app/src/hooks/useDeviceRegistration.ts:97`** - -Change: - -```ts - linkDeviceToUser(deviceId, userId).catch((e: unknown) => -``` - -to: - -```ts - linkDeviceToUser(deviceId).catch((e: unknown) => -``` - -The `if (!userId || !deviceId) return;` gate on line 96 stays unchanged — `userId` still gates _whether_ to link (a session must exist), it is just no longer passed. - -- [ ] **Step 5: Run the full devices test file to verify it passes** - -Run: `cd app && npx jest src/tests/notifications/devices.test.ts` -Expected: PASS — all `upsertDevice`, `updateNotifyTime`, and `linkDeviceToUser` cases green. - -- [ ] **Step 6: Commit** - -```bash -git add app/src/notifications/devices.ts app/src/hooks/useDeviceRegistration.ts app/src/tests/notifications/devices.test.ts -git commit -m "feat(app): linkDeviceToUser uses link_device_to_user RPC (JWT-derived user)" -``` - ---- - -## Task 4: Delete the dead `/api/account` endpoint - -**Files:** - -- Delete: `cron/api/account.ts` -- Delete: `cron/src/tests/account.test.ts` - -- [ ] **Step 1: Confirm nothing references the endpoint** - -Run (Grep tool, or): `cd cron && npx tsc --noEmit` after deletion in Step 2. First, sanity-check there is no import of the route: - -Run: `git grep -n "api/account" -- ':!docs'` -Expected: no matches in `app/` or `cron/src` source (only this plan / spec docs, which are excluded). If any source match appears, STOP and reassess. - -- [ ] **Step 2: Delete both files** - -```bash -git rm cron/api/account.ts cron/src/tests/account.test.ts -``` - -- [ ] **Step 3: Verify cron still typechecks and tests pass** - -Run: `cd cron && npx tsc --noEmit && npm test` -Expected: typecheck clean; Jest green with the `account` suite gone. - -- [ ] **Step 4: Commit** - -```bash -git commit -m "chore(cron): remove dead /api/account endpoint (writes go through RPCs)" -``` - ---- - -## Task 5: Full verification - -**Files:** none (verification only) - -- [ ] **Step 1: Typecheck both packages** - -Run: `cd cron && npx tsc --noEmit` -Expected: no errors. - -Run: `cd app && npx tsc --noEmit` -Expected: no errors. - -- [ ] **Step 2: Lint both packages** - -Run: `cd cron && npx eslint --ext .ts src` -Expected: no errors. - -Run: `cd app && npx eslint --ext .ts,.tsx src` -Expected: no errors. - -- [ ] **Step 3: Run the full test suites** - -Run: `cd cron && npm test` -Expected: green. - -Run: `cd app && npm test` -Expected: green. - -- [ ] **Step 4: Format check from repo root** - -Run: `npm run format:check` -Expected: pass (lint-staged formats on commit; this confirms nothing drifted). - -- [ ] **Step 5: Finish the branch** - -Use the **superpowers:finishing-a-development-branch** skill to verify tests, then present merge/PR options. Per repo policy, the PR targets `develop`. Before opening: run `/code-review`; because this is an auth/RLS/API change, also run `/security-review`. - ---- - -## Self-Review Notes - -- **Spec coverage:** Task 1 = RPCs + policy drop + DB verification; Tasks 2–3 = app RPC migration + call site; Task 4 = dead-code deletion; Task 5 = regression + rollout (one shot — schema and app land on the same branch/PR). `delete_my_account` and `register_device` are untouched, matching the spec's out-of-scope list. -- **Signature consistency:** `update_notify_time(p_id, p_notify_at)` and `link_device_to_user(p_id)` are used identically in the SQL (Task 1), the app code (Tasks 2–3), and the tests. `linkDeviceToUser` is single-arg everywhere after Task 3 (helper, call site, tests). -- **The `API_URL` / `EXPO_PUBLIC_API_URL` cleanup is intentionally excluded** (spec marks it optional/out-of-scope) to keep this change focused. diff --git a/docs/superpowers/plans/2026-06-15-notify-catch-up-windows.md b/docs/superpowers/plans/2026-06-15-notify-catch-up-windows.md deleted file mode 100644 index 9f96be3..0000000 --- a/docs/superpowers/plans/2026-06-15-notify-catch-up-windows.md +++ /dev/null @@ -1,715 +0,0 @@ -# Notify Catch-Up Windows Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Replace `jobs/notify.ts`'s fixed 30-minute window with a `(last_run, now]` catch-up window backed by a Supabase singleton, so devices are never dropped when the GitHub Actions schedule fires irregularly. - -**Architecture:** A one-row `notify_state` table stores `last_run_at`. Two `SECURITY DEFINER` Postgres RPCs hold the selection rule once: `peek_due_notifications()` (read-only, used by the workflow guard) and `claim_due_notifications()` (returns due FCM tokens **and** advances `last_run_at` atomically, used by the job). A new `sendDueNotifications()` in `cron/src/notify.ts` calls the claim RPC and dispatches; `jobs/notify.ts` becomes a thin wrapper. `notifyWindow.ts` and the `NOTIFY_WINDOW_*` plumbing are deleted. - -**Tech Stack:** TypeScript (ts-node), Supabase (`@supabase/supabase-js` + PL/pgSQL RPCs), firebase-admin (FCM), Jest, GitHub Actions. - -**Spec:** `docs/superpowers/specs/2026-06-15-notify-catch-up-windows-design.md` - -**Branch:** `feat/notify-catch-up-windows` (already checked out; the spec commit lives here). - ---- - -## File Structure - -| File | Change | Responsibility | -| ------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------- | -| `supabase/schema.sql` | Modify | Canonical reference: add `notify_state` table + seed + `peek_due_notifications()` + `claim_due_notifications()`. | -| Live Supabase project | Migrate | Apply the same DDL via `apply_migration` so the deployed DB matches. | -| `cron/src/notify.ts` | Modify | Add exported `sendDueNotifications()` (claim RPC → dispatch). | -| `cron/src/tests/notify.test.ts` | Modify | Add `rpc` to the Supabase mock; unit-test `sendDueNotifications`. | -| `cron/jobs/notify.ts` | Rewrite | Thin wrapper: `loadPulseConfig()` → `sendDueNotifications()` → log → exit. | -| `.github/workflows/notify.yml` | Modify | Guard calls `peek_due_notifications` RPC; remove `NOTIFY_WINDOW_*` env. | -| `cron/src/lib/notifyWindow.ts` | Delete | Window math no longer used. | -| `cron/src/tests/notifyWindow.test.ts` | Delete | Tests the deleted module. | -| `cron/e2e/notifyWindows.ts` | Create | Manual runner asserting `peek`/`claim` against a real Supabase project for the 4 key cases. | - ---- - -## Task 1: Add the `notify_state` singleton to `supabase/schema.sql` - -**Files:** - -- Modify: `supabase/schema.sql` (append after the `digests` block, before any trailing content) - -- [ ] **Step 1: Add the table + seed to `schema.sql`** - -Append this block to `supabase/schema.sql`: - -```sql --- ============================================================ --- notify_state --- Single-row store of the last time jobs/notify.ts processed a window. --- jobs/notify.ts notifies devices whose notify_at fell in (last_run_at, now], --- so no device is dropped when the GitHub Actions schedule fires irregularly. --- Seeded with now() so the first run after deploy sees a tiny window and does --- NOT mass-notify every device. Only the cron service-role key touches this --- table; the service role bypasses RLS, so no policy is defined. --- ============================================================ -CREATE TABLE IF NOT EXISTS notify_state ( - id BOOLEAN PRIMARY KEY DEFAULT TRUE CHECK (id), - last_run_at TIMESTAMPTZ NOT NULL DEFAULT now() -); - --- Exactly one row (id = TRUE); CHECK (id) + the PK forbid a second row. -INSERT INTO notify_state (id) VALUES (TRUE) ON CONFLICT (id) DO NOTHING; - -ALTER TABLE notify_state ENABLE ROW LEVEL SECURITY; -``` - -- [ ] **Step 2: Verify the SQL parses (lint-by-eye)** - -Re-read the appended block. Confirm: `CHECK (id)` present, `DEFAULT now()` on `last_run_at`, seed `INSERT ... ON CONFLICT (id) DO NOTHING`, `ENABLE ROW LEVEL SECURITY`. No trailing syntax errors. - -- [ ] **Step 3: Commit** - -```bash -git add supabase/schema.sql -git commit -m "feat(db): add notify_state singleton for catch-up windows" -``` - ---- - -## Task 2: Add the two RPCs to `supabase/schema.sql` - -**Files:** - -- Modify: `supabase/schema.sql` (append after the `notify_state` table block) - -- [ ] **Step 1: Add `peek_due_notifications()` to `schema.sql`** - -Append: - -```sql --- peek_due_notifications — read-only "is any device due since last_run_at?". --- Called by the .github/workflows/notify.yml guard step to skip checkout+npm on --- empty windows. A device is due iff the most recent occurrence of its notify_at --- (today if it has already passed in UTC, else yesterday) is later than --- last_run_at. notify_at is a UTC time-of-day, matching the rest of the cron. -CREATE OR REPLACE FUNCTION peek_due_notifications() -RETURNS boolean -LANGUAGE sql -SECURITY DEFINER -SET search_path = public -AS $$ - SELECT EXISTS ( - SELECT 1 - FROM devices d, notify_state s - WHERE d.notify_at IS NOT NULL - AND ( - (CASE - WHEN d.notify_at <= (now() AT TIME ZONE 'UTC')::time - THEN (now() AT TIME ZONE 'UTC')::date + d.notify_at - ELSE (now() AT TIME ZONE 'UTC')::date - 1 + d.notify_at - END) AT TIME ZONE 'UTC' - ) > s.last_run_at - ); -$$; - -GRANT EXECUTE ON FUNCTION peek_due_notifications() TO service_role; -``` - -- [ ] **Step 2: Add `claim_due_notifications()` to `schema.sql`** - -Append: - -```sql --- claim_due_notifications — returns the FCM tokens of all due devices AND --- advances last_run_at to now() atomically (FOR UPDATE serialises with the --- workflow's concurrency group). Advancing before dispatch gives at-most-once --- delivery, which is preferred over double-notify spam. Called by jobs/notify.ts. -CREATE OR REPLACE FUNCTION claim_due_notifications() -RETURNS TABLE (fcm_token text) -LANGUAGE plpgsql -SECURITY DEFINER -SET search_path = public -AS $$ -DECLARE - v_now timestamptz := now(); - v_last timestamptz; -BEGIN - SELECT s.last_run_at INTO v_last - FROM notify_state s - WHERE s.id - FOR UPDATE; - - RETURN QUERY - SELECT d.fcm_token - FROM devices d - WHERE d.notify_at IS NOT NULL - AND ( - (CASE - WHEN d.notify_at <= (v_now AT TIME ZONE 'UTC')::time - THEN (v_now AT TIME ZONE 'UTC')::date + d.notify_at - ELSE (v_now AT TIME ZONE 'UTC')::date - 1 + d.notify_at - END) AT TIME ZONE 'UTC' - ) > v_last; - - UPDATE notify_state SET last_run_at = v_now WHERE id; -END; -$$; - -GRANT EXECUTE ON FUNCTION claim_due_notifications() TO service_role; -``` - -- [ ] **Step 3: Verify by eye** - -Confirm both functions are `SECURITY DEFINER` with `SET search_path = public` (matching `register_device`), `claim` uses `FOR UPDATE` and the `UPDATE ... WHERE id` advance is the **last** statement, and the occurrence `CASE` is identical in both functions. - -- [ ] **Step 4: Commit** - -```bash -git add supabase/schema.sql -git commit -m "feat(db): add peek/claim due-notification RPCs" -``` - ---- - -## Task 3: Apply the migration to the live Supabase project - -**Files:** none (uses the Supabase MCP tool against the live project). - -> ⚠️ This mutates the live Supabase database. The DDL is additive (new table + new functions) and idempotent (`IF NOT EXISTS`, `CREATE OR REPLACE`, `ON CONFLICT DO NOTHING`), so re-running is safe. Nothing existing is dropped or altered. - -- [ ] **Step 1: Apply the migration** - -Call `mcp__supabase__apply_migration` with: - -- `name`: `notify_catch_up_windows` -- `query`: the **combined** SQL from Task 1 Step 1 + Task 2 Step 1 + Task 2 Step 2 (the `notify_state` table + seed + `ENABLE ROW LEVEL SECURITY`, then both `CREATE OR REPLACE FUNCTION ... GRANT` blocks), concatenated in that order. - -- [ ] **Step 2: Smoke-test the table and seed** - -Call `mcp__supabase__execute_sql` with: - -```sql -SELECT id, last_run_at FROM notify_state; -``` - -Expected: exactly one row, `id = true`, `last_run_at` ≈ now (within seconds). - -- [ ] **Step 3: Smoke-test the read-only RPC** - -Call `mcp__supabase__execute_sql` with: - -```sql -SELECT peek_due_notifications(); -``` - -Expected: returns `true` or `false` without error (value depends on current devices; correctness is asserted in Task 7). Do **not** call `claim_due_notifications()` here — it would advance `last_run_at`. - ---- - -## Task 4: Add `sendDueNotifications()` to `cron/src/notify.ts` (TDD) - -**Files:** - -- Modify: `cron/src/tests/notify.test.ts` (add `rpc` to mock; new `describe`) -- Modify: `cron/src/notify.ts` (new exported function) - -- [ ] **Step 1: Add `rpc` to the Supabase mock** - -In `cron/src/tests/notify.test.ts`, extend the `mockSupabase` object (currently lines ~11-18) so it includes an `rpc` mock: - -```ts -const mockSupabase = { - from: jest.fn().mockReturnThis(), - delete: jest.fn().mockReturnThis(), - lt: jest.fn().mockResolvedValue({ error: null }), - upsert: jest.fn().mockResolvedValue({ error: null }), - select: jest.fn().mockResolvedValue({ data: [], error: null }), - in: jest.fn().mockResolvedValue({ error: null }), - rpc: jest.fn().mockResolvedValue({ data: [], error: null }), -}; -``` - -- [ ] **Step 2: Import the new function and write the failing tests** - -Update the import line at the top of `cron/src/tests/notify.test.ts`: - -```ts -import { - persistDigests, - persistGlobalDigest, - dispatchFcm, - sendNotifications, - sendDueNotifications, -} from '../notify'; -``` - -Append this `describe` block at the end of `cron/src/tests/notify.test.ts`: - -```ts -// ── sendDueNotifications ───────────────────────────────────────────────────────── - -describe('sendDueNotifications', () => { - beforeEach(() => { - jest.clearAllMocks(); - mockSupabase.from.mockReturnThis(); - mockSupabase.in.mockResolvedValue({ error: null }); - mockSupabase.rpc.mockResolvedValue({ data: [], error: null }); - process.env.SUPABASE_URL = 'https://test.supabase.co'; - process.env.SUPABASE_SECRET_KEY = 'test-key'; - process.env.FIREBASE_PROJECT_ID = 'test-project'; - process.env.FIREBASE_CLIENT_EMAIL = 'test@test.iam.gserviceaccount.com'; - process.env.FIREBASE_PRIVATE_KEY = - '-----BEGIN PRIVATE KEY-----\\ntest\\n-----END PRIVATE KEY-----'; - }); - - it('claims due tokens via the claim_due_notifications RPC and dispatches them', async () => { - mockSupabase.rpc.mockResolvedValueOnce({ - data: [{ fcm_token: 'tok-a' }, { fcm_token: 'tok-b' }], - error: null, - }); - mockSendEachForMulticast.mockResolvedValueOnce({ - successCount: 2, - responses: [{ error: null }, { error: null }], - }); - - const result = await sendDueNotifications(); - - expect(mockSupabase.rpc).toHaveBeenCalledWith('claim_due_notifications'); - expect(mockSendEachForMulticast).toHaveBeenCalledWith( - expect.objectContaining({ tokens: ['tok-a', 'tok-b'] }), - ); - expect(result).toEqual({ sent: 2, total: 2 }); - }); - - it('skips dispatch and returns zero when no devices are due', async () => { - mockSupabase.rpc.mockResolvedValueOnce({ data: [], error: null }); - - const result = await sendDueNotifications(); - - expect(mockSendEachForMulticast).not.toHaveBeenCalled(); - expect(result).toEqual({ sent: 0, total: 0 }); - }); - - it('treats null RPC data as no devices due', async () => { - mockSupabase.rpc.mockResolvedValueOnce({ data: null, error: null }); - - const result = await sendDueNotifications(); - - expect(mockSendEachForMulticast).not.toHaveBeenCalled(); - expect(result).toEqual({ sent: 0, total: 0 }); - }); - - it('throws when the claim RPC returns an error', async () => { - mockSupabase.rpc.mockResolvedValueOnce({ data: null, error: { message: 'rpc failed' } }); - - await expect(sendDueNotifications()).rejects.toThrow( - 'Failed to claim due notifications: rpc failed', - ); - }); -}); -``` - -- [ ] **Step 3: Run the tests to verify they fail** - -Run: `cd cron && npx jest src/tests/notify.test.ts -t sendDueNotifications` -Expected: FAIL — `sendDueNotifications` is not exported / `is not a function`. - -- [ ] **Step 4: Implement `sendDueNotifications()` in `cron/src/notify.ts`** - -Add this exported function to `cron/src/notify.ts` (place it just above `sendNotifications`, near the other dispatch helpers): - -```ts -/** - * Notify devices whose notify_at fell in (last_run_at, now]. The - * claim_due_notifications RPC returns the due FCM tokens and advances - * last_run_at atomically, so unreliable cron firing never drops a device. - * Used by jobs/notify.ts. - */ -export async function sendDueNotifications(): Promise<{ sent: number; total: number }> { - const log = getLogger('notify'); - const db = buildClient(); - - const { data, error } = await db.rpc('claim_due_notifications'); - if (error) { - throw new Error(`Failed to claim due notifications: ${error.message}`); - } - - const tokens = ((data ?? []) as Array<{ fcm_token: string }>).map((d) => d.fcm_token); - if (tokens.length === 0) { - log.info('No devices due'); - return { sent: 0, total: 0 }; - } - - return dispatchFcm(tokens); -} -``` - -- [ ] **Step 5: Run the tests to verify they pass** - -Run: `cd cron && npx jest src/tests/notify.test.ts -t sendDueNotifications` -Expected: PASS (4 passing). - -- [ ] **Step 6: Run the full notify suite to confirm no regressions** - -Run: `cd cron && npx jest src/tests/notify.test.ts` -Expected: PASS (all existing + 4 new). - -- [ ] **Step 7: Commit** - -```bash -git add cron/src/notify.ts cron/src/tests/notify.test.ts -git commit -m "feat(cron): add sendDueNotifications backed by claim RPC" -``` - ---- - -## Task 5: Rewrite `cron/jobs/notify.ts` as a thin wrapper - -**Files:** - -- Rewrite: `cron/jobs/notify.ts` - -- [ ] **Step 1: Replace the file contents** - -Overwrite `cron/jobs/notify.ts` with: - -```ts -import { loadPulseConfig } from '../src/config'; -import { sendDueNotifications } from '../src/notify'; -import { getLogger } from '../src/logging'; - -/** - * GitHub Actions cron job (.github/workflows/notify.yml) — notify devices whose - * notify_at fell in (last_run_at, now]. The catch-up window means a device is - * never dropped when the schedule fires irregularly; in the reliable every-30-min - * case it matches the old fixed-window behaviour. Devices with notify_at = NULL - * are handled by jobs/daily-digest.ts instead. - * - * Run from cron/: `npx ts-node -r tsconfig-paths/register jobs/notify.ts` - */ -async function main(): Promise { - loadPulseConfig(); - - const log = getLogger('notify-cron'); - - try { - const { sent, total } = await sendDueNotifications(); - log.info(`Sent ${sent}/${total} notifications`); - } catch (err) { - log.error(`Unhandled error: ${String(err)}`); - process.exit(1); - } - - // firebase-admin / supabase keep sockets open — without an explicit exit the - // Actions job would hang until its timeout after the work is done. - process.exit(0); -} - -void main(); -``` - -- [ ] **Step 2: Typecheck the cron package** - -Run: `cd cron && npx tsc --noEmit` -Expected: no errors. (`notifyWindow` is still present at this point; it is deleted in Task 7. The rewritten `notify.ts` no longer imports it, which is fine.) - -- [ ] **Step 3: Commit** - -```bash -git add cron/jobs/notify.ts -git commit -m "refactor(cron): notify.ts delegates to sendDueNotifications" -``` - ---- - -## Task 6: Update the `notify.yml` guard to use the RPC - -**Files:** - -- Modify: `.github/workflows/notify.yml` - -- [ ] **Step 1: Replace the guard step** - -In `.github/workflows/notify.yml`, replace the entire `- name: Check window for devices` step (currently the `id: guard` block) with: - -```yaml -# Most runs have no due devices. Ask Supabase first (no checkout/install -# needed) via the peek RPC and skip the heavy steps when nothing is due. -# The RPC encapsulates the (last_run_at, now] selection rule shared with -# the job. Fails OPEN: any guard error runs the job. -- name: Check for due devices - id: guard - env: - SUPABASE_URL: ${{ secrets.SUPABASE_URL }} - SUPABASE_SECRET_KEY: ${{ secrets.SUPABASE_SECRET_KEY }} - run: | - node -e ' - const url = `${process.env.SUPABASE_URL}/rest/v1/rpc/peek_due_notifications`; - fetch(url, { - method: "POST", - headers: { - apikey: process.env.SUPABASE_SECRET_KEY, - Authorization: `Bearer ${process.env.SUPABASE_SECRET_KEY}`, - "Content-Type": "application/json", - }, - body: "{}", - }) - .then((r) => (r.ok ? r.json() : Promise.reject(new Error(`HTTP ${r.status}`)))) - .then((due) => { - console.log(`due: ${due}`); - process.stdout.write(`has=${due === true}\n`); - }) - .catch((e) => { - console.log(`guard error (${e.message}) — failing open`); - process.stdout.write("has=true\n"); - }); - ' | tee guard.out - grep "^has=" guard.out >> "$GITHUB_OUTPUT" || echo "has=true" >> "$GITHUB_OUTPUT" -``` - -- [ ] **Step 2: Remove the `NOTIFY_WINDOW_*` env from the job step** - -In the same file, in the `- uses: ./.github/actions/run-cron-job` step's `env:` block, delete these two lines: - -```yaml -NOTIFY_WINDOW_START: ${{ steps.guard.outputs.start }} -NOTIFY_WINDOW_END: ${{ steps.guard.outputs.end }} -``` - -The remaining `env:` keys (`SUPABASE_*`, `FIREBASE_*`) and `with: { job: jobs/notify.ts }` are unchanged. - -- [ ] **Step 3: Verify by eye** - -Confirm: the guard no longer emits `start=`/`end=` lines; the `if: steps.guard.outputs.has == 'true'` conditions on `checkout` and `run-cron-job` are unchanged; the schedule (`*/30 * * * *`), `concurrency`, and `timeout-minutes` are untouched. `.github/workflows/daily-digest.yml` is not modified. - -- [ ] **Step 4: Commit** - -```bash -git add .github/workflows/notify.yml -git commit -m "feat(ci): notify guard uses peek_due_notifications RPC" -``` - ---- - -## Task 7: Delete `notifyWindow.ts` and its test - -**Files:** - -- Delete: `cron/src/lib/notifyWindow.ts` -- Delete: `cron/src/tests/notifyWindow.test.ts` - -- [ ] **Step 1: Confirm nothing else imports `notifyWindow`** - -Use the Grep tool (or `rg`) to search the repo for `notifyWindow|NOTIFY_WINDOW`. -Expected: matches only in the two files being deleted (`cron/src/lib/notifyWindow.ts`, `cron/src/tests/notifyWindow.test.ts`). If `cron/jobs/notify.ts` or `.github/workflows/notify.yml` still reference either, Tasks 5/6 were not applied — fix those first. - -- [ ] **Step 2: Delete both files** - -```bash -git rm cron/src/lib/notifyWindow.ts cron/src/tests/notifyWindow.test.ts -``` - -- [ ] **Step 3: Typecheck and run the full cron suite** - -Run: `cd cron && npx tsc --noEmit && npx jest` -Expected: no type errors; all tests pass; no reference to the deleted module. - -- [ ] **Step 4: Commit** - -```bash -git commit -m "chore(cron): remove unused notifyWindow window logic" -``` - ---- - -## Task 8: Add the `e2e/notifyWindows.ts` manual runner - -**Files:** - -- Create: `cron/e2e/notifyWindows.ts` - -> This runner needs live `SUPABASE_URL` / `SUPABASE_SECRET_KEY` (loaded via `../src/bootstrap`, matching the other `e2e/` runners). It mutates the project: it snapshots and restores `notify_state.last_run_at`, and inserts then deletes device rows keyed by a unique `e2e-notify-` token prefix. Run against a dev project when possible. - -- [ ] **Step 1: Create the runner** - -Create `cron/e2e/notifyWindows.ts`: - -```ts -import '../src/bootstrap'; -import { loadPulseConfig } from '../src/config'; -import { getLogger } from '../src/logging'; -import { buildClient } from '../src/notify'; - -const log = getLogger('e2e:notifyWindows'); - -// Fixed UUIDs + token prefix so cleanup can target exactly these rows. -const PREFIX = 'e2e-notify-'; -const DEV_A = '00000000-0000-4000-8000-0000000000a1'; // notify_at in-window -const DEV_B = '00000000-0000-4000-8000-0000000000b2'; // notify_at out-of-window - -function pad(n: number): string { - return String(n).padStart(2, '0'); -} - -/** UTC HH:MM:SS string `offsetMin` minutes from `base`. */ -function utcTime(base: Date, offsetMin: number): string { - const d = new Date(base.getTime() + offsetMin * 60_000); - return `${pad(d.getUTCHours())}:${pad(d.getUTCMinutes())}:${pad(d.getUTCSeconds())}`; -} - -async function main(): Promise { - loadPulseConfig(); - const db = buildClient(); - - // Snapshot last_run_at so we can restore it after the destructive claim test. - const { data: snap, error: snapErr } = await db - .from('notify_state') - .select('last_run_at') - .eq('id', true) - .single(); - if (snapErr) throw new Error(`Failed to read notify_state: ${snapErr.message}`); - const savedLastRun = snap!.last_run_at as string; - log.info(`Saved last_run_at = ${savedLastRun}`); - - let failures = 0; - const check = (name: string, ok: boolean) => { - log.info(`${ok ? 'PASS' : 'FAIL'} — ${name}`); - if (!ok) failures += 1; - }; - - try { - const now = new Date(); - - // Device A: notify_at 5 min ago (inside a (last_run, now] window that - // starts 30 min ago). Device B: notify_at 5 min in the future (outside). - await db.from('devices').upsert([ - { id: DEV_A, fcm_token: `${PREFIX}a`, notify_at: utcTime(now, -5) }, - { id: DEV_B, fcm_token: `${PREFIX}b`, notify_at: utcTime(now, +5) }, - ]); - - // Set last_run_at to 30 min ago: A's time is inside (last_run, now], B's is not. - await db - .from('notify_state') - .update({ last_run_at: new Date(now.getTime() - 30 * 60_000).toISOString() }) - .eq('id', true); - - // peek must report due (A qualifies). - const { data: peek, error: peekErr } = await db.rpc('peek_due_notifications'); - if (peekErr) throw new Error(`peek failed: ${peekErr.message}`); - check('normal window: peek returns true when a device is due', peek === true); - - // claim returns A's token (not B's) and advances last_run_at. - const { data: claimed, error: claimErr } = await db.rpc('claim_due_notifications'); - if (claimErr) throw new Error(`claim failed: ${claimErr.message}`); - const tokens = ((claimed ?? []) as Array<{ fcm_token: string }>).map((r) => r.fcm_token); - check('normal window: claim returns the in-window device', tokens.includes(`${PREFIX}a`)); - check('normal window: claim excludes the out-of-window device', !tokens.includes(`${PREFIX}b`)); - - // After claim, last_run_at advanced to ~now, so the same A is no longer due. - const { data: peek2, error: peek2Err } = await db.rpc('peek_due_notifications'); - if (peek2Err) throw new Error(`peek2 failed: ${peek2Err.message}`); - check('idempotency: peek returns false immediately after claim', peek2 === false); - - // >24h outage: last_run far in the past → every device is due once. - await db - .from('notify_state') - .update({ last_run_at: new Date(now.getTime() - 48 * 3600 * 1000).toISOString() }) - .eq('id', true); - const { data: claimedAll, error: allErr } = await db.rpc('claim_due_notifications'); - if (allErr) throw new Error(`claim (outage) failed: ${allErr.message}`); - const allTokens = ((claimedAll ?? []) as Array<{ fcm_token: string }>).map((r) => r.fcm_token); - check( - '>24h outage: both test devices are claimed', - allTokens.includes(`${PREFIX}a`) && allTokens.includes(`${PREFIX}b`), - ); - } finally { - // Cleanup: remove test devices and restore the original last_run_at. - await db.from('devices').delete().in('id', [DEV_A, DEV_B]); - await db.from('notify_state').update({ last_run_at: savedLastRun }).eq('id', true); - log.info(`Restored last_run_at = ${savedLastRun}; removed test devices`); - } - - if (failures > 0) { - log.error(`${failures} check(s) failed`); - process.exit(1); - } - log.info('All notify-window checks passed'); - process.exit(0); -} - -main().catch((err) => { - log.error(String(err)); - process.exit(1); -}); -``` - -- [ ] **Step 2: Typecheck** - -Run: `cd cron && npx tsc --noEmit` -Expected: no errors. - -- [ ] **Step 3: Run the e2e runner against a live project (manual)** - -Run: `cd cron && npx ts-node -r tsconfig-paths/register e2e/notifyWindows.ts` -Expected: `PASS` for every check, then `All notify-window checks passed`, exit 0. `last_run_at` is restored and the `e2e-notify-*` device rows are deleted. - -> If run against the production project, note this momentarily inserts two devices with placeholder FCM tokens. They are deleted in the `finally` block and never receive a push (the runner never calls `dispatchFcm`). - -- [ ] **Step 4: Commit** - -```bash -git add cron/e2e/notifyWindows.ts -git commit -m "test(cron): e2e runner for peek/claim notify windows" -``` - ---- - -## Task 9: Final verification - -**Files:** none (verification only). - -- [ ] **Step 1: Typecheck, lint, test, format** - -Run each and confirm clean: - -```bash -cd cron && npx tsc --noEmit -cd cron && npx eslint --ext .ts src -cd cron && npm test -``` - -```bash -# from repo root -npm run format:check -``` - -Expected: typecheck clean; eslint clean; Jest green (notify suite includes the new `sendDueNotifications` tests; no `notifyWindow` suite remains); format check passes. If `format:check` flags the new/edited files, run `npm run format` and amend the relevant commit. - -- [ ] **Step 2: Confirm the deployed DB matches** - -Call `mcp__supabase__execute_sql` with: - -```sql -SELECT - to_regclass('public.notify_state') IS NOT NULL AS has_table, - to_regprocedure('public.peek_due_notifications()') IS NOT NULL AS has_peek, - to_regprocedure('public.claim_due_notifications()') IS NOT NULL AS has_claim; -``` - -Expected: all three columns `true`. - -- [ ] **Step 3: Push the branch and open a PR to `develop`** - -Per `CLAUDE.md`: changes touch notifications + CI, so run `/code-review` and `/security-review` before opening. Then: - -```bash -git push -u origin feat/notify-catch-up-windows -``` - -Open a PR targeting **`develop`** (confirm the base branch). Include in the body: the spec link, the at-most-once delivery trade-off, and a note that the migration was already applied to the live project via `apply_migration`. - ---- - -## Self-Review Notes (for the implementer) - -- **Spec coverage:** `notify_state` table (Task 1), `peek`/`claim` RPCs (Task 2), live migration + seed (Task 3), thin `notify.ts` via `sendDueNotifications` (Tasks 4–5), guard RPC swap + `NOTIFY_WINDOW_*` removal (Task 6), `notifyWindow.ts` + test deletion (Task 7), option-(a) tests = mocked unit tests (Task 4) + `e2e/` runner (Task 8). `daily-digest.yml`/`daily-digest.ts` deliberately untouched. -- **Type consistency:** the RPC name `claim_due_notifications` and the `{ fcm_token: string }[]` row shape are identical across `sendDueNotifications` (Task 4), its tests (Task 4), and the e2e runner (Task 8). `peek_due_notifications` returns `boolean`, consumed as `due === true` in the guard (Task 6) and the runner (Task 8). -- **At-most-once:** `claim` advances `last_run_at` as its final statement, before dispatch — matches the spec's stated trade-off. diff --git a/docs/superpowers/specs/2026-06-15-devices-rls-lockdown-design.md b/docs/superpowers/specs/2026-06-15-devices-rls-lockdown-design.md deleted file mode 100644 index 8b6ebfa..0000000 --- a/docs/superpowers/specs/2026-06-15-devices-rls-lockdown-design.md +++ /dev/null @@ -1,275 +0,0 @@ -# Devices Table RLS Lockdown — Design - -**Date:** 2026-06-15 -**Status:** Approved (brainstorming) → ready for implementation plan -**Branch:** `feat/devices-rls-lockdown` - -## Goal - -Remove the wide-open Row Level Security policies on the `devices` table and route -every client write through `SECURITY DEFINER` RPCs, so the shipped publishable -(anon) key can no longer read or tamper with the table directly. - -## Background — current state - -`devices` has RLS enabled but three fully-open policies: - -```sql -CREATE POLICY "device self-select" ON devices FOR SELECT USING (true); -CREATE POLICY "device self-register" ON devices FOR INSERT WITH CHECK (true); -CREATE POLICY "device self-update" ON devices FOR UPDATE USING (true) WITH CHECK (true); -``` - -The app (publishable/anon key) touches `devices` through three paths: - -| App function | Mechanism | Policy relied on | -| ------------------ | ----------------------------------------- | --------------------------- | -| `upsertDevice` | `register_device` RPC (SECURITY DEFINER) | none — definer bypasses RLS | -| `linkDeviceToUser` | direct `UPDATE … SET user_id WHERE id=` | `device self-update` | -| `updateNotifyTime` | direct `UPDATE … SET notify_at WHERE id=` | `device self-update` | - -The `.select('id')` RETURNING tacked onto the two direct updates is what needs -the `device self-select` policy. - -### Why `auth.uid()` RLS does not fit - -Devices register and set `notify_at` **before login** — notifications work for -logged-out users. With the anon key, `auth.uid()` is `NULL` in those calls, so a -`user_id = auth.uid()` policy would block registration and notify-time writes for -every anonymous device (the common case). The data model is device-centric, not -user-centric. - -### Threat closed - -With `USING (true)` SELECT, anyone who extracts the publishable key from the -shipped APK can `SELECT *` and harvest every `fcm_token` ↔ `user_id` mapping, and -blind-`UPDATE`/`INSERT` arbitrary rows. This design removes that. - -## Design - -### Security model - -Keep RLS **enabled** on `devices` and **drop all three policies**. With no -policies, RLS denies all direct `anon`/`authenticated` table access (no SELECT, -UPDATE, or INSERT). The service-role key (cron) bypasses RLS and is unaffected. - -Every write flows through a `SECURITY DEFINER` RPC — the pattern `register_device` -already established. The per-install device UUID (random v4, stored in MMKV) is -the unguessable capability that authorizes touching a row. An attacker with the -publishable key can no longer enumerate the table or blind-update rows; they would -need to already know a specific device UUID. - -### Database changes (`supabase/schema.sql`) - -**1. `register_device`** — unchanged (already definer, already granted to anon). - -**2. New `update_notify_time(p_id uuid, p_notify_at time) RETURNS boolean`:** - -```sql -CREATE OR REPLACE FUNCTION update_notify_time(p_id uuid, p_notify_at time) -RETURNS boolean -LANGUAGE plpgsql -SECURITY DEFINER -SET search_path = public -AS $$ -BEGIN - UPDATE devices - SET notify_at = p_notify_at, updated_at = now() - WHERE id = p_id; - RETURN FOUND; -- false = no such device row (preserves the app's "not registered yet" warning) -END; -$$; - -REVOKE EXECUTE ON FUNCTION update_notify_time(uuid, time) FROM PUBLIC; -GRANT EXECUTE ON FUNCTION update_notify_time(uuid, time) TO anon, authenticated, service_role; -``` - -**3. New `link_device_to_user(p_id uuid) RETURNS boolean`:** - -`user_id` is **not** a parameter — it is derived from the caller's JWT via -`auth.uid()`, so a device can only ever be linked to the authenticated caller's -own identity. (Confirmed safe: the app client has `persistSession: true` and the -app only calls this when a session exists, so supabase-js attaches the user JWT -and `auth.uid()` resolves.) - -```sql -CREATE OR REPLACE FUNCTION link_device_to_user(p_id uuid) -RETURNS boolean -LANGUAGE plpgsql -SECURITY DEFINER -SET search_path = public -AS $$ -DECLARE - v_uid uuid := auth.uid(); -BEGIN - IF v_uid IS NULL THEN - RAISE EXCEPTION 'link_device_to_user requires an authenticated user'; - END IF; - UPDATE devices - SET user_id = v_uid, updated_at = now() - WHERE id = p_id; - RETURN FOUND; -- false = device row not present yet (drives the app's retry) -END; -$$; - -REVOKE EXECUTE ON FUNCTION link_device_to_user(uuid) FROM PUBLIC; -GRANT EXECUTE ON FUNCTION link_device_to_user(uuid) TO authenticated, service_role; -``` - -`link_device_to_user` is granted to `authenticated` only (not `anon`) — it -requires an authenticated session by definition. - -**4. Drop the open policies:** - -```sql -DROP POLICY IF EXISTS "device self-select" ON devices; -DROP POLICY IF EXISTS "device self-register" ON devices; -DROP POLICY IF EXISTS "device self-update" ON devices; -``` - -RLS stays `ENABLE`d, so the table is now deny-all for direct anon/authenticated -access. The schema-file comments on the old policies are replaced with a note -explaining the RPC-only write surface. - -### App changes (`app/src/notifications/devices.ts`) - -**`updateNotifyTime`** — replace the direct update with the RPC; branch on the -returned boolean instead of `data.length`: - -```ts -const { data, error } = await supabase.rpc('update_notify_time', { - p_id: deviceId, - p_notify_at: notifyAt, -}); -if (error) { - log.warn(`updateNotifyTime failed: ${error.message}`); -} else if (data !== true) { - log.warn( - `updateNotifyTime: no device row for ${deviceId.slice(0, 8)}… — notify_at not saved (device may not have registered yet)`, - ); -} else { - log.debug(`notify_at updated for device ${deviceId.slice(0, 8)}…`); -} -``` - -**`linkDeviceToUser`** — drop the `userId` parameter (now derived server-side from -the JWT); keep the 3× retry, switching the 0-row check to `data !== true`: - -```ts -export async function linkDeviceToUser(deviceId: string): Promise { - const supabase = getSupabase(); - if (!supabase) return; - - const MAX_ATTEMPTS = 3; - const RETRY_DELAY_MS = 2000; - - for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) { - const { data, error } = await supabase.rpc('link_device_to_user', { p_id: deviceId }); - if (error) { - log.warn(`linkDeviceToUser failed: ${error.message}`); - return; - } - if (data === true) { - log.info(`device ${deviceId.slice(0, 8)}… linked`); - return; - } - if (attempt < MAX_ATTEMPTS) { - await new Promise((resolve) => setTimeout(resolve, RETRY_DELAY_MS)); - } - } - log.warn( - `linkDeviceToUser: no device row for ${deviceId.slice(0, 8)}… after ${MAX_ATTEMPTS} attempts`, - ); -} -``` - -**`useDeviceRegistration.ts`** — update the call site from -`linkDeviceToUser(deviceId, userId)` to `linkDeviceToUser(deviceId)`. The `userId` -gate (only call when a session exists) stays. - -After these edits, `app/src/notifications/devices.ts` no longer references -`.from('devices')` at all. - -### Dead-code removal - -`cron/api/account.ts` is never called by the app: - -- Account deletion uses the `delete_my_account` RPC (`useSupabaseAuth.ts:92`). -- Device registration uses the `register_device` RPC (`devices.ts`). -- `API_URL` (`app/src/config.ts:21`) is declared but never fetched. - -Delete: - -- `cron/api/account.ts` -- `cron/src/tests/account.test.ts` - -Optional follow-up cleanup (not required for this change): remove the now-dead -`API_URL` constant and its `EXPO_PUBLIC_API_URL` reference from the app config and -`.env.example`. Listed but out of scope to keep this change focused. - -## Testing - -### App unit tests (`app/src/tests/notifications/devices.test.ts`) - -Rewrite the cases that currently mock `.from('devices').update().eq().select()`: - -- `updateNotifyTime` → mock `supabase.rpc('update_notify_time', …)`: - - returns `{ data: true }` → debug success, no warn. - - returns `{ data: false }` → "not registered yet" warn. - - returns `{ error }` → failure warn. - - skips entirely when no cached FCM token (unchanged behavior). -- `linkDeviceToUser` → mock `supabase.rpc('link_device_to_user', …)`: - - `{ data: true }` first try → linked, single call. - - `{ data: false }` then `{ data: true }` → retried, eventually linked. - - `{ data: false }` ×3 → gives up with warn. - - `{ error }` → immediate return, no retry. - -### Database verification (run once against the live project) - -Run via the Supabase SQL editor (or `mcp__supabase.execute_sql`) to prove the -lockdown. `SET ROLE` simulates the anon/authenticated PostgREST roles: - -```sql --- anon: direct SELECT must be denied (no rows / permission), proving harvest is closed. -SET ROLE anon; -SELECT count(*) FROM devices; -- expect 0 rows visible (RLS denies) -RESET ROLE; - --- anon: register + notify-time RPCs must be callable. -SET ROLE anon; -SELECT register_device('00000000-0000-4000-8000-0000000000c3', 'rls-test-token', NULL); -SELECT update_notify_time('00000000-0000-4000-8000-0000000000c3', '09:00'); -RESET ROLE; - --- anon: link RPC must be rejected (no auth.uid()). -SET ROLE anon; -SELECT link_device_to_user('00000000-0000-4000-8000-0000000000c3'); -- expect EXCEPTION -RESET ROLE; - --- cleanup -DELETE FROM devices WHERE id = '00000000-0000-4000-8000-0000000000c3'; -``` - -(Document the expected outcome of each statement in the plan; this is a manual -verification, not an automated test, because the cron environment holds only the -service-role key, not the publishable key.) - -### Regression - -- `cron npm test` — green after deleting `account.test.ts` (the suite no longer - references the route). -- `app npm test` — green with the rewritten `devices.test.ts`. -- Typecheck + lint both packages. - -## Rollout - -Single shot — the user is the sole user, so no phased client migration is needed. -Apply the schema (add RPCs + grants, drop policies) and ship the app update -together. Order within the migration: create/replace the RPCs and grants first, -then drop the policies, so the table is never left writable-by-nobody mid-migration. - -## Out of scope - -- Removing the `API_URL` / `EXPO_PUBLIC_API_URL` config (optional cleanup noted above). -- Any change to `register_device`, `delete_my_account`, or the notify-window RPCs. -- `user_preferences` / `digests` / `notify_state` RLS (already correct). diff --git a/docs/superpowers/specs/2026-06-15-notify-catch-up-windows-design.md b/docs/superpowers/specs/2026-06-15-notify-catch-up-windows-design.md deleted file mode 100644 index 8b887dc..0000000 --- a/docs/superpowers/specs/2026-06-15-notify-catch-up-windows-design.md +++ /dev/null @@ -1,336 +0,0 @@ -# Notify catch-up windows — design - -**Date:** 2026-06-15 -**Status:** Approved, pending implementation plan -**Area:** `cron/` (notify job) + `supabase/` schema - -## Problem - -`jobs/notify.ts` notifies devices whose `notify_at` (a `TIME` of day) falls in a -fixed 30-minute window ending at "now". This is only correct if the GitHub -Actions schedule (`*/30 * * * *` in `.github/workflows/notify.yml`) actually -fires every 30 minutes. - -It does not. Observed Notify runs on 2026-06-14/15 fired roughly 3–4 times in a -day (e.g. 02:15, 07:38, 14:22) instead of 48 times. Every device whose -`notify_at` fell inside one of those multi-hour gaps was never queried and never -notified. The reporting user received **zero** notifications for this reason. - -`daily-digest.ts` handles devices with `notify_at = NULL` ("notify me when the -digest is ready") and is **out of scope** — it stays exactly as is. The two -workflows remain separate. - -## Goal - -Replace the fixed 30-minute window with a window of `(last_run, now]`, so no -time-of-day can fall through a gap between unreliable runs. Preserve behaviour in -the reliable case: if the schedule fires every 30 minutes, `last_run ≈ now − 30 -min` and the same devices are selected — no regression. - -A secondary goal is to remove now-unnecessary logic (window snapping, the -`NOTIFY_WINDOW_*` env plumbing, `notifyWindow.ts`). - -## Key decisions - -1. **Run memory: a global last-run timestamp.** A singleton `notify_state` row - holds `last_run_at`. Chosen over a per-device `last_notified_at` column - because it is contained entirely to `cron/` + Supabase (no app changes) and - matches the "between this run and the last run" model directly. - -2. **Selection logic lives once, in Supabase RPCs.** Both the cost-saving guard - step and the job must agree on "who is due". Putting the date math in SQL lets - both call the same function. This deletes `notifyWindow.ts` and the - `NOTIFY_WINDOW_START/END` env plumbing entirely. - -3. **At-most-once delivery.** `last_run_at` advances in the same RPC call that - returns the due tokens, before dispatch. A hard FCM outage can therefore drop - one window's notifications. This is preferred over double-notify spam and is - orthogonal to the actual failure (the runner not firing, not FCM failing). - Per-token failures are already tolerated by `dispatchFcm`, which returns - `sent/total`. - -## The selection rule - -Instead of windows with explicit midnight-wrap and ">24h outage" branches, a -single per-device predicate covers every case: - -> A device is **due** iff the **most recent occurrence of its `notify_at` time -> at-or-before now** is **later than `last_run_at`**. - -``` -most_recent_occurrence(T, now) = - today@T if T <= now's UTC time-of-day -- already happened today - yesterday@T otherwise -- hasn't happened yet today - -device is due ⟺ most_recent_occurrence(notify_at, now) > last_run_at -``` - -`notify_at` is treated as a **UTC** time-of-day, consistent with the existing -`HH:MM` UTC comparisons in the current code. `now()` and `last_run_at` are -`timestamptz`. The occurrence is built in UTC and compared as an absolute -instant. - -### Why one formula is enough - -| Case | Behaviour | Correct? | -| --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | -| Reliable 30-min cadence | `last_run ≈ now − 30min`; same devices as today's window | ✓ no regression | -| Window crosses midnight (last run 23:50, now 00:43) | a 23:55 user → yesterday@23:55 > 23:50 ✓; a 00:30 user → today@00:30 > 23:50 ✓; a 22:00 user → yesterday@22:00 < 23:50, not due ✓ | ✓ | -| Outage > 24h | every device's most-recent occurrence is after `last_run`, so each is selected **exactly once** (a single instant per device); the next run advances `last_run` past it | ✓ no duplicates | - -The half-open boundary flips from today's `[start, end)` to `(last_run, now]` -(exclusive start, inclusive end). A device sitting exactly on a boundary instant -is negligible and not a concern. - -## Components - -### 1. `supabase/schema.sql` — `notify_state` singleton - -```sql -CREATE TABLE IF NOT EXISTS notify_state ( - id BOOLEAN PRIMARY KEY DEFAULT TRUE CHECK (id), - last_run_at TIMESTAMPTZ NOT NULL DEFAULT now() -); - --- Exactly one row. Seeded with now() so the first run after deploy sees a tiny --- window and does NOT mass-notify every device. -INSERT INTO notify_state (id) VALUES (TRUE) ON CONFLICT (id) DO NOTHING; - -ALTER TABLE notify_state ENABLE ROW LEVEL SECURITY; --- No policy: only the cron service-role key touches this table, and the service --- role bypasses RLS. -``` - -The `CHECK (id)` plus `id` being the primary key guarantees the table can hold at -most one row (`id = TRUE`). - -### 2. `supabase/schema.sql` — two RPCs - -Both `SECURITY DEFINER, SET search_path = public`, matching `register_device`. -The occurrence math is written once and reused. - -**`peek_due_notifications() → boolean`** (read-only; called by the guard): - -```sql -CREATE OR REPLACE FUNCTION peek_due_notifications() -RETURNS boolean -LANGUAGE sql -SECURITY DEFINER -SET search_path = public -AS $$ - SELECT EXISTS ( - SELECT 1 - FROM devices d, notify_state s - WHERE d.notify_at IS NOT NULL - AND ( - (CASE - WHEN d.notify_at <= (now() AT TIME ZONE 'UTC')::time - THEN (now() AT TIME ZONE 'UTC')::date + d.notify_at - ELSE (now() AT TIME ZONE 'UTC')::date - 1 + d.notify_at - END) AT TIME ZONE 'UTC' - ) > s.last_run_at - ); -$$; -``` - -**`claim_due_notifications() → setof (fcm_token text)`** (called by the job; -returns due tokens **and** advances `last_run_at` atomically): - -```sql -CREATE OR REPLACE FUNCTION claim_due_notifications() -RETURNS TABLE (fcm_token text) -LANGUAGE plpgsql -SECURITY DEFINER -SET search_path = public -AS $$ -DECLARE - v_now timestamptz := now(); - v_last timestamptz; -BEGIN - SELECT s.last_run_at INTO v_last - FROM notify_state s - WHERE s.id - FOR UPDATE; - - RETURN QUERY - SELECT d.fcm_token - FROM devices d - WHERE d.notify_at IS NOT NULL - AND ( - (CASE - WHEN d.notify_at <= (v_now AT TIME ZONE 'UTC')::time - THEN (v_now AT TIME ZONE 'UTC')::date + d.notify_at - ELSE (v_now AT TIME ZONE 'UTC')::date - 1 + d.notify_at - END) AT TIME ZONE 'UTC' - ) > v_last; - - UPDATE notify_state SET last_run_at = v_now WHERE id; -END; -$$; -``` - -`FOR UPDATE` plus the workflow's `concurrency: { group: notify, -cancel-in-progress: false }` serialise runs, so two claims can never overlap. - -**`last_run_at` advancement semantics:** - -- The guard returning _empty_ → job is **skipped** → `claim` is **not** called → - `last_run_at` does **not** advance → the window grows until something is due. - This is the catch-up mechanism. -- The guard returning _due_ (or failing open) → job runs → `claim` advances - `last_run_at` to `v_now`. A spurious fail-open run on a truly empty window - advances `last_run` but loses nothing, because nobody was due before `v_now`. - -### 3. `cron/jobs/notify.ts` — shrinks to a thin RPC call - -Removes the `notifyWindow` import, the window log line, and the `.from('devices')` -query. New shape (preserving the existing logger, error handling, and -`process.exit` pattern): - -```ts -import { loadPulseConfig } from '../src/config'; -import { buildClient, dispatchFcm } from '../src/notify'; -import { getLogger } from '../src/logging'; - -async function main(): Promise { - loadPulseConfig(); - const log = getLogger('notify-cron'); - - try { - const db = buildClient(); - - const { data, error } = await db.rpc('claim_due_notifications'); - if (error) { - throw new Error(`Failed to claim due notifications: ${error.message}`); - } - - const tokens = (data ?? []).map((d) => d.fcm_token as string); - if (!tokens.length) { - log.info('No devices due'); - } else { - const { sent, total } = await dispatchFcm(tokens); - log.info(`Sent ${sent}/${total} notifications`); - } - } catch (err) { - log.error(`Unhandled error: ${String(err)}`); - process.exit(1); - } - - process.exit(0); -} - -void main(); -``` - -The doc-comment header is updated to describe the catch-up window model. - -### 4. `.github/workflows/notify.yml` — guard swaps query for RPC - -The guard `node -e` block now `POST`s to `/rest/v1/rpc/peek_due_notifications` -and sets `has=`, still **failing open** on any error: - -```js -const url = `${process.env.SUPABASE_URL}/rest/v1/rpc/peek_due_notifications`; -fetch(url, { - method: 'POST', - headers: { - apikey: process.env.SUPABASE_SECRET_KEY, - Authorization: `Bearer ${process.env.SUPABASE_SECRET_KEY}`, - 'Content-Type': 'application/json', - }, - body: '{}', -}) - .then((r) => (r.ok ? r.json() : Promise.reject(new Error(`HTTP ${r.status}`)))) - .then((due) => { - console.log(`due: ${due}`); - process.stdout.write(`has=${due === true}\n`); - }) - .catch((e) => { - console.log(`guard error (${e.message}) — failing open`); - process.stdout.write('has=true\n'); - }); -``` - -The `NOTIFY_WINDOW_START` / `NOTIFY_WINDOW_END` env vars are removed from the -`./.github/actions/run-cron-job` step. The schedule (`*/30 * * * *`), -`concurrency`, `timeout-minutes`, and the cost-floor comments are unchanged. The -guard step's purpose (skip checkout + npm on empty runs to stay at the 1-minute -billing floor) is preserved. - -`.github/workflows/daily-digest.yml` is **unchanged**. - -### 5. Deletions - -- `cron/src/lib/notifyWindow.ts` -- its test (`cron/src/tests/notifyWindow.test.ts`) -- all `NOTIFY_WINDOW_*` references - -## Data flow - -``` -GHA schedule fires (irregularly) - │ - ▼ -guard: POST peek_due_notifications() - │ false → skip job, last_run NOT advanced, window grows - │ true / error(fail-open) ↓ - ▼ -checkout + npm + run jobs/notify.ts - │ - ▼ -claim_due_notifications() ── FOR UPDATE on notify_state - │ returns due fcm_tokens - │ advances last_run_at = now() (atomic, before dispatch) - ▼ -dispatchFcm(tokens) → log sent/total -``` - -## Error handling - -- **Guard error** → fail open (`has=true`), job runs; a spurious run on an empty - window advances `last_run` harmlessly. -- **`claim` RPC error** → `notify.ts` throws, logs, `process.exit(1)`; - `last_run_at` not advanced (the `UPDATE` is in the same failed transaction); - next run retries the same window. -- **Dispatch error** → `dispatchFcm` tolerates per-token failures and reports - `sent/total`. A total FCM outage drops at most one window (at-most-once, by - design); affected devices' next occurrence is tomorrow. -- **Empty result** → log "No devices due", exit 0. - -## Testing (option a) - -The date logic lives in SQL, which the repo cannot unit-test (Jest, no Postgres -in CI; convention is "integration needing live keys → `e2e/` manual runners"). - -1. **Unit tests** for the new thin `notify.ts` with a mocked Supabase client: - - claims and dispatches the returned tokens, - - empty result → no dispatch, logs "No devices due", - - `rpc` error → throws / exits non-zero. - Mirror existing `cron` test patterns and mocking style. - -2. **`e2e/` runner** that seeds `notify_state` + `devices` against the real - Supabase project and asserts `peek` / `claim` for the four key cases: - normal cadence, midnight wrap, >24h outage, boundary. Run manually (needs the - live service key), per repo strategy. - -No pure-TS oracle of the SQL rule (rejected option b) — the SQL is the single -source of truth. - -## Deployment - -`supabase/schema.sql` is the canonical reference; the new table and two functions -must also be applied to the live Supabase project (via `apply_migration` / the -Supabase dashboard) before the updated workflow runs. The `notify_state` seed -row must exist (seeded with `now()`) so the first post-deploy run does not -mass-notify. - -## Out of scope - -- `daily-digest.ts` and the `notify_at = NULL` digest-time path. -- Any app / React Native changes (`notify_at` is still owned by the app's - `updateNotifyTime`; no new columns). -- Per-device delivery tracking. - -``` - -``` diff --git a/package.json b/package.json index bf4f293..0c16496 100644 --- a/package.json +++ b/package.json @@ -19,7 +19,7 @@ "lint-staged": { "cron/**/*.ts": [ "prettier --write", - "bash -c 'cd cron && npx eslint --ext .ts src api jobs'", + "bash -c 'cd cron && npx eslint --ext .ts src jobs e2e'", "bash -c 'npx tsc --noEmit --project cron/tsconfig.json'" ], "app/**/*.{ts,tsx}": [ diff --git a/supabase/README.md b/supabase/README.md index 2d30b3b..4b67d40 100644 --- a/supabase/README.md +++ b/supabase/README.md @@ -1,32 +1,34 @@ # Pulse — Supabase +Schema reference only — the single source of truth is `schema.sql`. No application code lives here. + ## Apply schema -Run `schema.sql` via the Supabase dashboard SQL editor or `apply_migration` MCP tool. Safe to re-apply (`IF NOT EXISTS`). +Run `schema.sql` via the Supabase dashboard SQL editor or the `apply_migration` MCP tool. Safe to re-apply: tables use `CREATE TABLE IF NOT EXISTS`, RPCs use `CREATE OR REPLACE`, policies are `DROP ... IF EXISTS` before recreate. --- ## Tables -**`devices`** — one row per app install. UUID generated by the app on first launch; stable across FCM token rotations, resets on reinstall. +**`devices`** — one row per app install. UUID generated by the app on first launch; stable across FCM token rotations, resets on reinstall. All client writes go through SECURITY DEFINER RPCs (see below); the table carries no open RLS policy. | Column | Type | Notes | | ------------ | -------------------- | ----------------------------------------------------------------------------- | | `id` | UUID PK | App-generated | -| `fcm_token` | TEXT | Updated via `onTokenRefresh` | -| `notify_at` | TIME | Nullable — null = notify at digest-ready time; set = notify at preferred time | -| `user_id` | UUID FK → auth.users | Stamped after login; nullable | +| `fcm_token` | TEXT NOT NULL | UNIQUE (`devices_fcm_token_key`). Updated via `onTokenRefresh` | +| `notify_at` | TIME | Nullable — null = notify at cron time (MVP default); set = preferred UTC time | +| `user_id` | UUID FK → auth.users | Stamped after login; nullable. `ON DELETE SET NULL` | | `updated_at` | TIMESTAMPTZ | | **`user_preferences`** — one row per authenticated user. -| Column | Type | Notes | -| ------------- | -------------------- | ----------------------------- | -| `user_id` | UUID PK → auth.users | RLS: `auth.uid() = user_id` | -| `preferences` | JSONB | Full `UserPreferences` object | -| `updated_at` | TIMESTAMPTZ | | +| Column | Type | Notes | +| ------------- | -------------------- | ------------------------------------------------ | +| `user_id` | UUID PK → auth.users | RLS: `auth.uid() = user_id`. `ON DELETE CASCADE` | +| `preferences` | JSONB | Full `UserPreferences` object. Default `'{}'` | +| `updated_at` | TIMESTAMPTZ | | -**`digests`** — one row per region per date, shared across all users. Cron upserts on `(region, date)` — safe to re-run. Rows older than `evictDBDays` are deleted before each daily write. +**`digests`** — one row per region per date, shared across all users. Cron upserts on `(region, date)` — safe to re-run. Rows older than 7 days are evicted before each daily write. PK `(region, date)` covers the main app query (`WHERE region = ANY($1) AND date = $2`); max ~56 rows (8 regions × 7 days), no extra index. | Column | Type | Notes | | ------------ | ----------- | --------------------------- | @@ -35,14 +37,54 @@ Run `schema.sql` via the Supabase dashboard SQL editor or `apply_migration` MCP | `payload` | JSONB | `{ headlines: Headline[] }` | | `created_at` | TIMESTAMPTZ | | +**`notify_state`** — single-row store of the last window processed by `jobs/notify.ts`. The job notifies devices whose `notify_at` fell in `(last_run_at, now]`, so no device is dropped when the GitHub Actions schedule fires irregularly. Seeded with `now()` so the first run after deploy sees a tiny window (no mass-notify). Cron service-role only. + +| Column | Type | Notes | +| ------------- | ----------- | -------------------------------------------------- | +| `id` | BOOLEAN PK | `DEFAULT TRUE CHECK (id)` — forces exactly one row | +| `last_run_at` | TIMESTAMPTZ | `DEFAULT now()` | + +**`pipeline_runs`** — one row per daily-digest run; durable observability/cost record written by `jobs/daily-digest.ts` (see `cron/src/runReport.ts`, RunReport `pulse.run.v1`). Promoted scalar columns make cost-over-time queries cheap; the full structured report lives in `report`. Cron service-role only (no client read path). + +| Column | Type | Notes | +| ------------------- | ---------------------- | ---------------------------------------------- | +| `id` | BIGINT PK | `GENERATED ALWAYS AS IDENTITY` | +| `run_at` | TIMESTAMPTZ NOT NULL | Indexed `DESC` (`pipeline_runs_run_at_idx`) | +| `status` | TEXT NOT NULL | | +| `total_cost_usd` | NUMERIC(10,6) NOT NULL | | +| `total_tokens` | INTEGER NOT NULL | | +| `regions_succeeded` | SMALLINT NOT NULL | | +| `regions_failed` | SMALLINT NOT NULL | | +| `duration_ms` | INTEGER | Nullable | +| `report` | JSONB NOT NULL | Full per-region quality / cost / notify report | +| `created_at` | TIMESTAMPTZ | `DEFAULT now()` | + --- ## RLS -Two-key model: publishable key in the app, secret key server-side only. +Two-key model: publishable key in the app, secret (service-role) key server-side only. The service role bypasses RLS, so cron-only tables define no policy and stay private. + +| Table | Rule | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------- | +| `digests` | SELECT open to all (`public read`, publishable key). Writes via secret key (bypasses RLS). | +| `devices` | No open policy. RLS denies every direct anon/authenticated SELECT/INSERT/UPDATE. All client access via the RPCs below. | +| `user_preferences` | SELECT/INSERT/UPDATE restricted to `auth.uid() = user_id` (app publishable key). | +| `notify_state` | No policy — cron service-role only. | +| `pipeline_runs` | No policy — cron service-role only. | + +--- + +## RPCs (SECURITY DEFINER) + +All client writes to `devices` go through these instead of direct table access, which lets the table stay fully RLS-locked. + +| Function | Grant | Purpose | +| -------------------------------------- | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | +| `register_device(id, token, user_id?)` | anon, authenticated, service_role | Canonical registration write (app + cron). Upserts by stable `id`; evicts any other row holding the same token to collapse reinstall ghosts. | +| `update_notify_time(id, notify_at)` | anon, authenticated, service_role | Only path for the app to change a device's `notify_at`. Returns `FOUND`. | +| `link_device_to_user(id)` | authenticated, service_role | Stamps `auth.uid()` (not a parameter) onto a device row — caller can only link to self. | +| `peek_due_notifications()` | service_role | Read-only "is any device due since `last_run_at`?". Used by the notify workflow guard. | +| `claim_due_notifications()` | service_role | Returns due FCM tokens AND advances `last_run_at` atomically (`FOR UPDATE`, at-most-once). | -| Table | Rule | -| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `digests` | SELECT open to all (publishable key). Writes via secret key (bypasses RLS). | -| `devices` | SELECT/INSERT/UPDATE open to publishable key (MVP). SELECT required by PostgREST for upsert RETURNING. See TODO.md V2 for Edge Function upgrade that locks this down. | -| `user_preferences` | SELECT/INSERT/UPDATE restricted to `auth.uid() = user_id` (app publishable key). | +**Gotcha — anon auto-grant.** Supabase's `ALTER DEFAULT PRIVILEGES` auto-grants `EXECUTE` on every new public function directly to `anon` (and `authenticated`/`service_role`), so `REVOKE ... FROM PUBLIC` alone leaves `anon` able to call it. Service-only functions (`peek_due_notifications`, `claim_due_notifications`) and `link_device_to_user` therefore revoke from `anon` explicitly: `REVOKE EXECUTE ... FROM PUBLIC, anon[, authenticated]`. diff --git a/supabase/schema.sql b/supabase/schema.sql index 8d88a4b..99f26fc 100644 --- a/supabase/schema.sql +++ b/supabase/schema.sql @@ -263,3 +263,32 @@ $$; -- never be reachable with the public anon key. Revoke the default PUBLIC grant. REVOKE EXECUTE ON FUNCTION claim_due_notifications() FROM PUBLIC, anon, authenticated; GRANT EXECUTE ON FUNCTION claim_due_notifications() TO service_role; + +-- ============================================================ +-- pipeline_runs +-- One row per daily-digest run — durable observability/cost record written by +-- jobs/daily-digest.ts (see cron/src/runReport.ts, RunReport `pulse.run.v1`). +-- Promoted scalar columns (run_at, status, total_cost_usd, total_tokens, +-- regions_*) make cost-over-time queries cheap; the full structured report +-- (per-region quality, cost breakdown, notify outcome) lives in `report` jsonb. +-- SELECT run_at, status, total_cost_usd, total_tokens +-- FROM pipeline_runs ORDER BY run_at DESC LIMIT 30; +-- Only the cron service-role key writes/reads this; the service role bypasses +-- RLS, so no policy is defined and the table stays private (no client read path). +-- ============================================================ +CREATE TABLE IF NOT EXISTS pipeline_runs ( + id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + run_at TIMESTAMPTZ NOT NULL, + status TEXT NOT NULL, + total_cost_usd NUMERIC(10,6) NOT NULL, + total_tokens INTEGER NOT NULL, + regions_succeeded SMALLINT NOT NULL, + regions_failed SMALLINT NOT NULL, + duration_ms INTEGER, + report JSONB NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +CREATE INDEX IF NOT EXISTS pipeline_runs_run_at_idx ON pipeline_runs (run_at DESC); + +ALTER TABLE pipeline_runs ENABLE ROW LEVEL SECURITY; diff --git a/todo.md b/todo.md index 32c138f..9eb7b86 100644 --- a/todo.md +++ b/todo.md @@ -2,16 +2,6 @@ ## V1 -**Phase 1 — Harden what's deployed** - -- [ ] Tighten `devices` table RLS — replace `USING (true)` / `WITH CHECK (true)` with `user_id = auth.uid()`, or make `/api/account` the only writer and lock the table down -- _Open Q: do clients ever write `devices` directly, or can the route be the sole writer?_ - -**Phase 2 — Observability: cost logging refactor** - -- [ ] Keep human-readable log lines, but have each cron run emit one structured JSON object of all logged/cost data, shaped for HTTP transmission (~80% there; needs a consistent schema + cleanup) -- _Open Q: where does the JSON go — route response, Supabase table, or external sink? That drives the schema._ - **Phase 3 — Image durability (server-side proxy/cache)** Client caching (expo-image) is already solved; this is purely source-URL rot.