Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 19 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,32 +24,36 @@ Reading the news is either too time-consuming or too algorithmically poisoned. M
## Architecture

```
Vercel Cron — /api/daily-digest (05:00 UTC daily)
GitHub Actions — daily-digest.yml (06:00 UTC daily)
→ Perplexity Sonar API × N regions (parallel)
→ format into digest JSON
→ Claude ranks headlines per region (+ cross-region global digest)
→ store in Supabase
→ Firebase Cloud Messaging ping (null-notify_at devices)

Vercel Cron — /api/notify (every 30 min)
→ FCM ping to devices whose notify_at matches the current window
GitHub Actions — notify.yml (every 30 min)
→ FCM ping to devices whose notify_at falls in (last_run_at, now] — a
catch-up window so no scheduled device is dropped when the cron fires late
```

| Component | Role |
| ------------------- | ------------------------------------------------------------------------------------------ |
| Perplexity Sonar | News source — one call per region; handles sourcing, recency, deduplication, summarization |
| Supabase | Authoritative store for digests and device tokens |
| Claude (Anthropic) | Ranks headlines by importance per region, and selects cross-region global top stories |
| Supabase | Authoritative store for digests, device tokens, and cron run reports |
| Firebase (FCM) | Lightweight "digest ready" ping; full content fetched from Supabase on tap |
| React Native (Expo) | Android-first app; local AsyncStorage cache + Supabase for recovery |
| GitHub Actions | Runs the two cron jobs (digest, notify) and CI |

---

## Subsystems

| Directory | README |
| ----------- | ---------------------------------------------------------------------- |
| `cron/` | [cron/README.md](cron/README.md) — pipeline, config, how to run |
| `app/` | [app/README.md](app/README.md) — Expo setup, feature phases, data flow |
| `supabase/` | [supabase/README.md](supabase/README.md) — schema, RLS, V2 additions |
| Directory | README |
| ----------- | ----------------------------------------------------------------------------- |
| `cron/` | [cron/README.md](cron/README.md) — pipeline, config, run-report observability |
| `app/` | [app/README.md](app/README.md) — Expo setup, navigation, data flow |
| `supabase/` | [supabase/README.md](supabase/README.md) — schema, RLS, definer RPCs |
| `shared/` | [shared/CLAUDE.md](shared/CLAUDE.md) — region constants, config, shared types |

---

Expand All @@ -59,11 +63,14 @@ Vercel Cron — /api/notify (every 30 min)
| ---------------- | ------------------------------------------------------------------------------------------------------- |
| Platform | Android first |
| News source | Perplexity Sonar API — one call per region, parallel |
| Ranking | Claude reorders headlines per region by importance, and selects a cross-region global digest |
| Scheduling | GitHub Actions cron (best-effort); notify uses a `(last_run_at, now]` catch-up window for late fires |
| Push strategy | Cron writes to Supabase, then FCM ping; app fetches on tap |
| Digest storage | One row per region per date, shared across all users |
| Device identity | UUID generated on first launch; resets on reinstall |
| Auth | Supabase Auth — email/password; `user_preferences` keyed on `auth.uid()` (Phase 4, done) |
| Device writes | Publishable key + open INSERT/UPDATE RLS (MVP); V2: Edge Function with secret key |
| Auth | Supabase Auth — email/password; `user_preferences` keyed on `auth.uid()` |
| Device writes | RLS-locked `devices` table; all client access via SECURITY DEFINER RPCs (no open anon writes) |
| Observability | Each pipeline run persists a report row to `pipeline_runs` (cost, tokens, per-region success/failure) |
| Source diversity | Prompt-driven (preferred outlets hint per region); domain allow-lists rejected — caused empty results |
| URL quality | Pattern filter on model output; HTTP HEAD validation rejected — news sites return 403 for section pages |
| Bias control | Prompt-driven neutrality; `user_location.country` steers regional sourcing |
74 changes: 20 additions & 54 deletions app/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,75 +26,33 @@ npm install

Copy `.env.example` to `.env` inside `app/` and fill in:

| Variable | Description |
| ------------------------------- | ------------------------------- |
| `EXPO_PUBLIC_SUPABASE_URL` | Supabase project URL |
| `EXPO_PUBLIC_SUPABASE_ANON_KEY` | Supabase anon (publishable) key |
| Variable | Description |
| -------------------------- | ------------------------------- |
| `EXPO_PUBLIC_SUPABASE_URL` | Supabase project URL |
| `EXPO_PUBLIC_SUPABASE_KEY` | Supabase publishable (anon) key |

FCM config lives in `app/android/app/google-services.json`, not env vars.

---

## Android dev setup on WSL2 (one-time)

The Android SDK lives on Windows; WSL2 needs to reach it.

**1. Add to `~/.bashrc`:**

```bash
export ANDROID_HOME=/mnt/c/Users/HP/AppData/Local/Android/Sdk
export JAVA_HOME="/mnt/c/Program Files/Android/Android Studio/jbr"
export PATH=$PATH:$ANDROID_HOME/platform-tools:$ANDROID_HOME/tools/bin:$ANDROID_HOME/emulator:$JAVA_HOME/bin
```

**2. Create an `adb` wrapper** so WSL2 uses the Windows ADB server (which can reach your phone):

```bash
mkdir -p ~/.local/bin
cat > ~/.local/bin/adb << 'EOF'
#!/bin/bash
exec /mnt/c/Users/HP/AppData/Local/Android/Sdk/platform-tools/adb.exe "$@"
EOF
chmod +x ~/.local/bin/adb
```

**3. Pair your phone (one-time per device):**

- Phone: Settings → Developer Options → Wireless debugging → **Pair device with pairing code**
- Note the pairing port and 6-digit code
- In Windows CMD (not WSL2):

```cmd
adb pair 192.168.64.4:<pairing-port>
```

Enter the code. You'll see `Successfully paired`.
FCM config lives in `app/google-services.json` (referenced from `app.json` as `./google-services.json`), not env vars. The file is gitignored — never commit it.

---

## Connecting your phone (every session)

Your phone's connection port changes each session. Run the helper script:
Wireless debugging port changes each session. Use the helper script:

```bash
./scripts/adb-connect.sh
```

Or manually — check the port in Settings → Developer Options → Wireless debugging, then in Windows CMD:

```cmd
adb connect 192.168.64.4:<connection-port>
```

Verify WSL2 sees it:
Or manually — check the port in Settings → Developer Options → Wireless debugging, then:

```bash
adb devices # should list 192.168.64.4:<port> device
adb connect <phone-ip>:<connection-port>
adb devices # should list <phone-ip>:<port> device
```

---

## Running (added by app/foundation slice)
## Running

This app uses native modules (`@react-native-firebase/messaging`), so **Expo Go will not work** — a custom dev client is required.

Expand All @@ -112,16 +70,24 @@ npx expo start --dev-client # subsequent runs — scan QR in the installed d
```bash
npm run build # tsc --noEmit (typecheck)
npm run lint # ESLint on src/
npm test # Jest (available after app/foundation slice)
npm test # Jest
```

---

## Navigation

No React Navigation. The root `App.tsx` conditionally renders screens based on the Zustand `nav` slice (`splash` / `digest` / `settings` / `login`), gated by an `appState` machine (`booting` → `prefs-loading` → `ready`, plus `maintenance` and `update-required` stub screens). Nav state (screen + day index) persists to MMKV with a TTL; the open article is transient.

---

## Data flow

**Digests:** cache-first (MMKV) with a configurable stale window for today. Notification tap forces a full remote fetch. Past dates are immutable — never re-fetched.

**Auth:** `useSupabaseAuth` manages session via Supabase. MMKV persists the session across restarts. Password-reset deep links (`pulse://reset-password`) use PKCE or implicit token flow depending on what Supabase sends.
**Auth:** `useSupabaseAuth` manages session via Supabase. MMKV persists the session across restarts.

**Deep links:** handled in `useDeepLinkRecovery` via `expo-linking` — both cold-start (`Linking.getInitialURL()`) and warm (`addEventListener('url', …)`). Two schemes are recognized: `pulse://reset-password` (password recovery → routes to the reset screen) and `pulse://confirm` (sign-up confirmation → establishes the session, which routes into the app). Each is exchanged via PKCE (`exchangeCodeForSession`) or implicit token flow (`setSession`) depending on what Supabase sends. Duplicate URLs are de-duped.

**Preferences:** keyed on `session.user.id`. Local writes are immediate. Supabase push is batched — flushed on settings close and on app background.

Expand Down
29 changes: 23 additions & 6 deletions cron/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Pulse — Cron Pipeline

Daily digest pipeline: fetches regional headlines from Perplexity, stores results in Supabase, and fires FCM push notifications to registered devices.
Daily digest pipeline: fetches regional headlines from Perplexity, ranks them with Claude, stores results in Supabase, and fires FCM push notifications to registered devices. Runs as GitHub Actions cron jobs (`.github/workflows/daily-digest.yml`, `.github/workflows/notify.yml`).

---

Expand Down Expand Up @@ -47,6 +47,7 @@ npm run e2e:countryRanking -- US GB DE # per-region rank on selected region
| Variable | Description |
| ----------------------- | ----------------------------------------------------- |
| `PERPLEXITY_API_KEY` | Perplexity Sonar API key |
| `ANTHROPIC_API_KEY` | Claude API key (headline ranking) |
| `SUPABASE_URL` | Supabase project URL |
| `SUPABASE_SECRET_KEY` | Service-role key (bypasses RLS) |
| `FIREBASE_PROJECT_ID` | Firebase project ID |
Expand All @@ -70,17 +71,33 @@ loadPulseConfig() → createSource()
→ runFetchPipeline()
→ resolveRegions()
→ fetchDigest() × N [Promise.allSettled, staggered]
→ Perplexity retry loop
→ Perplexity retry loop [429/5xx, exponential backoff]
→ parseHeadlines() → URL filter + slug dedup + topic dedup
→ rankHeadlines() [Claude — per-region reorder]
→ deduplicateAcrossDigests() [null out images shared across regions]
→ persistDigests() [upsert to Supabase]
→ rankGlobalHeadlines() [Claude — cross-region top stories]
→ rankGlobalHeadlines() [Claude — cross-region top stories, if enabled]
→ persistGlobalDigest()
→ sendNotifications() [FCM multicast]
→ writeRunLog()
→ dispatchFcm() [FCM multicast]
→ buildRunReport() → formatRunReportSummary() (log line) → persistRunReport() [→ pipeline_runs]
→ writeRunLog() [JSONL time-series, only if log.qualityLog]
```

`api/daily-digest.ts` follows the same path but only FCM-pings devices with `notify_at = NULL`. `api/notify.ts` handles devices with a specific scheduled time.
The two cron jobs share `runFetchPipeline`:

- `jobs/daily-digest.ts` runs the full pipeline above, then FCM-pings devices with `notify_at = NULL` ("notify me when the digest is ready").
- `jobs/notify.ts` runs every ~30 min and notifies devices whose `notify_at` falls in `(last_run_at, now]` — a catch-up window so no scheduled device is dropped when the cron fires irregularly.

---

## Run-report observability

Every `daily-digest` run assembles a versioned `RunReport` (`schema: pulse.run.v1`) — the durable, queryable companion to the human-readable log lines. It aggregates region success/failure, headline counts, fetch + ranking token cost, notify outcome, and duration.

- A one-line summary (`formatRunReportSummary`) is logged via Winston.
- The full report is best-effort persisted to `public.pipeline_runs` (`persistRunReport`): columns `run_at`, `status` (`ok`/`partial`), `total_cost_usd`, `total_tokens`, `regions_succeeded`, `regions_failed`, `duration_ms`, plus the whole `report` as JSON. Persistence failures are logged and swallowed — observability never breaks the digest pipeline.

Bump `schema` on any breaking shape change so downstream queries can branch on the version.

---

Expand Down
2 changes: 1 addition & 1 deletion cron/e2e/fetch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ async function main() {
printTotals(retryStats, resolvedRegions.length, fetchUsage, rankingUsage, config.api.fetch.count);

if (config.api.ranking.global.enabled) {
const globalHeadlines = await rankGlobalHeadlines(digests, config);
const { headlines: globalHeadlines } = await rankGlobalHeadlines(digests, config);
printGlobalHeadlines(globalHeadlines);
}

Expand Down
2 changes: 1 addition & 1 deletion cron/e2e/full.ts
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ async function main() {
await persistDigests(digests, config);

if (config.api.ranking.global.enabled) {
const globalHeadlines = await rankGlobalHeadlines(digests, config);
const { headlines: globalHeadlines } = await rankGlobalHeadlines(digests, config);
if (globalHeadlines.length > 0) {
await persistGlobalDigest(globalHeadlines);
}
Expand Down
2 changes: 1 addition & 1 deletion cron/e2e/globalRanking.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ async function main() {
digests.forEach((d) => log.info(` ${d.region.padEnd(20)} ${d.headlines.length} headlines`));

log.info('\nRunning global ranking…');
const globalHeadlines = await rankGlobalHeadlines(digests, config);
const { headlines: globalHeadlines } = await rankGlobalHeadlines(digests, config);
printGlobalHeadlines(globalHeadlines);
}

Expand Down
43 changes: 33 additions & 10 deletions cron/jobs/daily-digest.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,20 @@
import { loadPulseConfig, createSource } from '../src/config';
import { persistDigests, persistGlobalDigest, buildClient, dispatchFcm } from '../src/notify';
import { rankGlobalHeadlines } from '../src/rankHeadlines';
import { getLogger } from '../src/logging';
import { getLogger, formatError } from '../src/logging';
import type { DigestUsage } from '../src/types';
import {
buildRunLog,
deduplicateAcrossDigests,
runFetchPipeline,
writeRunLog,
} from '../src/pipeline';
import {
buildRunReport,
formatRunReportSummary,
persistRunReport,
type NotifyOutcome,
} from '../src/runReport';

/**
* GitHub Actions cron job (.github/workflows/daily-digest.yml) — fetch all
Expand All @@ -25,6 +32,8 @@ async function main(): Promise<void> {
const source = createSource(config);

try {
log.info(`Starting daily digest — ${config.api.regions.length} region(s) configured`);

const {
resolvedRegions,
digests: fetchedDigests,
Expand All @@ -34,7 +43,7 @@ async function main(): Promise<void> {
const digests = deduplicateAcrossDigests(fetchedDigests);

errors.forEach((error) =>
log.error(`Region fetch failed: ${error.region}: ${String(error.reason)}`),
log.error(`Region fetch failed: ${error.region}: ${formatError(error.reason)}`),
);

if (digests.length === 0) {
Expand All @@ -43,8 +52,10 @@ async function main(): Promise<void> {

await persistDigests(digests, config);

let globalRankingUsage: DigestUsage | null = null;
if (config.api.ranking.global.enabled) {
const globalHeadlines = await rankGlobalHeadlines(digests, config);
const { headlines: globalHeadlines, usage } = await rankGlobalHeadlines(digests, config);
globalRankingUsage = usage;

if (globalHeadlines.length > 0) {
await persistGlobalDigest(globalHeadlines);
Expand All @@ -58,28 +69,40 @@ async function main(): Promise<void> {
.select('fcm_token')
.is('notify_at', null);

let notify: NotifyOutcome = { devicesTargeted: 0, sent: 0, failed: 0 };
if (error) {
log.warn(`Failed to read null-notify_at devices: ${error.message}`);
} else if (devices?.length) {
const tokens = devices.map((d) => d.fcm_token as string);
const regions = digests.map((d) => d.region).join(',');

await dispatchFcm(tokens, regions);
const result = await dispatchFcm(tokens, regions);
notify = {
devicesTargeted: result.total,
sent: result.sent,
failed: result.total - result.sent,
};
}

const totalTokens = digests.reduce((sum, d) => sum + (d.usage?.totalTokens ?? 0), 0);

log.info(`Done — ${digests.length}/${resolvedRegions.length} regions, ${totalTokens} tokens`);
const report = buildRunReport({
config,
resolvedRegions,
digests,
errors,
startTime,
globalRankingUsage,
notify,
});
log.info(formatRunReportSummary(report));
await persistRunReport(report);

if (config.log.qualityLog) {
const runLog = buildRunLog(config, resolvedRegions, digests, errors, startTime);

const logPath = writeRunLog(runLog, resolvedRegions);

log.info(`Quality log → ${logPath}`);
}
} catch (err) {
log.error(`Unhandled error: ${String(err)}`);
log.error(`Unhandled error: ${formatError(err)}`);
process.exit(1);
}

Expand Down
5 changes: 3 additions & 2 deletions cron/jobs/notify.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { loadPulseConfig } from '../src/config';
import { sendDueNotifications } from '../src/notify';
import { getLogger } from '../src/logging';
import { getLogger, formatError } from '../src/logging';

/**
* GitHub Actions cron job (.github/workflows/notify.yml) — notify devices whose
Expand All @@ -17,10 +17,11 @@ async function main(): Promise<void> {
const log = getLogger('notify-cron');

try {
log.info('Starting due-notification run');
const { sent, total } = await sendDueNotifications();
log.info(`Sent ${sent}/${total} notifications`);
} catch (err) {
log.error(`Unhandled error: ${String(err)}`);
log.error(`Unhandled error: ${formatError(err)}`);
process.exit(1);
}

Expand Down
2 changes: 1 addition & 1 deletion cron/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"version": "0.1.0",
"scripts": {
"build": "tsc --noEmit",
"lint": "eslint --ext .ts src api jobs",
"lint": "eslint --ext .ts src jobs e2e",
"test": "jest",
"test:watch": "jest --watch",
"test:coverage": "jest --coverage",
Expand Down
2 changes: 1 addition & 1 deletion cron/src/lib/parseHeadlines.ts
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ export async function parseHeadlines(
if (seenInBatch.has(headline.url)) return false;
const slug = urlSlug(headline.url);
if (slug && usedSlugs.has(slug)) {
log.info(`Slug duplicate filtered: ${headline.url}`);
log.debug(`Slug duplicate filtered: ${headline.url}`);
return false;
}
seenInBatch.add(headline.url);
Expand Down
Loading
Loading