Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ SLACK_CHANNEL_ID=
DISCOURSE_BASE_URL=https://forum.example.org
DISCOURSE_API_KEY=your_api_key_here
DISCOURSE_API_USERNAME=system
# DISCOURSE_LOOKBACK_HOURS=24

# Gemini LLM
GEMINI_API_KEY=
Expand All @@ -26,6 +25,9 @@ GEMINI_API_KEY=

# Digest Config
# DIGEST_WINDOW_HOURS=24
# No digest on Sat/Sun (UTC); Monday looks back DIGEST_WINDOW_HOURS + 48 instead.
# Set false to restore plain daily behaviour (weekend runs, no Monday extension).
# SKIP_WEEKEND=true

# Triage
# Comma-separated projects this org maintains; bug reports against them rate high.
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/daily-digest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ jobs:
MAX_SUMMARY_TOKENS: ${{ vars.MAX_SUMMARY_TOKENS }}
MAX_INPUT_CHARS_PER_GROUP: ${{ vars.MAX_INPUT_CHARS_PER_GROUP }}
DIGEST_WINDOW_HOURS: ${{ vars.DIGEST_WINDOW_HOURS }}
SKIP_WEEKEND: ${{ vars.SKIP_WEEKEND }}
MIN_MESSAGE_LENGTH: ${{ vars.MIN_MESSAGE_LENGTH }}
EXCLUDE_COMMANDS: ${{ vars.EXCLUDE_COMMANDS }}
EXCLUDE_LINK_ONLY: ${{ vars.EXCLUDE_LINK_ONLY }}
Expand Down
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,4 +72,5 @@ All config flows through `loadConfig()` in `src/config/index.ts` (Zod schema ove
Runs as a **GitHub Actions** workflow (`.github/workflows/daily-digest.yml`), **not** a hosted service. Key facts for debugging "why didn't my change take effect":
- The digest job gates on `if: github.ref == 'refs/heads/main'` — **only `main` runs in production.** A change merged elsewhere won't appear in the digest until it reaches `main`.
- The trigger is `workflow_dispatch` only (no in-workflow cron); a **GCP Cloud Scheduler** job calls the dispatch API daily (`scripts/setup_cloud_scheduler_dispatch.sh`). Feature branches PR directly into `main`.
- **The schedule is daily but the digest is weekday-only.** Scheduler still fires seven days a week; `getRunSchedule()` (`src/utils/time.ts`) makes the app no-op on Sat/Sun (UTC) and extends Monday's lookback by `WEEKEND_EXTRA_HOURS` (48) so the weekend rolls into it. A green weekend run with no Slack post is correct, not a failure. `SKIP_WEEKEND=false` reverts the whole policy — weekend runs happen *and* Monday loses its extension.
- Operational procedures (incident playbook, required secrets/variables, recovery) live in `docs/production-runbook.md`.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Synapse is an intelligent community digest bot designed to aggregate, summarize,

- **Multi-Source Ingestion**: Seamlessly ingests messages from **Discord** channels and **Discourse** forums.
- **Intelligent Summarization**: Powered by the **Vercel AI SDK**, Synapse uses advanced LLMs (like Google Gemini) to generate concise, context-aware summaries of conversations.
- **Slack Destination**: Delivers beautifully formatted daily digests directly to your **Slack** workspace.
- **Slack Destination**: Delivers beautifully formatted weekday digests directly to your **Slack** workspace — weekends are skipped and rolled into Monday's digest.
- **Extensible Design**: Built on a modular architecture, making it easily extensible to support additional sources (e.g., GitHub, Telegram) and destinations (e.g., Email, Notion).

## Architecture
Expand Down
43 changes: 40 additions & 3 deletions docs/production-runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,27 @@ Why this setup:
## 2) Trigger Strategy

Primary trigger:
- Cloud Scheduler invokes `workflow_dispatch` daily.
- Cloud Scheduler invokes `workflow_dispatch` daily, seven days a week.

Fallback trigger:
- Keep the existing `schedule` block in the workflow as a best-effort backup.
- The workflow is `workflow_dispatch` only — it has no `schedule` block. Cloud Scheduler is the sole automated trigger.
- Keep `workflow_dispatch` enabled for manual recovery.

## 2a) Weekday-Only Digests

Cloud Scheduler still fires every day, but **the application itself no-ops on Saturday and
Sunday (UTC)** and rolls the weekend into Monday:

- Sat/Sun: the run starts, logs `Weekend (UTC) — skipping digest run`, and exits 0 before
contacting Discord, Discourse, Gemini, or Slack. **A green run with no Slack post is the
expected outcome on those days, not an incident.**
- Mon: the lookback window is `DIGEST_WINDOW_HOURS + 48` (72h by default), covering
Fri 13:00 → Mon 13:00 UTC with no gap. Its Slack header shows a date range rather than a
single date.
- `SKIP_WEEKEND` (repo Variable, default `true`) controls the whole policy. Setting it
`false` restores plain daily behaviour — weekend runs happen *and* Monday loses its
extension. Use it to force a one-off weekend run, then set it back.

## 3) Prerequisites

- GCP project with Cloud Scheduler API enabled.
Expand Down Expand Up @@ -83,6 +98,7 @@ Required workflow variables (minimum):
- `GEMINI_MODEL`
- `MAX_SUMMARY_TOKENS`
- `DIGEST_WINDOW_HOURS`
- `SKIP_WEEKEND` (default `true`; see §2a)
- `MIN_MESSAGE_LENGTH`
- `EXCLUDE_COMMANDS`
- `EXCLUDE_LINK_ONLY`
Expand All @@ -93,13 +109,22 @@ Required workflow variables (minimum):
Check these once per day:
- Cloud Scheduler job execution status is successful.
- A corresponding successful run exists in the `Daily Digest` workflow.
- Slack digest arrived in the expected channel.
- Slack digest arrived in the expected channel — **Monday through Friday only.**

On Saturday and Sunday, expect a successful workflow run and **no Slack post** (see §2a).
Verify the run logged the weekend-skip line; a weekend run that posted a digest means
`SKIP_WEEKEND` has been left `false`.

## 7) Alerting Baseline

Set up two alerts:
- **Trigger failure alert**: Cloud Scheduler job failed.
- **Digest missing alert**: no successful `Daily Digest` run or no Slack post within expected window.
**Suppress the "no Slack post" half on Saturday and Sunday (UTC)** — it would otherwise fire
every weekend by design. The "no successful run" half still applies all seven days.

A missing **Monday** digest is the most costly failure mode: it drops three days of community
activity, not one. Escalate it accordingly and use the recovery procedure in §9.

Target response objective:
- Detect failure within 30 minutes of scheduled run.
Expand Down Expand Up @@ -150,6 +175,18 @@ If a daily run is missed:
3. Confirm Slack post delivery.
4. Document incident summary and corrective action.

A manual dispatch uses the same rolling window as a scheduled one, so recovering a *late* run
needs a wider window to reach back to the missed period.

If a **Monday** run is missed, it was carrying three days of content and a plain re-dispatch on
Tuesday would only look back 24h. The job holds no state between runs, so widen the window
manually:
1. Set repo Variable `DIGEST_WINDOW_HOURS` to `96` (Fri 13:00 → Tue 13:00).
2. Manually dispatch `Daily Digest`.
3. Confirm Slack post delivery, then **restore `DIGEST_WINDOW_HOURS` to `24`.**

Do not set `SKIP_WEEKEND=false` for this — that would also strip the Monday extension.

## 10) Security & Maintenance

- Rotate GitHub PAT on a fixed schedule.
Expand Down
23 changes: 20 additions & 3 deletions src/DigestPipeline.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import { Source, Destination, Processor } from "./core/interfaces";
import { NormalizedMessage, DigestContext } from "./core/types";
import { Config } from "./config";
import { logger } from "./utils/logger";
import { getUtcDailyWindowFrom } from "./utils/time";
import { getRunSchedule } from "./utils/time";
import { applyMessageFilters } from "./utils/filters";
import { buildDigestBlocks, formatDigest } from "./utils/format";
import { DigestItem } from "./core/schemas";
Expand Down Expand Up @@ -52,7 +52,24 @@ export class DigestPipeline {

async run() {
logger.info("Starting digest pipeline...");
const { start, end, dateTitle } = getUtcDailyWindowFrom(new Date());

const now = new Date();
const { skip, extraHours } = getRunSchedule(now, this.config.SKIP_WEEKEND);
if (skip) {
logger.info("Weekend (UTC) — skipping digest run; content rolls into Monday.");
return;
}

// The window described in the header is the window actually fetched:
// a rolling `now - windowHours`, not a calendar day. On Monday that
// spans the weekend, so the title carries a range instead of one date.
const windowHours = this.config.DIGEST_WINDOW_HOURS + extraHours;
const end = now;
const start = new Date(end.getTime() - windowHours * 60 * 60 * 1000);
Comment thread
alchemydc marked this conversation as resolved.
const isoDate = (d: Date) => d.toISOString().slice(0, 10);
const dateTitle = extraHours > 0
? `${isoDate(start)} → ${isoDate(end)}`
: isoDate(end);
const context: DigestContext = { start, end, dateTitle };

const allMessages: NormalizedMessage[] = [];
Expand All @@ -62,7 +79,7 @@ export class DigestPipeline {
if (source.isEnabled()) {
logger.info(`Fetching from source: ${source.name}`);
try {
const messages = await source.fetchMessages(this.config.DIGEST_WINDOW_HOURS);
const messages = await source.fetchMessages(windowHours);
logger.info(`Fetched ${messages.length} messages from ${source.name}`);
allMessages.push(...messages);
} catch (err: any) {
Expand Down
18 changes: 10 additions & 8 deletions src/config/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,14 @@ const ConfigSchema = z.object({
z.number().int().min(1).default(24)
),

// Suppress Saturday/Sunday runs and roll the weekend into Monday's digest
// (Monday looks back DIGEST_WINDOW_HOURS + 48). Setting this false restores
// plain daily behaviour: weekend runs happen and Monday gets no extension.
SKIP_WEEKEND: z.preprocess(
toBool,
z.boolean().default(true)
),

LOG_LEVEL: z.preprocess(toStr, z.string().default("info")),

MIN_MESSAGE_LENGTH: z.preprocess(
Expand Down Expand Up @@ -101,11 +109,6 @@ const ConfigSchema = z.object({
DISCOURSE_API_KEY: z.preprocess(toStr, z.string()).optional(),
DISCOURSE_API_USERNAME: z.preprocess(toStr, z.string()).optional(),

DISCOURSE_LOOKBACK_HOURS: z.preprocess(
toNum,
z.number().int().min(1)
).optional(),

DISCOURSE_MAX_TOPICS: z.preprocess(
toNum,
z.number().int().min(1)
Expand All @@ -127,6 +130,7 @@ export type Config = {
MAX_INPUT_CHARS_PER_GROUP: number;
DRY_RUN: boolean;
DIGEST_WINDOW_HOURS: number;
SKIP_WEEKEND: boolean;
LOG_LEVEL: string;
MIN_MESSAGE_LENGTH: number;
EXCLUDE_COMMANDS: boolean;
Expand All @@ -137,7 +141,6 @@ export type Config = {
DISCOURSE_BASE_URL?: string;
DISCOURSE_API_KEY?: string;
DISCOURSE_API_USERNAME?: string;
DISCOURSE_LOOKBACK_HOURS?: number;
DISCOURSE_MAX_TOPICS?: number;


Expand Down Expand Up @@ -178,7 +181,6 @@ export function loadConfig(): Config {
DISCOURSE_BASE_URL: discoBase,
DISCOURSE_API_KEY: raw.DISCOURSE_API_KEY,
DISCOURSE_API_USERNAME: raw.DISCOURSE_API_USERNAME,
DISCOURSE_LOOKBACK_HOURS: raw.DISCOURSE_LOOKBACK_HOURS,
DISCOURSE_MAX_TOPICS: raw.DISCOURSE_MAX_TOPICS,

// derived enablement
Expand All @@ -197,6 +199,7 @@ export function loadConfig(): Config {
geminiModel: config.GEMINI_MODEL,
dryRun: config.DRY_RUN,
digestWindowHours: config.DIGEST_WINDOW_HOURS,
skipWeekend: config.SKIP_WEEKEND,
maxSummaryTokens: config.MAX_SUMMARY_TOKENS,
maxInputCharsPerGroup: config.MAX_INPUT_CHARS_PER_GROUP,
logLevel: config.LOG_LEVEL,
Expand All @@ -217,7 +220,6 @@ export function loadConfig(): Config {
enabled: config.DISCOURSE_ENABLED,
baseUrl: config.DISCOURSE_BASE_URL ? new URL(config.DISCOURSE_BASE_URL).hostname : undefined,
maxTopics: config.DISCOURSE_MAX_TOPICS ?? null,
lookbackHours: config.DISCOURSE_LOOKBACK_HOURS ?? null,
},
secrets: {
GEMINI_API_KEY: mask(process.env.GEMINI_API_KEY || ""),
Expand Down
3 changes: 3 additions & 0 deletions src/core/interfaces.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ import { DigestItem } from "./schemas";
export interface Source {
name: string;
isEnabled(): boolean;
// Every source uses the same lookback, so the window advertised in the
// digest header is exactly the window fetched. Includes the Monday weekend
// catch-up when applicable.
fetchMessages(windowHours: number): Promise<NormalizedMessage[]>;
}

Expand Down
5 changes: 2 additions & 3 deletions src/services/discourse/DiscourseSource.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,10 @@ export class DiscourseSource implements Source {
if (!this.isEnabled()) return [];

const now = Date.now();
const lookHours = this.config.DISCOURSE_LOOKBACK_HOURS ?? windowHours;
const since = now - lookHours * 60 * 60 * 1000;
const since = now - windowHours * 60 * 60 * 1000;
const maxTopics = this.config.DISCOURSE_MAX_TOPICS ?? 50;

logger.debug(`Discourse fetch: since=${new Date(since).toISOString()} now=${new Date(now).toISOString()} lookHours=${lookHours}`);
logger.debug(`Discourse fetch: since=${new Date(since).toISOString()} now=${new Date(now).toISOString()} windowHours=${windowHours}`);

const categoryMap = await this.fetchCategories();
const messages: NormalizedMessage[] = [];
Expand Down
7 changes: 7 additions & 0 deletions src/services/llm/AiSdkProcessor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -253,6 +253,13 @@ export class AiSdkProcessor implements Processor {
break;
}
}
// Truncation drops the OLDEST messages. On a Monday roll-up that is
// Friday's content, so make it visible rather than silent.
if (out.length < messages.length) {
logger.warn(
`Truncated group: kept ${out.length}/${messages.length} messages (limit ${maxChars} chars)`
);
}
return out;
}
}
5 changes: 4 additions & 1 deletion src/utils/format.ts
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,10 @@ export function buildDigestBlocks(params: {
logger.debug("[DEBUG] buildDigestBlocks items count:", params.items.length);
}

const range = `Time window: ${params.dateTitle} 00:00–${params.end.toISOString().slice(0, 10)} 00:00 UTC`;
// Describe the window actually fetched (a rolling span ending now), not a
// calendar day — on Mondays this spans the weekend.
const fmtUtc = (d: Date) => d.toISOString().slice(0, 16).replace("T", " ");
const range = `Time window: ${fmtUtc(params.start)}–${fmtUtc(params.end)} UTC`;

const legend = "🔴 urgent · 🟡 notable · ⚪ routine";

Expand Down
34 changes: 20 additions & 14 deletions src/utils/time.ts
Original file line number Diff line number Diff line change
@@ -1,17 +1,23 @@
// utils/time.ts
export function getDigestWindow(hours: number): { start: Date; end: Date } {
const end = new Date();
const start = new Date(end.getTime() - hours * 60 * 60 * 1000);
return { start, end };
}

// Returns UTC daily window (midnight to midnight) for a given UTC date
export function getUtcDailyWindowFrom(candidateUtc: Date): { start: Date; end: Date; dateTitle: string } {
const y = candidateUtc.getUTCFullYear();
const m = candidateUtc.getUTCMonth();
const d = candidateUtc.getUTCDate();
const start = new Date(Date.UTC(y, m, d, 0, 0, 0));
const end = new Date(start.getTime() + 24 * 60 * 60 * 1000);
const dateTitle = start.toISOString().slice(0, 10);
return { start, end, dateTitle };
// Sat + Sun: the two runs the weekend skip suppresses. Monday's window is
// extended by exactly this much so the skipped days are covered with no gap.
export const WEEKEND_EXTRA_HOURS = 48;

// Decides whether this run happens at all, and how far back it looks.
// The weekday is read in UTC so the decision never depends on the host's local
// timezone. "Weekend" therefore means the UTC weekend, which will not match the
// local calendar day everywhere — the digest itself is UTC-based throughout.
// `skipWeekend` gates the whole policy — false restores plain daily behaviour
// (weekend runs happen AND Monday gets no extension).
export function getRunSchedule(
now: Date,
skipWeekend: boolean
): { skip: boolean; extraHours: number } {
if (!skipWeekend) return { skip: false, extraHours: 0 };

const day = now.getUTCDay(); // 0 = Sun, 1 = Mon, 6 = Sat
if (day === 0 || day === 6) return { skip: true, extraHours: 0 };

return { skip: false, extraHours: day === 1 ? WEEKEND_EXTRA_HOURS : 0 };
}
1 change: 0 additions & 1 deletion test/integration/discourse.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,6 @@ describe('DiscourseSource Integration', () => {
DISCOURSE_BASE_URL: 'https://forum.example.com',
DISCOURSE_API_KEY: 'key',
DISCOURSE_API_USERNAME: 'user',
DISCOURSE_LOOKBACK_HOURS: 24,
} as any;
discourseSource = new DiscourseSource(config);
});
Expand Down
6 changes: 6 additions & 0 deletions test/unit/config.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ describe('Config Service', () => {
delete process.env.MAX_SUMMARY_TOKENS;
delete process.env.DRY_RUN;
delete process.env.DIGEST_WINDOW_HOURS;
delete process.env.SKIP_WEEKEND;
delete process.env.LOG_LEVEL;
delete process.env.MIN_MESSAGE_LENGTH;
delete process.env.EXCLUDE_COMMANDS;
Expand All @@ -32,6 +33,7 @@ describe('Config Service', () => {
expect(config.MAX_SUMMARY_TOKENS).toBe(4000);
expect(config.DRY_RUN).toBe(true);
expect(config.DIGEST_WINDOW_HOURS).toBe(24);
expect(config.SKIP_WEEKEND).toBe(true);
expect(config.LOG_LEVEL).toBe('info');
expect(config.MIN_MESSAGE_LENGTH).toBe(20);
expect(config.EXCLUDE_COMMANDS).toBe(true);
Expand All @@ -49,6 +51,7 @@ describe('Config Service', () => {
process.env.MAX_INPUT_CHARS_PER_GROUP = '';
process.env.DRY_RUN = '';
process.env.DIGEST_WINDOW_HOURS = '';
process.env.SKIP_WEEKEND = '';
process.env.LOG_LEVEL = '';
process.env.MIN_MESSAGE_LENGTH = '';
process.env.EXCLUDE_COMMANDS = '';
Expand All @@ -65,6 +68,7 @@ describe('Config Service', () => {
expect(config.MAINTAINED_PROJECTS).toEqual([]);
expect(config.DRY_RUN).toBe(true);
expect(config.DIGEST_WINDOW_HOURS).toBe(24);
expect(config.SKIP_WEEKEND).toBe(true);
expect(config.LOG_LEVEL).toBe('info');
expect(config.MIN_MESSAGE_LENGTH).toBe(20);
expect(config.EXCLUDE_COMMANDS).toBe(true);
Expand All @@ -78,6 +82,7 @@ describe('Config Service', () => {
process.env.MAX_SUMMARY_TOKENS = '2000';
process.env.DRY_RUN = 'false';
process.env.DIGEST_WINDOW_HOURS = '48';
process.env.SKIP_WEEKEND = 'false';
process.env.LOG_LEVEL = 'debug';
process.env.MIN_MESSAGE_LENGTH = '10';
process.env.EXCLUDE_COMMANDS = 'false';
Expand All @@ -89,6 +94,7 @@ describe('Config Service', () => {
expect(config.MAX_SUMMARY_TOKENS).toBe(2000);
expect(config.DRY_RUN).toBe(false);
expect(config.DIGEST_WINDOW_HOURS).toBe(48);
expect(config.SKIP_WEEKEND).toBe(false);
expect(config.LOG_LEVEL).toBe('debug');
expect(config.MIN_MESSAGE_LENGTH).toBe(10);
expect(config.EXCLUDE_COMMANDS).toBe(false);
Expand Down
Loading