Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@ GEMINI_API_KEY=

# Digest Config
# DIGEST_WINDOW_HOURS=24
# No digest on Sat/Sun (UTC); Monday looks back DIGEST_WINDOW_HOURS + 48 instead.
# Set false to restore plain daily behaviour (weekend runs, no Monday extension).
# SKIP_WEEKEND=true

# Triage
# Comma-separated projects this org maintains; bug reports against them rate high.
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/daily-digest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ jobs:
MAX_SUMMARY_TOKENS: ${{ vars.MAX_SUMMARY_TOKENS }}
MAX_INPUT_CHARS_PER_GROUP: ${{ vars.MAX_INPUT_CHARS_PER_GROUP }}
DIGEST_WINDOW_HOURS: ${{ vars.DIGEST_WINDOW_HOURS }}
SKIP_WEEKEND: ${{ vars.SKIP_WEEKEND }}
MIN_MESSAGE_LENGTH: ${{ vars.MIN_MESSAGE_LENGTH }}
EXCLUDE_COMMANDS: ${{ vars.EXCLUDE_COMMANDS }}
EXCLUDE_LINK_ONLY: ${{ vars.EXCLUDE_LINK_ONLY }}
Expand Down
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,4 +72,5 @@ All config flows through `loadConfig()` in `src/config/index.ts` (Zod schema ove
Runs as a **GitHub Actions** workflow (`.github/workflows/daily-digest.yml`), **not** a hosted service. Key facts for debugging "why didn't my change take effect":
- The digest job gates on `if: github.ref == 'refs/heads/main'` — **only `main` runs in production.** A change merged elsewhere won't appear in the digest until it reaches `main`.
- The trigger is `workflow_dispatch` only (no in-workflow cron); a **GCP Cloud Scheduler** job calls the dispatch API daily (`scripts/setup_cloud_scheduler_dispatch.sh`). Feature branches PR directly into `main`.
- **The schedule is daily but the digest is weekday-only.** Scheduler still fires seven days a week; `getRunSchedule()` (`src/utils/time.ts`) makes the app no-op on Sat/Sun (UTC) and extends Monday's lookback by `WEEKEND_EXTRA_HOURS` (48) so the weekend rolls into it. A green weekend run with no Slack post is correct, not a failure. `SKIP_WEEKEND=false` reverts the whole policy — weekend runs happen *and* Monday loses its extension.
- Operational procedures (incident playbook, required secrets/variables, recovery) live in `docs/production-runbook.md`.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Synapse is an intelligent community digest bot designed to aggregate, summarize,

- **Multi-Source Ingestion**: Seamlessly ingests messages from **Discord** channels and **Discourse** forums.
- **Intelligent Summarization**: Powered by the **Vercel AI SDK**, Synapse uses advanced LLMs (like Google Gemini) to generate concise, context-aware summaries of conversations.
- **Slack Destination**: Delivers beautifully formatted daily digests directly to your **Slack** workspace.
- **Slack Destination**: Delivers beautifully formatted weekday digests directly to your **Slack** workspace — weekends are skipped and rolled into Monday's digest.
- **Extensible Design**: Built on a modular architecture, making it easily extensible to support additional sources (e.g., GitHub, Telegram) and destinations (e.g., Email, Notion).

## Architecture
Expand Down
43 changes: 40 additions & 3 deletions docs/production-runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,27 @@ Why this setup:
## 2) Trigger Strategy

Primary trigger:
- Cloud Scheduler invokes `workflow_dispatch` daily.
- Cloud Scheduler invokes `workflow_dispatch` daily, seven days a week.

Fallback trigger:
- Keep the existing `schedule` block in the workflow as a best-effort backup.
- The workflow is `workflow_dispatch` only — it has no `schedule` block. Cloud Scheduler is the sole automated trigger.
- Keep `workflow_dispatch` enabled for manual recovery.

## 2a) Weekday-Only Digests

Cloud Scheduler still fires every day, but **the application itself no-ops on Saturday and
Sunday (UTC)** and rolls the weekend into Monday:

- Sat/Sun: the run starts, logs `Weekend (UTC) — skipping digest run`, and exits 0 before
contacting Discord, Discourse, Gemini, or Slack. **A green run with no Slack post is the
expected outcome on those days, not an incident.**
- Mon: the lookback window is `DIGEST_WINDOW_HOURS + 48` (72h by default), covering
Fri 13:00 → Mon 13:00 UTC with no gap. Its Slack header shows a date range rather than a
single date.
- `SKIP_WEEKEND` (repo Variable, default `true`) controls the whole policy. Setting it
`false` restores plain daily behaviour — weekend runs happen *and* Monday loses its
extension. Use it to force a one-off weekend run, then set it back.

## 3) Prerequisites

- GCP project with Cloud Scheduler API enabled.
Expand Down Expand Up @@ -83,6 +98,7 @@ Required workflow variables (minimum):
- `GEMINI_MODEL`
- `MAX_SUMMARY_TOKENS`
- `DIGEST_WINDOW_HOURS`
- `SKIP_WEEKEND` (default `true`; see §2a)
- `MIN_MESSAGE_LENGTH`
- `EXCLUDE_COMMANDS`
- `EXCLUDE_LINK_ONLY`
Expand All @@ -93,13 +109,22 @@ Required workflow variables (minimum):
Check these once per day:
- Cloud Scheduler job execution status is successful.
- A corresponding successful run exists in the `Daily Digest` workflow.
- Slack digest arrived in the expected channel.
- Slack digest arrived in the expected channel — **Monday through Friday only.**

On Saturday and Sunday, expect a successful workflow run and **no Slack post** (see §2a).
Verify the run logged the weekend-skip line; a weekend run that posted a digest means
`SKIP_WEEKEND` has been left `false`.

## 7) Alerting Baseline

Set up two alerts:
- **Trigger failure alert**: Cloud Scheduler job failed.
- **Digest missing alert**: no successful `Daily Digest` run or no Slack post within expected window.
**Suppress the "no Slack post" half on Saturday and Sunday (UTC)** — it would otherwise fire
every weekend by design. The "no successful run" half still applies all seven days.

A missing **Monday** digest is the most costly failure mode: it drops three days of community
activity, not one. Escalate it accordingly and use the recovery procedure in §9.

Target response objective:
- Detect failure within 30 minutes of scheduled run.
Expand Down Expand Up @@ -150,6 +175,18 @@ If a daily run is missed:
3. Confirm Slack post delivery.
4. Document incident summary and corrective action.

A manual dispatch uses the same rolling window as a scheduled one, so recovering a *late* run
needs a wider window to reach back to the missed period.

If a **Monday** run is missed, it was carrying three days of content and a plain re-dispatch on
Tuesday would only look back 24h. The job holds no state between runs, so widen the window
manually:
1. Set repo Variable `DIGEST_WINDOW_HOURS` to `96` (Fri 13:00 → Tue 13:00).
2. Manually dispatch `Daily Digest`.
3. Confirm Slack post delivery, then **restore `DIGEST_WINDOW_HOURS` to `24`.**

Do not set `SKIP_WEEKEND=false` for this — that would also strip the Monday extension.

## 10) Security & Maintenance

- Rotate GitHub PAT on a fixed schedule.
Expand Down
23 changes: 20 additions & 3 deletions src/DigestPipeline.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import { Source, Destination, Processor } from "./core/interfaces";
import { NormalizedMessage, DigestContext } from "./core/types";
import { Config } from "./config";
import { logger } from "./utils/logger";
import { getUtcDailyWindowFrom } from "./utils/time";
import { getRunSchedule } from "./utils/time";
import { applyMessageFilters } from "./utils/filters";
import { buildDigestBlocks, formatDigest } from "./utils/format";
import { DigestItem } from "./core/schemas";
Expand Down Expand Up @@ -52,7 +52,24 @@ export class DigestPipeline {

async run() {
logger.info("Starting digest pipeline...");
const { start, end, dateTitle } = getUtcDailyWindowFrom(new Date());

const now = new Date();
const { skip, extraHours } = getRunSchedule(now, this.config.SKIP_WEEKEND);
if (skip) {
logger.info("Weekend (UTC) — skipping digest run; content rolls into Monday.");
return;
}

// The window described in the header is the window actually fetched:
// a rolling `now - windowHours`, not a calendar day. On Monday that
// spans the weekend, so the title carries a range instead of one date.
const windowHours = this.config.DIGEST_WINDOW_HOURS + extraHours;
const end = now;
const start = new Date(end.getTime() - windowHours * 60 * 60 * 1000);
Comment thread
alchemydc marked this conversation as resolved.
const isoDate = (d: Date) => d.toISOString().slice(0, 10);
const dateTitle = extraHours > 0
? `${isoDate(start)} → ${isoDate(end)}`
: isoDate(end);
const context: DigestContext = { start, end, dateTitle };

const allMessages: NormalizedMessage[] = [];
Expand All @@ -62,7 +79,7 @@ export class DigestPipeline {
if (source.isEnabled()) {
logger.info(`Fetching from source: ${source.name}`);
try {
const messages = await source.fetchMessages(this.config.DIGEST_WINDOW_HOURS);
const messages = await source.fetchMessages(this.config.DIGEST_WINDOW_HOURS, extraHours);
logger.info(`Fetched ${messages.length} messages from ${source.name}`);
allMessages.push(...messages);
} catch (err: any) {
Expand Down
10 changes: 10 additions & 0 deletions src/config/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,14 @@ const ConfigSchema = z.object({
z.number().int().min(1).default(24)
),

// Suppress Saturday/Sunday runs and roll the weekend into Monday's digest
// (Monday looks back DIGEST_WINDOW_HOURS + 48). Setting this false restores
// plain daily behaviour: weekend runs happen and Monday gets no extension.
SKIP_WEEKEND: z.preprocess(
toBool,
z.boolean().default(true)
),

LOG_LEVEL: z.preprocess(toStr, z.string().default("info")),

MIN_MESSAGE_LENGTH: z.preprocess(
Expand Down Expand Up @@ -127,6 +135,7 @@ export type Config = {
MAX_INPUT_CHARS_PER_GROUP: number;
DRY_RUN: boolean;
DIGEST_WINDOW_HOURS: number;
SKIP_WEEKEND: boolean;
LOG_LEVEL: string;
MIN_MESSAGE_LENGTH: number;
EXCLUDE_COMMANDS: boolean;
Expand Down Expand Up @@ -197,6 +206,7 @@ export function loadConfig(): Config {
geminiModel: config.GEMINI_MODEL,
dryRun: config.DRY_RUN,
digestWindowHours: config.DIGEST_WINDOW_HOURS,
skipWeekend: config.SKIP_WEEKEND,
maxSummaryTokens: config.MAX_SUMMARY_TOKENS,
maxInputCharsPerGroup: config.MAX_INPUT_CHARS_PER_GROUP,
logLevel: config.LOG_LEVEL,
Expand Down
5 changes: 4 additions & 1 deletion src/core/interfaces.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,10 @@ import { DigestItem } from "./schemas";
export interface Source {
name: string;
isEnabled(): boolean;
fetchMessages(windowHours: number): Promise<NormalizedMessage[]>;
// `extraHours` is the weekend catch-up added on Mondays. It is kept separate
// from `windowHours` so a source with its own lookback override (Discourse)
// can apply the extension on top of that override rather than lose it.
fetchMessages(windowHours: number, extraHours?: number): Promise<NormalizedMessage[]>;
}

export interface Destination {
Expand Down
4 changes: 2 additions & 2 deletions src/services/discord/DiscordSource.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ export class DiscordSource implements Source {
return this.config.DISCORD_ENABLED;
}

async fetchMessages(windowHours: number): Promise<NormalizedMessage[]> {
async fetchMessages(windowHours: number, extraHours = 0): Promise<NormalizedMessage[]> {
if (!this.isEnabled()) return [];

const token = this.config.DISCORD_TOKEN;
Expand All @@ -46,7 +46,7 @@ export class DiscordSource implements Source {
}

const now = Date.now();
const since = now - windowHours * 60 * 60 * 1000;
const since = now - (windowHours + extraHours) * 60 * 60 * 1000;
const messages: NormalizedMessage[] = [];

try {
Expand Down
7 changes: 5 additions & 2 deletions src/services/discourse/DiscourseSource.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,14 @@ export class DiscourseSource implements Source {
return this.config.DISCOURSE_ENABLED;
}

async fetchMessages(windowHours: number): Promise<NormalizedMessage[]> {
async fetchMessages(windowHours: number, extraHours = 0): Promise<NormalizedMessage[]> {
if (!this.isEnabled()) return [];

const now = Date.now();
const lookHours = this.config.DISCOURSE_LOOKBACK_HOURS ?? windowHours;
// The forum-specific override replaces the base window, but the weekend
// catch-up still stacks on top of it — otherwise a configured override
// would silently drop the whole weekend from Monday's digest.
const lookHours = (this.config.DISCOURSE_LOOKBACK_HOURS ?? windowHours) + extraHours;
const since = now - lookHours * 60 * 60 * 1000;
const maxTopics = this.config.DISCOURSE_MAX_TOPICS ?? 50;

Expand Down
7 changes: 7 additions & 0 deletions src/services/llm/AiSdkProcessor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -253,6 +253,13 @@ export class AiSdkProcessor implements Processor {
break;
}
}
// Truncation drops the OLDEST messages. On a Monday roll-up that is
// Friday's content, so make it visible rather than silent.
if (out.length < messages.length) {
logger.warn(
`Truncated group: kept ${out.length}/${messages.length} messages (limit ${maxChars} chars)`
);
}
return out;
}
}
5 changes: 4 additions & 1 deletion src/utils/format.ts
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,10 @@ export function buildDigestBlocks(params: {
logger.debug("[DEBUG] buildDigestBlocks items count:", params.items.length);
}

const range = `Time window: ${params.dateTitle} 00:00–${params.end.toISOString().slice(0, 10)} 00:00 UTC`;
// Describe the window actually fetched (a rolling span ending now), not a
// calendar day — on Mondays this spans the weekend.
const fmtUtc = (d: Date) => d.toISOString().slice(0, 16).replace("T", " ");
const range = `Time window: ${fmtUtc(params.start)}–${fmtUtc(params.end)} UTC`;

const legend = "🔴 urgent · 🟡 notable · ⚪ routine";

Expand Down
33 changes: 19 additions & 14 deletions src/utils/time.ts
Original file line number Diff line number Diff line change
@@ -1,17 +1,22 @@
// utils/time.ts
export function getDigestWindow(hours: number): { start: Date; end: Date } {
const end = new Date();
const start = new Date(end.getTime() - hours * 60 * 60 * 1000);
return { start, end };
}

// Returns UTC daily window (midnight to midnight) for a given UTC date
export function getUtcDailyWindowFrom(candidateUtc: Date): { start: Date; end: Date; dateTitle: string } {
const y = candidateUtc.getUTCFullYear();
const m = candidateUtc.getUTCMonth();
const d = candidateUtc.getUTCDate();
const start = new Date(Date.UTC(y, m, d, 0, 0, 0));
const end = new Date(start.getTime() + 24 * 60 * 60 * 1000);
const dateTitle = start.toISOString().slice(0, 10);
return { start, end, dateTitle };
// Sat + Sun: the two runs the weekend skip suppresses. Monday's window is
// extended by exactly this much so the skipped days are covered with no gap.
export const WEEKEND_EXTRA_HOURS = 48;

// Decides whether this run happens at all, and how far back it looks.
// Weekday is read in UTC: the job fires at 13:00 UTC, far from any midnight
// boundary, so no populated timezone disagrees about which day it is.
// `skipWeekend` gates the whole policy — false restores plain daily behaviour
// (weekend runs happen AND Monday gets no extension).
Comment thread
alchemydc marked this conversation as resolved.
export function getRunSchedule(
now: Date,
skipWeekend: boolean
): { skip: boolean; extraHours: number } {
if (!skipWeekend) return { skip: false, extraHours: 0 };

const day = now.getUTCDay(); // 0 = Sun, 1 = Mon, 6 = Sat
if (day === 0 || day === 6) return { skip: true, extraHours: 0 };

return { skip: false, extraHours: day === 1 ? WEEKEND_EXTRA_HOURS : 0 };
}
6 changes: 6 additions & 0 deletions test/unit/config.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ describe('Config Service', () => {
delete process.env.MAX_SUMMARY_TOKENS;
delete process.env.DRY_RUN;
delete process.env.DIGEST_WINDOW_HOURS;
delete process.env.SKIP_WEEKEND;
delete process.env.LOG_LEVEL;
delete process.env.MIN_MESSAGE_LENGTH;
delete process.env.EXCLUDE_COMMANDS;
Expand All @@ -32,6 +33,7 @@ describe('Config Service', () => {
expect(config.MAX_SUMMARY_TOKENS).toBe(4000);
expect(config.DRY_RUN).toBe(true);
expect(config.DIGEST_WINDOW_HOURS).toBe(24);
expect(config.SKIP_WEEKEND).toBe(true);
expect(config.LOG_LEVEL).toBe('info');
expect(config.MIN_MESSAGE_LENGTH).toBe(20);
expect(config.EXCLUDE_COMMANDS).toBe(true);
Expand All @@ -49,6 +51,7 @@ describe('Config Service', () => {
process.env.MAX_INPUT_CHARS_PER_GROUP = '';
process.env.DRY_RUN = '';
process.env.DIGEST_WINDOW_HOURS = '';
process.env.SKIP_WEEKEND = '';
process.env.LOG_LEVEL = '';
process.env.MIN_MESSAGE_LENGTH = '';
process.env.EXCLUDE_COMMANDS = '';
Expand All @@ -65,6 +68,7 @@ describe('Config Service', () => {
expect(config.MAINTAINED_PROJECTS).toEqual([]);
expect(config.DRY_RUN).toBe(true);
expect(config.DIGEST_WINDOW_HOURS).toBe(24);
expect(config.SKIP_WEEKEND).toBe(true);
expect(config.LOG_LEVEL).toBe('info');
expect(config.MIN_MESSAGE_LENGTH).toBe(20);
expect(config.EXCLUDE_COMMANDS).toBe(true);
Expand All @@ -78,6 +82,7 @@ describe('Config Service', () => {
process.env.MAX_SUMMARY_TOKENS = '2000';
process.env.DRY_RUN = 'false';
process.env.DIGEST_WINDOW_HOURS = '48';
process.env.SKIP_WEEKEND = 'false';
process.env.LOG_LEVEL = 'debug';
process.env.MIN_MESSAGE_LENGTH = '10';
process.env.EXCLUDE_COMMANDS = 'false';
Expand All @@ -89,6 +94,7 @@ describe('Config Service', () => {
expect(config.MAX_SUMMARY_TOKENS).toBe(2000);
expect(config.DRY_RUN).toBe(false);
expect(config.DIGEST_WINDOW_HOURS).toBe(48);
expect(config.SKIP_WEEKEND).toBe(false);
expect(config.LOG_LEVEL).toBe('debug');
expect(config.MIN_MESSAGE_LENGTH).toBe(10);
expect(config.EXCLUDE_COMMANDS).toBe(false);
Expand Down
Loading