Skip to content

Latest commit

 

History

History
2140 lines (1948 loc) · 168 KB

ARCHITECTURE.md

File metadata and controls

2140 lines (1948 loc) · 168 KB

helix-opspile architecture

opspile is a unified inbox for an ops team. It ingests every channel the team works in on Slack, Microsoft Teams and Discord, keeps the threads in Cloudflare R2, and tracks read/unread state per person and per team, so that you can see which threads somebody else has already seen and you have not.

It follows the spirit of klickhaus: a small static UI in plain JavaScript, fed by separate ingestion services, with the data store doing the heavy lifting.

Everything runs on Cloudflare (Workers, R2, D1, Durable Objects, Workers AI, Vectorize, Artifacts, AI Gateway) in a single account; the thread agents' model is Claude on Amazon Bedrock, reached through AI Gateway.

The four piles

flowchart LR
  S[Slack Events API] --> R
  T[Microsoft Graph<br/>change notifications] --> R
  D[Discord REST API<br/>polled every minute] --> R
  B[Backfill from history APIs] --> R
  R[(1. raw events<br/>R2, append-only)] --> C
  C[(2. channel folders<br/>R2, one file per message)] --> I
  C <--> K[(3. knowledge bases<br/>Artifacts, one repo per org / team)]
  I[(4. fast index<br/>D1 + Durable Objects)] --> UI[opspile UI]
  UI -- read events --> I
Loading

Each pile is derived from the one before it. Pile 2 never talks to the platforms directly, so it, and everything after it, can be rebuilt by replaying pile 1.

1. Raw events

One R2 bucket, append-only, holding events exactly as the platform API delivered them. When Slack or Microsoft add fields, we keep them; normalization happens downstream. The one exception is credentials: Slack's deprecated verification token and Graph's clientState are removed before an event is stored.

<source>/<tenant>/<channel>/<yyyy>/<mm>/<dd>/<eventTs>-<eventId>.json
  • The key is derived from the platform's event id, so redelivered events (Slack retries with the same event_id, Graph may notify twice) overwrite themselves. Ingestion is idempotent.
  • Slack delivers edits as message_changed (with the full new message), deletions as message_deleted, and reactions as separate events. All of them are stored. A file uploaded to a Slack Connect channel arrives as a stub (file_access: check_file_info: an id and a file type, no name or link); the normalizer looks it up once with files.info and keeps the answer here, next to the channel's events, as slack/<tenant>/<channel>/files/<file id>.json ({ type: 'file_info', file }, without shares, which quotes other channels), so a replay asks Slack nothing. It is not an event and is skipped. A file Slack does not show the app is not kept, and the message says a file of that type is there.
  • Teams change notifications usually carry only the resource path. We store the notification and the message we then fetch as two objects. Graph notifications have no event id, so a notification is keyed by its own hash (notification-<sha256>.json), and a fetched message by its version (<lastModified ms>-<messageId>.json). The notification's resource names the message; the message's metadata names the notification. The files of a Teams message are kept here too, as bytes: the images pasted into it (hosted contents, fetched with the app's Graph token) and the files attached to it (SharePoint or OneDrive links, fetched through Graph's /shares with Files.Read.All), under teams/<tenant>/<team>/<channel>/files/<sha256 of the URL> with their content type and name (src/teamsfiles.js). The normalizer copies them when it first sees the message; the message keeps the platform's link, and an image attached as a file is shown where Teams shows it. The inbox's media proxy serves the copies, only to those who may see the message's thread (SharePoint files only from there, since the app could read any of them), and agents' read_file reads them. They are not events and are skipped; one Graph does not hand over is not kept, and the file stays a link.
  • Discord is polled (see Ingestion): one Durable Object (DiscordPoller) stores each state of a message it reads (new, edited, other reactions) and each deletion it finds as an envelope { t, via, guild, d, thread?, root?, reactions?, at? }. d is what Discord sent; around it is what only the poller knew at the time: the thread a message was posted in (Discord names just the thread's channel id), the conversation an inline reply belongs to, who reacted (messages carry counts only) and since when the poller has seen each reaction, and when it found a message gone. Keys are <message id>-<hash of the envelope> and <message id>-delete, so the same state read twice shares a key.
  • Backfill pages from the history APIs land here too, marked via: backfill (Discord's: via: history).

2. Channel folders

channels/<org>/<team>/<source>-<channelId>/<rootId>/<messageId>.md

Every channel belongs to one org (a customer, a partner or a vendor) and one team (the ops team that owns it). The mapping is a small rules engine over rows in D1: each rule matches the channel id, the channel name, the Teams team name or a Slack Connect team id with a regular expression, and sets the team, the org or the kind (channel, alert, ignore); the lowest priority wins. Naming conventions (aem-<customer> exists on both platforms) are rules, and so are the overrides for vendor channels, alert channels and irregular names. Channels that no rule matches are filed under _unassigned. A manual assignment of one channel is a rule too: its channel_id, priority 10.

Shared channels. A channel can be in several teams' inboxes, but it is filed under one team, its first: the rules pick it, and its folders, thread rows (threads.team), knowledge base and agents are that team's. The other teams follow it through channel_teams (source, channel id, team): the channel's threads are in their inboxes, and their members read, reply, react, resolve and draft there; read state in a thread is that of the team it is looked at from. Adding a team to a channel moves the channel only for its first team, when the admin team (or nobody) holds it; added to a channel another team holds, the new team shares it and nothing moves. Handing a channel back to the admin team is always a move. The shares are configuration, like the rules, so a replay of pile 1 keeps them.

An org is a slug. What it is called, its icon (a domain, whose favicon is shown, or an image URL) and whether it is a customer, a partner or a vendor are a row in orgs; an org without one is a customer shown by its slug and the favicon of its usual mail domain. The inbox and the agents show the name and the type.

The org list in the admin pages shows each org with the icon the inbox resolves for it (its own, else the favicon of its usual mail domain: the most common one among the people outside Adobe who post in its threads, free mail and tenant domains left out), and an empty square where there is none, so the orgs without an icon stand out (and can be listed alone). Favicons come through /api/admin/favicon, which answers 404 where Google's favicon service has only its globe. An org's page suggests icons (src/favicons.js): the favicons of the websites its mail domains, slug and name suggest (<slug>.com, www.<slug>.com, <name>.com, …) and the icons those sites name (<link rel="icon">, apple-touch-icon), each image with what Clef thinks of it: whether the site is the org's own (not a namesake's, an agency's or a parked domain's; Clef is told each site's title where it can be read) and whether the image is a logo rather than a stock icon. Picking one makes it the org's icon. Sites are asked with a browser's user agent (many answer a bot's with an error page); an .ico of bitmaps, which Clef cannot read, is decoded into a PNG for it (src/ico.js), and an SVG icon is offered without Clef's verdict.

Administration. The admin team manages all of this at /inbox/admin: ops teams and their members (found by name or email among the people and platform accounts opspile knows; names come from the platforms and are not edited there), the rules that file channels under a team (a name prefix like aem-, a Teams team, a Slack Connect team), manual assignments of channels to a team or an org, the channels shared with more teams, and each org's name, icon and type. A team's page suggests channels for it among the ones the admin team holds (the rule of last resort files every channel no other rule claims there): the ones where the team's members post, react or are members (listed from Slack, or from the Teams team, into channel_members), ranked tf-idf style so that someone active in most of the admin team's channels (an admin, say) counts for little and a member active in a few counts the most, and the ones Clef gives the team. The cron asks Clef about a batch of the admin team's channels every run, one question set per channel (channel_labels): which ops team should follow it (each described by its rules, some of its channels and its members, or none of them), and whether it is a conversation, an alert feed or not worth following. Clef is told the channel's name, its Teams team, its topic, its customer, how busy it is, the Adobe people who post in it and are in it with their ops teams, the technologies seen, and its recent threads' titles; it looks again when the channel has new activity, the ops teams change, or the label is a week old. Every rule change applies at once (the page names the channels it moved; the API still answers dryRun with what a change would move), and the page sends its changes one after the other, so many can be made in a row. Applied, each moved channel is re-filed by a refile job on the NORMALIZE queue: its thread folders move to channels/<new org>/<new team>/… (an object already at the new place, from a message that arrived meanwhile, is newer and kept), its thread rows take the new org, team and kind, and the new org and team get their knowledge bases. A job starts from where each thread is filed in D1, and also moves what is still where it was when the job was queued (a message filed meanwhile points the row at the new place), so several changes to a channel in a row end up where the last one says. A large channel moves a few hundred files per job.

Team managers. A member of a team can be its manager: they get the admin pages and the knowledge-base editor for that team only, where they add and remove members, promote them to manager or demote them, and edit the team's knowledge base (its systemprompt.md too). Teams, rules, assignments, orgs and the customers' knowledge bases stay with the admin team. A team can be self-managed: every member is a manager, and so is everyone who joins. Turning that off keeps the roles as they are; from then on new members join as members, and managers can be demoted. The role is team_members.role, the flag teams.self_managed; only the admin team sets the flag.

Channel requests. A team manager does not assign channels, but asks for them: their team's page has the same channel search the admin team adds channels with (every channel, by name, Teams team, customer or id), and picking one sends a request, "add this channel to this team", with an optional note (channel_requests, migration 0028; one open request per channel and team). The admin team sees the open ones under Requests in the admin pages, their number on Requests in every page's header. Opening one runs the manual assignment as a dry run and says what confirming would do: re-file the channel under the team (and which threads move), or share it with the team when another team holds it. Confirm then makes that assignment, exactly as adding the channel on the team's page does, and records who confirmed it and what it did; Decline takes an optional reason. The manager sees each request on their team's page: pending, confirmed (moved or shared), declined with the reason, or withdrawn (they may take an open one back). An admin who adds a requested channel to the team directly answers the request with it, and so does a rule that moves the channel there. Confirming closes the request first, so one declined or withdrawn meanwhile moves nothing, and a failed assignment opens it again. Nothing is sent by mail or chat.

Channel names and authors are not in the events, so the normalizer looks them up (conversations.info and users.info on Slack, Graph on Teams) and caches them in D1 for a day. These are its only platform calls. Files only move forward to a newer version, and a tombstone is final, so a replay of pile 1 in any order rebuilds the same pile 2. A rule changed in D1 by hand needs pile 2 emptied and replayed; one changed in the admin UI re-files the channels it moves (below).

A renamed channel is re-filed the same way, by the same refile jobs: the rules read a channel's name, its Teams team's name and its Slack Connect teams, so a channel renamed to a name a rule maps to an org (or another team, or a kind) takes its threads along. The normalizer notices it when a Slack rename event arrives, when a Teams notice says the channel or its team was renamed (it looks the channel up again then), and when a lookup (a day after the last, from a message or the cron) finds them changed; Discord renames are found by the lookups. It queues the moves only when the rules classify the channel differently than before, before it files the message that came with the change. A manual assignment is a rule, so a rename does not undo it (a channel assigned to a team keeps it, and takes the org its new name gives); the teams the channel is shared with stay. A rename event older than what D1 knows of the channel (a replay) is left out.

One markdown file per message, in a folder per platform thread (the root message's id; the root message is <rootId>/<rootId>.md). Message bodies are markdown themselves (Slack mrkdwn and Teams HTML are converted; a Teams system notice, whose body is an empty <systemEventMessage/>, is written from its eventDetail: "Meeting ended after 50 s. Only Ada joined.", with the names looked up where Graph leaves them out), so keeping each message in its own file avoids nesting one document inside another, and makes every file a natural retrieval chunk. There is no per-thread rollup file; reading a thread means listing its folder.

Folder snapshots are a read cache beside the folders, not part of pile 2's record: snapshots/<folder>.json holds every file of one folder, parsed, with its R2 version (src/filecache.js). A Worker may wait on only six R2 calls at a time, so an isolate that had not read a thread before read a thread of 300 messages in some fifty rounds (three seconds); with the snapshot it lists the folder, reads the snapshot, and reads only the files whose version the snapshot does not have, then writes the snapshot again behind the answer. A file is never taken from a snapshot for a version it does not have, so a stale or missing snapshot costs reads, never correctness; re-filing a folder deletes its snapshot, and emptying pile 2 may drop them all.

---
source: slack
thread: slack:<enterpriseId>:<channelId>:<thread_ts>
logicalThread: slack:<enterpriseId>:<channelId>:<other_thread_ts>   # only if re-threaded
id: <ts>
at: 2026-10-14T07:37:26Z
author: { kind: person, name: …, org: …, staff: true }
editedAt: …
deletedAt: …
raw: [<raw object key>, …]
permalink: https://…
---

The message body, as markdown.
thread key   slack:<enterpriseId>:<channelId>:<thread_ts>
             teams:<tenantId>:<teamId>:<channelId>:<rootMessageId>
             discord:<guildId>:<channelId>:<rootMessageId>
  • A message that quotes another (Slack: one forwarded with it, is_share, or the preview Slack adds under a link to a message, is_msg_unfurl) has it below its text and above its files, as a markdown quote whose first line is its author, linked to it with the title "Quoted message, <ISO time>" (quoteMarkdown in src/markdown.js). The inbox shows it as a card. Its words are not its author's: they count for no mentions, nor for the vectors of what people talk about, and editing the message leaves the quote alone.
  • New messages are new objects, so concurrent replies never contend for the same file.
  • An edit rewrites exactly one file. Earlier versions remain in pile 1.
  • A deletion keeps the file as a tombstone (deletedAt set, body removed), so links and read watermarks stay valid.
  • A Slack permalink names the channel's workspace (https://<workspace>.slack.com/…, from team.info, kept in channels.domain): a bare slack.com link opens in whichever workspace the browser used last, and people are often in several. A channel of the whole Enterprise org, or one shared between its workspaces, links to the workspace most channels live in. Links filed before a channel's workspace was known are rewritten in D1 when it becomes known.

Enrichment on write. When a message is copied into this pile:

  • Clef (a decision model) classifies the thread: who has to act next (us, customer, vendor, nobody), what kind of thread it is (incident, security, question, request, alert), and urgency. Thread-level results live in the index (pile 4).
  • Deterministic extraction handles structured tokens: CVE ids, URLs, ticket links, org/site identifiers.
  • Clairic (an extraction model) handles loose facts, and only values above a confidence threshold are kept.

Re-threading. People often post a follow-up as a new root message instead of replying, or split one request over several messages, and alert channels post the same alert again and again. A root by the same person within three minutes of their previous root in the channel, with nobody else's root between them, continues it (a burst), unless that root is deleted by then (people delete a message and post it again, and the new one is the thread), or someone has answered it in its thread already (then the new one is a message of its own, as the next of several questions each in its thread). A root that only greets ("Hello Ada", "Hi team :wave:": a salutation, the people it is addressed to and emoji, greets) says nothing of its own, and its author's next root within 15 minutes continues it the same way: people greet, then take minutes to write what it is about, and Clef reads a greeting as a conversation of its own. A root that says nothing but @-mentions ("@", "@Ada Lovelace", at most emoji and punctuation besides, or not even a mention: "?", ":eyes:"; mentionsOnly) continues its author's previous root within those 15 minutes the same way, with the same conditions (nobody else's root between them, that root not deleted and not answered in its thread yet): its author is calling the bot or a colleague to the question they just asked, and it has no words for Clef or vector search to place it by. A mention is @ and up to four capitalized words, what the platforms write for one, so "@Ada can you check" says something and is Clef's to decide. Somebody else's bare mention is not theirs to continue: one of a conversation's people, @-mentioned by name, is an addressed reply (below), and the same mention by its own asker, after somebody else's root, or later than 15 minutes, is Clef's. In the case that showed it, a customer asked whether a tool had an issue, then pinged the bot three and a half minutes later, 30 seconds past BURST_MS, and the ping's own entry got an agent that asked the customer what he needed. When a root message arrives, its skeleton (the text without timestamps, long numbers, hex and UUID ids, query strings and user agents; short numbers like status codes stay) is embedded and the most similar earlier roots in the same channel, within 14 days, are found by vector search.

  • Alert channels: an exact repeat of the skeleton or a near-identical root (cosine ≥ 0.97) is a match outright; otherwise Clef chooses between up to four candidates above 0.82 and "a new topic", told how long before each was posted, and a match needs a probability of at least 0.7. Candidates of the same group share their probability.
  • Conversation channels: in a chat, answers, thanks, nudges and escalations posted as new messages ("done, you have access now", "this is a blocker, please help") share few words with what they follow up, people answer hours later, and the same words ("thanks!") are not a repeat. So Clef reads the channel's latest three conversations active in the last three days and the groups of up to four similar roots (cosine ≥ 0.6) as a transcript: each conversation's first message and its latest two before the new root, marked with the conversation's letter and how long before they were posted, then the new message, and says which conversation it continues or that it is a new topic. A match needs a probability of at least 0.8, and 0.99 for a conversation with replies in its platform threads (MIN_CLEF_THREADED), and for any conversation in a channel that replies in threads (of its roots in the last 30 days, at least five, half or more got a person's reply in their own: a question that is still waiting for its first reply there is held in threads all the same): people continue a chat with more top-level messages where the chat is top-level messages, and where they reply in threads, a new top-level message is nearly always a topic of its own, however close it reads. Folded into conversations held in threads at 0.8, new roots got replies of their own as often as the channel's new topics; folded into ones without, half as often. In channels that reply in threads, those folded at Clef's usual certainty got replies of their own twice as often as the other folds. Comparing the new root with each candidate on its own spread Clef's answer over several candidates and left most of a chat in pieces. Platform notices ("… has joined the channel") are left alone.
  • Replies posted top-level: somebody who answers what they just read in the channel's box, rather than in the thread's, posts a reply as a root: it @-mentions the person it answers, and it comes at once. A root that @-mentions, by name, somebody who spoke in one of the conversations Clef reads, less than five minutes after that conversation's latest message, is its addressed reply: Clef's usual certainty in a conversation (0.8) is enough to fold it, whatever the channel's habit and the conversation's thread replies, and even when it has replies of its own already. The fold is logical_via = 'reply', and, like a burst member, it stays in its group when people answer it in its platform thread (below): that thread is where the conversation went on, so whoever answers the reply answers the group. The lower bar is only for the conversation the root addresses: where Clef picks another one, it needs MIN_CLEF_THREADED as before. Of production's roots over two weeks, 13 were addressed replies, and Clef gave 0.8 or more to three of them: a chat's answer and its follow-up, and a staff answer to a customer. The ten it kept apart say so themselves ("in other news", "another ticket", a question of their own), which is what the five minutes are for: an @-mention of whoever was active last, hours later, is as often a topic of its own.
  • Quotes: a root that quotes a message (below) stays on its own, whatever Clef or a burst would say. Forwarding a message into the channel, or linking to one, starts a topic on purpose: whoever meant to continue that conversation would have replied in it, and the text that goes with a quote ("first step in that direction") reads like a follow-up of whatever was said last. The root after it can still continue it. In the transcripts Clef reads, a quote is > <author> wrote: and its text.
  • Thread marks: a root that announces a thread (🧵, :thread:) stays on its own too, and out of a burst: nearly all of them are answered in their thread.
  • Replies of its own (de-threading): a conversation's root that people answer in its own platform thread holds a conversation of its own, however much it reads like a follow-up (a different customer raising a related problem by @-mentioning whoever was active last). One that has a person's reply when it is decided is not folded. One that gets its first after it was folded is split back out (dethread, on that reply): its row loses logical_thread and gets logical_via = 'split', which re-threading (forced, or a burst before it) leaves alone for good, its root file loses logicalThread, the roots that followed it in a burst go with it, and the open inboxes reload the group (split in the room's thread event). So the group's entry stops interleaving the two platform threads, and its replies go to its own platform thread again (src/post.js target, the entry's latest). A burst member, an addressed reply (logical_via = 'reply', above) and alerts stay in their group.
  • Replies the inbox routed: a group's replies go to the platform thread it was last active in, which may be a folded root's. Once one does (a person's from the inbox, the app's, or a thread agent's), the group's conversation goes on there, where the channel sees it: that root gets routed_at (routeReply, written before the post) and stays in its group for good. Its replies do not split it out, neither the inbox's own nor those of the people who answer them there, and a forced re-threading leaves it alone. Splitting it would leave the answer the channel read in an entry without the question it answered.

A match sets logicalThread on that one message file and on its thread row, always pointing at the group's first thread. The file stays where it is and the platform thread is never modified, so undoing the match is a one-field change. Replays arrive in any order, so an alert that is matched also pulls in near-identical later alerts that had found nothing earlier, and a root pulls in the next root by the same person within three minutes (15 after a greeting, or for a root that only mentions). A conversation's root is decided from the conversations before it, so re-threading a channel again goes oldest root first (scripts/replay.mjs --rethread --channel <id>). The inbox shows a group as one entry: the first thread's title, the latest activity and the latest member's label, the most pressing attention of any member, and all members' messages in order. A first thread whose messages were all deleted after a member folded into it has no activity, title or link: the next member's stand in. A root is re-threaded 30 seconds after it arrives, so it can be in the inbox on its own until then: a match tells the open inboxes which threads it folded, which take those threads' own entries off the list once the group's is there (as does a reload of the first page that has them in a group, should the announcement be missed), one that has a folded thread open shows its group instead, and opening a member's key (a link, a tab) opens its group too.

Vector search uses Workers AI embeddings in Vectorize, written at ingestion time so that a message can be matched seconds after it arrives.

A second index, CONTENT (src/vectors.js), holds what was said, for search over topics and who talks about them: one vector per message a person posted in a channel thread (type: message, with the author's person, staff and author key) and one per such thread (type: thread), the mean of the embeddings of its title and messages in pieces of about 512 tokens (a long thread keeps its first piece and its latest). Both carry channel, thread, org, team and at; the text stays in pile 2. Two minutes after a person posts, or a message is deleted, the thread is vectorized: each message keeps a hash of the text its vector was made from (messages.embedded), so only new and edited messages are embedded again, and a deleted one loses its vector. The cron enqueues the threads whose vectors are older than their latest message (threads.vectorized_through), which also fills the index the first time. Alert threads (a bot's posts) get no vectors here. A re-filed thread's vectors are made again (its hashes cleared, a vectorize job queued), so that they carry its new team and org; re-filed as an alert or ignored, it loses them.

A third index, SEARCH (src/fulltext.js), is for full-text search: an AI Search instance with built-in storage, one item per message of a channel thread (people's and bots', not system notices), its markdown body as it is. It carries team, channel (a hash of the channel key: AI Search filters read the first 64 bytes of a value, and a Teams channel key is longer), message (<thread>|<id>) and at. The vectors find what a message is about; BM25 here finds the words it has, an error string, a hostname or an id, which an embedding blurs. Its items are kept in step by the same job as the vectors: each message keeps a hash of its team and text (messages.indexed) and its item's id (messages.search_item), so an edit uploads one item again (a re-upload under the same name replaces it), a deletion takes it away, and a re-filed thread's items are uploaded again under the new team (re-filed as an alert or ignored, it loses them). A failure leaves a message's row as it was, so the next run tries again, and does not keep the vectors from being made; an item whose deletion failed keeps its id until it is gone. Search only counts and shows an item whose message D1 has and is not deleted, so a deleted message's words are never found, whatever is left in the index. A job uploads at most 100; threads.indexed_through tells the cron's sweep which threads are behind. AI Search's own vectors (the instance is hybrid) are not used yet: the CONTENT vectors already do that part.

3. Knowledge bases

One Artifacts repository per org and one per team: a pile of markdown files with git history.

kb-org-<slug>    what we know about this customer or vendor, across all teams
kb-team-<slug>   product and process knowledge for this team, across all customers

The repositories follow the Open Knowledge Format (OKF v0.2), the formalised LLM wiki pattern: one markdown file per concept (a site, a known issue, a playbook, a decision), YAML frontmatter with a required type, links between concepts, an index.md per directory and a log.md at the root. Provenance is in each concept's frontmatter: sources lists the threads (permalinks) a concept was learned from, with who said it and when, and footnotes tie claims to them; generated says which agent wrote it and when, and verified will record staff review. Every change is a commit, so a bad memory is a git revert.

Thread agents are the writers. A thread's agent writes its team's and its customer's knowledge base, and those of the teams its channel is shared with (channel_teams); every other team's it reads but cannot write (src/agent/repos.js, accessibleRepos). Code, not the agent, writes index.md and log.md and stamps generated, and a write that is not a well-formed concept is refused. For now every write is trusted and lands on main; staff review comes later, against the git history and the trust tiers OKF derives from verified.

Agent instructions. Each knowledge base may have a systemprompt.md at its root (a concept of type System Prompt). Every thread agent appends the bodies of those that apply to it, its team's first, then those of the teams its channel is shared with, then the customer's, to the end of its system prompt, read again before each model request; HTML comments in them are left out. Only people write them, whoever may edit the knowledge base, in the knowledge-base editor: agents' writes to systemprompt.md are refused, so a prompt injection cannot change what later agents are told.

Editor. Members of the admin team, and team managers for the knowledge bases of their teams and of the customers their teams own, have an editor at /inbox/kb: every knowledge base, its files, its history, and an editor for systemprompt.md and the concepts. The members of every ops team see every team's knowledge base there read-only (its files, history, thread agents, tools and MCP servers: the API answers them GETs only, and says per knowledge base whether they may change it, may), as every thread agent reads them, and the customers' their teams own the same way. A team owns the customers with a channel filed under it or shared with it (threads.team, channel_teams; ignored channels aside), so ownership follows the channels when they are re-filed or shared; src/kbaccess.js says who looks and who changes. What the members change is only their own settings, whether the agent's thinking shows in their inbox (see In the inbox, below). It saves as you type: a pause of a second (or leaving the file, or ⌘S) commits the file, authored by the person and stamped generated: { by: human:<email> }, through the same write path as the agents' (OKF checks, indexes and log written by code). A save names the blob it was written from, so a colleague's or an agent's newer version is never overwritten unseen: the editor offers to load theirs or save over it. index.md and log.md are read-only there. It asks only in the page (no native dialogs, which a debugger attached over CDP dismisses), and only when edits could not be saved or before a delete; the one exception is leaving the page (Back, reload, closing the tab) with edits that could not be saved, where only the browser's own prompt can ask.

The list of knowledge bases has the ops teams first, then the customers and vendors, by activity (the default: most entries first, then most tools and MCP servers turned on for their threads) or by name, as the person last chose in that browser. An entry is a file other than index.md, log.md and systemprompt.md; knowledge_bases.entries keeps the count, which every write updates (and opening a knowledge base in the editor, when its listing is complete), and the cron counts the knowledge bases not counted yet, a batch per run, over their whole tree.

Thread agents per channel. The editor shows, for each knowledge base, the channels it learns from (a team's: the team's channels; a customer's: theirs, in every team) and whether each one's thread agent is on; the list of knowledge bases marks those with an agent in any of their channels. The admin team, and the managers of the team that owns a channel (the team of its latest thread), turn its agent on or off there, right away (no question: it is one channel, and switches back as easily). The switch is a row in agent_channels with who set it and when. A team's knowledge base also has the team's default, On by default (agent_teams, after a question in the page, since it switches many channels at once): on, every channel of the team that nobody switched has an agent, the channels it gets later too. A new team starts with it on (the admin team's name for it in changed_by), and with the agent tools computer and extraction_rules and the global servers browser and github on for the team (those marked newTeams); the teams from before that have them as they had them, and deleting a team deletes them. A channel nobody switched has an agent when its team's default or AGENT_CHANNELS, the deployment's default, turns it on. Alert channels have no agents. Turned off, a channel's agents stop at their next message and skip their wrap-up; their drafts and what they filed stay.

MCP servers. Next to the thread agents, the editor lists a knowledge base's MCP servers: those of its team (kb-team-*) or its customer (kb-org-*), which the agents of that team's or customer's threads may call from code mode (see Thread agents). Whoever may edit the knowledge base adds, checks, edits, switches and deletes them. A server is a name, an https:// URL, what it is for (the agents are told), request headers for its credentials (sent once, sealed with TOKEN_KEY, then shown by name only) and optionally the only tools agents may call. Adding or changing one connects to it and lists its tools. Above them are the global servers, with a switch for the team (in a team's knowledge base) and one per channel the knowledge base learns from. In a team's knowledge base the per-channel switches fold, the channels switched on their own first and then by name, with a filter when there are many; in a customer's they are all there is, so they are shown unfolded.

Every page has the same header (ui/header.js, ui/header.css): a switch between Inbox and Agents (this editor: the agents' knowledge bases, channels, MCP servers and instructions) for every member of a team, Teams too for team managers, and for the admin team Orgs and Requests too, the sections of /inbox/admin, Topics (/inbox/topics, below) and COGS (/inbox/cogs, below). Each opens as it was left in that browser tab: the header keeps every page's address (sessionStorage) whenever it changes, so Inbox comes back with its team, filters, sort, unread, the open thread and "view as", Agents with its knowledge base and file, and each admin section with what was picked in it. From the inbox, what is on screen picks what the others open on: Agents and Orgs the customer on screen (the open thread's, or the one customer filtered on; Agents its knowledge base if it has one, else the team's), Teams the team picked (in the team views, for whoever may manage it). With nothing on screen for them, they open as they were left, and Agents, not opened yet in the tab, on the team's. The inbox, not opened yet, opens from a team's knowledge base on that team. (back=, the inbox's address on links made before, is still taken.) The editor's URL names the knowledge base and the file (or the history), and each one opened is a history entry, so the browser's Back and Forward step through them; going between the admin sections is one too. The editor itself is always the admin's own, also when they came from a "view as" inbox. On a phone the switch is the page you are on, with a menu of the others. Your menu, on every page, has Appearance: the color scheme, the system's (the default), light or dark, kept per browser.

The repositories are created through the Worker's Artifacts binding (one namespace per environment, helix-opspile and helix-opspile-stage) as soon as their org or team is known: when the normalizer discovers a channel, it makes sure the repositories for the channel's org and team exist, and the cron creates one for every team in D1, so a team that was added by hand gets one before it has any channel. knowledge_bases in D1 records the repositories that exist and their git remotes. Internal slugs (_unassigned) get none.

4. Fast index

  • D1 holds the queryable index: containers, threads (last activity, last speaker, classification), people and their platform identities, ops teams, and read watermarks. It answers the inbox queries, for example "everything waiting on us for my team" or "everything from this customer in the last 7 days". It keeps one piece of text: each message's first 200 characters as one line of plain text (messages.preview, written by the normalizer), and the latest one of every thread (threads.last_text, derived with the other last_* columns), so the inbox's rows show the latest reply without reading the thread. Threads indexed before it was kept (migration 0035) get it from the cron, a few hundred per run, the most recent first (src/previews.js, from the latest message's file in pile 2).
  • One Durable Object per ops team holds live read state. UI clients connect over WebSocket, read events fan out to everyone on the team immediately, and the object writes them back to D1 in batches. New messages, reactions and drafts are announced in the room of every team whose inbox has the thread (a shared channel's are in several). A message posted in the last hour that mentions people it did not mention before (a new message, or an edit that adds a mention) names them in its announcement, with who wrote it, for their notifications; a backfill, a replay or a late catch-up names nobody. An inbox on screen reloads its first page (and the open thread, if it changed) half a second after an announcement. A tab in the background does not: it notes which threads changed and catches up when it is shown again, unless a bell watches the inbox it shows (bells are checked on its reloads). The thread agent's status is not polled from the background either. A socket that was down has missed what its room announced, so when it comes back the inbox and the open thread reload, in the background tab's case once it is shown.
  • The index is derived: a messages row per pile-2 file, and a threads row recomputed from its messages on every write (last activity by a person, last speaker and whether they are staff, title, message count), so it is the same in any replay order. Alert threads use the last message of any kind, because they consist of bot posts.
  • The UI is one static page, served by the Worker at /inbox behind Cloudflare Access: facet cards (status, type, next actor, urgency, kind, customer, platform) on the left, threads in fixed-height rows in the middle, and the thread as a chat on the right (customers on the left, Adobe on the right, with reactions and images; only your own messages are blue, colleagues' are grey like the customer's). Avatars are the Slack profile picture or the Gravatar for the email; the customer's icon is the favicon of their mail domain. Reactions are kept in D1 (reactions) from Slack's reaction events and from the reactions Teams reports on each message.
  • Search is the box over the thread list (/ jumps to it): one search over the view on screen, a team's inbox or the personal view (view=personal), on every platform at once, Slack, Teams and Discord alike (GET /api/search, src/search.js). It looks three ways and adds them up. By meaning: the query is embedded (bge, with its instruction for short queries) and the CONTENT index answers with the team's message and thread vectors and those of the channels shared with it; a match needs a similarity of 0.6 and to be within 0.12 of the best one. By the words of a message: SEARCH with BM25, every word in one message (Porter stemming: "purged" finds "purge"), the recent boosted; the best scores 0.7, the others by their BM25 score relative to it. By the words of the thread: every word of the query in the title, the channel or the customer (the more of them in the title, the better), or in a fact (a URL, ticket, CVE, site, the problem). Members of a re-threaded group count for its entry, and D1 decides what the viewer sees, as for the inbox. The results take the list's place, best match first, each with the message that matched (its words marked); opening one scrolls the thread to that message. The filter cards count and filter the results, j and k step through them, and Esc or Clear brings the inbox back.
  • A click on a facet value picks it; Shift-click, or the ≠ beside it, excludes it (negative filters, as in klickhaus): picked values of a card are OR'd, excluded ones AND'd, and the cards AND'd with each other. The URL keeps both, f.<facet>=a|b and not.<facet>=c; a picked value is tinted and outlined in the accent colour, an excluded one is struck through, and a plain click stops excluding it.
  • The thread list has the personal view and a view per team the viewer is in, switched above the list (for a member of two teams: Personal · Admin · Helix). Personal (view=personal, and the default: any address without team=) has the conversations in the channels you posted or were mentioned in, whichever team they are filed under (all of their threads, not only the ones you are in; no alerts), puts the latest activity first (a person's last post, or for a thread only bots posted in its last message) and keeps only the customer and platform cards (one customer is picked at a time); filters on the cards it does not show do not apply there, and are kept for the team views. Unless the expert view is on (the user menu, the expert setting), it is the simple one: a column of customers by logo and name instead of the cards, and the list by customer until one is picked. A customer's ≠ in that column, or a Shift-click on it, excludes it as on the cards (not.customer= in the URL, and in a bell on the view): its threads leave the list, and it stays at the end of the column, struck through and out of the red badge of All customers, with or without threads, until a click brings it back. All customers drops the pick and what is excluded. Besides its cards it has one switch, Mentions (the @ at the top right of the customers; mentions=1 in the URL), keeps the threads that mention you, read or not (mentioned, all the thread's mentions, has you); with unread only, the ones the red badge counts. The customers without such a thread leave the column while it is on. Team views do not have it: their Involves card has Me (you posted or were mentioned). Its live updates come from the rooms of your teams and of the teams those channels are filed under; in the rooms of other teams, only about those channels (scoped sockets, src/room.js). Someone on no team (no-team mode, SIGNIN_DOMAINS) has only this view. A team's view (team=<slug>) puts what needs you most first and has every facet card. First comes the status (unseen and critical, unseen and hot, you, waiting too long, customer silent, team on it, the rest); within a status, the score of ui/rank.js, boosted trees over what the inbox knows of a thread (how long ago, who is waited on, urgency, resolved, kind, …) learned from Clef's pairwise judgements of production threads, from what people open in the list and from which customers get answered first (npm run rank, see the README), plus how close the thread is to you (mentioned you, you posted, 🛟 or 👀 by a colleague). Opening a thread by clicking it in the list records the pick, with the threads on screen and their features, in rank_events (POST /api/rank/pick, migration 0040), for that training. A facet value with threads you have not read has its label in bold, and a red badge counts the messages that mention you and that you have not read yet (Slack's and Teams' mention count); a thread's "N replies" is bold and blue while it has replies you have not read.
  • Channels a message names (#name, or #C… where Slack sent the id without the name, as it does for bots) are resolved when the thread is read (src/channelrefs.js, the channels of /api/thread), so channels opspile learned about later are named too: the thread shows #name, linked to the inbox that has the channel (the view on screen if it does, else your team's, else Personal) with the channel's customer picked, or to the channel on Slack or Discord when it is in none of yours. Inbox titles and last texts get the names of the ids. Links to a channel (not to a message in it) are resolved the same way, on every platform (src/channelurl.js, the channel_links of /api/thread): a Teams channel's link in a Slack message, or a Slack channel's in a Teams post, opens the channel's threads in the inbox, and a bare link shows #name. A link to a channel that is in none of your inboxes stays a link to its platform.
  • The list stays put while it is the same list (the same view, filters, unread only and Mentions): reading a thread neither moves nor hides its row. Every thread it has shown stays in it, read or not (unread only is "unread when the list was opened"); the rows in view, and those scrolled past, keep their order, and the rows below them are sorted as the view sorts. Threads that come in since (new ones, or with unread only, read ones with a new message) are added at the end of a team's list, except critical and hot ones, which come in right below the rows you have seen; in the personal view, threads with activity since that you have not seen in it come to the top. Another filter or view, unread only, or a click on the view already shown sorts it afresh; the thread open then stays, if it passes the filters.
  • Notifications are shown by the browser while an inbox is open (a tab, or the installed app, also in the background), through the service worker, so that a click opens the thread in the inbox already open, or in a new one. There are three kinds, settings of the person in D1 (person_settings, GET/PUT /api/settings), so every device has them: a message that mentions you (on unless turned off in the user menu; the room's announcement says whom it mentions), a new message from someone else in a thread of yours (you posted in it, were mentioned in it or put 👀 on it, in any thread of its group; on unless turned off, and the announcement says whose thread it is; bots' messages and edits do not count, and a message that mentions you notifies as a mention), and a bell on a view. The bell above the thread list turns notifications on for the view on screen with its filters (picked and excluded); a thread of that view that gets a message you have not read then notifies you. The inbox listens to the rooms of the teams its notifications need (all of yours, for mentions, your threads and Personal), checks a bell on the inbox on screen when it reloads, and fetches the first page of a belled view that is not on screen when its room announces activity. The menu lists the bells, each opening its view or turning it off. Inboxes in several tabs show one notification per thread, and none for the thread you are reading, nor while viewing as someone else. The first visit asks once (a toast) whether the browser may show them; browsers only ask on a click.
  • The toolbar is the same on the inbox and the admin pages: the pages and label review on the left, and on the right whoever is signed in, with the view on screen under their name. Their name opens a menu with "View as" (admin team; while it is on, a chip next to the name says whose inbox this is and leaves it in one click), the notifications, the keyboard shortcuts, and Log out, which is Access's /cdn-cgi/access/logout. Live updates are not shown while they work; the toolbar says "Reconnecting…" when a WebSocket has been down for a few seconds.

5. Stats (klickhaus)

Not a pile of its own but a copy of pile 4 for statistics: one row per message in ClickHouse, for klickhaus' opspile view (opspile.html), which has the facets, the chart and the long time ranges of klickhaus' other views (src/clickhouse.js). Every thread in D1 is exported, of every team and kind (channels the rules ignore never get there).

  • No text. A row is the message's metadata (when, which platform, channel, team and the teams it is shared with, org; its author's kind, relation, name, mail domain and ops teams; whether it was edited, its reactions, mentions and files) and its thread's classification: the title, whether it is resolved, who it waits on, Clef's label (who acts next, kind, urgency and its level, resolved_p) and the facts (tickets, sites, CVEs, URLs, the problem). Its url opens the thread in the inbox's personal view.
  • As the inbox shows it. The label, title, resolution and facts are the inbox entry's: a re-threaded group's latest member's label (with reviews), its first thread's title, the facts of every member, and resolved by a staff ✅ on a member's root (until a customer writes again) or else by Clef (resolved_p ≥ 0.5). Who it waits on is the inbox's waiting_on: Clef's next actor, except that a thread where Adobe posted last never waits on us, an unlabeled one waits on us when the customer posted last, and a 🙏 after the last message leaves nothing pending. The urgency level is the inbox's bucket (none is low). So exporting a thread exports its group; a group of more than 100 threads exports only the threads that changed and its first.
  • One table, rows replaced. opspile_messages_raw is a ReplacingMergeTree keyed by (timestamp, thread, message_id); _version is the export's time, so the latest export of a message wins, and _deleted makes it a tombstone: a deleted message, and every message of a thread re-filed as ignored. klickhaus reads it through a view with FINAL. Rows are kept 18 months.
  • Written by the Worker. An export reads D1 only, and POSTs INSERT … FORMAT JSONEachRow with async_insert=1&wait_for_async_insert=1 to CLICKHOUSE_URL as a user that may only insert there (credentials in headers). What D1 did not keep before (the author's kind, edited, the number of files: messages.author_kind, edited, attachments, migration 0041) is read from pile 2 once and kept; the normalizer writes it for every new message.
  • When. A message written queues an export of its thread three minutes later, after the label job; the label job queues one when it writes a label, and so do re-threading, a reaction and a re-file. A job queued before the thread's latest export is dropped, so a burst exports once. Each thread remembers what was exported (threads.exported_through, the row's updated_at, and exported_at); the cron enqueues up to 1,500 threads a run, newest first, whose rows are behind: a message written since, a label, a review or re-threading since the export, or a re-file (which clears the watermark). That is the backfill too, and the retry: a failed insert leaves the watermark as it was. Without the CLICKHOUSE_* bindings nothing is exported.
  • Not re-exported by itself: a renamed org or a changed type, a partner domain, team membership and shared teams. A thread's rows take them on with its next export.

In the inbox, the header has Stats for members of the teams in KLICKHAUS_TEAMS: it opens KLICKHAUS_URL/opspile.html in a new tab, filtered to the customer and team on screen.

Read state

Read state is a watermark per person per thread: "has read up to message timestamp X".

  • Unread for me: the thread's last message is newer than my watermark.
  • Seen by somebody else, not by me: another member of my team has a watermark at or past the last message, and I do not.
  • Unseen by the whole team: nobody on the team has a watermark at or past the last message.

Neither platform exposes other users' read state for channels, and each exposes the signed-in user's own state only coarsely, so opspile owns read state. Most of it is inferred rather than clicked:

Signal Effect
Person posts a message in the thread Their watermark moves to that message
Person reacts to a message Their watermark moves to that message (Slack reaction_added; Teams reports reactions on the message)
Person opens the thread in opspile Their watermark moves to the last message
A message is edited, or a reaction is added by someone else No change; edits never make a thread unread
Bot or system message Stored and shown, but does not count as new activity; a system notice is no reply either, and titles a thread only when nobody said anything in it

Status piles

The inbox sorts every thread into one pile, most pressing first: critical (critical urgency, nobody at Adobe has seen it), hot (high urgency, unseen by Adobe), you (you posted or were mentioned, and the customer spoke last or you were mentioned), waiting (waiting on us longer than 15 min / 1 h / 3 h for critical / high / normal), silent (waiting on the customer or vendor longer than 30 min / 1 h / 1 day: follow up or close; for normal and low urgency, Saturdays and Sundays in the viewer's time zone do not count), team (your team is in it and all is well) and quiet. Who a thread waits on is Clef's next_actor, except that a thread where Adobe posted last never waits on us (the label can be stale or wrong), and an unlabeled thread waits on us when the customer posted last; the API sends it as waiting_on. Admins can look at the inbox as another person, read-only.

Attention

Read state answers "who has seen this"; attention ranks what that means for the person looking, most pressing first:

  1. Nobody at Adobe has seen it: no staff member at all (in any team or none) has posted, reacted or opened the thread since its last message. Staff become people in D1 the first time they post, so their reads count even outside the ops teams.
  2. You were mentioned after you last read the thread (a platform mention, or your full name in the text).
  3. A teammate was mentioned after they last read it.
  4. Nobody on your team has seen it.
  5. You have not seen it.

Alert threads skip the first level: they are bot posts and nobody outside is waiting.

Backfill

A reply whose root has never been seen (it was posted before ingestion started, or before the bot joined) triggers one request for the thread's history: conversations.replies on Slack, the message and its /replies on Graph. Every message lands in pile 1 marked via: backfill and is normalized like a live event, so pile 2 cannot tell them apart. The backfills table makes the request idempotent. An error that stays (the bot is not in the channel) ends a request. Any other leaves it open: the queue retries it, and the cron sends out what is still open after an hour, an hour apart, until the thread has been quiet for a week.

Labels and facts

Two minutes after a person posts, the thread is labeled (later posts reuse the same label job, and an unchanged thread is never labeled twice):

  • Clef answers four questions: who acts next (us, customer, vendor, nobody), what kind of thread it is (incident, security, question, request, access, people, meeting, other; alerts are alert without asking), urgency (0–3) and whether it is resolved. The state it reads is the thread as text, with every author marked as Adobe staff or customer. access is about permissions and getting into tools; people joining, leaving, being introduced or away are people; scheduling and sharing a call's notes are meeting.
  • In the same call, Clef answers a yes/no question per topic (src/label.js TOPICS): authoring and content, sign-in and permissions, project plan and go-live, caching and delivery, domains and certificates, performance, preview and publishing, SEO, URLs and redirects, code and deployment, blocks and front-end code, errors and traffic, limits and quotas, security and compliance, roadmap and feedback. The kind says what the thread asks for, the topics what concern it is about, and the tech stack (below) which products: an incident and a question can both be about caching, on Akamai or Fastly. Topics name no products. A thread keeps the topics at 0.55 or above (labels.topics): the best single cut on 225 threads labeled by hand (macro F1 0.82; the ten topics before, kept at 0.8, had 0.56 and left 59% of the threads with no topic, these 31%, the hand labels 24%). The list came from the topic map. Clef's answers move a little with the other questions of a call, so the four questions above and the topics are asked in a call of their own, as they were measured, and the tech stack in calls of its own: an entry added or changed does not move the topics' answers. Measure again when the questions or the topics change. A label from before shows golive as project, and commerce and integrations, products now, not at all, until the thread is labeled again.
  • Extraction rules (src/extract.js) find facts and detect technologies: regular expressions, or scripts, which are configuration of the ops teams (extraction_rules), not code. The admin team's apply to every channel, any other team's to the channels it owns, on top. The admin team's start as migration 0043 wrote them: CVE ids, ticket keys, Edge Delivery org/site pairs (in hostnames, including aem.network and aem.reviews, da.live paths and admin API paths), the admin API operation, AEM Cloud Service pNNN-eNNN environments, GitHub repos, aem.live documentation pages and tools, configuration files (cdn.yaml, redirects.xlsx, …), HTTP status codes of failures, response headers that name a request or an error, support cases, and URLs (not chat links and attachments); and detections: an Edge Delivery or Document Authoring link, an AEM Cloud Service, Assets or App Builder host, an Akamai error reference, a Cloudflare ray id, a Fastly or CloudFront header, another git host. A rule is a pattern with a value template ($2/$1), a case and an exclude pattern, that writes a fact of its field or detects technology field. A regular expression cannot be stopped once it runs (and a Worker's clock stands still while it does), so a pattern that repeats a group that itself repeats without bound, as (a+)+, is refused when it is saved and when it runs: that shape backtracks without end on text that almost matches. A script is an ES module whose default export gets the thread and returns [{ field, value }, { tech }]; it runs in a Dynamic Worker of its own (LOADER), without network access, within two seconds. Rules are edited on the Agents page (Extraction rules of a team's knowledge base), with the MCP server's tools (list_extraction_rules, save_extraction_rule, …), and by thread agents where the team turned on their extraction_rules tool (see "Agent tools").
  • A language model (llama-3.3-70b-instruct) writes a title of at most eight words. For an alert group it names the specific failure (the error or failing request), once per group. The title leaves out what the inbox shows next to it: the customer, the channel, the platform, the kind of thread and Edge Delivery itself. When the model still names the customer or the product, it is asked once more, told which names to leave out, and its second answer is kept whole (the names are never cut out of a title, which leaves broken phrases such as "Headless options in without App Builder"). When the model finds nothing to name ("None", "No title available"), the thread has no title, and the inbox shows its first words.
  • Clef also answers yes/no questions about the tech stack the thread shows (Edge Delivery, Document Authoring, Universal Editor, the CDN, the content source, …): the technologies of the teams' configuration (tech_stack), each asked as "Does this thread show that the customer uses ?". The admin team's are asked in every channel; a team's own are asked in its channels too, and one with the id of an admin team's replaces it there (turned off, it is not asked there). Clef gets at most 24 questions per call; the rest are asked in calls of their own, at the same time. The channel keeps the highest probability any thread showed (a rule's detection counts as 1), so every thread of a customer shows their stack; the label keeps what the thread itself showed (labels.tech, migration 0059), so its header can name that first. The inbox shows a team's own name for a technology in its inbox and threads, the admin team's elsewhere. An entry also has aliases (tech_stack.aliases, migration 0057): the names it goes by in threads ("AEMaaCS", "IMS Org", "EdgeWorkers"), which match product names to it and are not asked of Clef. A feature people ask about by its own name is an entry, not an alias: the AEM Sidekick was an alias of Edge Delivery until migration 0061, so Clef never asked about it, the Topics page drew it as Edge Delivery, and a thread about it showed nothing of its own. Every entry is part of a product (tech_stack.product, migration 0056): the Adobe product it belongs to, by its name on Adobe's product list (Edge Delivery, Document Authoring, the AEM Sidekick, Universal Editor, Experience Workspace, AEM as a Cloud Service, MSM and Edge Functions are Experience Manager Sites; Dynamic Media is Experience Manager Assets; CX Enterprise Coworker, Brand Concierge, Brand Visibility, Journey Optimizer, … are their own), or a category for other companies' (CDN, Code hosting, Consent, …). Clef is asked about each entry; the product groups them, on the Agents page and the Topics page, where an entry is a feature of its product. Migration 0056 added what the product census (Clairic over all 822 channel threads with a person's message) found the threads naming that no entry covered, and the Adobe products they named that had none; it also put the names people use into the descriptions of others (Google Tag Manager and Tealium are their own entry now, Adobe's analytics another), where nobody had changed them since migration 0043. They are what Clef's topics about commerce, assets, Sites Optimizer, integrations and the inbox would have been.
  • The inbox filters by kind, topic and tech stack (the channel's, at 0.8 or above), and so do the MCP server's list_threads and the inbox's WebMCP filter_threads.
  • Clairic extracts the loose facts: the problem in the customer's words, the affected URL, the error, a deadline and a go-live date. Only values at or above a confidence threshold are kept (0.8, and 0.6 for the free-text problem). In the same call it lists the products the thread names (product facts), for the topic map to match against the tech stack. Clairic lists nearly every name in the state, so productFacts (src/label.js) keeps few of them: only its spans count (counted in code points, as Clairic counts; a string here counts UTF-16 units, one more per emoji), not those in the header or an author line, not a mention (@Name) or a piece of a word, not the names of the thread's people and customer, nor generic words (src/products.js GENERIC). Of the rest it keeps what the tech stack covers, one name per entry, and at most six it does not, the most named first, when they look like the name of a product (productLike: a proper name of at most four words, not an id, a variable, a header, a host name, a short acronym, a possessive, a phrase that ends in a lower-case noun, or a word the thread also writes in lower case). The list's one confidence says nothing (it shrinks as the list grows) and is not used. On the census's 822 threads: two names a thread instead of six, two in three of them names of known products (two in five before, and a sixth generic words), the names nobody knows yet a third, and 83% of the products a hand-made alias table finds. A product name says the thread mentions it; Clef's tech questions still say what the customer uses.

Label review. Members of the admin team can review labels in the inbox: a queue of current labels nobody has reviewed, least confident first, and per question the right answer. A review is calibration data (Clef's agreement rate and most common corrections per question) and overrides the model's answer for that label version; when the thread changes and is labeled again, the model answers again and the old review only counts as history.

Topic maps

Clef's topics are asked: each is a question someone wrote, and a thread about something nobody asked about gets none. A topic map (src/topicmap.js, the Topics page) finds what the threads are about from what was said in them instead, for the admin team, to see the common themes, who works on which and for whom, and where Clef's topics miss: what to ask Clef next.

  • The threads: channel threads with a CONTENT thread vector, at least three messages, and not a meeting or people thread (whose vectors say "let's meet" more than what about). Their vectors are centred (the mean taken away, then of unit length again): bge's vectors share a large common part, and without it k-means makes a few big clusters and many of one thread.
  • k-means (spherical, cosine; k-means++ seeds, the best of four runs, seeded so the same threads make the same map) into about one topic per twelve threads, at most 30. A few hundred threads take a quarter of a second.
  • One call names every topic (two to five words) and says in a sentence what its customers need: it reads all of them at once, each as the titles of its fifteen most typical threads (nearest its centre), the Clef topics and the products they have, with how many threads have each, so the names tell the topics apart. The model is the agents' (Claude on Bedrock, through AI Gateway, counted for COGS) where agents are configured, else the language model of the titles. A name that is vague (made of words such as "issues", "management", "content delivery"), too long, or another topic's, and a topic it did not name, are named on their own from the titles. No name names a customer; one given twice gets a number.
  • With them, per thread: the Adobe people who posted (staff, by people.id, with their messages), the customer (orgs), Clef's topics (labels.topics), the tags on the page, and the products Clairic named. Each product is matched against the tech stack (every team's enabled entries, src/products.js coveredBy): an entry covers it when its id, name or one of its aliases is the product's; or is in it as words, where the product is written as a name ("Akamai Property Manager" is Akamai's, "en-to launch" is not Adobe Launch's); or its description names it where it writes a name, cut at commas, "or", "such as" and brackets ("a consent manager such as OneTrust or Cookiebot" names OneTrust and Cookiebot, not "consent manager"). A word alone is no name. A covered product is named as its entry, and drawn as the product the entry is part of (its features listed when it is picked); one no entry covers is a gap, named as most threads write it. "AEM" and "Adobe" in front do not count, except where a name is generic without them ("AEM Author"), and generic names (API, CDN, JSON, Slack, …) are left out.
  • The cron builds one when the newest is a week old, and the admin team with Rebuild; a row of topic_maps each (the map as JSON), the newest five kept.
  • The expert finder reads the newest (src/topicindex.js): which topic each thread is in and who worked each topic's threads, so it can tell a thread agent which topics a person's similar threads are in, and list who works a topic (list_topics).

The page draws the map as one force-directed graph (its own, after d3-force's model): a node per person, topic, tag and customer, as big as its threads, a line between two that share threads, thicker the more (with the topics shown, a line joins a topic to the others, and a person to a customer; the topics are the hubs). One kind shown alone is joined to itself: two people, tags or products by the threads they share, and two topics or two customers, which never share a thread, by the people they share (as many people as the threads a line needs, each with that many threads in both). The filters are a team, the kinds shown and the threads a line needs. A node picked shows what it is joined to and its threads, each a link into the inbox; without a pick, the panel lists the gaps first (Not in the tech stack: the products no entry names, by their threads; those two customers or more name, then, folded, those of one customer, which are as likely a brand or a system of theirs, and a count of the names of one thread; adding one on the Agents page, or its name to an entry's description, makes Clef ask about it), then the topics with the share of their threads Clef tagged at all, the fewest first on a click: the topics Clef rarely tags are what its topic questions miss. A gap is drawn as a hollow ring.

Replying

Replies are written in the inbox and sent as the person, never as a bot: each person connects their own Slack account (user scopes chat:write and reactions:write) and Microsoft account (delegated ChannelMessage.Send, which also allows reactions) once. The account must be theirs (the same person in D1, or the same email); tokens are encrypted with TOKEN_KEY, Microsoft tokens are refreshed when they expire. A reply goes to the latest platform thread of an inbox entry and comes back through the normal ingestion path, like any other message. Replies are written in Markdown; on Slack they are sent in mrkdwn (**bold** as *bold*, links as <url|label>, bullets as •).

A reply shows in the thread at once, as it will look once sent (the person's name, avatar and blue bubble from their latest message on the platform), with a thin bar under it, as in iMessage. The platform takes a reply in about a second (POST /api/post); there is no real progress to report, so the bar eases towards 90% and completes and fades once the post returns. The bubble then stays until the message comes back through ingestion (some ten seconds later: the Slack event, the normalize queue's batch, the live update), and the message takes its place unchanged. Replies to a thread go out one after the other, in the order they were written. One that failed says why, with Retry and Discard; it stays until then. A page that lost the answer cannot know whether the post went out, and a platform would take the same post twice, so a reply carries a key, the same on every retry (post_keys, migration 0049, src/postkeys.js; the cron deletes keys after a day). A retry that finds the first answer gets it, without posting. One that finds the first try unfinished says the reply may have been sent, and offers Send again, under a new key, so that sending it twice is the person's decision. A reply the platform refused is not kept, so its Retry posts. The MCP tools send no key. Leaving the page while a reply is still on its way (a reload, closing the tab) asks first, with the browser's "Leave page?": the bubble lives in the page only, and the post may be cut off.

Emoji names. A :name: in a reply goes out as the platform knows it (src/shortcodes.js): Slack gets Slack's name, Teams and Discord the character. Another platform's name or spelling is Slack's (:facepalm: is :face_palm:, :thumbs-up: is :thumbsup:, from the GitHub and Discord names in src/emoji.js and by leaving out separators, src/emojinames.js), and a workspace's own emoji (one somebody reacted with, and on Slack every one of emoji.list, bot scope emoji:read, remembered per isolate for ten minutes) stays. For any other name, Clef picks the closest emoji from at most 16 candidates (the most alike by name and CLDR keywords, then common ones), at most three names per reply, within 2.5 seconds; it needs a probability of 0.5, else the name stays as typed. On Slack, Clef is asked only once emoji.list answered: without it (no emoji:read, Slack failing; tried again after a minute) a name that is not a standard one may be a custom emoji and stays as typed. Answers are remembered per isolate. Code (fenced blocks of three or more backticks or tildes, code spans of any number of backticks) and addresses (bare, autolinks in <…>, link destinations) are left alone. Edits (POST /api/edit) go the same way. The inbox shows every platform's names as emoji the same way.

Custom emoji in the inbox. GET /api/emoji/custom gives the page what emoji.list says, aliases followed: a custom emoji shows as its image, in messages, titles, previews, reactions and the thread list's moods, and an alias of a standard emoji as that emoji. The images load straight from Slack's CDN, which wants no sign-in; one that fails shows as :name:. Reactions of other platforms never use them: Teams stores like, which a workspace may have an emoji of its own for. emoji.list covers the workspace of the bot token only, so the other side's emoji in a Slack Connect channel have no image. For such a name, and any other nobody has, the page asks GET /api/emoji/closest: the emoji Clef picks as in a reply, three names a request and 40 a page load, shown with a dotted underline and a tooltip that says what it stands for. Only once emoji.list answered, since the name may be on it, and only for a :name: on its own, so urn:aem:sites stays as it is. Previews in plain text, like notifications, keep the name.

Emoji suggestions. Typing : and two more characters of a name in the reply box (or in a message being edited) suggests up to eight emoji, as typing @ suggests people. First, at once, those whose Slack name, or another platform's name for them, has what was typed: the whole name, then names it starts, then names with a part it starts, then any, one name per emoji (matchEmoji, src/emojinames.js), in a Slack thread the workspace's custom emoji too. When that leaves room, Clef fills the list (GET /api/emoji): it ranks the candidates above by how likely each is what is being typed, with no threshold, and the list takes them in that order, skipping the emoji it already has, until it is full. Picking one writes :name: . A ranking is the same for everyone: the Worker keeps it in the data center's cache for a day, and the browser keeps it too.

Files. A reply can carry files (src/attach.js): the 📎 button beside the reply box, files dropped anywhere on the open thread (which shows where they go, or why they cannot), or an image pasted into the reply box (a screenshot, named for when it was taken). They wait as thumbnails and chips above the box, each with an ×, and go in the same message as the text, up to 10 at a time. What a platform takes is a rule the Worker sends with GET /api/me, which the page checks before a file is added and the Worker again before it uploads:

as the person as the opspile app
Slack any file up to 20 MB, 25 MB in all, with the user scope files:write (files.getUploadURLExternal, then files.completeUploadExternal in the thread, the text as initial_comment) the same, with the bot token (bot scope files:write)
Teams images (PNG, JPEG, GIF) inline as hosted contents of the reply, up to 3 MB in all (Graph takes 4 MB of base64); other files and larger images uploaded to the channel's SharePoint folder and attached by reference, with the delegated Files.ReadWrite.All images inline only (Bot Framework data URIs, up to 1 MB): Teams lets bots send other files in personal chats only
Discord none: opspile does not post as the person any file, 10 MB in all, in the message (files[n])

The scopes for uploads are asked for when they are first needed: a person whose token lacks them is told why when they attach a file, with a link that connects the account again with file access (/api/auth/<platform>/start?files=1); a Microsoft token is refreshed with the scopes it was granted. Where nothing can be attached the button is dimmed and says why. The reply shows its files while it is being sent, and once it comes back through ingestion, as any message's files are shown.

The one exception is posting as the opspile app (POST /api/post/app, src/postapp.js, used by the MCP tool post_as_bot and the WebMCP tool of the same name): the same post as a thread agent's post_message (see "Agent tools"), and allowed exactly where that is, in channels whose Post messages switch is on (the channel's, else its team's) on a platform whose bot is set up, for staff of the thread's teams. The switch says whether the app speaks in a channel at all, whoever writes the text. The post names no person (who wrote it is kept in app_posts only). The people picked with @ (or that the shared draft it sends mentions) are mentioned as in a reply: <@U…> on Slack, in text and in the markdown block, which keeps the id form as a mention; <@id> on Discord, whose allowed_mentions ping those people and nobody else (no @everyone, no roles); on Teams an <at>Name</at> in the bot's text with a mention entity by the person's Entra object id, which the Bot Framework takes. A picked Teams bot stays plain text there: Teams gives no bot what another bot writes. It comes back through ingestion by the app, and a thread agent there is told about it like any message.

Editing. A message sent from opspile, or from the platform, can be edited in the inbox the way it was sent (POST /api/edit, src/edit.js): hovering one of your own messages shows ✎, which turns the bubble into an editor (Enter saves, Escape cancels, @ suggests people as in the reply box). Your own messages are edited as you, with your token: Slack chat.update (chat:write covers your own messages) and Teams PATCH on the message, which needs the delegated ChannelMessage.ReadWrite; that takes the tenant's admin consent, so it is asked for only once TEAMS_EDITS says it was given. A message you posted as the opspile app is recorded in app_posts (user state, like drafts) and is yours to edit as the app, where the app may still post: Slack chat.update with the bot token, the Teams bot's update activity, Discord PATCH with the bot, with mentions as when it was posted. Nobody else's messages, and nothing on Discord as a person, are offered. The accounts the message mentioned (@Name in its markdown, one person by that name, by their account for the thread, picked as for a thread agent's post: src/post.js mentionsByName) stay mentioned. Nothing is written to the piles: the edit comes back through ingestion like every edit made on the platform, and the inbox shows the new text and "(edited)" meanwhile. A Slack change without edited (a bot's chat.update) that changes the text counts as an edit too.

Discord threads are read-only in the inbox: Discord lets only bots post and react through its API, and opspile acts as the person or not at all. They have a link to reply in Discord, no reply box, reactions or ✅; Discord's own ✅ by staff still resolves them.

Mentions. @ in a reply suggests the people opspile knows on the thread's platform, and on Teams also the bots of the thread's Teams team (bot_teams in D1): the bots installed there, which the cron lists from Graph (installedApps) for every Teams team with a channel in D1, and any bot that posted, or was mentioned, in one of its channels. Bots people used come first. A bot nobody has mentioned in a team yet can be mentioned from the inbox all the same; an @name typed without picking it goes out as plain text, which the bot never receives. A picked bot is sent as an application mention, which the bot receives exactly like a mention typed in Teams, from the person who wrote the reply. That is how a team member is added from the inbox: @<admin bot> add someone@example.com asks the bot that manages the Teams team, which answers in the thread (and in turn comes back through ingestion). A bot is mentioned by the name people use for it (handle, the mention text), which can differ from the name it posts under; for an installed bot that is the app's name, which is what Teams writes when someone mentions it.

What follows the @ matches, in any case and without accents, the start of any word of a person's name, of their other names (identities.aliases: the Slack login and display name, the Discord username) or of their mail address before the @ (src/mention-match.js). People outside Adobe who posted in the thread come first. Then come the Adobe people who know what the thread is about (src/expertise.js): the thread's CONTENT vector (a re-threaded group's mean; its title embedded while it has none yet) is matched against the staff messages of every channel, the 50 closest that are no less than 0.65 similar and within 0.15 of the closest, and each author gets what their five closest have over the weakest that counts (their messages on every platform together, when D1 knows them as one person). The viewer's own messages do not count. The affinities are kept for ten minutes per version of the thread's vectors, as every typed letter asks again. Then anyone else who posted in the thread, the members of its channel, and everyone else; among those, the higher affinity first, then a match at the start of the name before one further in, then staff, then by name. The suggestion says "Adobe · related" for such an expert. A person with several accounts is suggested once, by their account for the thread (above, "One account a person"), and a mention of another of theirs goes to that one. The members are channel_members: Slack's conversations.members (on Teams, the team's members from Graph, where the app may list them) and whoever posted in the channel (on Teams, in any channel of the team). Opening the suggestions lists a channel's members when that is a week old, and looks up the members opspile has never seen (users.info), so a colleague who is in the channel but never posted anywhere opspile reads can be mentioned too.

Mentioning someone who is not in the channel. Slack notifies nobody of a mention in a channel they are not in, so a Slack message the inbox posts first adds the colleagues it mentions to the channel (src/invite.js): as a person (POST /api/post, a reply or a new thread), as the opspile app (POST /api/post/app) and from an agent's post_message. Only Adobe people (staff on the account or its person), not bots, not the one who posts, and up to ten a message: anyone else may be a customer, and is never put in another customer's channel. The opspile Slack app adds them, one conversations.invite each (bot scopes channels:write.invites, and groups:write.invites for private channels), so nobody has to connect their account again; Slack answers already_in_channel for whoever is in already, so no listing of members is read first. The post goes out either way, after the invites. The answer says whom the bot added (invited) and whom Slack would not add (uninvited, with Slack's error: in another workspace of the grid, a guest); the page says so in a toast, and the agent and the MCP client in their result. A call that fails as such (the scope missing, the app not in the channel) refuses nobody and is only logged (route: invite). Teams adds people to a team through an approval flow of its own, and a message written in Slack itself is left to Slack, which asks there; a thread agent still points out a mention of someone not in the channel (src/agent/members.js).

The suggestions that open with the @ bring along everything they are picked from (pool of GET /api/mentionable: the accounts of the workspace, one a person, each marked as in the thread, in the channel or by its affinity, and the bots), and the page ranks what is typed after it itself, with the Worker's own suggestMentions (in the markdown bundle), so completing a name asks the Worker nothing. Until the pool is there, each typed letter asks the Worker, as before. The page keeps a thread's pool for the page's life, and asks for it again in the background once it is two minutes old, or when a message came in in its channel since (the room's thread event: it may be an admin bot's confirmation that someone was added, or the platform's notice of it); the suggestions follow the new pool when it comes. A name the kept pool has nobody for asks the Worker at once, when the pool is more than five seconds old. The Worker, for a name that matches nobody, lists the channel's members again when that is more than 30 seconds old, and waits for it: someone just added who never posted is not in D1 until then (a Teams notice that they were added names them, and so records them, but not every channel gets one), and is in the answer when the listing found them. The Worker reads the thread's place first, then whether the viewer may see it and all the suggestions need at once: one round trip to D1 after the place, not up to four.

The thread's header. Line 1 is the customer's icon with the platform's on it, the title, and the channel's name at the end where the line has room for it. The icon opens the thread on the platform; a re-threaded group lists its posts instead (the first and the latest eleven of a large one), each with its root's link. Hovering it names the customer and its type. Line 2 has two dropdowns, each named by what matters most for the thread:

  • Tech stack: what the thread shows (labels.tech of its members, the likeliest first), then the rest of the channel's (channel_tech), then Clef's topics of the thread. A thread labeled before labels kept their technologies counts those it set the channel's highest probability for (channel_tech.thread), until it is labeled again. The dropdown is named by what the thread shows, else by its first topic, and only then by the channel's likeliest: that is the customer's, and a da.live link in one thread would otherwise name every other thread of the channel Document Authoring.
  • Links: the pages mentioned in the thread (Clairic's affected_url first, then the rules' url facts), then each Edge Delivery site's production URL (main--<site>--<org>.aem.live), then links to documentation and code (aem.live, github.com, adobe.com hosts without --), then tickets, CVEs (linked to cve.org) and support cases. The first one is the dropdown's name and opens with a click; ▾ lists them all.

A dropdown is left out when it has nothing to list. Whether the thread is resolved is not in the header: the ✅ on its root says so, and so do the list and the Send button's menu. After the dropdowns comes whom a thread agent suggests pulling in ("Agent suggests Lena Berg", below), when one did; the expert finder's own ranking is not shown.

Expert finder. Adobe colleagues worth pulling into a thread who are not in it yet, each with why: "knows about this topic", "typically responds in about 4 hours" (GET /api/experts, src/experts.js). The inbox does not show its ranking: line 2 of a thread's header named the top one ("Lena Berg knows about this topic"), with a ▾ list of up to eight, each with ✕ for "not the right person", and that was taken out, because its picks were mostly wrong and made the inbox look less capable than it is. The header shows only what a thread agent picked, for a reason (suggest_expert, below): it asks GET /api/experts?suggested=1, which reads those suggestions and ranks and records nothing. The ranking is left for the thread agents (list_experts) and the API; @ mentions rank people by what they know (src/expertise.js, above) on their own.

It speaks only when someone is needed. Once Adobe has the thread in hand (three messages of staff in it, IN_HAND: a first answer may only say who will look), its own people are the ones to ask, and the most active in the channel are mostly who is in it already: GET /api/experts then lists only an agent's suggestion, if there is one. The exception is the person who was mentioned and has not posted since, who may be looking for whom to hand it to. The list is still computed and recorded (held, below), for the training to see whom the team pulled in anyway. A thread agent's list_experts always gets the full list.

Five signals add up, each scaled to its best person: the threads someone worked in this channel and for this customer in the last 180 days (D1; the customer's counts half, as it mostly repeats the channel's), the staff messages closest to the thread by meaning (CONTENT, as for the mention suggestions) and by words (SEARCH, the title with BM25, for messages that have two neighbouring words of it, plural or not, neither of them as common as "is": a search for any word lets one word like "media" or "dev" say someone knows the topic). On 60 production threads where a customer asked and staff answered, looking only at what was there before the question, the first to answer was in the top 3 of 36 of them and in the top 5 of 38; the vectors alone had 22 and 27, channel history alone 33 and 33. Labels' topics changed nothing and are not used. Asking for neighbouring words, on 70 such threads, moved the first to answer from first place on 13 of them to 18, from the top 3 of 25 to 28 and from the top 5 of 28 to 31; any two words of the title got 12, 28 and 32, and no words at all 18, 27 and 30.

The fifth is similar threads: the channel threads of any channel closest to this one as a whole (CONTENT thread vectors, similarThreads in src/expertise.js, those no less than 0.6 similar and within 0.08 of the closest), and each Adobe person who posted in them gets what their three closest have over the weakest that counts. The meaning signal only sees what someone wrote: whoever answered a customer's question with "is this still happening?" said nothing about the topic, and only the thread as a whole says what it was about. On 65 production threads (August to October, a customer asked without mentioning anyone from Adobe, Adobe answered; only what was there before the question, the root message's vector as the thread's, and without the words, which cannot be replayed), the first to answer was first, in the top 3 and in the top 5 of 17, 23 and 28 of them before, and 15, 27 and 30 with it; on the 48 of those where they had not worked the channel before, 4, 8 and 11 before and 2, 11 and 13 with it. Someone whose similar threads count says so ("worked on 2 similar threads"), with how many, when they last posted in one and the topics of the topic map those are in (similar); the list says which topics the thread's closest threads are in (topics, for the agents: GET /api/experts answers without them).

A thread agent can say what the thread is about (list_experts with about), when its title and first message say it vaguely (a customer's name for a feature is rarely the product's): the meaning and similar threads go by that sentence instead of the thread's vector, embedded as a query (with bge's Represent this sentence for searching relevant passages: in front, QUERY_PREFIX: without it, a short sentence is closest to what every thread has in common), and the words by it instead of the title. On the 65 threads with their titles as about it ranked about as well (15, 23 and 30); on a vague one checked by hand, the colleague who had answered the same question in another channel came first with it, and was not listed without it. list_topics gives an agent the topic map itself: the topics, with what their customers need and their size, and for one of them the Adobe people who worked its threads and could be pulled in (topicExperts, with the same exclusions as the list). Topic names never name a customer, so they may say why someone fits to an agent of any team (src/topicindex.js reads the newest map, at most every ten minutes per isolate).

The list leaves out the viewer, whoever posted or is mentioned in the thread (what is said in the thread itself does not count either), anyone who has not posted for 90 days, and anyone without an account on the thread's platform, who could not be mentioned. Someone with several accounts there is listed once, by their account for the thread ("One account a person"), with the signals of all of them.

A response time is the median between a mention and the person's next post in that thread (a re-threaded group as one), over the last 180 days, in calendar time, nights and weekends included. It is only said with at least three answered mentions, most of those made. "Not in this channel" is only said with a complete list of the channel's members, as for find_member: most channels have none, and then nothing is claimed. Such a person ranks lower, as their mention may not reach them; GET /api/experts lists the members in the background when that is a week old, as the mention suggestions do. The answer has names, numbers and the topic map's names only, never message text or a thread's title: the signals read channels of every team, which neither the viewer nor a thread agent may read. It is kept for ten minutes per version of the thread, per isolate.

A thread agent can pull someone in too: list_experts reads the same list, and suggest_expert puts a colleague at the top of it, with its reason (expert_suggestions, migration 0045; user state, like drafts). Nobody is told or mentioned; a person decides. It takes only staff with an account on the thread's platform, three per thread at most, and a suggestion goes away once the person is in the thread. It tells the open inboxes (a room event experts), and the header shows it as "Agent suggests …": clicking it mentions them in the reply box, as picking them after @ does, and ▾ lists each suggestion with its reason when there are several. Nothing is sent: the reply stays the person's to write.

What the finder learns from. History says who answered threads, but not what the finder showed when, nor what people made of it, and the signals cannot be worked out again for a past moment. So each list GET /api/experts computes for someone is kept (expert_events, migration 0047, src/expertevents.js): every candidate it ranked, up to 20 and not only the eight it answers with, by account and person, with the numbers that placed them (each signal scaled and as found, the score, topic, channel membership, response times, an agent's suggestion) and the finder's version (FINDER in src/experts.js), and held when it answered with none of it, never what anyone wrote. The answer has its id (list), and the page said what happened to it (POST /api/experts/event): shown once each time the thread was opened with it in the header, open for its ▾ list, pick of someone (their place comes from the list, not the page), reject for ✕, and dismiss when the ▾ list was closed, or left for another thread or page, with nobody picked: the clearest "none of these". Since the header lost the finder the inbox sends none of these and asks for no lists, so new ones come only from other callers of the API; the rejects recorded before still leave those people out of that person's lists of the thread. Whom the person mentioned afterwards, and who answered, are in mentions and messages already, so they are joined in training, not recorded twice. Nothing is recorded while viewing as someone else. GET /api/experts/events (admin team) gives them out a page at a time; npm run rank -- experts, and the weekly run, fetch them into .rank/expert-events.jsonl, and the cron deletes them after 90 days (EXPERT_EVENTS_DAYS), all but the rejects, which the finder goes by (migration 0048 indexes them).

Reactions are made the same way, from a small picker or by clicking a reaction a message already has (which adds yours, or takes it off). The Worker records a reaction in D1 as soon as the platform accepts it, exactly as the platform's event will, so it shows at once; the event then changes nothing. A removal only takes away a reaction made before it, and a later add moves the reaction's time forward, so a remove-and-add comes out right in either order.

Resolved

✅ or ✔️ on a thread's root message (white_check_mark / heavy_check_mark on Slack, ✅ / ✔️ on Teams) by a staff member means resolved. e, or the Resolve button that appears when the pointer rests on Send for a second, adds ✅ as the person; a ✅ put on a reply in the inbox goes on the root as well (and comes off both). It lands on the root of the entry's latest platform thread; undo takes the person's own ✅ off again (someone else's ✅ is theirs to remove). It overrides Clef's resolved_p, and a thread resolved this way is not in the "you", "waiting" or "silent" status piles.

A customer message after the ✅ reopens the thread: it is open until Clef has labeled the thread with that message in it, and from then on Clef decides again (so "thanks, works now" does not keep it open), until staff resolve it again, which moves the ✅ (off and on again) so that it is newer than the customer's message. Staff and bot messages after the ✅ change nothing. A ✅ without a time (from a backfilled Slack message) counts from the root message's time.

The models see reactions too: each message is followed by who reacted with what (customer or Adobe staff), and a reaction relabels the thread, so a customer's ✅ or 👍 on the answer can make Clef call it resolved.

New threads

The ✎ button next to the list's bell opens a composer in the pane, where a thread would show. What it posts goes through POST /api/post as a reply does, as the person, with the same text, mentions and files, but names a channel instead of a thread, so no thread exists yet. The inbox has no channel filter: the channel is known when the customer and platform filters leave one channel of the view on screen (channelsInView, over GET /api/channels). The other filters pick threads, which can be in any channel. Otherwise the composer's header searches every channel the person may post in, the view's first, and Post stays off until one is picked; POST /api/post itself refuses a post that names neither a thread nor a channel. Alert and ignored channels are not offered, nor is Discord, where the inbox cannot post as the person. Nothing is written to D1: the thread comes back through ingestion like any message, some ten seconds later (one in ten takes 20 seconds or more: the normalize queue's delivery), and the page opens it once /api/thread has it, as soon as the team's room announces it. Until then the pane shows the thread as it will look, from the moment Post is pressed: its title, channel and message, with the bar a reply being sent has, and a reply box: what is sent from it shows at once too, and goes out as soon as the platform has named the first message, into that thread on the platform (POST /api/post with channel and root), in the order it was written. Nothing waits for the inbox to have the thread. A post that fails takes the replies written meanwhile back into the composer, after its text. The channels to pick from are asked for when the pointer reaches ✎.

Drafts

A reply can wait as a draft: Save draft appears above Send, next to Resolve, when the pointer rests on Send for a second (or Send is pressed for a second on touch), and ⌘S / Ctrl+S in the reply box does the same. An inbox entry carries at most one draft, and it is shared: everyone in the teams that follow the channel (the one it is filed under, and those it is shared with) sees it, and can change, send or discard it. Admins, who can open any thread, see its draft but change only their own teams' drafts.

  • Drafts live in D1 (drafts), keyed by the inbox entry (a re-threaded group's first thread). They are user state, like watermarks: rebuilding the index from pile 2 keeps them. A draft on a thread that is later folded into a group shows on the group, and moves to the group's key when it is saved again.
  • A draft's updated_at is its version. A save names the version its text was written from, and a draft that changed meanwhile is not overwritten: the colleague's draft is shown, with an offer to overwrite it. A discard does the same. A save from a draft that is gone (discarded or sent) saves nothing either, and offers to save it as a new one.
  • The draft waits on its own line under the reply box, never in it: who saved it (a colleague, you, or the thread agent), when, and how it starts, on one line, rendered as the message will be (its Markdown, mentions and emoji, as in the thread). View draft opens the line to all of it, and Hide draft closes it again, with a click or a tap alike; a mouse resting on the line for 300 ms opens it too, until it leaves. The line grows upwards, so it stays under the pointer, instead of moving away from it and closing again, over and over. The reply box is yours, and a draft never replaces what you typed. Edit takes the draft into the box to edit; with text there already, Add to reply puts it after that text. View draft, Edit and Discard are icons (their names are their labels and tooltips), on the line's first row with who saved the draft; the sends are labelled, past a hairline, and get a row of their own at the bottom of a narrow line.
  • The line sends the draft as it is: Send as me, or Send as bot (as the opspile app, POST /api/post/app) where the thread says the viewer may (can_post_as_bot: the channel's Post messages switch is on and its platform's bot is set up). On Discord, and before the Slack or Teams account is connected, Send as bot is the only way to send it from the inbox.
  • Sending a reply written from the draft uses it up (unless it changed since), from the line or from the box. Saves and discards are fanned out over the rooms of every team that follows the channel, so every open inbox follows at once; a reply box that took the draft in and still holds it unchanged follows the draft, one with your own changes keeps them (and the line says the draft changed since). What you type and do not save stays with its thread in the tab.
  • On a phone (a screen up to 699 px wide) the draft's line and the command cards (see "Agent tools") would leave the thread a few lines of the screen, so they are in a drawer (renderLines, ui/inbox.js). Under the reply box only its tab shows: one line that says what waits there ("1 command to decide", "Draft by …"), outlined as a command's card while one waits for a decision. A tap opens the lines as a sheet from the screen's bottom edge, over the thread, the rest dimmed. ✕, a tap beside the sheet and Escape close it; so do Edit and the sends, which happen in the thread, and opening another thread. Run, Deny, View draft and Discard leave it open, and Undo of a discard shows over it. It is the same markup on every screen: on a wider one the stylesheet shows no tab and makes the drawer no box (display: contents), so the lines are the reply box's own.
  • In the inbox, an entry with a draft shows a pencil in place of its status dot (as a resolved one shows ✓), in the status's color and outlined while the team has not read the thread; the personal inbox shows none. The Draft card filters by drafts saved last by you, by a colleague, or none. A draft does not change the status pile. The outline is the pencil's own shape, not the ring a dot has for "our move": a ringed pencil would mean two things in one column. A draft that waits on a thread nobody has read yet is what the outline points out; "our move" stays in the pencil's tooltip, after the draft.
  • A draft can carry sources (drafts.sources, src/sources.js): what an agent says it rests on, as { url, title?, quote? }, at most 10, http(s) only. They are for the person reviewing the draft: they stay in the inbox and the tools, and no reply ever carries them. Whoever may see a draft sees its sources; sign-in (Cloudflare Access) keeps that to staff. The tools (update_draft, post_draft) replace a draft's sources with each save, so they describe its text; a person's edit in the inbox keeps them, and Undo of a discard brings them back.
  • The sources sit on a row of the draft's line. Resting the pointer on one (or moving the keyboard focus to it) shows it beside the draft's line, with the agent's quote above it; the reply box keeps what it held (the line's View draft shows all of the draft); clicking it opens it in a new tab. GET /api/draft/source tells the inbox how: a browser cannot tell why a frame stayed empty, so the Worker fetches the page, without the viewer's credentials, and reads its X-Frame-Options and frame-ancestors. A page that may be framed shows in a sandboxed, credentialless frame (no top navigation, no forms, no referrer). Markdown and text are rendered, and so is what a page that refuses frames says in its HTML (pageMarkdown, which sees what the server sent, not what scripts build). Anything else, such as a page behind a sign-in, is a reason and a link. The route fetches only the sources of the thread's current draft, follows at most 4 redirects, each checked (public names, never an address or the inbox's own host), and reads at most 512 KB.
  • A source that links to a Slack or Teams message is never fetched (the platform's page needs a sign-in a frame does not have). It is the message as opspile has it (view: 'message', src/messagelinks.js), for a viewer who may see its channel, and the inbox shows it as the card a link to that message in a thread shows (src/ui-markdown.js), in a popover only as tall as the card. Its time shows the message in the inbox: where it is in the open thread, else its thread, opened at it. A message opspile does not have, or the viewer may not see, is a reason.

Steering reactions

Resting the pointer on a message for 300 ms shows the quick reactions ✅ 👀 🙏 👍 😀 🚀 🛟. Three of them steer the inbox, on any message of the thread:

  • 👀 seen: the person who put it is on it. For them it counts like having posted (higher); for the rest of the team, a colleague's 👀 since the customer last wrote puts the thread in "team on it" and halves its score in the team view, unless it mentions them.
  • 🙏 thanks: after the last message, nothing is pending (waiting_on is nobody).
  • 🛟 help: a colleague asks the team to look: the thread is in "you" for everyone else and its score in the team view doubles, until it is resolved.

Thread agents

One agent per inbox entry (a thread, or a re-threaded group under its first thread's key), in the channels where agents are on: the admin team and team managers switch them per channel in the knowledge-base editor, and their team's default and AGENT_CHANNELS turn on the rest. Each is a Durable Object (ThreadAgent) running Pi through the Agents SDK's PiHarness: the transcript, tool calls and retries live in the object's SQLite, so a run survives evictions and deploys, and a model request that fails is retried. Prompt caching asks for the one-hour TTL (cacheRetention: 'long'). Turns are often minutes apart, past pi's five-minute default, and Bedrock bills a later hit inside the hour as a read. But every write costs 2× the input price for an hour against 1.25× for five minutes, so whether that pays depends on how far apart turns are. Experiments (src/experiments.js) settle it: for a fixed window, each thread is in one arm by a hash of its key (salted with the experiment, the same arm all along; knowledge-base agents take no part), and the provider asks for the arm's retention at every request (retainingApi, src/agent/model.js), so an agent that is running at the window's edges switches on its next request. The cache experiment puts half the threads on short from 2026-11-07 to 2026-11-14 (UTC); outside it every thread is on long. What an agent in an arm uses is counted for COGS with the arm in the detail (below).

When it runs. Every new message of the entry pokes the agent through the NORMALIZE queue (after 15 seconds, so a burst arrives together; a new root's after it is re-threaded, so that a root folded into a group pokes the group's agent, not one of its own); it reads the messages after its watermark from pile 2 and gets them as one input, the whole thread on first contact. A new message during a run joins that run. A day after the latest message, its own posts included, a schedule wakes it once more to wrap up: file what was learned, or put a follow-up in the draft. It is also told, as an input of their own, that a person discarded its draft, and how a command it asked a person to run ended (see "Agent tools").

In the inbox. The open thread shows the agent's latest turn under its last message, on the Adobe side, as "Opspile Agent" with opspile's icon: while it works, a spinner and what it is doing (reading the thread, checking a web page, updating the draft…); once done, its closing note for the team, which the prompt asks for at the end of every turn (1 to 3 sentences: what it found and what it changed). It is outlined, not filled, since nobody posted it. The turn is read from the agent's transcript (src/agent/turn.js). A finished one is also kept in D1 (agent_turns, src/agent/kept.js) and comes with the thread (agent_turn in GET /api/thread, and so in the threads the inbox fetches ahead), so it shows with the messages instead of from the Durable Object a moment after them. The agent writes it a moment after its answer (or a failed request), and whenever status() finds a finished turn it has not kept (one that stopped half way and counts as failed now). The page asks GET /api/agent when the thread opens or changes, and every few seconds while the agent works or a new message waits for it; what the agent says wins over the kept turn when it is about a later one. A turn in progress is only ever the agent's to say, so it comes in after the messages. Its avatar opens the sidebar (below), and its name the agent settings of the thread's team (Agents, its thread agents, at the thread's channel).

Whether it shows is each person's own choice, in their settings (person_settings): traces, on unless they turn it off, for every channel, and channelTraces, by channel id, where they chose otherwise, which wins. The inbox's user menu changes both (the open thread's channel); Agents does too, a select on each channel with an agent. Hidden, the page does not ask GET /api/agent at all.

The sidebar. Anyone who may work a thread can talk to its agent in the inbox, about the thread, without the channel seeing any of it (POST /api/agent/sidebar, src/sidebar.js). Each message goes into the agent's own conversation as an input of its own (ThreadAgent.sidebar, src/agent/sidebar.js), after the run it is in, never steered into one; so the agent remembers it in later turns. New messages of the thread do not steer into it either: while a sidebar message is pending, a delivery follows it as a run of its own. It goes into the thread's history too: sidebar_messages in D1, the person's message when it is sent, the agent's answer when the turn ends (the turn's closing text, up to 8000 characters, kept by keepTurn instead of the thread's note, which stays the turn before). Every finished sidebar turn in the transcript is kept, not only the latest: an input queued behind one starts at the boundary its answer is committed at. Nothing of it is posted: while the agent's latest input is a sidebar message, post_message, add_reaction and request_membership refuse (the admin bots answer in the channel), and so do they when the transcript cannot be read. The input starts with a line that names the message, and only an input that starts with it is one: a customer quoting it is told inside a delivery. The inboxes of the thread's teams hear of each message and answer through the team's room ({ type: 'sidebar' }).

In the inbox, the agent's icon enters sidebar mode: its avatar on its note, the button with it beside the reply box, Continue in the sidebar on an earlier conversation, or a. Each run of sidebar messages is one line between the thread's messages where it happened ("Lena Berg had a sidebar conversation with Thread Agent · 4 messages · not posted"), for everyone who sees the thread in the inbox. Where the conversation itself shows depends on the room (ui/inbox.js, "the sidebar"):

  • Beside the thread, where the list, the thread and a panel fit side by side: 1180px from the list's left edge to the content area's right (the list's least 380, 440 for the thread, the panel's least 360), measured, because the filters fold to a rail on some screens. While the panel shows, the content area is shared 3 : 5 : 3 between the list, the thread and the panel, so the thread stays the widest by far and the list gives up room too, down to its least width; the panel is 360px to 560px. The panel (#side) has every sidebar conversation of the thread, each under when it started, the agent working on its answer, and a box of its own, so the thread keeps its reply box and can be read and answered meanwhile. A line in the thread opens the panel at its conversation, also for someone who may not talk to the agent (viewing as someone else): they read there, without a box.
  • In the thread, on a narrower screen: a line unfolds its conversation where it is, the one you are in stays open at the bottom, with the agent working on its answer, and the reply box turns into the sidebar's.

Both are purple, say that only the team sees it and that nothing goes to the platform, and are left by ✕ (and Esc, and opening another thread). A window that grows or shrinks moves the sidebar you are in from one to the other, with what is typed in its box.

Traces. The admin team sees everything an agent did with the MCP tool get_thread_trace: the thread's messages and the agent's steps in one timeline (src/agent/trace.js, read from the same transcript through the agent's trace()): each input, its thinking and text, every tool call with its arguments and result, the MCP server calls its code mode scripts made (pi keeps no record of a script's calls, so the agent keeps its latest 200 in its storage, mcp-calls: when, which tool, the arguments clipped, the outcome, never the result), prompt and tool changes, and the tokens and cost per step. An agent that does not answer leaves the thread's messages, with agent.error. It covers the transcript since the agent's last reset; long text is clipped.

Prompt injection. Before an agent sees a message that Adobe staff did not write (customers, partners, bots), Clef answers one yes/no question: does it try to instruct or reconfigure an AI that reads it? A message at or above p = 0.5, or one the check could not answer, is withheld: the agent only learns that it exists. Verdicts are kept per message in the agent's storage, so each message is checked once. A check that could not answer is not a verdict: the queue tries the delivery again a few times before the agent is told the message is withheld, and reading the thread asks again. Text the agent does see is fenced and labelled as written by people outside its control.

What it can do.

Tool
read_thread the whole entry, flagged messages withheld; with thread (a key, or a link to one of its messages as people paste them), another conversation of the same channel, its first and latest 100 messages
read_file a file attached to a message of the entry (src/agent/files.js), named by its link as the message shows it or by a Slack file id, never any other: fetched with the app's credentials (the Slack bot's files.info and token, the Discord bot's refreshed links, the copies of Teams files in pile 1), up to 10 MB. Images (PNG, JPEG, GIF, WebP up to 3.75 MB and 8000 pixels a side, read from their headers) go to the model as images, larger ones through the Markdown conversion; an image the model would refuse never reaches it (src/agent/images.js), since the API refuses the whole request and the image would stay in the history: the provider also leaves such images out of every request (over 2000 pixels a side when it has more than 20 images); plain text as it is; PDF, Word, Excel, OpenDocument, Numbers, HTML, XML and CSV through Workers AI's Markdown conversion (AI.toMarkdown). Text comes in parts of 30,000 characters (offset), and every part goes through Clef's injection check first, as a message would; a flagged file is withheld. Teams files are read from their copies in pile 1 (one without a copy cannot be read). Files of withheld messages cannot be read.
list_channel_threads the other conversations of the agent's channel, latest activity first (a re-threaded group as one, no join messages), paged by before; with query, those whose title has it, then those whose root is similar to it (Vectorize, the roots' embeddings from re-threading). Only its own channel: an agent reads another channel's threads only with read_any_thread
read_any_thread only where it is turned on (see "Agent tools"): any thread of the inbox, by its key or a link to one of its messages, its first and latest 99 messages, flagged messages withheld
list_experts, list_topics, suggest_expert colleagues not in the thread yet who could help, with the similar threads they worked on and the topics those are in (about: what the thread is about, in the agent's words), the topic map's topics and who works each, and a suggestion to the team to pull one in (see "Expert finder"). It tells nobody: a person decides
find_member whether someone is in the agent's channel (on Teams, its Teams team), by name or mail address (src/mention-match.js), from the platform's list of members in channel_members, listed again when over an hour old; two accounts of one mail address are one match. With no complete list to go by (a failed listing, a Teams shared channel, over 2000 members) it says it cannot tell
get_draft, update_draft, discard_draft the entry's shared draft and its sources, saved as the person opspile agent with the same version check as a person's save. discard_draft removes a draft the agent saved last when it is out of date (staff answered themselves); a colleague's draft it leaves alone. Unless post_message is on, this and suggest_expert are its only writes outside the knowledge bases: it cannot post, react or resolve. A person reads the draft and sends it.
post_message only where it is turned on (see "Agent tools"): posts a reply in the thread as the opspile app
request_membership only where it is turned on (see "Agent tools"): asks the platform's admin bot to add people to the channel, or remove them, for the author of a message of the thread that asked; the bot's admins approve
add_reaction with post_message: reacts to a message of the thread with an emoji, as the opspile app, on Slack and Discord
bash only where the computer is turned on (see "Agent tools"): a shell with files of its own
request_bash with the computer: asks a person on the team to run a command in that shell, with values the person fills in; it answers at once, and the agent is told later how it ended
git_ls, git_read, git_grep, git_log the team's and the customer's knowledge base (kb-team, kb-org), every other team's (kb-team-<slug>), and the GitHub repositories listed for the team, the customer or everyone in agent_repos (to start: this repository)
kb_write create, replace or delete concepts in one commit, in the team's or the customer's knowledge base or that of a team the channel is shared with; the other teams' are read only
codemode a JavaScript script that calls the tools above, and the team's and the customer's MCP servers, as await tools.<name>(args); only what the script outputs reaches the model

Mentions of people who are not in. When a message the agent is told about mentions someone on the platform (a via = 'id' row in mentions; a name in the text does not count) who is not in the channel, or on Teams the Teams team, the agent is told so below the messages and says so in its note, so a person can add them. src/agent/members.js goes by channel_members, and lists them again first when that is over an hour old, so someone added since is not called out. It says nothing when it cannot be sure: a failed listing (on Teams, without TeamMember.Read.All), nobody listed, a Teams shared channel (the list is the team's, and others are in the channel), or over 2000 members. The member of the mention suggestions is no use for this: it is also true of anyone who posted.

GitHub repositories are read from Artifacts mirrors: the cron imports each listed repository into a fresh Artifacts repo, promotes it once it is readable and deletes the mirror it replaces, at most every six hours. Reads need no GitHub token and hit no rate limit; private repositories cannot be imported yet.

Code mode and MCP servers. The codemode tool is pi-durable's code mode: the script runs in a QuickJS sandbox on the agent's own thread (createInlineCodemodeSandbox; one object per thread, so a script that computes holds up only its own agent, for at most its interrupt budget), every call it makes is a nested tool call, and a script interrupted by an eviction resumes where it was without making its finished calls again. Each ops team and each customer can have MCP servers (mcp_servers, maintained in the knowledge-base editor next to the per-channel switch; credentials are request headers sealed with TOKEN_KEY, or an OAuth grant, below). Each server is on or off for every channel of its team or customer, and a channel can have a switch of its own that wins (mcp_server_switches), as the global servers' do. An agent gets the servers of its thread's team and customer that are on in its channel, and connects only to those: each tool becomes mcp__<server>__<tool>, callable only from scripts, never offered to the model as a tool of its own, and each server is a code mode namespace that describeNamespace() and searchTools() find. So a server with a hundred tools costs the prompt a few lines, and a script can call several and filter what they return. A server's allow list limits the tools agents get; a tool that declares itself read-only may run again after an eviction, any other is reported to the model as interrupted instead. The list of servers and tools (no credentials) is kept in the agent's storage, so a restarted agent has the same tools at once; before the agent is told anything it is checked against D1, and the servers are asked again when it changed or after an hour. MCP results come from systems outside opspile; the prompt says they are data, not instructions, like customer messages.

MCP servers with OAuth. A server can sign in with OAuth instead of headers (auth 'oauth', src/agent/mcpoauth.js); opspile is then the OAuth client. Someone who manages the server connects it in the editor: opspile asks the server without credentials and follows its 401 to the protected resource metadata (RFC 9728) and its authorization server's metadata (RFC 8414 or OpenID), and signs in as the client staff gave, else with opspile's client ID metadata document (/oauth/client.json, public; its URL is the client id) where the authorization server advertises client_id_metadata_document_supported, else as a client it registers there (RFC 7591); it sends the browser to the authorization endpoint with PKCE (S256), a resource indicator and a state whose hash finds the server again. The callback, /api/kb/mcp/oauth/callback, is behind Access, and only the person who started may finish, within ten minutes; the grant (client, tokens, the authorization server's metadata) is kept sealed in mcp_servers.oauth, with who connected it and when in plain columns. The agents act as that person at the server. Agents and the editor's check get the token from the transport's auth provider: the stored one until two minutes before it expires, else a refresh, and a refresh too when the server answers 401. Many agents share one grant and refresh tokens may be single-use (opspile's own are), so only the holder of a 30-second lease in D1 (oauth_lock) refreshes, and the others wait and read its result. Disconnecting and connecting again do not wait for the lease, so a refresh only writes over the grant it started from: one that comes back to another grant leaves it alone, and revokes the tokens it got. A grant the server rejects (invalid_grant) is dropped with the reason, which the editor and the agents' prompt show until someone connects again; a new URL, client or scope, a switch back to headers, Disconnect and deleting the server revoke it where the server has a revocation endpoint. The cron refreshes grants nobody refreshed for a week, so refresh tokens that expire unused (opspile's own after 30 days) stay alive. opspile's own /mcp is such a server, so agents can be given what a person sees in the inbox.

Global MCP servers

Some servers are not a team's or a customer's but opspile's own, built in (src/agent/globals.js) and set up once: browser and github. A new team starts with both on; otherwise they are off until the admin team or a team's managers turn one on, for an ops team or for a single channel (mcp_global_switches; a channel's switch wins over its team's, and a channel without one follows its team). An agent gets those on for its thread's team or channel, next to the team's and the customer's, as mcp__browser, mcp__github; its id there is global:<key>. Their tools are those the last check found (mcp_global_tools), so an agent does not ask a server or mint a token just to list them; the editor's Check and the cron, once a day, ask again. The prompt says that these are opspile's own, and that customer data does not go into websites or URLs that are not the customer's or Adobe's.

browser: Cloudflare Browser Run. Cloudflare's own hosted Browser Run MCP server (https://browser.mcp.cloudflare.com/mcp), with an account API token that may only use Browser Run (BROWSER_RUN_TOKEN, Browser Run Write; account-scoped, so the server finds the account itself). Its tools load a page in Browser Run's Chrome and return it as markdown or HTML, a screenshot, a PDF, its links, elements by CSS selector or JSON extracted from it, or crawl a site. Each call loads the page anew, signed in nowhere: there is no session to click through. The tools that list and kill the account's browser sessions are left out of what agents get.

github: GitHub's MCP server, read-only, as the opspile GitHub App. The server is GitHub's hosted one, at https://api.githubcopilot.com/mcp/readonly with X-MCP-Readonly and X-MCP-Lockdown (public issue content from people without push access stays hidden: text from strangers that nobody screened). It authenticates with installation tokens of the opspile GitHub App (src/github.js), so it reads exactly the repositories the app is installed on, all of an account's or a selection, and nothing else; the app asks for read permissions only. The app is public, so that customers' organizations can install it too; since anyone can, an installation is used only after the admin team allowed it in the editor (github_installations), and a revoked one gets no more tokens. The admin team creates the app from the editor with GitHub's manifest flow: the page posts the manifest to GitHub, GitHub asks to confirm and sends them back with a code (and a state opspile signed for them), and opspile trades the code for the app and its private key, which stays in D1 (github_apps), sealed with TOKEN_KEY; then GitHub's installation page. Nobody copies a key. A token lives an hour: the Worker signs a JWT with the key (RS256, Web Crypto; GitHub's PKCS#1 key is wrapped as PKCS#8), trades it for a token per installation, keeps it per isolate until shortly before it expires, and an agent's client is replaced before its token runs out. An app installed on several accounts is a server per account, github_<account>.

Writes clone the knowledge base into memory (isomorphic-git), apply the change, rewrite the indexes and the log, commit and push with a short-lived write token. Many agents write the same team knowledge base, so a rejected push starts over from a fresh clone; a change that names the blob it was written from (base) fails instead of overwriting a newer version.

Seeding. Staff can teach an agent: POST /api/agent/seed (or the MCP tool seed_agent) with a thread hands the note to that thread's agent, and with a team and a customer slug to a knowledge-base agent (kb:<team>:<org>) that has the same tools minus the thread and the draft. Staff notes are trusted and not checked by Clef.

Left at that, the agent picks the knowledge base. kb (team or org, on either form) chooses it, and code holds the agent to it: the note goes into a session of the same Durable Object pinned to that knowledge base (seed-pin:<session> in its storage, src/agent/seed.js), and kb_write checks the calling session, so a code-mode script is refused like the model. A session and not a flag, because the agent's conversation goes on meanwhile: a turn already running may write anywhere, and a new customer message steers into whatever is running. The session starts without the transcript, reads the thread with read_thread when it needs to, and its turns show in neither the thread's latest turn nor the trace; the thread's agent meets the note in the knowledge base. A thread's kb is only ever that thread's own team or customer. kb leaves agent names alone: { team, kb: "team" } needs no customer and reaches kb:<team>: (the org left empty), which has no kb-org at all, and only the chosen knowledge base is created when it is missing.

Discarded drafts. When a person discards a draft the agent saved (DELETE /api/draft, the inbox's Discard), the agent is told who did and what the draft said, as feedback: it looks for why it was not wanted, does not save it again, and files a lesson in kb-team when there is one. It hears it 30 seconds later, after the inbox's Undo has run out, and not at all when the draft came back (the same text saved again by a person; src/agent/discard.js). Drafts a person saved last, drafts a reply used up, and drafts the agent discarded itself are not told.

The model. Claude Opus 5.5 on Amazon Bedrock (the inference profile us.anthropic.claude-opus-5-5 in us-east-2), through Cloudflare AI Gateway: Bedrock's bedrock-runtime endpoint speaks the Anthropic Messages API at /anthropic, AI Gateway passes it through, and the Bedrock API key goes in Authorization: Bearer. (bedrock-mantle speaks it too, but serves only older Claude models in that region.) The gateway meters every request but keeps no logs: prompts carry customer messages, so request logging is off (turn it off on any new gateway too).

Replay. Agents are not part of the replay path. Their output is user state (drafts) or pile 3 (git history); replaying pile 1 pokes them, but they find nothing past their watermark.

Agent tools

Four tools are off until they are turned on (a new team starts with computer and extraction_rules on for the team). Three by the admin team or a team's managers, as the global MCP servers are: for an ops team or a single channel (agent_tool_switches; a channel's switch wins over its team's), in the knowledge-base editor (Tools and MCP servers, src/agent/agenttools.js). Before an agent is told anything it reads the switches of its thread's team and channel; the tools it has are kept in its storage and installed as their own extension (opspile-tools, right after opspile's own), which a change replaces in place. Each comes with a section of the system prompt on how to use it.

post_message. The agent posts a reply in the latest platform thread of its entry right away (src/agent/posting.js), as the opspile app, never as a person: on Slack with the bot token (chat.postMessage in the thread, the Markdown in a markdown block, which Slack renders itself, and text in mrkdwn for notifications and for the message file), on Discord with the bot, in the Discord thread when the conversation is one and as an inline reply to its first message otherwise, with mentions switched off (but for the people a person posting as the app picked, see "Replying"). On Teams with the opspile bot (src/teamsbot.js): Microsoft Graph lets no app post to a channel, but a Bot Framework bot can, through the Bot Connector, in a team it is installed in. The bot is the opspile Entra app itself, single tenant, registered as an Azure Bot with the Teams channel, and packaged as a notification-only Teams app in the org catalog (TEAMS_APP_ID). A post replies in the channel thread (<channel id>;messageid=<root id>) with a Bot Framework token of the app; when the bot is not in the team yet, opspile installs the app there through Graph (TeamsAppInstallation.ReadWriteSelfForTeam.All, which lets the app install only itself) and posts again. Its messaging endpoint (/teams/bot) answers every activity and keeps nothing: opspile reads Teams through Graph. A platform whose bot is not set up does not get the tool. The prompt asks the agent to post only what the team would send as it is, and to keep anything uncertain or sensitive in the draft. A post may name the draft version it sends, which uses the draft up. The reply comes back through the platform's events like any message; the agent keeps the ids of its latest posts and is not told about them, so it never answers itself.

With post_message the agent also reacts (add_reaction, reactAsApp in src/agent/posting.js) to a message of its entry, as the app: on Slack reactions.add with the bot token (bot scope reactions:write), on Discord PUT …/reactions/<emoji>/@me as the bot (Add Reactions), in the channel or Discord thread the message is in. It names an emoji by its Slack name in any platform's spelling or as the character; Slack also takes a workspace's own, Discord gets the character. Not on Teams: Graph's setReaction is delegated only, and the Bot Framework has no call for a bot to react, so the prompt tells an agent on Teams it cannot. 👀, 🙏, 🛟 and ✅ (✔️) are refused: they steer the inbox or resolve a thread, and a bot's 🙏 would clear "waiting on us". The reaction comes back through the platform's events like any other; the agent is not told about reactions.

computer. pi's own bash tool, run on Cloudflare Computer (@cloudflare/computer, src/agent/computer.js). The ThreadAgent object holds a Workspace: a filesystem in its own SQLite, so each thread has a working directory (/workspace) whose files last from one turn to the next, and nobody else's. Commands run in a WorkerShellBackend: just-bash in a Dynamic Worker (the LOADER binding), which reaches the files back over Workers RPC through WorkspaceServiceProxy (the Worker exports it) and the object's __getWorkspaceStub(). The shell has the text tools (cat, grep, sed, awk, sort, find, jq…) and curl, with network access through the Worker's SecretGateway (egress http-gateway, no filter; see "Computer secrets" below), and no npm, node or python. What curl fetches is untrusted, like customer messages; nothing stops an agent from sending what it read, so the prompt says not to send anything to sites that are not the customer's or Adobe's. The bridge is an ExecutionEnv for pi-durable whose exec is workspace.runtime.exec; pi's file tools are not installed, so the rest of the environment answers not_supported.

/workspace/uploads is read-only and holds the files attached to the thread's messages (src/threadfiles.js). Every file is copied to the FILES bucket, under <thread>/<sha256 of the file's id>, by a queue job when its message arrives, or at the latest before the agent is told about it. It is kept for 30 days: the bucket's lifecycle rule deletes it, and a file whose message is older is neither copied nor shown. The mount is filled by uploadsMount into the object's SQLite, so each change of the list takes a new Workspace, and a schedule (expireUploads) empties it when the oldest file expires, even if the agent sleeps. Filling the mount makes /workspace, and the Workspace refuses any change to a directory a read-only mount is in, even a mkdir -p of one that exists: the computer makes /workspace only when it is missing (ensureCwd). Files of withheld messages stay out of it, as they do of read_file, but a file itself is not checked for prompt injection here, and a raw HAR holds its recorder's credentials: the prompt says not to quote them, and code does not stop the shell from reading or sending them.

Computer secrets. API keys and tokens of a team or a customer (agent_secrets, migration 0060, src/agent/secrets.js), kept in the Computer tool of its knowledge base by whoever edits it (/api/kb/tools/secrets, src/secretadmin.js), sealed with TOKEN_KEY and never shown again. A team's wins over the customer's of the same name. The agent never holds a value: its prompt lists the names, hosts and descriptions, and pi's bash gets each as a variable whose value is a placeholder ($ACME_API_KEY is opspile-secret-ACME_API_KEY), through the tool's prepare. The Workspace's egress is http-gateway with SecretGateway (src/agent/gateway.js, exported by the Worker), a WorkerEntrypoint that becomes the Dynamic Worker's globalOutbound: every fetch of the shell, curl's too, arrives there, with the agent's name in its props, which the shell cannot touch. A request that names no placeholder goes out as it is. One that does is sent with the values in place of the placeholders (in the URL, the headers, a Basic Authorization, a textual body up to 1 MiB; percent-encoded in the URL and a form body, escaped in JSON and XML, a "form" that parses as JSON as JSON, which is what curl -d sends without a content type; so a + does not arrive as a space), only over HTTPS to a host the secret lists, and with redirect: manual, so a redirect comes back to the shell and its next request is checked again; what comes back has the values replaced by their placeholders, in its headers and its body, as it is or escaped (scrubber, which also finds a value split between chunks). The prompt and the shell's variables come from the agent's team and customer as they are now too, not from its cached target. A placeholder for another host, over HTTP, or for a secret the agent's team and customer do not have is refused with a 403, and nothing is sent. The gateway reads the secrets from D1 on each such request, by the agent's team and customer as they are now, so a deleted secret or a moved thread counts at once. What it does not stop: a host on the list that publishes what it receives (the agent could post the key into a ticket of the API it is for), and a value the host sends back encoded otherwise.

Commands a person runs. With the computer comes request_bash (src/agent/bashrequest.js; the record is bash_requests, migration 0062, src/bashrequests.js): the agent asks a person on the team to run a command in its shell, for what needs a credential it does not have, or a person's go-ahead. It gives the command, its reason, a timeout (how long the command may run: 60 seconds unless it says, 180 at most, the shell's own limit), a deadline (until when the request stays open: a duration from now such as 4h, or a time; a day unless it says, a week at most) and env, the variables a person is to fill in, by name, each with what it is for. The tool answers at once, and the agent goes on: nothing waits on the request, and at most 5 of a thread wait or run at a time. A thread's agent only: a knowledge-base agent has no thread to show the request under.

The inbox shows each request as a card between the reply box and the draft's line (bash_requests in GET /api/thread; bashCard in ui/inbox.js): its id, which is how the agent names it in its notes, the command (highlighted as shell, to be read before it is run), the reason, a password field per variable, Run and Deny, for whoever may work the thread; Deny and Run stay in sight at the cards' bottom edge while a tall card scrolls. A person can have the page remember what they fill in, by variable name, on that device: encrypted (AES-GCM) in localStorage, with a key made in the browser that cannot be exported and is kept in IndexedDB; the next request with a variable of that name has it filled in, and Forget removes it. POST /api/bash/run and /api/bash/deny go to the agent's own object (runBash, denyBash), which decides: the request has to be pending still (the first click wins, the second is told who was first), the computer still on, every variable filled in. The run is the agent's computer's, as its own bash: the same Workspace, working directory and team secrets (as placeholders), with the person's values as variables of that one run. run answers 202 once the command has started; the page hears how it ended through the team's room, as it hears of every change of a request ({ type: 'bash' }), and keeps the request for an hour, on one line, saying how it ended (a nonzero exit, a timeout and a failure marked) and who decided. An agent often asks a few times until a command works, so of the requests that ended only the latest shows (bashCards), with a button for the earlier ones (2 earlier), all in one box. The card itself says that a command runs or was denied; a toast only says what went wrong. On a phone the cards are in the drawer the draft's line is in (see "Drafts").

How a request ended comes back to the agent as an input of its own, as a discarded draft does (harness.submit, with the request's id as the operation, so once): what the command printed and its exit code; that it was stopped at its timeout, with what it printed until then; that the computer failed; that a person denied it, and who; or that nobody decided before the deadline, which a schedule of the agent's object closes. A watchdog schedule, armed before each run, settles a run whose object restarted under it as failed (docs/STATE-MACHINES.md).

The values are kept nowhere: not in D1, not in the object's storage, not in the transcript, not in a log. They go from the page to the Worker, to the agent's object and into the shell's environment, for one run. What the agent is told of the output has each value of 8 characters or more replaced by its variable's name ([API_TOKEN]), as it is and as it would be escaped in a URL, JSON or XML (hider); the shell's own record of the run, which Cloudflare Computer keeps for an hour, is deleted, and output it saved to a file is rewritten the same way (removed, over 4 MB). Unlike the team's secrets, these are real values in the shell, not placeholders the gateway fills in, so that a person's own token works with any host and any tool. What that does not stop: the command itself can write a value to a file, which the agent can read afterwards, send it to any host, or print it encoded. The person who runs the command is the check, as with a command a colleague asks them to run: the card shows it whole, and says so.

The shell reads a file whole, in one isolate: above about 30 MB grep no longer finds it, and jq runs out of memory on a large one. So files over 30 MB stay out (200 MB a thread), and INDEX.txt says which. A HAR also gets <name>.requests.ndjson (src/har.js): one line per request, no bodies or cookies, credentials hidden, read as a stream, whatever the HAR's size.

extraction_rules. The agent reads, tests and changes its team's extraction rules and tech stack (list_extraction_rules, test_extraction_rule, save_extraction_rule, delete_extraction_rule, save_tech, delete_tech; see "Labels and facts"), as opspile-agent/<model>. Testing a script rule (MCP test_extraction_rule, the page's Test) runs code in a Dynamic Worker, so it is for those who may save the rule, as saving is. What it saves applies to every channel of its team from the next labeling of a thread; it cannot change the admin team's, which apply to every channel. Its section of the prompt asks it to add a rule only for what recurs and the existing rules miss, to test it first, and never to put a customer's name or a secret into one.

any_thread. Agents read only their own channel's threads, but staff in an internal channel paste links to customers' threads elsewhere. With this tool the agent has read_any_thread: any thread of the inbox, whatever its channel, team or customer, by its key or a link to one of its messages (resolveAnyThread, src/agent/history.js), except in channels the rules ignore. It has no search: a person points it to a thread. The messages go through Clef's injection check like any others, and each thread an agent reads is kept in agent_thread_reads, since a transcript can be reset.

Only the admin team switches it, one channel at a time (no team switch), and only in an internal channel: filed under an org of type internal, and not shared (channels.shared, channels.connected). A channel that stops being one loses the tool, whoever turned it on: it is checked with the switch before each read, and a channel opspile has no details of does not count. Neither says that nobody outside is in the channel (guests are not recorded), so the org's type alone does not turn it on. The prompt asks the agent to keep other customers' details out of the knowledge bases and out of what it posts; that and the channel's other tools are not restricted by code: what it read can reach them, bash and the browser server included.

membership. Customers and colleagues ask in a thread for someone to be added to the channel, and on each platform the admins have a bot for that: admin-bot (teams-admin-agent) on Teams, AEM Ops Automation (the slack-automation relay) on Slack. A request that does not reach the bot properly (a plain "@admin, can you add …") is easy to miss, and the thread does not wait on us when Adobe posted last. With this tool the agent files it (request_membership, src/agent/membership.js), naming the author of the message that asked. Neither bot listens to other bots (Teams delivers no bot's message to another bot; the relay ignores bot messages), so opspile calls each through a service binding to its MembershipRequests entrypoint, on the same Cloudflare account; a named entrypoint answers no HTTP, so no secret is shared:

  • Teams (TEAMS_ADMIN): submit({ action, emails, teamId, conversationId, requester, via }) files it as an @admin-bot add would: an approval card in the admins' Slack channel per person not yet as asked, and admin-bot's answer in the Teams thread, then and once reviewed.
  • Slack (SLACK_AUTOMATION): the agent posts the request in MEMBERSHIP_SLACK_CHANNEL (#general) as the opspile bot, mentioning only MEMBERSHIP_SLACK_BOT (the relay reads every other <@…> as a person to add), with the asker's name and a link to the thread, then hands that message to submit({ channel, ts, text, requester, via }). The relay threads its confirmations under it; the asker confirms if they are a member of the workspace, an admin otherwise.

Nobody is added until it is approved, and the prompt says not to tell anyone otherwise.

COGS

What the inbox costs, at list prices, for the admin team (/inbox/cogs, GET /api/admin/cogs): per service, per message and thread, per customer and team, per day, and per month for what is kept. Three sources are rolled up into one table, cogs_usage: a row per day, source, service, metric, detail (a model, a bucket, an index), customer and team, with a quantity. Prices are not stored: the dashboard prices the quantities with src/cogs/prices.js, so a corrected price applies to the past too, and a quantity without a price (a model added since) is listed and costs nothing.

The meter (src/cogs/meter.js). Every invocation has a Meter: a request, a queue batch (a scope per job), a cron run, and each Durable Object (an agent, a team's room, the Discord poller). A scope is an environment with the same bindings, counted as they answer: D1 rows read and written (meta; a first() counts one row read, since D1 does not say), R2 Class A and B operations per bucket, queue operations (three per 64 KB of a message: write, read, delete), Workers AI tokens per model (the model's usage, else a quarter of the characters it was given), Vectorize dimensions queried and written, AI Search queries (full-text or semantic) and the tokens it ingests; and one Workers request per invocation (a queue batch's shared by its jobs). Code adds what no binding shows: an agent's Bedrock tokens and the time its global browser server's calls take. Whom the work was for is the scope's subject: a request's thread parameter, a queue job's thread, a raw event's channel once it is filed (processKey names the customer and team its rules gave), an agent's thread (or kb:<team>:<org>), a room's team; else nobody, shared. At the end of the invocation (a request's after its waitUntil work too; a Durable Object's 15 seconds after it counted, or a room's at the end of each broadcast and flush, so it can hibernate) the subjects are resolved (a thread's customer and team from D1, once per invocation) and written to Analytics Engine (USAGE), one data point per customer, team, service, metric and detail, never more than the 250 an invocation may write (details fold into their metric first). Counting never fails the work: a binding's answer is returned as it came. Without USAGE (tests, the preview) the environment is the plain one.

An agent's tokens. pi keeps a ledger of what each session's model requests used (usage(), per model and tool, cumulative). The agent reads it after each model round (a hook on pi's generation task) and before it is told something, and counts the difference since the last time (kept in its storage, cogs:usage) as Bedrock input, output, cache read and cache write tokens (five-minute and one-hour writes apart, as Bedrock prices them), and what pi's catalog says they cost, which prices a model the list does not know. While an experiment runs, the arm the agent is in follows the model in the detail (us.anthropic.claude-opus-5-5#cache=short), which the prices read past. An agent that was never metered starts from its ledger as it is: what it used before is not counted on the day COGS started. Once a day it writes the size of its SQLite (cogs_objects), for the agents' storage.

Cloudflare's analytics (src/cogs/rollup.js, the Analytics SQL binding ANALYTICS, no token) for what the Worker cannot see, about opspile's own resources only, by name (COGS_RESOURCES, written by the deploy config; the account has other projects):

  • Durable Objects (events.durableObjectsInvocations): requests and wall time per object of the Worker's script, as GB-seconds of 128 MB. An agent's object is named by its thread, so its duration is its customer's. A team room's WebSocket connections hibernate, so its http invocations (the connections) count as requests and not as duration. Invocations that overlap are billed once; the wall times are added up, so this is an upper bound.
  • The Worker's CPU time and log events (logs.workersLogs): CPU from the invocations' own events (fetch, queue, scheduled), shared.
  • Artifacts operations per repository (events.artifactsEventsByNamespaceAccount, without token minting): a knowledge base's are its customer's or team's (knowledge_bases), the mirrors shared.
  • What the buckets (states.r2CombinedStorage) and indexes (states.vectorizeStorage, times 768 dimensions) hold, per day.

A query that fails leaves the day's rows of its services as they were, and the others are written.

D1: the day's messages and the threads started, per customer and team (volume), and, once a day, what only D1 knows is kept: its own size (meta.size_after), the agents' storage (cogs_objects), the full-text items (their number times the average upload of the last 30 days) and the Workers Paid plan.

The Access API (seatRows, with ACCESS_API_TOKEN and ACCESS_ACCOUNT_ID), once a day too, and written as d1 rows: the account's Zero Trust seats (users with access_seat or gateway_seat), one seats each, every one at $7 (the Free plan's 50 are not taken off: past them, Pay-as-you-go bills every seat). A holder whose email is a person on ops teams is those teams', split evenly; anyone else's seat is shared. The emails stay in the run. A failed read writes nothing, and the next run tries again.

The rollup. The cron rolls up today and yesterday every run, and up to three of the last 30 days that have no rows of a source yet (backfilling what Cloudflare's analytics and D1 still have; the meter's counts start with it). A day of a source is replaced whole, writing only the rows that changed and deleting the ones that are gone, so a run that changed nothing writes nothing.

By the hour. For the chart of a short period, the flow of the meter and of Cloudflare's analytics is rolled up by the UTC hour too (toHour(timestamp)), into cogs_hours: the same queries grouped by the hour as well, without what the buckets and indexes hold (stock), and without D1's messages and storage. Separate queries, so a day with many objects active in many hours cannot truncate the daily rows the totals come from. The same days, replaced the same way, and up to three of the last seven before yesterday without hours yet; hours older than eight days are deleted.

The agents' drafts (src/draftevents.js, draft_events, migration 0058). Every save, send and discard of a shared draft (src/drafts.js) is recorded with who did it, the entry's customer and team, and the draft it was about (its created_at, which stays through its versions). A draft remembers who saved its first version (origin) and the latest text that person saved (origin_body); a send says how it went out (as the person, as the opspile app, by the agent's post_message) and how much the text sent differs from that, by word (changed, 0 to 1). So an agent's draft a person sent is accepted: as it was (changed 0) or edited (in the reply box, or saved over by a colleague first). One a person discarded is discarded, one the agent discarded itself (discard_draft) withdrawn. An Undo saves the same text again: a save with no draft, by the person who discarded one with that text (its hash, never the text) in the minute before, deletes the discard's event and goes on as that draft, with its origin. The agent's post_message that used no draft up is recorded as posted. The report reads the events as they are (they are few), per customer and team; nothing is rolled up.

The report (src/cogs/report.js). Flow costs what the period used; per message and per thread it is divided by the period's messages and new threads of what is selected. These estimates, and a month at this rate, count from the first whole day of the source that started last (each source's first day is partial, and before it messages were counted without all of their cost), and a month at this rate divides by the time so far (estimateWindow: today counts its hours). A period that ends before then has no estimates. Stock costs a month: each series at its average over the days it was measured. Shared rows are shared out by the period's messages: a team's (a room's) over that team's customers, nobody's over everyone; then the filters apply, so a customer's cost includes its share. The chart has about as many columns whatever the period: by the hour for a week or less (from cogs_hours, while it has them, up to the current hour), by the day up to 45 days, by the week (from Monday, cut to the period) for longer; the totals and tables always come from the days. Its columns are by service, and asked (by), by customer or team too: a customer's own rows and its share of the shared ones, the ten biggest customers of what is selected apiece (or those picked) and the rest as (other), so the answer stays small. Where Cloudflare's analytics and the meter count the same metric on the same day, Cloudflare's count is used. Per accepted draft (the flow over the agents' drafts a person sent) and per agent reply (over those and the agents' own posts) count from the first whole day of the drafts' events as well (drafts.estimate), so that a period before they were recorded does not divide by none; a service's per accepted draft is its own flow, so Bedrock's is what the agents' model cost per accepted draft. Costs are at the margin: the allowances of a plan are listed with the prices, not subtracted. An experiment's arms are set side by side (experiments, of what is selected): per arm, the Bedrock tokens and cost, and the prompt (input, cache reads and writes), which a cache policy does not change, so what the prompt cost per million tokens compares the arms however the threads fell between them.

People

Authors are normalized to { kind: person | bot | system, org, staff }.

  • On Slack, bot posts carry bot_id and no user, and Slack Connect marks people from another workspace with user_team.
  • On Discord, bots carry author.bot, webhooks webhook_id. Discord shares no email with bots, so a member is staff when they have one of the DISCORD_STAFF_ROLES (identities.staff), or when they connected the account in the inbox (OAuth identify), which links it to their person: their posts move their watermark and count as theirs.
  • On Teams, the support tenant invites everyone as a guest, staff included, so userType and tenantId do not distinguish staff from customers. The organization is derived from the person's mail domain instead. Bot posts carry from.application; a bot (applicationIdentityType: bot) keeps its id as its identity, webhooks and connectors stay anonymous.

One account a person (src/accounts.js). Someone can have several accounts in one workspace or tenant: on Slack, besides their account of the Enterprise Grid, one of another workspace of the grid, or their own workspace's account seen through Slack Connect (a shared channel's user_team, kept as identities.home), or a guest account. D1 already knows them as one: identities are linked to their person by mail address (identities.person), and accounts of people outside Adobe, who have no people row, are one person's when they have one mail address. Wherever people are listed or ranked for a thread, each person is there once, by their account for the thread, and their other accounts count as that one: the mention suggestions and their pool (GET /api/mentionable), the mentions a reply, an edit or an agent's @Name makes (knownMentions, mentionsByName, and so the colleagues a post adds to the channel), the expert finder (findExperts, GET /api/experts, list_experts), the agents' suggestions (suggest_expert) and the ✕ on one, and find_member. The account for the thread is, of the person's, the first by:

  1. of the thread's own workspace or tenant (no other home, or the channel's workspace) and in its channel (listed in channel_members, or posted there; on Teams, in its Teams team);
  2. in its channel at all, then of its own workspace at all: a mention reaches someone in the channel, and the inbox can add a colleague of its own workspace to it (an account of another workspace it cannot);
  3. a full member before a guest (identities.guest, migration 0061: Slack's is_restricted or is_ultra_restricted, Graph's userType; NULL until the account is looked up again);
  4. the one that posted in the last 90 days, then the one with the most messages in the last 180, in the most channels, that posted last, and by id.

What the signals say of the person adds up over all of their accounts and goes to that one: the finder's history, response times and affinities are by person already (messages.person, mentions.person); a suggestion is in the thread or channel when any of their accounts is, finds them by the names and logins of all of them, and has the affinity of the person (or, for someone D1 has no person for, of their accounts together). An agent's suggestion is kept as the person's account for the thread, and one kept by another of their accounts (made before) shows as it; a ✕ on any of them leaves the person out.

Profile cards. Hovering over (focusing, or tapping) a name or avatar in the inbox shows what opspile knows about the person: name, picture, platform account, job title and company (Slack profile title, Graph jobTitle and companyName, cached with the identity), mail domain, ops teams, the channel's org, and above all who they are to us:

  • Adobe: staff (a person marked staff, or a mail domain in STAFF_DOMAINS);
  • Partner: their mail domain, or a parent domain, is in the partner_domains table (agencies and system integrators, maintained by hand, like the rules);
  • Customer: anyone else; the card notes when their domain is not the usual customer domain of the channel, which is how an unlisted partner shows up;
  • Bot.

A list of partner domains is explicit and never wrong, where guessing from "not staff and not the customer's domain" would call a customer's second domain a partner. The cards travel with the inbox and thread payloads (profiles, keyed by messages.author_key), so a hover costs no request, and emails stay on the server.

Platform facts that shape the design

These were measured against the live APIs during exploration.

  • Both platforms are two-level: a root message and a flat list of replies, no deeper nesting. A root without replies is a thread of length one.
  • Slack message ids are ts strings, unique per channel. A thread is keyed by the root's ts; replies carry thread_ts and parent_user_id. The root's latest_reply changes whenever a reply arrives, which makes change detection cheap.
  • Teams message ids are epoch-millisecond strings, unique per channel. Replies carry replyToId. The root's lastModifiedDateTime does not change when replies arrive, so new activity cannot be detected by polling roots.
  • Bots edit their own messages later, for example to add a ticket link, which is why edits never make a thread unread.
  • Discord is two-level in another way. A channel's messages are flat, and a thread is a channel of its own (channel_id is the thread's id, its parent_id the channel). A thread started from a message has that message's id, and a forum post's first message has the post's id, so the thread id is the root message id: a thread's messages are filed under its parent channel and the thread id, next to the message it started from. People also answer with an inline reply (type 19, message_reference) instead of a thread; that reply joins the conversation of the message it answers, and a thread started from such a reply joins it too. Message ids are snowflakes (ms since 2015 shifted by 22 bits), unique across Discord. Joins, pins, boosts and "started a thread" notices are not stored in pile 2.
  • Alert channels consist entirely of bot posts without replies. They are filed as kind: alert and counted, rather than shown as individual unread threads.

Ingestion

Two new apps:

  • A Slack app, installed into the enterprise grid, subscribed to message, edit, deletion, and reaction events. Slack only delivers message events for channels the app is a member of, so the app has to be added to every ingested channel, including Slack Connect channels. With SLACK_JOIN_TEAMS it adds itself to every public channel of those workspaces (src/autojoin.js, bot scope channels:join): a channel as it is created or unarchived, and on the cron, at most 40 a run, every other public channel it is not in. Channels Slack refuses (app_cannot_join_channel, say) are tried again after a week (slack_joins). Private channels still need an invite; channels the rules mark ignore are not joined.

  • A Microsoft Entra app registration in the support tenant with application permissions to read teams, channels and channel messages. One tenant-wide change-notification subscription on /teams/getAllMessages delivers root messages and replies. It carries no resource data (so no encryption certificate) and lives at most 60 minutes, so a cron renews it every 20 minutes. Microsoft stopped metering Teams APIs on 2025-08-25, and creating the subscription needed no protected-API approval (measured 2026-10-20). Polling is not an alternative: /messages/delta and /messages return root messages only, never replies.

  • A Discord application with a bot in each server opspile reads (DISCORD_GUILDS), permissions View Channels and Read Message History and the privileged Message Content intent, which REST reads need as well. Discord delivers messages only over its gateway WebSocket, which would keep a Durable Object in memory, and billed, around the clock; a support server gets a few messages a day, so opspile polls instead. The poller object's alarm asks every minute for each server's channels and active threads, which name their latest message, and reads the messages after the last one stored where there are newer ones: two requests when nothing happened. An archived thread that gets a message is active again, so archived threads are read only on the first poll, DISCORD_HISTORY_DAYS back. Every fifth poll it reads the conversations active in the last day again and compares them with what it saw: an edit or other reactions are stored as a new state of the message (a reaction timed when it was first seen), a message no longer there as a deletion. The object stores each thread's parent, each reply's root, the last message id per channel and thread, and its view of the last day's messages; the cron pokes it, so a stopped alarm restarts.

Failed jobs. The NORMALIZE queue retries a job five times. Its last failure is recorded in dead_jobs (migration 0050, src/deadjobs.js; ids and keys only, kept for 30 days), which /diag/dead lists and sends out again. Most jobs need nobody to do that: the cron's sweeps find vectors, full-text items and exports that are behind, roots whose re-threading died (skeleton_hash set, rethreaded_at not), recent threads with a person's message after their label, and open backfills. Re-threading and labels are swept once an hour, a batch spread out over minutes, and a job that died within the day waits. A dead refile does not heal by itself, since sweeping would apply today's rules to every channel, and waits to be sent out again.

Open questions

  • Slack Connect: does adding the app to a shared channel require approval from the other workspace?
  • Re-threading: what confidence threshold keeps false merges rare, measured on real follow-up posts?