[Tracking] Durable knowledge and agents #627

North star

Make Fragment the coordination and interface layer for durable, user-owned knowledge and agents, with replaceable compute, harnesses, storage, and derived views.

A person should be able to use an enrolled computer from their phone, edit and search a protected brain, run a long-lived agent, restore from customer-owned backup, and later connect to a self-hosted coordinator without changing the agent's identity or the content's keys.

This is an umbrella tracker, not one implementation task. Child issues should be independently useful, and PRs should retain the one-concern-per-PR rule.

The three separations

The brain audit, the durable-agent work, and the user-owned compute direction are the same architectural bet:

1. Sources vs. derived views. Brains, chats, and apps are authoritative sources; search indexes, mounts, blogs, dashboards, and automations are rebuildable consumers.
2. Logical computers vs. execution hosts. A person's stable logical computer can be hosted by Cloudflare today or an enrolled Mac mini tomorrow. Hosting is a binding, not the computer's or agent's identity.
3. Durable commitments vs. ephemeral execution. Goals, approvals, artifacts, and effect receipts survive; an individual Hermes turn does not need to resume mid-loop.

The intended product is not "Hermes with checkpointed LLM loops" or "a brain with more features." It is a durable control plane for knowledge and agency where execution can move without moving authority.

State boundaries to preserve

Write these distinctions into each relevant design rather than inventing one universal state format:

• Shared durable state: identities, SOUL.md, portable memories and skills, source documents, goals, artifacts, approvals, and effect receipts.
• Harness-native state: Hermes sessions, compression, native databases, and tool configuration. Preserve it and keep it opaque to Fragment.
• Host-local state: scratch, installed environments, caches, and machine-specific files.
• Derived state: search indexes, rendered sites, dashboards, snapshots, and other rebuildable views.

Another harness should be able to inherit identity, knowledge, goals, and receipts at a turn boundary while starting a fresh native session. Do not require a canonical cross-harness transcript or make Hermes' databases the platform contract.

A brain is shared knowledge, not Hermes' session store or the agent's memory of record. Hermes' native learning is a feature to preserve, not a bug to abstract away.

What the brain audit found

Fragment already covers the central knowledge workflow:

• Markdown wikis, source notes, citations, and agent authoring.
• File sync, offline working copies, watching, and conflict preservation.
• A direct browser reader and lexical section search.
• Built-in assets/blobs, generic jobs and triggers, sharing, and a common CLI and agent skill.

The important gaps for richer brains and eventual fbrain replacement are:

• Search over current, unsynced local edits and hybrid lexical/semantic retrieval.
• Folder-limited guests and source-backed shared-folder mounts.
• Organization content ownership, multiple administrators, coordinated offboarding, and recovery.
• Customer-keyed content and private processing.
• Verified import/migration, including access scopes, principals, assets, omissions, and search fidelity.

The access boundary should remain the fragment, not a path inside a fragment. If two folders have different audiences, make them separate fragments and compose them. Adding path ACLs would spread through file reads, storage tokens, sync, history, blobs, search, and exports.

A collection or workspace can reference those fragments under local aliases. Adding a reference grants no access; every source still authorizes its own readers. Initially, mounts should be read-only and source-managed.

Two migration warnings are particularly important:

• Fragment organizations currently govern billing and seats, not fragment ownership or content administration.
• New brain fragments currently default to link sharing unless explicitly created as restricted. Migration must never accidentally widen access.

A plain Markdown copy is not an organizational migration. It does not preserve folder policy, principal identity, mounts, history/provenance, external asset authorization, or a reviewable rollback. Also, the deleted old fbrain browser client and unshipped readable-export helpers should not be treated as current migration surfaces.

Derived views are the broader primitive

The useful generalization is an explicit dependency between fragments:

• A source exposes a named, bounded interface.
• A consumer is explicitly authorized to consume it.
• Changes carry source identity and revision/cursor information.
• A notification causes reconciliation through the consumer's existing jobs; it is not the sole durable copy of the content.
• The consumer records the incorporated revision and can rebuild from current source state.

The first proof should be published Markdown posts in a private brain rendered by an explicitly public blog. This forces authorization, intentional disclosure, revisions, deletion, and rebuilding to become real. The same relationship can later support a read-only mount, search index, dashboard, or automation without the cell asking whether a source is a brain.

Distinguish two products:

• A private alternate reader checks every visitor's source authority.
• A publishing view intentionally discloses selected source content to the destination's audience.

Authorization to publish is authorization to disclose. Revocation stops future reads; it cannot recall published copies.

Search should separate a common searchable-document shape from ranking. Files, chat messages, and app records can all produce documents with source, ID, version, location, and text. Lexical and semantic indexes consume those documents. Indexes remain derived, scoped to their source's access boundary, and must not return stale excerpts as current truth.

Start local: search the CLI working tree as it exists now, including unsynced additions, edits, deletions, and conflicts. Add semantic retrieval behind a pluggable local or explicitly trusted remote provider. Do not begin with a shared vector service or a broad browser capability that lets an app read everything its owner reads.

Computer identity and hosting

The current model already separates the agent fragment's identity from the container. Preserve that. The smallest sound attached-host model is:

• Keep one stable logical computer per person.
• Enroll devices with device-generated host keys; only public keys leave the devices.
• Bind exactly one active host to the logical computer at a time.
• Persist an activation_epoch for every host transition.
• Check that epoch on signing, model, credential, subscription, port, checkpoint, and readiness/commit operations, not only at connection time.

A device key authenticates a host. It is not the agent identity, the logical computer identity, or automatically a decryption authority.

An attached Mac mini must not become a second logical computer: that would unnecessarily disturb agent assignment, computer-scoped signing, connections, joined delivery, export, and wipe semantics.

The first useful attached slice should include:

• Explicit owner enrollment and attachment.
• An outbound-only authenticated transport.
• The existing Hermes image and bridge with persistent attached /data.
• Central computer-scoped agent signing and provider-secret brokering.
• Ordinary chat, terminal/file tools, routines while online, and model access.
• Honest online/offline, local-disk, and backup status.

Start with a fresh attached computer. Add movement of an existing computer only after verified checkpoints and restore work. Do not begin with a general scheduler, active-active hosts, load balancing, live migration, or automatic failover.

Connection is not readiness, a local restart is not a cloud restore, and an unacknowledged stop is not sleep. Initially it is fine to defer attached desktop/ports, automatic sleep, and remote wipe guarantees rather than misrepresent them.

For Apple Silicon, native ARM64 packaging or an explicitly supported emulation path is a prerequisite for full Hermes/tool compatibility. /data portability does not prove installed binaries are portable.

Checkpoints and movement

A real attached recovery contract needs a versioned checkpoint manifest covering /data and /data/work as one generation:

• Use the existing hold and Hermes' consistent database backup behavior.
• Commit a checkpoint only after every piece uploads and only for the current computer, device, epoch, generation, and save sequence.
• Let the logical Computer DO select the current checkpoint and reject late writers.
• Restore behind the existing restore gate and integrity checks.
• Retain fallback checkpoints and perform actual restore drills.
• Move hosts manually from a verified checkpoint, fencing the old epoch before the new one becomes active.

Backup upload alone is not durability. Until restore is proven, say local disk; no ongoing cloud backup.

The turn journal remains a backstop against replaying claimed turns, not a complete split-brain fence: it cannot undo an email, purchase, provider call, or write already performed by a disconnected machine.

Truer durability without flattening Hermes

Fragment already has durable saves, restore fallback, stable identities, authoritative chats and turn journals, scheduled records, and Workflow jobs. Its deliberate one-life-per-turn rule chooses a visible lost turn over rerunning possible external effects. Keep that rule.

The goal is: make commitments durable, not processes permanent.

Hermes should continue to own:

• Reasoning, native sessions, and context compression.
• Terminal, browser, desktop, and other computer-use tools.
• Model fallback and its native tool behavior.
• Personality and self-improving memories and skills.
• Outcome-level, non-technical owner questions.
• Bot Mode and visible agent handoffs.

Fragment should own durable goals, timing, source records, artifacts, approvals, and evidence of effects.

Near-term durability work

1. Durable goals as data. Store desired outcome, status, next action, wake condition, artifacts, and effect IDs in Fragment state. Scheduled records bring in Hermes; every invocation is a fresh turn that checkpoints its progress. Start as a template and Hermes skill, not a Hermes-specific cell API.
2. Restart-safe waits. Finish P6: a card outlives its original turn, and a late answer starts a new turn with the question, answer, and recovery context. Stop keeping a computer awake merely to wait.
3. Restart-safe handoffs. Reconcile unsettled Bot Mode deliveries from the teammate's journal when the keeper starts; wake the sender from that receipt rather than a dead process.
4. Prompt self-sync. Use supported hooks to mark memory/skill changes dirty and run the existing three-way sync promptly. Surface persistent sync failure. Preserve Hermes' native learning rather than replacing it with a separate memory stack.
5. Truthful tool events and receipts. Use Hermes' supported lifecycle/tool hooks beside Relay to record structured outcomes. For consequential effects, retain goal/turn/tool IDs, status, idempotency key, and result/artifact reference. Recovery should ask what happened, not infer it from progress text.
6. Reversible upgrades. Preserve the save from which an image upgrade started outside the three-save rotation, so an older image is not asked to read a newer schema without a matching checkpoint.

Use stable idempotency keys and provider reconciliation where possible. Do not promise generic exactly-once external effects: a retry is safe only when the upstream honors idempotency or the effect can be checked.

Later, continue the existing local terminal-backend to separate-sandbox path. That improves execution boundaries and state placement, but does not by itself make arbitrary commands idempotent or model turns resumable.

Privacy and custody are separate milestones

Provider-managed disk encryption and platform-sealed secrets are not customer-owned content access. Equally, attached compute alone is not end-to-end encryption or local-only inference.

Use explicit product modes:

• Hosted: Fragment hosts compute and may process plaintext.
• Attached: the user hosts compute; the coordinator may still process plaintext content and chat.
• Private: customer-keyed content and endpoint-controlled processing/inference leave the coordinator with ciphertext and explicit metadata.
• Self-hosted coordination: the customer can replace the coordinator as well.

The practical private-knowledge direction is one versioned encrypted content format with customer-controlled recipient grants:

• Customer devices authorize recipients and organization recovery.
• Hosted processors are optional, explicit recipients.
• Host enrollment and decryption grants remain separate.
• Account recovery or a registry edit must not silently manufacture decryption access.
• Loss of every customer and recovery key is honestly unrecoverable.

Start with protected knowledge fragments, not transparent encryption of arbitrary fragment SQL, jobs, and channels. Protect sensitive derivatives too: indexes, embeddings, workflow inputs/results, agent memories, logs, saves, exports, chat excerpts, summaries, and notifications can all disclose source content.

Inference and customer-backup connections should originate at the trusted endpoint in private mode. Local inference and explicitly trusted TEE inference can share that endpoint contract. External key services and attested processing are possible later options, not prerequisites for the first encrypted content format.

The honest target is zero routine custody of plaintext and content-decryption keys, not zero stored data and not a claim that an operator could never subvert a delivered client or authorized runtime.

Sequenced roadmap

Stage 1: useful on the current hosted architecture

• CLI local-current brain search, lexical first and hybrid second.
• Collection manifests and read-only CLI composition of independently shared fragments.
• One durable-goal template/skill using existing operations, schedules, and jobs.
• P6 late-answer continuation, Bot Mode reconciliation, and prompt memory/skill sync as separate PRs.
• Write down the source/revision, computer/device/epoch, and goal/turn/receipt contracts before generalizing them.

Exit: a multi-day goal survives active and waiting restarts; late approvals and handoffs settle once; unsynced brain edits are searchable; separately shared fragments compose without widening access.

Stage 2: prove derived views and one attached host

• Named source interfaces, explicit dependencies, revisioned notifications, and full rebuild.
• Private-brain published Markdown to public blog as the first derived-view vertical slice.
• Fresh attached Hermes computer on one enrolled host, with outbound transport and activation fencing.
• Structured Hermes lifecycle/tool events beside Relay.
• Truthful attached-mode limitations; no automatic failover or general scheduler.

Exit: a phone chats with Hermes on an attached host without replay after reconnect; native Hermes behavior remains; a blog reconciles from source revisions after missed notifications; dependency revocation stops future reads.

Stage 3: real recovery and customer-keyed knowledge

• Versioned, fenced checkpoints and restore drills; manual host movement from verified saves.
• Reversible Hermes image upgrades.
• Encrypted knowledge content, customer/recovery recipients, and customer-owned encrypted backups.
• Private local search and derived processing on an authorized computer; endpoint-direct inference.
• Organization content ownership/administration/offboarding/recovery before organizational migration.

Exit: hosted compute can be disabled while an attached computer runs the agent and private search; a protected brain restores from customer-owned backup; routine storage administration cannot read it.

Stage 4: replaceable coordination, more harnesses, and migration

• Private encrypted chat and endpoint interfaces.
• Self-hosted coordinator interoperability without changing identities or content keys.
• A second harness implementing the generic turn/goal/approval/receipt contracts; harness changes only at turn boundaries.
• Verified fbrain organizational migration after governance, recovery, access mapping, assets, and search fidelity are ready.

A migration report should include source-to-destination scopes and principals, content counts/hashes, omissions and locked inputs, asset results, link/search checks, resume boundaries, and rollback ownership.

Explicit non-goals for the early stages

• Directory ACLs inside a fragment repo.
• A general distributed scheduler, active-active Hermes, automatic failover, or live migration.
• Replaying an interrupted Hermes turn or enabling auto-continue as the durability model.
• A canonical cross-harness Hermes-session format.
• A Workflow-only replacement that sacrifices Hermes' native tools, sessions, learning, and desktop.
• Treating always-on, snapshots, or a vector database as durability.
• Transparent encryption of every fragment primitive in one redesign.
• A broad read everything the owner reads browser capability.
• Organizational migration before access, ownership, and recovery can be preserved.

Initial vertical slices / child issues

1. Brain: search current working trees. Local lexical search with sync disposition; semantic indexing as a follow-up.
2. Brain: compose collections. Manifest and read-only CLI browsing/search over independently shared fragments.
3. Agents: advance one durable goal through fresh turns. Template/skill, one schedule, one approval, one idempotent effect.
4. Agents: preserve approvals across restarts. P6 as its own issue/PR.
5. Agents: reconcile interrupted Bot Mode deliveries. Separate issue/PR.
6. Computers: enroll one attached host. Fresh logical-computer binding, one epoch, no migration or failover.
7. Derived views: render published brain Markdown as a blog. First cross-fragment source/dependency/revision proof.

Keep this tracker as the north star, invariants, dependency map, and index. Detailed implementation checklists belong in child issues.

Overall acceptance test

Disconnect hosted compute. From a phone, use an enrolled computer to converse with the same agent, advance a durable goal, and edit/search a protected brain. Answer a late approval without duplicating its effect. Restore the brain and computer state from customer-owned backup. Move to a self-hosted coordinator without changing the agent's identity or content keys. Where a second harness is used, hand off only at a turn boundary and preserve the shared goal and evidence, not Hermes' private session.

Current-tree references

These references describe the starting point at commit 01d5b4474618995a79fde69a574fa30ea1f6549c, not completed roadmap features:

• docs/durable-computers.md — one authority per fact, one life per turn, save/restore, deferred approvals, terminal/sandbox and connector directions.
• docs/computers.md and docs/bridge.md — computer/guest and bridge contracts, /data and /data/work, Hermes behavior.
• docs/chat-records.md — authoritative chat/work records, routines, turn claims.
• docs/api.md — access, files, subscriptions, jobs, schedules, exports, and current search surfaces.
• docs/hermes-relay.md — Relay limitations and supported Hermes lifecycle/tool hooks.
• images/bridge/src/runtime/mod.rs — runtime-neutral bridge boundary.
• images/hermes/boot/src/sync.rs — three-way agent-self synchronization.
• cell/src/runs_on.rs — agent identity, assignment, signing, and wake delivery.
• crates/core/src/computer.rs and cell/src/computer.rs — logical computer lifecycle, generations, saves, and current Cloudflare couplings.
• templates/brain/README.md and templates/brain/app.mjs — current brain access, files, generation-fenced derived index, and search.
• docs/technical-debt-ledger.md — known upgrade-rollback, tool-event, approval, Bot Mode, and hosting gaps.