SWIP-065: Self-indexed feeds - #106
Open
zelig wants to merge 10 commits into
Open
Conversation
Signed as a feed update, carried as the bare index: every BPS message under an explicit-publisher FEED_TOPIC cohort is simultaneously a live message and a retrievable feed update. Payloads are whirl-only pot nodes, so the feed carries its own index — one chunk per update, full history from the latest message. Settles dedup (address rule suffices), makes gaps recoverable, and turns SWIP-61 dual-parenting into a choice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gap-revealing frame carries the pot over everything missed, and fork references are content addresses, so one already-verified signature authenticates the whole recovery — hash checks, not signature checks. Feed-address retrieval demoted to the no-pot-profile fallback and the plain-feed-follower path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sequential indices advance by exactly 1 and iteration order is publication order, so the descent assigns indices by counting from the known anchor: the explicit IDX on the BPS frame, or the lookup index a plain-feed client just used. No index needs to ride in the pot nodes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- KEY defaults to timestamp: SOC addresses are already the O(1) secondary index by update index, so an IDX-keyed pot would duplicate the feed; the timestamp key buys seeking by time. - Fork refs = wrapped CACs confirmed; SOC-address alternative rejected outright (also duplicates the by-index path). - Pot profile optional confirmed; bare cohorts recover per missed index. - Equivocation: detection and error signalling made normative via the anchor check (SOC at last known index wraps the pot node in hand, implicitly validating the whole pot since join); response stays with the application; new conformance item 5. - Multi-publisher: one feed per publisher, merge language dropped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Dual parenting relaxed to not-required (pure extra cost for a subscriber content with recovery latency); SWIP-60 and SWIP-61 get follow-up amendments after this SWIP settles, both now listed definitively in Backwards compatibility. - New contrast paragraph in Validation: without a detectable order (ANCHOR binding) the republication guard is wrapped-address dedup, sound only if the application puts an index in the payload anyway — self-indexing moves that requirement into the construction. - Per-subscriber parenting choice made definitive: coherent because dual parenting is the costlier-but-faster alternative and latency tolerance is local. No resilience flag; CohortSpec untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Persistence header elevated: publish-is-upload named the pivotal switch of the SWIP. - Reorder mode scoped away: meaningful only for non-self-indexed cohorts over actual multihop trees; belongs to those SWIPs. - SOC upload MAY phase in later; wrapped chunk is the operative upload. - Pot maintenance: already exists in bee-js. - bps-history: for self-indexed cohorts the problem dissolves — the first verified frame carries the pot over the entire history, so history:true collapses into client-side descent. - O(log n) left unqualified; POT reference placeholder awaits the overleaf link. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
zelig
added a commit
that referenced
this pull request
Aug 9, 2026
Wrapped-CAC dedup under ANCHOR guards against unsolicited republication of old SOCs, and is sound only if the application guarantees distinct payloads - i.e. includes some index in the payload (per the SWIP-65 discussion: without self-indexing the sequence requirement moves above the protocol, unspecified). The two 'SWIP-65 (forthcoming)' anchors now link PR #106. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
zelig
added a commit
that referenced
this pull request
Aug 9, 2026
Per SWIP-65 (PR #106): a delivery gap on a self-indexed cohort is detectable and recoverable from storage, so the contract is satisfied by deliver-or-recover. The second parent becomes a per-subscriber latency/cost choice - single-parented nodes were already well-formed, and a single-parented relay endangers only itself. Conformance items 5-6 bind only nodes that maintain a second parent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 9, 2026
Nothing in self-indexed feeds requires BPS. Restructured accordingly:
the construction is an ordinary sequential feed (full id in the SOC),
pot layer / recovery / anchor check / persistence are all specified
transport-free ('follower', not 'subscriber'), and everything
SWIP-60-specific is consolidated into one 'Carriage over BPS' section
(bare-index rule, broker validation, API amendments). requires: 60
dropped from the frontmatter; new rationale lead 'Why is this not a
BPS SWIP?'; conformance reordered generic-first with BPS items 6-7.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The worked example now shows two code blocks: the generic player (watch/seek — any feed client, no BPS session) and the BPS-carried live loop (on_frame with gap backfill by descent). The backfill is the generic recovery mechanism verbatim, stated as such. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR introduces SWIP-065, specifying a “self-indexed feed” pattern where each sequential feed update embeds a whirl-only pot node that indexes the full history, enabling gap detection/recovery and efficient seeking from the latest update alone, with an optional carriage rule for BPS (SWIP-60/61).
Changes:
- Adds a new standards-track SWIP describing the self-indexed feed construction, payload/index format, and follower recovery behavior.
- Specifies BPS carriage using bare indices on the wire and explains dedup/recovery implications for SWIP-60/61.
- Defines conformance and backwards-compatibility expectations for implementations.
Suppressed comments (3)
SWIPs/swip-65.md:75
OWNERis referenced again here, but the section defines the owner asO. This should match the symbol used in the address derivation above.
follows it, and update `j` is individually addressable at `H( H(TOPIC ‖ j) ‖ OWNER )`.
SWIPs/swip-65.md:325
MAXis used here but isn't defined in the pseudocode or earlier notation. Using the same explicit maximum key described above (2^64 - 1) keeps the example self-contained.
live_window ← first k of Iter(n, MAX, DESC)
SWIPs/swip-65.md:161
- This inline-code expression uses Unicode minus (
−). Using ASCII-keeps the spec easier to copy into implementations and consistent with other SWIPs.
later index — is evidence of `IDX − w − 1` missed updates, and, with the pot profile,
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| author: Viktor Trón (@zelig), Viktor Tóth (@nugaon) | ||
| discussions-to: https://discord.gg/Q6BvSkCv | ||
| status: Draft | ||
| type: Standards Track (Interface, Networking) |
Comment on lines
+68
to
+71
| id = H(TOPIC ‖ IDX) // the signed id | ||
| a = H(id ‖ OWNER) // SOC address in storage | ||
| sig = SIG_O( H( id ‖ H_BMT(SPAN ‖ PAYLOAD) ) ) | ||
| c = id ‖ sig ‖ SPAN ‖ PAYLOAD |
|
|
||
| ### The construction: an ordinary sequential feed | ||
|
|
||
| Update `i` by owner `O` on topic `TOPIC` is the SOC ⟨`a`, `c`⟩: |
Comment on lines
+113
to
+114
| every node exactly once. Descending iteration from key `2^64 − 1` yields | ||
| newest-first: the live window is its first `k` elements. |
Comment on lines
+322
to
+323
| for j in w+1 .. IDX−1: # hash-verified via fork refs, | ||
| deliver(j, nth_newest(n, IDX−j)) # indices by counting, no sig checks |
Comment on lines
+387
to
+388
| non-self-indexed bindings; equivocation *response* policy (detection and error signalling are normative | ||
| above); multi-publisher cohorts need nothing here — one self-indexed feed has one |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A self-indexed feed is an ordinary sequential feed whose update payloads are nodes of a whirl-only pot over all previous updates: the feed carries its own index. One chunk per update, full history, seeking and recovery from the latest update alone, zero auxiliary chunks. The pattern is transport-agnostic — nothing in it requires BPS; any feed client can publish and follow one.
Carried over BPS (SWIP-60, #104) it does double duty — every live message is simultaneously a retrievable feed update, transmitted as the bare index (signed id =
keccak256(topic ‖ index), topic implicit in the channel), so a gap is visible at the very next frame. This is the sequential construction SWIP-60 promised, and the only SWIP-60-specific part of the spec.What it settles:
CohortSpecuntouched.swarm-postage-batch-idon the WS session makes publish-is-upload: an ephemeral live stream becomes a persistent, seekable, recoverable feed. For a plain feed publisher, persistence is simply what publishing is.history: truecollapses into client-side descent.Worked example: HLS-style live video — the pot is the manifest; live window by descending iteration, seek in O(log n), the final update alone is the VOD artifact playable by any feed client; nothing republished, nothing polled. BPS adds push latency at the live edge; nothing structural.
Companion amendments landed on #104 (ANCHOR-binding dedup soundness requires an application-level payload index) and #105 (conformance items 5–6 relaxed for self-indexed cohorts).
DISC: NO — SOCs, feeds, postage, push-sync and retrieval used exactly as they are; a chunk-payload convention plus client behaviour.
🤖 Generated with Claude Code