Skip to content

feat(endpoint-microsub): PR 1 - Core Microsub server with channels and timeline - #944

Merged
paulrobertlloyd merged 17 commits into
mainfrom
microsub/pr1-core-channels-timeline
Oct 10, 2026
Merged

paulrobertlloyd merged 17 commits into
mainfrom
microsub/pr1-core-channels-timeline

Conversation

@rmdes

@rmdes rmdes commented Sep 5, 2026 •

Copy link
Copy Markdown
Collaborator

Part one of seven: the core Microsub server — channels and the timeline. Scoped deliberately small so it is reviewable; the remaining six are written and follow behind it.

+3511 / −0 across 32 files, all new. No existing behaviour is touched.

What it adds

lib/controllers/microsub.js    action dispatch for the Microsub endpoint
lib/controllers/channels.js    channels: list, create, update, delete, order
lib/controllers/timeline.js    timeline reads, marking read/unread, removal
lib/storage/channels.js        channel persistence
lib/storage/items.js           item persistence, using getObjectId from @indiekit/util
lib/utils/auth.js              token verification and scope checks
lib/utils/pagination.js        cursor-based paging
lib/utils/uid.js               channel and item identifiers
lib/utils/validation.js        request validation
locales/en.json                strings

17 test files cover them.

Verification

Green: 895 tests, 894 pass, 0 fail on the full monorepo suite, endpoint-microsub included.

Worth stating plainly because it was not true until today. This is the successor to #829, which had carried a red check since February for two reasons, neither in this code:

Both are now cleared — this branches from this repo, and feat/microsub carries #914's fix as of e4c2821, byte-identical to main. So the six parts that follow will not inherit a red check either.

One open question

Should this target main rather than feat/microsub? I asked on #829 in August and it got lost in the write-access thread. feat/microsub is 32 commits behind main and, apart from that one backported fix, has no commits of its own — so if all seven parts land there it will keep drifting. Happy either way; it just needs deciding once rather than seven times.

Notes for review

  • @indiekit/util's getObjectId is used throughout rather than the plug-in declaring mongodb itself, per your note on feat(endpoint-microsub): PR 1 - Core Microsub server with channels and timeline #829 — there is no mongodb dependency here, so nothing to drift when the host's version moves.
  • Running the suite locally needs PASSWORD_SECRET set alongside SECRET, as build.yml does. Without it the session-backed tests redirect rather than fail cleanly, which is not obvious from the output.

@rmdes

rmdes commented Sep 5, 2026 •

Copy link
Copy Markdown
Collaborator Author

Correcting my comment I had the shape of this right but missed the history, which matters for how it gets fixed.

feat/microsub is at ba8e19a7, which is the commit that introduced the Prettier failure (#910: "has failed on packages/endpoint-share/README.md since ba8e19a, when the rel="share-url" example was added"). So the branch was pinned at the break and had received neither fix.

More importantly, the fix on main is not "run Prettier". #910 did that and you pushed back:

We should stick to one line and avoid the non-self-closing tag. Prettier's HTML linting is a bit suspect, and we disable it in .prettierignore … Let's use <!-- prettier-ignore --> instead.

#914 then did it your way. The branch already had the one-line, non-self-closing form you asked for — all it lacked was the marker, so anyone clearing this with prettier --write would have reintroduced the split, self-closing version you rejected.

Now fixed on feat/microsub in 2d7579e, which takes main's version of the file wholesale — a one-line addition, byte-identical to main. Not a reformat.

That said, the retarget question still stands on its own merits: feat/microsub is 32 commits behind main with no commits of its own beyond that one, so if the seven Microsub parts are going to land there it will keep drifting. Happy either way — just say which.

@rmdes rmdes closed this Sep 5, 2026
@rmdes rmdes reopened this Sep 5, 2026
@rmdes rmdes closed this Sep 5, 2026
@rmdes rmdes reopened this Sep 5, 2026
@rmdes

rmdes commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator Author

Finally !! damned this was convoluted to make it work but now we're good to go !

@rmdes rmdes added the plugin-endpoint Endpoint plug-in label Sep 7, 2026
@paulrobertlloyd
paulrobertlloyd changed the base branch from feat/microsub to main October 5, 2026 22:43
@paulrobertlloyd
paulrobertlloyd force-pushed the microsub/pr1-core-channels-timeline branch from 83593bf to 288f026 Compare October 5, 2026 22:47
@paulrobertlloyd

Copy link
Copy Markdown
Collaborator

@rmdes I’ve rebased this against main; once this PR is approved, let’s merge it into main, and release this feature piecemeal. Two remaining things to address, I think:

  • It’d be good if the Microsub endpoint could use shared pagination cursor methods. That’d no doubt need some tweaking to our existing methods, but perhaps less so once Paginate by properties.uid #975 lands?
  • It’d also be good to ensure we update the Microsub section of the Supported IndieWeb specifications page on the website. Might you be able to list all the constituent components of the specification, and mark those that this PR ticks off (much like for Micropub on the same page)

I’d like to get this into the next beta release, Beta 30, if possible.

@rmdes
rmdes force-pushed the microsub/pr1-core-channels-timeline branch from 4f16c2f to d52ee6f Compare October 9, 2026 19:15
rmdes and others added 12 commits October 10, 2026 08:45
…imeline

This PR adds the foundational Microsub endpoint with:

**Microsub API:**
- GET/POST ?action=channels - list, create, update, delete, reorder channels
- GET/POST ?action=timeline - list items, mark read/unread, remove

**Storage:**
- MongoDB collections for channels and items
- Cursor-based pagination for timeline
- Per-user channel ordering and read state tracking

**Features:**
- Follows Microsub spec for channel and timeline actions
- Testable with existing Microsub clients (Monocle, Indigenous, etc.)
- Multi-user support via userId from session/token

This is PR 1 of 6 for the Microsub implementation. Future PRs will add:
- PR 2: Feed discovery and subscription
- PR 3: Feed fetching and parsing
- PR 4: Reader UI
- PR 5: Compose and Micropub integration
- PR 6: Settings and filtering

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The plug-in declared mongodb ^6.0.0 while indiekit declares ^7.4.0, so npm
installed a nested copy of the driver. ObjectId values created by the plug-in
came from bson 6 but were passed to collections served by bson 7, which threw
BSONVersionError in markItemsRead, markItemsUnread and removeItems.
Fixes unicorn/prefer-await, unicorn/prefer-number-coercion,
unicorn/consistent-boolean-name, unicorn/no-computed-property-existence-check
and jsdoc/reject-function-type, and removes unused eslint-disable directives.

Satisfying unicorn/prefer-await means init() now awaits index creation rather
than leaving it to run in the background, so plug-in initialisation waits for
indexes to be created. Errors are still caught and warned about, and the plug-in
loader already awaits init().
Unit tests cover lib/utils and lib/storage, mirroring the structure of lib/.
Controllers are covered by integration tests, as in other endpoint plug-ins.
Replaces the direct mongodb import with @indiekit/util's getObjectId, as
suggested in review. The plug-in no longer declares mongodb at all, so its
driver version can't drift from the host's — @indiekit/util owns that pin.

This supersedes the earlier version bump, which fixed the same mismatch by
matching the pin by hand and would have needed maintaining.
`generateChannelUid` built its own string from `Math.random()`. `randomString`
from `@indiekit/util` does the same job with `randomBytes`, which is what a
channel identifier should be using.

That changes the alphabet from `[a-z0-9]` to base64url, so the tests asserting
lowercase now assert URL-safe characters instead — that was the actual
requirement, since a uid appears in Microsub request URLs.

Replaces the one `console.info` in the package with `debug`, matching
endpoint-micropub and endpoint-media, and declares the dependency.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WGHR7MuyvBaDbAFfAGUxeT
@rmdes
rmdes force-pushed the microsub/pr1-core-channels-timeline branch from d52ee6f to 7992fbe Compare October 10, 2026 06:46
@rmdes

rmdes commented Oct 10, 2026

Copy link
Copy Markdown
Collaborator Author

Docs PR for the specifications page: #978.

Timeline items get an `id`, a UUIDv7 stamped with their publication date,
listed and paged through util's `getCursor` like posts and media are since
#975. The cursor gains two options for it: a filter every page is confined
to (the channel) and the key path to order on. Clients see that id as the
Microsub `_id` and can mark or remove items by it; the Mongo `_id` no
longer leaves the storage layer. The endpoint's own cursor encoding and
query builders go.
Comment thread packages/endpoint-microsub/lib/storage/items.js Outdated
Items point at their channel by the channel's uid, the string clients
already use, so nothing in the package needs an ObjectId any more: no
casting, no mongodb dependency, and indexes on strings only, in line
with paginating and indexing on properties.uid elsewhere.
Comment thread packages/endpoint-microsub/lib/utils/pagination.js
Comment thread packages/endpoint-microsub/lib/utils/uid.js Outdated
Comment thread packages/endpoint-microsub/lib/utils/uid.js Outdated
Comment thread packages/endpoint-microsub/lib/utils/validation.js Outdated
Comment thread packages/endpoint-microsub/lib/utils/validation.js Outdated
Comment thread packages/endpoint-microsub/lib/controllers/timeline.js Outdated
Comment thread packages/endpoint-microsub/lib/controllers/timeline.js Outdated
Comment thread packages/endpoint-microsub/lib/storage/channels.js
Comment thread packages/endpoint-microsub/lib/storage/items.js Outdated
Comment thread packages/endpoint-microsub/lib/utils/auth.js Outdated
- Errors use the shared localisable strings (BadRequestError.missingParameter,
  invalidValue, missingProperty) through the localiser the validators now
  take; channel not found and the name limit are package-scoped locale keys,
  and the unused title keys are gone.
- uuidv7At moves to @indiekit/util, shared by the uid backfill and the
  Microsub timeline; the channel uid wrapper is a direct randomString(24).
- One collections module replaces the two duplicated collection getters.
- The user is publication.me: a session carries only a token and a scope,
  so the fallback chain never found anything else.
@rmdes

rmdes commented Oct 10, 2026

Copy link
Copy Markdown
Collaborator Author

Both done in this PR now. Timeline items carry an id, a UUIDv7 stamped with their published date, and page through util's getCursor like posts and media since #975. getCursor takes a filter and a key for it; defaults unchanged. The endpoint's own cursor code is gone. Clients see that id as the Microsub _id. The specifications page is #978.

Your review is addressed in 8332296: localisable error strings, uuidv7At shared from util, one collections module, channels referenced by uid with no ObjectId left, and the user is publication.me. Every thread has a note on what changed.

@paulrobertlloyd paulrobertlloyd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, just a few more issues! 😬

Comment thread packages/endpoint-microsub/lib/storage/channels.js Outdated
Comment thread packages/endpoint-microsub/lib/storage/channels.js Outdated
Comment thread packages/endpoint-microsub/lib/storage/channels.js Outdated
Comment thread packages/endpoint-microsub/lib/utils/validation.js Outdated
Comment thread packages/endpoint-microsub/lib/utils/validation.js Outdated
Comment thread packages/endpoint-microsub/lib/utils/validation.js Outdated
Comment thread packages/endpoint-microsub/lib/utils/validation.js Outdated
Comment thread packages/endpoint-microsub/index.js Outdated
Comment thread packages/endpoint-microsub/index.js Outdated
@paulrobertlloyd paulrobertlloyd added this to the v1.0 milestone Oct 10, 2026
…nes like its siblings

Other endpoints announce nothing at init and core already logs each
collection it adds under debug, so the console lines go; index creation
throws like core's does rather than warn and carry on.
@paulrobertlloyd
paulrobertlloyd force-pushed the microsub/pr1-core-channels-timeline branch from 9cb2128 to 66802a0 Compare October 10, 2026 16:44
@paulrobertlloyd
paulrobertlloyd merged commit b6a74a5 into main Oct 10, 2026
2 checks passed
@paulrobertlloyd
paulrobertlloyd deleted the microsub/pr1-core-channels-timeline branch October 10, 2026 16:54
@paulrobertlloyd

Copy link
Copy Markdown
Collaborator

Thanks for much for (the first part of) this contribution @rmdes – we got there eventually! I’ll release this in Beta 30, with the following PRs to be included in future beta releases. 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

plugin-endpoint Endpoint plug-in

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants