docs: add public API architecture plan and ADRs

[epipav]

No description provided.

[Copilot]

Pull request overview

Adds a documentation set for the planned LFX Insights Public API, including a detailed project plan, an architecture review packet, and a set of ADRs intended to lock down key contract and infrastructure decisions for a standalone /api service.

Changes:

• Introduces a comprehensive public API project plan covering architecture, epics, rollout stages, and observability strategy.
• Adds an “architecture review” doc set (overview, decisions, domain context) to present the proposal for approval.
• Adds ADRs formalizing major decisions (framework, versioning contract, auth model, caching, docs stack, etc.).

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
docs/PUBLIC_API_PLAN.md	Full project plan for the standalone public API service (architecture, epics, rollout, metrics).
docs/CONTEXT.md	Canonical domain language + wire-format conventions for the public API.
docs/architecture-review/01-overview.md	Architecture review overview document for stakeholders/approvers.
docs/architecture-review/02-decisions.md	Summary of “ADR-bar” decisions with links to ADRs.
docs/architecture-review/03-context.md	Architecture-review version of the canonical domain context.
docs/adr/0001-fastify-over-nestjs.md	ADR selecting Fastify over NestJS/Express/Hono.
docs/adr/0002-api-at-repo-root.md	ADR placing the service at repo-root api/ (not under workers).
docs/adr/0003-tolerant-reader-versioning.md	ADR defining /v1-alpha → /v1 stability and additive-only contract.
docs/adr/0004-server-to-server-cors-deny.md	ADR for server-to-server only (CORS) posture in v1.
docs/adr/0005-tiers-control-rate-limits-only.md	ADR for tier impact limited to rate limits in v1.
docs/adr/0006-long-lived-api-keys.md	ADR for long-lived, manually-rotated API keys.
docs/adr/0007-collections-only-permission-check.md	ADR limiting permission checks to Collections endpoints only.
docs/adr/0008-typebox-code-first-openapi.md	ADR choosing TypeBox code-first schemas as OpenAPI source.
docs/adr/0009-api-key-required-for-all-requests.md	ADR requiring API key auth for every request (no anonymous access).
docs/adr/0010-billing-bundled-with-lfx-membership.md	ADR bundling API access with existing LFX membership.
docs/adr/0011-pagination-page-pagesize-zero-indexed.md	ADR standardizing zero-indexed page/pageSize pagination.
docs/adr/0012-url-port-strategy-hybrid.md	ADR describing hybrid URL port/rename strategy.
docs/adr/0013-origin-cache-only-private-cache-control.md	ADR specifying origin-only Redis caching + Cache-Control: private.
docs/adr/0014-camelcase-json-iso8601-dates.md	ADR standardizing camelCase JSON + ISO-8601 UTC timestamps.
docs/adr/0015-api-keys-stored-in-auth0.md	ADR storing/managing API keys in Auth0 via Management API.
docs/adr/0016-vitepress-scalar-api-docs.md	ADR choosing VitePress + Scalar for API docs in api/docs/.
docs/adr/0017-collections-queries-not-shared.md	ADR keeping Collections SQL read queries in /api (not shared).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[emsearcy]

Left some comments throughout. A couple high-level notes:

Are you familiar with https://github.com/linuxfoundation/lfx-architecture-decisions/tree/main/decisions ? In particular I didn't see anything here about logging framework.

(Caveat, ADR-0003 requires sending both OTEL and DataDog-formatted trace and span IDs (both 64-bit unsigned ints, as dd.trace_id and dd.span_id) -- this is out of date; Datadog now handles proper OTEL trace/span IDs natively.)

Also, while I get that you may all be Javascript devs, and I think overall your choices are really good, we often times prefer Go when possible for backend services, because it tends to be easier to maintain long term (forward portability guarantees). But I get it if JS is deemed easier by y'all.

[emsearcy]

This is in conflict with ADR 15. Auth0 cannot provide long-lived API keys. It can either provide client_credential grants (exchange a long-lived client ID and secret for a short-lived access token) or authorization_code + refresh_token grants (login to get a refresh token, exchange the refresh token for a short-lived access token).

So either we need to change ADR 15 to adopt Postgres storage of long-lived keys, or change this one and adopt OAuth2 patterns earlier than v2.

[joanagmaia]

@jonathimer imo this would also be a business decision as it will affect user experience:

Long-lived keys (change ADR 15, store in Postgres):

• User generates a key once in the dashboard, pastes it into their app/script, and it just works forever (or until they revoke it).
• No login flow, no token refresh logic on their side.
• Same key works across machines, CI, scripts — easy to share within a team (which is also the security downside).
• If the key leaks, it's valid until someone manually revokes it.
• Not tied up with OAuth2.

OAuth2 now (change this ADR):

• Two sub-flavors depending on the use case:
  • Machine-to-machine (client credentials): user gets a client ID + secret, their code exchanges them for a short-lived token and refreshes it when it expires. Extra code on their side, but standard.
  • User-facing (auth code + refresh): user goes through a login/consent screen in the browser, app stores a refresh token. Better for apps acting on behalf of a user, but means no headless "paste and go."
• Tokens expire, so leaks have a limited blast radius.
• More upfront integration work for the user, but it's a known pattern.

[epipav]

You're right that this was internally inconsistent. We've adopted the OAuth2 refresh token + short-lived access token model rather than trying to force long-lived tokens through Auth0.
What changed:

• ADR-0006 is renamed and rewritten as 0006-refresh-and-short-lived-access-tokens.md. The "API key" a customer receives is now explicitly a refresh token issued by LFX Self-Serve at app.lfx.dev/settings.
• Customers exchange it at POST api.insights.linuxfoundation.org/v1/auth/token — a thin proxy Insights exposes to Self-Serve's token endpoint — to get a short-lived access token (~15 min). The Bearer value sent to the Insights API is always the short-lived access token. - ADR-0015 updated to align: keys are refresh tokens from LFX Self-Serve, not Auth0-managed credentials.

[emsearcy]

This is OK, but I would challenge that opaque pagination tokens/cursors are a better design choice when you're starting from scratch. Also I recommend that any pagination standard also define sorting mechanism (hard coded vs. user-selected, but it needs to be defined).

• The user cannot "cheat" and grab the first page to get a total count, then grab all remaining pages in parallel: requests are forced to be serialized
• Cursors can (and should!) be implemented to provide atomicity, which page offset numbers cannot: the cursor acts as a sort of session to ensure consistency while paging. offset based pagination can introduce repeats or missed data entries. That is: between fetching page N and page N+1, if an item is added which, based on your sort semantics, adds or removes data in pages 0-N, it shifts the results by 1 either way. A lost entry is caused when the item which would have been the first item of page N+1 falls back into the last item of page N, but we don't see it because we already fetched page N, and a duplicate is caused by an insertion causing the last item of page N (that we already fetched) to also be the first item of page N+1

[epipav]

Agreed on both points. Two changes:

1. Cursor-based pagination is now the standard — ADR-0011 is renamed and rewritten as 0011-pagination-cursor-based.md. We dropped the zero-indexed page/pageSize offset model. Your atomicity example (insert between page N and N+1 → first item of N+1 falls back to last of N, silently skipped) is incorporated verbatim into the ADR as reason Frontend init #1.
2. Sorting is now defined: each paginated endpoint declares a closed allow-list of accepted sort values in its TypeBox schema. Wire format: ?sort=field_direction (e.g. name_asc, commits_desc) — same convention as the existing Nuxt layer. Every accepted value must be index-backed. Removing an allowed value or changing an endpoint's default sort is a breaking change under ADR-0003. Defined in the Sort order section of ADR-0011 and in the wire-format conventions in CONTEXT.md.

[emsearcy]

FWIW, Auth0 JWTs do not carry organization information at present. Also need to define "belongs to". There are org-admin authorized individuals (key contacts for memberships, mostly). If the scope of "belongs to" is intended to capture all employees, note that we leave the realm of authorized identities and have moved over to, essentially, self-attestations (I can say I work for anyone). Alignment of employee identity (and ongoing validation thereof) by known employer domains is not presently in scope for LFX as far as I know.

[epipav]

On the JWT org claim: the org claim is now explicitly attributed to the LFX Self-Serve access token, not an Auth0 JWT. The auth model no longer involves Auth0 at the Insights API layer — customers get a refresh token from app.lfx.dev/settings, exchange it at POST /v1/auth/token (an Insights-proxied endpoint to Self-Serve), and receive a short-lived access token that Self-Serve signs and that Insights JWKS-verifies.
On "belongs to": narrowed to "authorized Key Contact of an organization with an active LFX membership — not every employee or self-attested affiliate." The Key Contact check lives in Self-Serve (OpenFGA against v2_organization entities), but the precise moment depends on an open product question (ADR-0015 Q1):

• If a new Insights-scoped refresh token is issued: check happens at issuance — non-Key-Contacts can't obtain the token at all.
• If the existing PAT is reused: everyone holds a PAT already, so the check moves to POST /v1/auth/token exchange time — Self-Serve refuses to mint an Insights access token for non-Key-Contacts even if they possess the PAT.

Either way, the check is in Self-Serve, and the Insights API doesn't touch a membership system. Updated in CONTEXT.md, 03-context.md, ADR-0010, and the Relationships section.
One assumption we'd like your input on: the spec assumes the LFX Self-Serve access token carries org and tier claims that Insights can read at request time for rate limiting. If those claims can't be added, the enforcement model changes significantly. Can you confirm whether Self-Serve can include these?

[emsearcy]

OTEL defines the standard for trace IDs and span IDs, AND how they are propagated. We should not invent our own. This is imperative for universal tracing.

[epipav]

👍 What changed (ADR-0019, ADR-0018, CONTEXT.md):

• X-Request-Id response header dropped entirely.
• W3C traceparent is now the sole HTTP propagation channel — honoured inbound, injected outbound automatically by the OTel SDK. No custom header.
• The only customer-facing exposure of the trace ID is requestId inside the error envelope JSON — that's the value a customer quotes in a support ticket, and it lives in a schema we already own, not a new HTTP header.

[emsearcy]

No where in the spec is it defined how the membership check is wired up. (e.g. if this is using OpenFGA relationships against v2_organization entities, as discussed in F2F?) ADR 10 references this but points to Public API plan, and I don't see anything in this file actually defining it.

Ideally our spec would be prescriptive about the implementation. Or at least a statement "this is out of scope" or explains the human contract needed to fulfill the behavior -- so that AI implementation doesn't go off the rails and try to build something "wrong".

[epipav]

Added a "Where membership is enforced" section to ADR-0010 that's explicit about the implementation

[joanagmaia]

Hey @jonathimer can you review the main files of this architecture proposal to double check if it's aligned with the initial PRD spec? It would basically be all files outside of docs/adr.
To confirm:

• Where users generate their API Key.
• Tier access limitation -> On the proposal only rate limiting.
• Endpoints access -> At the moment all endpoints available in v1.

Also left you a comment based on a comment from Eric to try to understand how we should define tiers and org mapping with the user.

[joanagmaia]

@jonathimer we some input on how users should create or have access to their API Keys.
What we proposed in this document was: "users are able to get a long-lived token on Insights with their org entitlements". With this information we would be able to also have access to the tier.

• @emsearcy mentioned that this is not possible as of today. In order for Eric to be able to advise us on how to get the token and the required info, we would need to understand:
  • If "belongs to an organization" implies all employees of an org
  • OR only allowlist-authorized individuals (key contacts or similar manual curated list)

#1879 (comment)
From Eric:

FWIW, Auth0 JWTs do not carry organization information at present. Also need to define "belongs to". There are org-admin authorized individuals (key contacts for memberships, mostly). If the scope of "belongs to" is intended to capture all employees, note that we leave the realm of authorized identities and have moved over to, essentially, self-attestations (I can say I work for anyone). Alignment of employee identity (and ongoing validation thereof) by known employer domains is not presently in scope for LFX as far as I know.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 9 comments.

[epipav]

@emsearcy, thanks for the review

Answered the individual inline comments in their respective threads.

• About logging framework: Added ADR-0018 (structured JSON logging via pino), inheriting lfx-architecture-decisions/0002.
• dd.trace_id / dd.span_id: Dropped — Datadog now ingests OTel-format trace/span IDs natively. ADR-0019 updated to reflect this, inheriting lfx-architecture-decisions/0003. W3C traceparent is the sole propagation channel.

[emsearcy]

I'm still concerned with the ambiguity of the API keys. Self Service (which is an Angular UI) should not be handling token exchange. Refresh tokens are probably not the best option for this, either. We probably would be better off with our own signed tokens, or possibly some kind of token service for long-lived tokens. I'll chat with the SSO team in DevOps this coming week.

[emsearcy]

Self service doesn't have a /token endpoint?. Auth0 is our IdP, and it's the one that exchanges refresh tokens for access tokens.

Next, our refresh tokens typically are configured with auto-rotation, which means each time you turn in the refresh token, you get not only an access token, but also your next refresh token.

This is a protection against lost/stolen tokens: if somebody else uses your refresh token, and you attempt to exchange the same refresh token, both are automatically invalidated, forcing a new interactive login.

All of these flows are based on the expectation that refresh tokens are automatically managed in the application state of an application (like a mobile app or desktop app) that can both handle an interactive login and manage its own state.

[emsearcy]

correct, though I will add, revocation of a JWT is a thing, but it's usually done by creating a denylist: a denied JWT (even an access token!) has its unique ID (jti) added to a revocation list. Since JWTs would be expected to expire, you only need to keep it on the revocation list for the token lifetime—this avoids unbounded revocation lists. Of course, that doesn't make sense with an "expiration-less" JWT.

really one would be more likely to simple use a database of API keys, like GitHub PATs, rather than JWTs, for non-expiring credentials.

[emsearcy]

What does Key Contact gating mean in this context? If you are providing different tiers, for example: are you only expecting this to gate on Key Contacts? or also to provide a "tier" claim in the ticket?

[emsearcy]

This is a big assumption—this is not trivial. Self Service does not create access tokens—so this is more of a need for some kind of Auth0 extensibility (like CDP roles is today). Also, projects have different tier names/structures (like Premier/Silver for ASWF): how do you want to handle that for the product side? Or, is claim only showing the organization's LF membership tier, e.g. I'm a Gold at CNCF but Silver at LF, so I get "Silver"?

[emsearcy]

Hi — assuming we end up with JWTs in the final model - you might instead consider the claim http://lfx.dev/claims/username which we often add to access tokens to carry an LFID (the one you see with an auth0| prefix is NOT reliably a LF username. It is unique (and immutable) per user, so you can use it for internal access if you need to, but it is a violation of our LFID integration guide to attempt to extract an LFID by stripping the "auth0|" prefix.

You might consider using user.id or enduser.id for the span attribute, as both of the these are semantic conventions (and are already indexed facets in Datadog!)

https://opentelemetry.io/docs/specs/semconv/registry/attributes/enduser/
https://opentelemetry.io/docs/specs/semconv/registry/attributes/user/

(I think "user" is more used in applications or webapp RUM, and "enduser" is more appropriate for APIs, but I could be mistaken)