Token API: server-side kNN /similarity endpoint + concurrency guard (don't starve live users)

docs/superpowers/specs/2026-06-28-similarity-endpoint.md

@@ -0,0 +1,160 @@
+# Token Similarity (k-NN) Endpoint — Design
+
+**Date:** 2026-06-28
+**Branch:** `feature/similarity-endpoint`
+**Driver:** Live API feedback (MMF/Sharkbook) — the **critical** scaling bottleneck. To find matches an
+agent must download every catalog annotation's 2,152-float vector (≈0.5–0.9 s/annotation; a 200-record
+page ≈176 s; >10k unwalkable). Wildbook already does OpenSearch kNN internally for matching
+(`Annotation.getMatchQuery`, nested `knn` on `embeddings.vector` filtered by method/version). Surface
+that as a read-only token endpoint so the agent gets nearest neighbors **without downloading vectors**.
+
+## Endpoint
+`POST /api/v3/similarity` — token-auth (same `tokenAuthSearch` Shiro filter as search), read-only. A new
+path (not under `/api/v3/search/*`, which the SearchApi servlet owns).
+
+### Request
+```json
+{ "annotationId": "<uuid>", "k": 50, "methodVersion": "msv4.1",
+  "taxonomy": "Equus quagga", "locationId": "Mbuluzi Game Reserve", "excludeSameEncounter": true }
+```
+- `annotationId` (**required**) — the query annotation. The server reads ITS stored embedding; the
+  caller never sends a vector (v1 has no raw-vector input — avoids untrusted vectors and is the whole
+  point: no vector download).
+- `k` (optional, default 50, **max 100**) — number of neighbors.
+- `method` (optional, default `miewid`) / `methodVersion` (optional) — which embedding to compare. If
+  omitted and the annotation has exactly one (method,version) embedding, use it; if it has several and
+  `methodVersion` is omitted → 400 (ambiguous).
+- `taxonomy` (optional) / `locationId` (optional) — explicit scope filters, applied as exact term
+  filters on `encounterTaxonomy` / `encounterLocationId` inside the candidate filter. **v1 uses these
+  explicit, safe scope params rather than an arbitrary caller `query`** — this avoids needing to
+  validate a free-form ranking-oracle filter (deferred to a future version with a strict allow-list).
+- `excludeSameEncounter` (optional, default false) — also drop candidates sharing the query's
+  `encounterId` (the skills' anti-inflation rule).
+
+### Behavior
+1. Resolve the query annotation. If the caller can't see it (ACL) or it doesn't exist → **404**, the
+   two indistinguishable (no existence oracle, mirroring media/resolve).
+2. Get its embedding for (method, methodVersion). None → **400** `"no embedding for that
+   method/version"`.
+3. Build the nested `knn` query (vector + `k`) filtered by `embeddings.method`/`methodVersion`
+   (reuse the `getMatchQuery` construction).
+4. **Enforce correctness server-side:** candidates are constrained to the **same `viewpoint`** as the
+   query annotation (skip this constraint only if the query's viewpoint is null), and the same
+   method/version (already in the knn filter). Always **exclude the query annotation itself**; if
+   `excludeSameEncounter`, also exclude its `encounterId`.
+5. **ACL-scope candidates:** wrap the candidate query with `applyAclFilter` for non-admin (admins
+   unfiltered), so neighbors are only annotations the caller may see.
+6. Merge the optional caller `query` as an additional `filter`.
+7. Execute via `queryPit` with size = k; return the top-k.
+
+### Response
+```json
+{ "query": { "annotationId": "...", "viewpoint": "left", "method": "miewid", "methodVersion": "msv4.1" },
+  "neighbors": [ { "id": "...", "score": 0.87, "encounterId": "...", "individualId": "...",
+                   "viewpoint": "left", "iaClass": "zebra", "methodVersion": ["msv4.1"] }, ... ] }
+```
+- Each neighbor is the **sanitized** annotation doc (ACL-scrubbed, like search hits) **with the
+  `embeddings` field stripped** + a `score` (the OpenSearch kNN cosine similarity). Stripping
+  `embeddings` is essential: it keeps the payload tiny (the whole point) and never returns vectors.
+- `score` lets the agent threshold/rank (calibrate per the skill guidance).
+
+## Security
+- Token-auth, read-only; candidates ACL-scoped via `applyAclFilter`; query annotation must be visible
+  (else 404, no oracle).
+- **Never returns vectors** (embeddings stripped from neighbor `_source`).
+- `k` bounded (≤100); optional `query` validated (no scripts/aggs/runtime_mappings); same-viewpoint +
+  same-method/version enforced (prevents cross-latent-space comparison the skills warn against).
+- Self (and optionally same-encounter) excluded so the top hit isn't the query itself.
+
+## Metrics (REQUIRED — this is user-run re-ID compute on our servers)
+A similarity call is a re-identification query executed on our OpenSearch. It creates **no `ia.Task`**,
+so the existing identification gauges (`MetricsBot` JDOQL counts of `Task` rows with
+`ibeis.identification`/`pipeline_root`/`graph`) will NOT see it. Add a dedicated metric so this re-ID
+channel is visible in `/metrics` and impact reporting:
+- A Prometheus **Counter** on `io.prometheus.client.CollectorRegistry.defaultRegistry` (the same
+  registry `MetricsResource`/`WildbookMetrics` expose), e.g. `wildbook_token_reid_queries_total`
+  (label: `context`), incremented once per successfully-served similarity query. Optionally also a
+  neighbors-returned counter and a latency histogram.
+- Keep it **distinct** from the pipeline `wildbook_identification_tasks*` gauges — token-API re-ID is a
+  separate, synchronous, agent-driven modality; conflating the two corrupts identification counts (cf.
+  the v2-detection-gauge pollution incident). Increment only on a real query that ran the kNN (not on
+  400/404/validation rejections).
+- Counter resets on restart (standard Prometheus semantics; the scrape/TSDB accumulates). If a durable
+  all-time total is also wanted, additionally persist a running total (e.g. a `SystemValue`) — optional.
+
+## Reuse / files
+- New servlet `org.ecocean.api.SimilarityApi extends ApiBase` (`doPost`); web.xml servlet + url-pattern
+  `/api/v3/similarity`; Shiro `[urls]` `/api/v3/similarity = tokenAuthSearch`.
+- New `OpenSearch.knnQuery(...)` (or reuse `Annotation.getMatchQuery` building blocks) to assemble the
+  nested-knn + filters; reuse `applyAclFilter`, `queryPit`, `sanitizeDoc`.
+- Strip `embeddings` from each returned `_source` before `sanitizeDoc`/output.
+
+## Out of scope (future)
+- Raw-vector input; cross-viewpoint search; paging beyond top-k; multi-annotation batch; returning the
+  query vector. A `/api/v3/agent-skill/find-missed-matches` rewrite to use this endpoint (huge speedup)
+  is a follow-up doc PR once this lands.
+
+## Design review (BINDING — incorporated)
+
+**PREREQUISITE — RESOLVED (verified empirically on zebra.wildme.org 2026-06-28).** Through the existing
+`/api/v3/search/annotation` (which forwards the body to OpenSearch), a nested `knn` with an inner
+`filter` is accepted (HTTP 200) and the filter is **honored**: filtering to the opposite `viewpoint`
+returned 10 `left` neighbors instead of the natural `right` ones, and the filter worked on BOTH a nested
+subfield (`embeddings.methodVersion`) and a PARENT field (`viewpoint`). So the deployed engine supports
+efficient filtered nested kNN; build on native `knn.filter` (parent + nested filters), no mapping
+migration needed. **Heavy-compute note:** kNN is the most expensive token operation — it must be behind
+the token-API concurrency guard (see the separate request-isolation workstream) so it cannot starve
+interactive Wildbook search/matching.
+
+(Original prerequisite text, now satisfied:) verify the deployed engine supports efficient *filtered
+nested* kNN. The
+`embeddings.vector` mapping is `knn_vector` with **no `method.engine`** specified, so filtered-kNN
+support is engine/version-dependent and unconfirmed. The endpoint MUST NOT ship on a global-ANN +
+outer-`bool.filter` (post-filter) basis — that returns fewer than `k`, lets hidden/out-of-scope vectors
+suppress visible neighbors, and is a recall/leak hazard. Resolve one of:
+  (a) confirm the live OpenSearch honors `knn.filter` on the nested `embeddings.vector` (empirically:
+      POST a `knn`+`filter` body through the existing `/api/v3/search/annotation` and check it filters);
+  (b) migrate the mapping to a filter-capable engine (Lucene) + reindex;
+  (c) fall back to exact filtered scoring or a flattened embedding-row index.
+Do not implement until (a) or (b) holds.
+
+**Corrected query shape:** put ACL, viewpoint, method/version, caller scope, and self/same-encounter
+exclusion **inside the native `knn.filter`** (the efficient filtered-ANN path), NOT only an outer bool.
+Keep an outer ACL `bool.filter` as defense-in-depth only.
+
+**Vector non-leak (defense in depth):** set `_source.excludes:["embeddings"]` on the request, reject any
+caller `_source`/`fields`/`docvalue_fields`, AND `remove("embeddings")` from each doc before output.
+
+**Query-annotation visibility gating (no-oracle):** mirror `MediaResolveApi` exactly — token context
+only; parse first; for a non-admin run the singleton id-eligibility ACL query (fresh PIT) to confirm
+visibility BEFORE any DB/vector load; not-visible and not-found return an identical **404** (never a
+distinct "no embedding"/timing signal for hidden ids).
+
+**Caller `query` is now a ranking oracle → strict validation** (stricter than annotation search): allow
+filter-only DSL on a **field allow-list**; reject scripts, aggs, runtime_mappings, `_source`, `fields`,
+sort, post_filter, rescore, `nested`/`embeddings`/`vector`, ACL fields, terms-lookup, and overly
+large/deep clauses (extend the existing `DENY_FEATURES`/`nodeAllowed` deny-list).
+
+**Deterministic method resolution:** do NOT reuse `getEmbeddingByMethod(null,null)` (returns first of a
+set). Require exactly one eligible `(method, methodVersion)` after defaults (method=miewid); else 400.
+
+**Viewpoint-null → 400** (don't silently broaden latent space). **k:** default 50, range 1–100, else 400.
+
+**Cost:** fresh PIT; reject huge/deep caller filters; note token/IP rate-limiting as a platform follow-up.
+
+## Open questions (RESOLVED by review)
+- Path `/api/v3/similarity` (not under `/api/v3/search/*`). • 404 for not-visible/not-found query
+  annotation. • Always the query annotation's viewpoint (no override) in v1; null → 400. • k default 50,
+  max 100, out-of-range → 400. • Do NOT rely on outer `bool.filter`; use native `knn.filter`.
+
+## Original open questions (superseded by the resolutions above)
+1. Endpoint path/name: `/api/v3/similarity` vs `/api/v3/search/similar` (the latter collides with the
+   SearchApi `/api/v3/search/*` mapping → avoid). Confirm `/api/v3/similarity`.
+2. Not-visible/not-found query annotation: 404 (chosen) vs 200-empty — which better preserves no-oracle
+   while being ergonomic?
+3. Should same-viewpoint enforcement be overridable via an explicit `viewpoint` param, or always the
+   query's viewpoint? (Lean: always the query's in v1; overridable later.)
+4. `k` max (100 chosen) and default (50) — reasonable for re-ID shortlists?
+5. Does the OpenSearch nested-knn query honor an outer `bool.filter` (ACL + viewpoint + scope) so the
+   ANN search is restricted to the filtered candidate set (efficient + correct), or does kNN run global
+   then post-filter? (Implementation must confirm the filter applies to the ANN candidate set.)

docs/superpowers/specs/2026-06-28-token-api-concurrency-guard.md

@@ -0,0 +1,131 @@
+# Token-API Concurrency Guard — Design
+
+**Date:** 2026-06-28
+**Branch:** `feature/similarity-endpoint` (ships under PR5, ahead of/with the kNN endpoint)
+**Driver:** Operator requirement — a remote agent may make a *massive volume* of token-API calls (that's
+fine), but it **must not interfere with interactive Wildbook use**. Token search/kNN hit the SAME
+OpenSearch the live UI and the live re-ID matching pipeline use; one sequential agent already triggered
+intermittent `500`s under sustained paging. kNN (PR5) is the heaviest op. We need **resource isolation**,
+not a volume quota.
+
+## Approach: bound *concurrency*, not volume
+A per-instance (per-JVM) **semaphore** caps how many token-API requests execute *simultaneously*. The
+agent may send unlimited requests; only N run at once; the rest get **`429 Too Many Requests` +
+`Retry-After`** and back off. This bounds the agent's instantaneous footprint on OpenSearch/Tomcat,
+leaving headroom for live users, without limiting total work (the agent self-paces). Volume is fine;
+crowding-out is prevented.
+
+### Two pools
+- **GENERAL** — every *token* search / `media/resolve` / aggregation request acquires one permit.
+- **SIMILARITY** — the kNN `/api/v3/similarity` endpoint acquires from a **separate, smaller** pool
+  (kNN is CPU-heaviest). Independent pools (one acquire per request) — no double-acquire, no deadlock.
+Defaults (per instance, tunable): `GENERAL = 16`, `SIMILARITY = 4`, `Retry-After = 5s`. Read via the
+existing `getConfigurationValue(key, default)` mechanism (e.g. `tokenApiMaxConcurrent`,
+`tokenApiMaxConcurrentSimilarity`, `tokenApiRetryAfterSeconds`); fall back to defaults.
+
+### Only token requests are throttled
+The guard applies **only when the request is token-authenticated** (`TOKEN_AUTH_ATTR`). Interactive
+session/UI traffic (the live users we're protecting) is never throttled. Non-token requests don't
+consume a permit.
+
+### Mechanism: in-servlet, not a filter
+Each token-serving servlet acquires/releases around its work (robust, testable, no Shiro-ordering
+dependence):
+```
+if (tokenAuth) {
+    if (!TokenApiConcurrency.tryAcquire(kind)) { write 429 + Retry-After; return; }
+    try { ...handle... } finally { TokenApiConcurrency.release(kind); }
+}
+```
+- `tryAcquire` is **non-blocking** (immediate) — never hold a Tomcat thread waiting (waiting threads are
+  themselves a way to starve the pool). Acquire as late as possible but before the expensive
+  OpenSearch/DB work (`SearchApi`: before `queryPit`; `MediaResolveApi`: before the resolve loop;
+  `SimilarityApi`: at entry, before the kNN). Release in `finally` on every path (success, 4xx, 5xx,
+  exception) so permits never leak.
+- Wiring: `SearchApi` + `MediaResolveApi` → `Kind.GENERAL`; `SimilarityApi` → `Kind.SIMILARITY`.
+
+### 429 response
+`HTTP 429`, header `Retry-After: <seconds>`, body `{"error":"token API busy, retry after N seconds"}`.
+No `_source`/data. Distinct from `403`/`401`/`400`.
+
+### Metrics (ties into the re-ID metrics work)
+On the Prometheus `defaultRegistry`:
+- `wildbook_token_api_inflight{pool=}` — gauge of permits currently held (inc on acquire, dec on
+  release).
+- `wildbook_token_api_rejected_total{pool=}` — counter of `429`s.
+This makes saturation observable (ops can see when agents are hitting the cap and tune the budget).
+
+## Security / correctness
+- Permits released in `finally` (no leak on exceptions/early returns).
+- Per-JVM semaphore: in a fleet, each instance self-protects (load-balancer spreads load; per-instance
+  caps still bound each node's contribution to the shared OpenSearch). A distributed limiter is overkill.
+- Fail-open vs fail-closed on guard error: the guard itself must never 500 a request; if anything in the
+  guard throws, proceed (fail-open) — the guard is a safety valve, not an auth gate.
+- Does not change auth/ACL; orthogonal to the per-request cost ceilings (10k window, k≤100, etc.) which
+  remain and bound each permit's cost.
+
+## Out of scope (documented)
+- Distributed/cluster-wide limiting; a dedicated OpenSearch read replica / lower-priority search pool
+  (the stronger infra isolation) — recommended as a platform follow-up, not in this PR.
+- Per-token / per-IP fairness (one noisy token vs many) — v1 is a global per-instance cap; per-token
+  fairness can come later if needed.
+- Short-wait acquire (vs instant 429) — start instant; revisit if 429 churn is high.
+
+## Testing
+- `TokenApiConcurrency`: acquire up to N succeeds; the N+1th fails; release restores a permit; pools are
+  independent (exhausting SIMILARITY doesn't block GENERAL). Config override changes sizes; bad config →
+  defaults. Metrics inflight inc/dec and rejected increments.
+- Servlet-level: with the pool exhausted (acquire all permits in the test), a token request returns
+  `429` + `Retry-After` and does NOT call `queryPit`/resolve; a session (non-token) request is NOT
+  throttled even when the pool is exhausted; permit released after a normal request (next succeeds);
+  permit released after an exception path.
+
+## Design review — incorporated (BINDING)
+
+- **Scope honesty:** this is *bounded concurrency* for OpenSearch isolation, not absolute isolation.
+  It does not protect Tomcat threads / Postgres / body-parsing from a hostile flood (rejected token
+  requests still parse + do user lookup before the 429). Document a WAF/edge rate-limit + an OpenSearch
+  `thread_pool`/search-backpressure layer as the complementary platform follow-ups.
+- **AutoCloseable permit handle (no leak / no over-release):** `tryAcquire(kind)` returns a `Permit`
+  (`AutoCloseable`, or null if not acquired). Use `try (Permit p = …) { … }`; never call bare
+  `Semaphore.release()` (over-releases if acquire failed). Handles `MediaResolveApi`'s IOException
+  rethrow safely.
+- **Acquire placement (cover ALL OpenSearch work, skip 4xx gates):**
+  - `SearchApi`: acquire AFTER the individual-token 400 gate, BEFORE `new OpenSearch()`/`deletePit`/
+    `queryPit` (so 401/403/404/405/400/agg-400 never consume a permit, but the whole OpenSearch path is
+    covered).
+  - `MediaResolveApi`: acquire AFTER token/context/body/`currentUser`/`isAdmin`, BEFORE computing
+    `visible` (which itself calls `gatedVisibleIds` → `deletePit`+`queryPit`).
+  - `SimilarityApi`: at entry to the kNN execution.
+- **Lower defaults** (one sequential agent already caused 500s): `GENERAL = 8`, `SIMILARITY = 2`
+  (configurable). Note the budgets are **additive** (≤10 concurrent token OpenSearch ops/instance).
+  Clamp invalid config to ≥1; **restart required** to resize; these are node/webapp-wide
+  `OpenSearch.properties` settings read once via `getConfigurationValue`.
+- **Instant 429** (no wait); GENERAL pool also covers aggregation requests in v1 (their per-request cost
+  is already capped: terms-only, size≤1000).
+- **Idempotent metrics registration:** register the gauge/counter once (guard against duplicate
+  registration across redeploys/multiple webapp contexts — e.g. catch the already-registered case or use
+  a singleton holder). Update the inflight gauge only through the permit lifecycle. Low-cardinality
+  labels (`pool` only).
+
+## Related interference vector — shared PIT cache (IMPORTANT, partly addressed here)
+`OpenSearch.PIT_CACHE` is a **static `HashMap` keyed by index**, shared by token AND interactive/UI
+search. Each token request calls `deletePit(indexName)`, which deletes the cached point-in-time the UI
+may be mid-search on — so a single token call can disrupt a live search, independent of concurrency. The
+unsynchronized HashMap is also a cross-thread hazard.
+- **In this PR:** the new `SimilarityApi` uses a **request-scoped PIT** (create → use → delete its own
+  id; never touches `PIT_CACHE`), so the heaviest new endpoint adds no shared-PIT contention.
+- **Follow-up PR (recommended, possibly urgent — affects the already-built token search/`media/resolve`):**
+  move those token paths to request-scoped PITs too (and/or make `PIT_CACHE` concurrency-safe). Done as a
+  focused PR because it touches shared search infra used by the live UI. The concurrency guard only
+  *bounds* how many token requests churn the shared PIT at once; it does not stop a single one from
+  deleting a UI PIT.
+
+## Open questions (for review)
+1. Defaults: `GENERAL=16`, `SIMILARITY=4` per instance — reasonable starting headroom, or derive from
+   CPU count? (Lean: fixed configurable defaults; ops tune per instance size.)
+2. Instant `429` vs a short bounded wait (e.g. 100ms) before rejecting — which better avoids 429 churn
+   without risking thread pile-up? (Lean: instant.)
+3. Should aggregation requests share the GENERAL pool or get their own (they can be heavier than a plain
+   term search)? (Lean: GENERAL for v1; split if needed.)
+4. Per-JVM semaphore acceptable given the fleet, or is per-token/per-IP fairness needed in v1?