Skip to content
This repository was archived by the owner on Oct 14, 2026. It is now read-only.

Commit 9eb4e73

Browse files
tb-adbeTanase Butcaru
andauthored
fix(data-access): document the semantic-index helpers in the README (LLMO-7445) (#1948)
## Summary - Documents the semantic-index helpers (`syncOpportunitySemantic`, `copyEntityVectors`, and the batched `lookupOpportunitiesByVectors` / `getQueryEmbeddings` / `upsertQueryEmbeddings` / `touchQueryEmbeddings`) in the data-access README, next to the URL Index section. - Docs-only `fix:` commit, intentionally scoped to force a patch release of `@adobe/spacecat-shared-data-access` (same approach as #1929). The 4.33.0 release from #1942 was tagged (`@adobe/spacecat-shared-data-access-v4.33.0`, release commit `8a21efec`) but its npm publish failed with `E422` (sigstore provenance rejected while the repo was private, run https://github.com/adobe/spacecat-shared/actions/runs/36001440888). A re-run skips data-access because the tag already exists, so the batched helpers are on `main` but not on npm. This release (4.33.1) publishes them. ## Test plan - [x] README-only change; no code changes. - [x] Integration tests ECR image version bump. Ref: LLMO-7445 --------- Co-authored-by: Tanase Butcaru <tbutcaru@adobe.com>
1 parent 9644766 commit 9eb4e73

2 files changed

Lines changed: 39 additions & 3 deletions

File tree

‎packages/spacecat-shared-data-access/README.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,40 @@ const matches = await lookupEntityIdsByUrl(postgrestClient, {
140140

141141
See the package `CLAUDE.md` ("URL Index") for the canonicalization, single-writer, and site-scoping contracts.
142142

143+
## Semantic Index
144+
145+
For the topic vector index (`opportunity_semantic_embedding`) and the global query-embedding cache (`semantic_query_embedding`), use the shared helpers so the writer and reader share one text normalization and one storage path. They store and read pre-embedded vectors; embedding is the caller's job (e.g. `AzureEmbeddingClient` from `@adobe/spacecat-shared-gpt-client`).
146+
147+
```js
148+
import {
149+
syncOpportunitySemantic, copyEntityVectors, lookupOpportunitiesByVectors,
150+
getQueryEmbeddings, upsertQueryEmbeddings, touchQueryEmbeddings,
151+
} from '@adobe/spacecat-shared-data-access';
152+
153+
const { postgrestClient } = dataAccess.services;
154+
const scope = { model: 'azure/text-embedding-3-small', dims: 1536 };
155+
156+
// writer (needs the postgrest_writer role): full-replace an opportunity's topic vectors
157+
await syncOpportunitySemantic(postgrestClient, {
158+
siteId, entityId, entityType, sourceType: 'topic', sources: [{ text, vector, ...scope }],
159+
});
160+
161+
// writer: copy an opportunity's vector rows to another opportunity in the same site
162+
await copyEntityVectors(postgrestClient, { siteId, fromEntityId, toEntityId });
163+
164+
// query cache, batched: one entry per text, in input order (null on a miss)
165+
const cached = await getQueryEmbeddings(postgrestClient, { texts, ...scope });
166+
await upsertQueryEmbeddings(postgrestClient, { entries: [{ text, vector }], ...scope });
167+
await touchQueryEmbeddings(postgrestClient, { textHashes, ...scope });
168+
169+
// reader, batched: one best-first list of { entityId, entityType, score } per query vector
170+
const matches = await lookupOpportunitiesByVectors(postgrestClient, {
171+
siteId, sourceType: 'topic', vectors, k: 10, minScore: 0.5, ...scope,
172+
});
173+
```
174+
175+
The batched helpers group their calls internally (search in groups of up to 20 vectors, cache reads/touches in groups of 50 hashes), so callers pass whole lists rather than looping per item.
176+
143177
## Field Mapping Behavior
144178
145179
Public model API remains camelCase while Postgres/PostgREST tables are snake_case.

‎packages/spacecat-shared-data-access/test/it/postgrest/docker-compose.yml‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,8 @@ name: spacecat-data-access-it
22

33
services:
44
db:
5-
image: postgres:16-alpine
5+
# Needs the `vector` extension (data-service >= v5.134.0); same image the data-service compose uses.
6+
image: 682033462621.dkr.ecr.us-east-1.amazonaws.com/mysticat-pgvector:pg16
67
environment:
78
POSTGRES_USER: postgres
89
POSTGRES_PASSWORD: postgres
@@ -36,16 +37,17 @@ services:
3637
# v5.80.0 had aged out. Bumped straight to the current tip rather than
3738
# the oldest still-available tag (v5.104.0) since both are far past v5.80.0 and
3839
# pinning to the tip buys the most runway before the next forced bump.
40+
# v5.137.0 (2026-09-25): v5.115.0 aged out of ECR; also carries the semantic-index schema.
3941
#
4042
# To obtain or refresh the digest:
4143
# aws ecr describe-images \
4244
# --profile <your-profile> --region us-east-1 \
4345
# --repository-name mysticat-data-service \
44-
# --image-ids imageTag=v5.115.0 \
46+
# --image-ids imageTag=v5.137.0 \
4547
# --query 'imageDetails[0].imageDigest' --output text
4648
# Then export MYSTICAT_DATA_SERVICE_DIGEST=sha256:<the-digest> before npm run test:it,
4749
# or set it in the .env file consumed by docker compose. Leave unset for local dev.
48-
image: ${MYSTICAT_DATA_SERVICE_REPOSITORY:-682033462621.dkr.ecr.us-east-1.amazonaws.com/mysticat-data-service}:${MYSTICAT_DATA_SERVICE_TAG:-v5.115.0}${MYSTICAT_DATA_SERVICE_DIGEST:+@${MYSTICAT_DATA_SERVICE_DIGEST}}
50+
image: ${MYSTICAT_DATA_SERVICE_REPOSITORY:-682033462621.dkr.ecr.us-east-1.amazonaws.com/mysticat-data-service}:${MYSTICAT_DATA_SERVICE_TAG:-v5.137.0}${MYSTICAT_DATA_SERVICE_DIGEST:+@${MYSTICAT_DATA_SERVICE_DIGEST}}
4951
depends_on:
5052
db:
5153
condition: service_healthy

0 commit comments

Comments
 (0)