This repository was archived by the owner on Oct 14, 2026. It is now read-only.
Repository navigation
Commit 3479136
feat(data-access): batched semantic index writer and suggestion lookup (LLMO-7445) (#1966)
<!-- mysticat-pr-skill -->
## 1. Abstract
Moves the semantic-index helpers to the redesigned tables: one batched
writer and two readers for opportunities and suggestions. The type
registries are removed from utils.
## 2. Reasoning
The first version of the semantic index had separate shapes per table, a
`source_type` column, and type registries in utils, so every new
producer or text kind needed a shared release. Each producer would also
have had to embed, batch, and dedupe on its own. The data-service
redesign
([#1153](adobe/mysticat-data-service#1153))
gives all three tables the same text columns and splits `source_type`
into `match_type` and `match_field_type`. This PR moves the shared
helpers to that shape and puts the embed and write logic in one place.
## 3. High-level overview of the changes
**utils**
- Removes `OPPORTUNITY_SEMANTIC_SOURCE_TYPES`,
`OPPORTUNITY_SEMANTIC_ENTITY_TYPES`, and the suggestion equivalents,
along with their `.d.ts` types. This reverts #1959.
**data-access** (`semantic-index.utils.js`)
- **The match type is never a parameter.** Each dimension has its own
functions that set `match_type` internally, so callers can't index or
search under the wrong lookup. Today that's topics; claims follow the
same pattern when that dimension lands. `SEMANTIC_MATCH_TYPES` is
internal.
- `indexSemanticTopics(pg, embeddingClient, { target, siteId, entities:
[{ entityId, entityType, fields: [{ matchFieldType, texts }] }] })`
replaces `syncOpportunitySemantic`. `target` is `opportunity` or
`suggestion`.
- It full-replaces each (entity, `matchFieldType`) group: it keeps
unchanged texts, embeds and inserts new ones, and deletes stale ones.
Fields that aren't listed (and other match types) are left alone, and
`texts: []` clears a field.
- A stored row from another embedding model or dims counts as stale: it
is re-embedded and overwritten in place on the entity's next refresh (a
model change is a hard cutover per entity; mysticat-data-service#1153
review).
- Texts are deduped across entities, looked up in the query cache first,
and the misses are embedded in batches of 256.
- It reads `semantic_query_embedding` but never writes to it.
- `embedQueries(pg, embeddingClient, texts, { timeoutMs, log })` is the
query side. It normalizes the texts, serves cached vectors, embeds the
distinct misses in one call within a time budget, and returns the cache
writes as a best-effort `cacheWrites` promise.
- It rejects empty or over-2048-character texts (once normalized) and
more than `MAX_QUERY_TEXTS` (256) texts with a `ValidationError`, and
treats a cache read failure as all misses (logged and returned as
`cacheError`).
- `lookupOpportunitiesByTopic` and the new `lookupSuggestionsByTopic`
replace `lookupOpportunitiesByVectors`. They take optional
`matchFieldTypes` / `entityTypes` filters (free-form, deduped, max 100
values).
- `lookupSuggestionsByTopic` checks `statuses` against
`Suggestion.STATUSES` and adds `opportunityStatuses`.
- No function takes the embedding generation: the writer, `embedQueries`
and both lookups read `SEMANTIC_MATCHING_CONFIG.embedding` themselves.
- `siteId` and `entityId` must be UUIDs, and each writer field reports a
`rejected` count (texts dropped as empty or too long).
- `entityType` and `matchFieldType` are free-form, length-bounded
strings, so there are no registries left.
- New exports: `SEMANTIC_TARGETS`, `SEMANTIC_SEARCH_RPCS` (target →
RPC), `MAX_QUERY_TEXTS` and `EmbeddingUnavailableError`.
- `getQueryEmbeddings`, `upsertQueryEmbeddings` and
`touchQueryEmbeddings` are now internal, so only `embedQueries` writes
the cache that the writer and readers trust. The cache row stores the
normalized `text`.
**Release type.** This is a breaking change to the semantic API in both
packages, released as `feat` (minor). No external code consumes these
helpers yet: the writer in audit-worker is behind
`OFFSITE_SEMANTIC_INDEX_ENABLED` (off), consumers pin exact versions,
and their PRs adopting this API come next. The old helpers also stop
working once mysticat-data-service#1153 runs.
Removed exports (to list in the squash commit body):
- utils: `OPPORTUNITY_SEMANTIC_SOURCE_TYPES`,
`OPPORTUNITY_SEMANTIC_ENTITY_TYPES`, `SUGGESTION_SEMANTIC_SOURCE_TYPES`,
`SUGGESTION_SEMANTIC_ENTITY_TYPES` and their `.d.ts` types;
- data-access: `syncOpportunitySemantic`,
`lookupOpportunitiesByVectors`, `SEMANTIC_SEARCH_RPC`, the re-exported
`OPPORTUNITY_SEMANTIC_SOURCE_TYPES` /
`OPPORTUNITY_SEMANTIC_ENTITY_TYPES`, and the cache helpers
`getQueryEmbeddings` / `upsertQueryEmbeddings` / `touchQueryEmbeddings`
(no consumer imports them); the lookups no longer accept `model` /
`dims`.
## 4. Required information
- Jira / issue:
[LLMO-7445](https://jira.corp.adobe.com/browse/LLMO-7445)
- ADR(s): `docs/adr/0002-embedding-client-and-semantic-index-utils.md`
(Decision 4 amended; outdated paragraphs marked superseded)
- Other:
[mysticat-data-service#1153](adobe/mysticat-data-service#1153)
(table and RPC redesign)
## 6. Affected / used mysticat-workspace projects
- mysticat-data-service: contract. The helpers write the redesigned
tables and call the new RPC signatures (`p_match_type`,
`p_match_field_types`) from #1153.
- spacecat-audit-worker: consumed. It switches its writer to
`indexSemanticTopics` when it bumps data-access
([spacecat-audit-worker#27](https://github.com/Adobe-AEM-Sites/spacecat-audit-worker/pull/27),
draft; the writer is flag-off today).
- spacecat-api-service: consumed. The by-topics endpoint switches to
`embedQueries`, `lookupOpportunitiesByTopic` and `matchFieldTypes`, and
drops `sourceTypes`, when it bumps data-access
([spacecat-api-service#79](https://github.com/Adobe-AEM-Sites/spacecat-api-service/pull/79),
draft).
## 8. Test plan
(a) Nothing beyond unit tests yet. The helpers are first exercised end
to end when the audit-worker and api-service follow-ups run against a
local data-service with #1153 applied.
(b) dev: after #1153 is deployed and the consumer PRs ship, enable
`OFFSITE_SEMANTIC_INDEX_ENABLED` and run an offsite audit. Check that
rows land in `opportunity_semantic_embedding` with `match_type='topic'`,
then call the by-topics endpoint and confirm it returns matches.
## 9. Deployment & merge order
-
[mysticat-data-service#1153](adobe/mysticat-data-service#1153):
depends-on. Its RPC signatures must be deployed before any consumer uses
this release.
- spacecat-audit-worker / spacecat-api-service follow-ups: blocked by
this PR (they bump to its release).
Order: data-service #1153 → this PR (release) → audit-worker +
api-service bumps.
🤖 Generated with
[review-kit](https://github.com/adobe/experience-success-skills)
---------
Co-authored-by: Tanase Butcaru <tbutcaru@adobe.com>1 parent c784590 commit 3479136
12 files changed
Lines changed: 1356 additions & 996 deletions
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
142 | 142 | | |
143 | 143 | | |
144 | 144 | | |
145 | | - | |
| 145 | + | |
146 | 146 | | |
147 | 147 | | |
148 | 148 | | |
149 | | - | |
150 | | - | |
151 | | - | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
152 | 152 | | |
153 | 153 | | |
154 | 154 | | |
155 | | - | |
156 | | - | |
157 | | - | |
158 | 155 | | |
159 | | - | |
160 | | - | |
161 | | - | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
162 | 166 | | |
| 167 | + | |
| 168 | + | |
163 | 169 | | |
164 | | - | |
165 | | - | |
166 | | - | |
167 | | - | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
168 | 174 | | |
169 | 175 | | |
170 | | - | |
171 | | - | |
172 | | - | |
173 | | - | |
174 | | - | |
175 | | - | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
176 | 187 | | |
177 | 188 | | |
178 | 189 | | |
179 | | - | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
180 | 193 | | |
181 | | - | |
| 194 | + | |
182 | 195 | | |
183 | | - | |
| 196 | + | |
184 | 197 | | |
185 | 198 | | |
186 | 199 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
| 55 | + | |
55 | 56 | | |
| 57 | + | |
56 | 58 | | |
57 | | - | |
58 | 59 | | |
59 | 60 | | |
| 61 | + | |
| 62 | + | |
60 | 63 | | |
61 | | - | |
62 | | - | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
63 | 68 | | |
| 69 | + | |
64 | 70 | | |
65 | 71 | | |
66 | 72 | | |
67 | 73 | | |
68 | 74 | | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
74 | 79 | | |
75 | 80 | | |
76 | 81 | | |
| |||
0 commit comments