Skip to content
This repository was archived by the owner on Oct 14, 2026. It is now read-only.

Commit c784590

Browse files
tb-adbeTanase Butcaru
andauthored
refactor(data-access): remove copyEntityVectors and COPY_VECTORS_RPC (LLMO-7445) (#1964)
<!-- mysticat-pr-skill --> ## 1. Abstract Removes the `copyEntityVectors` helper and the `COPY_VECTORS_RPC` constant from data-access's semantic-index utils. ## 2. Reasoning The helper's only caller was the offsite snapshot vector copy in audit-worker, which is being removed (topic vectors are no longer copied to the `IGNORED` superseded-run snapshot). With no caller left, the helper and the `wrpc_copy_opportunity_semantic_vectors` RPC it wraps are dropped, so the upcoming semantic-index redesign doesn't have to carry them. ## 3. High-level overview of the changes - `copyEntityVectors` and `COPY_VECTORS_RPC` are removed from `semantic-index.utils.js` and from the package exports. - The README example and the ADR-0002 description are updated. - No other semantic-index helper changes behaviour. - **Compatibility:** this removes a published export. Its only known consumer, audit-worker, stops importing it in the PR linked below, so this is not marked as a breaking change. A consumer still importing it would fail at import time after upgrading. ## 4. Required information - Jira / issue: [LLMO-7445](https://jira.corp.adobe.com/browse/LLMO-7445) - ADR(s): `docs/adr/0002-embedding-client-and-semantic-index-utils.md` ## 6. Affected / used mysticat-workspace projects - spacecat-audit-worker: consumed. It must no longer import `copyEntityVectors` before it upgrades to this release (done in its PR below). - mysticat-data-service: contract. `wrpc_copy_opportunity_semantic_vectors` loses its last client wrapper; dropped in the companion migration PR. ## 8. Test plan (a) No runtime behaviour to verify beyond the removed helper. No local e2e was run for this PR. (b) After release: `grep -r copyEntityVectors` in the consuming repos (audit-worker, api-service) should return nothing once audit-worker's PR is merged. ## 9. Deployment & merge order - Depends on: https://github.com/Adobe-AEM-Sites/spacecat-audit-worker/pull/25 (removes the only import). Merge that first. - Related: the mysticat-data-service PR dropping `wrpc_copy_opportunity_semantic_vectors`. - Order: audit-worker PR (merged and deployed), then this PR, then the data-service migration. 🤖 Generated with [review-kit](https://github.com/adobe/experience-success-skills) Co-authored-by: Tanase Butcaru <tbutcaru@adobe.com>
1 parent ca6e9aa commit c784590

5 files changed

Lines changed: 3 additions & 98 deletions

File tree

‎docs/adr/0002-embedding-client-and-semantic-index-utils.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ Following ADR-0001's test (*true for every consumer → shared; specific to one
4545

4646
### 4. Semantic-index helpers live in `data-access`, mirroring the URL index
4747

48-
`semantic-index.utils.js` mirrors `url-index.utils.js`: `syncOpportunitySemantic` (full-replace per `source_type`), `lookupOpportunitiesByVectors` (ANN read RPC wrapper, many query vectors per call), the batched `semantic_query_embedding` cache helpers (`getQueryEmbeddings` / `upsertQueryEmbeddings` / `touchQueryEmbeddings`), and `copyEntityVectors` (a thin wrapper over the `wrpc_copy_opportunity_semantic_vectors` **server-side** write RPC — the copy runs as `INSERT … SELECT`, no rows materialized in the caller). Text normalization + hashing live in this layer so a value written always matches a later read/re-sync. **The shared layer stores and reads pre-embedded vectors; it never embeds** — embedding is the writer's (audit-worker's) job via the client above.
48+
`semantic-index.utils.js` mirrors `url-index.utils.js`: `syncOpportunitySemantic` (full-replace per `source_type`), `lookupOpportunitiesByVectors` (ANN read RPC wrapper, many query vectors per call), and the batched `semantic_query_embedding` cache helpers (`getQueryEmbeddings` / `upsertQueryEmbeddings` / `touchQueryEmbeddings`). (`copyEntityVectors`, a wrapper over the `wrpc_copy_opportunity_semantic_vectors` RPC, was removed with its only caller, the offsite snapshot vector copy.) Text normalization + hashing live in this layer so a value written always matches a later read/re-sync. **The shared layer stores and reads pre-embedded vectors; it never embeds** — embedding is the writer's (audit-worker's) job via the client above.
4949

5050
**Type registries.** `OPPORTUNITY_SEMANTIC_SOURCE_TYPES` (kinds of embedded text: `topic`, later `claim`) and `OPPORTUNITY_SEMANTIC_ENTITY_TYPES` (opportunity types, from `OPPORTUNITY_TYPES`) are defined in `spacecat-shared-utils` (so non-Node consumers such as the UI can import them and their types) and re-exported from this layer, and this library's writer stores and reader accepts only their values; anything else is a `ValidationError`. The reader takes `sourceTypes` (required, non-empty) and optional `entityTypes` (omitted or empty = all), deduped, so a list can never exceed its registry. The DB does not duplicate the registries: the RPC only rejects empty, multi-dimensional, or NULL-element lists and lists over 100 elements (a length bound, not a registry mirror; `NULL` entity types = all; the reader sends `NULL` when `entityTypes` is omitted or empty). A team that wants its opportunities searchable adds its source kind and/or opportunity type to the registries, indexes with them, and callers opt in by passing them.
5151

‎packages/spacecat-shared-data-access/README.md‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -146,7 +146,7 @@ For the topic vector index (`opportunity_semantic_embedding`) and the global que
146146
147147
```js
148148
import {
149-
syncOpportunitySemantic, copyEntityVectors, lookupOpportunitiesByVectors,
149+
syncOpportunitySemantic, lookupOpportunitiesByVectors,
150150
getQueryEmbeddings, upsertQueryEmbeddings, touchQueryEmbeddings,
151151
OPPORTUNITY_SEMANTIC_SOURCE_TYPES, OPPORTUNITY_SEMANTIC_ENTITY_TYPES, SEMANTIC_MATCHING_CONFIG,
152152
} from '@adobe/spacecat-shared-data-access';
@@ -161,9 +161,6 @@ await syncOpportunitySemantic(postgrestClient, {
161161
siteId, entityId, entityType: CITED_ANALYSIS, sourceType: TOPIC, sources: [{ text, vector, ...scope }],
162162
});
163163

164-
// writer: copy an opportunity's vector rows to another opportunity in the same site
165-
await copyEntityVectors(postgrestClient, { siteId, fromEntityId, toEntityId });
166-
167164
// query cache, batched: one entry per text, in input order (null on a miss)
168165
const cached = await getQueryEmbeddings(postgrestClient, { texts, ...scope });
169166
await upsertQueryEmbeddings(postgrestClient, { entries: [{ text, vector }], ...scope });

‎packages/spacecat-shared-data-access/src/service/index.js‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,6 @@ export {
5555
SEMANTIC_INDEX_TABLES,
5656
QUERY_EMBEDDING_TABLE,
5757
SEMANTIC_SEARCH_RPC,
58-
COPY_VECTORS_RPC,
5958
SEMANTIC_CHUNK_SIZE,
6059
QUERY_HASH_CHUNK_SIZE,
6160
MAX_SOURCE_TEXT_LENGTH,
@@ -68,7 +67,6 @@ export {
6867
serializeVector,
6968
parseVector,
7069
syncOpportunitySemantic,
71-
copyEntityVectors,
7270
lookupOpportunitiesByVectors,
7371
getQueryEmbeddings,
7472
upsertQueryEmbeddings,

‎packages/spacecat-shared-data-access/src/util/semantic-index.utils.js‎

Lines changed: 1 addition & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -30,11 +30,10 @@ import { DEFAULT_PAGE_SIZE, rpcError } from './postgrest.utils.js';
3030
* format: changing it desyncs stored rows from new lookups.
3131
*/
3232

33-
/** Tables the sync/copy helpers are allowed to touch. */
33+
/** Tables the sync helpers are allowed to touch. */
3434
export const SEMANTIC_INDEX_TABLES = Object.freeze(['opportunity_semantic_embedding']);
3535
export const QUERY_EMBEDDING_TABLE = 'semantic_query_embedding';
3636
export const SEMANTIC_SEARCH_RPC = 'rpc_opportunity_semantic_search';
37-
export const COPY_VECTORS_RPC = 'wrpc_copy_opportunity_semantic_vectors';
3837

3938
/** Multi-row op chunk size; smaller than the URL index's because vectors inflate the payload. */
4039
export const SEMANTIC_CHUNK_SIZE = 20;
@@ -395,37 +394,6 @@ export async function syncOpportunitySemantic(postgrestClient, {
395394
return rows.length;
396395
}
397396

398-
/**
399-
* Copy all vector rows from one opportunity to another within a site, server-side via the copy
400-
* RPC, so identical content is re-pointed rather than re-embedded. Idempotent. Requires the
401-
* `postgrest_writer` role.
402-
*
403-
* @param {object} postgrestClient - `@supabase/postgrest-js` client
404-
* @param {object} params
405-
* @param {string} params.siteId - the site both opportunities belong to
406-
* @param {string} params.fromEntityId - source opportunity id
407-
* @param {string} params.toEntityId - destination opportunity id
408-
* @returns {Promise<number>} rows inserted (0 when already copied)
409-
*/
410-
export async function copyEntityVectors(postgrestClient, {
411-
siteId, fromEntityId, toEntityId,
412-
} = {}) {
413-
assertClient(postgrestClient);
414-
assertId(siteId, 'siteId');
415-
assertId(fromEntityId, 'fromEntityId');
416-
assertId(toEntityId, 'toEntityId');
417-
418-
const { data, error } = await postgrestClient.rpc(COPY_VECTORS_RPC, {
419-
p_site_id: siteId,
420-
p_from_entity_id: fromEntityId,
421-
p_to_entity_id: toEntityId,
422-
});
423-
if (error) {
424-
throw rpcError(`Failed to copy semantic vectors for entity ${fromEntityId}`, COPY_VECTORS_RPC, { siteId, fromEntityId, toEntityId }, error);
425-
}
426-
return data ?? 0;
427-
}
428-
429397
/**
430398
* Site-scoped nearest-neighbour search for many query vectors: for each one, the opportunities
431399
* whose vectors best match it, one row per opportunity (best score). Via RPC because PostgREST

‎packages/spacecat-shared-data-access/test/unit/util/semantic-index.utils.test.js‎

Lines changed: 0 additions & 58 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,6 @@ import {
1919
SEMANTIC_INDEX_TABLES,
2020
QUERY_EMBEDDING_TABLE,
2121
SEMANTIC_SEARCH_RPC,
22-
COPY_VECTORS_RPC,
2322
OPPORTUNITY_SEMANTIC_SOURCE_TYPES,
2423
OPPORTUNITY_SEMANTIC_ENTITY_TYPES,
2524
SEMANTIC_MATCHING_CONFIG,
@@ -29,7 +28,6 @@ import {
2928
serializeVector,
3029
parseVector,
3130
syncOpportunitySemantic,
32-
copyEntityVectors,
3331
lookupOpportunitiesByVectors,
3432
getQueryEmbeddings,
3533
upsertQueryEmbeddings,
@@ -221,7 +219,6 @@ describe('semantic-index.utils', () => {
221219
expect(SEMANTIC_INDEX_TABLES).to.deep.equal(['opportunity_semantic_embedding']);
222220
expect(QUERY_EMBEDDING_TABLE).to.equal('semantic_query_embedding');
223221
expect(SEMANTIC_SEARCH_RPC).to.equal('rpc_opportunity_semantic_search');
224-
expect(COPY_VECTORS_RPC).to.equal('wrpc_copy_opportunity_semantic_vectors');
225222
});
226223
});
227224

@@ -510,61 +507,6 @@ describe('semantic-index.utils', () => {
510507
});
511508
});
512509

513-
describe('copyEntityVectors', () => {
514-
it('validates the client + ids', async () => {
515-
await expect(copyEntityVectors({}, { siteId: SITE_ID, fromEntityId: 'a', toEntityId: 'b' }))
516-
.to.be.rejectedWith(ValidationError, 'postgrestClient is required');
517-
await expect(copyEntityVectors(makeClient(), { fromEntityId: 'a', toEntityId: 'b' }))
518-
.to.be.rejectedWith(ValidationError, 'siteId is required');
519-
await expect(copyEntityVectors(makeClient(), { siteId: SITE_ID, toEntityId: 'b' }))
520-
.to.be.rejectedWith(ValidationError, 'fromEntityId is required');
521-
await expect(copyEntityVectors(makeClient(), { siteId: SITE_ID, fromEntityId: 'a' }))
522-
.to.be.rejectedWith(ValidationError, 'toEntityId is required');
523-
});
524-
525-
it('calls the copy RPC and returns the inserted count', async () => {
526-
const client = makeClient({ rpcResult: { data: 3, error: null } });
527-
const n = await copyEntityVectors(client, {
528-
siteId: SITE_ID, fromEntityId: 'a', toEntityId: 'b',
529-
});
530-
expect(n).to.equal(3);
531-
expect(client.calls.rpc).to.deep.equal([{
532-
name: 'wrpc_copy_opportunity_semantic_vectors',
533-
params: { p_site_id: SITE_ID, p_from_entity_id: 'a', p_to_entity_id: 'b' },
534-
}]);
535-
});
536-
537-
it('returns 0 when the RPC reports no inserted rows (null data)', async () => {
538-
const client = makeClient({ rpcResult: { data: null, error: null } });
539-
const n = await copyEntityVectors(client, {
540-
siteId: SITE_ID, fromEntityId: 'a', toEntityId: 'b',
541-
});
542-
expect(n).to.equal(0);
543-
});
544-
545-
it('wraps an RPC error', async () => {
546-
const client = makeClient({ rpcResult: { data: null, error: { message: 'boom' } } });
547-
const err = await copyEntityVectors(client, {
548-
siteId: SITE_ID, fromEntityId: 'a', toEntityId: 'b',
549-
}).catch((e) => e);
550-
expect(err).to.be.instanceOf(DataAccessError);
551-
expect(err.message).to.include('Failed to copy semantic vectors');
552-
expect(err.message).to.not.include('signature not found');
553-
});
554-
555-
it('adds the deploy-ordering hint when PostgREST cannot find the copy RPC', async () => {
556-
const cause = { code: 'PGRST202', message: 'Could not find the function' };
557-
const client = makeClient({ rpcResult: { data: null, error: cause } });
558-
const err = await copyEntityVectors(client, {
559-
siteId: SITE_ID, fromEntityId: 'a', toEntityId: 'b',
560-
}).catch((e) => e);
561-
expect(err).to.be.instanceOf(DataAccessError);
562-
expect(err.message).to.equal(`Failed to copy semantic vectors for entity a: ${COPY_VECTORS_RPC} ${PGRST202_HINT}`);
563-
expect(err.details).to.deep.equal({ siteId: SITE_ID, fromEntityId: 'a', toEntityId: 'b' });
564-
expect(err.cause).to.equal(cause);
565-
});
566-
});
567-
568510
describe('lookupOpportunitiesByVectors', () => {
569511
const base = {
570512
siteId: SITE_ID, sourceTypes: [SOURCE_TYPE], model: MODEL, dims: 2,

0 commit comments

Comments
 (0)