Skip to content

feat: expose embedding chunk provenance - #209

Draft
sanikolaev wants to merge 9 commits into
masterfrom
feat/embedding-chunk-provenance
Draft

sanikolaev wants to merge 9 commits into
masterfrom
feat/embedding-chunk-provenance

Conversation

@sanikolaev

@sanikolaev sanikolaev commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator
  • preserve UTF-8 byte spans for automatically chunked text embeddings
  • return the winning row-local vector slot from grouped KNN search
  • validate FFI ownership, row offsets, spans, and vector-count bounds
  • update HNSW to upstream commit 72bb56474fc8f7942a433ba310db44391c075dd3 from feat: return the winning vector in grouped search hnswlib#2
  • bump embeddings ABI to 10 and KNN ABI to 19 without changing the KNN storage format

truncate and mean intentionally return no source spans because neither vector represents one truthful source chunk.

Blocked by manticoresoftware/hnswlib#4

Related issue: manticoresoftware/manticoresearch#4968

CI-related changes

BUILD_TESTING=ON now enables the standalone, daemon-free KNN tests:

  • knn_graphrepair
  • knn_multivector_slot
  • knn_embedding_span

These tests run directly in MCL CI and do not require a Manticore Search checkout.

Manticore Search daemon ubertests remain available when MANTICORE_LOCATOR points to a daemon checkout. If it is not set, only the daemon ubertests are skipped; the standalone MCL tests are still configured and run. This does not restore the removed manticore_src.txt cross-repository autodiscovery.

@sanikolaev
sanikolaev marked this pull request as draft September 30, 2026 10:22
@sanikolaev
sanikolaev requested a review from glookka October 2, 2026 06:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant