Skip to content

feat: expose KNN chunk provenance - #4966

Draft
sanikolaev wants to merge 20 commits into
mainfrom
feat/knn-chunk-provenance
Draft

sanikolaev wants to merge 20 commits into
mainfrom
feat/knn-chunk-provenance

Conversation

@sanikolaev

@sanikolaev sanikolaev commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator
  • persist UTF-8 byte spans for automatically generated embedding chunks
  • return winning row-local vector slot through ANN, fullscan, exact rescore, hybrid, and distributed search
  • expose knn_chunk_index(), knn_chunk_start(), and knn_chunk_end() in SQL and _knn_chunk over HTTP
  • preserve provenance through RAM/disk, flush, merge, optimize, and restart
  • return -1 when source provenance is unavailable, including manual FLOAT_VECTOR_ARRAY values
  • keep .spknn storage unchanged; provenance belongs to primary row/blob storage

Related issue: #4968

Related PRs:

  1. HNSW PR: feat: return the winning vector in grouped search hnswlib#4
    • head: a1166683c062593925ea85b7538e60d13e1ef040
  2. Columnar PR: feat: expose embedding chunk provenance columnar#209
    • provisional head: de27c015d11e768063a24c5930b593ed1b042c1b
  3. This PR pins that exact Columnar SHA through the mcl gitlink.

Both downstream pins will be replaced with canonical upstream merge SHAs after explicit merge approval.

@sanikolaev
sanikolaev requested a review from glookka October 2, 2026 07:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant