-
Notifications
You must be signed in to change notification settings - Fork 0
Sync commits Feat/vector engine and fix critical issues #35
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 154 commits
Commits
Show all changes
158 commits
Select commit
Hold shift + click to select a range
e42124f
feat(59-01): create vector module skeleton and AlignedBuffer type
TinDang97 75e2488
feat(59-01): scalar distance kernels and DistanceTable OnceLock dispatch
TinDang97 18bf96b
docs(59-01): update .planning submodule for 59-01 completion
TinDang97 e9a299b
feat(59-02): AVX2+FMA and AVX-512 distance kernels
TinDang97 6c0131c
feat(59-02): NEON kernels and DistanceTable SIMD dispatch wiring
TinDang97 2685417
docs(59-02): update .planning submodule for 59-02 completion
TinDang97 9116dc1
bench(59-03): add Criterion benchmarks for distance kernels
TinDang97 25715d1
test(59-03): add exhaustive SIMD-vs-scalar correctness tests
TinDang97 a278596
docs(59-03): update .planning submodule ref
TinDang97 52e0e23
fix(59-03): replace deprecated criterion::black_box with std::hint::b…
TinDang97 d2f9e2e
fix(59): close verification gaps — add core newtypes, SAFETY comments
TinDang97 c9edb54
docs(phase-59): complete SIMD vector distance engine
TinDang97 1eb4309
feat(60-01): FWHT scalar+AVX2 and Lloyd-Max 4-bit codebook
TinDang97 4ef1837
feat(60-01): TurboQuant MSE encoder/decoder with nibble packing
TinDang97 121214f
docs(60-01): complete TurboQuant 4-bit encoder plan
TinDang97 9ff685f
feat(60-02): CollectionMetadata with XXHash64 checksum and TQ ADC sca…
TinDang97 2688c24
feat(60-02): wire TQ ADC into DistanceTable and init FWHT dispatch
TinDang97 0e9a2e5
docs(60-02): update .planning submodule ref
TinDang97 97b8d6b
docs(phase-60): complete TurboQuant 4-bit encoder
TinDang97 318ee6f
feat(61-01): HnswGraph struct with BFS reorder and dual prefetch
TinDang97 f64ab31
feat(61-01): HnswBuilder with insert, neighbor pruning, and BFS reorder
TinDang97 6e50d0c
docs(61-01): update .planning submodule for 61-01 completion
TinDang97 4718a1c
fix(60): remove unused imports in collection.rs and tq_adc.rs
TinDang97 7a5221a
test(61-02): add BitVec and SearchScratch with TDD tests
TinDang97 619ff25
feat(61-02): HNSW beam search with dual prefetch and TQ-ADC distance
TinDang97 18d505a
docs(61-02): update .planning submodule for 61-02 completion
TinDang97 bd9fb1c
fix(61): remove unused variables in HNSW graph and build tests
TinDang97 a45a171
docs(phase-61): complete HNSW core
TinDang97 7667b0c
feat(62-01): segment architecture with MutableSegment, ImmutableSegme…
TinDang97 016b39c
docs(62-01): update .planning submodule for segment architecture plan
TinDang97 8c48869
feat(62-02): compaction pipeline with HNSW+TQ encoding and recall ver…
TinDang97 8d42bb7
docs(62-02): update .planning submodule for compaction pipeline compl…
TinDang97 ecddba2
docs(phase-62): complete segment architecture
TinDang97 54a44f3
feat(63-01): add VectorStore, VectorIndex, IndexMeta types
TinDang97 5ed8b15
feat(63-01): add FT.CREATE/FT.DROPINDEX/FT.INFO commands and Shard in…
TinDang97 307e2c4
docs(63-01): update .planning submodule ref
TinDang97 76de2f8
feat(63-02): add FT.SEARCH handler, SQ quantization, ShardMessage::Ve…
TinDang97 9db69e4
feat(63-02): SPSC handler FT.* interception + HSET auto-indexing hook
TinDang97 15a6412
docs(63-02): update .planning submodule ref
TinDang97 27faf90
feat(63-03): add merge_search_results and scatter_vector_search coord…
TinDang97 77fbbd8
test(63-03): end-to-end integration tests for FT.CREATE + HSET + FT.S…
TinDang97 76717d4
docs(63-03): update .planning submodule for 63-03 completion
TinDang97 9b88118
fix(vector): add DEL/HDEL/UNLINK auto-delete hook for vector indexes
TinDang97 c38adf1
feat(vector): route FT.* commands from connection handlers to VectorS…
TinDang97 a682ecb
docs(phase-63): complete FT.* Redis commands + shard integration
TinDang97 f342973
feat(64-01): add FilterExpr AST and PayloadIndex with Roaring bitmaps
TinDang97 472fc2a
feat(64-01): add cost-based filter strategy selection
TinDang97 16ffbab
docs(64-01): update .planning submodule ref
TinDang97 d422f5c
test(64-02): add failing tests for filtered search functions
TinDang97 0b8822e
feat(64-02): filtered search functions with ACORN 2-hop expansion
TinDang97 e3ce43d
test(64-02): add failing tests for FILTER clause parsing and PayloadI…
TinDang97 9c96625
feat(64-02): FILTER clause parsing + PayloadIndex wiring in FT.SEARCH
TinDang97 ba5e0ee
docs(64-02): update .planning submodule ref
TinDang97 84e0d50
fix(64): populate PayloadIndex during HSET auto-indexing
TinDang97 881396d
docs(phase-64): complete filtered search engine
TinDang97 b09b784
feat(65-01): TransactionManager with begin/commit/abort/acquire_write
TinDang97 45798ca
feat(65-01): MVCC visibility check for snapshot isolation entry filte…
TinDang97 2855544
test(65-02): add failing MVCC search + VectorStore txn_manager tests
TinDang97 b5b2db8
feat(65-02): MVCC-aware mutable segment search + TransactionManager o…
TinDang97 38952ba
test(65-02): add failing SegmentHolder search_mvcc + dirty set merge …
TinDang97 5014bea
feat(65-02): thread MVCC through SegmentHolder search + dirty set mer…
TinDang97 e70ef84
docs(65-02): update .planning submodule ref
TinDang97 5849d9a
docs(phase-65): complete MVCC transaction protocol
TinDang97 18570d2
feat(66-01): VectorWalRecord enum with CRC32 framing and HnswGraph se…
TinDang97 d2eee8e
feat(66-01): immutable segment disk I/O with checksum verification an…
TinDang97 94691e1
docs(66-01): update planning submodule for plan completion
TinDang97 bcb4913
feat(66-02): WAL writer extension + vector crash recovery engine
TinDang97 dc7148b
feat(66-02): shard restore integration + VectorStore pending segments
TinDang97 3a1f4aa
docs(66-02): update planning submodule for plan completion
TinDang97 95177fa
docs(phase-66): complete vector persistence
TinDang97 f9e8918
feat(67-01): IVF data structures + FAISS-interleaved PostingList + LU…
TinDang97 99cde81
feat(67-01): VPSHUFB FastScan AVX2 kernel + scalar fallback + dispatch
TinDang97 f7a413d
docs(67-01): complete IVF segment + VPSHUFB FastScan plan
TinDang97 01bff13
feat(67-02): IVF search, k-means clustering, and build_ivf_segment
TinDang97 2756eb2
feat(67-02): SegmentHolder fan-out integration for IVF segments
TinDang97 4ab3aad
docs(67-02): update .planning submodule ref
TinDang97 4ba0b7c
docs(phase-67): complete IVF + FastScan
TinDang97 3331334
feat(68-01): add gpu-cuda feature flag and build.rs CUDA detection
TinDang97 e24516a
feat(68-01): add GPU module tree with types, context, CAGRA build, an…
TinDang97 b7b7cfa
docs(68-01): update .planning submodule ref
TinDang97 a50404b
feat(68-02): GPU-aware compaction pipeline with CPU fallback
TinDang97 87eafc3
docs(phase-68): complete GPU pipeline
TinDang97 9753bec
test(69-01): add vector engine stress tests (10K interleaved ops + co…
TinDang97 75b2170
feat(69-02): Criterion benchmarks for HNSW build/search and FWHT tran…
TinDang97 40f1a63
test(69-01): add vector engine edge case and FT.* command hardening t…
TinDang97 ffb5232
test(69-02): memory audit validating per-vector overhead budget
TinDang97 7bdc4fb
docs(phase-69): complete vector engine hardening
TinDang97 e2acb0b
chore: update Cargo.lock for vector engine dependencies (arc-swap, ro…
TinDang97 c749c20
docs: add phase 70 — production benchmark, profiling, monitoring, tuning
TinDang97 4440ee1
feat(70-01): add vector metrics module with atomic counters
TinDang97 cb34b72
feat(70-02): create vector engine benchmark script
TinDang97 5a70ab5
feat(70-02): add flamegraph profiling script and 768d Criterion bench…
TinDang97 54f80ff
feat(70-01): wire vector metrics into FT commands and INFO output
TinDang97 bbbbd0f
docs(70-02): update .planning submodule for benchmark infrastructure …
TinDang97 17b7b8b
docs(70-01): update .planning submodule for vector metrics plan
TinDang97 b02e585
perf(70-03): optimize HNSW search hot path for >10% QPS improvement
TinDang97 204e9a2
docs(phase-70): complete production benchmark and tuning
TinDang97 a400f6b
bench: add comprehensive vector engine production benchmark script
TinDang97 d2a1c48
bench: add competitive benchmark vs Redis 8.6.1 and Qdrant
TinDang97 4614773
fix(vector): critical recall bugs — BFS reorder + dimension-adaptive …
TinDang97 c781e74
fix(vector): root cause HNSW recall — BitVec bug + add f32 search path
TinDang97 55908a7
docs: add phase 71 — fix TurboQuant integration
TinDang97 d36d937
feat(71-01): wire ImmutableSegment search to f32 HNSW traversal
TinDang97 9204475
test(71-03): add QJL transform module with sign-bit random projection
TinDang97 70f8989
feat(71-03): implement TurboQuant_prod encoder and inner-product scorer
TinDang97 098e18b
docs(71-01): update .planning submodule for HNSW recall fix
TinDang97 e361d6d
docs(71-03): update .planning submodule for TurboQuant IP mode comple…
TinDang97 255db2b
feat(71-02): add TQ-ADC brute-force scan for exhaustive nearest neigh…
TinDang97 716bbb1
docs(71-02): update .planning submodule for TQ-ADC brute-force scan
TinDang97 568546b
fix(71-04): fix all 9 broken tests from plans 01-03, add f32 recall b…
TinDang97 713b637
docs(71-04): update .planning submodule for integration validation
TinDang97 dedce36
docs(phase-71): complete TurboQuant integration fix
TinDang97 85570ca
docs: add phase 72 — multi-bit TQ, NEON tuning, benchmark
TinDang97 cbf6024
feat(72-01): multi-bit codebooks (1/2/3-bit) and variable bit packing
TinDang97 4ad5474
feat(72-02): extend QuantizationConfig with TurboQuant1/2/3 variants
TinDang97 c71a9cd
feat(72-01): multi-bit ADC distance computation (1/2/3-bit)
TinDang97 25f719f
feat(72-03): fix int8 NEON dispatch, add NEON FWHT butterfly
TinDang97 3b4cc95
docs(72-01): complete multi-bit TurboQuant codebooks and ADC plan
TinDang97 b8d93b7
docs(72-02): update .planning submodule for config/persistence plan
TinDang97 462fd68
feat(72-03): convert SmallVec upper layers to CSR format
TinDang97 00beb69
docs(72-03): update .planning submodule for ARM NEON + CSR plan
TinDang97 06f9178
feat(72-04): server-mode benchmark scripts and competitive results
TinDang97 287add5
docs(72-04): complete competitive benchmark plan
TinDang97 7987f3e
docs(phase-72): complete multi-bit TQ + benchmark
TinDang97 3b9a1d2
fix(vector): prevent SPSC self-send crash in FT.* command routing
TinDang97 c7b8070
fix: prevent panics in DashTable get_mut-after-insert and SPSC self-send
TinDang97 5fa2e97
feat(vector): enable FT.* commands on --shards 1 via shared VectorStore
TinDang97 c648a3f
bench: fair TCP benchmark infrastructure (Moon insert works, search c…
TinDang97 01aec25
fix(vector): complete server-mode vector search — 3 bugs fixed
TinDang97 f45cb46
docs: update .planning submodule (debug session)
TinDang97 2b23a81
fix(vector): TurboQuant codebook fix, memory optimization, 2-stage HN…
TinDang97 f5bcbf3
feat(vector): TQ-at-insert architecture — 5.5x memory reduction
TinDang97 2a27ea3
fix(vector): restore f32 reranking — recall 0.18 → 0.99
TinDang97 4b26e7f
fix(vector): mutable uses f32 L2 brute-force, not TQ-ADC
TinDang97 2bad801
feat(vector): TurboQuant_prod scoring — unbiased L2 without f32
TinDang97 36ee475
feat(vector): M=4 multi-projection QJL — recall 83% → 90%
TinDang97 4a056eb
perf(vector): SRHT replaces dense Gaussian QJL — 25x faster insert
TinDang97 5ea8c22
fix(vector): restore dense Gaussian QJL — recall 88% → 94.5%
TinDang97 f07ada6
feat(vector): sub-centroid refinement, multi-shard FT.*, fast insert
TinDang97 08a78c1
perf(vector): auto-compact mutable → HNSW on first search
TinDang97 10f16de
perf(vector): f32 HNSW build + LUT ADC + adaptive ef — recall 87→93%,…
TinDang97 56914cc
perf(vector): sub-centroid LUT in HNSW beam — eliminate rerank, +23% QPS
TinDang97 ecb3d4c
feat(vector): configurable EF_RUNTIME and COMPACT_THRESHOLD per index
TinDang97 82de46e
docs: vector engine report + user guide
TinDang97 4abd7f7
feat(vector): Light/Exact build modes — Light default, 5× less memory…
TinDang97 6f91362
docs: vector engine report + user guide
TinDang97 03512ed
fix: resolve merge conflicts with main (pub/sub branch)
TinDang97 5c8df60
fix: CI failures + address 23 code review findings
TinDang97 5afe4f7
fix: address 10 critical code review findings
TinDang97 3383bf5
fix: remove #[inline(always)] from target_feature functions (unstable…
TinDang97 95e5edf
fix: add # Safety docs to AVX2/AVX-512 unsafe fns, relax IVF recall t…
TinDang97 5721d6a
chore: clean up dead code, unused files, and test warnings
TinDang97 ceeb39d
Merge branch 'main' into feat/vector-engine
TinDang97 7dee89f
fix: address 15 Qodo review findings — 6 bugs, 9 rule violations
TinDang97 c74e379
fix: address inline review findings — scripts, init safety, codebook API
TinDang97 777be2b
fix: clarify codebook accessor docs, add debug_assert for non-4-bit u…
TinDang97 439a986
fix: remove remaining panic!/unwrap in library code, eliminate format…
TinDang97 6054c64
fix: fail fast on invalid codebook construction instead of swallowing…
TinDang97 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
❓ Verification inconclusive
Script executed:
Repository: pilotspace/moon
Repository: pilotspace/moon
Exit code: 0
stdout:
stderr:
Vector smoke test uses incompatible command format and loses binary data.
The server's FT.SEARCH parser (lines 660–699 in
src/command/vector_search/mod.rs) expects the format:where the query string is parsed to extract a parameter name, then the vector blob is retrieved via
extract_param_blob().The test at lines 685–691 uses the incompatible positional form:
Additionally, command substitution strips null bytes from the binary payload. The bash warning confirms this:
ignored null byte in input, and the payload is reduced from 16 bytes to 2 bytes before being sent to the server.The test cannot exercise the intended code path and will fail to parse correctly.
🤖 Prompt for AI Agents