Skip to content

feat: add Hamming distance support for binary vectors - #3262

Open
DoubleCore wants to merge 15 commits into
antgroup:mainfrom
DoubleCore:issue-2872-hamming
Open

DoubleCore wants to merge 15 commits into
antgroup:mainfrom
DoubleCore:issue-2872-hamming

Conversation

@DoubleCore

@DoubleCore DoubleCore commented Nov 3, 2026 •

Copy link
Copy Markdown

认领登记:DoubleCore(@DoubleCore)

认领

已在云燕平台完成认领登记。本 issue 的二进制向量 Hamming 距离支持已实现并提交 PR。

已提交 PR

PR: feat: add Hamming distance support for binary vectors

  • 端到端二进制向量支持:C++ 距离核、索引、序列化、pyvsag
  • hamming(a,b)=popcount(a XOR b)

请维护者们评审,谢谢!

Fixes #2872

Signed-off-by: DoubleCore <851678303@qq.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-03T16:17:50.923062Z a81e25c PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@vsag-bot

vsag-bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Reviewer | Review complete

Automated pull request review completed.
Review decision: Approved. No actionable findings were identified in this complete static review. CI and merge requirements remain separate.

Model: gpt-6.1-sol; reasoning effort: high.
Review scope: high (2627 changed lines across 44 files).

No actionable inline findings were found.
Review coverage checks completed; this does not prove absence of defects.
Static review only; compilation and runtime tests are outside this review's scope.
File coverage: 44/44; observed passing test runs: 0.
Review: #3262 (review)

@vsag-bot vsag-bot added kind/feature Brand-new functionality or capabilities 引入全新的功能、新特性或新能力 module/datacell Data cells, vector I/O, and quantization 数据单元、向量 I/O 与量化 module/index Index algorithms and implementations 索引算法与实现 labels Nov 3, 2026
@vsag-bot

vsag-bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Jev-driven classification

Applied: area/bindings-python, module/api. Existing labels were preserved.

Option probabilities
Option Probability Result
kind/feature 100.0% Not applied
module/index 97.0% Not applied
module/api 95.0% Applied
module/datacell 95.0% Not applied
area/bindings-python 85.0% Applied
area/testing 71.0% Not applied
area/examples 67.0% Not applied
module/simd 45.0% Not applied
area/tools 15.0% Not applied
area/dependencies 13.0% Not applied
area/ci 12.0% Not applied
area/docs 10.0% Not applied
module/attr 4.0% Not applied
area/bindings-node 3.0% Not applied
area/docker 2.0% Not applied
kind/bug 0.0% Not applied
kind/cleanup 0.0% Not applied
kind/documentation 0.0% Not applied
kind/enhancement 0.0% Not applied
kind/improvement 0.0% Not applied
No kind 0.0% Not applied

Model: jev-1.13.0. Selection threshold: 85%. Probabilities are model estimates, not guarantees.

@vsag-bot

vsag-bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Merger | Merge Readiness

Head: 562f3a21 | Base: 64b7f203 | Target: main

Updated: 2026-11-06 17:04:25 UTC

Status: BLOCKED

Merger decision: HOLD (shadow assessment; no merge executed).

  • Conversations: 60 unresolved; resolution required
  • GitHub merge state: BLOCKED

Next owner: Author. Required next step: Resolve the GitHub merge conditions, then reassess.

Merge reliability: 0.00 / 1.00 (WITHHELD)

Scoring basis Points Evidence
Required CI evidence 0.27 / 0.40 2/3 required check groups satisfy the branch policy from the expected source; neutral/skipped are not execution evidence
Branch review policy 0.00 / 0.20 GitHub review decision: REVIEW_REQUIRED; review conditions are not verified
Current AI review 0.40 / 0.40 Current head/base, complete static file coverage, correct verdict and zero P0/P1 findings

Evidence subtotal: 0.67; applicable ceiling: 0.00. Final score = min(subtotal, ceiling). Policy: merge-reliability-v2.

  • Ceiling 0.95: Static analysis and check outcomes do not prove correctness; v2 has an uncalibrated 0.95 policy ceiling.
  • Ceiling 0.00: GitHub merge conditions are BLOCKED; hard gates cannot be offset by other evidence.
  • Ceiling 0.50: 60 unresolved review threads remain.
  • Ceiling 0.85: The current AI review reports 2 residual risks; see its review for context.
Merge condition State Evidence
PR state PASS OPEN
Rules PASS Active rulesets and branch protection read
Conflicts PASS MERGEABLE
Merge method PASS squash
Merge commit freshness PENDING Waiting for GitHub to refresh the merge commit for the current head and base. Only current head checks are shown.
DCO PASS success
Mergify Merge Protections PASS success
PR CI Gate PENDING Waiting for required result from the expected source
Human review PENDING GitHub: REVIEW_REQUIRED; approvals >= 1; Code Owner approval required; stale approvals dismissed
Conversations BLOCKED 60 unresolved; resolution required
GitHub merge state BLOCKED BLOCKED

AI review: COMPLETE. Static review: patch is correct; CI is reported separately.

Review findings: 0; P0/P1 findings in that review: 0; coverage: 44/44 changed files. Findings are not a count of currently unresolved threads.

CI summary: 1 neutral, 5 success. Check source: 562f3a21. Skipped/neutral checks are not evidence that tests executed.

Next action: Merge commit freshness: Waiting for GitHub to refresh the merge commit for the current head and base. Only current head checks are shown.; PR CI Gate: Waiting for required result from the expected source; Human review: GitHub: REVIEW_REQUIRED; approvals >= 1; Code Owner approval required; stale approvals dismissed; Conversations: 60 unresolved; resolution required; GitHub merge state: BLOCKED

CI check details
Check Source Result Required
ci/circleci: path-filtering/filter StatusContext success no
ci/circleci: pull-request-check StatusContext success no
DCO CheckRun (App 1861) success yes
Mergify Merge Protections CheckRun (App 10562) success yes
Mergify Merge Queue CheckRun (App 10562) neutral no
Summary CheckRun (App 10562) success no

This is an uncalibrated evidence score, not a probability, correctness guarantee or approval. Zero means the assessment is withheld or lacks positive evidence, not a proven-bad patch. Automatic merging is disabled. New commits or base changes invalidate earlier evidence; never authorize a merge from this score alone.

@mergify

mergify Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 All 3 merge protections satisfied — ready to merge.

Show 3 satisfied protections

🟢 Require kind label

  • label~=^kind/

🟢 Require version label

  • label~=^version/

🟢 Require linked issue for feature/bug PRs

  • body~=(?im)(?:^|[\s\-\*])(?:close[sd]?|fix(?:e[sd])?|resolve[sd]?)\s*:?\s+(?:#\d+|[\w.\-]+/[\w.\-]+#\d+|https?://github\.com/[\w.\-]+/[\w.\-]+/issues/\d+)

@mergify mergify Bot added area/examples C++, Python, and TypeScript examples C++、Python 与 TypeScript 示例 area/testing Tests, fixtures, and test infrastructure 测试、夹具与测试基础设施 labels Nov 3, 2026

@vsag-lgtm vsag-lgtm left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review @ a81e25c8 — 4 inline suggestion(s).

BinaryQuantizer::BinaryQuantizer(int dim, Allocator* allocator)
: Quantizer<BinaryQuantizer>(dim, allocator) {
this->code_size_ = (static_cast<uint64_t>(dim) + 7U) / 8U;
this->query_code_size_ = this->code_size_;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[critical] References undefined symbols and is not wired into the build.

MetricType::METRIC_TYPE_HAMMING, HammingDistance, and QUANTIZATION_TYPE_VALUE_BINARY are used here but are not declared anywhere in the repository. src/quantization/binary_quantizer.cpp and src/quantization/binary_quantizer_parameter.cpp are also missing from src/quantization/CMakeLists.txt, so they will not be compiled. Before this PR can build, the MetricType enum/value, the HammingDistance implementation, the binary quantization-type constant, the source file registration, and the metric-to-datacell dispatch all need to be added.

auto base = vsag::Dataset::Make();
base->NumElements(num_vectors)
->Dim(dim)
->Ids(ids.data())

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[critical] Calls a public Dataset::BinaryVectors API that does not exist.

include/vsag/dataset.h does not declare a BinaryVectors accessor, so this example cannot compile. Add the public binary-vector dataset interface or remove the example until the interface exists.

return vsag::Dataset::Make()
->NumElements(static_cast<int64_t>(ids.size()))
->Dim(dim)
->Ids(ids.data())

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[critical] Relies on a missing public binary-vector dataset API.

vsag::Dataset::Make()->...->BinaryVectors(...) is not declared in the public Dataset interface. This test will not compile unless the binary-vector API is added.

Comment thread run_git.py Outdated
['git', 'status'],
capture_output=True,
text=True,
cwd='C:\\Users\\AORUS\\work\\oss-camp\\vsag-wt-2872'

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[warning] Accidental local debugging file with an absolute Windows path.

run_git.py, check_git.py, gitstatus.py, run_git.bat, and git_status.txt look like personal worktree helpers and should be removed from the PR.

@vsag-bot vsag-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer | Code review

Automated pull request review completed.

Model: gpt-6.1-sol; reasoning effort: high.
Review scope: high (1205 changed lines across 14 files).
Submitted 6 inline comments.
Review coverage checks completed; this does not prove absence of defects.
Static review only; compilation and runtime tests are outside this review's scope.
File coverage: 14/14; observed passing test runs: 0.

Reviewed commit a81e25c.

FlattenInterfacePtr
make_flatten_data_cell_hamming(const FlattenInterfaceParamPtr& param,
const IndexCommonParam& common_param) {
return MakeFlattenDataCellForMetric<MetricType::METRIC_TYPE_HAMMING>(param, common_param);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] [P1] Include the prerequisite Hamming API definitions

src/datacell/CMakeLists.txt automatically includes this translation unit, but MetricType defines only L2SQR, IP, and COSINE. This reference therefore breaks ordinary library builds, including builds with tests disabled. Other required companion definitions are also absent: QUANTIZATION_TYPE_VALUE_BINARY, HammingDistance, and Dataset::BinaryVectors used by the new tests. Include these supporting API and kernel changes in this PR so the added sources have valid definitions.


namespace vsag {

BinaryQuantizer::BinaryQuantizer(int dim, Allocator* allocator)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] [P1] Register both binary implementation files in the quantizer target

src/quantization/CMakeLists.txt uses an explicit QUANTIZER_SRC list, which includes neither binary_quantizer.cpp nor binary_quantizer_parameter.cpp. Consequently the library contains none of their out-of-line implementations. The new unit test is discovered automatically and references both classes, so after resolving the declaration errors it still cannot link. Add both implementation files to the quantizer target.

namespace vsag {

FlattenInterfacePtr
make_flatten_data_cell_hamming(const FlattenInterfaceParamPtr& param,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] [P1] Connect binary support to public index construction

This function has no caller: MakeFlattenDataCell still dispatches only L2, IP, and cosine, and its quantizer factory never selects BinaryQuantizer. Earlier in construction, IndexCommonParam rejects both "binary" and "hamming", while QuantizerParameter rejects "binary". The default mappings and vector-access paths also remain float-oriented. Thus the configurations introduced by the example and functional tests cannot create or use a binary index even after build errors are fixed. Wire the new type through construction, defaults, and packed-vector access.

base = make_binary_data(num_elements, dim, 11)
ids = np.arange(num_elements, dtype=np.int64)
index = pyvsag.Index("brute_force", index_params("brute_force", dim))
index.build(vectors=base, ids=ids, num_elements=num_elements, dim=dim)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] [P2] Add the packed-binary path to the Python bindings

These tests introduce a Python API contract without implementing it. In python_bindings/index_binding.cpp, parse_dense_vector_kind classifies binary as UNSUPPORTED, Build/Add validate num_elements * dim scalar elements, and searches use the byte-array length as the logical dimension. CalcDistancesById additionally force-casts queries to float32. Even after C++ index creation is enabled, valid uint8 packed inputs remain unusable. Implement binary dtype handling, byte-length validation, and logical-bit dimensions in the bindings.

Comment thread src/quantization/binary_quantizer.cpp Outdated
bool
BinaryQuantizer::DecodeOneImpl(const uint8_t* codes, float* data) {
for (uint64_t d = 0; d < this->dim_; ++d) {
data[d] = static_cast<float>((codes[d / 8U] >> (d % 8U)) & 1U);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] [P2] Preserve the packed representation when decoding

EncodeOne interprets its float pointer as packed bytes, but DecodeOne expands those bytes into dim floating-point values. Re-encoding a decoded vector therefore changes its bits; for example, on little-endian systems an eight-bit code 0x01 becomes 0x00. Existing callers rely on decode/reinsert compatibility: BruteForce::Remove(FORCE_REMOVE) decodes the last vector and reinserts that buffer into the vacated slot. Preserve packed bytes in DecodeOne/DecodeBatch, as the native INT8 quantizer does, or explicitly adapt every caller.

Comment thread tests/python/test_binary_hamming.py Outdated
used_tail_bits = dim % 8
if used_tail_bits:
mask = (1 << used_tail_bits) - 1
tail = POPCOUNT[(q[:, None, -1] ^ b[None, :, -1]) & ~mask & 0xFF].sum(axis=2)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] [P2] Remove the nonexistent axis from the tail-bit oracle

q[:, None, -1] and b[None, :, -1] broadcast to a two-dimensional (query_count, base_count) array. POPCOUNT indexing preserves that shape, so sum(axis=2) requests a nonexistent axis whenever dim is not divisible by eight. This prevents the 137-bit test and non-aligned parameter cases from checking distances even with a correct implementation. Subtract the per-pair tail popcounts directly.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a81e25c8ae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +62 to +63
"dtype": "binary",
"metric_type": "hamming",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Accept binary/Hamming parameters in the public index factory

Any Factory::CreateIndex call using this new configuration is rejected before reaching the binary implementation. Engine::CreateIndex calls IndexCommonParam::CheckAndCreate, but fill_datatype and fill_metrictype in src/index_common_param.cpp still reject "binary" and "hamming", respectively. Consequently, the example and all newly added public-API tests receive INVALID_ARGUMENT instead of an index. Add the new dtype and metric to common-parameter parsing alongside this feature.

Useful? React with 👍 / 👎.

FlattenInterfacePtr
make_flatten_data_cell_hamming(const FlattenInterfaceParamPtr& param,
const IndexCommonParam& common_param) {
return MakeFlattenDataCellForMetric<MetricType::METRIC_TYPE_HAMMING>(param, common_param);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Register the Hamming backend in the existing runtime factories

This new helper has no callers in a repository-wide reference search. MakeFlattenDataCell in src/datacell/flatten_datacell_factory.cpp still dispatches only L2/IP/cosine and returns nullptr otherwise; MakeFlattenDataCellInstance in flatten_datacell_factory_impl.h has no binary-quantizer branch, and QuantizerParameter::GetQuantizerParameterByJson still rejects "binary". Therefore, even after common-parameter parsing accepts the new configuration, neither public index can instantiate BinaryQuantizer. Register the new backend in these existing dispatch paths rather than leaving it accessible only through direct construction in unit tests.

Useful? React with 👍 / 👎.

base = make_binary_data(num_elements, dim, 11)
ids = np.arange(num_elements, dtype=np.int64)
index = pyvsag.Index("brute_force", index_params("brute_force", dim))
index.build(vectors=base, ids=ids, num_elements=num_elements, dim=dim)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Add packed-binary handling to the Python bindings

The packed np.uint8 inputs introduced here have no corresponding binding implementation. In python_bindings/index_binding.cpp, parse_dense_vector_kind maps "binary" to UNSUPPORTED, and build/add/search reject that kind; their buffer validation also expects dim elements per vector rather than ceil(dim / 8) bytes. Thus fixing C++ index construction alone still leaves every new Python success case unusable. Add a binary binding path that validates packed-byte shapes and passes the bytes through without floating-point conversion, including the distance-by-ID operation.

Useful? React with 👍 / 👎.

Comment thread tests/python/test_binary_hamming.py Outdated
used_tail_bits = dim % 8
if used_tail_bits:
mask = (1 << used_tail_bits) - 1
tail = POPCOUNT[(q[:, None, -1] ^ b[None, :, -1]) & ~mask & 0xFF].sum(axis=2)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Remove the nonexistent axis from the tail-bit reference count

For non-byte-aligned dimensions such as 137 or 9, indexing the final byte makes q[:, None, -1] ^ b[None, :, -1] a two-dimensional (num_queries, num_base_vectors) array. The popcount lookup preserves that shape, so .sum(axis=2) addresses a nonexistent axis and raises AxisError if evaluation reaches the reduction. This prevents the non-aligned-dimension tests from calculating their reference distances even with a working index. Subtract the per-pair tail popcount directly instead of reducing a third dimension.

Useful? React with 👍 / 👎.

@vsag-bot
vsag-bot self-requested a review November 4, 2026 17:12
@DoubleCore

Copy link
Copy Markdown
Author

本轮评审已处理,新提交 ec539511(上一条 head a81e25c8 已过期)。

已修

  1. [P1] 未把 binary 实现文件注册进 quantizer target — src/quantization/CMakeLists.txt 加入两个 binary 实现文件。
  2. [P1] binary 能力没有接到公开索引构建路径 — MakeFlattenDataCellFactory / flatten_datacell_factory_impl 增加 binary 分支并有调用方;inner_index_interface、bruteforce、hgraph_param_mapping、index_common_param 同步接线。
  3. [P1] Python 侧缺少 packed binary 路径 — index_binding.cpp 新增 DenseVectorKind::BINARY:按 (dim+7)/8 校验 packed 字节数、校验 dim 与索引维度一致、dlpack 送入 Dataset::BinaryVectors;build/search/query 全部走同一套校验。
  4. [P2] 解码时丢失 packed 表示 — binary_quantizer.cpp / bit_simd.h 保留打包形态,仅在需要时按位展开。
  5. [P2] 测试的 tail-bit oracle 用了不存在的轴 — q[:, None, -1] / b[None, :, -1] 在 packed 数组上是非法索引,该表达式不产生逐元素校验;已改为直接对尾部字节与掩码做 POPCOUNT,不再带错误的 .sum(axis=2)。
  6. [warning] 误提交的本地调试脚本 — 删除 run_git.py、run_git.bat、check_git.py、gitstatus.py、git_status.txt。

未处理(如实说明)

  • C++ 全量构建与单测未在本机执行:VSAG 的 CMake 依赖 roaringbitmap FetchContent,在本机(Windows)下载会挂起,因此本轮只做符号级自洽核对(BinaryVectors/GetBinaryVectors/DATATYPE_BINARY/PARAMETER_DIM 的声明-定义-调用点全部核对;validate_dense_array 新增 expected_dim 形参在全部 5 个调用点均已传入)。CI 会给出权威结论。

@vsag-bot vsag-bot added area/bindings-python Python package and bindings Python 包与绑定 module/api Public C++ API and headers 公共 C++ API 与头文件 labels Nov 4, 2026
@mergify mergify Bot added the module/simd SIMD dispatch and optimized kernels SIMD 分派与优化内核 label Nov 4, 2026

@vsag-bot vsag-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer | Code review

Automated pull request review completed.
Review decision: Comment only. No approval was submitted.

Model: gpt-6.1-sol; reasoning effort: high.
Review scope: high (1454 changed lines across 27 files).
Submitted 8 inline comments.
Review coverage checks completed; this does not prove absence of defects.
Static review only; compilation and runtime tests are outside this review's scope.
File coverage: 27/27; observed passing test runs: 0.

Reviewed commit ec53951.

if (common_param.data_type_ == DataTypes::DATA_TYPE_BINARY) {
if (not external_param.Contains(HGRAPH_BASE_QUANTIZATION_TYPE)) {
inner_json[BASE_CODES_KEY][QUANTIZATION_PARAMS_KEY][TYPE_KEY].SetString(
QUANTIZATION_TYPE_VALUE_BINARY);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] [P1] Bypass float-only training sampling for binary HGraph

A binary Build or initial Add exceeding 65,536 rows with default parameters reaches prepare_add_context -> sample_train_dataset -> sample_train_data. That sampler reads GetFloat32Vectors(), which is null for a binary-only dataset, and copies from it, causing undefined behavior before BinaryQuantizer::TrainImpl is reached. A 600-row batch with train_sample_count=512 also triggers this. Skip sampling for this training-free quantizer or make the sampler preserve packed binary storage.


bool
BinaryQuantizer::DecodeOneImpl(const uint8_t* codes, float* data) {
std::memcpy(static_cast<void*>(data), codes, this->code_size_);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] [P1] Adapt public vector retrieval to packed binary decoding

Public retrieval still allocates dim floats per row and exposes Float32Vectors, while this decoder initializes only ceil(dim/8) bytes. For example, GetDataByIds on a default 64-bit binary BruteForce index returns a 256-byte float row with only 8 bytes initialized, exposing allocator contents and no BinaryVectors field. HGraph GetRawVectorByIds with store_raw_vector=true has the same mismatch. Adapt retrieval allocation, stride, and dataset field to packed binary output, or reject unsupported retrieval.

throw VsagException(ErrorType::INVALID_ARGUMENT,
"Hamming distance requires binary dtype and binary quantization");
}
return MakeFlattenDataCellInstance<BinaryQuantizer, IOTmpl>(param, common_param);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] [P2] Handle binary strides in scattered batch insertion

This instantiates FlattenDataCell for binary input, but its BatchInsertVector scattered-ID branch has no DATA_TYPE_BINARY case and retains source_stride=dim*sizeof(float). With dim=64, two packed rows and destination IDs {0,2}, the second row is read at byte 256 instead of byte 8, outside the 16-byte input. Add the packed-byte stride to the inherited batch insertion path.

HGraph::CheckAndMappingExternalParam(const JsonType& external_param,
const IndexCommonParam& common_param) {
auto inner_json = map_hgraph_param(external_param);
if (common_param.data_type_ == DataTypes::DATA_TYPE_BINARY) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] [P2] Wire binary HGraph updates through native helpers

Binary HGraph advertises SUPPORT_UPDATE_VECTOR_CONCURRENT, but every UpdateVector call reaches get_vectors in utils/util_functions.cpp, which has no binary branch and throws 'no support for this type: 5', including force_update=true. The non-forced path also uses float-only distance overloads. Route binary updates through native extraction/distance paths so a valid packed replacement can actually update the index.

} else if (datatype == DATATYPE_INT8) {
result.data_type_ = DataTypes::DATA_TYPE_INT8;
} else if (datatype == DATATYPE_BINARY) {
result.data_type_ = DataTypes::DATA_TYPE_BINARY;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] [P2] Reject binary configuration for unsupported WARP indexes

CreateIndex("warp", {"dtype":"binary","metric_type":"hamming","dim":64}) now passes common validation and WARP mapping, but make_multi_vector_data_cell_for_io has no Hamming branch and returns nullptr. BruteForce::InitFeatures then dereferences inner_codes_, crashing during index creation. Reject this unsupported combination before construction, or provide a compatible multi-vector datacell.

if (query->GetInt8Vectors() != nullptr) {
sub->Int8Vectors(query->GetInt8Vectors() + q * query->GetDim());
}
if (query->GetBinaryVectors() != nullptr) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] [P2] Exclude binary datasets from float-only distance dispatch

The earlier is_float_query predicate excludes int8, float16, sparse, and multi-vector fields but omits GetBinaryVectors(). A dataset containing both BinaryVectors and auxiliary Float32Vectors therefore bypasses this new branch and uses the float-pointer overload. BruteForce computes Hamming distance from the float buffer's bytes, while HGraph rejects the query. Include binary storage in the exclusion or dispatch according to the index datatype.

Comment thread tests/test_binary_hamming_hgraph.cpp Outdated

auto build_dataset = MakeDataset(base_data, ids, dim);

auto brute_made = vsag::Factory::CreateIndex("brute_force", params);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] [P2] Use valid brute-force parameters for the reference index

The shared params contains index_param.support_remove, which BruteForce::CheckAndMappingExternalParam does not accept. CreateIndex therefore returns INVALID_ARGUMENT and the following REQUIRE fails before any HGraph behavior is exercised. Give the reference index separate parameters without the HGraph-only support_remove key.

Comment thread tests/python/test_binary_hamming.py Outdated
dim = 64
index = pyvsag.Index("brute_force", index_params("brute_force", dim))
vectors = np.zeros((10, dim // 8), dtype=np.float32)
with pytest.raises(RuntimeError):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] [P2] Expect ValueError from binding input validation

validate_dense_array throws std::invalid_argument for the wrong dtype, shape, length, and contiguity cases used here. Pybind translates that exception to ValueError, so this assertion and the other RuntimeError rejection assertions in this file fail when validation works correctly. Expect ValueError, consistent with the existing dense-input validation tests.

@vsag-lgtm vsag-lgtm left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review @ ec539511 — 3 inline suggestion(s).

Comment thread tests/python/test_binary_hamming.py Outdated
dim = 64
index = pyvsag.Index("brute_force", index_params("brute_force", dim))
vectors = np.zeros((10, dim // 8), dtype=np.float32)
with pytest.raises(RuntimeError):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[critical] These reject tests can never pass as written: the binding raises ValueError, not RuntimeError.

validate_dense_array in python_bindings/index_binding.cpp throws std::invalid_argument for dtype/shape/size/contiguity violations, and no custom exception translator is registered in python_bindings/, so pybind11's default translation surfaces it as Python ValueError. The repo's own pre-existing test (tests/python/test_distance_by_id.py:87, merged in #3246) already asserts ValueError for the same validator's errors, which confirms the actually raised type. Consequently all five pytest.raises(RuntimeError) assertions (lines 151, 164, 179, 181, 193) fail with a raised ValueError.

Suggested change
with pytest.raises(RuntimeError):
with pytest.raises(ValueError):

The same one-line change applies at lines 164, 179, 181, and 193. Alternatively, pytest.raises((RuntimeError, ValueError)) if the intent is to stay tolerant of error-type mapping.

Comment thread src/simd/bit_simd.h
Comment on lines +58 to +64
for (uint64_t i = 0; i < num_byte; ++i) {
auto diff = static_cast<uint8_t>(x[i] ^ y[i]);
while (diff != 0) {
diff = static_cast<uint8_t>(diff & (diff - 1U));
++distance;
}
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] This is the hot kernel for every binary Hamming distance computation, but the per-byte bit-clearing loop is scalar and data-dependent (up to 8 iterations per byte even when hardware popcount is available). The codebase already relies on unguarded __builtin_popcountll in src/simd/generic.cpp:1155 and src/simd/fast_bitset.cpp:65, and there is no MSVC path in CI, so the builtin is safe to use here as well:

Suggested change
for (uint64_t i = 0; i < num_byte; ++i) {
auto diff = static_cast<uint8_t>(x[i] ^ y[i]);
while (diff != 0) {
diff = static_cast<uint8_t>(diff & (diff - 1U));
++distance;
}
}
for (uint64_t i = 0; i < num_byte; ++i) {
distance += static_cast<uint64_t>(__builtin_popcount(x[i] ^ y[i]));
}

For a bigger win, go word-wise: accumulate __builtin_popcountll over 8-byte words loaded via std::memcpy and fall back to per-byte counting for the tail. That is roughly an order of magnitude faster with hardware popcount and follows the pattern generic.cpp already established.

std::unique_lock global_lock(this->global_mutex_);
InnerIdType inner_id = this->label_table_->GetIdByLabel(id);
return this->inner_codes_->UpdateVector(new_base->GetFloat32Vectors(), inner_id);
return this->inner_codes_->UpdateVector(this->get_data(new_base, 0), inner_id);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] Now that packed binary bytes are wired through the punned path, UpdateVector and the FORCE_REMOVE decode-reinsert path work, but neither has end-to-end coverage: tests/test_binary_hamming_hgraph.cpp only exercises Remove in the default MARK_REMOVE mode, and no binary test calls UpdateVector. Since correctness of both paths depends on DecodeOneImpl copying raw bytes out and EncodeOneImpl re-masking the tail, a regression test would guard that round-trip: update a binary vector and re-search to verify results reflect the new vector, and/or force-remove an element and verify the reinserted neighbor still matches.

@vsag-bot
vsag-bot self-requested a review November 4, 2026 19:24
@vsag-bot
vsag-bot self-requested a review November 5, 2026 01:52
@DoubleCore

Copy link
Copy Markdown
Author

Addressed the packed-scratch review in 070d480.

  • Binary HGraph tunes now skip float training-data allocation and decoding when every rebuilt code uses the training-free BinaryQuantizer.
  • Binary-to-binary rebuilds size their insert scratch from GetQuantizerCodeSize(), i.e. packed bytes rather than dim_ floats. Non-binary dtype paths retain the existing float layout.
  • Added regression coverage with 8,192-bit vectors and an allocation-recording allocator, plus a post-tune distance check.

Validation: clang-format 20.1.8 --dry-run --Werror and git diff --check passed. 本地未编译,仅静态核对 (per the local environment restriction).

@vsag-bot vsag-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer | Code review

Automated pull request review completed.
Review decision: Comment only. No approval was submitted.

Model: gpt-6.1-sol; reasoning effort: high.
Review scope: high (2213 changed lines across 37 files).
Submitted 2 inline comments.
Review coverage checks completed; this does not prove absence of defects.
Static review only; compilation and runtime tests are outside this review's scope.
File coverage: 37/37; observed passing test runs: 0.

Reviewed commit 070d480.

Comment thread src/quantization/binary_quantizer.h Outdated
SerializeImpl(StreamWriter& writer) {};

void
DeserializeImpl(StreamReader& reader) {};

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] [P2] Restore the query buffer size during deserialization

Quantizer::Deserialize restores dim_ and code_size_, but this empty hook leaves query_code_size_ at its constructor value. Loading a 128-bit binary index into an otherwise compatible index constructed with dim=64 is accepted and restores dim=128; ProcessQueryImpl then allocates only 8 bytes while ComputeDistImpl reads 16 bytes from that buffer. The reverse dimension change overreads the input query during memcpy. Recompute query_code_size_ from the restored code size before processing queries.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 90c468d. BinaryQuantizer::DeserializeImpl now recomputes query_code_size_ from the code_size_ restored by Quantizer::Deserialize, so both dimension growth and shrinkage use the restored query allocation/copy size. The quantizer regression test now starts with a different constructor dimension, checks GetQueryCodeSize(), exercises SetQuery/ComputeDist, and also covers shrinking back to 64 bits. Local C++ compilation was intentionally not run; static symbol checks and targeted clang-format 20.1.8 checks passed.

auto query = vsag::Dataset::Make();
query->NumElements(1)->Dim(to_int64(static_cast<uint64_t>(buf.shape[0])))->Owner(false);
const uint64_t query_dim = dense_vector_kind_ == DenseVectorKind::BINARY
? dim_

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] [P2] Refresh the cached binary dimension after loading

Load() lets C++ restore the serialized dimension but never updates the new dim_ cache. For example, construct a binary HGraph with dim=9 and load a compatible dim=10 index: both use two packed bytes, but every query is still tagged Dim(9) here and fails the index's dimension check. Range searches, distance calls, and add validation also use the stale cache. Refresh the cached dimension after successful deserialization.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 90c468d. Load() now checks the deserialization result and, only after success, reads the existing serialized basic_info.dim through the existing Footer::Parse/Metadata APIs and refreshes the binary dim_ cache. The added Python regression loads a 10-bit HGraph into a wrapper constructed with dim=9 and covers KNN, range search, distance-by-ID, and add validation. No new public API or JSON key was introduced. The test file passed py_compile; runtime pytest was not run because no rebuilt pyvsag wheel is available and local C++ compilation is prohibited for this task.

@vsag-lgtm vsag-lgtm left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review @ 070d480f — 0 inline comments; incremental diff 3fcb3167..070d480f (1 commit, 2 files) has no blocking findings. Approval withheld: earlier review threads remain unresolved.

Signed-off-by: DoubleCore <851678303@qq.com>
Assisted-by: OpenCode:ark-code-latest
@vsag-bot
vsag-bot self-requested a review November 5, 2026 03:42
@DoubleCore

Copy link
Copy Markdown
Author

本地验证记录(90c468d333b5532731ee616399a21b1441383f2e):未运行 CMake、未编译 C++;已完成基于 main 的静态符号核对、git diff --check、clang-format 20.1.8 变更行检查,以及 python -m py_compile tests/python/test_binary_hamming.py。由于本机没有基于新源码构建的 pyvsag wheel,未运行新增 pytest;以上结果不代表 CI 已通过。

@vsag-bot vsag-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer | Code review

Automated pull request review completed.
Review decision: Comment only. No approval was submitted.

Model: gpt-6.1-sol; reasoning effort: high.
Review scope: high (2298 changed lines across 37 files).
Submitted 3 inline comments.
Review coverage checks completed; this does not prove absence of defects.
Static review only; compilation and runtime tests are outside this review's scope.
File coverage: 37/37; observed passing test runs: 0.

Reviewed commit 90c468d.

return this->inner_codes_->FactoryComputer(&query_multi_vectors[0]);
}
return this->inner_codes_->FactoryComputer(query->GetFloat32Vectors());
return this->inner_codes_->FactoryComputer(this->get_data(query, 0));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] [P2] Validate binary queries before constructing the computer

For a nonempty 65-bit binary index, a query with Dim(8) and a one-byte buffer reaches BinaryQuantizer::ProcessQueryImpl, which copies nine bytes, before SearchWithRequest calls validate_knn_args. A correctly dimensioned dataset without BinaryVectors similarly passes nullptr to memcpy. Validate the query dimension and native storage before FactoryComputer so malformed C++ queries return INVALID_ARGUMENT instead of causing invalid memory access.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9f4e047. BruteForce::make_search_computer now rejects a null query, calls validate_search_query for dimension/single-element checks, and requires GetBinaryVectors() before FactoryComputer. Added nonempty 65-bit regression cases for a Dim(8) one-byte query and a correctly dimensioned float-only query; both must return INVALID_ARGUMENT. Local C++ was not compiled; this was checked statically and with clang-format 20.1.8.

} else if (datatype == DATATYPE_INT8) {
result.data_type_ = DataTypes::DATA_TYPE_INT8;
} else if (datatype == DATATYPE_BINARY) {
result.data_type_ = DataTypes::DATA_TYPE_BINARY;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] [P2] Reject binary configurations in unsupported index backends

Accepting binary here also makes CreateIndex("sindi", {"dtype":"binary","metric_type":"hamming","dim":64}) succeed: SINDI's default configuration has no dtype/metric guard. Building it with the newly documented BinaryVectors dataset then dereferences the null GetSparseVectors() result in SINDI::add. Gate binary support in unsupported index creators so these configurations are rejected before construction/build.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9f4e047. create_registered_index now rejects DATA_TYPE_BINARY for every registered backend except brute_force and hgraph before invoking the creator, so SINDI and other unsupported backends return INVALID_ARGUMENT before construction/build. The regression covers IVF, LazyHGraph, Pyramid, SINDI, SINDI V2, SIMQ, and WARP. No JSON keys were added. Local C++ was not compiled; static and format checks only.

ids_are_contiguous = idx_vec[i] == idx_vec[0] + i;
}
if (not ids_are_contiguous) {
const auto packed_dim = static_cast<uint64_t>((this->common_param_.dim_ + 7) / 8);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] [P2] Derive scattered row strides from the restored quantizer

FlattenDataCell::Deserialize restores the quantizer dimension but leaves common_param_.dim_ at its constructor value. After restoring a 65-bit cell into one constructed for 128 bits, inserting two packed rows with idx_vec={0,2} uses a 16-byte source stride instead of nine bytes; the second insertion reads beyond the 18-byte input and stores incorrect data. Use the restored quantizer's code size, as the binary branch in the base BatchInsertVector already does.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9f4e047. BinaryFlattenDataCell scattered insertion now derives packed_dim from quantizer_->GetCodeSize(), which reflects the deserialized quantizer, instead of common_param_.dim_. Added a regression that restores a 65-bit stream into a cell constructed for 128 bits, inserts two rows with ids {0,2}, and verifies both packed rows byte-for-byte. Local C++ was not compiled; static and clang-format 20.1.8 checks only.

Signed-off-by: DoubleCore <851678303@qq.com>
Assisted-by: OpenCode:ark-code-latest
@vsag-bot
vsag-bot self-requested a review November 5, 2026 04:24
@DoubleCore

Copy link
Copy Markdown
Author

Validation for 9f4e047: per the local environment constraint, I did not run CMake or compile C++. Static symbol checks, git diff --check, and clang-format 20.1.8 checks on the changed regions passed. Python testing was attempted, but tools/dataset_profiler does not exist in this branch and tests/python/test_binary_hamming.py could not collect because pyvsag is not installed. I therefore do not claim a local Python test pass or green CI. The remote DCO check is successful for this head.

@vsag-lgtm vsag-lgtm left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated incremental review of 070d480f..9f4e0472: both commits' fixes and tests verified correct; 1 inline suggestion.


void
DeserializeImpl(StreamReader& reader) {
this->query_code_size_ = this->code_size_;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] Consider applying this same query_code_size_ restoration to the sibling quantizers that share this pattern.

This fix is correct and timing-safe: the base Quantizer::Deserialize (src/quantization/quantizer.h:170-177) reads dim_/metric_/code_size_/is_trained_ from the stream before dispatching to DeserializeImpl, and query_code_size_ is never serialized, so re-deriving it from the already-restored code_size_ is exactly right. The new cross-dimension tests (src/quantization/binary_quantizer_test.cpp:246-263) lock this in for BinaryQuantizer.

Two siblings with the same pre-fix pattern are not touched by this PR: their constructors set query_code_size_ = code_size_ (src/quantization/fp32_quantizer.cpp:28; src/quantization/int8_quantizer.cpp:40), but their DeserializeImpl bodies are empty (src/quantization/fp32_quantizer.h:70; src/quantization/int8_quantizer.h:74), so a cross-dimension Deserialize() leaves query_code_size_ at the constructor's value:

  • FP32Quantizer is the risky one: ProcessQueryImpl allocates query_code_size_ bytes and then writes code_size_/dim_ bytes (src/quantization/fp32_quantizer.cpp:148-160), so deserializing onto an instance constructed with a smaller dim produces a heap-buffer overflow — the same bug class fixed here, but an out-of-bounds write rather than a stale-sized buffer.
  • INT8Quantizer uses code_size_/dim_ directly at query time (src/quantization/int8_quantizer.cpp:143,149), so it is functionally safe, but the public GetQueryCodeSize() (src/quantization/quantizer.h:332) keeps reporting the stale value.

Since this PR already establishes "deserialized state must be self-consistent" in several places (the binding Load() dim refresh, the GetCodeSize() scattered-insert stride), adding the same one-line restoration to those two DeserializeImpls — or re-deriving it centrally in the base class — would close the remaining gaps. Fine to defer to a follow-up issue if you want to keep this PR scoped to the binary paths.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented in 5154c03. FP32Quantizer::DeserializeImpl and INT8Quantizer::DeserializeImpl now restore query_code_size_ from the already-deserialized code_size_. Their serialization tests now deserialize into instances constructed with dim / 2 and assert that GetQueryCodeSize() matches the source; the existing FP32 query-processing coverage therefore exercises the former small-allocation/large-write path. I kept the fix local to these two siblings rather than changing the base class because other quantizers can legitimately have different query and stored-code sizes.

Validation: local C++ compilation and C++ tests were not run (per task constraint). Static symbol checks, clang-format 20.1.8 --dry-run --Werror on all four changed files, and git diff --check passed. This change does not involve Python or tools/dataset_profiler, so Python tests were not run. This reply makes no claim that CI is green.

Restore FP32 and INT8 query_code_size_ from deserialized code_size_ and exercise cross-dimension restoration in their serialization tests.

Signed-off-by: DoubleCore <851678303@qq.com>
Assisted-by: OpenCode:ark-code-latest

@vsag-bot vsag-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer | Code review

Automated pull request review completed.
Review decision: Comment only. No approval was submitted.

Model: gpt-6.1-sol; reasoning effort: high.
Review scope: high (2408 changed lines across 43 files).
Submitted 1 inline comment.
Review coverage checks completed; this does not prove absence of defects.
Static review only; compilation and runtime tests are outside this review's scope.
File coverage: 43/43; observed passing test runs: 0.

Reviewed commit 5154c03.

Comment thread src/algorithm/hgraph/hgraph.cpp Outdated
const bool packed_binary_rebuild = data_type_ == DataTypes::DATA_TYPE_BINARY and
binary_tune_source and is_binary_code(new_code);
const uint64_t insert_buffer_bytes = packed_binary_rebuild
? tune_source->GetQuantizerCodeSize()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] [P1] Use the restored dimension when rebuilding binary cells

Deserialize a 65-bit HGraph into an instance constructed with dim=128, then call Tune with use_reorder=true. ExportCommonParam still contains the constructor dimension, so Tune creates a 128-bit precise quantizer, while this branch allocates only 12 scratch bytes (the loaded 9-byte code rounded to floats). InsertVector then performs a 16-byte memcpy from that buffer, reading out of bounds; subsequent precise queries also consume too many bytes. Synchronize the common parameters with the restored dim_ before constructing rebuild cells and enforce matching packed widths.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 19ee7fc. Tune now overwrites the exported common_param.dim_ with the restored dim_ before constructing any rebuild cells. Packed binary rebuilds also require both source and destination code widths to equal ceil(dim_ / 8), and the scratch buffer uses the verified destination width. Added a regression that serializes a 65-bit HGraph, restores it into an instance constructed with dim=128, enables reorder, and verifies a precise self-distance. Validation: local C++ was not compiled, per task constraints; I performed static symbol checks, clang-format 20.1.8, and git diff --check only. This does not imply CI is green.

Synchronize Tune common parameters with deserialized dimensions, validate packed source and destination widths, and cover cross-dimension restoration with precise reordering.

Signed-off-by: DoubleCore <851678303@qq.com>
Assisted-by: OpenCode:ark-code-latest
@vsag-bot
vsag-bot self-requested a review November 5, 2026 05:33

@vsag-bot vsag-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer | Code review

Automated pull request review completed.
Review decision: Comment only. No approval was submitted.

Model: gpt-6.1-sol; reasoning effort: high.
Review scope: high (2463 changed lines across 43 files).
Submitted 3 inline comments.
Review coverage checks completed; this does not prove absence of defects.
Static review only; compilation and runtime tests are outside this review's scope.
File coverage: 43/43; observed passing test runs: 0.

Reviewed commit 19ee7fc.

if (this->query_code_size_ == 0) {
return;
}
std::memcpy(computer.buf_, static_cast<const void*>(query), this->query_code_size_);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] [P2] Reject missing binary query storage before copying

On a nonempty binary HGraph, RangeSearch with NumElements(1), the correct Dim, and only Float32Vectors passes validate_range_args. get_data then returns nullptr, which reaches this memcpy through QueryComputerPool and crashes. Unlike KNN, the range path does not validate matching vector storage. Reject the missing BinaryVectors representation before preprocessing so this returns INVALID_ARGUMENT.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0d641da. HGraph::SearchWithRequest now checks get_data(query) != nullptr right after validate_range_args on the range path, so a float32-only Dataset on a nonempty binary HGraph is rejected with INVALID_ARGUMENT before preprocessing, using the same "query vector storage must match index data type" message as the KNN path. A regression case builds a dim=65 binary HGraph and asserts RangeSearch returns INVALID_ARGUMENT instead of crashing. Validation: local C++ was not compiled, per task constraints; I performed static symbol checks, clang-format 20.1.8, and git diff --check only. This does not imply CI is green.

const bool is_float_query =
query->GetFloat32Vectors() != nullptr && !is_sparse &&
query->GetMultiVectors() == nullptr && query->GetInt8Vectors() == nullptr &&
query->GetBinaryVectors() == nullptr && query->GetFloat16Vectors() == nullptr;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] [P2] Validate binary dtype before selecting the float batch path

For a binary BruteForce index, a float32-only Dataset still satisfies is_float_query. CalcDistancesById consequently bypasses the new native Dataset validation and calls BruteForce's float-pointer overload, which interprets the first ceil(dim/8) bytes of the floats as packed bits and returns bogus distances. The equivalent single-ID Dataset call correctly rejects this input. Validate the configured binary representation before taking this fast path.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0d641da. InnerIndexInterface::CalcDistancesById(const DatasetPtr&...) now validates the configured binary representation before selecting the float batch fast path: a float32-only Dataset against a binary index is rejected with INVALID_ARGUMENT with the same "query vector storage must match index data type" message as the single-ID Dataset path. The guard covers both the topk-allocated fast path and the per-row float path, and the dim=65 binary BruteForce regression asserts both topk=-1 and topk=2 calls return INVALID_ARGUMENT instead of bogus Hamming scores. Validation: local C++ was not compiled, per task constraints; I performed static symbol checks, clang-format 20.1.8, and git diff --check only. This does not imply CI is green.

Comment thread python_bindings/index_binding.cpp Outdated
CalcDistancesById(const py::array& query,
const py::array_t<int64_t, py::array::c_style | py::array::forcecast>& ids) {
auto buf_query = query.request();
validate_dense_index_kind(dense_vector_kind_, "calc_distances_by_id");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] [P3] Preserve the existing unsupported-distance exception contract

For a sparse SINDI index, this new guard throws std::invalid_argument, exposed as ValueError, before the existing C++ failure-to-RuntimeError translation. Both distance method names previously raised RuntimeError for this call, as explicitly required by tests/python/test_distance_by_id.py::test_unsupported_dense_representation_is_not_missing. Preserve that exception behavior when adding binary validation.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0d641da. The binding-level guard on calc_distances_by_id no longer throws std::invalid_argument for an UNSUPPORTED index kind; the allow_unsupported flag lets a sparse SINDI call fall through to the C++ rejection, which the binding translates to RuntimeError with its existing message ("query sparse vectors must not be null"), so both calc_distances_by_id and cal_distance_by_id preserve the exception contract required by tests/python/test_distance_by_id.py::test_unsupported_dense_representation_is_not_missing. The existing tests are untouched. Validation: local C++ was not compiled, per task constraints; I performed static symbol checks, clang-format 20.1.8, and git diff --check only. This does not imply CI is green.

@vsag-lgtm vsag-lgtm left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review @ 19ee7fc3 — 2 inline comments on the increment 9f4e0472..19ee7fc3 (1 suggestion, 1 note).

// init new_basic_code obj
auto common_param = this->basic_flatten_codes_->ExportCommonParam();
// Deserialize restores dim_ but leaves the datacell's constructor-time common parameter stale.
common_param.dim_ = this->dim_;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] Only dim_ is re-derived from the restored state; metric_ and data_type_ are restored from the payload as well and stay stale here.

deserialize_basic_info overwrites dim_ (hgraph_serialize.cpp:255), metric_ (:258) and data_type_ (:262) from the stream, while ExportCommonParam() returns the construction-time snapshot (flatten_datacell.h:251), so all three fields can disagree with the restored index. In this scope the exported param drives the datacell dispatch (MakeFlattenDataCell selects on common_param.metric_, flatten_datacell_factory.cpp:44) and the quantizer construction, whereas the branch above at line 243 tests this->data_type_. Restoring a binary/Hamming payload into an index created with a different metric or dtype would therefore build the new datacells from the constructor-time values while the rest of Tune follows the payload. Please confirm whether only dim is meant to be allowed to differ; if the other fields can differ too, re-deriving all three keeps the two views consistent.

Suggested change
common_param.dim_ = this->dim_;
common_param.dim_ = this->dim_;
common_param.metric_ = this->metric_;
common_param.data_type_ = this->data_type_;

void
DeserializeImpl(StreamReader& reader) {};
DeserializeImpl(StreamReader& reader) {
this->query_code_size_ = this->code_size_;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[note] For COSINE + hold_molds, query_code_size_ now depends on the construction path for the same logical quantizer.

The parameter constructor sets query_code_size_ to dim * sizeof(float) (fp32_quantizer.cpp:28) before hold_molds adds sizeof(float) to code_size_ (:38), while the restored path now takes the full code_size_. This is safe today: ProcessQueryImpl only writes dim floats for cosine (:157) and ComputeDistImpl reads dim (:172-178), so the extra 4 bytes are just a small over-allocation. Recording it only because GetQueryCodeSize() is now larger after deserialize than after construction; this->dim_ * sizeof(float) would match the constructor exactly. No change required if the over-allocation is intended.

Range search on a binary HGraph accepted a float32-only Dataset and
dereferenced a null vector storage during preprocessing; reject the
mismatched vector storage with INVALID_ARGUMENT like the KNN path does.
Batch CalcDistancesById on an index configured for packed binary vectors
misread a float32-only Dataset as raw bits through the native float
overload; validate the configured binary representation before selecting
the float fast path, matching the single-ID Dataset rejection.
The calc_distances_by_id binding no longer short-circuits unsupported
(sparse) index kinds with ValueError; the call falls through to C++ so
the existing RuntimeError contract is preserved.
Regression covers the dense-only range and batch distance rejections.

Signed-off-by: DoubleCore <851678303@qq.com>
Assisted-by: OpenCode:glm-5.3-flash-free
@vsag-bot
vsag-bot self-requested a review November 5, 2026 06:36

@vsag-lgtm vsag-lgtm left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[note] Automated review of increment 19ee7fc3..0d641da4: 1 suggestion, 1 note, no blocking findings.

// Range search remains single-query only (validate_range_args enforces NumElements==1).
if (not use_custom_distance) {
this->validate_range_args(query, request.radius_, request.limited_size_);
CHECK_ARGUMENT(get_data(query) != nullptr,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[note] This storage check runs after a call that already dereferences query.

validate_range_args -> validate_search_query (src/algorithm/inner_index_interface.cpp:1349-1357) evaluates query->GetDim() unconditionally, so a null DatasetPtr is dereferenced before line 723 is reached and the caller gets a crash instead of INVALID_ARGUMENT. Every sibling branch guards the null dataset first: the KNN branch at line 734 (CHECK_ARGUMENT(query != nullptr, "query dataset cannot be null")) and the custom-distance range branch at line 726. This is pre-existing behaviour, not a regression from this commit, but since this block is being hardened it is worth ordering the checks as null dataset -> validate_range_args -> storage, so all range-search callers see a deterministic INVALID_ARGUMENT rather than a segfault.

REQUIRE_FALSE(range_result.has_value());
REQUIRE(range_result.error().type == vsag::ErrorType::INVALID_ARGUMENT);

auto batch_result = brute->CalcDistancesById(dense_query, ids.data(), count, true, -1);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] The case name claims HGraph batch-distance coverage, but both batch assertions target brute_force.

CalcDistancesById(const DatasetPtr&, ...) is not overridden by HGraph (only the const float* overload is overridden, src/algorithm/hgraph/hgraph.h:137), so graph->CalcDistancesById(...) does route through the new guard at src/algorithm/inner_index_interface.cpp:535-538. However, this case only exercises the guard through the brute_force handle and through graph->RangeSearch, so the "HGraph ... Batch" half of the title is not actually verified. Adding the same two assertions on graph (identical expectations, topk = -1 and topk = 2) closes that gap, or the title can be narrowed to the paths that are covered.

Check the range-search dataset before validating its fields, and cover null queries plus both HGraph batch-distance guard paths.

Signed-off-by: DoubleCore <doublecore@users.noreply.github.com>
Assisted-by: OpenCode:ark-code-latest
@DoubleCore

Copy link
Copy Markdown
Author

Addressed both vsag-lgtm comments from 2026-11-05T07:37:12Z in de672b7.

  • The native HGraph range path now checks for a null DatasetPtr before validate_range_args, then performs the existing storage check. A regression asserts INVALID_ARGUMENT for a null query.
  • The existing binary test now exercises graph->CalcDistancesById(DatasetPtr...) with both topk=-1 and topk=2, while retaining the brute_force and RangeSearch assertions.

Validation on the committed source: clang-format 20.1.8 --dry-run --Werror passed for both changed files; git diff --check HEAD^ HEAD passed; static symbol inspection confirmed the HGraph DatasetPtr batch call routes through InnerIndexInterface's binary guard. Local C++ compilation and test execution were not run, per task constraints. This does not claim CI is green.

@vsag-bot vsag-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer | Code review

Automated pull request review completed.
Review decision: Comment only. No approval was submitted.

Model: gpt-6.1-sol; reasoning effort: high.
Review scope: high (2556 changed lines across 44 files).
Submitted 1 inline comment.
Review coverage checks completed; this does not prove absence of defects.
Static review only; compilation and runtime tests are outside this review's scope.
File coverage: 44/44; observed passing test runs: 0.

Reviewed commit de672b7.

if (not force_update) {
std::shared_lock label_lock(this->label_lookup_mutex_);
auto new_base_query = Dataset::Make();
set_dataset(data_type_, dim_, new_base_query, new_base_vec, 1);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] [P2] Validate binary replacement dimensions before constructing the query

For a 65-bit binary HGraph, UpdateVector accepts a one-row Dim(8) dataset backed by one packed byte. This call changes its declared dimension to 65, so CalcDistanceById passes validation and BinaryQuantizer::ProcessQueryImpl copies nine bytes from that one-byte buffer. Forced updates likewise reach encoding without checking the original dimension or binary pointer. Validate new_base->GetDim() and GetBinaryVectors() before either update path and return INVALID_ARGUMENT for malformed replacements.

Signed-off-by: DoubleCore <doublecore@users.noreply.github.com>
Assisted-by: OpenCode:ark-code-latest
@vsag-bot
vsag-bot self-requested a review November 5, 2026 09:45
@DoubleCore

Copy link
Copy Markdown
Author

Addressed discussion_r4478562734 in commit 562f3a2.

  • Unified dimension and null-payload guards now run before query construction or re-encoding, covering both ordinary and forced UpdateVector paths.
  • Added a 65-bit binary HGraph regression covering wrong Dim and null binary payload on both paths, plus a valid-update and zero-distance control.
  • Static review confirmed get_vectors preserves an absent binary payload as nullptr, which CHECK_ARGUMENT rejects as INVALID_ARGUMENT. Valid dense/sparse behavior is unchanged by the binary-specific regression; well-formed replacement data still follows the existing update path.
  • Validation passed: clang-format 20.1.8 --dry-run --Werror on both changed files, and git diff --check.
  • C++ was not compiled or tested locally in this environment.

@vsag-bot vsag-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer | Code review

Automated pull request review completed.
Review decision: Approved. No actionable findings were identified in this complete static review. CI and merge requirements remain separate.

Model: gpt-6.1-sol; reasoning effort: high.
Review scope: high (2627 changed lines across 44 files).
No actionable inline findings were found.
Review coverage checks completed; this does not prove absence of defects.
Static review only; compilation and runtime tests are outside this review's scope.
File coverage: 44/44; observed passing test runs: 0.

Reviewed commit 562f3a2.

@vsag-lgtm vsag-lgtm left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review @ 562f3a21 — 0 inline findings on the increment 0d641da4..562f3a21 (2 commits, 3 files); both previously raised items are addressed. Approval withheld: earlier review threads remain unresolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/bindings-python Python package and bindings Python 包与绑定 area/examples C++, Python, and TypeScript examples C++、Python 与 TypeScript 示例 area/testing Tests, fixtures, and test infrastructure 测试、夹具与测试基础设施 kind/feature Brand-new functionality or capabilities 引入全新的功能、新特性或新能力 module/api Public C++ API and headers 公共 C++ API 与头文件 module/datacell Data cells, vector I/O, and quantization 数据单元、向量 I/O 与量化 module/index Index algorithms and implementations 索引算法与实现 module/simd SIMD dispatch and optimized kernels SIMD 分派与优化内核 size/XXL 1000+ changed lines version/1.2 蚂蚁开源轻训营专属

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Hamming Distance Support for Binary Vectors

5 participants