feat(hgraph): backport parallel PiPNN graph construction and adaptive alpha pruning to 1.0

[LightWant]

Summary

Backports the PiPNN feature from main onto the 1.0 release branch, following the established backport pattern on this branch (e.g. #3079, #3076).

Two upstream commits are ported:

Upstream	PR	Commit	Content
1	#2858	65e3ba36	Parallel PiPNN graph construction (opt-in graph_type: "pipnn")
2	#3049	dee3bd71	Opt-in adaptive alpha pruning for PiPNN final reservoir pruning

Scope: HGraph only

Upstream PiPNN supports both HGraph and Pyramid. This backport intentionally covers HGraph only.

On 1.0 the Pyramid PiPNN path depends on infrastructure that does not exist on this branch — PyramidPathStore (multi-path storage), PyramidBuildCache, IndexNode routing graphs / GraphBuildFunc, source-ID persistence, and optimized_build_codes_. Porting Pyramid PiPNN would mean backporting roughly six additional, later feature commits first. The docs state the HGraph-only boundary explicitly in both English and Chinese.

What is included

• src/impl/pipnn/ — the PiPNN batch graph builder plus the adaptive-pruning policy and their unit tests.
• HGraph wiring: GRAPH_TYPE_PIPNN, build_by_pipnn dispatch, the pipnn_* build parameters, parameter mapping/validation/JSON round-trip, and the Add() hand-off that routes the first batch through Build().
• Searcher fix: the entry point's own duplicate labels are now expanded. The entry point is marked visited before its neighbours are expanded, so its duplicates were previously unreachable from the traversal loop.
• alpha on HGraph; JsonWrapper::IsNumber() (main parity); src/utils/float_utils.h so the non-finite checks survive -Ofast, where std::isfinite folds away.
• Bilingual docs, the tools/eval/pipnn_parallel.yaml eval config, and build/monitor output tweaks.

1.0 adaptations (differences from upstream)

• Reuses the existing external_mapping parameter table instead of upstream's later build_default_hgraph_inner_param refactor.
• Keeps 1.0's cache/MCI/Training control flow; drops upstream-only try_optimized_build(), deduplicate_storage, and the fused-RaBitQ publish hooks, none of which exist here.
• Writes the contiguous batch-insert run via 1.0's flat io_->Write instead of upstream's layout-aware WriteRange.
• Uses 1.0's single non-template is_result_distance_eligible; drops upstream's use_custom_distance conjunct.
• Keeps only the PiPNN compatibility checks in CheckCompatibility (drops the upstream-only rabitq_fused_datacell field and the HGraph conjugate-graph guard).

Tests that depend on engine support absent from 1.0 are dropped rather than stubbed — specifically the fused RaBitQ datacell tests, deduplicated-storage tests, and query-computer accounting tests.

tests/test_hgraph_rabitq_split.cpp is restored to 1.0's three tests with only the PiPNN case appended. main had grown that file to 22 cases of unrelated drift (fused RaBitQ, MRLE, rerank statistics) that cannot run on 1.0; carrying them over produced four real failures locally, so they are excluded.

Testing

Built and tested on the 1.0 baseline (6c4d297d) with GCC 11.4, Release, -Ofast:

• Library and both test binaries build clean (libvsag_static, unittests, functests).
• PiPNN unit tests: 19 cases / 33469 assertions — all pass.
• HGraph unit tests: 32 cases / 1663 assertions — all pass.
• [rabitq_split][hgraph]: 4/4 pass, including the PiPNN case over the split RaBitQ datacell.

Regression baseline — the two failures below are pre-existing on an unmodified 1.0 tree and are not caused by this backport. I verified this by building origin/1.0 at 6c4d297d in a separate worktree and running it:

Suite	Unmodified 1.0	This branch
Unit tests	610 cases, 2 failed	640 cases, 2 failed
Failure 1	basic_searcher_test.cpp — non-finite-distance bridge	identical
Failure 2	parallel_searcher_test.cpp — non-finite-distance bridge	identical
Functional tests (partial, 77 tests each)	1 failed	1 failed, same test

Both pre-existing failures come from main commit 1527d13c (threshold filtering), which added those tests but was never backported to 1.0 — the tests are present on 1.0 without the accompanying implementation fix.

[LHT129]

PR Review Summary

This PR backports PiPNN parallel graph construction and adaptive alpha pruning from main to the 1.0 branch (4620 additions, 109 deletions across 45 files).

What Changed

• PiPNN graph builder (src/impl/pipnn/): New batch graph construction pipeline using direction-hash partitioning, BLAS-based distance computation (Sgemm/Ssyrk), and reservoir-based candidate management with adaptive pruning. This enables significantly faster index construction at high thread counts compared to ODescent.
• HGraph integration (src/algorithm/hgraph/): Added build_by_pipnn and build_by_batch_graph entry points, parameter parsing/validation, and batch insert optimization for contiguous IDs.
• ODescent parallelization (src/impl/odescent/): Parallelized add_reverse_edges and count_in_edges using SafeThreadPool.
• Infrastructure: Added Ssyrk BLAS wrapper, ParallelBatchInsertVector helper, batch scalar quantization encoding, IsFiniteFloatBits utility, and IsNumber() JSON method.
• Eval tooling: Rewrote MemoryPeakMonitor with background sampling thread for accurate peak RSS tracking; added CPU time tracking to DurationMonitor.
• Tests: New hgraph_pipnn_test.cpp (448 lines), pipnn_graph_builder_test.cpp (534 lines), pipnn_adaptive_pruning_test.cpp (280 lines), plus extensive additions to test_hgraph.cpp and test_hgraph_rabitq_split.cpp.
• Docs: PiPNN parameter documentation in English and Chinese, including the new pipnn_parallel.yaml benchmark config.

Overall Assessment

The code quality is high. The PiPNN implementation is well-structured with clear separation between the pipeline, partition strategy, and pruning policy. Thread safety is handled carefully with SafeThreadPool, per-point locking, and proper atomic/barrier usage. The test coverage is thorough across unit, integration, and end-to-end levels.

Issues Found (see inline comments for details)

1. std::isfinite under -Ofast (3 locations in pipnn_graph_builder.cpp): sanitize_distance, robust_prune/reservoir insertion, and prepare_sketches cosine normalization all use std::isfinite, which is unreliable when compiled with -ffast-math. Should use IsFiniteFloatBits from float_utils.h instead.

2. Implicit coupling in build_by_batch_graph: The all_rows_are_valid guard and batch insertion lambdas are implicitly coupled — a comment or assertion would help future readers.

3. [=] capture in ParallelBatchInsertVector: The default-by-value capture copies the FlattenInterfacePtr shared_ptr for every batch. Explicit captures would be clearer.

4. MemoryPeakMonitor thread lifecycle: The Start()/Stop() pair is not exception-safe by itself; callers must ensure Stop() runs on all paths. Acceptable for eval tooling but worth noting.

5. Parallel add_reverse_edges thread safety: Relies on graph implementation internal locking — a brief comment would clarify the contract.

Recommendation

The issues are minor and none are blocking. The std::isfinite → IsFiniteFloatBits change is the most impactful fix, as it affects correctness under common -Ofast builds. Everything else is code clarity / maintainability. This PR is ready to merge after addressing the std::isfinite issue.

[LHT129]

PR Review Summary

This is a comprehensive backport of parallel PiPNN graph construction and adaptive alpha pruning from main to the 1.0 branch. The PR has been thoroughly reviewed (43 inline comments, all resolved) and is in good shape.

What this PR does

• Adds PiPNN as a new graph construction algorithm for HGraph indexes (graph_type: "pipnn")
• Backports adaptive alpha pruning (PR #2933) for PiPNN final reservoir pruning
• Adds parallel batch insertion utilities for flatten datacells
• Fixes entry point duplicate label expansion in both basic and parallel searchers
• Adds BLAS Ssyrk wrapper for symmetric rank-k operations
• Adds IsFiniteFloatBits utility for NaN/Inf detection under -Ofast

Review dimensions

Correctness: All -Ofast hazards have been addressed — IsFiniteFloatBits is used consistently throughout the PiPNN pipeline. The searcher fix for entry point duplicate label expansion is correct and symmetric between BasicSearcher and ParallelSearcher. The entry_point_id_ fallback to inner_ids.front() is properly guarded.

Style: Consistent with the codebase conventions — 4-space indent, vsag namespace, .cpp suffix, uint64_t over size_t. New files follow the Apache 2.0 license header pattern.

Tests: Comprehensive test coverage including:

• Unit tests for the PiPNN graph builder (determinism, invariants, duplicate handling, submission failure recovery)
• Unit tests for adaptive pruning (relaxation, tightening, zero-step equivalence)
• Integration tests for HGraph PiPNN (all metrics, RaBitQ+SQ8, serialization, extra info, attribute filtering)
• Existing HGraph tests parametrized for PiPNN via HGRAPH_PR_DAILY_CASE_WITH_PIPNN

Performance: The parallel batch insertion avoids per-vector EncodeOne + SetVector overhead. The route graph ODescent construction copies parameters per iteration (faithful backport, author confirmed this is intentional for diff-comparability with main).

Security: No new security concerns. Input validation is thorough (parameter bounds, metric compatibility, data type checks).

Maintainability: Clear backport comments throughout explain where 1.0 diverges from main (no fused RaBitQ datacell, no validate_add_data(), no deduplicate_storage). The build_by_batch_graph refactoring cleanly shares orchestration between ODescent and PiPNN paths.

Verdict

The PR is well-structured, thoroughly tested, and all previously raised issues have been addressed. Ready to merge.

[vsag-bot]

Jev-driven classification

Applied: kind/feature. Existing labels were preserved.

Option probabilities

Option	Probability	Result
kind/feature	98.0%	Applied
module/index	98.0%	Not applied
module/datacell	90.0%	Not applied
module/api	84.0%	Not applied
area/testing	73.0%	Not applied
area/docs	67.0%	Not applied
area/ci	22.0%	Not applied
area/dependencies	22.0%	Not applied
area/tools	22.0%	Not applied
module/simd	10.0%	Not applied
area/examples	8.0%	Not applied
module/attr	6.0%	Not applied
area/bindings-python	4.0%	Not applied
area/bindings-node	3.0%	Not applied
kind/enhancement	2.0%	Not applied
area/docker	2.0%	Not applied
kind/bug	0.0%	Not applied
kind/cleanup	0.0%	Not applied
kind/documentation	0.0%	Not applied
kind/improvement	0.0%	Not applied
kind/perf	0.0%	Not applied
No kind	0.0%	Not applied

Model: jev-1.13.0. Selection threshold: 85%. Probabilities are model estimates, not guarantees.

[LHT129]

I have completed a thorough review of this backport PR. The code quality is high, and the 30 existing review comments from the earlier review round have addressed all the substantive issues I would have raised.

Summary of what was reviewed:

• src/impl/pipnn/ — PiPNN graph builder, adaptive pruning policy, and their unit tests (1523 + 125 + 87 + 49 lines of new implementation code)
• src/algorithm/hgraph/hgraph_build.cpp — HGraph integration: PiPNN dispatch, batch insert paths, route-graph ODescent construction
• src/algorithm/hgraph/hgraph_parameter.cpp / .h — Parameter parsing, validation, JSON round-trip, compatibility checks
• src/impl/searcher/basic_searcher.cpp / parallel_searcher.cpp — Entry-point duplicate label expansion fix
• src/utils/float_utils.h — IsFiniteFloatBits for -Ofast safety
• src/datacell/flatten_datacell.h — Contiguous batch insertion fast path
• src/datacell/flatten_build_utils.h — parallel_batch_insert_vector template
• src/impl/odescent/odescent_graph_builder.cpp — Parallelized add_reverse_edges and count_in_edges
• src/impl/blas/ — Ssyrk BLAS wrapper and test
• src/quantization/scalar_quantization/ — EncodeBatch refactor
• tools/eval/monitor/ — Background-sampling memory peak monitor and CPU-aware duration monitor
• Documentation (EN/ZH), eval configs, and test files

Key observations (all already addressed in existing threads):

1. IsFiniteFloatBits correctly replaces std::isfinite throughout — critical for -Ofast builds
2. The insert_candidate hash-bucket early-return is intentional and documented
3. add_reverse_edges parallelization uses atomic fetch_add for disjoint slot allocation — thread-safe
4. MemoryPeakMonitor destructor lifetime contract is documented
5. batch_insert_flatten_codes coupling between total and inner_ids size is explicitly asserted
6. Route-graph ODescent parameter copy is a known tradeoff (small struct, few iterations)
7. entry_point_id_ fallback to inner_ids.front() is guarded by the not inner_ids.empty() check

No new blocking issues found. The backport is clean, well-documented, and the 1.0 adaptations are clearly explained in both code comments and the PR description. The test coverage is thorough (19 PiPNN unit test cases, 32 HGraph unit test cases, 4 rabitq_split cases, all passing).

[LHT129]

Code Review Summary

I have completed a thorough review of this PR which backports parallel PiPNN graph construction and adaptive alpha pruning from main to the 1.0 release branch (46 files, +4844/-109).

Overall Assessment

The code quality is high. The PiPNN implementation is well-structured with clear separation of concerns (pipeline stages, partitioning, leaf building, pruning). The parallelization patterns (atomic work-stealing, pre-sized reverse edges with disjoint slots) are correct and well-documented. The -Ofast safety issue with std::isfinite has been properly addressed via IsFiniteFloatBits.

Key Observations

1. sanitize_distance is the single choke point for distance validation — all distances round-trip through BF16 order keys, so fixing IsFiniteFloatBits here covers the reservoir insertion and pruning paths. Confirmed no remaining std::isfinite calls in pipnn_graph_builder.cpp.

2. batch_insert_flatten_codes now has a CHECK_ARGUMENT that ties batch_count to inner_ids.size(), and all callers are guarded by all_rows_are_valid. The coupling is explicit rather than implicit.

3. add_reverse_edges parallelization uses atomic fetch_add to hand each writer a disjoint slot in pre-sized rows — no per-node locking required. The safety argument is documented in a comment.

4. EncodeBatchImpl allocates the normalized vector only when metric == COSINE (guarded by if constexpr), avoiding unnecessary allocation for L2/IP metrics.

5. entry_point_id_ assignment is guarded by not inner_ids.empty().

6. Test coverage is comprehensive: unit tests for PiPNN build, continue-add, factor, remove, duplicate-vector search, serialization, and RaBitQ split datacell integration.

Prior Review Comments

LHT129 has already provided extensive review feedback. All correctness issues have been addressed by the author. The remaining open items are performance suggestions (route graph make_shared hoisting, work-stealing batch size, power-of-2 assertion) that do not affect correctness.

No new substantive issues found.

[LHT129]

[suggestion] The pipnn_param.alpha = this->alpha; assignment in FromJson is overwritten at build time: build_by_batch_graph copies pipnn_param_ into a local pipnn_parameter and reassigns pipnn_parameter.alpha = this->alpha_; before use. The FromJson assignment only matters for ToJson() serialization. Consider either removing the FromJson assignment and letting build_by_batch_graph be the sole source of truth, or adding a comment explaining the dual-assignment intent.

[LHT129]

[note] count_in_edges uses Vector<std::atomic<int>> for the parallel count accumulator, while add_reverse_edges uses Vector<std::atomic<uint32_t>>. Both are functionally correct for practical graph sizes, but the signed int could theoretically overflow (UB) if data_num_ * max_degree exceeds INT_MAX. Using uint32_t would be consistent with add_reverse_edges and avoids the theoretical signed-overflow concern.

[LHT129]

[note] In the allocate_rows phase, reverse_counts[i].exchange(0, std::memory_order_relaxed) is used to reset the counter and size the reverse adjacency row. The return value of exchange is discarded — store(0, std::memory_order_relaxed) followed by load() (or just using the value already in resize) would be semantically clearer. The current form works correctly but reads as if the old value is needed when it is not.

[LHT129]

[note] The Vector<std::atomic<uint32_t>> reverse_counts(data_num_, allocator_) constructor value-initializes all elements to 0, so the subsequent initialize_counts parallel task (lines 184-189) writes 0 to already-zero atomics. This is a minor double-initialization. If the intent is to ensure a known state before the parallel count_edges phase, a comment would clarify; otherwise the initialize_counts phase could be removed. The same pattern appears in count_in_edges (line 333).

[LHT129]

[suggestion] The initialize_counts phase in both count_in_edges() and add_reverse_edges() is redundant: std::vector<std::atomic<T>>(n, alloc) value-initializes every element to zero (atomic value-initialization performs zero-initialization per [atomics.types.operations]/2). The explicit store(0, memory_order_relaxed) loop adds a full parallel phase with no effect. Removing it saves one parallelize_task barrier per call.

In add_reverse_edges, the three remaining phases (count → allocate → fill) are still correctly ordered because each parallelize_task drains all futures before returning.

[LHT129]

[suggestion] Same redundant initialize_counts as in add_reverse_edges: Vector<std::atomic<int>>(n, alloc) already zero-initializes all elements. The explicit store(0, memory_order_relaxed) loop is unnecessary.

[LHT129]

[suggestion] CheckCompatibility only validates graph_type when adaptive_pruning is enabled. When PiPNN is used without adaptive pruning, a PiPNN-built index could pass compatibility checks against a non-PiPNN config. The graph_type check should be unconditional for any PiPNN index, since the graph structure is fundamentally different from NSW/ODescent.

Suggested fix: move CHECK_FIELD_EQ(*this, *p, graph_type) outside the if (this->pipnn_param.adaptive_pruning) block, or add a separate unconditional check for graph_type.

[LHT129]

Code Review Summary

This is a large backport (+4733/-109, 46 files) that brings PiPNN parallel graph construction and adaptive alpha pruning from main to the 1.0 release branch. Overall the implementation quality is high — the -Ofast/std::isfinite issue has been addressed with IsFiniteFloatBits, thread safety is well-considered, and the existing review comments have been resolved.

Key Findings

1. flatten_datacell.h — EncodeBatch return value ignored
In the contiguous-ID fast path of BatchInsertVector, quantizer_->EncodeBatch(...) returns bool but the return value is discarded. If encoding fails (e.g., due to invalid input data), invalid codes are silently written to IO. The fallback path (per-row InsertVector) also ignores the return value, so this is a pre-existing pattern, but worth noting since the batch path writes a larger block of data in one shot.

2. hgraph_parameter.cpp — bottom_graph_param null safety
When graph_type == GRAPH_TYPE_VALUE_PIPNN, the code calls pipnn_param.Validate(this->bottom_graph_param->max_degree_) without an explicit null check on bottom_graph_param. In practice it is always initialized by this point, but a debug assertion would make the invariant explicit.

3. hgraph_build.cpp — Route graph ODescentParameter copy per iteration
When use_pipnn is true, current_route_odescent_param is created via std::make_shared<ODescentParameter>(*route_odescent_param) on every iteration of the route-graph loop. The only per-iteration mutation is block_size (and max_degree, which is set identically each time). The allocation could be hoisted outside the loop. (Already flagged by LHT129, acknowledged.)

4. Test coverage is adequate for the backport scope
hgraph_pipnn_test.cpp covers: all supported metrics (L2/IP/Cosine), parameter parsing/serialization, adaptive pruning validation, duplicate handling, and incremental Add after Build. The existing test_hgraph.cpp diff adds PiPNN variants of existing test cases. This is reasonable coverage for a backport.

5. No security concerns — the changes are purely algorithmic and data-structure additions with no new external input parsing beyond the existing JSON parameter framework.

Verdict

Requesting changes for the EncodeBatch return value issue (item 1). The other items are suggestions that can be addressed at the author's discretion.

[LHT129]

[note] quantizer_->EncodeBatch() returns bool but the return value is discarded here. If encoding fails, invalid codes are silently written to IO via io_->Write() on the next line. The per-row fallback path (InsertVector loop below) has the same pattern, so this is consistent with existing code, but the batch path writes a larger block in one shot — a failed encode is more impactful. Consider at least logging a warning on failure, or propagating the error.

[LHT129]

Code Review Summary

This is a large backport (+4733/-109, 46 files) that brings PiPNN parallel graph construction and adaptive alpha pruning from main to the 1.0 release branch. Overall the implementation quality is high — the -Ofast/std::isfinite issue has been addressed with IsFiniteFloatBits, thread safety is well-considered, and the existing review comments have been resolved.

Key Findings

1. flatten_datacell.h — EncodeBatch return value ignored
In the contiguous-ID fast path of BatchInsertVector, quantizer_->EncodeBatch(...) returns bool but the return value is discarded. If encoding fails (e.g., due to invalid input data), invalid codes are silently written to IO. The fallback path (per-row InsertVector) also ignores the return value, so this is a pre-existing pattern, but worth noting since the batch path writes a larger block of data in one shot.

2. hgraph_parameter.cpp — bottom_graph_param null safety
When graph_type == GRAPH_TYPE_VALUE_PIPNN, the code calls pipnn_param.Validate(this->bottom_graph_param->max_degree_) without an explicit null check on bottom_graph_param. In practice it is always initialized by this point, but a debug assertion would make the invariant explicit.

3. hgraph_build.cpp — Route graph ODescentParameter copy per iteration
When use_pipnn is true, current_route_odescent_param is created via std::make_shared<ODescentParameter>(*route_odescent_param) on every iteration of the route-graph loop. The only per-iteration mutation is block_size (and max_degree, which is set identically each time). The allocation could be hoisted outside the loop. (Already flagged by LHT129, acknowledged.)

4. Test coverage is adequate for the backport scope
hgraph_pipnn_test.cpp covers: all supported metrics (L2/IP/Cosine), parameter parsing/serialization, adaptive pruning validation, duplicate handling, and incremental Add after Build. The existing test_hgraph.cpp diff adds PiPNN variants of existing test cases. This is reasonable coverage for a backport.

5. No security concerns — the changes are purely algorithmic and data-structure additions with no new external input parsing beyond the existing JSON parameter framework.

Verdict

Requesting changes for the EncodeBatch return value issue (item 1). The other items are suggestions that can be addressed at the author's discretion.

[LHT129]

Code Review Summary

This is a large backport (+4733/-109, 46 files) that brings PiPNN parallel graph construction and adaptive alpha pruning from main to the 1.0 release branch. Overall the implementation quality is high.

Key Findings

1. flatten_datacell.h — EncodeBatch return value ignored
In the contiguous-ID fast path of BatchInsertVector, quantizer_->EncodeBatch(...) returns bool but the return value is discarded. If encoding fails, invalid codes are silently written to IO. The fallback path also ignores the return value, so this is a pre-existing pattern, but the batch path writes a larger block in one shot.

2. hgraph_parameter.cpp — bottom_graph_param null safety
When graph_type == GRAPH_TYPE_VALUE_PIPNN, the code calls pipnn_param.Validate(this->bottom_graph_param->max_degree_) without an explicit null check on bottom_graph_param. A debug assertion would make the invariant explicit.

3. hgraph_build.cpp — Route graph ODescentParameter copy per iteration
When use_pipnn is true, current_route_odescent_param is created via std::make_shared<ODescentParameter>(*route_odescent_param) on every iteration. The allocation could be hoisted outside the loop. (Already flagged by LHT129.)

4. Test coverage is adequate — hgraph_pipnn_test.cpp covers all supported metrics, parameter parsing, adaptive pruning, duplicates, and incremental Add after Build.

5. No security concerns.

Verdict

Requesting changes for the EncodeBatch return value issue (item 1). The other items are suggestions.