feat(hgraph): backport parallel PiPNN graph construction and adaptive alpha pruning to 1.0

README.md

@@ -39,6 +39,8 @@ last-reviewed: 2026-05-12
 
 VSAG is a vector indexing library used for similarity search. The indexing algorithm allows users to search through various sizes of vector sets, especially those that cannot fit in memory. The library also provides methods for generating parameters based on vector dimensions and data scale, allowing developers to use it without understanding the algorithm’s principles. VSAG is written in C++ and provides a Python wrapper package called [pyvsag](https://pypi.org/project/pyvsag/) and a Node.js/TypeScript binding package called `vsag`.
 
+HGraph supports experimental, opt-in adaptive alpha for PiPNN L2 builds; see the [HGraph parameter guide](docs/docs/en/src/indexes/hgraph.md).
+
 ## Performance
 The VSAG algorithm SINDI for sparse vector search achieves a breakthrough in performance, substantially outperforming previous **state-of-the-art (SOTA)** solutions. In our internal tests on a 40-core Intel(R) Xeon(R) Silver 4210R CPU, VSAG's QPS exceeds that of the previous SOTA algorithm, Zilliz, by 166% on the sparse-full(8M) at 98% recall. While the official ann-benchmarks on sparse track runs on an Azure Standard D8lds v5 VM, we plan to submit our results under the official benchmark environment soon to formally validate this performance leap.
 
@@ -329,6 +331,10 @@ VSAG referenced the following works during its implementation:
   Sebastiano Vigna  
    [PDF](https://dl.acm.org/doi/pdf/10.1145/2433396.2433409) | [DOI](https://doi.org/10.1145/2433396.2433409)
 
+3. PiPNN: Ultra-Scalable Graph-Based Nearest Neighbor Indexing [_arXiv_, 2026] \
+  Tobias Rubel, Richard Wen, Laxman Dhulipala, Lars Gottesbüren, Rajesh Jayaram, Jakub Łącki \
+   [PDF](https://arxiv.org/pdf/2602.21247) | [CODE](https://github.com/ParAlg/PiPNN)
+
 ## Contributors
 
 <!-- prettier-ignore-start -->

docs/docs/en/src/indexes/hgraph.md

@@ -17,8 +17,8 @@ default.
 1. **Graph construction.** Vectors are organised in a layered proximity graph; upper
    layers act as navigation aids, the bottom layer connects every data point to its
    nearest neighbours within a `max_degree` budget. The construction algorithm can be
-   either NSW-style insertion (`graph_type: "nsw"`, the default) or ODescent
-   (`graph_type: "odescent"`).
+   NSW-style insertion (`graph_type: "nsw"`, the default), ODescent
+   (`graph_type: "odescent"`), or PiPNN (`graph_type: "pipnn"`).
 2. **Quantization.** The base storage is compressed with a configurable quantizer
    (`base_quantization_type` — `fp32`, `fp16`, `bf16`, `sq8`, `sq4`, `sq8_uniform`, `sq4_uniform`,
    `pq`, `pqfs`, `rabitq`, `tq`). Optionally, a second high-precision copy is kept
@@ -67,7 +67,17 @@ most users need; the exhaustive list is in [Index Parameters](../resources/index
 | `base_quantization_type` | string | — (required) | `fp32`, `fp16`, `bf16`, `sq8`, `sq4`, `sq8_uniform`, `sq4_uniform`, `pq`, `pqfs`, `rabitq`, `tq` — see the [Quantization chapter](../quantization/README.md) for per-quantizer details |
 | `max_degree` | int | `64` | Maximum out-degree per graph node |
 | `ef_construction` | int | `400` | Candidate list size during build (higher = better recall, slower build) |
-| `graph_type` | string | `"nsw"` | Graph algorithm: `nsw` or `odescent` |
+| `alpha` | float | `1.0` | Final robust-pruning factor; PiPNN requires a finite value at least `1.0`. |
+| `graph_type` | string | `"nsw"` | Graph algorithm: `nsw`, `odescent`, or `pipnn` |
+| `pipnn_max_leaf_size` | int | `1024` | Maximum partition leaf size. Larger leaves increase candidate coverage and leaf distance work. |
+| `pipnn_min_leaf_size` | int | `64` | Target used when merging undersized leaves. |
+| `pipnn_leader_sample_rate` | float | `0.005` | Fraction sampled as partition leaders; each partition uses at least `2` and at most `1000` leaders, bounded by its point count. |
+| `pipnn_fanout` | int[] | `[10, 2]` | Number of nearest leader partitions joined at each listed partition level; deeper levels use `1`. |
+| `pipnn_leaf_neighbor_count` | int | `5` | Nearest candidates contributed per point in each leaf. This is the primary PiPNN quality/build-work knob; leaves smaller than `pipnn_min_leaf_size` use at least `4`. |
+| `pipnn_hash_plane_count` | int | `12` | Direction-hash bits, in `[1, 15]`; `max_degree <= 2^pipnn_hash_plane_count`. |
+| `pipnn_reservoir_size` | int | `64` | Candidate slots retained per point before final pruning; effective capacity is at least `max_degree`. |
+| `pipnn_adaptive_pruning` | bool | `false` | Experimental per-point alpha adjustment in PiPNN final pruning; L2 only. |
+| `pipnn_adaptive_pruning_adjust_step` | float | `0.06` | Alpha adjustment step; finite, non-negative, `alpha - 2*step > 0`, and `alpha + 3*step` finite when enabled. |
 | `use_reorder` | bool | `false` | Keep a high-precision copy and re-rank after the coarse search |
 | `precise_quantization_type` | string | `"fp32"` | Quantizer used for reordering (takes effect only with `use_reorder: true`) |
 | `base_pq_dim` | int | `1` | Number of PQ subspaces. When using `pq` / `pqfs`, set this explicitly instead of relying on the default. |
@@ -86,6 +96,34 @@ The optional [MCI companion](../advanced/hgraph_mci_companion.md) supports filte
 search. Set `mci_knng_source` in `index_param` to `hgraph` (default) or `odescent`
 to select its candidate graph source and enable the companion.
 
+### PiPNN build boundary
+
+Set `graph_type: "pipnn"` to use [PiPNN](https://arxiv.org/abs/2602.21247) for the initial,
+full `Build`. The PiPNN builder accepts dense `float32` input with `metric_type` set to `"l2"`,
+`"ip"`, or `"cosine"`. It builds the bottom graph from the original build vectors, then reuses
+HGraph's route layers, storage, search, filtering, reordering, incremental `Add`, removal, and
+serialization paths. The persistent base storage may use a supported quantizer such as `sq8`,
+including RaBitQ with SQ8 reorder. Cache-assisted build and deduplicated vector storage are not
+supported with PiPNN. The `pipnn_*` parameters in the table above tune this builder;
+`ef_construction` does not. The existing `alpha` parameter controls PiPNN's final robust pruning.
+
+`build_thread_count` parallelizes vector preparation, partitioning, candidate generation, and
+final pruning. Keep `OPENBLAS_NUM_THREADS=1` when benchmarking so BLAS threads do not obscure
+builder scaling. The reproducible `tools/eval/pipnn_parallel.yaml` workload compares NSW,
+ODescent, and PiPNN at a matched Recall@10 target on SIFT1M and query-validates every graph.
+
+PiPNN is available for HGraph only; the Pyramid index does not support `graph_type: "pipnn"`.
+
+### Experimental PiPNN adaptive alpha
+
+Set `pipnn_adaptive_pruning: true` under `index_param` to apply [PR #2933](https://github.com/antgroup/vsag/pull/2933)'s per-node count-based schedule to the final PiPNN reservoir pruning. This opt-in experiment supports L2 and affects the initial batch `Build` only. It leaves partitioning, leaf neighbors, relative hashes, reservoir contents, incremental `Add`, and ODescent route-layer construction unchanged.
+
+For target degree K, scan the existing ordered reservoir at the configured `alpha`, yielding accepted A and scanned rejects B. Unscanned tail candidates do not count toward B. With a nonzero step, if `0 < |A| < K`, keep A and retry B with alpha increased by 1, 2, or 3 steps for `K/|A| <= 1.5`, `(1.5, 3]`, or `> 3`. If `|A| = K`, clear A and rescan the complete original reservoir, including the tail, with alpha decreased by 2, 1, or 0 steps for `|B|/K < 2.5`, `[2.5, 5)`, or `>= 5`. If the tightened pass is short, keep its accepted neighbors and retry its new rejects at the original alpha. Rejected neighbors are never appended unconditionally.
+
+The strict rejection rule remains `alpha * d(selected, candidate) < d(source, candidate)`. PiPNN retains its BF16 source-distance cache, FP32 pairwise squared-L2 calculations, and `(distance, source-ID gap, ID)` order. Unlike the NSW selector's disabled fast path, PiPNN already prunes short reservoirs, so zero adjustment step reproduces its original single-pass adjacency. Greedy selection and the degree cap mean recall need not improve monotonically with alpha.
+
+The policy and step are serialized. Loading an enabled index requires matching policy, base alpha, and step. Missing fields mean disabled. The disabled path preserves legacy behavior. This is experimental: compare build cost and query throughput at matched recall.
+
 ## Supported input data types
 
 The `dtype` field in the top-level build config selects how `Dataset` interprets the raw vector

docs/docs/en/src/resources/index_parameters.md

@@ -39,6 +39,7 @@ HGraph places its build parameters under the generic `index_param` key (see
 |-------|---------|-------------|
 | `max_degree` | 16–48 | Maximum out-degree per node |
 | `ef_construction` | 200–500 | Candidate set size during build; larger = higher recall, slower build |
+| `alpha` | `1.0` | Final robust-pruning factor; PiPNN requires a finite value at least `1.0` |
 | `base_quantization_type` | `fp32` / `fp16` / `bf16` / `sq8` / `sq4` / `pq` | Quantization of the base storage — see the [Quantization chapter](../quantization/README.md) for all supported values |
 
 The optional `mci_knng_source` build parameter enables the MCI companion and selects
@@ -55,6 +56,27 @@ The opt-in `use_hybrid_traversal` search mode combines sparse and clique neighbo
 controlling per-expansion work and seeds. See the
 [MCI companion guide](../advanced/hgraph_mci_companion.md#dynamic-neighbor-traversal).
 
+### PiPNN build parameters
+
+When `graph_type` is `pipnn` on HGraph, the following build parameters are also accepted under
+`index_param`. PiPNN is available for HGraph only; the Pyramid index does not support
+`graph_type: "pipnn"`.
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `pipnn_max_leaf_size` | `1024` | Maximum partition leaf size, in `[2, 2147483647]`. |
+| `pipnn_min_leaf_size` | `64` | Target for merging undersized leaves; must be positive and no larger than `pipnn_max_leaf_size`. |
+| `pipnn_leader_sample_rate` | `0.005` | Fraction sampled as leaders, in `(0, 1]`; each partition uses `2` to `1000`, bounded by its point count. |
+| `pipnn_fanout` | `[10, 2]` | Positive nearest-leader fanout at successive partition levels; deeper levels use `1`. |
+| `pipnn_leaf_neighbor_count` | `5` | Positive nearest-candidate count per point and leaf; undersized leaves use at least `4`. |
+| `pipnn_hash_plane_count` | `12` | Direction-hash bits in `[1, 15]`; `max_degree <= 2^pipnn_hash_plane_count`. |
+| `pipnn_reservoir_size` | `64` | Positive per-point candidate capacity before final pruning; effective capacity is `max(value, max_degree)` and must not exceed `65535`. |
+
+The existing `alpha` field controls PiPNN's final robust pruning. `ef_construction` only tunes NSW
+and has no effect on PiPNN.
+
+HGraph additionally supports the experimental L2-only options `pipnn_adaptive_pruning` (default `false`) and `pipnn_adaptive_pruning_adjust_step` (default `0.06`). They adapt the final per-point pruning threshold during initial `Build`; see [HGraph adaptive PiPNN pruning](../indexes/hgraph.md). These options are not exposed by Pyramid.
+
 At search time:
 
 ```json
@@ -140,6 +162,9 @@ Pyramid supports organising multiple subgraphs by tag:
 }
 ```
 
+`graph_type: "pipnn"` is not supported by Pyramid; the PiPNN build parameters above apply to
+HGraph only.
+
 ## SINDI (sparse vectors)
 
 ```json

docs/docs/zh/src/indexes/hgraph.md

@@ -13,8 +13,8 @@ HGraph 都是推荐的默认索引。
 ## 工作原理
 
 1. **构图。** 向量被组织成层级近邻图：上层作为导航入口，底层连接每个数据点到在
-   `max_degree` 预算内的最近邻。构图算法可以是 NSW 风格插入（`graph_type: "nsw"`，默认）
-   或 ODescent（`graph_type: "odescent"`）。
+   `max_degree` 预算内的最近邻。构图算法可以是 NSW 风格插入（`graph_type: "nsw"`，默认）、
+   ODescent（`graph_type: "odescent"`）或 PiPNN（`graph_type: "pipnn"`）。
 2. **量化。** 底层存储使用可配置的量化器进行压缩（`base_quantization_type` —
    `fp32`、`fp16`、`bf16`、`sq8`、`sq4`、`sq8_uniform`、`sq4_uniform`、`pq`、`pqfs`、`rabitq`、`tq`）。
    可选地再保留一份高精度副本（`use_reorder: true` 搭配 `precise_quantization_type`），
@@ -61,7 +61,17 @@ auto result = index->KnnSearch(
 | `base_quantization_type` | string | —（必填） | `fp32`、`fp16`、`bf16`、`sq8`、`sq4`、`sq8_uniform`、`sq4_uniform`、`pq`、`pqfs`、`rabitq`、`tq` —— 各量化器细节见[量化章节](../quantization/README.md) |
 | `max_degree` | int | `64` | 图节点最大出度 |
 | `ef_construction` | int | `400` | 构建阶段的候选集大小（越大召回越高，构建越慢） |
-| `graph_type` | string | `"nsw"` | 构图算法：`nsw` 或 `odescent` |
+| `alpha` | float | `1.0` | 最终 robust pruning 的系数；PiPNN 要求有限且不小于 `1.0` |
+| `graph_type` | string | `"nsw"` | 构图算法：`nsw`、`odescent` 或 `pipnn` |
+| `pipnn_max_leaf_size` | int | `1024` | 分区叶子的最大点数；越大候选覆盖可能越高，叶内距离计算也越多 |
+| `pipnn_min_leaf_size` | int | `64` | 合并过小叶子时使用的目标规模 |
+| `pipnn_leader_sample_rate` | float | `0.005` | 分区 leader 的采样比率；每个分区最少使用 `2` 个、最多 `1000` 个 leader，且不超过分区点数 |
+| `pipnn_fanout` | int[] | `[10, 2]` | 每个列出的分区层级中，一个点加入的最近 leader 分区数；更深层使用 `1` |
+| `pipnn_leaf_neighbor_count` | int | `5` | 每个点在每个叶子中贡献的最近候选数，是 PiPNN 主要的质量/构建工作量旋钮；小于 `pipnn_min_leaf_size` 的叶子至少使用 `4` |
+| `pipnn_hash_plane_count` | int | `12` | 方向 hash 位数，范围 `[1, 15]`；`max_degree` 不能超过 `2` 的该次幂 |
+| `pipnn_reservoir_size` | int | `64` | 最终剪枝前每个点保留的候选槽数；实际容量至少为 `max_degree` |
+| `pipnn_adaptive_pruning` | bool | `false` | 实验性逐点动态 alpha，仅用于 PiPNN 最终剪枝，要求 L2 |
+| `pipnn_adaptive_pruning_adjust_step` | float | `0.06` | 调整步长；启用时要求有限、非负、`alpha - 2*step > 0`，且 `alpha + 3*step` 有限 |
 | `use_reorder` | bool | `false` | 是否额外保留一份高精度副本用于精排 |
 | `precise_quantization_type` | string | `"fp32"` | 精排使用的量化类型（仅在 `use_reorder: true` 时生效） |
 | `base_pq_dim` | int | `1` | PQ 子空间数（`pq` / `pqfs` 时必填） |
@@ -80,6 +90,33 @@ auto result = index->KnnSearch(
 在 `index_param` 中设置 `mci_knng_source` 为 `hgraph`（默认）或 `odescent`，
 即可选择候选图来源并启用挂件。
 
+### PiPNN 构建边界
+
+设置 `graph_type: "pipnn"` 可让初次全量 `Build` 使用
+[PiPNN](https://arxiv.org/abs/2602.21247)。PiPNN 构建器接收 `dtype: "float32"`、
+`metric_type: "l2"`、`"ip"` 或 `"cosine"` 的稠密向量输入，从原始构建向量生成底层图，并复用
+HGraph 现有的路由层、向量存储、搜索、过滤、精排、增量 `Add`、删除和序列化路径。持久化底层
+存储可以使用 `sq8` 等受支持的量化器，包括 RaBitQ 配合 SQ8 精排。缓存辅助构建和向量存储去重
+暂不支持 PiPNN。上表的 `pipnn_*` 参数用于调节该构建器，`ef_construction` 不生效；
+现有的 `alpha` 参数控制 PiPNN 的最终 robust pruning。
+
+`build_thread_count` 会并行化向量预处理、分区、候选边生成和最终剪枝。性能测试时应固定
+`OPENBLAS_NUM_THREADS=1`，避免 BLAS 线程影响构建线程扩展性。可复现配置
+`tools/eval/pipnn_parallel.yaml` 会在 SIFT1M 上以相同 Recall@10 目标比较 NSW、ODescent 和
+PiPNN，并通过查询验证每一份生成的图。
+
+PiPNN 仅适用于 HGraph；Pyramid 索引不支持 `graph_type: "pipnn"`。
+
+### 实验性 PiPNN 动态 alpha
+
+在 `index_param` 中设置 `pipnn_adaptive_pruning: true`，将 [PR #2933](https://github.com/antgroup/vsag/pull/2933) 的逐点计数反馈规则用于 PiPNN 最终 reservoir 剪枝。此实验功能默认关闭，仅支持 L2 和初始批量 `Build`；分区、leaf 内候选、relative hash、reservoir 内容、后续增量 `Add` 及 ODescent 路由层构建保持原有行为。
+
+设目标度数为 K，首轮按配置的 alpha 扫描原有有序 reservoir，得到接受集合 A 和已扫描的拒绝集合 B；未扫描尾部不计入 B。步长非零时，若 `0 < |A| < K`，保留 A，根据 `K/|A| <= 1.5`、`(1.5, 3]`、`> 3` 分别将 alpha 增加 1、2、3 个步长，重扫 B。若 `|A| = K`，根据 `|B|/K < 2.5`、`[2.5, 5)`、`>= 5` 分别将 alpha 减少 2、1、0 个步长，清空 A 并重扫完整原候选列表（包括首轮未扫描尾部）。收紧后若不足 K，保留当前接受项，按原 alpha 重扫收紧阶段新产生的拒绝集合。不会无条件补入被拒绝的邻居。
+
+拒绝条件仍为 `alpha * d(已选邻居, 候选) < d(源点, 候选)`。保留 PiPNN 的 BF16 源点距离缓存、FP32 两点平方 L2 计算和 `(距离, 源点 ID 差, ID)` 排序。PiPNN 原算法已对较短 reservoir 剪枝，因此零步长会复现其原单轮邻接结果。贪心选择和度数上限意味着 Recall 不一定随 alpha 单调增加。
+
+策略和步长会序列化；加载已启用的索引要求策略、基础 alpha、步长一致，旧参数缺失字段按关闭处理。关闭时保留原有行为。该功能属于实验性质，需对比构建开销和匹配召回率时的查询吞吐。
+
 ## 支持的输入数据类型
 
 顶层构建配置中的 `dtype` 字段决定 `Dataset` 如何解释原始向量字节。HGraph 支持四种输入类型，