@@ -61,21 +61,23 @@ The tool reads the layouts defined in [HDF5 Dataset Format](dataset_format.md):
   `uint32 len | uint32 ids[len] | float32 vals[len]` records, optionally indexed by
   `/<split>_offsets`. Sparse dimensionality is inferred as `max(feature id) + 1`.
 
-For dense one-byte unsigned HDF5 integers, the profiler reinterprets each byte as signed `INT8`
-(128..255 becomes -128..-1), matching `eval_performance`'s `H5::PredType::ALPHA_I8` reads. The
-report dtype and all numeric statistics therefore use the same values as the evaluator.
+For dense one-byte unsigned HDF5 integers, the profiler uses HDF5's numeric conversion to signed
+`INT8` (values above 127 saturate at 127), matching `eval_performance`'s `H5::PredType::ALPHA_I8`
+reads. The report dtype and all numeric statistics therefore use the same values as the evaluator.
 
 `multi_vector` files are rejected with an explanatory error. Data is streamed in bounded
 chunks: the profiler never materializes a complete dataset, and pairwise work is capped by
 the configured sample size, so no `O(N²)` matrix is ever allocated over the full dataset.
 Dense block sizes are derived from a working-memory budget rather than a fixed row count,
 sparse per-dimension accumulators stop at 10,000,000 inferred dimensions (the other metrics
 are still reported), and Euclidean distances switch to direct coordinate differences when
-the fast expanded form would cancel catastrophically. Full norm storage (8 bytes per vector)
-and exact-duplicate hash storage (estimated at 96 bytes per vector) each have a 256 MiB budget.
-If the row-count estimate exceeds either budget, the corresponding full-split norms/outliers or
-exact-duplicate fields are explicitly marked unavailable with a reason; no partial result is
-presented as full-dataset coverage, and streaming/sampled metrics are still reported.
+the fast expanded form would cancel catastrophically. Full norm and sparse nnz storage (8 bytes
+per vector each) and fixed-width exact-duplicate digests (16 bytes per vector) each have a 256 MiB
+retained-state budget. If the row-count estimate exceeds a budget, the corresponding full-split
+norms/outliers, nnz distribution, or exact-duplicate fields are explicitly marked unavailable with
+a reason; no partial result is presented as full-dataset coverage, and streaming/sampled metrics
+are still reported. Sparse offsets are validated in bounded windows; an index over 64 MiB is not
+retained and the reader falls back to bounded sequential decoding.
 
 To profile arrays you only have in memory — for example the same arrays you pass to
 `pyvsag` — first export them with h5py to the layout above; see
 
@@ -56,17 +56,18 @@ python -m dataset_profiler my_dataset.hdf5 \
   `uint32 len | uint32 ids[len] | float32 vals[len]`，可选地由 `/<split>_offsets` 索引。稀疏
   维度按 `max(特征 id) + 1` 推断。
 
-对于稠密的一字节无符号 HDF5 整数，分析工具会逐字节重解释为有符号 `INT8`（128..255 映射为
--128..-1），与 `eval_performance` 使用 `H5::PredType::ALPHA_I8` 读取的行为一致。因此，报告中的
+对于稠密的一字节无符号 HDF5 整数，分析工具通过 HDF5 数值转换读取为有符号 `INT8`（大于 127
+的值饱和为 127），与 `eval_performance` 使用 `H5::PredType::ALPHA_I8` 读取的行为一致。因此，报告中的
 dtype 与所有数值统计均使用 evaluator 实际看到的值。
 
 `multi_vector` 文件会被拒绝并给出说明。数据按有界分块流式读取：工具不会把完整数据集载入内存，
 两两距离计算受采样规模约束，绝不会为完整数据集分配 `O(N²)` 矩阵。稠密数据的分块行数由工作内存
 预算推导而非固定行数；稀疏每维累加器在推断维度超过 10,000,000 时停止（其余指标仍会输出）；
-当快速展开式计算会出现灾难性抵消时，欧氏距离改用坐标直接相减。全量范数存储（每向量 8 字节）
-与精确重复哈希存储（按每向量 96 字节估算）各有 256 MiB 预算。如果按行数估算超过任一预算，
-对应的全量范数/异常值或精确重复字段会明确标记为不可用并给出原因，不会把部分结果伪装成全量
-统计；其余流式与采样指标仍会输出。
+当快速展开式计算会出现灾难性抵消时，欧氏距离改用坐标直接相减。全量范数与稀疏 nnz 存储
+（各每向量 8 字节）以及定长精确重复摘要（每向量 16 字节）各有 256 MiB 保留状态预算。
+如果按行数估算超过任一预算，对应的全量范数/异常值、nnz 分布或精确重复字段会明确标记为不可用
+并给出原因，不会把部分结果伪装成全量统计；其余流式与采样指标仍会输出。稀疏 offsets 采用有界
+窗口校验；超过 64 MiB 的索引不会保留在内存中，读取器会回退到有界顺序解码。
 
 如果数据只存在于内存中（例如准备传给 `pyvsag` 的数组），可先用 h5py 按上述布局导出为 HDF5，
 参见 `tools/dataset_profiler/examples/README.md`。
 
@@ -51,12 +51,14 @@ dataset, and bounded memory for streaming sparse files.
   Full-dataset statistics (norms, per-dimension aggregates, duplicates,
   invalid counts, sparsity) come from a single streaming pass. Norm state is
   preallocated once (8 bytes per vector), never accumulated as block lists and
-  concatenated. Norm state and exact-duplicate hash state (estimated at 96
-  bytes per vector for the digest object and `Counter` entry) each have an
+  concatenated. Norm state, sparse nnz state (8 bytes per vector each), and
+  exact-duplicate digests (a preallocated 16-byte row per vector) each have an
   explicit 256 MiB retained-state budget. The row-count estimate is checked
   before allocation; when it exceeds the budget, the affected full-split
-  norms/outliers or exact-duplicate fields are marked `available: false` with
-  a reason and warning, while streaming and sampled metrics still appear.
+  norms/outliers, nnz distribution, or exact-duplicate fields are marked
+  unavailable with a reason and warning, while streaming totals and sampled
+  metrics still appear. Duplicate groups are counted with NumPy over the
+  fixed-width digest rows rather than a Python `Counter`.
 * **Distance computations**: Euclidean pairwise distances use the expanded
   norm/dot form when it is numerically stable and fall back to direct
   coordinate differences for pairs whose norms dwarf their distance (for
@@ -77,8 +79,11 @@ dataset, and bounded memory for streaming sparse files.
   convention (BLAS-dependent), which only affects projection scatter
   orientation, not the scalar anisotropy statistics.
 * **Sparse offsets**: results must not change between the `*_offsets`-indexed
-  and scan-only encodings of the same byte stream. Tests assert full-dataset
-  metric equality between the two paths.
+  and scan-only encodings of the same byte stream. Offset validation reads
+  bounded windows. An index larger than 64 MiB is not retained; the reader
+  uses the bounded sequential decoder and cross-checks its decoded record
+  count. Tests assert full-dataset metric equality between the paths and
+  exercise the oversized-index fallback.
 * **Invalid values**: NaN/Inf entries are excluded from numeric summaries and
   treated as zero in norms, pairwise math, and PCA input (documented per
   metric). An all-zero vector is a *valid* vector: it contributes to counts,
@@ -153,12 +158,13 @@ message: deterministic reports are still produced and a warning is printed
 * Near-duplicate detection is a bounded *indicator* over the sample, not a
   full-dataset guarantee.
 * The duplicate hash includes invalid bytes as-is (byte equality, not numeric
-  equality). Dense one-byte unsigned input is viewed as signed `INT8` before
-  hashing and numeric statistics, matching the evaluator's `ALPHA_I8` reads;
-  this preserves the bytes but maps 128..255 to -128..-1.
-* Full norm quantiles/outliers and exact duplicate counts are unavailable when
-  their estimated per-row state exceeds the 256 MiB per-structure budget; the
-  report marks these fields unavailable rather than returning partial results.
+  equality). Dense one-byte unsigned input is numerically converted by HDF5 to
+  signed `INT8` before hashing and numeric statistics, matching the evaluator's
+  `ALPHA_I8` reads; values above 127 saturate at 127 rather than wrapping.
+* Full norm quantiles/outliers, sparse nnz distributions, and exact duplicate
+  counts are unavailable when their estimated per-row state exceeds the 256 MiB
+  per-structure budget; the report marks these fields unavailable rather than
+  returning partial results.
 * k-means is an O(n·k) Lloyd loop over the 2-D projection; it is intended for
   structure hints, not production clustering.
 * Sparse dimensionality is inferred as `max(id) + 1`; a file with id gaps at
 
@@ -77,6 +77,8 @@
         "definition",
         "scope",
         "sample_size",
+        "available",
+        "reason",
         "count",
         "stats",
         "histogram",
@@ -86,6 +88,8 @@
         "definition",
         "scope",
         "sample_size",
+        "available",
+        "reason",
         "count",
         "outlier_count",
         "outlier_ratio",
@@ -123,6 +127,8 @@
         "density",
         "nnz_per_vector",
         "nnz_histogram",
+        "nnz_available",
+        "nnz_reason",
         "all_zero_vectors",
         "all_zero_vector_ratio",
     ),
 
@@ -18,7 +18,8 @@
 
 * **dense** (``type`` missing or ``"dense"``): ``/<split>`` is an ``(N, D)``
   matrix of ``INT8`` or ``FLOAT32`` values.  One-byte unsigned integers are
-  reinterpreted as ``INT8`` to match the evaluator's ``ALPHA_I8`` reads.
+  numerically converted to ``INT8`` by HDF5 to match the evaluator's
+  ``ALPHA_I8`` reads (out-of-range values saturate at 127).
 * **sparse** (``type="sparse"``): ``/<split>`` is a flat ``INT8`` byte stream.
   Each record is ``uint32 len | uint32 ids[len] | float32 vals[len]`` in
   little-endian order.  ``/<split>_offsets`` is an optional ``UINT64`` array of
@@ -38,6 +39,8 @@
 import numpy as np
 
 DEFAULT_CHUNK_BYTES = 64 * 1024 * 1024
+_OFFSET_INDEX_BUDGET_BYTES = DEFAULT_CHUNK_BYTES
+_OFFSET_VALIDATION_ROWS = 1024 * 1024
 _LEN_STRUCT = struct.Struct("<I")
 _LEN_SIZE = _LEN_STRUCT.size
 
@@ -175,27 +178,21 @@ def _read_metadata(self) -> DatasetMetadata:
                 )
             offsets_name = self.split + "_offsets"
             if offsets_name in f:
-                offsets = np.asarray(f[offsets_name][()], dtype=np.uint64)
-                if len(offsets) < 1:
-                    raise DatasetFormatError(
-                        "'/%s' must contain at least the trailing sentinel" % offsets_name
-                    )
-                if int(offsets[0]) != 0:
-                    raise DatasetFormatError(
-                        "'/%s' must start at byte offset 0" % offsets_name
-                    )
-                if np.any(offsets[1:] < offsets[:-1]):
-                    raise DatasetFormatError(
-                        "'/%s' must be non-decreasing" % offsets_name
-                    )
-                if int(offsets[-1]) != int(dset.shape[0]):
-                    raise DatasetFormatError(
-                        "'/%s' sentinel (%d) does not match '/%s' size (%d)"
-                        % (offsets_name, int(offsets[-1]), self.split, int(dset.shape[0]))
-                    )
-                count = len(offsets) - 1
+                offsets_dset = f[offsets_name]
+                offset_count = _validate_offsets(
+                    offsets_dset, int(dset.shape[0]), offsets_name
+                )
+                count = offset_count - 1
                 has_offsets = True
-                self._offsets = offsets
+                offset_bytes = offset_count * np.dtype(np.uint64).itemsize
+                if offset_bytes <= _OFFSET_INDEX_BUDGET_BYTES:
+                    self._offsets = np.asarray(offsets_dset[()], dtype=np.uint64)
+                else:
+                    warnings.append(
+                        "sparse '/%s' exceeds the %d-byte retained-index budget; "
+                        "using the bounded sequential decoder"
+                        % (offsets_name, _OFFSET_INDEX_BUDGET_BYTES)
+                    )
             else:
                 warnings.append(
                     "sparse dataset has no '/%s_offsets'; records are counted by scanning"
@@ -238,7 +235,7 @@ def iter_dense_blocks(
         total = int(dset.shape[0])
         for start in range(0, total, block_rows):
             end = min(start + block_rows, total)
-            yield start, _evaluator_dense_view(dset[start:end])
+            yield start, _read_evaluator_dense(dset, slice(start, end))
 
     def read_dense_rows(self, indices: Sequence[int]) -> np.ndarray:
         """Read selected rows from a dense matrix (indices must be sorted)."""
@@ -250,7 +247,7 @@ def read_dense_rows(self, indices: Sequence[int]) -> np.ndarray:
         if np.any(idx[1:] <= idx[:-1]):
             raise ValueError("dense row indices must be strictly increasing")
         dset = self._file[self.split]
-        return _evaluator_dense_view(dset[idx.tolist()])
+        return _read_evaluator_dense(dset, idx.tolist())
 
     # -- sparse --------------------------------------------------------
     def iter_sparse_records(
@@ -264,7 +261,13 @@ def iter_sparse_records(
         if self._offsets is not None:
             yield from self._iter_sparse_with_offsets(max_chunk_bytes)
         else:
-            yield from self._iter_sparse_without_offsets(max_chunk_bytes)
+            # With no index, or with an index too large to retain, decode the
+            # byte stream directly.  An oversized index still supplies the
+            # record count, which is cross-checked by the decoder.
+            yield from self._iter_sparse_without_offsets(
+                max_chunk_bytes,
+                expected_count=self.metadata.count if self.metadata.has_offsets else None,
+            )
 
     def _iter_sparse_with_offsets(
         self, max_chunk_bytes: int
@@ -297,7 +300,7 @@ def _iter_sparse_with_offsets(
             start = end
 
     def _iter_sparse_without_offsets(
-        self, max_chunk_bytes: int
+        self, max_chunk_bytes: int, expected_count: Optional[int] = None
     ) -> Iterator[Tuple[int, np.ndarray, np.ndarray]]:
         dset = self._file[self.split]
         total_bytes = int(dset.shape[0])
@@ -334,6 +337,11 @@ def _iter_sparse_without_offsets(
                 "sparse byte stream has %d trailing byte(s) that do not form a "
                 "complete record" % len(buffer)
             )
+        if expected_count is not None and index != expected_count:
+            raise DatasetFormatError(
+                "sparse decoded record count (%d) does not match '/%s_offsets' (%d)"
+                % (index, self.split, expected_count)
+            )
 
     def read_sparse_rows(
         self, indices: Sequence[int], max_chunk_bytes: int = DEFAULT_CHUNK_BYTES
@@ -357,12 +365,43 @@ def read_sparse_rows(
         return out
 
 
-def _evaluator_dense_view(values) -> np.ndarray:
-    """Match the evaluator's signed interpretation of one-byte dense values."""
-    array = np.asarray(values)
-    if array.dtype.kind == "u" and array.dtype.itemsize == 1:
-        return array.view(np.int8)
-    return array
+def _read_evaluator_dense(dset, selection) -> np.ndarray:
+    """Read dense values with the evaluator's HDF5 destination datatype."""
+    dtype = np.dtype(dset.dtype)
+    if dtype.kind == "u" and dtype.itemsize == 1:
+        # ALPHA_I8 is a signed destination type.  HDF5 performs a numeric
+        # conversion and saturates uint8 values above 127; viewing the bytes
+        # would instead wrap them to negative values.
+        return np.asarray(dset.astype(np.int8)[selection])
+    return np.asarray(dset[selection])
+
+
+def _validate_offsets(dset, total_bytes: int, name: str) -> int:
+    """Validate an on-disk offset index using bounded read windows."""
+    if len(dset.shape) != 1:
+        raise DatasetFormatError("'/%s' must be a 1-D offset array" % name)
+    count = int(dset.shape[0])
+    if count < 1:
+        raise DatasetFormatError(
+            "'/%s' must contain at least the trailing sentinel" % name
+        )
+    first = int(np.asarray(dset[0], dtype=np.uint64))
+    last = int(np.asarray(dset[count - 1], dtype=np.uint64))
+    if first != 0:
+        raise DatasetFormatError("'/%s' must start at byte offset 0" % name)
+    if last != int(total_bytes):
+        raise DatasetFormatError(
+            "'/%s' sentinel (%d) does not match the sparse byte stream size (%d)"
+            % (name, last, int(total_bytes))
+        )
+    previous = first
+    for start in range(1, count, _OFFSET_VALIDATION_ROWS):
+        end = min(start + _OFFSET_VALIDATION_ROWS, count)
+        window = np.asarray(dset[start:end], dtype=np.uint64)
+        if int(window[0]) < previous or np.any(window[1:] < window[:-1]):
+            raise DatasetFormatError("'/%s' must be non-decreasing" % name)
+        previous = int(window[-1])
+    return count
 
 
 def _describe_dtype(dtype: np.dtype) -> str: