diff --git a/docs/docs/en/src/SUMMARY.md b/docs/docs/en/src/SUMMARY.md index ceac850926..45a690a640 100644 --- a/docs/docs/en/src/SUMMARY.md +++ b/docs/docs/en/src/SUMMARY.md @@ -98,6 +98,7 @@ - [HDF5 Dataset Format](resources/dataset_format.md) - [Index Analysis Tool](resources/analyze_index.md) - [Compatibility Check Tool](resources/check_compatibility.md) +- [Dataset Feature Visualization Tool](resources/dataset_profiler.md) # Resources diff --git a/docs/docs/en/src/resources/dataset_profiler.md b/docs/docs/en/src/resources/dataset_profiler.md new file mode 100644 index 0000000000..b1e3373b73 --- /dev/null +++ b/docs/docs/en/src/resources/dataset_profiler.md @@ -0,0 +1,185 @@ +# Dataset Feature Visualization (`dataset_profiler`) + +VSAG ships tools for evaluating index performance (`eval_performance`) and for visualizing the +serialized layout of an index (`visualize_index`). The **dataset profiler** completes the picture +from the other side: it analyzes and visualizes the *feature characteristics of an input vector +dataset* — the vector norms, per-dimension value distributions, sparsity, duplicates, sampled +distance distributions, outliers, and anisotropy that influence index selection, quantization +quality, recall, and search performance. + +It is a pure-Python tool (`tools/dataset_profiler`) that understands the same dense and sparse +HDF5 files consumed by `eval_performance`, and it produces two artifacts: + +- a **versioned JSON report** with every metric tagged by definition, scope (full or sampled), + and sample size; +- a **self-contained interactive HTML report** (inline CSS/JS/SVG, opens offline); + +plus an **optional AI-assisted interpretation** that explains the *aggregate metrics* — never +raw vectors — and phrases each suggestion as a hypothesis backed by cited metric keys. + +## Installation + +The profiler requires Python 3.10+ with NumPy and h5py; it is not part of the C++ build. + +```bash +cd tools/dataset_profiler +pip install -r requirements.txt +``` + +## Quick start + +```bash +# Profile the train split with the default deterministic settings. +python -m dataset_profiler /path/to/my_dataset.hdf5 +``` + +By default this writes `my_dataset.profile.json` and `my_dataset.profile.html` next to the +input file. Use `--out-json` / `--out-html` to place them elsewhere (pass `-` to `--out-json` +to print the report to stdout instead). + +A richer invocation: + +```bash +python -m dataset_profiler my_dataset.hdf5 \ + --split train --seed 42 \ + --sample-size 512 --pairwise-sample-size 256 \ + --metric euclidean --clusters 8 \ + --out-json my_dataset.profile.json --out-html my_dataset.profile.html +``` + +`tools/dataset_profiler/examples/make_toy_datasets.py` regenerates the bundled demo datasets +and reports end to end, and the `tools/dataset_profiler/examples/output/` directory contains +the committed example JSON/HTML files. + +## Supported input formats + +The tool reads the layouts defined in [HDF5 Dataset Format](dataset_format.md): + +- **Dense** (`type` absent or `"dense"`): `/` is an `(N, D)` matrix of `INT8` or + `FLOAT32` values. +- **Sparse** (`type="sparse"`): `/` is a flat byte stream of + `uint32 len | uint32 ids[len] | float32 vals[len]` records, optionally indexed by + `/_offsets`. Sparse dimensionality is inferred as `max(feature id) + 1`. + +For dense one-byte unsigned HDF5 integers, the profiler uses HDF5's numeric conversion to signed +`INT8` (values above 127 saturate at 127), matching `eval_performance`'s `H5::PredType::ALPHA_I8` +reads. The report dtype and all numeric statistics therefore use the same values as the evaluator. + +`multi_vector` files are rejected with an explanatory error. Data is streamed in bounded +chunks: the profiler never materializes a complete dataset, and pairwise work is capped by +the configured sample size, so no `O(N²)` matrix is ever allocated over the full dataset. +Dense block sizes are derived from a working-memory budget rather than a fixed row count, +sparse per-dimension accumulators stop at 10,000,000 inferred dimensions (the other metrics +are still reported), and Euclidean distances switch to direct coordinate differences when +the fast expanded form would cancel catastrophically. Full norm and sparse nnz storage (8 bytes +per vector each) each have a 256 MiB retained-state budget. Exact-duplicate detection retains a +16-byte digest per vector but uses a peak-aware budget of 64 bytes per vector, also covering +`np.unique`'s sorted copy, sort index, unique output, and counts, against the same 256 MiB +full-split state budget. It is available through 4,194,304 vectors +(`256 × 1024 × 1024 / 64`, approximately 4.19 million) and becomes unavailable above that +threshold. If the row-count estimate exceeds a budget, the corresponding full-split norms/outliers, +nnz distribution, or exact-duplicate fields are explicitly marked unavailable with a reason; no +partial result is presented as full-dataset coverage, and streaming/sampled metrics are still +reported. Sparse offsets are validated in bounded windows; an index over 64 MiB is not retained and +the reader falls back to bounded sequential decoding. + +To profile arrays you only have in memory — for example the same arrays you pass to +`pyvsag` — first export them with h5py to the layout above; see +[tools/dataset_profiler/examples/README.md](https://github.com/antgroup/vsag/tree/main/tools/dataset_profiler/examples). + +## Metrics + +Every metric in the JSON report carries a human-readable `definition`, a `scope` of `full`, +`sample`, or `mixed`, and the `sample_size` used. The HTML report repeats this provenance per +section. + +| Metric | Scope | Description | +| --- | --- | --- | +| Dataset summary | full | Count, dimensionality, element type, file attributes. | +| `norms` | full | L2 norm distribution of every vector. | +| `dimensions` | full | Per-coordinate mean, std, min, max, zero ratio; top-variance and top-zero dimensions listed separately; long series are bucket-averaged and flagged. | +| `sparsity` | full | Zero/non-zero counts and density; NaN/Inf are excluded from finite non-zero counts and reported separately. For sparse files also stored entries and nnz-per-vector distribution. | +| `invalid_values` | full | NaN/Inf counts (values and affected vectors). Invalid entries are excluded from numeric summaries and treated as zero elsewhere. | +| `outliers` | full | Robust norm outliers via modified z-score (median/MAD, IQR fallback). | +| `duplicates` | mixed | Exact duplicates (full, byte-identical) plus a near-duplicate indicator over sampled pairs (`distance ≤ epsilon × median sampled distance`); inner-product inputs use non-negative Euclidean distance for this indicator. | +| `pairwise_distances` | sample | Distance distribution between pairs of a bounded random sample. | +| `nearest_neighbor_distances` | sample | Per sampled vector, distance to the nearest other sampled vector. | +| `anisotropy` | sample | Centered-covariance eigenvalue concentration: top-1/top-10 share, effective rank, mean pairwise cosine. | +| `clustering` | sample | Deterministic PCA-2 projection (scatter) and optional seeded k-means summary (`--clusters K`). | + +## Sampling and reproducibility + +- All sampling uses an explicit `--seed` (default 42) with a dedicated PCG64 generator; the + method string is recorded in the report. +- Sample indices are drawn with Floyd's algorithm (bounded by the sample size, streaming-safe); + a seeded permutation selects the pairwise subset without favoring low-index rows. +- `--sample-size` bounds distribution/reduction work; `--pairwise-sample-size` bounds the + distance matrix; both effective sizes are reported alongside the requested ones. Dense sample + materialization is additionally capped at 256 MiB for its float64 matrix; reaching that cap + produces a warning and reduced or unavailable sampled metrics. +- Two runs with identical input, configuration, sample sizes, and seed produce byte-identical + JSON reports (keys sorted, fixed indent). Sparse results are identical whether or not the + file provides `*_offsets`. + +## Privacy and offline behavior + +- The deterministic JSON and HTML reports are generated entirely offline. The HTML inlines all + assets and contacts no external service when opened. +- Raw vectors are never written into the report and never sent to an AI provider: the AI + request contains only aggregate metrics, sampling configuration, warnings, and bounded + non-sensitive metadata. Per-vector entries such as the PCA projection points and cluster + labels are filtered out of that payload. +- AI credentials come only from environment variables, are transmitted exclusively over https + in the `Authorization` header, and are never included in reports, command output, or error + messages (errors are redacted). HTTP redirects are rejected so credentials cannot be forwarded. + +## Optional AI-assisted interpretation + +AI analysis is off by default. Enable it with `--ai` (or by setting +`VSAG_DATASET_PROFILER_AI=1`) and configure it through the environment: + +```bash +export VSAG_AI_API_KEY=... # bearer token (fallback: OPENAI_API_KEY) +export VSAG_AI_MODEL=... # required model name +export VSAG_AI_BASE_URL=... # optional https OpenAI-compatible endpoint +python -m dataset_profiler my_dataset.hdf5 --ai +``` + +The provider receives the aggregate profile and must return a structured analysis: +`summary`, `findings`, `recommendations` (each citing evidence keys that exist in the report +and phrased as hypotheses with a `validation` step), `limitations`, and +`additional_measurements`. The profiler validates the response before merging it into the +report as `ai_analysis`; malformed responses, unknown evidence references, and provider +failures never block report generation — they are surfaced as an `ai_analysis.status = +"error"` entry in the JSON and a visible warning box in the HTML, plus a terminal warning. + +### Validating AI recommendations + +Treat every recommendation as a hypothesis, not a decision: + +1. Reproduce the measurement yourself — open the HTML report and check the cited metric keys + and their scope/sample size. +2. Run the suggested experiment in `eval_performance` (see the [Evaluation Tool](eval.md)) with + your real query set and ground truth; AI interpretation never replaces a reproducible + benchmark. +3. Re-profile after any data transformation (normalization, rotation, pruning) and compare + the before/after metric keys cited by the recommendation. + +## Tests and reference + +Run the tool's unit and integration tests with: + +```bash +cd tools/dataset_profiler && python -m pytest tests -q +``` + +Design notes — metric selection, sampling internals, privacy boundaries, and the AI +prompt/response schema — are in +[`tools/dataset_profiler/TECHNICAL_REPORT.md`](https://github.com/antgroup/vsag/tree/main/tools/dataset_profiler/TECHNICAL_REPORT.md). + +## Related + +- [HDF5 Dataset Format](dataset_format.md) +- [Evaluation Tool](eval.md) +- [Index Analysis Tool](analyze_index.md) +- [FAQ: how data distribution affects vector search](faq.md) diff --git a/docs/docs/zh/src/SUMMARY.md b/docs/docs/zh/src/SUMMARY.md index f720493ae7..ac32950d5b 100644 --- a/docs/docs/zh/src/SUMMARY.md +++ b/docs/docs/zh/src/SUMMARY.md @@ -98,6 +98,7 @@ - [HDF5 数据集格式](resources/dataset_format.md) - [索引分析工具](resources/analyze_index.md) - [兼容性检查工具](resources/check_compatibility.md) +- [数据集特征可视化工具](resources/dataset_profiler.md) # 资源 diff --git a/docs/docs/zh/src/resources/dataset_profiler.md b/docs/docs/zh/src/resources/dataset_profiler.md new file mode 100644 index 0000000000..02f6436b55 --- /dev/null +++ b/docs/docs/zh/src/resources/dataset_profiler.md @@ -0,0 +1,159 @@ +# 数据集特征可视化（`dataset_profiler`） + +VSAG 已提供索引性能评测工具（`eval_performance`）与索引序列化结构可视化工具 +（`visualize_index`）。**数据集特征分析工具**从另一个角度补全了信息：它分析并可视化*输入向量 +数据集本身的特征*——向量范数、各维度数值分布、稀疏度、重复数据、采样距离分布、异常值与各向异性， +这些因素会显著影响索引选择、量化质量、召回率与检索性能。 + +该工具是纯 Python 实现（位于 `tools/dataset_profiler`），直接读取 `eval_performance` 使用的稠密 +与稀疏 HDF5 文件，输出两类产物： + +- **带版本号的 JSON 报告**：每项指标都注明定义、统计范围（全量或采样）以及样本数量； +- **自包含的交互式 HTML 报告**：CSS/JS/SVG 全部内联，可离线打开； + +另有**可选的 AI 辅助解读**：只解释*聚合指标*（绝不接触原始向量），并把每条建议表述为附有指标 +依据的待验证假设。 + +## 安装 + +工具需要 Python 3.10+ 以及 NumPy、h5py，不参与 C++ 构建。 + +```bash +cd tools/dataset_profiler +pip install -r requirements.txt +``` + +## 快速开始 + +```bash +# 使用默认确定性配置分析 train 分片 +python -m dataset_profiler /path/to/my_dataset.hdf5 +``` + +默认在输入文件旁生成 `my_dataset.profile.json` 与 `my_dataset.profile.html`；用 +`--out-json` / `--out-html` 指定输出位置（`--out-json -` 表示将 JSON 打印到标准输出）。 + +更完整的调用示例： + +```bash +python -m dataset_profiler my_dataset.hdf5 \ + --split train --seed 42 \ + --sample-size 512 --pairwise-sample-size 256 \ + --metric euclidean --clusters 8 \ + --out-json my_dataset.profile.json --out-html my_dataset.profile.html +``` + +`tools/dataset_profiler/examples/make_toy_datasets.py` 可端到端重新生成自带的演示数据与报告； +`tools/dataset_profiler/examples/output/` 目录中保存了示例 JSON/HTML 文件。 + +## 支持的数据格式 + +工具读取[HDF5 数据集格式](dataset_format.md)中定义的两种布局： + +- **稠密**（无 `type` 属性或 `type="dense"`）：`/` 是 `(N, D)` 的 `INT8` 或 `FLOAT32` + 矩阵； +- **稀疏**（`type="sparse"`）：`/` 是扁平字节流，每条记录为 + `uint32 len | uint32 ids[len] | float32 vals[len]`，可选地由 `/_offsets` 索引。稀疏 + 维度按 `max(特征 id) + 1` 推断。 + +对于稠密的一字节无符号 HDF5 整数，分析工具通过 HDF5 数值转换读取为有符号 `INT8`（大于 127 +的值饱和为 127），与 `eval_performance` 使用 `H5::PredType::ALPHA_I8` 读取的行为一致。因此，报告中的 +dtype 与所有数值统计均使用 evaluator 实际看到的值。 + +`multi_vector` 文件会被拒绝并给出说明。数据按有界分块流式读取：工具不会把完整数据集载入内存， +两两距离计算受采样规模约束，绝不会为完整数据集分配 `O(N²)` 矩阵。稠密数据的分块行数由工作内存 +预算推导而非固定行数；稀疏每维累加器在推断维度超过 10,000,000 时停止（其余指标仍会输出）； +当快速展开式计算会出现灾难性抵消时，欧氏距离改用坐标直接相减。全量范数与稀疏 nnz 存储 +（各每向量 8 字节）各有 256 MiB 保留状态预算。精确重复检测为每个向量保留 16 字节摘要，但按 +每向量 64 字节的峰值内存进行预算，额外覆盖 `np.unique` 的排序副本、排序索引、去重输出和计数， +并使用同样的 256 MiB 全量状态预算。因此，精确重复检测在不超过 4,194,304 个向量 +（`256 × 1024 × 1024 / 64`，约 419 万）时可用，超过该阈值后变为不可用。 +如果按行数估算超过任一预算，对应的全量范数/异常值、nnz 分布或精确重复字段会明确标记为不可用 +并给出原因，不会把部分结果伪装成全量统计；其余流式与采样指标仍会输出。稀疏 offsets 采用有界 +窗口校验；超过 64 MiB 的索引不会保留在内存中，读取器会回退到有界顺序解码。 + +如果数据只存在于内存中（例如准备传给 `pyvsag` 的数组），可先用 h5py 按上述布局导出为 HDF5， +参见 `tools/dataset_profiler/examples/README.md`。 + +## 指标 + +JSON 报告中每项指标都带有人类可读的 `definition`、`scope`（`full`、`sample` 或 `mixed`）以及 +所用 `sample_size`；HTML 报告在每一节中重复这些溯源信息。 + +| 指标 | 范围 | 说明 | +| --- | --- | --- | +| 数据集概况 | 全量 | 向量数、维度、元素类型、文件属性。 | +| `norms` | 全量 | 每个向量的 L2 范数分布。 | +| `dimensions` | 全量 | 每个坐标维度的均值、标准差、最小/最大值、零值比例；单独列出方差最高与零值比例最高的维度；超长序列会按桶平均并标注。 | +| `sparsity` | 全量 | 零值/非零值计数与密度；NaN/Inf 不计入有限非零值，单独统计。稀疏文件还统计存储条目数与每向量非零元素（nnz）分布。 | +| `invalid_values` | 全量 | NaN/Inf 的数量（按值与受影响向量）。非法值不参与数值统计，在其余计算中按 0 处理。 | +| `outliers` | 全量 | 基于稳健修正 z 分数（median/MAD，IQR 兜底）的范数异常向量。 | +| `duplicates` | 混合 | 精确重复（全量，字节级一致），外加基于采样配对的近重复指示指标（`距离 ≤ epsilon × 采样距离中位数`）；内积输入在该指标中使用非负的欧氏距离。 | +| `pairwise_distances` | 采样 | 有界随机样本内两两距离的分布。 | +| `nearest_neighbor_distances` | 采样 | 每个采样向量到样本内最近邻的距离。 | +| `anisotropy` | 采样 | 中心化协方差特征值集中度：最大特征值占比、前 10 占比、有效秩、平均两两余弦相似度。 | +| `clustering` | 采样 | 确定性 PCA-2 投影散点图；可选的带种子 k-means 摘要（`--clusters K`）。 | + +## 采样与可复现性 + +- 所有采样均使用显式 `--seed`（默认 42）与独立 PCG64 生成器；采样方法字符串记录在报告中。 +- 样本下标用 Floyd 算法抽取（内存/时间受样本量约束，适配流式扫描）；再通过带种子的随机排列 + 选择两两距离子集，避免偏向低下标行。 +- `--sample-size` 约束分布与降维类计算，`--pairwise-sample-size` 约束距离矩阵；报告中同时记录 + 请求值与生效值。稠密样本的 float64 矩阵另有 256 MiB 物化上限；达到上限时会发出警告，并缩减 + 采样规模或将采样指标标记为不可用。 +- 相同输入、配置、采样规模与随机种子两次运行生成的 JSON 报告逐字节一致（键排序、固定缩进）。 + 稀疏文件的分析结果与是否提供 `*_offsets` 无关。 + +## 隐私与离线行为 + +- 确定性 JSON/HTML 报告完全离线生成；HTML 内联全部资源，打开时不访问任何外部服务。 +- 原始向量既不写入报告，也绝不发送给 AI 服务：AI 请求只包含聚合指标、采样配置、警告与有限的 + 非敏感元数据。PCA 投影点、聚类标签等逐向量字段会从该负载中过滤掉。 +- AI 凭据只来自环境变量，仅通过 https 的 `Authorization` 头传输，且绝不写入报告、命令输出或 + 错误信息（错误消息会被脱敏）。工具拒绝 HTTP 重定向，避免凭据被转发到其他端点。 + +## 可选的 AI 辅助解读 + +AI 分析默认关闭。使用 `--ai`（或设置 `VSAG_DATASET_PROFILER_AI=1`）显式启用，并通过环境变量 +配置： + +```bash +export VSAG_AI_API_KEY=... # 访问令牌（兜底：OPENAI_API_KEY） +export VSAG_AI_MODEL=... # 必填的模型名 +export VSAG_AI_BASE_URL=... # 可选，https 的 OpenAI 兼容端点 +python -m dataset_profiler my_dataset.hdf5 --ai +``` + +AI 收到聚合画像后必须返回结构化分析：`summary`、`findings`、`recommendations`（每条都要引用 +报告中真实存在的指标字段，以“假设 + `validation` 验证步骤”的形式表述）、`limitations` 与 +`additional_measurements`。分析器在把结果并入报告的 `ai_analysis` 字段前会做校验；格式错误的 +响应、虚构的指标引用以及服务失败都不会阻碍报告生成——它们在 JSON 中呈现为 +`ai_analysis.status = "error"` 条目，在 HTML 中呈现为醒目警告框，同时在终端打印警告。 + +### 如何验证 AI 建议 + +请把每条建议当作假设而非结论： + +1. 自行复核指标证据——打开 HTML 报告，核对其引用的指标键及统计范围/样本量。 +2. 用 `eval_performance`（见[性能评估工具](eval.md)）配合真实查询集与 ground truth 复现实验； + AI 解读永远不能替代可复现的基准测试。 +3. 任何数据变换（归一化、旋转、维度剪枝）之后重新画像，并对比建议所引用的前后指标键。 + +## 测试与参考 + +运行工具的单元测试与集成测试： + +```bash +cd tools/dataset_profiler && python -m pytest tests -q +``` + +设计说明（指标选择、采样内部机制、隐私边界、AI 提示与响应结构）见 +`tools/dataset_profiler/TECHNICAL_REPORT.md`。 + +## 相关文档 + +- [HDF5 数据集格式](dataset_format.md) +- [性能评估工具](eval.md) +- [索引分析工具](analyze_index.md) +- [FAQ：数据分布如何影响向量检索](faq.md) diff --git a/tools/README.md b/tools/README.md index d98103d7e8..f455e8745c 100644 --- a/tools/README.md +++ b/tools/README.md @@ -9,6 +9,7 @@ nearby entry points for local development and AI coding agents. | `check_compatibility` | [check_compatibility/README.md](check_compatibility/README.md) | [docs/docs/en/src/resources/check_compatibility.md](../docs/docs/en/src/resources/check_compatibility.md) | | | `eval_performance` | [eval/README.md](eval/README.md) | [docs/docs/en/src/resources/eval.md](../docs/docs/en/src/resources/eval.md) | | | `autotune` | [autotune/README.md](autotune/README.md) | [docs/docs/en/src/resources/autotune.md](../docs/docs/en/src/resources/autotune.md) | | +| `dataset_profiler` | [dataset_profiler/README.md](dataset_profiler/README.md) | [docs/docs/en/src/resources/dataset_profiler.md](../docs/docs/en/src/resources/dataset_profiler.md) | | Chinese documentation is available under `docs/docs/zh/src/resources/` and on the Chinese website paths under . diff --git a/tools/dataset_profiler/README.md b/tools/dataset_profiler/README.md new file mode 100644 index 0000000000..ae48edb6ef --- /dev/null +++ b/tools/dataset_profiler/README.md @@ -0,0 +1,76 @@ +# VSAG Dataset Profiler (`dataset_profiler`) + +A reproducible tool that profiles and visualizes the **feature characteristics of an input +vector dataset** — dimension distributions, value statistics, norms, sparsity, duplicates, +sampled pairwise distances, outliers, anisotropy, and optional clustering — for the dense and +sparse HDF5 formats already used by VSAG's evaluation tooling (and by the same arrays you hand +to `pyvsag`). + +The canonical documentation lives in the website source tree. + +- Local English docs: + [docs/docs/en/src/resources/dataset_profiler.md](../../docs/docs/en/src/resources/dataset_profiler.md) +- Local Chinese docs: + [docs/docs/zh/src/resources/dataset_profiler.md](../../docs/docs/zh/src/resources/dataset_profiler.md) +- HDF5 dataset format: + [docs/docs/en/src/resources/dataset_format.md](../../docs/docs/en/src/resources/dataset_format.md) +- Website: +- Chinese website: +- Design notes (metrics, sampling, privacy, AI schema): + [TECHNICAL_REPORT.md](TECHNICAL_REPORT.md) +- Example datasets and reports: [examples/](examples/README.md) + +## Quick start + +Requires Python 3.10+ with `numpy` and `h5py` (see [requirements.txt](requirements.txt)). No +C++ build is needed. + +```bash +cd tools/dataset_profiler +pip install -r requirements.txt + +# Deterministic JSON + self-contained HTML, written next to the input file +python -m dataset_profiler /path/to/my_dataset.hdf5 --split train --seed 42 + +# Bounded sampling, explicit metric, k-means over the PCA-2 projection +python -m dataset_profiler my_dataset.hdf5 \ + --sample-size 512 --pairwise-sample-size 256 \ + --metric euclidean --clusters 8 \ + --out-json my_dataset.profile.json --out-html my_dataset.profile.html +``` + +Regenerate the bundled demo end to end with: + +```bash +python examples/make_toy_datasets.py +``` + +## AI-assisted interpretation (optional) + +The deterministic reports never require network access. The optional AI layer summarizes the +**aggregate metrics** (never raw vectors) and cites metric keys as evidence. Enable it +explicitly with `--ai` and configure it entirely through environment variables: + +```bash +export VSAG_AI_API_KEY=... # bearer token; never stored in reports or logs +export VSAG_AI_MODEL=... # model name (OpenAI-compatible endpoint) +export VSAG_AI_BASE_URL=... # optional, https only +python -m dataset_profiler my_dataset.hdf5 --ai +``` + +If the provider is unavailable the JSON/HTML reports are still generated and a clear warning +is printed. AI output is validated: every finding/recommendation must reference metric keys +that exist in the report, and recommendations are phrased as hypotheses to validate with +`tools/eval`. See [TECHNICAL_REPORT.md](TECHNICAL_REPORT.md) for the prompt/response schema. + +## Tests + +```bash +pip install pytest +python -m pytest tests -q +``` + +## Repository checks + +This tool is pure Python and does not affect the C++ library, so `make fmt`/`make lint`/ +`make test` are not impacted by changes here; run the pytest suite above instead. diff --git a/tools/dataset_profiler/README_zh.md b/tools/dataset_profiler/README_zh.md new file mode 100644 index 0000000000..c61841097a --- /dev/null +++ b/tools/dataset_profiler/README_zh.md @@ -0,0 +1,66 @@ +# VSAG 数据集特征分析工具（`dataset_profiler`） + +一个可复现的数据集特征分析与可视化工具：分析并可视化输入向量数据集的**特征**——各维度数值分布、 +数值统计、向量范数、稀疏度、重复数据、采样向量间距离、异常值、各向异性以及可选的聚类摘要。 +支持 VSAG 评测工具（以及 `pyvsag` 工作流）现有的稠密和稀疏 HDF5 数据格式。 + +规范文档位于网站文档源码树中。 + +- 本地中文文档： + [docs/docs/zh/src/resources/dataset_profiler.md](../../docs/docs/zh/src/resources/dataset_profiler.md) +- 本地英文文档： + [docs/docs/en/src/resources/dataset_profiler.md](../../docs/docs/en/src/resources/dataset_profiler.md) +- HDF5 数据格式： + [docs/docs/zh/src/resources/dataset_format.md](../../docs/docs/zh/src/resources/dataset_format.md) +- 中文网站： +- 英文网站： +- 技术报告（指标、采样、隐私与 AI 结构说明）：[TECHNICAL_REPORT.md](TECHNICAL_REPORT.md) +- 示例数据与示例报告：[examples/](examples/README.md) + +## 快速开始 + +需要 Python 3.10+，并安装 `numpy` 与 `h5py`（见 [requirements.txt](requirements.txt)），无需 C++ +构建。 + +```bash +cd tools/dataset_profiler +pip install -r requirements.txt + +# 生成确定性 JSON 与自包含 HTML（默认写到输入文件旁边） +python -m dataset_profiler /path/to/my_dataset.hdf5 --split train --seed 42 + +# 限定采样规模、指定距离度量、并对 PCA-2 投影做 k-means +python -m dataset_profiler my_dataset.hdf5 \ + --sample-size 512 --pairwise-sample-size 256 \ + --metric euclidean --clusters 8 \ + --out-json my_dataset.profile.json --out-html my_dataset.profile.html +``` + +重新生成本目录自带的演示数据与报告： + +```bash +python examples/make_toy_datasets.py +``` + +## AI 辅助解读（可选） + +确定性报告的生成完全不需要网络。可选的 AI 层只基于**聚合指标**（绝不发送原始向量）生成解读， +并引用指标字段作为依据。通过 `--ai` 显式启用，并且完全通过环境变量配置： + +```bash +export VSAG_AI_API_KEY=... # 访问令牌；绝不写入报告或日志 +export VSAG_AI_MODEL=... # 模型名（OpenAI 兼容接口） +export VSAG_AI_BASE_URL=... # 可选，仅允许 https +python -m dataset_profiler my_dataset.hdf5 --ai +``` + +AI 服务不可用时，JSON/HTML 报告仍会正常生成，并在终端给出明确警告。AI 输出会经过校验：每项 +发现/建议必须引用报告中真实存在的指标字段，且建议以“待通过 `tools/eval` 基准验证的假设”的形式 +呈现。提示与响应结构见 [TECHNICAL_REPORT.md](TECHNICAL_REPORT.md)。 + +## 测试 + +```bash +pip install pytest +python -m pytest tests -q +``` diff --git a/tools/dataset_profiler/TECHNICAL_REPORT.md b/tools/dataset_profiler/TECHNICAL_REPORT.md new file mode 100644 index 0000000000..2d37594e49 --- /dev/null +++ b/tools/dataset_profiler/TECHNICAL_REPORT.md @@ -0,0 +1,177 @@ +# Dataset Feature Visualization Tool — Technical Report + +This report documents the design decisions behind `tools/dataset_profiler` +(`vsag-dataset-profiler`), implemented for issue #2862: metric selection, the +sampling design, privacy boundaries, the AI prompt/response schema, the +validation strategy, and known limitations. + +## 1. Metric selection + +The profiler computes the metric families requested in issue #2862. Every +metric carries a `definition`, a `scope` (`full`, `sample`, or `mixed`), and +the `sample_size` when applicable, so full-dataset facts and estimates are +never conflated. + +| Metric | Scope | Why it matters for index choice | +| --- | --- | --- | +| `norms` | full | Norm spread drives quantization range selection (SQ/PQ/FastScan) and the value of normalization or OPQ-style rotation. | +| `dimensions` (per-coordinate mean/std/min/max/zero-ratio + top lists) | full | Shows scale mismatch between coordinates (a warning sign for Euclidean indexes) and dimensions carrying no signal (candidates for pruning). | +| `sparsity` | full | Exact-zero ratio for dense files; stored entries, non-zero counts per vector (`nnz`), and density for sparse files. Predicts SINDI-style sparse index behavior and compression headroom. | +| `invalid_values` | full | NaN/Inf counts per dataset and per affected vector; indexes and quantizers silently degrade on invalid input. | +| `duplicates` | full (exact) + sample (near) | Exact duplicates are byte-identical vectors (full coverage via hashing). Near-duplicates are a bounded indicator: sampled pairs closer than `epsilon x median(sampled duplicate-detection distance)`, using Euclidean distance for IP inputs to keep the threshold non-negative. | +| `pairwise_distances` | sample | Distribution shape (median, spread, tails) informs distance-based thresholds and graph vs. IVF trade-offs. | +| `nearest_neighbor_distances` | sample | Small NN-distance gaps relative to typical distances indicate hard recall problems for graph indexes. | +| `outliers` | full | Robust modified z-score over norms (median/MAD, IQR fallback) — norm outliers distort shared quantization ranges. | +| `anisotropy` | sample | Eigenvalue concentration of the centered sample covariance (top-1 share, top-10 share, effective rank, mean pairwise cosine). Highly anisotropic data hurts cosine/IP quality and motivates transformation or different quantization. | +| `clustering` | sample + optional | Deterministic PCA-2 projection of the sample (scatter plot) and optional seeded k-means summary when `--clusters K` is given. | + +Excluded on purpose: ground-truth/kNN recall estimation (requires the VSAG +C++ library and belongs to `tools/eval`), and query-distribution analysis +(no queries exist in the dataset files). + +## 2. Sampling design + +Requirements: reproducible reports, no `O(N^2)` allocation over the full +dataset, and bounded memory for streaming sparse files. + +* **Generator**: `numpy.random.Generator(PCG64(seed))`, seeded explicitly by + `--seed` (default 42). No global NumPy state is used or modified. +* **Index selection**: Floyd's algorithm (`sample_indices`) draws `k` distinct + indices from `[0, n)` with memory/time bounded by `k`, not `n` — important + for sparse files without `*_offsets`, which are scanned streaming. +* **Pairwise bound**: a seeded permutation of the uniformly random sample + selects `min(pairwise_sample_size, sample_size, N)` rows for the pairwise + matrix. Sorting the selected positions afterward restores deterministic row + order without biasing the subset toward low dataset indices. The report + records requested vs. effective sizes and the exact method string. +* **Streaming**: dense files are read in row blocks sized from a 128 MiB + working-byte budget (at most 65536 rows), so a wide FLOAT32 matrix cannot + retain gigabytes per block; sparse files are decoded record-by-record from + bounded byte chunks (default 64 MiB). + Full-dataset statistics (norms, per-dimension aggregates, duplicates, + invalid counts, sparsity) come from a single streaming pass. Norm state is + preallocated once (8 bytes per vector), never accumulated as block lists and + concatenated. Norm state and sparse nnz state (8 bytes per vector each) have + an explicit 256 MiB state budget. Exact-duplicate detection preallocates a + 16-byte digest row per vector and uses a peak-aware 64-byte-per-vector budget + that also covers `np.unique`'s sorted copy, sort index, unique output, and + counts. Each row-count estimate is checked before allocation; when it exceeds + the budget, the affected full-split norms/outliers, nnz distribution, or + exact-duplicate fields are marked unavailable with a reason and warning, + while streaming totals and sampled metrics still appear. Duplicate groups + are counted with NumPy over the fixed-width digest rows rather than a Python + `Counter`. +* **Distance computations**: Euclidean pairwise distances use the expanded + norm/dot form when it is numerically stable and fall back to direct + coordinate differences for pairs whose norms dwarf their distance (for + example rows near `1e10` differing by `1`, where the expansion rounds to + zero). The sparse Euclidean branch always subtracts coordinates directly. + The mean pairwise cosine is derived from normalized-vector sums + (`||sum(u)||² - sum(||u||²)`) instead of an `O(M²)` Gram matrix. +* **Per-dimension arrays**: sparse absent coordinates contribute implicit zeros + to per-dimension statistics. Sparse accumulation stops at dimensionality + 10,000,000 (other metrics continue) to bound memory. Series longer than + `max_dimension_points` (default 1024) are bucket-averaged for storage, with + `source_length`, `bucket_size`, and `downsampled` flags recorded so charts + stay honest. +* **Determinism**: two runs with the same file, configuration, sample sizes, + and seed produce byte-identical JSON (`sort_keys`, fixed indent, + `allow_nan=False`). The committed example reports under `examples/output/` + are byte-reproducible. The one environment-dependent piece is SVD sign + convention (BLAS-dependent), which only affects projection scatter + orientation, not the scalar anisotropy statistics. +* **Sparse offsets**: results must not change between the `*_offsets`-indexed + and scan-only encodings of the same byte stream. Offset validation reads + bounded windows. An index larger than 64 MiB is not retained; the reader + uses the bounded sequential decoder and cross-checks its decoded record + count. Tests assert full-dataset metric equality between the paths and + exercise the oversized-index fallback. +* **Invalid values**: NaN/Inf entries are excluded from numeric summaries and + treated as zero in norms, pairwise math, and PCA input (documented per + metric). An all-zero vector is a *valid* vector: it contributes to counts, + norms (0), and the all-zero-vector counter, but is excluded from outlier + detection by the robust method's behavior. + +## 3. Privacy boundaries + +* **Default offline**: JSON/HTML generation never opens a network connection. +* **Aggregate-only AI payload**: the request is built from the report — + aggregate metrics, sampling config, warnings, and non-sensitive metadata + (split name, vector type, dtype, count/dim, distance). A field allowlist + drops per-vector entries (`clustering.projection_points`, which holds one + PCA coordinate pair per sampled vector, and `clustering.labels`, which holds + one cluster assignment per sampled vector). Raw vectors are not in the + report and therefore structurally cannot be sent; tests assert that source + rows never appear in the request body and that those keys are absent. +* **Credentials**: the API key is read from the environment (`VSAG_AI_API_KEY`, + fallback `OPENAI_API_KEY`, or a custom variable via `VSAG_AI_KEY_ENV`), + sent only in the `Authorization` header over https (plaintext URLs are + rejected), and never written to reports. Redirects are rejected so the + bearer token cannot be forwarded to another endpoint. Error messages are + redacted (`sk-…` patterns and bearer tokens) and truncated; tests assert keys + cannot leak through provider failures. +* **Self-contained HTML**: all CSS/JS/SVG are inlined; opening a report + contacts no external service. + +## 4. AI prompt/response schema + +The adapter is provider-neutral (any OpenAI-compatible `/chat/completions` +endpoint) and temperature is fixed at 0. + +Request: the system-style instructions plus +`{summary, findings[], recommendations[], limitations[], additional_measurements[]}`, where every +finding/recommendation is `{title, detail, evidence: [metric key], confidence: low|medium|high}` +and every recommendation additionally carries `validation` (how to test it in a reproducible +`tools/eval` run). The prompt embeds the list of allowed evidence keys (leaf paths enumerated +from `metrics.*`) and the aggregate JSON payload. + +Validation (client side, `parse_and_validate_response`): + +* response must parse as a single JSON object (markdown fences tolerated); +* `summary` non-empty string; `findings`/`recommendations` non-empty lists (<= 10 items); +* every item needs non-empty `evidence`, and **each cited key must resolve inside the report's + `metrics` subtree** — invented keys fail the whole response; +* recommendations must carry a `validation` string; +* `confidence` restricted to `low|medium|high`; strings are length-bounded. + +Any rejection degrades to `ai_analysis.status = "error"` with a redacted +message: deterministic reports are still produced and a warning is printed +(issue acceptance criteria 6 and 7). + +## 5. Validation strategy + +* **Unit tests** (`tests/`): sampling determinism; stats helpers; both HDF5 + layouts incl. corrupt/edge encodings; all profiler metric families incl. + determinism, sample bounds, metric aliasing, invalid values, and the + offsets/scan equivalence invariant; HTML self-containment and escaping; AI + schema validation and failure modes; CLI exit codes and output handling. +* **Integration**: the CLI is exercised end to end against generated dense and + sparse fixtures; `make_toy_datasets.py` reproduces the committed example + reports byte-for-byte. +* **Manual spot checks**: rendered charts were opened locally and cross-checked + against direct NumPy computations on the toy datasets (counts, zero ratios, + duplicate groups). + +## 6. Limitations + +* Sampled metrics (pairwise, NN distances, anisotropy, clustering) are + estimates: their report entries carry `scope: sample` and the sample size, + and the HTML repeats this per section. +* Near-duplicate detection is a bounded *indicator* over the sample, not a + full-dataset guarantee. +* The duplicate hash includes invalid bytes as-is (byte equality, not numeric + equality). Dense one-byte unsigned input is numerically converted by HDF5 to + signed `INT8` before hashing and numeric statistics, matching the evaluator's + `ALPHA_I8` reads; values above 127 saturate at 127 rather than wrapping. +* Full norm quantiles/outliers, sparse nnz distributions, and exact duplicate + counts are unavailable when their estimated per-row state exceeds the 256 MiB + per-structure budget; the report marks these fields unavailable rather than + returning partial results. +* k-means is an O(n·k) Lloyd loop over the 2-D projection; it is intended for + structure hints, not production clustering. +* Sparse dimensionality is inferred as `max(id) + 1`; a file with id gaps at + the top reports a smaller dim than its producer may have assumed (this is + what the evaluation loader implies as well). +* The AI layer follows the repository's issue #2862 boundary: it explains + measured metrics and does not inspect raw data, and its suggestions are + explicitly hypotheses to validate with `tools/eval` benchmarks. diff --git a/tools/dataset_profiler/dataset_profiler/__init__.py b/tools/dataset_profiler/dataset_profiler/__init__.py new file mode 100644 index 0000000000..b0d7e729ff --- /dev/null +++ b/tools/dataset_profiler/dataset_profiler/__init__.py @@ -0,0 +1,30 @@ +# Copyright 2024-present the vsag project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Deterministic profiling and visualization for VSAG-compatible HDF5 datasets. + +The public entry point is :func:`profile_dataset` in :mod:`dataset_profiler.profiler`; +the command-line wrapper lives in :mod:`dataset_profiler.cli`. +""" + +from __future__ import annotations + +TOOL_NAME = "vsag-dataset-profiler" +__version__ = "0.1.0" +SCHEMA_VERSION = "1.0" + +from .profiler import profile_dataset # noqa: E402 (after constants: report.py imports them back) +from .report import build_report # noqa: E402 + +__all__ = ["SCHEMA_VERSION", "TOOL_NAME", "__version__", "build_report", "profile_dataset"] diff --git a/tools/dataset_profiler/dataset_profiler/__main__.py b/tools/dataset_profiler/dataset_profiler/__main__.py new file mode 100644 index 0000000000..f52fc90a78 --- /dev/null +++ b/tools/dataset_profiler/dataset_profiler/__main__.py @@ -0,0 +1,24 @@ +# Copyright 2024-present the vsag project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Enable ``python -m dataset_profiler``.""" + +from __future__ import annotations + +import sys + +from .cli import main + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/dataset_profiler/dataset_profiler/ai.py b/tools/dataset_profiler/dataset_profiler/ai.py new file mode 100644 index 0000000000..b6e67965f3 --- /dev/null +++ b/tools/dataset_profiler/dataset_profiler/ai.py @@ -0,0 +1,631 @@ +# Copyright 2024-present the vsag project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Provider-neutral AI-assisted interpretation of a dataset profile. + +The adapter accepts the *structured, aggregated* profile report produced by +:func:`dataset_profiler.profiler.profile_dataset` and asks a language model to +explain it. Raw vectors are never part of the request: the payload contains +only aggregate metrics, sampling configuration, and bounded non-sensitive +metadata. + +Configuration is explicit and environment-based so that credentials never end +up in generated reports, command output, source code, or logs: + +* ``VSAG_AI_API_KEY`` (fallback: ``OPENAI_API_KEY``) -- bearer token. +* ``VSAG_AI_MODEL`` -- model name required when AI is enabled. +* ``VSAG_AI_BASE_URL`` (fallback: ``OPENAI_BASE_URL``) -- OpenAI-compatible + endpoint root, defaulting to ``https://api.openai.com/v1``. +* ``VSAG_AI_KEY_ENV`` -- name of the environment variable holding the key + (used when a caller wants a custom variable). + +Every finding and recommendation returned by the model must reference metric +keys that exist in the report; responses that do not parse, omit evidence, or +invent evidence keys are rejected. Provider failures never prevent the +deterministic reports from being written: :func:`analyze_report` degrades to a +``{"status": "error", ...}`` entry with a redacted message. +""" + +from __future__ import annotations + +import json +import os +import re +import urllib.error +import urllib.request +from typing import Any, Dict, List, Optional, Sequence + +AI_SCHEMA_VERSION = "1.0" +DEFAULT_BASE_URL = "https://api.openai.com/v1" +DEFAULT_TIMEOUT_SECONDS = 60.0 +MAX_LIST_ITEMS = 10 +MAX_TEXT_LENGTH = 2000 +VALID_CONFIDENCE = ("low", "medium", "high") + +_ERROR_REDACTIONS = ( + re.compile(r"sk-[A-Za-z0-9_\-]{8,}"), + re.compile(r"Bearer\s+[A-Za-z0-9._\-]{8,}"), +) + +# Field allowlists for the aggregate AI payload. Anything not listed here is +# never forwarded to the provider; arbitrary HDF5 object names and per-vector +# fields such as ``clustering.projection_points`` and ``clustering.labels`` are +# excluded. +_DATASET_METADATA_FIELDS = ( + "vector_type", + "dtype", + "count", + "dim", + "dim_inferred", + "distance_metric", + "has_offsets", +) +_AGGREGATE_METRIC_FIELDS: Dict[str, tuple] = { + "invalid_values": ( + "definition", + "scope", + "sample_size", + "count", + "total_values", + "invalid_values", + "vectors_with_invalid", + "invalid_ratio", + "vectors_with_invalid_ratio", + ), + "norms": ( + "definition", + "scope", + "sample_size", + "available", + "reason", + "count", + "stats", + "histogram", + "non_finite_norms", + ), + "outliers": ( + "definition", + "scope", + "sample_size", + "available", + "reason", + "count", + "outlier_count", + "outlier_ratio", + "median_norm", + "mad_norm", + "threshold", + "max_over_median_norm", + "method", + ), + "dimensions": ( + "definition", + "scope", + "sample_size", + "dim", + "summary", + "top_variance_dimensions", + "top_zero_dimensions", + "available", + "reason", + ), + "sparsity": ( + "definition", + "scope", + "sample_size", + "representation", + "count", + "dim", + "cells", + "total_values", + "zero_values", + "non_zero_values", + "stored_entries", + "explicit_zero_entries", + "zero_ratio", + "density", + "nnz_per_vector", + "nnz_histogram", + "nnz_available", + "nnz_reason", + "all_zero_vectors", + "all_zero_vector_ratio", + ), + "duplicates": ( + "definition", + "scope", + "sample_size", + "exact", + "near_duplicate", + "scope_note", + ), + "pairwise_distances": ( + "definition", + "scope", + "sample_size", + "metric", + "pair_count", + "requested_pairwise_sample_size", + "stats", + "histogram", + "matrix_is_bounded", + "dense_sample_materialized", + "sparse_sparse_distance_loops", + ), + "nearest_neighbor_distances": ( + "definition", + "scope", + "sample_size", + "available", + "reason", + "stats", + "histogram", + ), + "anisotropy": ( + "definition", + "scope", + "sample_size", + "available", + "reason", + "method", + "top1_eigenvalue_share", + "top10_eigenvalue_share", + "effective_rank", + "mean_pairwise_cosine", + "explained_variance_ratio", + "interpretation", + ), + "clustering": ( + "definition", + "scope", + "sample_size", + "available", + "reason", + "method", + "explained_variance_ratio_pc1", + "explained_variance_ratio_pc2", + "clustering_available", + "n_clusters", + "cluster_sizes", + "inertia", + "iterations", + ), +} + + +class AIAnalysisError(RuntimeError): + """Raised for configuration, transport, or response-validation failures.""" + + +class _RejectRedirectHandler(urllib.request.HTTPRedirectHandler): + """Reject redirects so bearer credentials cannot reach another endpoint.""" + + def redirect_request(self, req, fp, code, msg, headers, newurl): + raise AIAnalysisError( + "AI provider redirects are disabled; refusing HTTP %s redirect" % code + ) + + +# --------------------------------------------------------------------------- +# Prompt construction +# --------------------------------------------------------------------------- +_SYSTEM_INSTRUCTIONS = ( + "You are interpreting a deterministic statistical profile of a vector " + "dataset produced by the VSAG dataset profiler. You did NOT receive raw " + "vectors; you receive only aggregate metrics. Respond with a single JSON " + "object and nothing else, using exactly this schema:\n" + "{\n" + ' "summary": "",\n' + ' "findings": [{"title": str, "detail": str, "evidence": [metric_key],\n' + ' "confidence": "low"|"medium"|"high"}],\n' + ' "recommendations": [{"title": str, "detail": str, "evidence": [metric_key],\n' + ' "validation": "",\n' + ' "confidence": "low"|"medium"|"high"}],\n' + ' "limitations": [str],\n' + " \"additional_measurements\": [str]\n" + "}\n" + "Rules:\n" + "1. Every finding and recommendation must cite one or more evidence keys " + "from the provided list; never cite keys that are not listed.\n" + "2. Distinguish full-dataset metrics from sampled metrics; sampled " + "conclusions are hypotheses about the sample.\n" + "3. Recommendations must be phrased as hypotheses to validate with " + "reproducible VSAG benchmarks, never as guaranteed optimal " + "configurations.\n" + "4. Report uncertainty honestly in limitations and additional_measurements." +) + + +def evidence_metric_keys(report: Dict[str, Any]) -> List[str]: + """Return dotted ``metrics.*`` keys that evidence references may use.""" + from .stats import metric_keys + + return metric_keys(report) + + +def _resolve_evidence_key(report: Dict[str, Any], key: str) -> bool: + """Return True when ``key`` is a ``metrics.*`` path present in the report.""" + if not key.startswith("metrics."): + return False + node: Any = report.get("metrics", {}) + for part in key.split(".")[1:]: + if isinstance(node, dict) and part in node: + node = node[part] + else: + return False + return True + + +def aggregate_payload(report: Dict[str, Any]) -> Dict[str, Any]: + """Build the bounded, aggregate-only request payload from a report. + + Only aggregate metrics, sampling configuration, warnings, and non-sensitive + metadata are included. The full file path and any per-vector raw content + are never sent. + """ + dataset = report.get("dataset") or {} + return { + "schema_version": report.get("schema_version"), + "tool": report.get("tool"), + "split": report.get("input", {}).get("split"), + "dataset": { + key: dataset[key] for key in _DATASET_METADATA_FIELDS if key in dataset + }, + "config": report.get("config"), + "metrics": _aggregate_metrics(report.get("metrics") or {}), + "warnings": report.get("warnings"), + } + + +def _aggregate_metrics(metrics: Dict[str, Any]) -> Dict[str, Any]: + """Filter a metrics tree down to the aggregate allowlist. + + Per-vector payloads are excluded on purpose: ``clustering.projection_points`` + holds one PCA coordinate pair per sampled vector and ``clustering.labels`` + holds one cluster assignment per sampled vector, which together with the + accompanying summaries can expose individual rows. + """ + payload: Dict[str, Any] = {} + for name, metric in metrics.items(): + allowed = _AGGREGATE_METRIC_FIELDS.get(name) + if allowed is None or not isinstance(metric, dict): + continue + payload[name] = {key: metric[key] for key in allowed if key in metric} + return payload + + +def build_prompt(report: Dict[str, Any]) -> str: + """Render the user prompt sent to the provider (JSON payload + evidence keys).""" + payload = aggregate_payload(report) + keys = evidence_metric_keys(report) + parts = [ + _SYSTEM_INSTRUCTIONS, + "", + "Allowed evidence metric keys:", + json.dumps(keys, sort_keys=True), + "", + "Aggregate profile payload:", + json.dumps(payload, sort_keys=True, allow_nan=False), + ] + return "\n".join(parts) + + +# --------------------------------------------------------------------------- +# Response parsing / validation +# --------------------------------------------------------------------------- +_FENCE_RE = re.compile(r"^```[a-zA-Z]*\s*|\s*```$", re.MULTILINE) + + +def _extract_json_object(text: str) -> Dict[str, Any]: + if not isinstance(text, str) or not text.strip(): + raise AIAnalysisError("empty AI response") + cleaned = _FENCE_RE.sub("", text.strip()) + try: + parsed = json.loads(cleaned) + except json.JSONDecodeError: + start = cleaned.find("{") + end = cleaned.rfind("}") + if start == -1 or end <= start: + raise AIAnalysisError("AI response is not a JSON object") + try: + parsed = json.loads(cleaned[start : end + 1]) + except json.JSONDecodeError as exc: + raise AIAnalysisError("AI response is not valid JSON: %s" % exc) + if not isinstance(parsed, dict): + raise AIAnalysisError("AI response JSON must be an object") + return parsed + + +def _clean_text(value: Any, field: str) -> str: + if not isinstance(value, str) or not value.strip(): + raise AIAnalysisError("AI response field %r must be a non-empty string" % field) + return value.strip()[:MAX_TEXT_LENGTH] + + +def _clean_item( + item: Any, report: Dict[str, Any], field: str, index: int, require_validation: bool +) -> Dict[str, Any]: + where = "%s[%d]" % (field, index) + if not isinstance(item, dict): + raise AIAnalysisError("AI response %s must be an object" % where) + entry: Dict[str, Any] = { + "title": _clean_text(item.get("title"), where + ".title"), + "detail": _clean_text(item.get("detail"), where + ".detail"), + } + evidence = item.get("evidence") + if not isinstance(evidence, list) or not evidence: + raise AIAnalysisError("AI response %s must cite non-empty evidence" % where) + keys: List[str] = [] + for key in evidence: + if not isinstance(key, str) or not _resolve_evidence_key(report, key): + raise AIAnalysisError( + "AI response %s cites unknown evidence key %r" % (where, key) + ) + if key not in keys: + keys.append(key) + entry["evidence"] = keys + confidence = item.get("confidence", "medium") + if confidence not in VALID_CONFIDENCE: + raise AIAnalysisError( + "AI response %s.confidence must be one of %s" + % (where, ", ".join(VALID_CONFIDENCE)) + ) + entry["confidence"] = confidence + if require_validation: + entry["validation"] = _clean_text( + item.get("validation"), where + ".validation" + ) + return entry + + +def _clean_string_list(value: Any, field: str) -> List[str]: + if value is None: + return [] + if not isinstance(value, list) or len(value) > MAX_LIST_ITEMS: + raise AIAnalysisError("AI response %r must be a list of at most %d items" % (field, MAX_LIST_ITEMS)) + return [_clean_text(item, "%s item" % field) for item in value] + + +def parse_and_validate_response(text: str, report: Dict[str, Any]) -> Dict[str, Any]: + """Parse a raw provider response into the validated ``ai_analysis`` block. + + Raises :class:`AIAnalysisError` when the response is malformed, cites + evidence keys that do not exist in the report, or omits required fields. + """ + parsed = _extract_json_object(text) + result: Dict[str, Any] = {"summary": _clean_text(parsed.get("summary"), "summary")} + + for field, require_validation in (("findings", False), ("recommendations", True)): + items = parsed.get(field) + if not isinstance(items, list) or not items: + raise AIAnalysisError("AI response %r must be a non-empty list" % field) + if len(items) > MAX_LIST_ITEMS: + raise AIAnalysisError("AI response %r must have at most %d items" % (field, MAX_LIST_ITEMS)) + result[field] = [ + _clean_item(item, report, field, index, require_validation) + for index, item in enumerate(items) + ] + result["limitations"] = _clean_string_list(parsed.get("limitations"), "limitations") + result["additional_measurements"] = _clean_string_list( + parsed.get("additional_measurements"), "additional_measurements" + ) + return result + + +# --------------------------------------------------------------------------- +# Providers +# --------------------------------------------------------------------------- +class MockProvider: + """Deterministic provider that replays a canned response (tests/demos). + + ``secrets`` implements the same public ``redaction_secrets`` provider + contract as network-backed providers, allowing failure-path tests to verify + redaction without relying on private implementation attributes. + """ + + name = "mock" + + def __init__( + self, + response: str, + model: str = "mock-model", + secrets: str | Sequence[str] = (), + ) -> None: + self._response = response + self.model = model + self.redaction_secrets = ( + (secrets,) if isinstance(secrets, str) else tuple(secrets) + ) + self.last_prompt: Optional[str] = None + + def complete(self, prompt: str) -> str: + self.last_prompt = prompt + return self._response + + +class OpenAICompatibleProvider: + """Calls an OpenAI-compatible ``/chat/completions`` endpoint over HTTPS.""" + + name = "openai-compatible" + + def __init__( + self, + api_key: str, + model: str, + base_url: str = DEFAULT_BASE_URL, + timeout: float = DEFAULT_TIMEOUT_SECONDS, + ) -> None: + if not api_key: + raise AIAnalysisError("AI analysis enabled but no API key was found in the environment") + if any(ord(character) < 33 or ord(character) > 126 for character in api_key): + # HTTP authentication headers require visible ASCII. Reject an + # accidentally retained newline before urllib can include the full + # Authorization value in a transport exception. + raise AIAnalysisError( + "AI API key must contain only visible ASCII characters; remove " + "any accidental whitespace or control characters" + ) + if not model: + raise AIAnalysisError("AI analysis enabled but no model was configured (set VSAG_AI_MODEL or pass --ai-model)") + if not base_url.startswith("https://"): + raise AIAnalysisError( + "AI base URL must use https so the API key is never sent in the clear" + ) + self._api_key = api_key + # Public provider contract: analyze_report redacts these literal values + # from provider exceptions without inspecting private attributes. + self.redaction_secrets = (api_key,) + self.model = model + self.base_url = base_url.rstrip("/") + self.timeout = float(timeout) + self._opener = urllib.request.build_opener(_RejectRedirectHandler()) + + def complete(self, prompt: str) -> str: + request = urllib.request.Request( + self.base_url + "/chat/completions", + data=json.dumps( + { + "model": self.model, + "messages": [{"role": "user", "content": prompt}], + "temperature": 0, + }, + sort_keys=True, + ).encode("utf-8"), + headers={ + "Content-Type": "application/json", + "Authorization": "Bearer " + self._api_key, + }, + method="POST", + ) + try: + with self._opener.open(request, timeout=self.timeout) as response: + body = json.loads(response.read().decode("utf-8")) + except urllib.error.HTTPError as exc: + raise AIAnalysisError( + "AI provider returned HTTP %s for %s" % (exc.code, self.base_url) + ) + except (urllib.error.URLError, TimeoutError, OSError) as exc: + raise AIAnalysisError("AI provider is unreachable: %s" % type(exc).__name__) + except json.JSONDecodeError: + raise AIAnalysisError("AI provider returned a non-JSON response body") + try: + return str(body["choices"][0]["message"]["content"]) + except (KeyError, IndexError, TypeError) as exc: + raise AIAnalysisError("AI provider response is missing choices[0].message.content") + + +def provider_from_env( + env: Optional[Dict[str, str]] = None, + model: Optional[str] = None, + base_url: Optional[str] = None, + key_env: Optional[str] = None, + timeout: float = DEFAULT_TIMEOUT_SECONDS, +) -> OpenAICompatibleProvider: + """Build a provider from the environment; credentials are never persisted.""" + env = os.environ if env is None else env + key_name = key_env or env.get("VSAG_AI_KEY_ENV") or "VSAG_AI_API_KEY" + api_key = env[key_name] if key_name in env else env.get("OPENAI_API_KEY") or "" + resolved_model = model or env.get("VSAG_AI_MODEL") or "" + resolved_base = base_url or env.get("VSAG_AI_BASE_URL") or env.get("OPENAI_BASE_URL") or DEFAULT_BASE_URL + return OpenAICompatibleProvider( + api_key=api_key, + model=resolved_model, + base_url=resolved_base, + timeout=timeout, + ) + + +# --------------------------------------------------------------------------- +# Entry point +# --------------------------------------------------------------------------- +def _redact(message: str, secrets: Sequence[str] = ()) -> str: + # Provider exceptions can contain either the literal header value or its + # repr/JSON-escaped form. Redact the configured credential itself instead + # of relying on a provider-specific token prefix or character set. + variants = set() + for secret in secrets: + if not secret: + continue + candidates = ( + secret, + secret.strip(), + json.dumps(secret, ensure_ascii=True)[1:-1], + repr(secret)[1:-1], + secret.encode("unicode_escape").decode("ascii"), + ) + variants.update(candidate for candidate in candidates if candidate) + for secret in sorted(variants, key=len, reverse=True): + message = message.replace(secret, "[redacted]") + for pattern in _ERROR_REDACTIONS: + message = pattern.sub("[redacted]", message) + return message[:500] + + +def analyze_report( + report: Dict[str, Any], + provider: Optional[Any] = None, + env: Optional[Dict[str, str]] = None, + **provider_kwargs: Any +) -> Dict[str, Any]: + """Produce the ``ai_analysis`` block for ``report``. + + Never raises: configuration, transport, and validation failures are folded + into a ``{"status": "error"}`` block so the deterministic JSON and HTML + reports are still written. Providers document literal credentials for error + redaction through a public ``redaction_secrets`` sequence attribute. + """ + if provider is None: + try: + provider = provider_from_env(env=env, **provider_kwargs) + except AIAnalysisError as exc: + return {"status": "error", "provider": None, "error": _redact(str(exc))} + name = getattr(provider, "name", type(provider).__name__) + model = getattr(provider, "model", None) + configured_secrets = getattr(provider, "redaction_secrets", ()) + if isinstance(configured_secrets, str): + configured_secrets = (configured_secrets,) + secrets = ( + [secret for secret in configured_secrets if isinstance(secret, str)] + if isinstance(configured_secrets, Sequence) + else [] + ) + try: + raw = provider.complete(build_prompt(report)) + block = parse_and_validate_response(raw, report) + except AIAnalysisError as exc: + return { + "status": "error", + "provider": name, + "model": model, + "error": _redact(str(exc), secrets), + } + except Exception as exc: # defensive: providers may raise transport errors + return { + "status": "error", + "provider": name, + "model": model, + "error": _redact( + "unexpected AI failure: %s: %s" % (type(exc).__name__, exc), + secrets, + ), + } + block.update( + { + "status": "ok", + "provider": name, + "model": model, + "ai_schema_version": AI_SCHEMA_VERSION, + } + ) + return block diff --git a/tools/dataset_profiler/dataset_profiler/cli.py b/tools/dataset_profiler/dataset_profiler/cli.py new file mode 100644 index 0000000000..425327258d --- /dev/null +++ b/tools/dataset_profiler/dataset_profiler/cli.py @@ -0,0 +1,208 @@ +# Copyright 2024-present the vsag project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Command-line interface for the VSAG dataset profiler. + +The deterministic JSON and HTML reports are always produced offline. AI +interpretation is opt-in through ``--ai`` (or ``VSAG_DATASET_PROFILER_AI=1``) +and configured exclusively through environment variables so API keys never +appear in arguments, reports, or logs. +""" + +from __future__ import annotations + +import argparse +import os +import sys +from pathlib import Path +from typing import Dict, List, Optional + +from . import __version__ +from .ai import analyze_report +from .hdf5 import DEFAULT_CHUNK_BYTES, DatasetFormatError +from .html_report import render_html +from .profiler import VALID_METRICS, profile_dataset +from .stats import dump_json + +_TRUE_VALUES = ("1", "true", "yes", "on") + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + prog="dataset_profiler", + description=( + "Profile and visualize the feature characteristics of a " + "VSAG-compatible HDF5 dataset (dense or sparse)." + ), + ) + parser.add_argument("input", help="path to the .hdf5 dataset file") + parser.add_argument("--split", default="train", help="dataset split to profile (default: train)") + parser.add_argument("--seed", type=int, default=42, help="explicit random seed (default: 42)") + parser.add_argument( + "--sample-size", + type=int, + default=1024, + help="max vectors drawn for sampled metrics (default: 1024)", + ) + parser.add_argument( + "--pairwise-sample-size", + type=int, + default=512, + help="max vectors used for the bounded pairwise-distance matrix (default: 512)", + ) + parser.add_argument( + "--metric", + choices=VALID_METRICS, + default=None, + help="distance metric for pairwise statistics (default: file 'distance' attribute)", + ) + parser.add_argument( + "--clusters", + type=int, + default=0, + help="optional k-means cluster count on the PCA-2 projection (0 disables)", + ) + parser.add_argument( + "--near-duplicate-epsilon", + type=float, + default=1e-3, + help="relative pairwise-distance threshold for near-duplicate indicators (default: 1e-3)", + ) + parser.add_argument( + "--max-chunk-bytes", + type=int, + default=DEFAULT_CHUNK_BYTES, + help="streaming read chunk size for sparse records (default: 64 MiB)", + ) + parser.add_argument( + "--max-dimension-points", + type=int, + default=1024, + help="max per-dimension points stored per series (default: 1024)", + ) + parser.add_argument( + "--out-json", + metavar="PATH", + default=None, + help="write the JSON report here; pass '-' to print to stdout (default: .profile.json)", + ) + parser.add_argument( + "--out-html", + metavar="PATH", + default=None, + help=( + "write the self-contained HTML report here (default: .profile.html; " + "omitted when --out-json is given without --out-html)" + ), + ) + parser.add_argument( + "--ai", + action="store_true", + help="enable optional AI-assisted interpretation (also enabled by VSAG_DATASET_PROFILER_AI=1)", + ) + parser.add_argument("--ai-model", default=None, help="AI model name (default: $VSAG_AI_MODEL)") + parser.add_argument( + "--ai-base-url", default=None, help="OpenAI-compatible base URL (default: $VSAG_AI_BASE_URL)" + ) + parser.add_argument( + "--ai-key-env", + default=None, + help="name of the environment variable holding the API key (default: $VSAG_AI_KEY_ENV or VSAG_AI_API_KEY)", + ) + parser.add_argument( + "--ai-timeout", type=float, default=60.0, help="AI request timeout in seconds (default: 60)" + ) + parser.add_argument("--version", action="version", version="%(prog)s " + __version__) + return parser + + +def _ai_requested(args: argparse.Namespace, env: Dict[str, str]) -> bool: + if args.ai: + return True + return env.get("VSAG_DATASET_PROFILER_AI", "").strip().lower() in _TRUE_VALUES + + +def _default_output_path(input_path: str, suffix: str) -> Path: + return Path(input_path).with_name(Path(input_path).stem + ".profile." + suffix) + + +def _write_text(path: Path, text: str) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(text, encoding="utf-8") + + +def main(argv: Optional[List[str]] = None, env: Optional[Dict[str, str]] = None) -> int: + env = os.environ if env is None else env + args = build_parser().parse_args(argv) + + try: + report = profile_dataset( + args.input, + split=args.split, + seed=args.seed, + sample_size=args.sample_size, + pairwise_sample_size=args.pairwise_sample_size, + metric=args.metric, + clusters=args.clusters, + near_duplicate_relative_epsilon=args.near_duplicate_epsilon, + max_chunk_bytes=args.max_chunk_bytes, + max_dimension_points=args.max_dimension_points, + ) + except (DatasetFormatError, ValueError, OSError) as exc: + print("error: %s" % exc, file=sys.stderr) + return 1 + + if _ai_requested(args, env): + analysis = analyze_report( + report, + model=args.ai_model, + base_url=args.ai_base_url, + key_env=args.ai_key_env, + timeout=args.ai_timeout, + env=env, + ) + report["ai_analysis"] = analysis + if analysis.get("status") != "ok": + print( + "warning: AI analysis failed (%s); the deterministic JSON and HTML " + "reports were still generated" % analysis.get("error", "unknown error"), + file=sys.stderr, + ) + + for warning in report.get("warnings", []): + print("warning: %s" % warning, file=sys.stderr) + + json_text = dump_json(report) + if args.out_json == "-": + sys.stdout.write(json_text) + else: + json_path = Path(args.out_json) if args.out_json else _default_output_path(args.input, "json") + _write_text(json_path, json_text) + print("wrote %s" % json_path, file=sys.stderr) + + if args.out_html: + html_path = Path(args.out_html) + elif args.out_json is None: + html_path = _default_output_path(args.input, "html") + else: + html_path = None + if html_path is not None: + _write_text(html_path, render_html(report)) + print("wrote %s" % html_path, file=sys.stderr) + + return 0 + + +if __name__ == "__main__": # pragma: no cover + sys.exit(main()) diff --git a/tools/dataset_profiler/dataset_profiler/hdf5.py b/tools/dataset_profiler/dataset_profiler/hdf5.py new file mode 100644 index 0000000000..9c022772a9 --- /dev/null +++ b/tools/dataset_profiler/dataset_profiler/hdf5.py @@ -0,0 +1,490 @@ +# Copyright 2024-present the vsag project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Readers for the HDF5 layouts consumed by VSAG's evaluation tooling. + +Two layouts are supported, matching ``docs/docs/en/src/resources/dataset_format.md``: + +* **dense** (``type`` missing or ``"dense"``): ``/`` is an ``(N, D)`` + matrix of ``INT8`` or ``FLOAT32`` values. One-byte unsigned integers are + numerically converted to ``INT8`` by HDF5 to match the evaluator's + ``ALPHA_I8`` reads (out-of-range values saturate at 127). +* **sparse** (``type="sparse"``): ``/`` is a flat ``INT8`` byte stream. + Each record is ``uint32 len | uint32 ids[len] | float32 vals[len]`` in + little-endian order. ``/_offsets`` is an optional ``UINT64`` array of + record byte offsets with a trailing sentinel. + +The reader deliberately streams data in bounded chunks. Callers never need to +materialize a complete dataset in memory. +""" + +from __future__ import annotations + +import struct +from dataclasses import dataclass, field +from pathlib import Path +from typing import Dict, Iterator, List, Optional, Sequence, Tuple + +import numpy as np + +DEFAULT_CHUNK_BYTES = 64 * 1024 * 1024 +_OFFSET_INDEX_BUDGET_BYTES = DEFAULT_CHUNK_BYTES +_OFFSET_VALIDATION_ROWS = 1024 * 1024 +_LEN_STRUCT = struct.Struct(" Optional[str]: + """Read a scalar string attribute from an HDF5 object.""" + if name not in h5obj.attrs: + return None + value = h5obj.attrs[name] + if isinstance(value, np.ndarray): + if value.size != 1: + raise DatasetFormatError( + "attribute %r is not a scalar string (shape=%s)" % (name, value.shape) + ) + value = value.item() + elif isinstance(value, np.generic): + value = value.item() + if isinstance(value, bytes): + return value.decode("utf-8") + return str(value) + + +class HDF5Dataset: + """Streaming reader for one split of a VSAG-compatible HDF5 dataset.""" + + def __init__(self, path: str, split: str = "train") -> None: + import h5py + + self.path = Path(path) + if not self.path.is_file(): + raise DatasetFormatError("input file does not exist: %s" % self.path) + self.split = split + self._file = h5py.File(str(self.path), "r") + self._dataset = None + self._offsets = None + self._metadata: Optional[DatasetMetadata] = None + try: + self._metadata = self._read_metadata() + except Exception: + self._file.close() + raise + + # -- lifecycle ----------------------------------------------------- + def __enter__(self) -> "HDF5Dataset": + return self + + def __exit__(self, exc_type, exc, tb) -> None: + self.close() + + def close(self) -> None: + if self._file is not None: + self._file.close() + self._file = None + + @property + def metadata(self) -> DatasetMetadata: + assert self._metadata is not None + return self._metadata + + # -- metadata ------------------------------------------------------ + def _read_metadata(self) -> DatasetMetadata: + f = self._file + vector_type = read_str_attr(f, "type") or "dense" + vector_type = vector_type.strip().lower() + if vector_type == "multi_vector": + raise DatasetFormatError( + "multi_vector datasets are not supported by the dataset profiler; " + "profile the underlying flat sub-vector matrix or a dense/sparse export" + ) + if vector_type not in ("dense", "sparse"): + raise DatasetFormatError( + "unsupported HDF5 dataset type %r (expected 'dense' or 'sparse')" + % vector_type + ) + if self.split not in f: + raise DatasetFormatError( + "missing required HDF5 dataset '/%s'" % self.split + ) + + dset = f[self.split] + dtype = _describe_dtype(dset.dtype) + attributes = {} + for key in ("type", "distance", "multi_vector_dim"): + value = read_str_attr(f, key) + if value is not None: + attributes[key] = value + distance = attributes.get("distance") + warnings: List[str] = [] + + count: Optional[int] = None + dim: Optional[int] = None + dim_inferred = False + has_offsets = False + + if vector_type == "dense": + if len(dset.shape) != 2: + raise DatasetFormatError( + "dense dataset '/%s' must be 2-D, got shape %s" + % (self.split, dset.shape) + ) + count, dim = int(dset.shape[0]), int(dset.shape[1]) + if count <= 0: + raise DatasetFormatError("dense dataset '/%s' is empty" % self.split) + if dim <= 0: + raise DatasetFormatError( + "dense dataset '/%s' has non-positive dimensionality" % self.split + ) + else: + if len(dset.shape) != 1: + raise DatasetFormatError( + "sparse dataset '/%s' must be a flat 1-D byte stream, got shape %s" + % (self.split, dset.shape) + ) + offsets_name = self.split + "_offsets" + if offsets_name in f: + offsets_dset = f[offsets_name] + offset_count = _validate_offsets( + offsets_dset, int(dset.shape[0]), offsets_name + ) + count = offset_count - 1 + has_offsets = True + offset_bytes = offset_count * np.dtype(np.uint64).itemsize + if offset_bytes <= _OFFSET_INDEX_BUDGET_BYTES: + self._offsets = np.asarray(offsets_dset[()], dtype=np.uint64) + else: + warnings.append( + "sparse '/%s' exceeds the %d-byte retained-index budget; " + "using the bounded sequential decoder" + % (offsets_name, _OFFSET_INDEX_BUDGET_BYTES) + ) + else: + warnings.append( + "sparse dataset has no '/%s_offsets'; records are counted by scanning" + % self.split + ) + if count == 0: + raise DatasetFormatError("sparse dataset '/%s' is empty" % self.split) + dim_inferred = True + warnings.append( + "sparse dimensionality is inferred as max(feature id) + 1 during profiling" + ) + + return DatasetMetadata( + path=self.path, + file_name=self.path.name, + file_size_bytes=int(self.path.stat().st_size), + vector_type=vector_type, + split=self.split, + dtype=dtype, + count=count, + dim=dim, + dim_inferred=dim_inferred, + distance=distance, + has_offsets=has_offsets, + datasets=sorted(str(name) for name in f.keys()), + attributes=attributes, + warnings=warnings, + ) + + # -- dense --------------------------------------------------------- + def iter_dense_blocks( + self, block_rows: int = 65536 + ) -> Iterator[Tuple[int, np.ndarray]]: + """Yield ``(start_row, block)`` slices of the dense matrix.""" + if self.metadata.vector_type != "dense": + raise DatasetFormatError("iter_dense_blocks requires a dense dataset") + if block_rows <= 0: + raise ValueError("block_rows must be positive") + dset = self._file[self.split] + total = int(dset.shape[0]) + for start in range(0, total, block_rows): + end = min(start + block_rows, total) + yield start, _read_evaluator_dense(dset, slice(start, end)) + + def read_dense_rows(self, indices: Sequence[int]) -> np.ndarray: + """Read selected rows from a dense matrix (indices must be sorted).""" + if self.metadata.vector_type != "dense": + raise DatasetFormatError("read_dense_rows requires a dense dataset") + idx = np.asarray(indices, dtype=np.int64) + if idx.size == 0: + return np.empty((0, int(self.metadata.dim or 0)), dtype=np.float64) + if np.any(idx[1:] <= idx[:-1]): + raise ValueError("dense row indices must be strictly increasing") + dset = self._file[self.split] + return _read_evaluator_dense(dset, idx.tolist()) + + # -- sparse -------------------------------------------------------- + def iter_sparse_records( + self, max_chunk_bytes: int = DEFAULT_CHUNK_BYTES + ) -> Iterator[Tuple[int, np.ndarray, np.ndarray]]: + """Yield ``(index, ids, values)`` for every sparse vector in the split.""" + if self.metadata.vector_type != "sparse": + raise DatasetFormatError("iter_sparse_records requires a sparse dataset") + if max_chunk_bytes < 16: + raise ValueError("max_chunk_bytes is too small") + if self._offsets is not None: + yield from self._iter_sparse_with_offsets(max_chunk_bytes) + else: + # With no index, or with an index too large to retain, decode the + # byte stream directly. An oversized index still supplies record + # boundaries and the count, which the decoder cross-checks in windows. + yield from self._iter_sparse_without_offsets( + max_chunk_bytes, + expected_count=self.metadata.count if self.metadata.has_offsets else None, + validate_offsets=self.metadata.has_offsets, + ) + + def _iter_sparse_with_offsets( + self, max_chunk_bytes: int + ) -> Iterator[Tuple[int, np.ndarray, np.ndarray]]: + dset = self._file[self.split] + offsets = self._offsets + assert offsets is not None + total_records = len(offsets) - 1 + start = 0 + while start < total_records: + limit = int(offsets[start]) + int(max_chunk_bytes) + end = int(np.searchsorted(offsets, limit, side="right")) - 1 + if end <= start: + end = start + 1 + end = min(end, total_records) + byte_start = int(offsets[start]) + byte_end = int(offsets[end]) + chunk = np.asarray(dset[byte_start:byte_end]) + view = memoryview(chunk).cast("B") + pos = 0 + for index in range(start, end): + record_size = int(offsets[index + 1]) - int(offsets[index]) + ids, values, consumed = _decode_one_record(view, pos, record_size) + yield index, ids, values + pos += consumed + if pos != view.nbytes: + raise DatasetFormatError( + "sparse byte stream is inconsistent with '/%s_offsets'" % self.split + ) + start = end + + def _iter_sparse_without_offsets( + self, + max_chunk_bytes: int, + expected_count: Optional[int] = None, + validate_offsets: bool = False, + ) -> Iterator[Tuple[int, np.ndarray, np.ndarray]]: + dset = self._file[self.split] + offset_dset = self._file[self.split + "_offsets"] if validate_offsets else None + offset_window = np.empty(0, dtype=np.uint64) + offset_window_start = 0 + total_bytes = int(dset.shape[0]) + buffer = bytearray() + index = 0 + position = 0 + while position < total_bytes: + end = min(position + max_chunk_bytes, total_bytes) + buffer.extend(np.asarray(dset[position:end]).tobytes()) + position = end + offset = 0 + while True: + if offset + _LEN_SIZE > len(buffer): + break + (length,) = _LEN_STRUCT.unpack_from(buffer, offset) + record_size = _LEN_SIZE + 8 * int(length) + record_start = position - len(buffer) + offset + remaining = total_bytes - record_start + if record_size > remaining: + raise DatasetFormatError( + "sparse record is truncated: len=%d requires %d byte(s), " + "but only %d byte(s) remain" + % (int(length), record_size, remaining) + ) + if offset + record_size > len(buffer): + break + ids, values, consumed = _decode_one_record(buffer, offset, record_size) + if offset_dset is not None: + # Keep only one bounded offset window, but check both boundaries + # of every decoded record before exposing it to the caller. + if index + 1 >= offset_window_start + offset_window.size: + offset_window_start = ( + index // _OFFSET_VALIDATION_ROWS + ) * _OFFSET_VALIDATION_ROWS + offset_window_end = min( + offset_window_start + _OFFSET_VALIDATION_ROWS + 1, + int(offset_dset.shape[0]), + ) + offset_window = np.asarray( + offset_dset[offset_window_start:offset_window_end], + dtype=np.uint64, + ) + window_index = index - offset_window_start + stored_start = int(offset_window[window_index]) + stored_end = int(offset_window[window_index + 1]) + decoded_end = record_start + consumed + if record_start != stored_start or decoded_end != stored_end: + raise DatasetFormatError( + "sparse decoded record %d has byte range [%d, %d), but " + "'/%s_offsets' contains [%d, %d)" + % ( + index, + record_start, + decoded_end, + self.split, + stored_start, + stored_end, + ) + ) + yield index, ids, values + index += 1 + offset += consumed + del buffer[:offset] + if buffer: + raise DatasetFormatError( + "sparse byte stream has %d trailing byte(s) that do not form a " + "complete record" % len(buffer) + ) + if expected_count is not None and index != expected_count: + raise DatasetFormatError( + "sparse decoded record count (%d) does not match '/%s_offsets' (%d)" + % (index, self.split, expected_count) + ) + + def read_sparse_rows( + self, indices: Sequence[int], max_chunk_bytes: int = DEFAULT_CHUNK_BYTES + ) -> Dict[int, Tuple[np.ndarray, np.ndarray]]: + """Read selected sparse vectors into a mapping ``index -> (ids, values)``.""" + wanted = set(int(i) for i in indices) + if not wanted: + return {} + out: Dict[int, Tuple[np.ndarray, np.ndarray]] = {} + for index, ids, values in self.iter_sparse_records(max_chunk_bytes): + if index in wanted: + out[index] = (ids.copy(), values.copy()) + if len(out) == len(wanted): + break + missing = wanted.difference(out) + if missing: + raise DatasetFormatError( + "sparse dataset ended before sampled indices: %s" + % sorted(missing)[:5] + ) + return out + + +def _read_evaluator_dense(dset, selection) -> np.ndarray: + """Read dense values with the evaluator's HDF5 destination datatype.""" + dtype = np.dtype(dset.dtype) + if dtype.kind == "u" and dtype.itemsize == 1: + # ALPHA_I8 is a signed destination type. HDF5 performs a numeric + # conversion and saturates uint8 values above 127; viewing the bytes + # would instead wrap them to negative values. + return np.asarray(dset.astype(np.int8)[selection]) + return np.asarray(dset[selection]) + + +def _validate_offsets(dset, total_bytes: int, name: str) -> int: + """Validate an on-disk offset index using bounded read windows.""" + if len(dset.shape) != 1: + raise DatasetFormatError("'/%s' must be a 1-D offset array" % name) + count = int(dset.shape[0]) + if count < 1: + raise DatasetFormatError( + "'/%s' must contain at least the trailing sentinel" % name + ) + first = int(np.asarray(dset[0], dtype=np.uint64)) + last = int(np.asarray(dset[count - 1], dtype=np.uint64)) + if first != 0: + raise DatasetFormatError("'/%s' must start at byte offset 0" % name) + if last != int(total_bytes): + raise DatasetFormatError( + "'/%s' sentinel (%d) does not match the sparse byte stream size (%d)" + % (name, last, int(total_bytes)) + ) + previous = first + for start in range(1, count, _OFFSET_VALIDATION_ROWS): + end = min(start + _OFFSET_VALIDATION_ROWS, count) + window = np.asarray(dset[start:end], dtype=np.uint64) + if int(window[0]) < previous or np.any(window[1:] < window[:-1]): + raise DatasetFormatError("'/%s' must be non-decreasing" % name) + previous = int(window[-1]) + return count + + +def _describe_dtype(dtype: np.dtype) -> str: + dtype = np.dtype(dtype) + if dtype.kind in ("i", "u") and dtype.itemsize == 1: + return "INT8" + if dtype.kind == "f" and dtype.itemsize == 4: + return "FLOAT32" + raise DatasetFormatError( + "unsupported HDF5 element type %r; expected INT8 or FLOAT32" % dtype + ) + + +def _decode_one_record( + buffer, offset: int, expected_size: Optional[int] = None +) -> Tuple[np.ndarray, np.ndarray, int]: + """Decode one sparse record starting at ``offset``. + + Returns ``(ids, values, consumed_bytes)``. ``expected_size`` is supplied + when byte offsets are available and must match the encoded record exactly. + """ + view = memoryview(buffer).cast("B") + if offset + _LEN_SIZE > view.nbytes: + raise DatasetFormatError("sparse record is truncated before its length header") + (length,) = _LEN_STRUCT.unpack_from(view, offset) + record_size = _LEN_SIZE + 8 * int(length) + if expected_size is not None and record_size != expected_size: + raise DatasetFormatError( + "sparse record size mismatch: offsets say %d byte(s), record header " + "requires %d" % (expected_size, record_size) + ) + if offset + record_size > view.nbytes: + raise DatasetFormatError( + "sparse record is truncated: len=%d requires %d byte(s)" + % (int(length), record_size) + ) + ids_start = offset + _LEN_SIZE + vals_start = ids_start + 4 * int(length) + if length: + ids = np.frombuffer(view, dtype=" str: + """Render a complete standalone HTML document for ``report``.""" + metrics = report.get("metrics", {}) + dataset = report.get("dataset", {}) + config = report.get("config", {}) + file_name = _report_file_name(report) + body: List[str] = [] + body.append(_header(report, dataset, config)) + body.append(_overview_cards(report, metrics, dataset)) + body.append(_norms_section(metrics.get("norms", {}))) + body.append(_dimensions_section(metrics.get("dimensions", {}))) + body.append(_sparsity_section(metrics.get("sparsity", {}))) + body.append(_duplicates_section(metrics.get("duplicates", {}))) + body.append(_distance_sections(metrics)) + body.append(_anisotropy_section(metrics.get("anisotropy", {}))) + body.append(_clustering_section(metrics.get("clustering", {}))) + body.append(_ai_section(report.get("ai_analysis"))) + body.append(_warnings_section(report.get("warnings", []))) + body.append(_footer()) + + return ( + "\n" + '\n\n' + '\n' + '\n' + '\n' + "VSAG dataset profile: %s\n" + "\n" + "\n\n" + '
\n' + "%s\n" + "\n" + "\n\n" + ) % ( + _e(file_name), + _CSS, + "\n".join(part for part in body if part), + _JS, + ) + + +def _report_file_name(report: Dict[str, Any]) -> str: + """Return the input file name stored under ``report.input``.""" + return str(report.get("input", {}).get("file_name") or "dataset") + + +# --------------------------------------------------------------------------- +# Page sections +# --------------------------------------------------------------------------- +def _header(report: Dict[str, Any], dataset: Dict[str, Any], config: Dict[str, Any]) -> str: + title = "Dataset Feature Profile" + subtitle = "%s · %s vectors · %s dimensions · %s" % ( + _e(_report_file_name(report)), + _fmt(dataset.get("count")), + _fmt(dataset.get("dim")), + _e(str(dataset.get("vector_type", "dense")).upper()), + ) + meta = ( + 'schema %s' + 'seed %s' + 'sample %s' + 'pairwise sample %s' + 'metric %s' + % ( + _e(str(report.get("schema_version"))), + _e(str(config.get("seed"))), + _fmt(config.get("effective_sample_size")), + _fmt(config.get("effective_pairwise_sample_size")), + _e(str(config.get("metric"))), + ) + ) + return ( + '
' + "

%s

%s

" + '
%s
' + '

Deterministic offline report. No raw vectors were sent anywhere ' + "while generating this file.

" % (title, subtitle, meta) + ) + + +def _overview_cards( + report: Dict[str, Any], metrics: Dict[str, Any], dataset: Dict[str, Any] +) -> str: + config = report.get("config", {}) + norms = metrics.get("norms", {}).get("stats", {}) + sparsity = metrics.get("sparsity", {}) + duplicates = metrics.get("duplicates", {}).get("exact", {}) + outliers = metrics.get("outliers", {}) + anisotropy = metrics.get("anisotropy", {}) + invalid = metrics.get("invalid_values", {}) + cards = [ + ("Vectors", _fmt(dataset.get("count")), "full dataset"), + ( + "Dimensions", + _fmt(dataset.get("dim")) + (" (inferred)" if dataset.get("dim_inferred") else ""), + "full dataset", + ), + ("Element type", _e(str(dataset.get("dtype"))), "full dataset"), + ( + "Distance metric", + _e(str(config.get("metric") or dataset.get("distance_metric") or "n/a")), + "profiling metric", + ), + ("Norm median", _fmt(norms.get("p50")), "full dataset"), + ("Zero ratio", _pct(sparsity.get("zero_ratio")), "full dataset"), + ( + "Exact duplicate ratio", + _pct(duplicates.get("duplicate_ratio")), + "full dataset", + ), + ("Invalid values", _fmt(invalid.get("invalid_values")), "full dataset"), + ("Norm outlier ratio", _pct(outliers.get("outlier_ratio")), "full dataset"), + ( + "Top eigenvalue share", + _fmt(anisotropy.get("top1_eigenvalue_share")), + "sample", + ), + ] + items = "".join( + '
%s
' + '
%s
%s
' + % (label, value, scope) + for label, value, scope in cards + ) + return '

Overview

%s
' % items + + +def _norms_section(metric: Dict[str, Any]) -> str: + if not metric: + return "" + if metric.get("available") is False: + return _section( + "Vector norms", + metric, + '

%s

' % _e(metric.get("reason", "unavailable")), + ) + return _section( + "Vector norms", + metric, + _chart_histogram(metric.get("histogram", {}), "norm", "L2 norm", "Vectors"), + ) + + +def _dimensions_section(metric: Dict[str, Any]) -> str: + if not metric: + return "" + per_dim = metric.get("per_dimension", {}) + charts = [ + _chart_line( + per_dim.get("mean", {}), + "dimension mean", + "Coordinate", + "Mean", + "#2563eb", + ), + _chart_line( + per_dim.get("std", {}), + "dimension standard deviation", + "Coordinate", + "Std", + "#dc2626", + ), + _chart_line( + per_dim.get("zero_ratio", {}), + "dimension zero ratio", + "Coordinate", + "Zero ratio", + "#059669", + ), + ] + table = _top_dimension_table(metric) + return _section( + "Per-dimension distributions", + metric, + "".join(charts) + table, + extra_note=_downsample_note(per_dim), + ) + + +def _sparsity_section(metric: Dict[str, Any]) -> str: + if not metric: + return "" + total_cells = ( + metric.get("cells") if "cells" in metric else metric.get("total_values") + ) + # Sparse zero_values includes implicit zeros; explicit_zero_entries is only + # a legacy fallback. Checking key presence also preserves a dense count 0. + zero_values = ( + metric.get("zero_values") + if "zero_values" in metric + else metric.get("explicit_zero_entries") + ) + rows = [ + ("Representation", str(metric.get("representation"))), + ("Total cells", _fmt(total_cells)), + ("Zero values", _fmt(zero_values)), + ("Non-zero values", _fmt(metric.get("non_zero_values"))), + ("Density", _pct(metric.get("density"))), + ("Zero ratio", _pct(metric.get("zero_ratio"))), + ("All-zero vectors", _fmt(metric.get("all_zero_vectors"))), + ] + if metric.get("representation") == "sparse": + rows.append(("Stored entries", _fmt(metric.get("stored_entries")))) + charts = "" + if metric.get("nnz_histogram"): + charts = _chart_histogram( + metric["nnz_histogram"], "nnz", "Non-zero entries per vector", "Vectors" + ) + return _section( + "Sparsity", + metric, + charts, + extra_note=_table(rows), + ) + + +def _duplicates_section(metric: Dict[str, Any]) -> str: + if not metric: + return "" + exact = metric.get("exact", {}) + near = metric.get("near_duplicate", {}) + rows = [ + ("Unique vectors", _fmt(exact.get("unique_vectors"))), + ("Duplicate vectors (exact)", _fmt(exact.get("duplicate_vectors"))), + ("Duplicate ratio (exact)", _pct(exact.get("duplicate_ratio"))), + ("Duplicate groups", _fmt(exact.get("duplicate_groups"))), + ("Largest duplicate group", _fmt(exact.get("max_group_size"))), + ] + near_rows = [ + ("Near-duplicate sampled pairs", _fmt(near.get("near_duplicate_pairs"))), + ("Near-duplicate pair ratio", _pct(near.get("near_duplicate_ratio"))), + ("Threshold", _fmt(near.get("threshold"))), + ("Median sampled pairwise distance", _fmt(near.get("median_pairwise_distance"))), + ("Pairs sampled", _fmt(near.get("pair_count"))), + ] + scope_note = metric.get("scope_note", "") + return _section( + "Duplicate and near-duplicate indicators", + metric, + _table(rows) + "

Near-duplicate (sampled)

" + _table(near_rows), + extra_note=_e(scope_note), + ) + + +def _distance_sections(metrics: Dict[str, Any]) -> str: + pairwise = metrics.get("pairwise_distances", {}) + nearest = metrics.get("nearest_neighbor_distances", {}) + parts: List[str] = [] + if pairwise: + parts.append( + _section( + "Pairwise distance distribution", + pairwise, + _chart_histogram( + pairwise.get("histogram", {}), + "pairwise distance", + "Distance between sampled pairs", + "Pairs", + ) + + _metric_stats_table(pairwise.get("stats", {})), + extra_note="The distance matrix is bounded by the configured pairwise sample size.", + ) + ) + if nearest and nearest.get("available"): + parts.append( + _section( + "Nearest-neighbor distance distribution", + nearest, + _chart_histogram( + nearest.get("histogram", {}), + "nn distance", + "Distance to nearest other sampled vector", + "Vectors", + ) + + _metric_stats_table(nearest.get("stats", {})), + ) + ) + return "".join(parts) + + +def _anisotropy_section(metric: Dict[str, Any]) -> str: + if not metric: + return "" + if not metric.get("available"): + return _section( + "Anisotropy", metric, '

%s

' % _e(metric.get("reason", "unavailable")) + ) + ratios = metric.get("explained_variance_ratio", []) or [] + charts = _chart_bars( + ["PC%d" % (index + 1) for index in range(len(ratios))], + ratios, + "PC variance share", + "Explained variance ratio of leading principal components", + "#7c3aed", + ) + rows = [ + ("Top-1 eigenvalue share", _fmt(metric.get("top1_eigenvalue_share"))), + ("Top-10 eigenvalue share", _fmt(metric.get("top10_eigenvalue_share"))), + ("Effective rank", _fmt(metric.get("effective_rank"))), + ("Mean pairwise cosine", _fmt(metric.get("mean_pairwise_cosine"))), + ] + return _section( + "Anisotropy", + metric, + charts, + extra_note=_table(rows) + '

%s

' % _e(metric.get("interpretation", "")), + ) + + +def _clustering_section(metric: Dict[str, Any]) -> str: + if not metric: + return "" + if not metric.get("available"): + return _section( + "Projection and clustering", + metric, + '

%s

' % _e(metric.get("reason", "unavailable")), + ) + points = metric.get("projection_points", []) or [] + labels = metric.get("labels") if metric.get("clustering_available") else None + charts = _chart_scatter(points, labels, "PCA-2 projection of the seeded sample") + if metric.get("clustering_available"): + sizes = metric.get("cluster_sizes", []) or [] + charts += _chart_bars( + ["C%d" % index for index in range(len(sizes))], + sizes, + "Cluster size", + "k-means cluster sizes on the PCA-2 projection", + "#0891b2", + ) + charts += _table( + [ + ("Clusters", _fmt(metric.get("n_clusters"))), + ("Inertia", _fmt(metric.get("inertia"))), + ("Iterations", _fmt(metric.get("iterations"))), + ] + ) + else: + charts += '

%s

' % _e(metric.get("reason", "clustering disabled")) + return _section("Projection and clustering", metric, charts) + + +def _ai_section(analysis: Optional[Dict[str, Any]]) -> str: + if not analysis: + return ( + '

AI-assisted interpretation

' + '

Disabled. The deterministic JSON and HTML reports were ' + "generated without network access.

" + ) + status = analysis.get("status") + if status != "ok": + return ( + '

AI-assisted interpretation

' + '
AI analysis unavailable. ' + "The deterministic reports are still complete. %s
" + '

Reason: %s

' + % ( + _e(str(analysis.get("provider") or "provider")), + _e(str(analysis.get("error") or "unknown error")), + ) + ) + findings = analysis.get("findings", []) or [] + recommendations = analysis.get("recommendations", []) or [] + parts = [ + '

AI-assisted interpretation

', + '

Provider: %s · model: %s. AI output is derived only from the ' + "aggregate metrics in this report. Recommendations are hypotheses to validate " + "with reproducible VSAG benchmarks, not guaranteed optimal configurations.

" + % (_e(str(analysis.get("provider"))), _e(str(analysis.get("model")))), + ] + if analysis.get("summary"): + parts.append('

%s

' % _e(str(analysis["summary"]))) + parts.append("

Findings

") + if findings: + for item in findings: + parts.append(_ai_item(item, "Finding")) + else: + parts.append('

No findings returned.

') + parts.append("

Recommendations to validate

") + if recommendations: + for item in recommendations: + parts.append(_ai_item(item, "Hypothesis")) + else: + parts.append('

No recommendations returned.

') + for key, title in ( + ("limitations", "Limitations"), + ("additional_measurements", "Additional measurements needed"), + ): + values = analysis.get(key, []) or [] + if values: + parts.append("

%s

    %s
" % (title, "".join("
  • %s
    • " % _e(str(value)) for value in values))) + parts.append("
    ") + return "".join(parts) + + +def _ai_item(item: Dict[str, Any], label: str) -> str: + evidence = item.get("evidence", []) or [] + chips = "".join('%s' % _e(str(key)) for key in evidence) + validation = item.get("validation") + extra = "" + if validation: + extra = '

    Validation: %s

    ' % _e(str(validation)) + confidence = item.get("confidence") + if confidence: + extra += 'confidence: %s' % _e(str(confidence)) + return ( + '

    %s %s

    ' + "

    %s

    %s
    %s
    " + % (label, _e(str(item.get("title", ""))), _e(str(item.get("detail", ""))), chips, extra) + ) + + +def _warnings_section(warnings: Sequence[str]) -> str: + if not warnings: + return "" + items = "".join("
  • %s
    • " % _e(str(warning)) for warning in warnings) + return '

    Warnings

      %s
    ' % items + + +def _footer() -> str: + return ( + '

    Generated by vsag-dataset-profiler. Deterministic JSON and ' + "HTML outputs are produced offline. When AI analysis is enabled, only the bounded " + "aggregate metric payload described in the technical report is sent to the configured " + "provider; raw vectors and credentials are never included.

    " + ) + + +# --------------------------------------------------------------------------- +# Layout helpers +# --------------------------------------------------------------------------- +def _section( + title: str, + metric: Dict[str, Any], + chart_html: str, + extra_note: str = "", +) -> str: + scope = str(metric.get("scope", "full")) + sample_size = metric.get("sample_size") + sample_text = _fmt(sample_size) if sample_size is not None else "n/a" + definition = metric.get("definition", "") + note = '

    %s

    ' % extra_note if extra_note else "" + return ( + '

    %s

    ' + '
    Scope: %s' + 'Sample size: %s
    ' + '

    Definition: %s

    ' + "%s%s
    " + % (_e(title), _e(scope), _e(scope), _e(sample_text), _e(str(definition)), chart_html, note) + ) + + +def _table(rows: Iterable[Sequence[Any]]) -> str: + body = "".join( + "%s%s" % (_e(str(row[0])), _e(str(row[1]))) + for row in rows + if len(row) >= 2 + ) + return '%s
    ' % body + + +def _metric_stats_table(stats: Dict[str, Any]) -> str: + if not stats: + return "" + keys = ("count", "mean", "std", "min", "p1", "p5", "p25", "p50", "p75", "p95", "p99", "max") + return _table((key, _fmt(stats.get(key))) for key in keys if key in stats) + + +def _top_dimension_table(metric: Dict[str, Any]) -> str: + rows = metric.get("top_variance_dimensions", []) or [] + if not rows: + return "" + body = "".join( + "%d%s%s%s" + % ( + int(row.get("dimension", 0)), + _fmt(row.get("mean")), + _fmt(row.get("std")), + _pct(row.get("zero_ratio")), + ) + for row in rows + ) + return ( + "

    Highest-variance dimensions

    " + '' + "%s
    DimensionMeanStdZero ratio
    " % body + ) + + +def _downsample_note(per_dim: Dict[str, Any]) -> str: + reduced = any( + isinstance(view, dict) and view.get("downsampled") + for view in per_dim.values() + ) + if reduced: + return ( + "Some per-dimension series were bucket-averaged for display; the report " + "records source_length and bucket_size for each series." + ) + return "" + + +# --------------------------------------------------------------------------- +# SVG charts +# --------------------------------------------------------------------------- +def _chart_histogram( + data: Dict[str, Any], prefix: str, xlabel: str, ylabel: str +) -> str: + bins = data.get("bins") or [] + counts = data.get("counts") or [] + if len(bins) < 2 or not counts: + return '

    No finite values available for this chart.

    ' + width, height = 720, 280 + left, right, top, bottom = 56, 16, 18, 48 + plot_width = width - left - right + plot_height = height - top - bottom + max_count = max(counts) or 1 + bar_width = plot_width / float(len(counts)) + parts = [_svg_open(width, height)] + parts.append(_axes(left, top, plot_width, plot_height, ylabel)) + for index, count in enumerate(counts): + bar_height = (count / float(max_count)) * plot_height + x = left + index * bar_width + y = top + plot_height - bar_height + low, high = bins[index], bins[index + 1] + tip = "%s %.6g – %.6g: %d" % (prefix, low, high, count) + parts.append( + '%s' + % (x, y, max(bar_width - 1.0, 1.0), bar_height, _e(tip), _e(tip)) + ) + parts.append(_x_axis(left, top + plot_height, plot_width, bins[0], bins[-1], xlabel)) + parts.append(_axis_labels(left, top, plot_width, plot_height, max_count)) + parts.append("") + return "".join(parts) + + +def _chart_line( + data: Dict[str, Any], + title: str, + xlabel: str, + ylabel: str, + color: str, +) -> str: + values = data.get("values") if isinstance(data, dict) else None + if not values: + return '

    No per-dimension data available.

    ' + pairs = [(index, float(value)) for index, value in enumerate(values) if value is not None] + if len(pairs) < 2: + return '

    Not enough per-dimension points for %s.

    ' % _e(title) + pairs = _thin(pairs, _MAX_SVG_POINTS) + downsampled = bool(data.get("downsampled")) + source_length = int(data.get("source_length") or len(values)) + # Match stats.downsample's integer-linspace partition exactly. A single + # ceiling bucket size is wrong when source_length is not divisible by the + # number of output points. + boundaries = np.linspace( + 0, source_length, len(values) + 1, dtype=np.int64 + ) + + def source_range(index: int): + if not downsampled: + return index, index + start = boundaries[index] + end = boundaries[index + 1] - 1 + return start, max(start, end) + + def x_position(index: int) -> float: + start, end = source_range(index) + return (start + end) / 2.0 + + width, height = 720, 230 + left, right, top, bottom = 56, 16, 18, 44 + plot_width = width - left - right + plot_height = height - top - bottom + y_values = [value for _, value in pairs] + y_min, y_max = min(y_values), max(y_values) + if not y_max > y_min: + # Pad by a representable fraction of the magnitude: adding 1.0 to a + # value near 1e20 leaves the float64 bound unchanged. + magnitude = max(abs(y_max), abs(y_min)) + pad = max(magnitude * 1e-3, math.ulp(magnitude) * 4.0, 1e-12) + y_min -= pad + y_max += pad + x_max = max(source_range(index)[1] for index, _ in pairs) or 1 + + def sx(index: float) -> float: + return left + (index / float(x_max)) * plot_width + + def sy(value: float) -> float: + return top + plot_height - ((value - y_min) / (y_max - y_min)) * plot_height + + points = " ".join( + "%.2f,%.2f" % (sx(x_position(index)), sy(value)) for index, value in pairs + ) + parts = [_svg_open(width, height)] + parts.append(_axes(left, top, plot_width, plot_height, ylabel)) + parts.append( + '' + % (color, points) + ) + for index, value in pairs: + start, end = source_range(index) + if start == end: + tip = "dimension %d: %.6g" % (start, value) + else: + tip = "dimensions %d-%d: %.6g" % (start, end, value) + parts.append( + '%s' + % (sx(x_position(index)), sy(value), color, _e(tip), _e(tip)) + ) + parts.append(_x_axis(left, top + plot_height, plot_width, 0, x_max, xlabel)) + parts.append(_axis_labels(left, top, plot_width, plot_height, y_max, y_min)) + parts.append("") + caption = ( + '

    %s; %d plotted point(s)%s.

    ' + % ( + _e(title), + len(pairs), + "; bucket-averaged" if data.get("downsampled") else "", + ) + ) + return caption + "".join(parts) + + +def _chart_bars( + categories: Sequence[str], + values: Sequence[Any], + title: str, + ylabel: str, + color: str, +) -> str: + pairs = [ + (str(category), float(value)) + for category, value in zip(categories, values) + if value is not None + ] + if not pairs: + return '

    No data available for %s.

    ' % _e(title) + width, height = 720, 230 + left, right, top, bottom = 56, 16, 18, 46 + plot_width = width - left - right + plot_height = height - top - bottom + max_value = max(value for _, value in pairs) + if max_value <= 0: + max_value = 1.0 + bar_width = plot_width / float(len(pairs)) + parts = [_svg_open(width, height), _axes(left, top, plot_width, plot_height, ylabel)] + for index, (category, value) in enumerate(pairs): + bar_height = (value / max_value) * plot_height + x = left + index * bar_width + y = top + plot_height - bar_height + tip = "%s: %.6g" % (category, value) + parts.append( + '%s' + % (x, y, max(bar_width - 2.0, 1.0), bar_height, color, _e(tip), _e(tip)) + ) + parts.append( + '%s' + % (x + bar_width / 2.0, top + plot_height + 18, _e(category)) + ) + parts.append(_axis_labels(left, top, plot_width, plot_height, max_value)) + parts.append("") + return '

    %s.

    ' % _e(title) + "".join(parts) + + +def _chart_scatter( + points: Sequence[Sequence[Any]], labels: Optional[Sequence[int]], title: str +) -> str: + valid = [ + (float(point[0]), float(point[1]), index) + for index, point in enumerate(points) + if len(point) >= 2 and point[0] is not None and point[1] is not None + ] + if len(valid) < 2: + return '

    Not enough projected points for a scatter plot.

    ' + width, height = 720, 360 + left, right, top, bottom = 56, 20, 18, 40 + plot_width = width - left - right + plot_height = height - top - bottom + xs = [point[0] for point in valid] + ys = [point[1] for point in valid] + x_min, x_max = min(xs), max(xs) + y_min, y_max = min(ys), max(ys) + if math.isclose(x_min, x_max): + x_min -= 1.0 + x_max += 1.0 + if math.isclose(y_min, y_max): + y_min -= 1.0 + y_max += 1.0 + parts = [_svg_open(width, height)] + parts.append(_axes(left, top, plot_width, plot_height, "PC2")) + for x, y, index in valid: + px = left + ((x - x_min) / (x_max - x_min)) * plot_width + py = top + plot_height - ((y - y_min) / (y_max - y_min)) * plot_height + cluster = labels[index] if labels is not None and index < len(labels) else 0 + color = _PALETTE[int(cluster) % len(_PALETTE)] + tip = "sample %d · cluster %s · (%.4g, %.4g)" % (index, cluster, x, y) + parts.append( + '%s' + % (px, py, color, _e(tip), _e(tip)) + ) + parts.append( + 'PC1' + % (left + plot_width / 2.0, height - 8) + ) + parts.append("") + return '

    %s.

    ' % _e(title) + "".join(parts) + + +def _svg_open(width: int, height: int) -> str: + return ( + '' % (width, height) + ) + + +def _axes(left: int, top: int, width: int, height: int, ylabel: str) -> str: + return ( + '' + '' + '' + '%s' + % ( + left, + top, + width, + height, + left, + top, + left, + top + height, + left, + top + height, + left + width, + top + height, + top + height / 2.0, + _e(ylabel), + ) + ) + + +def _x_axis( + left: int, y: int, width: int, minimum: float, maximum: float, label: str +) -> str: + return ( + '%s' + '%s' + '%s' + % ( + left, + y + 16, + _fmt(minimum), + left + width, + y + 16, + _fmt(maximum), + left + width / 2.0, + y + 36, + _e(label), + ) + ) + + +def _axis_labels( + left: int, + top: int, + width: int, + height: int, + maximum: float, + minimum: float = 0.0, +) -> str: + return ( + '%s' + '%s' + % ( + left - 6, + top + 4, + _fmt(maximum), + left - 6, + top + height, + _fmt(minimum), + ) + ) + + +def _thin(pairs: List[Any], limit: int) -> List[Any]: + if len(pairs) <= limit: + return pairs + step = int(math.ceil(len(pairs) / float(limit))) + return pairs[::step] + + +# --------------------------------------------------------------------------- +# Formatting +# --------------------------------------------------------------------------- +def _e(value: str) -> str: + return html.escape(str(value), quote=True) + + +def _fmt(value: Any) -> str: + if value is None: + return "n/a" + if isinstance(value, bool): + return "yes" if value else "no" + if isinstance(value, int): + return str(value) + try: + number = float(value) + except (TypeError, ValueError): + return _e(str(value)) + if not math.isfinite(number): + return "n/a" + magnitude = abs(number) + if magnitude != 0 and (magnitude < 1e-3 or magnitude >= 1e6): + return "%.4e" % number + return "%.4g" % number + + +def _pct(value: Any) -> str: + if value is None: + return "n/a" + try: + number = float(value) + except (TypeError, ValueError): + return "n/a" + if not math.isfinite(number): + return "n/a" + return "%.3f%%" % (number * 100.0) + + +_CSS = """ +:root { color-scheme: light; } +* { box-sizing: border-box; } +body { margin: 0; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, + "Helvetica Neue", Arial, sans-serif; color: #172033; background: #f5f7fb; } +.hero { padding: 32px 40px 24px; background: linear-gradient(135deg, #172554, #1d4ed8); + color: #fff; } +.hero h1 { margin: 0 0 6px; font-size: 28px; } +.subtitle { margin: 0 0 14px; opacity: .9; } +.pills { display: flex; flex-wrap: wrap; gap: 8px; } +.pill { padding: 4px 10px; border-radius: 999px; background: rgba(255,255,255,.16); + font-size: 12px; } +.hero .pill { background: rgba(255,255,255,.16); } +.hero .note { color: rgba(255,255,255,.75); margin: 14px 0 0; font-size: 13px; } +section { margin: 24px 40px; padding: 22px 24px; background: #fff; border-radius: 14px; + box-shadow: 0 1px 2px rgba(15,23,42,.06); } +h2 { margin: 0 0 10px; font-size: 20px; } +h3 { margin: 18px 0 8px; font-size: 15px; } +h4 { margin: 0 0 6px; font-size: 15px; } +.cards { display: grid; grid-template-columns: repeat(auto-fit, minmax(150px, 1fr)); gap: 12px; } +.card { padding: 12px 14px; border-radius: 10px; background: #f8fafc; border: 1px solid #e2e8f0; } +.card-label { font-size: 12px; color: #64748b; text-transform: uppercase; letter-spacing: .04em; } +.card-value { font-size: 20px; font-weight: 650; margin: 4px 0; word-break: break-word; } +.card-scope { font-size: 11px; color: #94a3b8; } +.badges { display: flex; flex-wrap: wrap; gap: 8px; margin: 4px 0 8px; } +.badge { padding: 3px 9px; border-radius: 999px; background: #eff6ff; color: #1d4ed8; + font-size: 12px; font-weight: 600; } +.badge-full { background: #ecfdf5; color: #047857; } +.badge-sample { background: #fff7ed; color: #c2410c; } +.badge-mixed { background: #f5f3ff; color: #6d28d9; } +.definition { color: #475569; font-size: 13px; margin: 0 0 12px; } +.note, .muted { color: #64748b; font-size: 13px; } +.chart { width: 100%; max-width: 900px; height: auto; display: block; margin: 6px 0 18px; } +.chart-caption { color: #475569; font-size: 12px; margin: 2px 0 4px; } +.plot-bg { fill: #fbfdff; } +.axis { stroke: #cbd5e1; stroke-width: 1; } +.bar { fill: #2563eb; opacity: .85; cursor: default; } +.bar:hover, .point:hover { opacity: 1; } +.point { cursor: default; } +.line { stroke-linejoin: round; stroke-linecap: round; } +.axis-label, .x-label, .y-label { fill: #64748b; font-size: 11px; } +table { border-collapse: collapse; margin: 8px 0 12px; font-size: 13px; } +table.kv th { text-align: left; color: #475569; font-weight: 500; padding: 4px 18px 4px 0; } +table.kv td { padding: 4px 0; font-variant-numeric: tabular-nums; } +table.grid { width: 100%; max-width: 760px; } +table.grid th, table.grid td { border-bottom: 1px solid #e2e8f0; padding: 6px 10px; + text-align: left; } +table.grid th { color: #64748b; font-weight: 600; } +.warning { padding: 12px 14px; border-radius: 10px; background: #fff7ed; color: #9a3412; + border: 1px solid #fed7aa; margin: 8px 0; } +.warnings { color: #9a3412; font-size: 13px; } +.summary { font-size: 15px; color: #1e293b; } +.ai-item { border-left: 3px solid #2563eb; padding: 4px 0 4px 14px; margin: 14px 0; } +.ai-label { font-size: 11px; color: #1d4ed8; text-transform: uppercase; letter-spacing: .06em; } +.evidence-row { display: flex; flex-wrap: wrap; gap: 6px; margin: 6px 0; } +.evidence { background: #f1f5f9; color: #334155; border-radius: 4px; padding: 2px 6px; + font-size: 12px; } +.validation { font-size: 13px; color: #475569; } +.tooltip { position: fixed; pointer-events: none; background: #0f172a; color: #fff; + padding: 5px 8px; border-radius: 6px; font-size: 12px; opacity: 0; + transition: opacity .08s; z-index: 10; max-width: 320px; } +footer { color: #64748b; font-size: 12px; margin: 8px 40px 40px; } +code { font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; } +@media (max-width: 640px) { section, footer { margin-left: 12px; margin-right: 12px; } + .hero { padding: 24px 16px; } } +""" + +_JS = """ +(function () { + var tip = document.getElementById('tooltip'); + document.querySelectorAll('[data-tip]').forEach(function (node) { + node.addEventListener('mouseenter', function (event) { + tip.textContent = node.getAttribute('data-tip'); + tip.style.opacity = '1'; + }); + node.addEventListener('mousemove', function (event) { + tip.style.left = (event.clientX + 14) + 'px'; + tip.style.top = (event.clientY + 14) + 'px'; + }); + node.addEventListener('mouseleave', function () { tip.style.opacity = '0'; }); + }); +})(); +""" diff --git a/tools/dataset_profiler/dataset_profiler/profiler.py b/tools/dataset_profiler/dataset_profiler/profiler.py new file mode 100644 index 0000000000..5898266d91 --- /dev/null +++ b/tools/dataset_profiler/dataset_profiler/profiler.py @@ -0,0 +1,1718 @@ +# Copyright 2024-present the vsag project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Deterministic dataset profiling. + +The profiler makes a single streaming pass over the selected split for +full-dataset statistics and a bounded random sample for distribution, +pairwise-distance, anisotropy, and optional clustering statistics. No +all-pairs ``O(N^2)`` matrix is ever allocated for the complete dataset: pairwise +work is capped by ``pairwise_sample_size``. +""" + +from __future__ import annotations + +import hashlib +import math +from dataclasses import dataclass +from typing import Any, Dict, List, Optional, Sequence, Tuple + +import numpy as np + +from .hdf5 import DEFAULT_CHUNK_BYTES, HDF5Dataset, DatasetMetadata +from .report import DEFINITIONS, build_report, metric_entry +from .sampling import SAMPLING_METHOD, make_rng, sample_indices +from .stats import ( + HISTOGRAM_BINS, + as_finite_float, + downsample, + histogram, + summarize, + top_indices, +) + +VALID_METRICS = ("euclidean", "ip", "angular") +_DEFAULT_DENSE_METRIC = "euclidean" +_DEFAULT_SPARSE_METRIC = "ip" +_MAX_ANISOTROPY_DIM = 50000 +# One byte budget shared by every sampled-memory guard: the dense float64 +# sample, the dense expansion of a sparse sample, and the retained sparse +# sample rows below. Sparse samples must obey the same budget as dense ones +# even though their rows carry a variable number of entries. +_SAMPLE_MEMORY_BUDGET_BYTES = 256 * 1024 * 1024 +# A retained sparse sample keeps one (uint32, float32) copy per entry and later +# expands into one Python dict entry per finite value. Besides those 8 array +# bytes, the expansion creates Python int/float objects and allocates hash-table +# storage. Use a conservative per-entry estimate for all of them so nnz-heavy +# rows trip the budget before either representation can exhaust memory. +_SPARSE_SAMPLE_ENTRY_BYTES = 128 +_SPARSE_SAMPLE_ROW_OVERHEAD_BYTES = 128 +_MAX_SPARSE_STATS_DIMENSION = 10_000_000 +_PCA_SVD_MAX_DIM = 4096 +# Exact norm quantiles/outliers and exact duplicate groups need per-row state. +# Budget each full-split structure independently and disable its metric instead +# of allowing dataset-sized Python/NumPy state to grow without a limit. +_FULL_SPLIT_STATE_BUDGET_BYTES = 256 * 1024 * 1024 +_NORM_BYTES_PER_VECTOR = 8 +# Store fixed-width digests in a preallocated array instead of Python objects. +# np.unique additionally holds a sorted copy, sort index, unique output, and +# counts, bringing peak duplicate-detection state to about 64 bytes per vector. +_DUPLICATE_DIGEST_BYTES = 16 +_DUPLICATE_BYTES_PER_VECTOR = 4 * _DUPLICATE_DIGEST_BYTES +# Dense streaming blocks are sized by their working bytes, not by a fixed row +# count: every row keeps the raw block, a float64 copy, a finite mask, and +# several float64 temporaries alive at the same time. +_DENSE_BLOCK_BYTES = 128 * 1024 * 1024 +_DENSE_BLOCK_BYTES_PER_VALUE = 40 +_DENSE_BLOCK_MAX_ROWS = 65536 +# Relative tolerance under which the expanded Euclidean formula (||a||²+||b||²-2a·b) +# has lost every significant digit; such pairs are recomputed from differences. +_EUCLIDEAN_GRAM_REL_TOL = 1e-8 +# Normalization and dot products lose significant digits for nearly coincident +# angular vectors. Recompute pairs in this band from normalized differences +# instead of canonicalizing genuine small distances to zero. +_ANGULAR_DIRECT_TOL = 32 * np.finfo(np.float64).eps +# Row budget for the streaming mean-cosine reduction (normalized blocks only). +_MEAN_COSINE_BLOCK_ROWS = 8192 +_INVALID_SAMPLE_WARNING = ( + "sampled vectors contain NaN/Inf entries; sampled metrics replace them with " + "0.0, which can distort pairwise, nearest-neighbor, anisotropy, and clustering " + "results" +) + + +@dataclass +class ProfilerConfig: + """Configuration for :func:`profile_dataset`.""" + + seed: int = 42 + sample_size: int = 1024 + pairwise_sample_size: int = 512 + metric: Optional[str] = None + clusters: int = 0 + near_duplicate_relative_epsilon: float = 1e-3 + max_chunk_bytes: int = DEFAULT_CHUNK_BYTES + max_dimension_points: int = 1024 + + def validate(self) -> None: + if int(self.sample_size) <= 0: + raise ValueError("sample_size must be positive") + if int(self.pairwise_sample_size) <= 0: + raise ValueError("pairwise_sample_size must be positive") + if int(self.clusters) < 0: + raise ValueError("clusters must be non-negative") + if int(self.clusters) > 1000: + raise ValueError("clusters must be at most 1000") + if not math.isfinite(float(self.near_duplicate_relative_epsilon)): + raise ValueError("near_duplicate_relative_epsilon must be finite") + if float(self.near_duplicate_relative_epsilon) < 0: + raise ValueError("near_duplicate_relative_epsilon must be non-negative") + if int(self.max_chunk_bytes) < 16: + raise ValueError("max_chunk_bytes is too small") + if int(self.max_dimension_points) <= 0: + raise ValueError("max_dimension_points must be positive") + + +def profile_dataset( + path: str, + split: str = "train", + *, + seed: int = 42, + sample_size: int = 1024, + pairwise_sample_size: int = 512, + metric: Optional[str] = None, + clusters: int = 0, + near_duplicate_relative_epsilon: float = 1e-3, + max_chunk_bytes: int = DEFAULT_CHUNK_BYTES, + max_dimension_points: int = 1024, +) -> Dict[str, Any]: + """Profile one split of a dense or sparse VSAG-compatible HDF5 file. + + Args: + path: Path to the ``.hdf5`` file. + split: Dataset to profile (``"train"`` by default). + seed: Explicit seed for every sampling operation. + sample_size: Maximum number of vectors drawn for distribution and + dimensionality-reduction metrics. + pairwise_sample_size: Maximum number of sampled vectors used for the + pairwise-distance matrix. Must not exceed the complete dataset. + metric: ``"euclidean"``, ``"ip"``, or ``"angular"``. When omitted the + file's ``distance`` attribute is used, falling back to + ``"euclidean"`` for dense and ``"ip"`` for sparse files. + clusters: Optional k-means cluster count on the PCA-2 projection. + near_duplicate_relative_epsilon: Relative pairwise-distance threshold + used for the near-duplicate indicator. + max_chunk_bytes: Streaming read chunk size. + max_dimension_points: Maximum per-dimension points stored in reports; + longer arrays are bucket-averaged. + + Returns: + A JSON-serializable versioned report dictionary. + + Raises: + ValueError: For invalid configuration or unsupported/corrupt input. + """ + config = ProfilerConfig( + seed=seed, + sample_size=sample_size, + pairwise_sample_size=pairwise_sample_size, + metric=metric, + clusters=clusters, + near_duplicate_relative_epsilon=near_duplicate_relative_epsilon, + max_chunk_bytes=max_chunk_bytes, + max_dimension_points=max_dimension_points, + ) + config.validate() + + with HDF5Dataset(path, split=split) as dataset: + metadata = dataset.metadata + warnings: List[str] = list(metadata.warnings) + metric_name = _normalize_metric(config.metric, metadata, warnings) + rng = make_rng(config.seed) + + if metadata.vector_type == "dense": + metrics = _profile_dense(dataset, config, metric_name, rng, warnings) + else: + metrics, effective_sample, effective_pairwise = _profile_sparse( + dataset, config, metric_name, rng, warnings + ) + + requested_sample = int(config.sample_size) + requested_pairwise = int(config.pairwise_sample_size) + dataset_count = int(metadata.count or 0) + dataset_limited_sample = min(requested_sample, dataset_count) + if metadata.vector_type == "dense": + effective_sample = _dense_sample_size( + dataset_count, int(metadata.dim or 0), requested_sample + ) + effective_pairwise = min(requested_pairwise, effective_sample) + if dataset_limited_sample < requested_sample: + warnings.append( + "sample_size reduced from %d to %d to stay within the dataset size" + % (requested_sample, dataset_limited_sample) + ) + if ( + metadata.vector_type == "dense" + and effective_sample < dataset_limited_sample + ): + warnings.append( + "sample_size reduced from %d to %d to stay within the dense " + "sample memory budget" % (dataset_limited_sample, effective_sample) + ) + if effective_pairwise < requested_pairwise: + warnings.append( + "pairwise_sample_size reduced from %d to %d to stay within the " + "available sample/dataset size" + % (requested_pairwise, effective_pairwise) + ) + + report_config = { + "seed": int(config.seed), + "sample_size": requested_sample, + "pairwise_sample_size": requested_pairwise, + "effective_sample_size": effective_sample, + "effective_pairwise_sample_size": effective_pairwise, + "sampling_method": SAMPLING_METHOD, + "pairwise_sampling_method": ( + "uniformly random subset selected by a seeded permutation of the " + "sampled rows" + ), + "metric": metric_name, + "clusters": int(config.clusters), + "near_duplicate_relative_epsilon": float( + config.near_duplicate_relative_epsilon + ), + "max_dimension_points": int(config.max_dimension_points), + } + return build_report(metadata, report_config, metrics, warnings) + + +# --------------------------------------------------------------------------- +# Metric selection +# --------------------------------------------------------------------------- +def _normalize_metric( + requested: Optional[str], metadata: DatasetMetadata, warnings: List[str] +) -> str: + aliases = { + "l2": "euclidean", + "euclid": "euclidean", + "inner_product": "ip", + "dot": "ip", + "cosine": "angular", + "cosine_similarity": "angular", + } + candidate = requested if requested is not None else metadata.distance + if candidate is None: + fallback = _DEFAULT_SPARSE_METRIC if metadata.vector_type == "sparse" else _DEFAULT_DENSE_METRIC + warnings.append( + "no distance metric was provided and the file has no 'distance' attribute; " + "using %r" % fallback + ) + return fallback + normalized = aliases.get(str(candidate).strip().lower(), str(candidate).strip().lower()) + if normalized not in VALID_METRICS: + raise ValueError( + "unsupported distance metric %r (expected one of %s)" + % (candidate, ", ".join(VALID_METRICS)) + ) + if metadata.vector_type == "sparse" and normalized != "ip": + warnings.append( + "sparse VSAG evaluation supports only the 'ip' metric; %r is computed " + "here for profiling but is not a supported eval metric for sparse files" + % normalized + ) + return normalized + + +# --------------------------------------------------------------------------- +# Dense profiling +# --------------------------------------------------------------------------- +def _dense_block_rows(dim: int) -> int: + """Return a dense block row count whose working buffers fit the byte budget. + + The streaming dense pass converts each block to float64 and keeps several + float64/bool temporaries alive simultaneously, so a fixed row count would + retain gigabytes for wide vectors. Row counts are derived from + ``_DENSE_BLOCK_BYTES`` instead. + """ + if dim <= 0: + return _DENSE_BLOCK_MAX_ROWS + rows = _DENSE_BLOCK_BYTES // (int(dim) * _DENSE_BLOCK_BYTES_PER_VALUE) + return int(max(1, min(_DENSE_BLOCK_MAX_ROWS, rows))) + + +def _dense_sample_size(count: int, dim: int, requested: int) -> int: + """Cap the dense float64 sample by its materialized byte size.""" + dataset_limited = min(int(requested), int(count)) + if dim <= 0: + return int(dataset_limited) + max_rows = _SAMPLE_MEMORY_BUDGET_BYTES // (int(dim) * 8) + return int(min(dataset_limited, max_rows)) + + +def _sparse_sample_row_bytes(ids: np.ndarray) -> int: + """Estimate the bytes one sampled sparse row keeps after retention and + after the Python dict expansion performed by :func:`_pairwise_sparse`.""" + return ( + _SPARSE_SAMPLE_ROW_OVERHEAD_BYTES + int(ids.size) * _SPARSE_SAMPLE_ENTRY_BYTES + ) + + +def _sparse_sample_budget_reason( + next_row_bytes: int, retained_bytes: int, retained_rows: int +) -> str: + """Return why retention of the sampled sparse rows stopped.""" + return ( + "sparse sample limited to %d retained rows: continuing would exceed the " + "%d-byte sample memory budget (estimated %d total bytes so far plus the " + "pending row); the sampled sparse pairwise and distribution metrics are " + "unavailable, while streaming and full-split metrics are still reported" + % ( + retained_rows, + _SAMPLE_MEMORY_BUDGET_BYTES, + int(retained_bytes) + int(next_row_bytes), + ) + ) + + +def _full_split_state_reason( + metric_name: str, count: int, bytes_per_vector: int +) -> Optional[str]: + """Return why a full-split metric exceeds its retained-state budget.""" + estimated = int(count) * int(bytes_per_vector) + budget = int(_FULL_SPLIT_STATE_BUDGET_BYTES) + if estimated <= budget: + return None + return ( + "%s requires an estimated %d bytes for %d vectors (%d bytes per vector), " + "exceeding the %d-byte full-split state budget; this full-split metric is " + "unavailable, while streaming and sampled metrics are still reported" + % (metric_name, estimated, int(count), int(bytes_per_vector), budget) + ) + + +def _duplicate_digest_buffer( + count: int, unavailable_reason: Optional[str] +) -> Optional[np.ndarray]: + """Allocate fixed-width duplicate digests only when they fit the budget.""" + if unavailable_reason is not None: + return None + return np.empty((int(count), _DUPLICATE_DIGEST_BYTES), dtype=np.uint8) + + +def _profile_dense( + dataset: HDF5Dataset, + config: ProfilerConfig, + metric: str, + rng: np.random.Generator, + warnings: List[str], +) -> Dict[str, Any]: + metadata = dataset.metadata + count = int(metadata.count or 0) + dim = int(metadata.dim or 0) + sample_size = _dense_sample_size(count, dim, config.sample_size) + sample_idx = sample_indices(count, sample_size, rng) + pair_n = min(config.pairwise_sample_size, sample_idx.size) + # Sorting only restores row order after selection; the subset itself comes + # from a seeded permutation and is not biased toward low dataset indices. + pair_positions = np.sort(rng.permutation(sample_idx.size)[:pair_n]) + + dim_count = np.zeros(dim, dtype=np.float64) + dim_mean = np.zeros(dim, dtype=np.float64) + dim_m2 = np.zeros(dim, dtype=np.float64) + dim_zero = np.zeros(dim, dtype=np.float64) + dim_min = np.full(dim, np.inf, dtype=np.float64) + dim_max = np.full(dim, -np.inf, dtype=np.float64) + + norms_reason = _full_split_state_reason( + "norm distribution and outlier statistics", count, _NORM_BYTES_PER_VECTOR + ) + duplicates_reason = _full_split_state_reason( + "exact duplicate detection", count, _DUPLICATE_BYTES_PER_VECTOR + ) + for reason in (norms_reason, duplicates_reason): + if reason is not None: + warnings.append(reason) + hashes: Optional[np.ndarray] = _duplicate_digest_buffer(count, duplicates_reason) + norms: Optional[np.ndarray] = ( + None if norms_reason is not None else np.empty(count, dtype=np.float64) + ) + total_values = 0 + zero_values = 0 + nonzero_values = 0 + invalid_values = 0 + vectors_with_invalid = 0 + all_zero_vectors = 0 + + for start, block in dataset.iter_dense_blocks(block_rows=_dense_block_rows(dim)): + raw = np.asarray(block) + if raw.dtype.kind == "f": + finite = np.isfinite(raw) + else: + finite = np.ones(raw.shape, dtype=bool) + valid = np.where(finite, raw, 0).astype(np.float64, copy=False) + invalid_in_block = int((~finite).sum()) + invalid_values += invalid_in_block + if invalid_in_block: + vectors_with_invalid += int((~finite).any(axis=1).sum()) + block_zero_values = int(np.count_nonzero((raw == 0) & finite)) + total_values += int(raw.size) + zero_values += block_zero_values + nonzero_values += int(raw.size) - invalid_in_block - block_zero_values + + block_norms = np.sqrt(np.einsum("ij,ij->i", valid, valid)) + if norms is not None: + norms[start : start + raw.shape[0]] = block_norms + all_zero_vectors += int(np.count_nonzero(block_norms == 0)) + + # Merge each block with the running per-coordinate Welford state. In + # particular, do not subtract E[x]^2 from E[x^2]: that loses small + # variations around large FLOAT32 magnitudes. + block_count = finite.sum(axis=0).astype(np.float64) + with np.errstate(divide="ignore", invalid="ignore"): + block_mean = np.where( + block_count > 0, valid.sum(axis=0) / block_count, 0.0 + ) + centered = np.where(finite, valid - block_mean, 0.0) + block_m2 = np.einsum("ij,ij->j", centered, centered) + combined_count = dim_count + block_count + delta = block_mean - dim_mean + with np.errstate(divide="ignore", invalid="ignore"): + weight = np.where( + combined_count > 0, block_count / combined_count, 0.0 + ) + correction = np.where( + combined_count > 0, + delta * delta * dim_count * weight, + 0.0, + ) + dim_mean += delta * weight + dim_m2 += block_m2 + correction + dim_count = combined_count + dim_zero += ((raw == 0) & finite).sum(axis=0) + masked_min = np.where(finite, raw, np.inf).min(axis=0) + masked_max = np.where(finite, raw, -np.inf).max(axis=0) + dim_min = np.minimum(dim_min, masked_min) + dim_max = np.maximum(dim_max, masked_max) + + if hashes is not None: + for row_index, row in enumerate(raw): + contiguous = np.ascontiguousarray(row) + hashes[start + row_index] = np.frombuffer( + _digest(contiguous.tobytes()), dtype=np.uint8 + ) + + sample_matrix = dataset.read_dense_rows(sample_idx).astype(np.float64, copy=True) + finite_sample = np.isfinite(sample_matrix) + if not bool(finite_sample.all()): + warnings.append(_INVALID_SAMPLE_WARNING) + # Replace invalid entries in place to avoid a second full sample-sized + # allocation; full-dataset numeric summaries continue to exclude them. + sample_matrix[~finite_sample] = 0.0 + pair_matrix = sample_matrix[pair_positions] + pair_distances = _pairwise_dense(pair_matrix, metric) + pair_values = _upper_triangle(pair_distances) + duplicate_pair_values = pair_values + duplicate_metric = metric + if metric == "ip": + # IP distance (1 - dot) is negative for unnormalized duplicates and is + # therefore unsuitable for a non-negative near-duplicate threshold. + duplicate_pair_values = _upper_triangle(_pairwise_dense(pair_matrix, "euclidean")) + duplicate_metric = "euclidean" + + metrics: Dict[str, Any] = { + "invalid_values": _invalid_metric( + invalid_values, vectors_with_invalid, total_values, count + ), + "norms": _norms_metric(norms, norms_reason), + "outliers": _outliers_metric(norms, norms_reason), + "dimensions": _dimensions_metric( + dim_count, + dim_mean, + dim_m2, + dim_zero, + dim_min, + dim_max, + config.max_dimension_points, + warnings, + ), + "sparsity": _dense_sparsity_metric( + total_values, zero_values, nonzero_values, all_zero_vectors, count + ), + "duplicates": _duplicates_metric( + hashes, + count, + duplicate_pair_values, + config.near_duplicate_relative_epsilon, + duplicate_metric, + int(pair_matrix.shape[0]), + duplicates_reason, + ), + } + metrics.update( + _sampled_metrics( + pair_matrix=pair_distances, + sample_matrix=sample_matrix, + pair_values=pair_values, + metric=metric, + config=config, + rng=rng, + warnings=warnings, + ) + ) + return metrics + + +# --------------------------------------------------------------------------- +# Sparse profiling +# --------------------------------------------------------------------------- +def _profile_sparse( + dataset: HDF5Dataset, + config: ProfilerConfig, + metric: str, + rng: np.random.Generator, + warnings: List[str], +) -> Tuple[Dict[str, Any], int, int]: + metadata = dataset.metadata + if metadata.count is None: + counted = 0 + for _ in dataset.iter_sparse_records(max_chunk_bytes=config.max_chunk_bytes): + counted += 1 + metadata.count = counted + warnings.append( + "sparse file has no '%s_offsets'; the byte stream was scanned once to " + "count records before profiling" % metadata.split + ) + count = int(metadata.count or 0) + if count <= 0: + raise ValueError("sparse dataset '/%s' is empty" % metadata.split) + sample_idx = sample_indices(count, min(config.sample_size, count), rng) + pair_n = min(config.pairwise_sample_size, sample_idx.size) + pair_positions = np.sort(rng.permutation(sample_idx.size)[:pair_n]) + sample_set = set(int(index) for index in sample_idx) + pair_sample_set = set(int(sample_idx[position]) for position in pair_positions) + + accumulator: Optional[_DimAccumulator] = _DimAccumulator() + dimension_stats_reason: Optional[str] = None + dim = 0 + norms_reason = _full_split_state_reason( + "norm distribution and outlier statistics", count, _NORM_BYTES_PER_VECTOR + ) + duplicates_reason = _full_split_state_reason( + "exact duplicate detection", count, _DUPLICATE_BYTES_PER_VECTOR + ) + nnz_reason = _full_split_state_reason( + "sparse nnz-per-vector distribution", count, _NORM_BYTES_PER_VECTOR + ) + for reason in (norms_reason, duplicates_reason, nnz_reason): + if reason is not None: + warnings.append(reason) + hashes: Optional[np.ndarray] = _duplicate_digest_buffer(count, duplicates_reason) + # The record count is known before this pass (a no-offsets stream is counted + # above), so avoid retaining one tiny NumPy array per sparse vector. + norms: Optional[np.ndarray] = ( + None if norms_reason is not None else np.empty(count, dtype=np.float64) + ) + nnz: Optional[np.ndarray] = ( + None if nnz_reason is not None else np.empty(count, dtype=np.int64) + ) + stored_entries = 0 + nonzero_entries = 0 + explicit_zero_entries = 0 + invalid_values = 0 + vectors_with_invalid = 0 + all_zero_vectors = 0 + sample_records: List[Tuple[int, np.ndarray, np.ndarray]] = [] + # Sample retention is bounded by estimated bytes, not by the row count: + # valid sparse files can carry so many entries per vector that the full + # row count would exhaust memory before any report is produced. + retained_sample_bytes = 0 + sample_budget_reason: Optional[str] = None + + for index, raw_ids, raw_values in dataset.iter_sparse_records( + max_chunk_bytes=config.max_chunk_bytes + ): + if raw_ids.size: + order = np.argsort(raw_ids, kind="stable") + ids = raw_ids[order].astype(np.uint32, copy=False) + values = raw_values[order].astype(np.float32, copy=False) + else: + ids = raw_ids + values = raw_values + if ids.size > 1 and np.any(ids[1:] == ids[:-1]): + # Pairwise/PCA materialization keeps one value per feature, while + # duplicate observations would corrupt full per-dimension means. + # Reject the ambiguous encoding instead of mixing both semantics. + raise ValueError("sparse record %d contains repeated feature IDs" % index) + length = int(ids.size) + stored_entries += length + finite = np.isfinite(values) + valid = np.where(finite, values, 0).astype(np.float64, copy=False) + invalid_values += int((~finite).sum()) + if not bool(finite.all()): + vectors_with_invalid += 1 + nonzero = int(np.count_nonzero(valid)) + nonzero_entries += nonzero + explicit_zero_entries += int(np.count_nonzero((values == 0) & finite)) + norm = float(math.sqrt(float(np.dot(valid, valid)))) + if norms is not None: + norms[index] = norm + if nnz is not None: + nnz[index] = nonzero + if norm == 0.0: + all_zero_vectors += 1 + if ids.size: + dim = max(dim, int(ids[-1]) + 1) + if dim > _MAX_SPARSE_STATS_DIMENSION and accumulator is not None: + dimension_stats_reason = ( + "sparse dimensionality %d exceeds the per-dimension statistics " + "limit of %d; other metrics are still reported" + % (dim, _MAX_SPARSE_STATS_DIMENSION) + ) + warnings.append(dimension_stats_reason) + # Release the bounded accumulators before continuing the full pass. + accumulator = None + if accumulator is not None: + accumulator.add(ids, values, finite) + if hashes is not None: + payload = ids.astype(" _SAMPLE_MEMORY_BUDGET_BYTES: + sample_budget_reason = _sparse_sample_budget_reason( + row_bytes, retained_sample_bytes, len(sample_records) + ) + warnings.append(sample_budget_reason) + else: + retained_sample_bytes += row_bytes + sample_records.append((index, ids.copy(), values.copy())) + + if ( + sample_budget_reason is None + and len(sample_records) != len(sample_set) + ): + raise ValueError("sparse sampling failed: not all sampled records were found") + sample_records.sort(key=lambda item: item[0]) + pair_records = [ + record for record in sample_records if record[0] in pair_sample_set + ] + metadata.dim = dim + if dim == 0: + warnings.append( + "all sparse vectors are empty; dimensionality, density, and " + "per-dimension statistics are unavailable" + ) + + if accumulator is not None: + dim_count, dim_mean, dim_m2, dim_zero, dim_min, dim_max = ( + accumulator.arrays(count) + ) + dimensions = _dimensions_metric( + dim_count, + dim_mean, + dim_m2, + dim_zero, + dim_min, + dim_max, + config.max_dimension_points, + warnings, + ) + else: + dimensions = metric_entry( + DEFINITIONS["dimensions"], + "full", + None, + available=False, + reason=dimension_stats_reason, + dim=int(dim), + ) + if sample_budget_reason is None: + if any( + not bool(np.isfinite(values).all()) for _, _, values in sample_records + ): + warnings.append(_INVALID_SAMPLE_WARNING) + pair_matrix = _pairwise_sparse(pair_records, metric) + pair_values = _upper_triangle(pair_matrix) + duplicate_pair_values = pair_values + duplicate_metric = metric + if metric == "ip": + duplicate_pair_values = _upper_triangle( + _pairwise_sparse(pair_records, "euclidean") + ) + duplicate_metric = "euclidean" + sample_matrix = _sparse_sample_to_dense(sample_records, dim) + else: + pair_matrix = np.zeros((0, 0), dtype=np.float64) + pair_values = np.empty(0, dtype=np.float64) + duplicate_pair_values = pair_values + duplicate_metric = metric + sample_matrix = None + + metrics: Dict[str, Any] = { + "invalid_values": _invalid_metric( + invalid_values, vectors_with_invalid, count * dim if dim else None, count + ), + "norms": _norms_metric(norms, norms_reason), + "outliers": _outliers_metric(norms, norms_reason), + "dimensions": dimensions, + "sparsity": _sparse_sparsity_metric( + count, + dim, + stored_entries, + nonzero_entries, + invalid_values, + explicit_zero_entries, + nnz, + all_zero_vectors, + nnz_reason, + ), + "duplicates": _duplicates_metric( + hashes, + count, + duplicate_pair_values, + config.near_duplicate_relative_epsilon, + duplicate_metric, + int(pair_matrix.shape[0]), + duplicates_reason, + near_unavailable_reason=sample_budget_reason, + ), + } + metrics.update( + _sampled_metrics( + pair_matrix=pair_matrix, + sample_matrix=sample_matrix, + pair_values=pair_values, + metric=metric, + config=config, + rng=rng, + warnings=warnings, + sparse_pairwise=True, + unavailable_reason=sample_budget_reason, + ) + ) + effective_sample = 0 if sample_budget_reason is not None else len(sample_records) + return metrics, effective_sample, int(pair_matrix.shape[0]) + + +class _DimAccumulator: + """Bounded growable Welford accumulators for sparse dimension data.""" + + def __init__(self) -> None: + self._max_index = -1 + self.count = np.zeros(0, dtype=np.float64) + self.invalid = np.zeros(0, dtype=np.float64) + self.mean = np.zeros(0, dtype=np.float64) + self.m2 = np.zeros(0, dtype=np.float64) + self.zero = np.zeros(0, dtype=np.float64) + self.min = np.full(0, np.inf, dtype=np.float64) + self.max = np.full(0, -np.inf, dtype=np.float64) + + @property + def dim(self) -> int: + return self._max_index + 1 + + def _ensure(self, size: int) -> None: + if size <= self.count.size: + return + if size > _MAX_SPARSE_STATS_DIMENSION: + # The profiling loop normally disables dimension metrics before + # reaching this guard. Keep the accumulator itself safe as well. + raise ValueError("sparse dimension exceeds per-dimension statistics limit") + new_size = min( + max(size, max(16, self.count.size * 2)), _MAX_SPARSE_STATS_DIMENSION + ) + pad = new_size - self.count.size + self.count = np.concatenate([self.count, np.zeros(pad)]) + self.invalid = np.concatenate([self.invalid, np.zeros(pad)]) + self.mean = np.concatenate([self.mean, np.zeros(pad)]) + self.m2 = np.concatenate([self.m2, np.zeros(pad)]) + self.zero = np.concatenate([self.zero, np.zeros(pad)]) + self.min = np.concatenate([self.min, np.full(pad, np.inf)]) + self.max = np.concatenate([self.max, np.full(pad, -np.inf)]) + + def add(self, ids: np.ndarray, values: np.ndarray, finite: np.ndarray) -> None: + if ids.size == 0: + return + self._max_index = max(self._max_index, int(ids.max())) + self._ensure(self._max_index + 1) + all_indices = ids.astype(np.int64, copy=False) + if not bool(finite.all()): + np.add.at(self.invalid, all_indices[~finite], 1.0) + if not np.any(finite): + return + index = all_indices[finite] + vals = values[finite].astype(np.float64, copy=False) + + # Merge one stable per-record batch for each feature. np.unique also + # makes repeated feature IDs deterministic without scalar Welford loops. + unique, inverse = np.unique(index, return_inverse=True) + group_count = np.bincount(inverse).astype(np.float64) + group_sum = np.bincount(inverse, weights=vals) + group_mean = group_sum / group_count + centered = vals - group_mean[inverse] + group_m2 = np.bincount(inverse, weights=centered * centered) + old_count = self.count[unique] + old_mean = self.mean[unique] + combined_count = old_count + group_count + delta = group_mean - old_mean + weight = group_count / combined_count + self.mean[unique] = old_mean + delta * weight + self.m2[unique] = ( + self.m2[unique] + group_m2 + delta * delta * old_count * weight + ) + self.count[unique] = combined_count + + zero = vals == 0 + if np.any(zero): + np.add.at(self.zero, index[zero], 1.0) + np.minimum.at(self.min, index, vals) + np.maximum.at(self.max, index, vals) + + def arrays(self, vector_count: int): + """Merge implicit zeros stably while excluding stored invalid values.""" + dim = self.dim + finite_count = self.count[:dim] + invalid_count = self.invalid[:dim] + count = float(vector_count) - invalid_count + implicit_count = np.maximum(count - finite_count, 0.0) + mean = self.mean[:dim].copy() + m2 = self.m2[:dim].copy() + with np.errstate(divide="ignore", invalid="ignore"): + mean_weight = np.where(count > 0, finite_count / count, 0.0) + zero_correction = np.where( + count > 0, + mean * mean * finite_count * (implicit_count / count), + 0.0, + ) + mean *= mean_weight + m2 += zero_correction + zero = self.zero[:dim] + implicit_count + implicit_zero = implicit_count > 0 + minimum = self.min[:dim].copy() + maximum = self.max[:dim].copy() + minimum[implicit_zero] = np.minimum(minimum[implicit_zero], 0.0) + maximum[implicit_zero] = np.maximum(maximum[implicit_zero], 0.0) + return count, mean, m2, zero, minimum, maximum + + +# --------------------------------------------------------------------------- +# Shared metric builders +# --------------------------------------------------------------------------- +def _norms_metric( + norms: Optional[np.ndarray], unavailable_reason: Optional[str] = None +) -> Dict[str, Any]: + if norms is None: + return metric_entry( + DEFINITIONS["norms"], + "full", + None, + available=False, + reason=unavailable_reason or "full-split norm state is unavailable", + ) + stats = summarize(norms) + return metric_entry( + DEFINITIONS["norms"], + "full", + None, + count=int(norms.size), + stats=stats, + histogram=histogram(norms), + non_finite_norms=int(np.size(norms) - np.count_nonzero(np.isfinite(norms))), + ) + + +def _outliers_metric( + norms: Optional[np.ndarray], unavailable_reason: Optional[str] = None +) -> Dict[str, Any]: + if norms is None: + return metric_entry( + DEFINITIONS["outliers"], + "full", + None, + available=False, + reason=unavailable_reason or "full-split norm state is unavailable", + ) + values = np.asarray(norms, dtype=np.float64) + values = values[np.isfinite(values)] + if values.size == 0: + return metric_entry( + DEFINITIONS["outliers"], + "full", + None, + count=0, + outlier_count=0, + outlier_ratio=None, + median_norm=None, + mad_norm=None, + max_over_median_norm=None, + method="n/a", + ) + median = float(np.median(values)) + deviation = np.abs(values - median) + mad = float(np.median(deviation)) + method = "modified z-score (median/MAD)" + if mad > 0: + modified_z = 0.6745 * (values - median) / mad + mask = np.abs(modified_z) > 3.5 + threshold = 3.5 * mad / 0.6745 + else: + q1, q3 = np.percentile(values, [25, 75], method="linear") + iqr = float(q3 - q1) + if iqr > 0: + mask = (values < q1 - 3.0 * iqr) | (values > q3 + 3.0 * iqr) + method = "IQR fallback (MAD is zero)" + threshold = 3.0 * iqr + elif np.all(values == median): + mask = np.zeros(values.shape, dtype=bool) + method = "all norms identical; no outlier can be distinguished" + threshold = 0.0 + else: + # With a repeated central value, both robust spreads can be zero + # even though extreme values exist (for example 99 ones and 100). + mask = values != median + method = "median fallback (MAD and IQR are zero)" + threshold = 0.0 + outlier_count = int(np.count_nonzero(mask)) + max_over_median = float(values.max() / median) if median > 0 else None + return metric_entry( + DEFINITIONS["outliers"], + "full", + None, + count=int(values.size), + outlier_count=outlier_count, + outlier_ratio=as_finite_float(outlier_count / float(values.size)), + median_norm=as_finite_float(median), + mad_norm=as_finite_float(mad), + threshold=as_finite_float(threshold), + max_over_median_norm=as_finite_float(max_over_median), + method=method, + ) + + +def _invalid_metric( + invalid_values: int, + vectors_with_invalid: int, + total_values: Optional[int], + count: int, +) -> Dict[str, Any]: + total = int(total_values) if total_values is not None else None + ratio = None + if total: + ratio = invalid_values / float(total) + return metric_entry( + DEFINITIONS["invalid_values"], + "full", + None, + count=count, + total_values=total, + invalid_values=int(invalid_values), + vectors_with_invalid=int(vectors_with_invalid), + invalid_ratio=as_finite_float(ratio), + vectors_with_invalid_ratio=as_finite_float( + vectors_with_invalid / float(count) if count else None + ), + ) + + +def _dimensions_metric( + count: np.ndarray, + mean_value: np.ndarray, + m2: np.ndarray, + zero: np.ndarray, + minimum: np.ndarray, + maximum: np.ndarray, + max_points: int, + warnings: List[str], +) -> Dict[str, Any]: + mean = np.asarray(mean_value, dtype=np.float64).copy() + mean[count <= 0] = np.nan + with np.errstate(divide="ignore", invalid="ignore"): + variance = np.where(count > 0, m2 / count, np.nan) + std = np.sqrt(np.maximum(variance, 0)) + zero_ratio = np.where(count > 0, zero / count, np.nan) + minimum = np.where(np.isfinite(minimum), minimum, np.nan) + maximum = np.where(np.isfinite(maximum), maximum, np.nan) + + mean_view = downsample(mean, max_points) + std_view = downsample(std, max_points) + zero_view = downsample(zero_ratio, max_points) + min_view = downsample(minimum, max_points) + max_view = downsample(maximum, max_points) + if any(view["downsampled"] for view in (mean_view, std_view, zero_view)): + warnings.append( + "per-dimension arrays were bucket-averaged to at most %d points per " + "series; see per_dimension.downsampled in the report" % max_points + ) + + top_variance = [] + for index in top_indices(std, 10): + top_variance.append( + { + "dimension": int(index), + "std": as_finite_float(std[index]), + "mean": as_finite_float(mean[index]), + "zero_ratio": as_finite_float(zero_ratio[index]), + } + ) + top_zero = [] + for index in top_indices(zero_ratio, 10): + top_zero.append( + { + "dimension": int(index), + "zero_ratio": as_finite_float(zero_ratio[index]), + "mean": as_finite_float(mean[index]), + } + ) + + return metric_entry( + DEFINITIONS["dimensions"], + "full", + None, + dim=int(mean.size), + summary={ + "mean": summarize(mean), + "std": summarize(std), + "zero_ratio": summarize(zero_ratio), + }, + top_variance_dimensions=top_variance, + top_zero_dimensions=top_zero, + per_dimension={ + "mean": mean_view, + "std": std_view, + "zero_ratio": zero_view, + "min": min_view, + "max": max_view, + }, + ) + + +def _dense_sparsity_metric( + total_values: int, + zero_values: int, + nonzero_values: int, + all_zero_vectors: int, + count: int, +) -> Dict[str, Any]: + density = nonzero_values / float(total_values) if total_values else None + return metric_entry( + DEFINITIONS["sparsity"], + "full", + None, + representation="dense", + count=count, + total_values=int(total_values), + zero_values=int(zero_values), + non_zero_values=int(nonzero_values), + zero_ratio=as_finite_float(zero_values / float(total_values) if total_values else None), + density=as_finite_float(density), + all_zero_vectors=int(all_zero_vectors), + all_zero_vector_ratio=as_finite_float( + all_zero_vectors / float(count) if count else None + ), + ) + + +def _sparse_sparsity_metric( + count: int, + dim: int, + stored_entries: int, + nonzero_entries: int, + invalid_values: int, + explicit_zero_entries: int, + nnz_per_vector: Optional[np.ndarray], + all_zero_vectors: int, + nnz_unavailable_reason: Optional[str] = None, +) -> Dict[str, Any]: + cells = count * dim if dim else None + zero_values = ( + cells - nonzero_entries - invalid_values if cells is not None else None + ) + density = nonzero_entries / float(cells) if cells else None + return metric_entry( + DEFINITIONS["sparsity"], + "full", + None, + representation="sparse", + count=int(count), + dim=int(dim), + cells=cells, + stored_entries=int(stored_entries), + zero_values=int(zero_values) if zero_values is not None else None, + non_zero_values=int(nonzero_entries), + explicit_zero_entries=int(explicit_zero_entries), + zero_ratio=as_finite_float( + zero_values / float(cells) if zero_values is not None and cells else None + ), + density=as_finite_float(density), + nnz_per_vector=( + summarize(nnz_per_vector) if nnz_per_vector is not None else None + ), + nnz_histogram=( + histogram(nnz_per_vector, bins=min(HISTOGRAM_BINS, 20)) + if nnz_per_vector is not None + else None + ), + nnz_available=nnz_per_vector is not None, + nnz_reason=( + None + if nnz_per_vector is not None + else nnz_unavailable_reason or "full-split nnz state is unavailable" + ), + all_zero_vectors=int(all_zero_vectors), + all_zero_vector_ratio=as_finite_float( + all_zero_vectors / float(count) if count else None + ), + ) + + +def _duplicates_metric( + hashes: Optional[np.ndarray], + count: int, + pair_values: np.ndarray, + epsilon: float, + near_metric: str, + near_sample_size: int, + unavailable_reason: Optional[str] = None, + near_unavailable_reason: Optional[str] = None, +) -> Dict[str, Any]: + if hashes is None: + exact = { + "available": False, + "reason": unavailable_reason or "full-split duplicate state is unavailable", + "definition": "Vectors whose complete byte representation is identical.", + "scope": "full", + "sample_size": None, + "count": int(count), + "unique_vectors": None, + "duplicate_vectors": None, + "duplicate_ratio": None, + "duplicate_groups": None, + "max_group_size": None, + } + scope_note = ( + "exact duplicate detection is unavailable because the full-split state " + "budget was exceeded; the near-duplicate indicator still covers the " + "bounded pairwise sample" + ) + else: + # A one-dimensional void view lets NumPy compare complete 16-byte + # digests without a Python dict entry and integer object per vector. + digest_rows = np.ascontiguousarray(hashes).view( + np.dtype((np.void, _DUPLICATE_DIGEST_BYTES)) + ).reshape(-1) + _, group_sizes = np.unique(digest_rows, return_counts=True) + repeated = group_sizes[group_sizes > 1] + duplicate_vectors = int(np.sum(repeated - 1, dtype=np.int64)) + duplicate_groups = int(repeated.size) + max_group = int(group_sizes.max()) if group_sizes.size else 0 + exact = { + "definition": "Vectors whose complete byte representation is identical.", + "scope": "full", + "sample_size": None, + "count": int(count), + "unique_vectors": int(group_sizes.size), + "duplicate_vectors": int(duplicate_vectors), + "duplicate_ratio": as_finite_float( + duplicate_vectors / float(count) if count else None + ), + "duplicate_groups": int(duplicate_groups), + "max_group_size": int(max_group), + } + scope_note = ( + "exact duplicate detection covers the full split; the near-duplicate " + "indicator covers the bounded pairwise sample" + ) + near = _near_duplicate_summary( + pair_values, epsilon, near_metric, near_sample_size, near_unavailable_reason + ) + return metric_entry( + DEFINITIONS["duplicates"], + "mixed", + None, + exact=exact, + near_duplicate=near, + scope_note=scope_note, + ) + + +def _near_duplicate_summary( + pair_values: np.ndarray, + epsilon: float, + metric: str, + sample_size: int, + unavailable_reason: Optional[str] = None, +) -> Dict[str, Any]: + if pair_values.size == 0: + return { + "available": False, + "reason": unavailable_reason or "fewer than two sampled vectors", + "scope": "sample", + "sample_size": int(sample_size), + "pair_count": 0, + } + finite = pair_values[np.isfinite(pair_values)] + if finite.size == 0: + return { + "available": False, + "reason": "no finite sampled pairwise distances", + "scope": "sample", + "sample_size": int(sample_size), + "pair_count": int(pair_values.size), + } + median = float(np.median(finite)) + threshold = max(0.0, median * float(epsilon)) + near_count = int(np.count_nonzero(finite <= threshold)) + return { + "available": True, + "definition": ( + "Sampled pairs whose duplicate-detection distance is at most epsilon " + "times the sampled pairwise-distance median." + ), + "metric": metric, + "scope": "sample", + "sample_size": int(sample_size), + "pair_count": int(finite.size), + "relative_epsilon": as_finite_float(epsilon), + "median_pairwise_distance": as_finite_float(median), + "threshold": as_finite_float(threshold), + "near_duplicate_pairs": near_count, + "near_duplicate_ratio": as_finite_float(near_count / float(finite.size)), + } + + +def _sampled_metrics( + *, + pair_matrix: np.ndarray, + sample_matrix: Optional[np.ndarray], + pair_values: np.ndarray, + metric: str, + config: ProfilerConfig, + rng: np.random.Generator, + warnings: List[str], + sparse_pairwise: bool = False, + unavailable_reason: Optional[str] = None, +) -> Dict[str, Any]: + sample_size = int(pair_matrix.shape[0]) + pair_count = int(pair_values.size) + if unavailable_reason is not None: + return { + name: metric_entry( + DEFINITIONS[name], + "sample", + sample_size, + available=False, + reason=unavailable_reason, + ) + for name in ( + "pairwise_distances", + "nearest_neighbor_distances", + "anisotropy", + "clustering", + ) + } + pairwise = metric_entry( + DEFINITIONS["pairwise_distances"], + "sample", + sample_size, + metric=metric, + pair_count=pair_count, + requested_pairwise_sample_size=int(config.pairwise_sample_size), + stats=summarize(pair_values), + histogram=histogram(pair_values), + matrix_is_bounded=True, + dense_sample_materialized=bool(sample_matrix is not None), + sparse_sparse_distance_loops=bool(sparse_pairwise), + ) + nearest = _nearest_neighbor_metric(pair_matrix) + pca_result = ( + _pca_statistics(sample_matrix) if sample_matrix is not None else None + ) + anisotropy = _anisotropy_metric( + sample_matrix, pair_matrix, metric, warnings, pca_result + ) + clustering = _clustering_metric( + sample_matrix, config.clusters, rng, warnings, pca_result + ) + return { + "pairwise_distances": pairwise, + "nearest_neighbor_distances": nearest, + "anisotropy": anisotropy, + "clustering": clustering, + } + + +def _nearest_neighbor_metric(pair_matrix: np.ndarray) -> Dict[str, Any]: + sample_size = int(pair_matrix.shape[0]) + if sample_size < 2: + return metric_entry( + DEFINITIONS["nearest_neighbor_distances"], + "sample", + sample_size, + available=False, + reason="fewer than two sampled vectors", + ) + matrix = np.array(pair_matrix, dtype=np.float64, copy=True) + np.fill_diagonal(matrix, np.inf) + values = np.min(matrix, axis=1) + values = values[np.isfinite(values)] + return metric_entry( + DEFINITIONS["nearest_neighbor_distances"], + "sample", + sample_size, + available=True, + stats=summarize(values), + histogram=histogram(values), + ) + + +def _anisotropy_metric( + sample_matrix: Optional[np.ndarray], + pair_matrix: np.ndarray, + metric: str, + warnings: List[str], + pca_result: Optional[Tuple[np.ndarray, np.ndarray, np.ndarray]], +) -> Dict[str, Any]: + sample_size = ( + int(sample_matrix.shape[0]) + if sample_matrix is not None + else int(pair_matrix.shape[0]) + ) + if sample_matrix is None: + return metric_entry( + DEFINITIONS["anisotropy"], + "sample", + sample_size, + available=False, + reason=( + "the sampled vectors could not be materialized as a dense matrix " + "within the dimensionality/memory guard" + ), + ) + if sample_size < 3 or sample_matrix.shape[1] == 0: + return metric_entry( + DEFINITIONS["anisotropy"], + "sample", + sample_size, + available=False, + reason="needs at least three sampled vectors with non-zero dimensionality", + ) + result = pca_result + if result is None: + return metric_entry( + DEFINITIONS["anisotropy"], + "sample", + sample_size, + available=False, + reason="sample contains no finite variation", + ) + eigenvalues, projection, explained = result + total = float(np.sum(eigenvalues)) + if total > 0: + shares = eigenvalues / total + top1 = float(shares[0]) + top10 = float(np.sum(shares[:10])) + entropy = -float(np.sum(shares[shares > 0] * np.log(shares[shares > 0]))) + effective_rank = float(math.exp(entropy)) + else: + top1 = top10 = effective_rank = None + mean_cosine = _mean_pairwise_cosine(sample_matrix) + return metric_entry( + DEFINITIONS["anisotropy"], + "sample", + sample_size, + available=True, + method="eigendecomposition of centered sample covariance", + top1_eigenvalue_share=as_finite_float(top1), + top10_eigenvalue_share=as_finite_float(top10), + effective_rank=as_finite_float(effective_rank), + mean_pairwise_cosine=as_finite_float(mean_cosine), + explained_variance_ratio=[as_finite_float(value) for value in explained[:10]], + interpretation=( + "top1_eigenvalue_share near 1 indicates a dominant direction; " + "effective_rank near 1 indicates nearly one-dimensional variation" + ), + ) + + +def _clustering_metric( + sample_matrix: Optional[np.ndarray], + clusters: int, + rng: np.random.Generator, + warnings: List[str], + pca_result: Optional[Tuple[np.ndarray, np.ndarray, np.ndarray]], +) -> Dict[str, Any]: + sample_size = 0 if sample_matrix is None else int(sample_matrix.shape[0]) + if sample_matrix is None: + return metric_entry( + DEFINITIONS["clustering"], + "sample", + sample_size, + available=False, + reason="sampled vectors could not be materialized as a dense matrix", + ) + if sample_matrix.shape[0] < 3 or sample_matrix.shape[1] == 0: + return metric_entry( + DEFINITIONS["clustering"], + "sample", + sample_size, + available=False, + reason="needs at least three sampled vectors with non-zero dimensionality", + ) + result = pca_result + if result is None: + return metric_entry( + DEFINITIONS["clustering"], + "sample", + sample_size, + available=False, + reason="sample contains no finite variation", + ) + _, projection, explained = result + output: Dict[str, Any] = { + "available": True, + "method": "deterministic PCA-2 projection of the seeded sample", + "projection_points": [ + [as_finite_float(point[0]), as_finite_float(point[1])] + for point in projection + ], + "explained_variance_ratio_pc1": as_finite_float( + explained[0] if explained.size else None + ), + "explained_variance_ratio_pc2": as_finite_float( + explained[1] if explained.size > 1 else None + ), + } + if clusters <= 0: + output["clustering_available"] = False + output["reason"] = "disabled (pass --clusters K to enable k-means)" + return metric_entry( + DEFINITIONS["clustering"], "sample", sample_size, **output + ) + labels, sizes, inertia, iterations = _kmeans(projection, clusters, rng) + output.update( + { + "clustering_available": True, + "n_clusters": int(len(sizes)), + "cluster_sizes": [int(value) for value in sizes], + "inertia": as_finite_float(inertia), + "iterations": int(iterations), + "labels": [int(value) for value in labels], + } + ) + return metric_entry(DEFINITIONS["clustering"], "sample", sample_size, **output) + + +# --------------------------------------------------------------------------- +# Pairwise distances +# --------------------------------------------------------------------------- +def _pairwise_dense(matrix: np.ndarray, metric: str) -> np.ndarray: + matrix = np.asarray(matrix, dtype=np.float64) + if matrix.shape[0] == 0: + return np.zeros((0, 0), dtype=np.float64) + if metric == "euclidean": + return _dense_euclidean(matrix) + if metric == "ip": + return 1.0 - matrix @ matrix.T + if metric == "angular": + # Canonicalize by the largest coordinate first. Positive scalar + # multiples then have identical scaled coordinates, so independent + # norm rounding cannot give them a spurious non-zero distance. + scales = np.max(np.abs(matrix), axis=1) + safe_scales = np.where(scales > 0, scales, 1.0) + scaled = matrix / safe_scales[:, None] + norms = np.linalg.norm(scaled, axis=1) + safe = np.where(norms > 0, norms, 1.0) + normalized = scaled / safe[:, None] + similarity = normalized @ normalized.T + np.clip(similarity, -1.0, 1.0, out=similarity) + distances = 1.0 - similarity + zero_norms = norms == 0 + if np.any(zero_norms): + distances[zero_norms, :] = 1.0 + distances[:, zero_norms] = 1.0 + distances[np.ix_(zero_norms, zero_norms)] = 0.0 + unstable = np.triu(np.abs(distances) <= _ANGULAR_DIRECT_TOL, k=1) + for row in np.unique(np.nonzero(unstable)[0]): + columns = np.nonzero(unstable[row])[0] + differences = normalized[row] - normalized[columns] + direct = 0.5 * np.einsum("ij,ij->i", differences, differences) + distances[row, columns] = direct + distances[columns, row] = direct + np.fill_diagonal(distances, 0.0) + return distances + raise ValueError("unsupported metric %r" % metric) + + +def _pairwise_sparse( + records: Sequence[Tuple[int, np.ndarray, np.ndarray]], metric: str +) -> np.ndarray: + size = len(records) + matrix = np.zeros((size, size), dtype=np.float64) + maps: List[Dict[int, float]] = [] + norms = np.zeros(size, dtype=np.float64) + for position, (_, ids, values) in enumerate(records): + mapping = {int(key): float(value) for key, value in zip(ids, values) if np.isfinite(value)} + if metric == "angular": + # Match the dense canonicalization: proportional sparse vectors + # must have identical coordinates before their norms are rounded. + scale = max((abs(value) for value in mapping.values()), default=0.0) + if scale > 0: + mapping = {key: value / scale for key, value in mapping.items()} + maps.append(mapping) + norms[position] = math.sqrt(sum(value * value for value in mapping.values())) + for i in range(size): + for j in range(i + 1, size): + left = maps[i] + right = maps[j] + if len(left) > len(right): + left, right = right, left + dot = 0.0 + for key, value in left.items(): + other = right.get(key) + if other is not None: + dot += value * other + if metric == "ip": + distance = 1.0 - dot + elif metric == "euclidean": + # Direct coordinate differences avoid the cancellation that the + # expanded norm/dot form suffers for large-magnitude vectors. + squared = 0.0 + for key in left.keys() | right.keys(): + delta = left.get(key, 0.0) - right.get(key, 0.0) + squared += delta * delta + distance = math.sqrt(squared) + elif metric == "angular": + if norms[i] == 0.0 and norms[j] == 0.0: + distance = 0.0 + elif norms[i] == 0.0 or norms[j] == 0.0: + distance = 1.0 + else: + cosine = max(-1.0, min(1.0, dot / (norms[i] * norms[j]))) + distance = 1.0 - cosine + if abs(distance) <= _ANGULAR_DIRECT_TOL: + squared = 0.0 + for key in maps[i].keys() | maps[j].keys(): + delta = maps[i].get(key, 0.0) / norms[i] - maps[j].get( + key, 0.0 + ) / norms[j] + squared += delta * delta + distance = 0.5 * squared + else: + raise ValueError("unsupported metric %r" % metric) + matrix[i, j] = distance + matrix[j, i] = distance + return matrix + + +def _dense_euclidean(matrix: np.ndarray) -> np.ndarray: + """Return the Euclidean distance matrix without catastrophic cancellation. + + The fast expanded form is used first. When the norms of two rows dwarf + their distance (for example rows near 1e10 whose differences are 1), that + expansion rounds to zero, so every pair touching an unstable row is + recomputed from direct coordinate differences. + """ + squared_norms = np.einsum("ij,ij->i", matrix, matrix) + squared = squared_norms[:, None] + squared_norms[None, :] - 2.0 * ( + matrix @ matrix.T + ) + scale = squared_norms[:, None] + squared_norms[None, :] + unstable = (scale > 0) & (squared <= _EUCLIDEAN_GRAM_REL_TOL * scale) + # A zero self-distance always satisfies the relative test. It needs no + # correction and must not send every ordinary row through the O(M*D) + # direct-difference fallback. + np.fill_diagonal(unstable, False) + if np.any(unstable): + for row in np.unique(np.nonzero(unstable)[0]): + difference = matrix[row] - matrix + direct = np.einsum("ij,ij->i", difference, difference) + squared[row, :] = direct + squared[:, row] = direct + np.maximum(squared, 0.0, out=squared) + return np.sqrt(squared) + + +def _upper_triangle(matrix: np.ndarray) -> np.ndarray: + if matrix.shape[0] < 2: + return np.empty(0, dtype=np.float64) + return matrix[np.triu_indices(matrix.shape[0], k=1)] + + +def _sparse_sample_to_dense( + records: Sequence[Tuple[int, np.ndarray, np.ndarray]], dim: int +) -> Optional[np.ndarray]: + if dim <= 0 or not records: + return None + rows = len(records) + if rows * dim * 8 > _SAMPLE_MEMORY_BUDGET_BYTES or dim > _MAX_ANISOTROPY_DIM: + return None + matrix = np.zeros((rows, dim), dtype=np.float64) + for position, (_, ids, values) in enumerate(records): + if ids.size: + matrix[position, ids.astype(np.int64, copy=False)] = values + return np.where(np.isfinite(matrix), matrix, 0.0) + + +# --------------------------------------------------------------------------- +# PCA / clustering helpers +# --------------------------------------------------------------------------- +def _pca_statistics( + matrix: np.ndarray, +) -> Optional[Tuple[np.ndarray, np.ndarray, np.ndarray]]: + matrix = np.asarray(matrix, dtype=np.float64) + if matrix.shape[0] < 3 or matrix.shape[1] == 0: + return None + centered = matrix - np.mean(matrix, axis=0, keepdims=True) + if not np.any(np.isfinite(centered)) or not np.any(centered != 0.0): + return None + rows, dim = centered.shape + if dim <= _PCA_SVD_MAX_DIM: + _, singular, vt = np.linalg.svd(centered, full_matrices=False) + eigenvalues = (singular * singular) / float(max(rows - 1, 1)) + total = float(np.sum(eigenvalues)) + explained = eigenvalues / total if total > 0 else np.zeros_like(eigenvalues) + components = vt[:2] if vt.shape[0] >= 2 else vt + projection = centered @ components.T + elif rows <= dim: + # For tall-enough wide inputs the row Gram matrix is the smaller square + # decomposition and avoids a dim-by-dim covariance allocation. + gram = centered @ centered.T + eigenvalues_raw, eigenvectors = np.linalg.eigh(gram) + order = np.argsort(eigenvalues_raw)[::-1] + eigenvalues = np.maximum(eigenvalues_raw[order], 0.0) / float(max(rows - 1, 1)) + total = float(np.sum(eigenvalues)) + explained = eigenvalues / total if total > 0 else np.zeros_like(eigenvalues) + scores = eigenvectors[:, order[:2]] * np.sqrt(np.maximum(eigenvalues_raw[order[:2]], 0.0)) + projection = scores + else: + # When rows outnumber a wide feature dimension, the feature covariance + # is smaller than the rows-by-rows Gram matrix. + covariance = centered.T @ centered + eigenvalues_raw, eigenvectors = np.linalg.eigh(covariance) + order = np.argsort(eigenvalues_raw)[::-1] + eigenvalues = np.maximum(eigenvalues_raw[order], 0.0) / float(max(rows - 1, 1)) + total = float(np.sum(eigenvalues)) + explained = eigenvalues / total if total > 0 else np.zeros_like(eigenvalues) + components = eigenvectors[:, order[:2]] + projection = centered @ components + if projection.shape[1] < 2: + projection = np.pad(projection, ((0, 0), (0, 2 - projection.shape[1]))) + if projection.shape[0] == 0: + return None + return eigenvalues, projection, explained + + +def _mean_pairwise_cosine(matrix: np.ndarray) -> Optional[float]: + matrix = np.asarray(matrix, dtype=np.float64) + rows = int(matrix.shape[0]) + if rows < 2: + return None + # Sum identity: sum_{i != j} u_i . u_j = ||sum_i u_i||^2 - sum_i ||u_i||^2. + # This matches the pairwise definition (zero-norm rows contribute nothing) + # without materializing the O(M^2) Gram matrix of the reduction sample; + # normalized rows are also accumulated in bounded blocks. + total = np.zeros(int(matrix.shape[1]), dtype=np.float64) + squared = 0.0 + for start in range(0, rows, _MEAN_COSINE_BLOCK_ROWS): + chunk = matrix[start : start + _MEAN_COSINE_BLOCK_ROWS] + norms = np.linalg.norm(chunk, axis=1) + safe = np.where(norms > 0, norms, 1.0) + normalized = chunk / safe[:, None] + total += normalized.sum(axis=0) + squared += float(np.einsum("ij,ij->", normalized, normalized)) + mean = (float(total @ total) - squared) / float(rows * (rows - 1)) + if not math.isfinite(mean): + return None + return float(np.clip(mean, -1.0, 1.0)) + + +def _kmeans( + points: np.ndarray, k: int, rng: np.random.Generator, max_iterations: int = 50 +) -> Tuple[np.ndarray, np.ndarray, float, int]: + points = np.asarray(points, dtype=np.float64) + rows = points.shape[0] + k = max(1, min(int(k), rows)) + initial = sample_indices(rows, k, rng) + centers = points[initial].copy() + labels = np.zeros(rows, dtype=np.int64) + iterations = 0 + for iterations in range(1, max_iterations + 1): + distances = np.linalg.norm(points[:, None, :] - centers[None, :, :], axis=2) + new_labels = np.argmin(distances, axis=1) + if np.array_equal(new_labels, labels) and iterations > 1: + labels = new_labels + break + labels = new_labels + for cluster in range(k): + members = points[labels == cluster] + if members.size: + centers[cluster] = members.mean(axis=0) + else: + farthest = int(np.argmax(np.min(distances, axis=1))) + centers[cluster] = points[farthest] + distances = np.linalg.norm(points[:, None, :] - centers[None, :, :], axis=2) + labels = np.argmin(distances, axis=1) + inertia = float(np.sum(np.min(distances, axis=1) ** 2)) + sizes = np.bincount(labels, minlength=k) + return labels, sizes, inertia, iterations + + +# --------------------------------------------------------------------------- +# Misc +# --------------------------------------------------------------------------- +def _digest(payload: bytes) -> bytes: + return hashlib.blake2b(payload, digest_size=_DUPLICATE_DIGEST_BYTES).digest() diff --git a/tools/dataset_profiler/dataset_profiler/report.py b/tools/dataset_profiler/dataset_profiler/report.py new file mode 100644 index 0000000000..1110b9f538 --- /dev/null +++ b/tools/dataset_profiler/dataset_profiler/report.py @@ -0,0 +1,130 @@ +# Copyright 2024-present the vsag project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Versioned JSON report assembly and metric definitions. + +Every rendered metric carries a human-readable ``definition``, a ``scope`` +(``"full"`` or ``"sample"``), and the ``sample_size`` used when the scope is a +sample. This keeps full-dataset and sampled statistics visibly distinct in +both the JSON report and the HTML visualization. +""" + +from __future__ import annotations + +from typing import Any, Dict, List, Optional + +from . import SCHEMA_VERSION, TOOL_NAME, __version__ +from .hdf5 import DatasetMetadata + +DEFINITIONS: Dict[str, str] = { + "norms": ( + "L2 norm ||x||_2 of every vector, computed over all finite values in the " + "selected split." + ), + "dimensions": ( + "Per-coordinate count, mean, standard deviation, minimum, maximum, and " + "zero ratio over the selected split." + ), + "sparsity": ( + "Count and fraction of finite exactly-zero entries; non-zero values and " + "density count finite non-zero entries, while NaN/Inf values are reported " + "separately in invalid_values. For sparse vectors this also reports non-zero " + "entries per vector and density = nnz / (count * dim)." + ), + "duplicates": ( + "Exact duplicate detection by hashing each complete vector; near-duplicate " + "pairs use Euclidean distance for inner-product inputs and the chosen " + "distance otherwise, below a relative threshold derived from the sampled " + "duplicate-detection distance median." + ), + "pairwise_distances": ( + "Distribution of distances between distinct vectors in a bounded random " + "sample. The full M x M matrix is allocated only for the configured sample " + "size, never for the complete dataset." + ), + "nearest_neighbor_distances": ( + "For each vector in the bounded sample, the distance to its nearest other " + "vector in the same sample (diagonal excluded)." + ), + "invalid_values": ( + "Counts of NaN/Inf values in the selected split and the number of vectors " + "containing at least one such value. Invalid entries are excluded from " + "numeric summaries and treated as zero in norms." + ), + "outliers": ( + "Robust norm outliers: vectors whose norm has a modified z-score above 3.5, " + "using median and median absolute deviation. Invalid (NaN/Inf) value counts " + "are reported separately." + ), + "anisotropy": ( + "Eigenvalue concentration of the centered sample covariance: the largest " + "eigenvalue share, top-10 share, effective rank, and mean pairwise cosine " + "similarity. High top-1 share indicates a dominant direction." + ), + "clustering": ( + "Optional deterministic PCA-2 projection of the sample and optional k-means " + "cluster summary over that projection." + ), +} + + +def metric_entry( + definition: str, + scope: str, + sample_size: Optional[int] = None, + **fields: Any +) -> Dict[str, Any]: + """Create a metric dictionary with its provenance metadata.""" + entry: Dict[str, Any] = { + "definition": definition, + "scope": scope, + "sample_size": sample_size, + } + entry.update(fields) + return entry + + +def build_report( + metadata: DatasetMetadata, + config: Dict[str, Any], + metrics: Dict[str, Any], + warnings: List[str], +) -> Dict[str, Any]: + """Assemble the deterministic, versioned profile report.""" + return { + "schema_version": SCHEMA_VERSION, + "tool": { + "name": TOOL_NAME, + "version": __version__, + }, + "input": { + "file_name": metadata.file_name, + "file_size_bytes": metadata.file_size_bytes, + "split": metadata.split, + }, + "dataset": { + "vector_type": metadata.vector_type, + "dtype": metadata.dtype, + "count": metadata.count, + "dim": metadata.dim, + "dim_inferred": metadata.dim_inferred, + "distance_metric": metadata.distance, + "has_offsets": metadata.has_offsets, + "datasets": list(metadata.datasets), + "attributes": dict(metadata.attributes), + }, + "config": dict(config), + "metrics": metrics, + "warnings": list(warnings), + } diff --git a/tools/dataset_profiler/dataset_profiler/sampling.py b/tools/dataset_profiler/dataset_profiler/sampling.py new file mode 100644 index 0000000000..c7a15a6cac --- /dev/null +++ b/tools/dataset_profiler/dataset_profiler/sampling.py @@ -0,0 +1,79 @@ +# Copyright 2024-present the vsag project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Deterministic index sampling helpers. + +Sampling must be reproducible across runs, hosts, and whether or not a sparse +file carries its optional ``*_offsets`` datasets. All helpers therefore use a +caller-supplied :class:`numpy.random.Generator` seeded with an explicit integer +and do not depend on global NumPy state. +""" + +from __future__ import annotations + +from typing import Optional + +import numpy as np + +SAMPLING_METHOD = ( + "numpy.random.Generator(PCG64); Floyd's algorithm without replacement; " + "indices sorted ascending before use" +) + + +def make_rng(seed: int) -> np.random.Generator: + """Create the deterministic random generator used by the profiler.""" + return np.random.Generator(np.random.PCG64(int(seed))) + + +def sample_indices(n: int, k: int, rng: np.random.Generator) -> np.ndarray: + """Return ``k`` distinct indices from ``[0, n)`` in ascending order. + + Floyd's algorithm is used so that memory and runtime are bounded by ``k`` + rather than by ``n``, which matters for sparse datasets whose offsets are + not available and which are scanned streaming. ``k >= n`` returns every + index. + """ + n = int(n) + k = int(k) + if n < 0: + raise ValueError("n must be non-negative") + if k < 0: + raise ValueError("k must be non-negative") + if k >= n: + return np.arange(n, dtype=np.int64) + selected = set() + for j in range(n - k, n): + t = int(rng.integers(0, j + 1)) + if t in selected: + selected.add(j) + else: + selected.add(t) + return np.array(sorted(selected), dtype=np.int64) + + +def take_prefix_sample(sample: np.ndarray, limit: Optional[int]) -> np.ndarray: + """Bound a previously drawn sample by ``limit`` rows. + + ``sample`` is already a uniformly random subset of the dataset, so taking + its first rows is itself a deterministic uniform sample. This keeps the + pairwise-distance phase bounded by ``pairwise_sample_size`` without + allocating another index set. + """ + if limit is None: + return sample + limit = int(limit) + if limit <= 0: + raise ValueError("pairwise sample size must be positive") + return sample[:limit] diff --git a/tools/dataset_profiler/dataset_profiler/stats.py b/tools/dataset_profiler/dataset_profiler/stats.py new file mode 100644 index 0000000000..8094921d5c --- /dev/null +++ b/tools/dataset_profiler/dataset_profiler/stats.py @@ -0,0 +1,217 @@ +# Copyright 2024-present the vsag project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Small numeric helpers shared by the profiler and report writers.""" + +from __future__ import annotations + +import json +import math +from typing import Any, Dict, Iterable, List, Optional + +import numpy as np + +PERCENTILES = (1.0, 5.0, 25.0, 50.0, 75.0, 95.0, 99.0) +HISTOGRAM_BINS = 20 + + +def as_finite_float(value: Any) -> Optional[float]: + """Convert a NumPy or Python number to a JSON-safe float.""" + if value is None: + return None + try: + number = float(value) + except (TypeError, ValueError): + return None + if not math.isfinite(number): + return None + # Preserve small finite magnitudes: fixed-decimal rounding would serialize + # values such as 5e-11 as zero in JSON, HTML, and AI payloads. + return number + + +def _as_float64_array(values: Iterable[float]) -> np.ndarray: + """Convert values without boxing ndarray elements into a Python list.""" + if isinstance(values, np.ndarray): + return np.asarray(values, dtype=np.float64) + return np.fromiter(values, dtype=np.float64) + + +def summarize(values: Iterable[float]) -> Dict[str, Any]: + """Return count/mean/std/min/max/percentiles for finite values.""" + array = _as_float64_array(values) + array = array[np.isfinite(array)] + out: Dict[str, Any] = {"count": int(array.size)} + if array.size == 0: + out.update( + { + "mean": None, + "std": None, + "min": None, + "max": None, + } + ) + for percentile in PERCENTILES: + out["p%g" % percentile] = None + return out + out.update( + { + "mean": as_finite_float(np.mean(array)), + "std": as_finite_float(np.std(array)), + "min": as_finite_float(np.min(array)), + "max": as_finite_float(np.max(array)), + } + ) + for percentile in PERCENTILES: + key = "p%g" % percentile + out[key] = as_finite_float(np.percentile(array, percentile, method="linear")) + return out + + +def histogram(values: Iterable[float], bins: int = HISTOGRAM_BINS) -> Dict[str, Any]: + """Return histogram edges/counts safe for JSON and SVG rendering.""" + array = _as_float64_array(values) + array = array[np.isfinite(array)] + if array.size == 0: + return {"bins": [], "counts": [], "bin_count": 0} + bin_count = int(bins) + if np.all(array == array[0]): + # NumPy's automatic constant-range padding is +/-0.5, which rounds + # away at large magnitudes. Supply a scale-aware representable range. + value = float(array[0]) + padding = max(abs(value) * 1e-3, math.ulp(value) * 4.0, 0.5) + lower = value - padding + upper = value + padding + if math.isfinite(lower) and math.isfinite(upper) and lower < upper: + counts, edges = np.histogram( + array, bins=bin_count, range=(lower, upper) + ) + else: + # A one-ulp range cannot be split into ``bin_count`` bins. Build + # explicit, one-sided edges instead, always stepping toward the + # finite interior so even +/-DBL_MAX cannot overflow. + direction = -np.inf if value > 0 else np.inf + edges = np.empty(bin_count + 1, dtype=np.float64) + edges[0] = value + for index in range(1, bin_count + 1): + edges[index] = np.nextafter(edges[index - 1], direction) + if direction == -np.inf: + edges = edges[::-1] + counts, edges = np.histogram(array, bins=edges) + else: + # Automatic edges can repeat when a nonconstant range spans only a few + # float64 ULPs. Depending on the NumPy version, np.histogram either + # raises or returns duplicate edges. Probe only the linspace candidates + # instead, then scan the full array once with the largest usable edges. + first_edge = np.min(array) + last_edge = np.max(array) + while True: + edges = np.linspace(first_edge, last_edge, bin_count + 1) + if bin_count <= 1 or np.all(edges[1:] > edges[:-1]): + break + bin_count -= 1 + counts, edges = np.histogram(array, bins=edges) + return { + "bins": [as_finite_float(edge) for edge in edges], + "counts": [int(count) for count in counts], + "bin_count": bin_count, + } + + +def downsample(values: np.ndarray, max_points: int = 1024) -> Dict[str, Any]: + """Downsample a 1-D array for report rendering. + + Returns the (possibly reduced) values plus metadata describing whether the + array was reduced and how many source dimensions each output point covers. + """ + array = np.asarray(values) + length = int(array.size) + if length <= max_points: + return { + "values": [as_finite_float(value) for value in array], + "source_length": length, + "downsampled": False, + "bucket_size": 1, + } + boundaries = np.linspace(0, length, max_points + 1, dtype=np.int64) + reduced: List[Optional[float]] = [] + for start, end in zip(boundaries[:-1], boundaries[1:]): + if end <= start: + reduced.append(None) + continue + chunk = np.asarray(array[start:end], dtype=np.float64) + finite = chunk[np.isfinite(chunk)] + reduced.append(as_finite_float(np.mean(finite)) if finite.size else None) + return { + "values": reduced, + "source_length": length, + "downsampled": True, + "bucket_size": int(math.ceil(length / float(max_points))), + } + + +def top_indices(values: np.ndarray, k: int, largest: bool = True) -> List[int]: + """Return indices of the top-``k`` entries, ignoring non-finite values.""" + array = np.asarray(values, dtype=np.float64) + finite = np.isfinite(array) + if not np.any(finite): + return [] + candidates = np.where(finite)[0] + order = np.argsort(array[candidates], kind="stable") + if largest: + order = order[::-1] + return [int(candidates[i]) for i in order[: int(k)]] + + +def jsonable(value: Any) -> Any: + """Recursively convert NumPy values and non-finite floats for JSON.""" + if isinstance(value, dict): + return {str(key): jsonable(item) for key, item in value.items()} + if isinstance(value, (list, tuple)): + return [jsonable(item) for item in value] + if isinstance(value, np.ndarray): + return [jsonable(item) for item in value.tolist()] + if isinstance(value, (np.integer,)): + return int(value) + if isinstance(value, (np.floating, float)): + return as_finite_float(value) + if isinstance(value, (np.bool_, bool)): + return bool(value) + return value + + +def dump_json(report: Dict[str, Any]) -> str: + """Serialize a report deterministically (sorted keys, fixed indentation).""" + return json.dumps( + jsonable(report), indent=2, sort_keys=True, ensure_ascii=False, allow_nan=False + ) + "\n" + + +def metric_keys(report: Dict[str, Any]) -> List[str]: + """Enumerate dotted evidence keys for every metric leaf in a report.""" + keys: List[str] = [] + + def walk(node: Any, prefix: str) -> None: + if isinstance(node, dict): + for key in sorted(node): + walk(node[key], "%s.%s" % (prefix, key) if prefix else str(key)) + return + if isinstance(node, (list, tuple)): + if node and all(not isinstance(item, (dict, list, tuple)) for item in node): + keys.append(prefix) + return + keys.append(prefix) + + walk(report.get("metrics", {}), "metrics") + return keys diff --git a/tools/dataset_profiler/examples/.gitignore b/tools/dataset_profiler/examples/.gitignore new file mode 100644 index 0000000000..ed8e9547b4 --- /dev/null +++ b/tools/dataset_profiler/examples/.gitignore @@ -0,0 +1,2 @@ +# Generated demo datasets (regenerate with make_toy_datasets.py) +output/*.hdf5 diff --git a/tools/dataset_profiler/examples/README.md b/tools/dataset_profiler/examples/README.md new file mode 100644 index 0000000000..be6acf993e --- /dev/null +++ b/tools/dataset_profiler/examples/README.md @@ -0,0 +1,46 @@ +# Dataset profiler examples + +Reproducible demo datasets and reports for `vsag-dataset-profiler`. + +## Contents + +| File | Description | +| --- | --- | +| `make_toy_datasets.py` | Generates the toy datasets below and profiles them with a fixed seed. | +| `output/toy_dense.hdf5` | 120 clustered `FLOAT32` vectors (16-D) with duplicates, an all-zero vector, an outlier, and one NaN (generated locally; not committed). | +| `output/toy_sparse.hdf5` | 82 sparse records with `train_offsets`, including a duplicate and an empty record (generated locally; not committed). | +| `output/toy_dense.profile.json` | Versioned deterministic JSON report for the dense file. | +| `output/toy_dense.profile.html` | Self-contained interactive HTML report for the dense file. | +| `output/toy_sparse.profile.json` / `output/toy_sparse.profile.html` | The same pair for the sparse file. | + +## Reproduce + +```bash +cd tools/dataset_profiler +pip install -r requirements.txt +python examples/make_toy_datasets.py # writes into examples/output/ +``` + +The generation seed and the profiling seed are fixed in the script, so re-running it on the +same NumPy build reproduces the committed reports byte-for-byte (verified with NumPy 2.4 / +Python 3.11; SVD sign conventions may differ across BLAS builds). + +## Profile your own dataset + +```bash +python -m dataset_profiler /path/to/your.hdf5 --split train --seed 42 \ + --out-json your.profile.json --out-html your.profile.html +``` + +To profile arrays that you currently only have in memory (for example the same arrays you +pass to `pyvsag`), first export them to the HDF5 layout described in +[`dataset_format.md`](../../../docs/docs/en/src/resources/dataset_format.md): + +```python +import h5py, numpy as np + +vectors = np.load("train.npy").astype("float32") # your (N, D) array +with h5py.File("my_dataset.hdf5", "w") as f: + f.attrs["distance"] = "euclidean" + f.create_dataset("train", data=vectors) +``` diff --git a/tools/dataset_profiler/examples/make_toy_datasets.py b/tools/dataset_profiler/examples/make_toy_datasets.py new file mode 100644 index 0000000000..e871c2a407 --- /dev/null +++ b/tools/dataset_profiler/examples/make_toy_datasets.py @@ -0,0 +1,145 @@ +# Copyright 2024-present the vsag project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Generate reproducible toy dense/sparse HDF5 datasets and profile reports. + +This is the demo entry point for the dataset profiler. It writes: + +* ``toy_dense.hdf5`` -- clustered float32 vectors with duplicates, an + all-zero vector, an outlier, and one NaN value; +* ``toy_sparse.hdf5`` -- sparse records with ``train_offsets``, a duplicate + record, and one empty record; +* ``toy_dense.profile.{json,html}`` and ``toy_sparse.profile.{json,html}`` + produced from the two files with a fixed seed. + +Run from anywhere:: + + python tools/dataset_profiler/examples/make_toy_datasets.py --outdir out + +Everything is deterministic for a given NumPy build; committed example reports +under ``examples/output/`` were produced with NumPy 2.4 / Python 3.11. +""" + +from __future__ import annotations + +import argparse +import struct +import sys +from pathlib import Path + +import numpy as np + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from dataset_profiler.html_report import render_html # noqa: E402 +from dataset_profiler.profiler import profile_dataset # noqa: E402 +from dataset_profiler.stats import dump_json # noqa: E402 + +SEED = 20260107 +_PROFILE_ARGS = dict(seed=42, sample_size=64, pairwise_sample_size=32, clusters=4) + + +def write_dense_dataset(path: Path) -> None: + import h5py + + rng = np.random.Generator(np.random.PCG64(SEED)) + centers = np.asarray( + [ + rng.normal(0.0, 1.0, size=16), + rng.normal(0.0, 1.0, size=16) + 3.5, + rng.normal(0.0, 1.0, size=16) - 3.5, + ], + dtype=np.float32, + ) + blocks = [] + for center in centers: + blocks.append((center[None, :] + 0.4 * rng.normal(0.0, 1.0, size=(40, 16))).astype(np.float32)) + data = np.concatenate(blocks, axis=0) + data[5] = data[0] # exact duplicate + data[9] = data[0] + data[11] = 0.0 # all-zero vector + data[13] *= 300.0 # outlier norms + data[17, 3] = np.float32("nan") # one invalid value + with h5py.File(str(path), "w") as f: + f.attrs["distance"] = "euclidean" + f.create_dataset("train", data=data) + f.create_dataset("test", data=data[:8]) + + +def _sparse_records(rng: np.random.Generator): + records = [] + for _ in range(80): + length = int(rng.integers(3, 12)) + ids = np.sort(rng.choice(128, size=length, replace=False)).astype(np.uint32) + values = rng.uniform(0.05, 1.0, size=length).astype(np.float32) + records.append((ids, values)) + return records + + +def write_sparse_dataset(path: Path) -> None: + import h5py + + rng = np.random.Generator(np.random.PCG64(SEED + 1)) + records = _sparse_records(rng) + records.append((records[0][0].copy(), records[0][1].copy())) # duplicate record + records.append((np.zeros(0, dtype=np.uint32), np.zeros(0, dtype=np.float32))) # empty record + + stream = bytearray() + offsets = [] + for ids, values in records: + offsets.append(len(stream)) + stream += struct.pack(" None: + report = profile_dataset(str(path), **_PROFILE_ARGS) + stem = path.with_suffix("").name + json_path = outdir / (stem + ".profile.json") + html_path = outdir / (stem + ".profile.html") + json_path.write_text(dump_json(report), encoding="utf-8") + html_path.write_text(render_html(report), encoding="utf-8") + print("wrote %s (%d warnings)" % (json_path, len(report["warnings"]))) + print("wrote %s" % html_path) + + +def main(argv=None) -> int: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument( + "--outdir", + default=str(Path(__file__).resolve().parent / "output"), + help="directory for the generated .hdf5 files and reports (default: examples/output)", + ) + args = parser.parse_args(argv) + outdir = Path(args.outdir) + outdir.mkdir(parents=True, exist_ok=True) + + dense = outdir / "toy_dense.hdf5" + sparse = outdir / "toy_sparse.hdf5" + write_dense_dataset(dense) + write_sparse_dataset(sparse) + profile_to(dense, outdir) + profile_to(sparse, outdir) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/dataset_profiler/examples/output/toy_dense.profile.html b/tools/dataset_profiler/examples/output/toy_dense.profile.html new file mode 100644 index 0000000000..c45c22e1d9 --- /dev/null +++ b/tools/dataset_profiler/examples/output/toy_dense.profile.html @@ -0,0 +1,106 @@ + + + + + + +VSAG dataset profile: toy_dense.hdf5 + + + +
    +

    Dataset Feature Profile

    toy_dense.hdf5 · 120 vectors · 16 dimensions · DENSE

    schema 1.0seed 42sample 64pairwise sample 32metric euclidean

    Deterministic offline report. No raw vectors were sent anywhere while generating this file.

    +

    Overview

    Vectors
    120
    full dataset
    Dimensions
    16
    full dataset
    Element type
    FLOAT32
    full dataset
    Distance metric
    euclidean
    profiling metric
    Norm median
    15.4
    full dataset
    Zero ratio
    0.833%
    full dataset
    Exact duplicate ratio
    1.667%
    full dataset
    Invalid values
    1
    full dataset
    Norm outlier ratio
    33.333%
    full dataset
    Top eigenvalue share
    0.995
    sample
    +

    Vector norms

    Scope: fullSample size: n/a

    Definition: L2 norm ||x||_2 of every vector, computed over all finite values in the selected split.

    Vectorsnorm 0 – 68.8885: 119norm 68.8885 – 137.777: 0norm 137.777 – 206.665: 0norm 206.665 – 275.554: 0norm 275.554 – 344.442: 0norm 344.442 – 413.331: 0norm 413.331 – 482.219: 0norm 482.219 – 551.108: 0norm 551.108 – 619.996: 0norm 619.996 – 688.885: 0norm 688.885 – 757.773: 0norm 757.773 – 826.661: 0norm 826.661 – 895.55: 0norm 895.55 – 964.438: 0norm 964.438 – 1033.33: 0norm 1033.33 – 1102.22: 0norm 1102.22 – 1171.1: 0norm 1171.1 – 1239.99: 0norm 1239.99 – 1308.88: 0norm 1308.88 – 1377.77: 101378L2 norm1190
    +

    Per-dimension distributions

    Scope: fullSample size: n/a

    Definition: Per-coordinate count, mean, standard deviation, minimum, maximum, and zero ratio over the selected split.

    dimension mean; 16 plotted point(s).

    Meandimension 0: -7.66733dimension 1: -3.03324dimension 2: -1.89663dimension 3: 2.47286dimension 4: -0.345023dimension 5: -2.48299dimension 6: 2.80795dimension 7: 2.69899dimension 8: 1.69641dimension 9: -1.65213dimension 10: -1.10091dimension 11: 1.46622dimension 12: -0.802047dimension 13: -6.76611dimension 14: -1.93973dimension 15: -3.50863015Coordinate2.808-7.667

    dimension standard deviation; 16 plotted point(s).

    Stddimension 0: 78.1093dimension 1: 29.8536dimension 2: 23.8668dimension 3: 19.1549dimension 4: 4.08406dimension 5: 24.2897dimension 6: 23.7184dimension 7: 29.8089dimension 8: 13.2363dimension 9: 8.97964dimension 10: 8.10975dimension 11: 18.765dimension 12: 8.51057dimension 13: 56.9357dimension 14: 26.1913dimension 15: 34.1021015Coordinate78.114.084

    dimension zero ratio; 16 plotted point(s).

    Zero ratiodimension 0: 0.00833333dimension 1: 0.00833333dimension 2: 0.00833333dimension 3: 0.00840336dimension 4: 0.00833333dimension 5: 0.00833333dimension 6: 0.00833333dimension 7: 0.00833333dimension 8: 0.00833333dimension 9: 0.00833333dimension 10: 0.00833333dimension 11: 0.00833333dimension 12: 0.00833333dimension 13: 0.00833333dimension 14: 0.00833333dimension 15: 0.00833333015Coordinate0.0084030.008333

    Highest-variance dimensions

    DimensionMeanStdZero ratio
    0-7.66778.110.833%
    13-6.76656.940.833%
    15-3.50934.10.833%
    1-3.03329.850.833%
    72.69929.810.833%
    14-1.9426.190.833%
    5-2.48324.290.833%
    2-1.89723.870.833%
    62.80823.720.833%
    32.47319.150.840%
    +

    Sparsity

    Scope: fullSample size: n/a

    Definition: Count and fraction of finite exactly-zero entries; non-zero values and density count finite non-zero entries, while NaN/Inf values are reported separately in invalid_values. For sparse vectors this also reports non-zero entries per vector and density = nnz / (count * dim).

    Representationdense
    Total cells1920
    Zero values16
    Non-zero values1903
    Density99.115%
    Zero ratio0.833%
    All-zero vectors1

    +

    Duplicate and near-duplicate indicators

    Scope: mixedSample size: n/a

    Definition: Exact duplicate detection by hashing each complete vector; near-duplicate pairs use Euclidean distance for inner-product inputs and the chosen distance otherwise, below a relative threshold derived from the sampled duplicate-detection distance median.

    Unique vectors118
    Duplicate vectors (exact)2
    Duplicate ratio (exact)1.667%
    Duplicate groups1
    Largest duplicate group3

    Near-duplicate (sampled)

    Near-duplicate sampled pairs0
    Near-duplicate pair ratio0.000%
    Threshold0.01586
    Median sampled pairwise distance15.86
    Pairs sampled496

    exact duplicate detection covers the full split; the near-duplicate indicator covers the bounded pairwise sample

    +

    Pairwise distance distribution

    Scope: sampleSample size: 32

    Definition: Distribution of distances between distinct vectors in a bounded random sample. The full M x M matrix is allocated only for the configured sample size, never for the complete dataset.

    Pairspairwise distance 1.11464 – 2.63555: 149pairwise distance 2.63555 – 4.15645: 8pairwise distance 4.15645 – 5.67736: 0pairwise distance 5.67736 – 7.19827: 0pairwise distance 7.19827 – 8.71917: 0pairwise distance 8.71917 – 10.2401: 0pairwise distance 10.2401 – 11.761: 0pairwise distance 11.761 – 13.2819: 0pairwise distance 13.2819 – 14.8028: 5pairwise distance 14.8028 – 16.3237: 125pairwise distance 16.3237 – 17.8446: 110pairwise distance 17.8446 – 19.3655: 0pairwise distance 19.3655 – 20.8864: 0pairwise distance 20.8864 – 22.4073: 0pairwise distance 22.4073 – 23.9282: 0pairwise distance 23.9282 – 25.4491: 0pairwise distance 25.4491 – 26.97: 0pairwise distance 26.97 – 28.491: 0pairwise distance 28.491 – 30.0119: 9pairwise distance 30.0119 – 31.5328: 901.11531.53Distance between sampled pairs1490
    count496
    mean14.6
    std10.12
    min1.115
    p11.493
    p51.725
    p252.383
    p5015.86
    p7517.19
    p9530.98
    p9931.34
    max31.53

    The distance matrix is bounded by the configured pairwise sample size.

    Nearest-neighbor distance distribution

    Scope: sampleSample size: 32

    Definition: For each vector in the bounded sample, the distance to its nearest other vector in the same sample (diagonal excluded).

    Vectorsnn distance 1.11464 – 1.1563: 2nn distance 1.1563 – 1.19796: 0nn distance 1.19796 – 1.23962: 0nn distance 1.23962 – 1.28128: 2nn distance 1.28128 – 1.32293: 1nn distance 1.32293 – 1.36459: 2nn distance 1.36459 – 1.40625: 0nn distance 1.40625 – 1.44791: 0nn distance 1.44791 – 1.48957: 1nn distance 1.48957 – 1.53123: 1nn distance 1.53123 – 1.57289: 2nn distance 1.57289 – 1.61454: 5nn distance 1.61454 – 1.6562: 4nn distance 1.6562 – 1.69786: 0nn distance 1.69786 – 1.73952: 3nn distance 1.73952 – 1.78118: 3nn distance 1.78118 – 1.82284: 1nn distance 1.82284 – 1.86449: 2nn distance 1.86449 – 1.90615: 1nn distance 1.90615 – 1.94781: 21.1151.948Distance to nearest other sampled vector50
    count32
    mean1.596
    std0.2218
    min1.115
    p11.115
    p51.189
    p251.491
    p501.612
    p751.751
    p951.916
    p991.945
    max1.948
    +

    Anisotropy

    Scope: sampleSample size: 64

    Definition: Eigenvalue concentration of the centered sample covariance: the largest eigenvalue share, top-10 share, effective rank, and mean pairwise cosine similarity. High top-1 share indicates a dominant direction.

    PC variance share.

    Explained variance ratio of leading principal componentsPC1: 0.994974PC1PC2: 0.00489536PC2PC3: 6.9245e-05PC3PC4: 9.13283e-06PC4PC5: 7.263e-06PC5PC6: 6.62456e-06PC6PC7: 6.29738e-06PC7PC8: 6.21929e-06PC8PC9: 5.03303e-06PC9PC10: 4.58376e-06PC100.9950

    Top-1 eigenvalue share0.995
    Top-10 eigenvalue share1
    Effective rank1.033
    Mean pairwise cosine0.08111

    top1_eigenvalue_share near 1 indicates a dominant direction; effective_rank near 1 indicates nearly one-dimensional variation

    +

    Projection and clustering

    Scope: sampleSample size: 64

    Definition: Optional deterministic PCA-2 projection of the sample and optional k-means cluster summary over that projection.

    PCA-2 projection of the seeded sample.

    PC2sample 0 · cluster 1 · (-18.7, -0.101)sample 1 · cluster 1 · (-18.77, -0.1246)sample 2 · cluster 1 · (-19.33, -0.9355)sample 3 · cluster 1 · (-18.52, -0.3024)sample 4 · cluster 1 · (-18.77, -0.1246)sample 5 · cluster 0 · (1355, -2.693)sample 6 · cluster 1 · (-19.26, 0.09897)sample 7 · cluster 1 · (-18.72, -0.1534)sample 8 · cluster 1 · (-18.91, 0.08807)sample 9 · cluster 1 · (-18.35, -0.2309)sample 10 · cluster 1 · (-18.89, -0.5661)sample 11 · cluster 1 · (-18.42, -0.6214)sample 12 · cluster 1 · (-19.49, -0.8252)sample 13 · cluster 1 · (-18.91, -0.1685)sample 14 · cluster 1 · (-19.26, -0.9399)sample 15 · cluster 1 · (-19.38, -0.29)sample 16 · cluster 1 · (-19.16, -0.69)sample 17 · cluster 1 · (-18.91, 0.06722)sample 18 · cluster 1 · (-18.6, -0.3961)sample 19 · cluster 2 · (-28.12, -13.04)sample 20 · cluster 2 · (-28, -14.16)sample 21 · cluster 3 · (-28.79, -13.65)sample 22 · cluster 2 · (-27.58, -13.7)sample 23 · cluster 2 · (-28.22, -13.24)sample 24 · cluster 3 · (-28.92, -13.95)sample 25 · cluster 2 · (-28.11, -13.31)sample 26 · cluster 3 · (-28.2, -13.97)sample 27 · cluster 2 · (-27.9, -13.61)sample 28 · cluster 2 · (-27.73, -13.64)sample 29 · cluster 3 · (-28.23, -14.28)sample 30 · cluster 2 · (-28.05, -13.5)sample 31 · cluster 3 · (-28.74, -13.55)sample 32 · cluster 3 · (-28.84, -13.56)sample 33 · cluster 3 · (-28.53, -13.87)sample 34 · cluster 3 · (-28.03, -14.4)sample 35 · cluster 3 · (-28.51, -13.73)sample 36 · cluster 2 · (-27.77, -13.69)sample 37 · cluster 3 · (-28.25, -14.14)sample 38 · cluster 3 · (-28.69, -13.65)sample 39 · cluster 2 · (-28.06, -13.85)sample 40 · cluster 2 · (-27.72, -13.97)sample 41 · cluster 3 · (-28.29, -14.06)sample 42 · cluster 1 · (-16.81, 14.92)sample 43 · cluster 1 · (-17.27, 14.98)sample 44 · cluster 1 · (-16.26, 14.55)sample 45 · cluster 1 · (-16.75, 14.51)sample 46 · cluster 1 · (-15.44, 15.09)sample 47 · cluster 1 · (-16.97, 14.54)sample 48 · cluster 1 · (-16.73, 15.43)sample 49 · cluster 1 · (-16.19, 14.27)sample 50 · cluster 1 · (-17.19, 14.07)sample 51 · cluster 1 · (-16.15, 14.66)sample 52 · cluster 1 · (-16.69, 15.01)sample 53 · cluster 1 · (-17.16, 14.69)sample 54 · cluster 1 · (-16.38, 15.33)sample 55 · cluster 1 · (-16.79, 15.01)sample 56 · cluster 1 · (-16.63, 14.96)sample 57 · cluster 1 · (-16.69, 15.06)sample 58 · cluster 1 · (-16.78, 14.36)sample 59 · cluster 1 · (-16.28, 14.92)sample 60 · cluster 1 · (-16.93, 14.97)sample 61 · cluster 1 · (-16.32, 14.5)sample 62 · cluster 1 · (-16.55, 14.81)sample 63 · cluster 1 · (-16.41, 14.73)PC1

    Cluster size.

    k-means cluster sizes on the PCA-2 projectionC0: 1C0C1: 40C1C2: 11C2C3: 12C3400
    Clusters4
    Inertia2334
    Iterations3
    +

    AI-assisted interpretation

    Disabled. The deterministic JSON and HTML reports were generated without network access.

    +

    Warnings

    • sampled vectors contain NaN/Inf entries; sampled metrics replace them with 0.0, which can distort pairwise, nearest-neighbor, anisotropy, and clustering results
    +

    Generated by vsag-dataset-profiler. Deterministic JSON and HTML outputs are produced offline. When AI analysis is enabled, only the bounded aggregate metric payload described in the technical report is sent to the configured provider; raw vectors and credentials are never included.

    + + + diff --git a/tools/dataset_profiler/examples/output/toy_dense.profile.json b/tools/dataset_profiler/examples/output/toy_dense.profile.json new file mode 100644 index 0000000000..f7a9e6d36b --- /dev/null +++ b/tools/dataset_profiler/examples/output/toy_dense.profile.json @@ -0,0 +1,972 @@ +{ + "config": { + "clusters": 4, + "effective_pairwise_sample_size": 32, + "effective_sample_size": 64, + "max_dimension_points": 1024, + "metric": "euclidean", + "near_duplicate_relative_epsilon": 0.001, + "pairwise_sample_size": 32, + "pairwise_sampling_method": "uniformly random subset selected by a seeded permutation of the sampled rows", + "sample_size": 64, + "sampling_method": "numpy.random.Generator(PCG64); Floyd's algorithm without replacement; indices sorted ascending before use", + "seed": 42 + }, + "dataset": { + "attributes": { + "distance": "euclidean" + }, + "count": 120, + "datasets": [ + "test", + "train" + ], + "dim": 16, + "dim_inferred": false, + "distance_metric": "euclidean", + "dtype": "FLOAT32", + "has_offsets": false, + "vector_type": "dense" + }, + "input": { + "file_name": "toy_dense.hdf5", + "file_size_bytes": 14336, + "split": "train" + }, + "metrics": { + "anisotropy": { + "available": true, + "definition": "Eigenvalue concentration of the centered sample covariance: the largest eigenvalue share, top-10 share, effective rank, and mean pairwise cosine similarity. High top-1 share indicates a dominant direction.", + "effective_rank": 1.0329965616385048, + "explained_variance_ratio": [ + 0.9949739695781928, + 0.004895360499264164, + 6.92450088586453e-05, + 9.13283391803934e-06, + 7.263004934639645e-06, + 6.624561846720796e-06, + 6.29737871941663e-06, + 6.219294295534202e-06, + 5.033033167079361e-06, + 4.58376244986289e-06 + ], + "interpretation": "top1_eigenvalue_share near 1 indicates a dominant direction; effective_rank near 1 indicates nearly one-dimensional variation", + "mean_pairwise_cosine": 0.0811067732379052, + "method": "eigendecomposition of centered sample covariance", + "sample_size": 64, + "scope": "sample", + "top10_eigenvalue_share": 0.9999837289556469, + "top1_eigenvalue_share": 0.9949739695781928 + }, + "clustering": { + "available": true, + "cluster_sizes": [ + 1, + 40, + 11, + 12 + ], + "clustering_available": true, + "definition": "Optional deterministic PCA-2 projection of the sample and optional k-means cluster summary over that projection.", + "explained_variance_ratio_pc1": 0.9949739695781928, + "explained_variance_ratio_pc2": 0.004895360499264164, + "inertia": 2334.0304705889184, + "iterations": 3, + "labels": [ + 1, + 1, + 1, + 1, + 1, + 0, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 2, + 2, + 3, + 2, + 2, + 3, + 2, + 3, + 2, + 2, + 3, + 2, + 3, + 3, + 3, + 3, + 3, + 2, + 3, + 3, + 2, + 2, + 3, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1 + ], + "method": "deterministic PCA-2 projection of the seeded sample", + "n_clusters": 4, + "projection_points": [ + [ + -18.695403001555583, + -0.10101175304085905 + ], + [ + -18.769532937984355, + -0.12464157401082006 + ], + [ + -19.33228049906296, + -0.9355128672244951 + ], + [ + -18.518979545484385, + -0.3024165193174056 + ], + [ + -18.769532937984355, + -0.12464157401082006 + ], + [ + 1354.986379702136, + -2.6929492833837325 + ], + [ + -19.264730381651916, + 0.098966097363606 + ], + [ + -18.71504625804072, + -0.1534110145652537 + ], + [ + -18.905889648963157, + 0.08807300820456172 + ], + [ + -18.346705895591818, + -0.2309442611352232 + ], + [ + -18.889026647154406, + -0.5660627726873928 + ], + [ + -18.418970512306984, + -0.6213816338874713 + ], + [ + -19.493204349778274, + -0.825174418426528 + ], + [ + -18.91263020980252, + -0.16853820121749657 + ], + [ + -19.264724668469906, + -0.9399309701524784 + ], + [ + -19.379425136952854, + -0.2900073497302022 + ], + [ + -19.16465098775358, + -0.6900419377129746 + ], + [ + -18.909405608402327, + 0.06722347076233509 + ], + [ + -18.600290246635122, + -0.3960761844816398 + ], + [ + -28.11537055060114, + -13.040556842409714 + ], + [ + -27.99572582148709, + -14.158501150083541 + ], + [ + -28.790436450319053, + -13.645616902565994 + ], + [ + -27.57978327187732, + -13.697967039602158 + ], + [ + -28.22407576061271, + -13.240087844319053 + ], + [ + -28.919998079002834, + -13.949671791654794 + ], + [ + -28.107913526262294, + -13.312765718887771 + ], + [ + -28.203758679635932, + -13.967478213529393 + ], + [ + -27.90072200277777, + -13.609004810912644 + ], + [ + -27.72788973222946, + -13.6389722795964 + ], + [ + -28.228664735558386, + -14.283874099598954 + ], + [ + -28.04649601644018, + -13.497469264725522 + ], + [ + -28.74339341788303, + -13.549554600479954 + ], + [ + -28.836531360538352, + -13.555018798907522 + ], + [ + -28.53037350993945, + -13.869599042225309 + ], + [ + -28.03470465629059, + -14.396250782843133 + ], + [ + -28.513675697224066, + -13.72749896254574 + ], + [ + -27.769352206030085, + -13.686837454109183 + ], + [ + -28.247062982103927, + -14.144079795020877 + ], + [ + -28.68662192043679, + -13.65426951940266 + ], + [ + -28.06182338014036, + -13.845396554779677 + ], + [ + -27.71740127906813, + -13.971714209586551 + ], + [ + -28.291501318634477, + -14.062034389101296 + ], + [ + -16.814886884345675, + 14.918647061672836 + ], + [ + -17.269215431713988, + 14.983839357566543 + ], + [ + -16.25834302592752, + 14.550771222237369 + ], + [ + -16.748385398791605, + 14.51300637687291 + ], + [ + -15.443135024325354, + 15.091725685356169 + ], + [ + -16.972143647427444, + 14.537468414769409 + ], + [ + -16.728219509951508, + 15.42953034649609 + ], + [ + -16.192340173300366, + 14.274698459521362 + ], + [ + -17.19322218295534, + 14.065718659948613 + ], + [ + -16.14720286658612, + 14.662807560622438 + ], + [ + -16.68752783901762, + 15.0076819334923 + ], + [ + -17.160130104898585, + 14.694205053401758 + ], + [ + -16.375204343682203, + 15.332733022245694 + ], + [ + -16.794500195387783, + 15.014946686830436 + ], + [ + -16.63478616231834, + 14.964349324323148 + ], + [ + -16.693390961897535, + 15.064750587076436 + ], + [ + -16.77521089039028, + 14.363963164972336 + ], + [ + -16.27807494372119, + 14.923352640989856 + ], + [ + -16.927504523607567, + 14.973162629146604 + ], + [ + -16.316966509053344, + 14.503898170007778 + ], + [ + -16.546196676381594, + 14.807853824500826 + ], + [ + -16.406086577786425, + 14.73358962349123 + ] + ], + "sample_size": 64, + "scope": "sample" + }, + "dimensions": { + "definition": "Per-coordinate count, mean, standard deviation, minimum, maximum, and zero ratio over the selected split.", + "dim": 16, + "per_dimension": { + "max": { + "bucket_size": 1, + "downsampled": false, + "source_length": 16, + "values": [ + 4.977198123931885, + 5.71360445022583, + 5.12531852722168, + 207.5524444580078, + 4.443779468536377, + 3.3880538940429688, + 258.8473205566406, + 325.96600341796875, + 142.93264770507812, + 4.768925189971924, + 2.9417226314544678, + 202.784423828125, + 5.191808700561523, + 3.283911943435669, + 5.215048789978027, + 4.44017219543457 + ] + }, + "mean": { + "bucket_size": 1, + "downsampled": false, + "source_length": 16, + "values": [ + -7.667329511046409, + -3.033241882446843, + -1.896628301590681, + 2.4728561803289373, + -0.34502301930915563, + -2.4829891121635836, + 2.8079502251930535, + 2.698991995553176, + 1.6964068959156673, + -1.6521264026096711, + -1.1009128115101097, + 1.4662248463680347, + -0.802047459280584, + -6.766114973028501, + -1.9397269538526112, + -3.5086287669837475 + ] + }, + "min": { + "bucket_size": 1, + "downsampled": false, + "source_length": 16, + "values": [ + -858.962646484375, + -325.7226257324219, + -260.0692138671875, + -3.7851216793060303, + -28.776506423950195, + -266.34765625, + -4.427289962768555, + -5.071621894836426, + -3.877328395843506, + -88.47776794433594, + -86.15872955322266, + -5.425839900970459, + -85.19577026367188, + -626.9149780273438, + -285.9913635253906, + -373.79412841796875 + ] + }, + "std": { + "bucket_size": 1, + "downsampled": false, + "source_length": 16, + "values": [ + 78.10926294950829, + 29.8535987506539, + 23.866797221092025, + 19.154852376201013, + 4.084058167250529, + 24.28966896949042, + 23.718413736138544, + 29.808896796502967, + 13.23627950524428, + 8.979640993394002, + 8.109751923872745, + 18.76500523457064, + 8.510566769752593, + 56.93568622823231, + 26.191342975590096, + 34.102054196788686 + ] + }, + "zero_ratio": { + "bucket_size": 1, + "downsampled": false, + "source_length": 16, + "values": [ + 0.008333333333333333, + 0.008333333333333333, + 0.008333333333333333, + 0.008403361344537815, + 0.008333333333333333, + 0.008333333333333333, + 0.008333333333333333, + 0.008333333333333333, + 0.008333333333333333, + 0.008333333333333333, + 0.008333333333333333, + 0.008333333333333333, + 0.008333333333333333, + 0.008333333333333333, + 0.008333333333333333, + 0.008333333333333333 + ] + } + }, + "sample_size": null, + "scope": "full", + "summary": { + "mean": { + "count": 16, + "max": 2.8079502251930535, + "mean": -1.2532711906539393, + "min": -7.667329511046409, + "p1": -7.532147330343723, + "p25": -2.6205523047343986, + "p5": -6.991418607532978, + "p50": -1.3765196070598904, + "p75": 1.523770358754943, + "p95": 2.7262315529631453, + "p99": 2.7916064907470717, + "std": 3.0154691254943824 + }, + "std": { + "count": 16, + "max": 78.10926294950829, + "mean": 25.482242299642692, + "min": 4.084058167250529, + "p1": 4.687912230743861, + "p25": 12.17211987728171, + "p5": 7.10332848471719, + "p50": 23.792605478615286, + "p75": 29.820072285040702, + "p95": 62.22908040855131, + "p99": 74.93322644131689, + "std": 18.428710901483733 + }, + "zero_ratio": { + "count": 16, + "max": 0.008403361344537815, + "mean": 0.008337710084033613, + "min": 0.008333333333333333, + "p1": 0.008333333333333333, + "p25": 0.008333333333333333, + "p5": 0.008333333333333333, + "p50": 0.008333333333333333, + "p75": 0.008333333333333333, + "p95": 0.008350840336134454, + "p99": 0.008392857142857143, + "std": 1.6951082572686462e-05 + } + }, + "top_variance_dimensions": [ + { + "dimension": 0, + "mean": -7.667329511046409, + "std": 78.10926294950829, + "zero_ratio": 0.008333333333333333 + }, + { + "dimension": 13, + "mean": -6.766114973028501, + "std": 56.93568622823231, + "zero_ratio": 0.008333333333333333 + }, + { + "dimension": 15, + "mean": -3.5086287669837475, + "std": 34.102054196788686, + "zero_ratio": 0.008333333333333333 + }, + { + "dimension": 1, + "mean": -3.033241882446843, + "std": 29.8535987506539, + "zero_ratio": 0.008333333333333333 + }, + { + "dimension": 7, + "mean": 2.698991995553176, + "std": 29.808896796502967, + "zero_ratio": 0.008333333333333333 + }, + { + "dimension": 14, + "mean": -1.9397269538526112, + "std": 26.191342975590096, + "zero_ratio": 0.008333333333333333 + }, + { + "dimension": 5, + "mean": -2.4829891121635836, + "std": 24.28966896949042, + "zero_ratio": 0.008333333333333333 + }, + { + "dimension": 2, + "mean": -1.896628301590681, + "std": 23.866797221092025, + "zero_ratio": 0.008333333333333333 + }, + { + "dimension": 6, + "mean": 2.8079502251930535, + "std": 23.718413736138544, + "zero_ratio": 0.008333333333333333 + }, + { + "dimension": 3, + "mean": 2.4728561803289373, + "std": 19.154852376201013, + "zero_ratio": 0.008403361344537815 + } + ], + "top_zero_dimensions": [ + { + "dimension": 3, + "mean": 2.4728561803289373, + "zero_ratio": 0.008403361344537815 + }, + { + "dimension": 15, + "mean": -3.5086287669837475, + "zero_ratio": 0.008333333333333333 + }, + { + "dimension": 14, + "mean": -1.9397269538526112, + "zero_ratio": 0.008333333333333333 + }, + { + "dimension": 13, + "mean": -6.766114973028501, + "zero_ratio": 0.008333333333333333 + }, + { + "dimension": 12, + "mean": -0.802047459280584, + "zero_ratio": 0.008333333333333333 + }, + { + "dimension": 11, + "mean": 1.4662248463680347, + "zero_ratio": 0.008333333333333333 + }, + { + "dimension": 10, + "mean": -1.1009128115101097, + "zero_ratio": 0.008333333333333333 + }, + { + "dimension": 9, + "mean": -1.6521264026096711, + "zero_ratio": 0.008333333333333333 + }, + { + "dimension": 8, + "mean": 1.6964068959156673, + "zero_ratio": 0.008333333333333333 + }, + { + "dimension": 7, + "mean": 2.698991995553176, + "zero_ratio": 0.008333333333333333 + } + ] + }, + "duplicates": { + "definition": "Exact duplicate detection by hashing each complete vector; near-duplicate pairs use Euclidean distance for inner-product inputs and the chosen distance otherwise, below a relative threshold derived from the sampled duplicate-detection distance median.", + "exact": { + "count": 120, + "definition": "Vectors whose complete byte representation is identical.", + "duplicate_groups": 1, + "duplicate_ratio": 0.016666666666666666, + "duplicate_vectors": 2, + "max_group_size": 3, + "sample_size": null, + "scope": "full", + "unique_vectors": 118 + }, + "near_duplicate": { + "available": true, + "definition": "Sampled pairs whose duplicate-detection distance is at most epsilon times the sampled pairwise-distance median.", + "median_pairwise_distance": 15.856817157369147, + "metric": "euclidean", + "near_duplicate_pairs": 0, + "near_duplicate_ratio": 0.0, + "pair_count": 496, + "relative_epsilon": 0.001, + "sample_size": 32, + "scope": "sample", + "threshold": 0.015856817157369148 + }, + "sample_size": null, + "scope": "mixed", + "scope_note": "exact duplicate detection covers the full split; the near-duplicate indicator covers the bounded pairwise sample" + }, + "invalid_values": { + "count": 120, + "definition": "Counts of NaN/Inf values in the selected split and the number of vectors containing at least one such value. Invalid entries are excluded from numeric summaries and treated as zero in norms.", + "invalid_ratio": 0.0005208333333333333, + "invalid_values": 1, + "sample_size": null, + "scope": "full", + "total_values": 1920, + "vectors_with_invalid": 1, + "vectors_with_invalid_ratio": 0.008333333333333333 + }, + "nearest_neighbor_distances": { + "available": true, + "definition": "For each vector in the bounded sample, the distance to its nearest other vector in the same sample (diagonal excluded).", + "histogram": { + "bin_count": 20, + "bins": [ + 1.1146412798770295, + 1.1562998027865923, + 1.197958325696155, + 1.2396168486057177, + 1.2812753715152805, + 1.322933894424843, + 1.3645924173344057, + 1.4062509402439685, + 1.4479094631535312, + 1.489567986063094, + 1.5312265089726567, + 1.5728850318822194, + 1.6145435547917821, + 1.6562020777013449, + 1.6978606006109076, + 1.7395191235204703, + 1.781177646430033, + 1.8228361693395958, + 1.8644946922491585, + 1.906153215158721, + 1.9478117380682838 + ], + "counts": [ + 2, + 0, + 0, + 2, + 1, + 2, + 0, + 0, + 1, + 1, + 2, + 5, + 4, + 0, + 3, + 3, + 1, + 2, + 1, + 2 + ] + }, + "sample_size": 32, + "scope": "sample", + "stats": { + "count": 32, + "max": 1.9478117380682838, + "mean": 1.5962322847234698, + "min": 1.1146412798770295, + "p1": 1.1146412798770295, + "p25": 1.4912582074975094, + "p5": 1.1892903890564392, + "p50": 1.6116257271718863, + "p75": 1.7509658512115271, + "p95": 1.9157255817084389, + "p99": 1.945191410466244, + "std": 0.22181336335330268 + } + }, + "norms": { + "count": 120, + "definition": "L2 norm ||x||_2 of every vector, computed over all finite values in the selected split.", + "histogram": { + "bin_count": 20, + "bins": [ + 0.0, + 68.8884556375297, + 137.7769112750594, + 206.6653669125891, + 275.5538225501188, + 344.4422781876485, + 413.3307338251782, + 482.2191894627079, + 551.1076451002376, + 619.9961007377673, + 688.884556375297, + 757.7730120128267, + 826.6614676503564, + 895.5499232878861, + 964.4383789254158, + 1033.3268345629456, + 1102.2152902004752, + 1171.103745838005, + 1239.9922014755346, + 1308.8806571130644, + 1377.769112750594 + ], + "counts": [ + 119, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 1 + ] + }, + "non_finite_norms": 0, + "sample_size": null, + "scope": "full", + "stats": { + "count": 120, + "max": 1377.769112750594, + "mean": 23.36713710329417, + "min": 0.0, + "p1": 3.4883344002846957, + "p25": 4.598193064968365, + "p5": 3.839000151636664, + "p50": 15.397560383722155, + "p75": 16.043222996068575, + "p95": 16.63487385967905, + "p99": 16.937213056072363, + "std": 124.27743636189794 + } + }, + "outliers": { + "count": 120, + "definition": "Robust norm outliers: vectors whose norm has a modified z-score above 3.5, using median and median absolute deviation. Invalid (NaN/Inf) value counts are reported separately.", + "mad_norm": 0.9384653936093876, + "max_over_median_norm": 89.47970187583293, + "median_norm": 15.397560383722155, + "method": "modified z-score (median/MAD)", + "outlier_count": 40, + "outlier_ratio": 0.3333333333333333, + "sample_size": null, + "scope": "full", + "threshold": 4.869724058758869 + }, + "pairwise_distances": { + "definition": "Distribution of distances between distinct vectors in a bounded random sample. The full M x M matrix is allocated only for the configured sample size, never for the complete dataset.", + "dense_sample_materialized": true, + "histogram": { + "bin_count": 20, + "bins": [ + 1.1146412798770295, + 2.635547628959658, + 4.156453978042286, + 5.677360327124915, + 7.198266676207544, + 8.719173025290171, + 10.2400793743728, + 11.760985723455429, + 13.281892072538056, + 14.802798421620684, + 16.323704770703316, + 17.844611119785945, + 19.36551746886857, + 20.8864238179512, + 22.40733016703383, + 23.928236516116456, + 25.449142865199086, + 26.970049214281715, + 28.49095556336434, + 30.01186191244697, + 31.5327682615296 + ], + "counts": [ + 149, + 8, + 0, + 0, + 0, + 0, + 0, + 0, + 5, + 125, + 110, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 9, + 90 + ] + }, + "matrix_is_bounded": true, + "metric": "euclidean", + "pair_count": 496, + "requested_pairwise_sample_size": 32, + "sample_size": 32, + "scope": "sample", + "sparse_sparse_distance_loops": false, + "stats": { + "count": 496, + "max": 31.5327682615296, + "mean": 14.601497390500318, + "min": 1.1146412798770295, + "p1": 1.4926785045174533, + "p25": 2.3832882818400134, + "p5": 1.7253295057198694, + "p50": 15.856817157369147, + "p75": 17.19127849277004, + "p95": 30.97918795637365, + "p99": 31.34371021181013, + "std": 10.123166853475572 + } + }, + "sparsity": { + "all_zero_vector_ratio": 0.008333333333333333, + "all_zero_vectors": 1, + "count": 120, + "definition": "Count and fraction of finite exactly-zero entries; non-zero values and density count finite non-zero entries, while NaN/Inf values are reported separately in invalid_values. For sparse vectors this also reports non-zero entries per vector and density = nnz / (count * dim).", + "density": 0.9911458333333333, + "non_zero_values": 1903, + "representation": "dense", + "sample_size": null, + "scope": "full", + "total_values": 1920, + "zero_ratio": 0.008333333333333333, + "zero_values": 16 + } + }, + "schema_version": "1.0", + "tool": { + "name": "vsag-dataset-profiler", + "version": "0.1.0" + }, + "warnings": [ + "sampled vectors contain NaN/Inf entries; sampled metrics replace them with 0.0, which can distort pairwise, nearest-neighbor, anisotropy, and clustering results" + ] +} diff --git a/tools/dataset_profiler/examples/output/toy_sparse.profile.html b/tools/dataset_profiler/examples/output/toy_sparse.profile.html new file mode 100644 index 0000000000..5b08a89cb3 --- /dev/null +++ b/tools/dataset_profiler/examples/output/toy_sparse.profile.html @@ -0,0 +1,106 @@ + + + + + + +VSAG dataset profile: toy_sparse.hdf5 + + + +
    +

    Dataset Feature Profile

    toy_sparse.hdf5 · 82 vectors · 128 dimensions · SPARSE

    schema 1.0seed 42sample 64pairwise sample 32metric ip

    Deterministic offline report. No raw vectors were sent anywhere while generating this file.

    +

    Overview

    Vectors
    82
    full dataset
    Dimensions
    128 (inferred)
    full dataset
    Element type
    INT8
    full dataset
    Distance metric
    ip
    profiling metric
    Norm median
    1.507
    full dataset
    Zero ratio
    94.950%
    full dataset
    Exact duplicate ratio
    1.220%
    full dataset
    Invalid values
    0
    full dataset
    Norm outlier ratio
    1.220%
    full dataset
    Top eigenvalue share
    0.05201
    sample
    +

    Vector norms

    Scope: fullSample size: n/a

    Definition: L2 norm ||x||_2 of every vector, computed over all finite values in the selected split.

    Vectorsnorm 0 – 0.1134: 1norm 0.1134 – 0.2268: 0norm 0.2268 – 0.340199: 0norm 0.340199 – 0.453599: 0norm 0.453599 – 0.566999: 1norm 0.566999 – 0.680399: 1norm 0.680399 – 0.793799: 1norm 0.793799 – 0.907198: 3norm 0.907198 – 1.0206: 6norm 1.0206 – 1.134: 2norm 1.134 – 1.2474: 7norm 1.2474 – 1.3608: 9norm 1.3608 – 1.4742: 9norm 1.4742 – 1.5876: 7norm 1.5876 – 1.701: 10norm 1.701 – 1.8144: 9norm 1.8144 – 1.9278: 6norm 1.9278 – 2.0412: 2norm 2.0412 – 2.1546: 6norm 2.1546 – 2.268: 202.268L2 norm100
    +

    Per-dimension distributions

    Scope: fullSample size: n/a

    Definition: Per-coordinate count, mean, standard deviation, minimum, maximum, and zero ratio over the selected split.

    dimension mean; 128 plotted point(s).

    Meandimension 0: 0.0113014dimension 1: 0.0257751dimension 2: 0.0561368dimension 3: 0.018313dimension 4: 0.0269997dimension 5: 0.0185367dimension 6: 0.0371437dimension 7: 0.0135045dimension 8: 0.00670347dimension 9: 0.0302741dimension 10: 0.0241525dimension 11: 0.0185959dimension 12: 0.051726dimension 13: 0.0460903dimension 14: 0.0240339dimension 15: 0.022955dimension 16: 0.00413032dimension 17: 0.0254266dimension 18: 0.0126993dimension 19: 0.0408748dimension 20: 0.0096224dimension 21: 0.0451287dimension 22: 0.0163127dimension 23: 0.0303798dimension 24: 0.0233611dimension 25: 0.0409702dimension 26: 0.00669524dimension 27: 0.033723dimension 28: 0.0438825dimension 29: 0.017442dimension 30: 0.0239852dimension 31: 0.0491976dimension 32: 0.022641dimension 33: 0.0261856dimension 34: 0.0149408dimension 35: 0.031781dimension 36: 0.0485814dimension 37: 0.0158925dimension 38: 0.0502251dimension 39: 0.0265135dimension 40: 0.0382096dimension 41: 0.024272dimension 42: 0.0330465dimension 43: 0.0319438dimension 44: 0.0224175dimension 45: 0.0349538dimension 46: 0.0105928dimension 47: 0.0246587dimension 48: 0.0317068dimension 49: 0.0401586dimension 50: 0.0197657dimension 51: 0.0069083dimension 52: 0.0136332dimension 53: 0.0118885dimension 54: 0.0256885dimension 55: 0.0370393dimension 56: 0dimension 57: 0.0149782dimension 58: 0.0187499dimension 59: 0.0450571dimension 60: 0.0200332dimension 61: 0.00931685dimension 62: 0.0317986dimension 63: 0.0233447dimension 64: 0.0414952dimension 65: 0.0124228dimension 66: 0.0246133dimension 67: 0.0116226dimension 68: 0.012328dimension 69: 0.0614125dimension 70: 0.0572769dimension 71: 0.0511069dimension 72: 0.0175383dimension 73: 0.0160194dimension 74: 0.0550786dimension 75: 0.040714dimension 76: 0.0401803dimension 77: 0.0243449dimension 78: 0.00859104dimension 79: 0.0302052dimension 80: 0.030138dimension 81: 0.0203635dimension 82: 0.017773dimension 83: 0.0189621dimension 84: 0.0237152dimension 85: 0.0280097dimension 86: 0.042951dimension 87: 0.027848dimension 88: 0.0296936dimension 89: 0.0133585dimension 90: 0.0429856dimension 91: 0.0320764dimension 92: 0.0278494dimension 93: 0.0442415dimension 94: 0.0111141dimension 95: 0.0339272dimension 96: 0.0207639dimension 97: 0.00327984dimension 98: 0.0480488dimension 99: 0.022781dimension 100: 0.0385218dimension 101: 0.00777072dimension 102: 0.0356469dimension 103: 0.0103142dimension 104: 0.0122732dimension 105: 0.00650846dimension 106: 0.0254101dimension 107: 0.00755439dimension 108: 0.00744183dimension 109: 0.0213257dimension 110: 0.0392835dimension 111: 0.0292306dimension 112: 0.0110371dimension 113: 0.00768599dimension 114: 0.0569517dimension 115: 0.0470726dimension 116: 0.0371416dimension 117: 0.0250312dimension 118: 0.0562887dimension 119: 0.034093dimension 120: 0.0728355dimension 121: 0.0315901dimension 122: 0dimension 123: 0.026808dimension 124: 0.0231166dimension 125: 0.016297dimension 126: 0.0238899dimension 127: 0.01030920127Coordinate0.072840

    dimension standard deviation; 128 plotted point(s).

    Stddimension 0: 0.0830651dimension 1: 0.116702dimension 2: 0.20274dimension 3: 0.107216dimension 4: 0.141239dimension 5: 0.0953123dimension 6: 0.151615dimension 7: 0.0812609dimension 8: 0.0603312dimension 9: 0.14573dimension 10: 0.115405dimension 11: 0.0877409dimension 12: 0.187593dimension 13: 0.189282dimension 14: 0.103689dimension 15: 0.131855dimension 16: 0.0371729dimension 17: 0.13494dimension 18: 0.0803174dimension 19: 0.167354dimension 20: 0.0866016dimension 21: 0.179501dimension 22: 0.11142dimension 23: 0.14257dimension 24: 0.117271dimension 25: 0.147818dimension 26: 0.0463238dimension 27: 0.141487dimension 28: 0.171247dimension 29: 0.0957511dimension 30: 0.107236dimension 31: 0.179964dimension 32: 0.114471dimension 33: 0.123818dimension 34: 0.0837208dimension 35: 0.157341dimension 36: 0.189677dimension 37: 0.114129dimension 38: 0.179852dimension 39: 0.126076dimension 40: 0.142763dimension 41: 0.129044dimension 42: 0.159565dimension 43: 0.138718dimension 44: 0.11899dimension 45: 0.141597dimension 46: 0.087567dimension 47: 0.149815dimension 48: 0.144897dimension 49: 0.157945dimension 50: 0.0959941dimension 51: 0.0621747dimension 52: 0.0717573dimension 53: 0.106996dimension 54: 0.120002dimension 55: 0.150641dimension 56: 0dimension 57: 0.0956055dimension 58: 0.0944471dimension 59: 0.172498dimension 60: 0.127649dimension 61: 0.0575833dimension 62: 0.147816dimension 63: 0.120309dimension 64: 0.172381dimension 65: 0.0840841dimension 66: 0.107205dimension 67: 0.0677013dimension 68: 0.0704271dimension 69: 0.202696dimension 70: 0.193186dimension 71: 0.185782dimension 72: 0.110398dimension 73: 0.0816863dimension 74: 0.200304dimension 75: 0.164654dimension 76: 0.152166dimension 77: 0.13526dimension 78: 0.0612277dimension 79: 0.132675dimension 80: 0.138735dimension 81: 0.113394dimension 82: 0.0958237dimension 83: 0.0882797dimension 84: 0.123453dimension 85: 0.132488dimension 86: 0.165444dimension 87: 0.129244dimension 88: 0.138504dimension 89: 0.108446dimension 90: 0.163737dimension 91: 0.131187dimension 92: 0.117508dimension 93: 0.178221dimension 94: 0.0605513dimension 95: 0.138521dimension 96: 0.115436dimension 97: 0.0207952dimension 98: 0.193403dimension 99: 0.127942dimension 100: 0.175996dimension 101: 0.0493222dimension 102: 0.151783dimension 103: 0.0534741dimension 104: 0.0796461dimension 105: 0.0420943dimension 106: 0.119189dimension 107: 0.0679895dimension 108: 0.0669765dimension 109: 0.126155dimension 110: 0.167156dimension 111: 0.141966dimension 112: 0.0698465dimension 113: 0.0414749dimension 114: 0.200666dimension 115: 0.167697dimension 116: 0.155809dimension 117: 0.130895dimension 118: 0.190268dimension 119: 0.12795dimension 120: 0.227107dimension 121: 0.141158dimension 122: 0dimension 123: 0.134451dimension 124: 0.119217dimension 125: 0.105012dimension 126: 0.141042dimension 127: 0.06973130127Coordinate0.22710

    dimension zero ratio; 128 plotted point(s).

    Zero ratiodimension 0: 0.97561dimension 1: 0.939024dimension 2: 0.914634dimension 3: 0.95122dimension 4: 0.963415dimension 5: 0.963415dimension 6: 0.939024dimension 7: 0.963415dimension 8: 0.987805dimension 9: 0.95122dimension 10: 0.95122dimension 11: 0.95122dimension 12: 0.926829dimension 13: 0.939024dimension 14: 0.939024dimension 15: 0.963415dimension 16: 0.987805dimension 17: 0.963415dimension 18: 0.97561dimension 19: 0.926829dimension 20: 0.987805dimension 21: 0.939024dimension 22: 0.97561dimension 23: 0.95122dimension 24: 0.95122dimension 25: 0.926829dimension 26: 0.97561dimension 27: 0.926829dimension 28: 0.926829dimension 29: 0.963415dimension 30: 0.95122dimension 31: 0.914634dimension 32: 0.95122dimension 33: 0.939024dimension 34: 0.963415dimension 35: 0.95122dimension 36: 0.926829dimension 37: 0.97561dimension 38: 0.926829dimension 39: 0.95122dimension 40: 0.902439dimension 41: 0.963415dimension 42: 0.95122dimension 43: 0.926829dimension 44: 0.939024dimension 45: 0.926829dimension 46: 0.97561dimension 47: 0.963415dimension 48: 0.95122dimension 49: 0.926829dimension 50: 0.95122dimension 51: 0.987805dimension 52: 0.963415dimension 53: 0.987805dimension 54: 0.95122dimension 55: 0.914634dimension 56: 1dimension 57: 0.97561dimension 58: 0.95122dimension 59: 0.902439dimension 60: 0.97561dimension 61: 0.963415dimension 62: 0.926829dimension 63: 0.963415dimension 64: 0.926829dimension 65: 0.97561dimension 66: 0.926829dimension 67: 0.963415dimension 68: 0.963415dimension 69: 0.902439dimension 70: 0.914634dimension 71: 0.926829dimension 72: 0.963415dimension 73: 0.95122dimension 74: 0.926829dimension 75: 0.926829dimension 76: 0.914634dimension 77: 0.963415dimension 78: 0.97561dimension 79: 0.926829dimension 80: 0.939024dimension 81: 0.95122dimension 82: 0.963415dimension 83: 0.939024dimension 84: 0.963415dimension 85: 0.95122dimension 86: 0.926829dimension 87: 0.926829dimension 88: 0.939024dimension 89: 0.97561dimension 90: 0.914634dimension 91: 0.926829dimension 92: 0.926829dimension 93: 0.939024dimension 94: 0.963415dimension 95: 0.926829dimension 96: 0.963415dimension 97: 0.97561dimension 98: 0.939024dimension 99: 0.963415dimension 100: 0.95122dimension 101: 0.97561dimension 102: 0.939024dimension 103: 0.963415dimension 104: 0.963415dimension 105: 0.97561dimension 106: 0.95122dimension 107: 0.987805dimension 108: 0.987805dimension 109: 0.963415dimension 110: 0.939024dimension 111: 0.939024dimension 112: 0.97561dimension 113: 0.963415dimension 114: 0.914634dimension 115: 0.914634dimension 116: 0.939024dimension 117: 0.963415dimension 118: 0.914634dimension 119: 0.926829dimension 120: 0.890244dimension 121: 0.939024dimension 122: 1dimension 123: 0.95122dimension 124: 0.939024dimension 125: 0.97561dimension 126: 0.963415dimension 127: 0.975610127Coordinate10.8902

    Highest-variance dimensions

    DimensionMeanStdZero ratio
    1200.072840.227189.024%
    20.056140.202791.463%
    690.061410.202790.244%
    1140.056950.200791.463%
    740.055080.200392.683%
    980.048050.193493.902%
    700.057280.193291.463%
    1180.056290.190391.463%
    360.048580.189792.683%
    130.046090.189393.902%
    +

    Sparsity

    Scope: fullSample size: n/a

    Definition: Count and fraction of finite exactly-zero entries; non-zero values and density count finite non-zero entries, while NaN/Inf values are reported separately in invalid_values. For sparse vectors this also reports non-zero entries per vector and density = nnz / (count * dim).

    Vectorsnnz 0 – 0.55: 1nnz 0.55 – 1.1: 0nnz 1.1 – 1.65: 0nnz 1.65 – 2.2: 0nnz 2.2 – 2.75: 0nnz 2.75 – 3.3: 12nnz 3.3 – 3.85: 0nnz 3.85 – 4.4: 5nnz 4.4 – 4.95: 0nnz 4.95 – 5.5: 10nnz 5.5 – 6.05: 12nnz 6.05 – 6.6: 0nnz 6.6 – 7.15: 15nnz 7.15 – 7.7: 0nnz 7.7 – 8.25: 8nnz 8.25 – 8.8: 0nnz 8.8 – 9.35: 9nnz 9.35 – 9.9: 0nnz 9.9 – 10.45: 8nnz 10.45 – 11: 2011Non-zero entries per vector150

    Representationsparse
    Total cells10496
    Zero values9966
    Non-zero values530
    Density5.050%
    Zero ratio94.950%
    All-zero vectors1
    Stored entries530

    +

    Duplicate and near-duplicate indicators

    Scope: mixedSample size: n/a

    Definition: Exact duplicate detection by hashing each complete vector; near-duplicate pairs use Euclidean distance for inner-product inputs and the chosen distance otherwise, below a relative threshold derived from the sampled duplicate-detection distance median.

    Unique vectors81
    Duplicate vectors (exact)1
    Duplicate ratio (exact)1.220%
    Duplicate groups1
    Largest duplicate group2

    Near-duplicate (sampled)

    Near-duplicate sampled pairs0
    Near-duplicate pair ratio0.000%
    Threshold0.00214
    Median sampled pairwise distance2.14
    Pairs sampled496

    exact duplicate detection covers the full split; the near-duplicate indicator covers the bounded pairwise sample

    +

    Pairwise distance distribution

    Scope: sampleSample size: 32

    Definition: Distribution of distances between distinct vectors in a bounded random sample. The full M x M matrix is allocated only for the configured sample size, never for the complete dataset.

    Pairspairwise distance -0.320178 – -0.254169: 1pairwise distance -0.254169 – -0.18816: 0pairwise distance -0.18816 – -0.122152: 0pairwise distance -0.122152 – -0.0561426: 1pairwise distance -0.0561426 – 0.00986629: 0pairwise distance 0.00986629 – 0.0758752: 1pairwise distance 0.0758752 – 0.141884: 0pairwise distance 0.141884 – 0.207893: 4pairwise distance 0.207893 – 0.273902: 7pairwise distance 0.273902 – 0.339911: 6pairwise distance 0.339911 – 0.40592: 5pairwise distance 0.40592 – 0.471929: 7pairwise distance 0.471929 – 0.537938: 3pairwise distance 0.537938 – 0.603947: 6pairwise distance 0.603947 – 0.669955: 12pairwise distance 0.669955 – 0.735964: 12pairwise distance 0.735964 – 0.801973: 10pairwise distance 0.801973 – 0.867982: 17pairwise distance 0.867982 – 0.933991: 24pairwise distance 0.933991 – 1: 380-0.32021Distance between sampled pairs3800
    count496
    mean0.9132
    std0.1982
    min-0.3202
    p10.1826
    p50.4108
    p250.9622
    p501
    p751
    p951
    p991
    max1

    The distance matrix is bounded by the configured pairwise sample size.

    Nearest-neighbor distance distribution

    Scope: sampleSample size: 32

    Definition: For each vector in the bounded sample, the distance to its nearest other vector in the same sample (diagonal excluded).

    Vectorsnn distance -0.320178 – -0.263937: 2nn distance -0.263937 – -0.207695: 0nn distance -0.207695 – -0.151453: 0nn distance -0.151453 – -0.0952116: 0nn distance -0.0952116 – -0.03897: 2nn distance -0.03897 – 0.0172717: 0nn distance 0.0172717 – 0.0735133: 2nn distance 0.0735133 – 0.129755: 0nn distance 0.129755 – 0.185997: 5nn distance 0.185997 – 0.242238: 5nn distance 0.242238 – 0.29848: 4nn distance 0.29848 – 0.354722: 4nn distance 0.354722 – 0.410963: 0nn distance 0.410963 – 0.467205: 1nn distance 0.467205 – 0.523447: 0nn distance 0.523447 – 0.579688: 0nn distance 0.579688 – 0.63593: 1nn distance 0.63593 – 0.692172: 3nn distance 0.692172 – 0.748413: 0nn distance 0.748413 – 0.804655: 3-0.32020.8047Distance to nearest other sampled vector50
    count32
    mean0.2801
    std0.2746
    min-0.3202
    p1-0.3202
    p5-0.1879
    p250.1805
    p500.241
    p750.3549
    p950.7625
    p990.7946
    max0.8047
    +

    Anisotropy

    Scope: sampleSample size: 64

    Definition: Eigenvalue concentration of the centered sample covariance: the largest eigenvalue share, top-10 share, effective rank, and mean pairwise cosine similarity. High top-1 share indicates a dominant direction.

    PC variance share.

    Explained variance ratio of leading principal componentsPC1: 0.052015PC1PC2: 0.0486358PC2PC3: 0.0441679PC3PC4: 0.0423136PC4PC5: 0.0407092PC5PC6: 0.0375497PC6PC7: 0.0363682PC7PC8: 0.0348908PC8PC9: 0.0332851PC9PC10: 0.032031PC100.052010

    Top-1 eigenvalue share0.05201
    Top-10 eigenvalue share0.402
    Effective rank44.49
    Mean pairwise cosine0.03976

    top1_eigenvalue_share near 1 indicates a dominant direction; effective_rank near 1 indicates nearly one-dimensional variation

    +

    Projection and clustering

    Scope: sampleSample size: 64

    Definition: Optional deterministic PCA-2 projection of the sample and optional k-means cluster summary over that projection.

    PCA-2 projection of the seeded sample.

    PC2sample 0 · cluster 2 · (-0.2074, 0.0621)sample 1 · cluster 0 · (0.4255, -0.2312)sample 2 · cluster 2 · (0.1147, 0.2976)sample 3 · cluster 2 · (0.2604, -0.05114)sample 4 · cluster 2 · (0.287, 0.09492)sample 5 · cluster 1 · (-0.1675, 0.5852)sample 6 · cluster 0 · (0.4325, -0.7935)sample 7 · cluster 2 · (-0.0954, 0.08669)sample 8 · cluster 0 · (0.2253, -0.1363)sample 9 · cluster 2 · (0.1493, 0.3217)sample 10 · cluster 0 · (0.05836, -0.5619)sample 11 · cluster 1 · (0.3705, 1.344)sample 12 · cluster 2 · (0.3431, 0.1843)sample 13 · cluster 2 · (0.1784, 0.1858)sample 14 · cluster 2 · (0.1479, -0.1267)sample 15 · cluster 2 · (0.04694, 0.07188)sample 16 · cluster 0 · (0.3102, -0.3629)sample 17 · cluster 1 · (0.01943, 0.4048)sample 18 · cluster 2 · (0.04315, -0.0326)sample 19 · cluster 2 · (-0.07432, 0.2404)sample 20 · cluster 2 · (0.412, 0.3519)sample 21 · cluster 2 · (0.1907, 0.2199)sample 22 · cluster 0 · (0.3769, -0.1633)sample 23 · cluster 2 · (0.04047, -0.07175)sample 24 · cluster 2 · (0.06355, -0.0297)sample 25 · cluster 2 · (-0.05814, -0.165)sample 26 · cluster 2 · (0.1085, -0.03378)sample 27 · cluster 2 · (0.1536, 0.1648)sample 28 · cluster 2 · (-0.2555, -0.2617)sample 29 · cluster 0 · (0.1479, -0.2841)sample 30 · cluster 2 · (0.1168, -0.09275)sample 31 · cluster 3 · (-1.157, -0.6058)sample 32 · cluster 2 · (-0.03001, 0.007666)sample 33 · cluster 2 · (0.1507, -0.01816)sample 34 · cluster 1 · (-0.5668, 0.4042)sample 35 · cluster 0 · (0.5639, -0.7593)sample 36 · cluster 2 · (-0.1724, -0.1714)sample 37 · cluster 0 · (0.01463, -0.2576)sample 38 · cluster 2 · (-0.1148, -0.07657)sample 39 · cluster 2 · (-0.2272, 0.00779)sample 40 · cluster 2 · (-0.2842, 0.1307)sample 41 · cluster 2 · (0.001078, 0.02287)sample 42 · cluster 2 · (-0.03994, -0.01782)sample 43 · cluster 2 · (0.01169, -0.06964)sample 44 · cluster 3 · (-1.494, 0.4443)sample 45 · cluster 2 · (0.2219, 0.106)sample 46 · cluster 1 · (-0.1675, 0.4062)sample 47 · cluster 2 · (0.09983, 0.2796)sample 48 · cluster 2 · (0.246, 0.2805)sample 49 · cluster 2 · (0.06127, -0.03594)sample 50 · cluster 0 · (0.1744, -0.1532)sample 51 · cluster 1 · (-0.1333, 0.455)sample 52 · cluster 2 · (-0.03992, -0.1276)sample 53 · cluster 0 · (0.2469, -0.1592)sample 54 · cluster 3 · (-0.5786, -0.6908)sample 55 · cluster 3 · (-0.7242, -0.1654)sample 56 · cluster 2 · (0.1911, 0.0401)sample 57 · cluster 2 · (0.1584, 0.1788)sample 58 · cluster 0 · (-0.07516, -0.4171)sample 59 · cluster 2 · (-0.1381, 0.02606)sample 60 · cluster 2 · (-0.07748, 0.0146)sample 61 · cluster 3 · (-0.5164, -0.1084)sample 62 · cluster 0 · (0.1823, -0.1728)sample 63 · cluster 2 · (0.04839, -0.01546)PC1

    Cluster size.

    k-means cluster sizes on the PCA-2 projectionC0: 13C0C1: 6C1C2: 40C2C3: 5C3400
    Clusters4
    Inertia5.621
    Iterations7
    +

    AI-assisted interpretation

    Disabled. The deterministic JSON and HTML reports were generated without network access.

    +

    Warnings

    • sparse dimensionality is inferred as max(feature id) + 1 during profiling
    +

    Generated by vsag-dataset-profiler. Deterministic JSON and HTML outputs are produced offline. When AI analysis is enabled, only the bounded aggregate metric payload described in the technical report is sent to the configured provider; raw vectors and credentials are never included.

    + + + diff --git a/tools/dataset_profiler/examples/output/toy_sparse.profile.json b/tools/dataset_profiler/examples/output/toy_sparse.profile.json new file mode 100644 index 0000000000..0e81a99e19 --- /dev/null +++ b/tools/dataset_profiler/examples/output/toy_sparse.profile.json @@ -0,0 +1,1600 @@ +{ + "config": { + "clusters": 4, + "effective_pairwise_sample_size": 32, + "effective_sample_size": 64, + "max_dimension_points": 1024, + "metric": "ip", + "near_duplicate_relative_epsilon": 0.001, + "pairwise_sample_size": 32, + "pairwise_sampling_method": "uniformly random subset selected by a seeded permutation of the sampled rows", + "sample_size": 64, + "sampling_method": "numpy.random.Generator(PCG64); Floyd's algorithm without replacement; indices sorted ascending before use", + "seed": 42 + }, + "dataset": { + "attributes": { + "distance": "ip", + "type": "sparse" + }, + "count": 82, + "datasets": [ + "train", + "train_offsets" + ], + "dim": 128, + "dim_inferred": true, + "distance_metric": "ip", + "dtype": "INT8", + "has_offsets": true, + "vector_type": "sparse" + }, + "input": { + "file_name": "toy_sparse.hdf5", + "file_size_bytes": 11376, + "split": "train" + }, + "metrics": { + "anisotropy": { + "available": true, + "definition": "Eigenvalue concentration of the centered sample covariance: the largest eigenvalue share, top-10 share, effective rank, and mean pairwise cosine similarity. High top-1 share indicates a dominant direction.", + "effective_rank": 44.49290281102398, + "explained_variance_ratio": [ + 0.052014962100113186, + 0.048635838698198465, + 0.04416794726143196, + 0.04231359641360622, + 0.04070918348682084, + 0.03754969488148461, + 0.03636820582146442, + 0.03489076513085486, + 0.03328509138315907, + 0.032030952831737354 + ], + "interpretation": "top1_eigenvalue_share near 1 indicates a dominant direction; effective_rank near 1 indicates nearly one-dimensional variation", + "mean_pairwise_cosine": 0.03975827258536121, + "method": "eigendecomposition of centered sample covariance", + "sample_size": 64, + "scope": "sample", + "top10_eigenvalue_share": 0.40196623800887094, + "top1_eigenvalue_share": 0.052014962100113186 + }, + "clustering": { + "available": true, + "cluster_sizes": [ + 13, + 6, + 40, + 5 + ], + "clustering_available": true, + "definition": "Optional deterministic PCA-2 projection of the sample and optional k-means cluster summary over that projection.", + "explained_variance_ratio_pc1": 0.052014962100113186, + "explained_variance_ratio_pc2": 0.048635838698198465, + "inertia": 5.621113595456645, + "iterations": 7, + "labels": [ + 2, + 0, + 2, + 2, + 2, + 1, + 0, + 2, + 0, + 2, + 0, + 1, + 2, + 2, + 2, + 2, + 0, + 1, + 2, + 2, + 2, + 2, + 0, + 2, + 2, + 2, + 2, + 2, + 2, + 0, + 2, + 3, + 2, + 2, + 1, + 0, + 2, + 0, + 2, + 2, + 2, + 2, + 2, + 2, + 3, + 2, + 1, + 2, + 2, + 2, + 0, + 1, + 2, + 0, + 3, + 3, + 2, + 2, + 0, + 2, + 2, + 3, + 0, + 2 + ], + "method": "deterministic PCA-2 projection of the seeded sample", + "n_clusters": 4, + "projection_points": [ + [ + -0.20743608523355045, + 0.06210093917944295 + ], + [ + 0.42545566824266035, + -0.231201338943911 + ], + [ + 0.11468421883663907, + 0.29761042640821156 + ], + [ + 0.26039439987043167, + -0.051144723977761694 + ], + [ + 0.2870408663767685, + 0.09491624562561052 + ], + [ + -0.1675107465796664, + 0.5851585581408076 + ], + [ + 0.43250827377913376, + -0.7935439774425732 + ], + [ + -0.09540021061314025, + 0.08668725133713698 + ], + [ + 0.2253492459380464, + -0.1363241874694267 + ], + [ + 0.14929014203761765, + 0.321748119275456 + ], + [ + 0.05835862809236886, + -0.5618525897019667 + ], + [ + 0.3704847742609134, + 1.3441928768999738 + ], + [ + 0.3430639839632944, + 0.1842834298057345 + ], + [ + 0.17837020086293814, + 0.18582591848037616 + ], + [ + 0.1478903282737174, + -0.1267350713129171 + ], + [ + 0.046939783454339495, + 0.0718809987969001 + ], + [ + 0.31017280874569114, + -0.3628608163416557 + ], + [ + 0.019430225253440597, + 0.40482078987856934 + ], + [ + 0.043147765188429756, + -0.0325996155480084 + ], + [ + -0.07432094075905923, + 0.2403716026547468 + ], + [ + 0.41196251057158634, + 0.35188451966495066 + ], + [ + 0.1907088407796459, + 0.2199320081003718 + ], + [ + 0.3768885956213379, + -0.1632909096294965 + ], + [ + 0.0404718432224404, + -0.07175298157038554 + ], + [ + 0.06355372630419565, + -0.029699996369756634 + ], + [ + -0.058137222740139285, + -0.16497472519446937 + ], + [ + 0.1084825321911692, + -0.03378480769767541 + ], + [ + 0.15359066424617296, + 0.1647614816415337 + ], + [ + -0.25545975318773206, + -0.26168229236860574 + ], + [ + 0.14789302727798714, + -0.2840616990317078 + ], + [ + 0.11682538804377063, + -0.09275227495711638 + ], + [ + -1.1574043119844808, + -0.6057935280063844 + ], + [ + -0.03000561590701989, + 0.007665719915319429 + ], + [ + 0.1506932320863538, + -0.018161386298097854 + ], + [ + -0.5668010522318626, + 0.40416993637141724 + ], + [ + 0.5638622320681854, + -0.759276404595125 + ], + [ + -0.17241273218107447, + -0.1713834235158628 + ], + [ + 0.01462918387284451, + -0.2576171591730882 + ], + [ + -0.11481037671856113, + -0.0765653693205332 + ], + [ + -0.2272291144619133, + 0.007790419161002329 + ], + [ + -0.2841907883905598, + 0.13066152952837454 + ], + [ + 0.0010780840511975462, + 0.022866848611144144 + ], + [ + -0.03993504358459449, + -0.017820799532544257 + ], + [ + 0.01168896889712102, + -0.06964253173439303 + ], + [ + -1.4937783826302278, + 0.4443105465699255 + ], + [ + 0.22188089925431226, + 0.10597993681952367 + ], + [ + -0.16745751639928805, + 0.4061736676236113 + ], + [ + 0.09983457399067378, + 0.27958152567517786 + ], + [ + 0.245961189715901, + 0.2804590230659894 + ], + [ + 0.061266816540201476, + -0.035941856978787226 + ], + [ + 0.17444306266660886, + -0.15315376421606142 + ], + [ + -0.1332881794526313, + 0.4550050977859015 + ], + [ + -0.03992481287283805, + -0.1275806434304765 + ], + [ + 0.2468631435801579, + -0.15917488601501642 + ], + [ + -0.5785715368917542, + -0.6908260264403346 + ], + [ + -0.7241649150559837, + -0.16540462815993232 + ], + [ + 0.19107157159143523, + 0.04010070578665015 + ], + [ + 0.15843539254363503, + 0.17881277791149275 + ], + [ + -0.07516265771610249, + -0.4170592831575677 + ], + [ + -0.1380610193453818, + 0.026061426348770247 + ], + [ + -0.07748067532353244, + 0.014599814637943114 + ], + [ + -0.5163809415913643, + -0.1084490531430182 + ], + [ + 0.18226408745753653, + -0.17284017346183486 + ], + [ + 0.04839375210155645, + -0.015461216965573083 + ] + ], + "sample_size": 64, + "scope": "sample" + }, + "dimensions": { + "definition": "Per-coordinate count, mean, standard deviation, minimum, maximum, and zero ratio over the selected split.", + "dim": 128, + "per_dimension": { + "max": { + "bucket_size": 1, + "downsampled": false, + "source_length": 128, + "values": [ + 0.7343350648880005, + 0.7845785617828369, + 0.9450693130493164, + 0.9087703824043274, + 0.9172006249427795, + 0.5456408858299255, + 0.8825763463973999, + 0.6055675148963928, + 0.5496842265129089, + 0.885227620601654, + 0.7364864945411682, + 0.6013668179512024, + 0.9674977660179138, + 0.9813886880874634, + 0.5725422501564026, + 0.8749192357063293, + 0.3386862576007843, + 0.8638743758201599, + 0.5206712484359741, + 0.9174684882164001, + 0.789036750793457, + 0.9198509454727173, + 0.9382211565971375, + 0.8807173371315002, + 0.8334482908248901, + 0.6490805745124817, + 0.3947819471359253, + 0.991331160068512, + 0.9548744559288025, + 0.6426075100898743, + 0.5653858780860901, + 0.9505579471588135, + 0.8324854969978333, + 0.943615198135376, + 0.6046254634857178, + 0.9522503018379211, + 0.9944417476654053, + 0.997749388217926, + 0.8164845108985901, + 0.8145623803138733, + 0.8171408176422119, + 0.8649344444274902, + 0.9594299793243408, + 0.8458189368247986, + 0.8465406894683838, + 0.8771272301673889, + 0.795369029045105, + 0.9885457754135132, + 0.8804170489311218, + 0.8487966060638428, + 0.6558961868286133, + 0.5664803385734558, + 0.464032381772995, + 0.9748530983924866, + 0.7827348709106445, + 0.9808060526847839, + 0.0, + 0.6967467665672302, + 0.62018883228302, + 0.961921751499176, + 0.9207697510719299, + 0.4908132553100586, + 0.9679073691368103, + 0.7023905515670776, + 0.9872914552688599, + 0.7011198401451111, + 0.7895970940589905, + 0.49931809306144714, + 0.5483108162879944, + 0.9551632404327393, + 0.9703733921051025, + 0.8762781620025635, + 0.9443708658218384, + 0.6394898891448975, + 0.9740490317344666, + 0.954216718673706, + 0.8031226992607117, + 0.8691160678863525, + 0.5329544544219971, + 0.8791018128395081, + 0.8925344944000244, + 0.9431511759757996, + 0.6961179375648499, + 0.6055589318275452, + 0.7920624017715454, + 0.9317213296890259, + 0.9043184518814087, + 0.8180569410324097, + 0.9086676239967346, + 0.9830385446548462, + 0.8793079257011414, + 0.8285038471221924, + 0.7087154984474182, + 0.9503145813941956, + 0.42963907122612, + 0.934410035610199, + 0.8958642482757568, + 0.1438571810722351, + 0.8861127495765686, + 0.9586595296859741, + 0.9565818905830383, + 0.3452453315258026, + 0.8307809233665466, + 0.33026570081710815, + 0.692362368106842, + 0.3232252597808838, + 0.8057090044021606, + 0.6194599866867065, + 0.6102302074432373, + 0.8579896092414856, + 0.9574557542800903, + 0.9476794600486755, + 0.46798890829086304, + 0.30401772260665894, + 0.9515742659568787, + 0.8649401664733887, + 0.8878250122070312, + 0.8365238308906555, + 0.916469931602478, + 0.6679425835609436, + 0.9597930312156677, + 0.8517646193504333, + 0.0, + 0.9427135586738586, + 0.8920515775680542, + 0.7968640327453613, + 0.9191786050796509, + 0.5809857249259949 + ] + }, + "mean": { + "bucket_size": 1, + "downsampled": false, + "source_length": 128, + "values": [ + 0.011301381013742307, + 0.02577509004168394, + 0.05613677367204573, + 0.018312954684583153, + 0.02699974470022248, + 0.018536699981224244, + 0.03714365639337679, + 0.013504487803069554, + 0.006703466176986694, + 0.03027408152091794, + 0.02415252858545722, + 0.018595858863214167, + 0.051725985800347665, + 0.04609027323199481, + 0.02403394423606919, + 0.022955021480234657, + 0.0041303202146437115, + 0.025426591678363517, + 0.012699298742340832, + 0.04087478263167346, + 0.009622399399920208, + 0.04512866313864545, + 0.016312717664532544, + 0.030379843057655707, + 0.023361134274703703, + 0.040970179729345364, + 0.006695238010185521, + 0.03372296981695222, + 0.04388253699715544, + 0.017441997804292817, + 0.023985244878908484, + 0.04919757839383148, + 0.022640970149418203, + 0.026185619576675135, + 0.014940790105156782, + 0.031780969805833774, + 0.048581353684024116, + 0.015892542353490503, + 0.05022509214354724, + 0.026513452210077428, + 0.038209578431234126, + 0.02427198792376169, + 0.033046480540822196, + 0.031943784526935436, + 0.02241745309495344, + 0.034953844801681795, + 0.010592794363818518, + 0.024658737204423766, + 0.03170679982115583, + 0.040158630326026824, + 0.019765731583281262, + 0.0069082968118714125, + 0.01363316650797681, + 0.011888452419420568, + 0.025688520655399418, + 0.037039317463229346, + 0.0, + 0.014978184205729788, + 0.01874992006072184, + 0.04505707891430797, + 0.02003318652874086, + 0.009316846546603412, + 0.031798589429477365, + 0.02334465050115818, + 0.041495198883661405, + 0.012422784799482764, + 0.024613311468828013, + 0.011622593624562751, + 0.012327981067866813, + 0.061412511257136745, + 0.05727685697195008, + 0.051106945407099834, + 0.01753830182843092, + 0.016019442095989135, + 0.055078556624854474, + 0.040714035673839286, + 0.04018033931895, + 0.024344939284208344, + 0.00859104078717348, + 0.030205224890534465, + 0.030138026286916034, + 0.020363509291555823, + 0.017773036549731, + 0.01896210896169267, + 0.023715168237686157, + 0.028009732685438018, + 0.042950964192064796, + 0.027848017379278087, + 0.029693637860984334, + 0.013358497219841656, + 0.042985602213842115, + 0.032076403135206634, + 0.027849414999165188, + 0.04424152178008382, + 0.011114095587555954, + 0.033927153523375345, + 0.02076387369051212, + 0.0032798422545921514, + 0.048048753563950705, + 0.022780965014201838, + 0.03852183062855791, + 0.007770716780569496, + 0.03564690907553929, + 0.010314210522465589, + 0.012273189316435557, + 0.006508457587986458, + 0.025410121897371806, + 0.007554390081545202, + 0.00744183179808826, + 0.021325675361767046, + 0.03928353710145485, + 0.02923055819985343, + 0.011037102196274734, + 0.007685985143591718, + 0.05695165094079041, + 0.047072607933021184, + 0.03714163928497128, + 0.025031168286393325, + 0.05628872281167566, + 0.034093007808778344, + 0.07283548701827121, + 0.03159006503297061, + 0.0, + 0.02680800091929552, + 0.023116634477202483, + 0.01629700021045964, + 0.0238898936204794, + 0.010309220814123386 + ] + }, + "min": { + "bucket_size": 1, + "downsampled": false, + "source_length": 128, + "values": [ + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0 + ] + }, + "std": { + "bucket_size": 1, + "downsampled": false, + "source_length": 128, + "values": [ + 0.08306507965294531, + 0.11670238772292027, + 0.20273995499711636, + 0.10721633996891657, + 0.14123894991930439, + 0.09531233492872516, + 0.151615007779749, + 0.08126089068613881, + 0.06033119559288025, + 0.14572986416669143, + 0.11540529889769062, + 0.08774092835102866, + 0.18759333746854454, + 0.189281817752319, + 0.1036887312980423, + 0.13185531239076592, + 0.0371728819317934, + 0.13494004813830354, + 0.08031741742541762, + 0.16735405517805943, + 0.08660159459928186, + 0.1795013352875239, + 0.11141966571825412, + 0.14256970651181225, + 0.11727082490913769, + 0.14781840663220455, + 0.046323778287328476, + 0.14148688171242935, + 0.1712474163766646, + 0.09575109877422945, + 0.10723558324061695, + 0.17996424546534517, + 0.11447128396687127, + 0.12381811719696724, + 0.08372082668151817, + 0.15734078610847282, + 0.18967731071443955, + 0.11412904005040583, + 0.1798522782986329, + 0.1260764629014871, + 0.14276322650701892, + 0.12904384728444251, + 0.15956530135313285, + 0.13871826145019625, + 0.11898959339731975, + 0.14159715628043246, + 0.08756704803441406, + 0.1498152862340504, + 0.14489676334208976, + 0.15794461542766722, + 0.09599413456929705, + 0.06217467130684271, + 0.07175733865562052, + 0.10699607177478511, + 0.1200020862384663, + 0.15064086456093145, + 0.0, + 0.09560551745973996, + 0.09444709777701649, + 0.17249828025655184, + 0.12764862330802937, + 0.05758327644344536, + 0.14781618349702827, + 0.1203085504579423, + 0.17238088764990456, + 0.08408411807908603, + 0.10720518938478403, + 0.06770126295096246, + 0.07042708552256519, + 0.2026955077393008, + 0.19318561782187912, + 0.18578216120122676, + 0.11039843597039173, + 0.08168633440705382, + 0.20030419898542823, + 0.16465395068967073, + 0.1521658926087923, + 0.13526043773550278, + 0.06122771823801578, + 0.13267485018789182, + 0.13873463970530492, + 0.11339394458917508, + 0.09582365682004741, + 0.08827972016709022, + 0.1234527578853575, + 0.13248837764006685, + 0.16544396443283388, + 0.1292438940878139, + 0.1385044128662626, + 0.10844557457048122, + 0.16373731831220467, + 0.13118685227161753, + 0.11750795801546907, + 0.1782205586693177, + 0.06055133004665685, + 0.1385206584447068, + 0.11543610707372215, + 0.020795245595403376, + 0.19340278920178894, + 0.12794237701706623, + 0.17599621071748167, + 0.04932219377048738, + 0.15178264341969314, + 0.05347414577764921, + 0.0796460601694308, + 0.042094252026081316, + 0.11918943606178048, + 0.06798951073390681, + 0.06697648618279434, + 0.1261549817150462, + 0.16715555405409024, + 0.14196617069222636, + 0.06984654872646281, + 0.041474934633061344, + 0.20066563309278548, + 0.16769745416957604, + 0.15580866324652792, + 0.13089465065398526, + 0.1902677716025027, + 0.12794963467288217, + 0.22710726808136664, + 0.14115782376178862, + 0.0, + 0.134450557380297, + 0.11921671651575079, + 0.10501237838602857, + 0.14104180652391296, + 0.06973129162395626 + ] + }, + "zero_ratio": { + "bucket_size": 1, + "downsampled": false, + "source_length": 128, + "values": [ + 0.975609756097561, + 0.9390243902439024, + 0.9146341463414634, + 0.9512195121951219, + 0.9634146341463414, + 0.9634146341463414, + 0.9390243902439024, + 0.9634146341463414, + 0.9878048780487805, + 0.9512195121951219, + 0.9512195121951219, + 0.9512195121951219, + 0.926829268292683, + 0.9390243902439024, + 0.9390243902439024, + 0.9634146341463414, + 0.9878048780487805, + 0.9634146341463414, + 0.975609756097561, + 0.926829268292683, + 0.9878048780487805, + 0.9390243902439024, + 0.975609756097561, + 0.9512195121951219, + 0.9512195121951219, + 0.926829268292683, + 0.975609756097561, + 0.926829268292683, + 0.926829268292683, + 0.9634146341463414, + 0.9512195121951219, + 0.9146341463414634, + 0.9512195121951219, + 0.9390243902439024, + 0.9634146341463414, + 0.9512195121951219, + 0.926829268292683, + 0.975609756097561, + 0.926829268292683, + 0.9512195121951219, + 0.9024390243902439, + 0.9634146341463414, + 0.9512195121951219, + 0.926829268292683, + 0.9390243902439024, + 0.926829268292683, + 0.975609756097561, + 0.9634146341463414, + 0.9512195121951219, + 0.926829268292683, + 0.9512195121951219, + 0.9878048780487805, + 0.9634146341463414, + 0.9878048780487805, + 0.9512195121951219, + 0.9146341463414634, + 1.0, + 0.975609756097561, + 0.9512195121951219, + 0.9024390243902439, + 0.975609756097561, + 0.9634146341463414, + 0.926829268292683, + 0.9634146341463414, + 0.926829268292683, + 0.975609756097561, + 0.926829268292683, + 0.9634146341463414, + 0.9634146341463414, + 0.9024390243902439, + 0.9146341463414634, + 0.926829268292683, + 0.9634146341463414, + 0.9512195121951219, + 0.926829268292683, + 0.926829268292683, + 0.9146341463414634, + 0.9634146341463414, + 0.975609756097561, + 0.926829268292683, + 0.9390243902439024, + 0.9512195121951219, + 0.9634146341463414, + 0.9390243902439024, + 0.9634146341463414, + 0.9512195121951219, + 0.926829268292683, + 0.926829268292683, + 0.9390243902439024, + 0.975609756097561, + 0.9146341463414634, + 0.926829268292683, + 0.926829268292683, + 0.9390243902439024, + 0.9634146341463414, + 0.926829268292683, + 0.9634146341463414, + 0.975609756097561, + 0.9390243902439024, + 0.9634146341463414, + 0.9512195121951219, + 0.975609756097561, + 0.9390243902439024, + 0.9634146341463414, + 0.9634146341463414, + 0.975609756097561, + 0.9512195121951219, + 0.9878048780487805, + 0.9878048780487805, + 0.9634146341463414, + 0.9390243902439024, + 0.9390243902439024, + 0.975609756097561, + 0.9634146341463414, + 0.9146341463414634, + 0.9146341463414634, + 0.9390243902439024, + 0.9634146341463414, + 0.9146341463414634, + 0.926829268292683, + 0.8902439024390244, + 0.9390243902439024, + 1.0, + 0.9512195121951219, + 0.9390243902439024, + 0.975609756097561, + 0.9634146341463414, + 0.975609756097561 + ] + } + }, + "sample_size": null, + "scope": "full", + "summary": { + "mean": { + "count": 128, + "max": 0.07283548701827121, + "mean": 0.026791064613839476, + "min": 0.0, + "p1": 0.000885557408739881, + "p25": 0.015663952816550326, + "p5": 0.006775156899196346, + "p50": 0.02463602433662589, + "p75": 0.03706489791866483, + "p95": 0.053905156836277064, + "p99": 0.06029588460013636, + "std": 0.01465340927849492 + }, + "std": { + "count": 128, + "max": 0.22710726808136664, + "mean": 0.12264241279901432, + "min": 0.0, + "p1": 0.005614716310758912, + "p25": 0.09290525337453492, + "p5": 0.0473732237064341, + "p50": 0.12690180251153776, + "p75": 0.15165691668973505, + "p95": 0.19216437164509734, + "p99": 0.20272795423750617, + "std": 0.04489103572590744 + }, + "zero_ratio": { + "count": 128, + "max": 1.0, + "mean": 0.9495045731707318, + "min": 0.8902439024390244, + "p1": 0.9024390243902439, + "p25": 0.926829268292683, + "p5": 0.9146341463414634, + "p50": 0.9512195121951219, + "p75": 0.9634146341463414, + "p95": 0.9878048780487805, + "p99": 0.9967073170731707, + "std": 0.022903139062312908 + } + }, + "top_variance_dimensions": [ + { + "dimension": 120, + "mean": 0.07283548701827121, + "std": 0.22710726808136664, + "zero_ratio": 0.8902439024390244 + }, + { + "dimension": 2, + "mean": 0.05613677367204573, + "std": 0.20273995499711636, + "zero_ratio": 0.9146341463414634 + }, + { + "dimension": 69, + "mean": 0.061412511257136745, + "std": 0.2026955077393008, + "zero_ratio": 0.9024390243902439 + }, + { + "dimension": 114, + "mean": 0.05695165094079041, + "std": 0.20066563309278548, + "zero_ratio": 0.9146341463414634 + }, + { + "dimension": 74, + "mean": 0.055078556624854474, + "std": 0.20030419898542823, + "zero_ratio": 0.926829268292683 + }, + { + "dimension": 98, + "mean": 0.048048753563950705, + "std": 0.19340278920178894, + "zero_ratio": 0.9390243902439024 + }, + { + "dimension": 70, + "mean": 0.05727685697195008, + "std": 0.19318561782187912, + "zero_ratio": 0.9146341463414634 + }, + { + "dimension": 118, + "mean": 0.05628872281167566, + "std": 0.1902677716025027, + "zero_ratio": 0.9146341463414634 + }, + { + "dimension": 36, + "mean": 0.048581353684024116, + "std": 0.18967731071443955, + "zero_ratio": 0.926829268292683 + }, + { + "dimension": 13, + "mean": 0.04609027323199481, + "std": 0.189281817752319, + "zero_ratio": 0.9390243902439024 + } + ], + "top_zero_dimensions": [ + { + "dimension": 122, + "mean": 0.0, + "zero_ratio": 1.0 + }, + { + "dimension": 56, + "mean": 0.0, + "zero_ratio": 1.0 + }, + { + "dimension": 108, + "mean": 0.00744183179808826, + "zero_ratio": 0.9878048780487805 + }, + { + "dimension": 107, + "mean": 0.007554390081545202, + "zero_ratio": 0.9878048780487805 + }, + { + "dimension": 53, + "mean": 0.011888452419420568, + "zero_ratio": 0.9878048780487805 + }, + { + "dimension": 51, + "mean": 0.0069082968118714125, + "zero_ratio": 0.9878048780487805 + }, + { + "dimension": 20, + "mean": 0.009622399399920208, + "zero_ratio": 0.9878048780487805 + }, + { + "dimension": 16, + "mean": 0.0041303202146437115, + "zero_ratio": 0.9878048780487805 + }, + { + "dimension": 8, + "mean": 0.006703466176986694, + "zero_ratio": 0.9878048780487805 + }, + { + "dimension": 127, + "mean": 0.010309220814123386, + "zero_ratio": 0.975609756097561 + } + ] + }, + "duplicates": { + "definition": "Exact duplicate detection by hashing each complete vector; near-duplicate pairs use Euclidean distance for inner-product inputs and the chosen distance otherwise, below a relative threshold derived from the sampled duplicate-detection distance median.", + "exact": { + "count": 82, + "definition": "Vectors whose complete byte representation is identical.", + "duplicate_groups": 1, + "duplicate_ratio": 0.012195121951219513, + "duplicate_vectors": 1, + "max_group_size": 2, + "sample_size": null, + "scope": "full", + "unique_vectors": 81 + }, + "near_duplicate": { + "available": true, + "definition": "Sampled pairs whose duplicate-detection distance is at most epsilon times the sampled pairwise-distance median.", + "median_pairwise_distance": 2.1404403169274016, + "metric": "euclidean", + "near_duplicate_pairs": 0, + "near_duplicate_ratio": 0.0, + "pair_count": 496, + "relative_epsilon": 0.001, + "sample_size": 32, + "scope": "sample", + "threshold": 0.0021404403169274014 + }, + "sample_size": null, + "scope": "mixed", + "scope_note": "exact duplicate detection covers the full split; the near-duplicate indicator covers the bounded pairwise sample" + }, + "invalid_values": { + "count": 82, + "definition": "Counts of NaN/Inf values in the selected split and the number of vectors containing at least one such value. Invalid entries are excluded from numeric summaries and treated as zero in norms.", + "invalid_ratio": 0.0, + "invalid_values": 0, + "sample_size": null, + "scope": "full", + "total_values": 10496, + "vectors_with_invalid": 0, + "vectors_with_invalid_ratio": 0.0 + }, + "nearest_neighbor_distances": { + "available": true, + "definition": "For each vector in the bounded sample, the distance to its nearest other vector in the same sample (diagonal excluded).", + "histogram": { + "bin_count": 20, + "bins": [ + -0.3201782801408797, + -0.26393662219866226, + -0.20769496425644488, + -0.1514533063142275, + -0.09521164837201007, + -0.038969990429792656, + 0.017271667512424704, + 0.07351332545464212, + 0.12975498339685954, + 0.1859966413390769, + 0.24223829928129437, + 0.2984799572235117, + 0.3547216151657291, + 0.41096327310794656, + 0.4672049310501639, + 0.5234465889923814, + 0.5796882469345987, + 0.6359299048768161, + 0.6921715628190335, + 0.748413220761251, + 0.8046548787034684 + ], + "counts": [ + 2, + 0, + 0, + 0, + 2, + 0, + 2, + 0, + 5, + 5, + 4, + 4, + 0, + 1, + 0, + 0, + 1, + 3, + 0, + 3 + ] + }, + "sample_size": 32, + "scope": "sample", + "stats": { + "count": 32, + "max": 0.8046548787034684, + "mean": 0.28006498574368643, + "min": -0.3201782801408797, + "p1": -0.3201782801408797, + "p25": 0.18049477848073447, + "p5": -0.18794500711724174, + "p50": 0.24096922320408964, + "p75": 0.35489981731960274, + "p95": 0.7624898795557888, + "p99": 0.7945659924604402, + "std": 0.27461634501677396 + } + }, + "norms": { + "count": 82, + "definition": "L2 norm ||x||_2 of every vector, computed over all finite values in the selected split.", + "histogram": { + "bin_count": 20, + "bins": [ + 0.0, + 0.11339979561368205, + 0.2267995912273641, + 0.3401993868410461, + 0.4535991824547282, + 0.5669989780684103, + 0.6803987736820922, + 0.7937985692957743, + 0.9071983649094564, + 1.0205981605231385, + 1.1339979561368205, + 1.2473977517505026, + 1.3607975473641845, + 1.4741973429778665, + 1.5875971385915486, + 1.7009969342052307, + 1.8143967298189128, + 1.9277965254325948, + 2.041196321046277, + 2.1545961166599588, + 2.267995912273641 + ], + "counts": [ + 1, + 0, + 0, + 0, + 1, + 1, + 1, + 3, + 6, + 2, + 7, + 9, + 9, + 7, + 10, + 9, + 6, + 2, + 6, + 2 + ] + }, + "non_finite_norms": 0, + "sample_size": null, + "scope": "full", + "stats": { + "count": 82, + "max": 2.267995912273641, + "mean": 1.4603532581317715, + "min": 0.0, + "p1": 0.3827605177131066, + "p25": 1.2327860776593744, + "p5": 0.8307837390171599, + "p50": 1.5066505307092668, + "p75": 1.738605616387497, + "p95": 2.081467246823919, + "p99": 2.2069716737233103, + "std": 0.4122392123060818 + } + }, + "outliers": { + "count": 82, + "definition": "Robust norm outliers: vectors whose norm has a modified z-score above 3.5, using median and median absolute deviation. Invalid (NaN/Inf) value counts are reported separately.", + "mad_norm": 0.25464266906928934, + "max_over_median_norm": 1.5053231429892142, + "median_norm": 1.5066505307092668, + "method": "modified z-score (median/MAD)", + "outlier_count": 1, + "outlier_ratio": 0.012195121951219513, + "sample_size": null, + "scope": "full", + "threshold": 1.3213481715974984 + }, + "pairwise_distances": { + "definition": "Distribution of distances between distinct vectors in a bounded random sample. The full M x M matrix is allocated only for the configured sample size, never for the complete dataset.", + "dense_sample_materialized": true, + "histogram": { + "bin_count": 20, + "bins": [ + -0.3201782801408797, + -0.2541693661338357, + -0.18816045212679172, + -0.12215153811974774, + -0.056142624112703765, + 0.009866289894340241, + 0.07587520390138419, + 0.14188411790842814, + 0.20789303191547215, + 0.27390194592251615, + 0.33991085992956016, + 0.40591977393660406, + 0.47192868794364806, + 0.5379376019506921, + 0.603946515957736, + 0.66995542996478, + 0.735964343971824, + 0.8019732579788679, + 0.867982171985912, + 0.9339910859929559, + 1.0 + ], + "counts": [ + 1, + 0, + 0, + 1, + 0, + 1, + 0, + 4, + 7, + 6, + 5, + 7, + 3, + 6, + 12, + 12, + 10, + 17, + 24, + 380 + ] + }, + "matrix_is_bounded": true, + "metric": "ip", + "pair_count": 496, + "requested_pairwise_sample_size": 32, + "sample_size": 32, + "scope": "sample", + "sparse_sparse_distance_loops": true, + "stats": { + "count": 496, + "max": 1.0, + "mean": 0.9131720688583472, + "min": -0.3201782801408797, + "p1": 0.18258860116424103, + "p25": 0.9621584448001095, + "p5": 0.41084182397992564, + "p50": 1.0, + "p75": 1.0, + "p95": 1.0, + "p99": 1.0, + "std": 0.19819997980418136 + } + }, + "sparsity": { + "all_zero_vector_ratio": 0.012195121951219513, + "all_zero_vectors": 1, + "cells": 10496, + "count": 82, + "definition": "Count and fraction of finite exactly-zero entries; non-zero values and density count finite non-zero entries, while NaN/Inf values are reported separately in invalid_values. For sparse vectors this also reports non-zero entries per vector and density = nnz / (count * dim).", + "density": 0.050495426829268296, + "dim": 128, + "explicit_zero_entries": 0, + "nnz_available": true, + "nnz_histogram": { + "bin_count": 20, + "bins": [ + 0.0, + 0.55, + 1.1, + 1.6500000000000001, + 2.2, + 2.75, + 3.3000000000000003, + 3.8500000000000005, + 4.4, + 4.95, + 5.5, + 6.050000000000001, + 6.6000000000000005, + 7.15, + 7.700000000000001, + 8.25, + 8.8, + 9.350000000000001, + 9.9, + 10.450000000000001, + 11.0 + ], + "counts": [ + 1, + 0, + 0, + 0, + 0, + 12, + 0, + 5, + 0, + 10, + 12, + 0, + 15, + 0, + 8, + 0, + 9, + 0, + 8, + 2 + ] + }, + "nnz_per_vector": { + "count": 82, + "max": 11.0, + "mean": 6.463414634146342, + "min": 0.0, + "p1": 2.43, + "p25": 5.0, + "p5": 3.0, + "p50": 7.0, + "p75": 8.0, + "p95": 10.0, + "p99": 11.0, + "std": 2.3746379541825693 + }, + "nnz_reason": null, + "non_zero_values": 530, + "representation": "sparse", + "sample_size": null, + "scope": "full", + "stored_entries": 530, + "zero_ratio": 0.9495045731707317, + "zero_values": 9966 + } + }, + "schema_version": "1.0", + "tool": { + "name": "vsag-dataset-profiler", + "version": "0.1.0" + }, + "warnings": [ + "sparse dimensionality is inferred as max(feature id) + 1 during profiling" + ] +} diff --git a/tools/dataset_profiler/requirements.txt b/tools/dataset_profiler/requirements.txt new file mode 100644 index 0000000000..774fcf2775 --- /dev/null +++ b/tools/dataset_profiler/requirements.txt @@ -0,0 +1,7 @@ +# Runtime dependencies for the VSAG dataset profiler (tools/dataset_profiler). +numpy>=1.24 +h5py>=3.8 + +# For the test suite only: +# pip install pytest +# python -m pytest tests -q diff --git a/tools/dataset_profiler/tests/conftest.py b/tools/dataset_profiler/tests/conftest.py new file mode 100644 index 0000000000..348bd2a889 --- /dev/null +++ b/tools/dataset_profiler/tests/conftest.py @@ -0,0 +1,125 @@ +# Copyright 2024-present the vsag project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Shared fixtures for the dataset profiler tests.""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import numpy as np +import pytest + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + + +def encode_sparse_record(ids, values) -> bytes: + """Encode one VSAG sparse record: uint32 len | uint32 ids | float32 vals.""" + ids = np.asarray(ids, dtype=" np.random.Generator: + return np.random.Generator(np.random.PCG64(20260101)) + + +@pytest.fixture +def dense_matrix(rng) -> np.ndarray: + """Small deterministic dense matrix with duplicates, NaN, and an outlier.""" + base = rng.normal(0.0, 1.0, size=(60, 12)).astype(np.float32) + # exact duplicates of rows 0..1 + base[2] = base[0] + base[5] = base[1] + # NaN and Inf land in row 7 + base[7, 0] = np.float32("nan") + base[7, 1] = np.float32("inf") + # one huge outlier row + base[9] = base[9] * 500.0 + # an all-zero row + base[11] = 0.0 + return base + + +@pytest.fixture +def dense_path(tmp_path, dense_matrix) -> Path: + import h5py + + path = tmp_path / "dense_toy.hdf5" + with h5py.File(str(path), "w") as f: + f.attrs["distance"] = "l2" + f.create_dataset("train", data=dense_matrix) + f.create_dataset("test", data=dense_matrix[:6]) + return path + + +@pytest.fixture +def sparse_records(): + """Deterministic list of (ids, values) sparse vectors.""" + rows = [] + gen = np.random.Generator(np.random.PCG64(99)) + for _ in range(40): + length = int(gen.integers(2, 8)) + ids = np.sort(gen.choice(48, size=length, replace=False)).astype(np.uint32) + values = gen.uniform(0.1, 1.0, size=length).astype(np.float32) + rows.append((ids, values)) + # duplicate of the first record and one empty record + rows.append((rows[0][0].copy(), rows[0][1].copy())) + rows.append((np.zeros(0, dtype=np.uint32), np.zeros(0, dtype=np.float32))) + return rows + + +def _sparse_bytes_and_offsets(rows): + stream = bytearray() + offsets = [] + for ids, values in rows: + offsets.append(len(stream)) + stream += encode_sparse_record(ids, values) + offsets.append(len(stream)) + return bytes(stream), offsets + + +@pytest.fixture +def sparse_path(tmp_path, sparse_records) -> Path: + import h5py + + stream, offsets = _sparse_bytes_and_offsets(sparse_records) + path = tmp_path / "sparse_toy.hdf5" + with h5py.File(str(path), "w") as f: + f.attrs["type"] = "sparse" + f.attrs["distance"] = "ip" + f.create_dataset("train", data=np.frombuffer(stream, dtype=np.int8).copy()) + f.create_dataset("train_offsets", data=np.asarray(offsets, dtype=np.uint64)) + return path + + +@pytest.fixture +def sparse_no_offsets_path(tmp_path, sparse_records) -> Path: + import h5py + + stream, _ = _sparse_bytes_and_offsets(sparse_records) + path = tmp_path / "sparse_no_offsets.hdf5" + with h5py.File(str(path), "w") as f: + f.attrs["type"] = "sparse" + f.create_dataset("train", data=np.frombuffer(stream, dtype=np.int8).copy()) + return path diff --git a/tools/dataset_profiler/tests/test_ai.py b/tools/dataset_profiler/tests/test_ai.py new file mode 100644 index 0000000000..220790f533 --- /dev/null +++ b/tools/dataset_profiler/tests/test_ai.py @@ -0,0 +1,403 @@ +# Copyright 2024-present the vsag project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for the optional AI interpretation adapter.""" + +from __future__ import annotations + +import json + +import pytest + +from dataset_profiler import ai +from dataset_profiler.ai import ( + AIAnalysisError, + MockProvider, + OpenAICompatibleProvider, + aggregate_payload, + analyze_report, + build_prompt, + parse_and_validate_response, + provider_from_env, +) +from dataset_profiler.profiler import profile_dataset +from dataset_profiler.stats import dump_json + + +def _report(dense_path): + return profile_dataset(str(dense_path), sample_size=24, pairwise_sample_size=12) + + +def _canned_response(): + return json.dumps( + { + "summary": "The sample is anisotropic with a small duplicate count.", + "findings": [ + { + "title": "Dominant direction", + "detail": "PC1 explains most variance.", + "evidence": ["metrics.anisotropy.top1_eigenvalue_share"], + "confidence": "medium", + } + ], + "recommendations": [ + { + "title": "Consider HGraph", + "detail": "Hypothesis: a graph index may help recall.", + "evidence": ["metrics.norms.stats.p50", "metrics.pairwise_distances.stats.p50"], + "validation": "Compare recall and QPS in tools/eval before adopting.", + "confidence": "low", + } + ], + "limitations": ["Sampled metrics only."], + "additional_measurements": ["Full pairwise distance run."], + } + ) + + +# --------------------------------------------------------------------------- +# Prompt / payload privacy boundary +# --------------------------------------------------------------------------- +def test_prompt_contains_only_aggregate_metrics(dense_path, dense_matrix): + report = _report(dense_path) + prompt = build_prompt(report) + payload = aggregate_payload(report) + assert set(payload) == { + "schema_version", + "tool", + "split", + "dataset", + "config", + "metrics", + "warnings", + } + assert "Allowed evidence metric keys:" in prompt + # no raw vector row is ever part of the prompt + for row in dense_matrix[:3]: + assert repr(row.tolist()) not in prompt + assert "raw_vectors" not in prompt + + +def test_prompt_is_jsonable(dense_path): + report = _report(dense_path) + payload = aggregate_payload(report) + json.dumps(payload, allow_nan=False, sort_keys=True) + + +def test_aggregate_payload_omits_arbitrary_dataset_object_names(dense_path): + report = _report(dense_path) + report["dataset"]["datasets"].append("customer-private-label") + report["dataset"]["attributes"]["private_label"] = "customer-secret" + + payload = aggregate_payload(report) + dataset = payload["dataset"] + assert set(dataset) == { + "vector_type", + "dtype", + "count", + "dim", + "dim_inferred", + "distance_metric", + "has_offsets", + } + assert "datasets" not in dataset + assert "attributes" not in dataset + prompt = build_prompt(report) + assert "customer-private-label" not in prompt + assert "customer-secret" not in prompt + + +def test_aggregate_payload_excludes_per_vector_projections(dense_path): + report = profile_dataset( + str(dense_path), sample_size=24, pairwise_sample_size=12, clusters=3 + ) + assert report["metrics"]["clustering"]["projection_points"] + assert report["metrics"]["clustering"]["labels"] + payload = aggregate_payload(report) + clustering = payload["metrics"]["clustering"] + assert "projection_points" not in clustering + assert "labels" not in clustering + # aggregate summary fields are still forwarded + assert "cluster_sizes" in clustering + assert clustering["n_clusters"] == 3 + prompt = build_prompt(report) + assert "projection_points" not in prompt + + + +def test_aggregate_payload_forwards_budget_unavailability(): + report = { + "metrics": { + "norms": {"available": False, "reason": "norm budget exceeded"}, + "outliers": {"available": False, "reason": "norm budget exceeded"}, + "sparsity": { + "nnz_available": False, + "nnz_reason": "nnz budget exceeded", + }, + } + } + metrics = aggregate_payload(report)["metrics"] + for name in ("norms", "outliers"): + assert metrics[name]["available"] is False + assert metrics[name]["reason"] == "norm budget exceeded" + assert metrics["sparsity"]["nnz_available"] is False + assert metrics["sparsity"]["nnz_reason"] == "nnz budget exceeded" + + +# --------------------------------------------------------------------------- +# Response validation +# --------------------------------------------------------------------------- +def test_valid_response_is_parsed(dense_path): + report = _report(dense_path) + block = parse_and_validate_response(_canned_response(), report) + assert block["summary"].startswith("The sample is anisotropic") + assert block["findings"][0]["evidence"] == ["metrics.anisotropy.top1_eigenvalue_share"] + assert block["recommendations"][0]["validation"].startswith("Compare recall") + assert block["recommendations"][0]["confidence"] == "low" + + +def test_response_inside_markdown_fences_is_parsed(dense_path): + report = _report(dense_path) + fenced = "```json\n" + _canned_response() + "\n```" + block = parse_and_validate_response(fenced, report) + assert block["findings"] + + +def test_malformed_responses_are_rejected(dense_path): + report = _report(dense_path) + bad_cases = [ + "", + "not json at all", + json.dumps([1, 2, 3]), + json.dumps({"findings": [{"title": "x", "detail": "y", "evidence": ["metrics.norms.stats.mean"]}], + "recommendations": []}), # missing summary / empty recommendations + json.dumps({"summary": "s", "findings": [], "recommendations": []}), + json.dumps({"summary": "s", "findings": "oops", "recommendations": []}), + ] + for text in bad_cases: + with pytest.raises(AIAnalysisError): + parse_and_validate_response(text, report) + + +def test_evidence_must_reference_real_metric_keys(dense_path): + report = _report(dense_path) + response = json.loads(_canned_response()) + response["findings"][0]["evidence"] = ["metrics.does_not_exist"] + with pytest.raises(AIAnalysisError, match="unknown evidence key"): + parse_and_validate_response(json.dumps(response), report) + + response = json.loads(_canned_response()) + response["findings"][0]["evidence"] = [] + with pytest.raises(AIAnalysisError, match="non-empty evidence"): + parse_and_validate_response(json.dumps(response), report) + + response = json.loads(_canned_response()) + response["recommendations"][0].pop("validation") + with pytest.raises(AIAnalysisError, match="validation"): + parse_and_validate_response(json.dumps(response), report) + + response = json.loads(_canned_response()) + response["findings"][0]["confidence"] = "certain" + with pytest.raises(AIAnalysisError, match="confidence"): + parse_and_validate_response(json.dumps(response), report) + + +# --------------------------------------------------------------------------- +# Providers +# --------------------------------------------------------------------------- +def test_analyze_report_ok_path(dense_path): + report = _report(dense_path) + provider = MockProvider(_canned_response()) + block = analyze_report(report, provider=provider) + assert block["status"] == "ok" + assert block["provider"] == "mock" + assert block["model"] == "mock-model" + assert provider.last_prompt is not None + # AI block never breaks JSON serialization of the full report + report["ai_analysis"] = block + json.loads(dump_json(report)) + + +def test_analyze_report_degrades_on_provider_failure(dense_path): + class FailingProvider: + name = "failing" + model = "m" + + def complete(self, prompt): + raise ai.AIAnalysisError("AI provider is unreachable") + + report = _report(dense_path) + block = analyze_report(report, provider=FailingProvider()) + assert block["status"] == "error" + assert "unreachable" in block["error"] + + +def test_analyze_report_redacts_secrets_in_errors(dense_path): + class LeakyProvider: + name = "leaky" + model = "m" + + def complete(self, prompt): + raise RuntimeError("auth failed with sk-abcdefghijkl12345678") + + report = _report(dense_path) + block = analyze_report(report, provider=LeakyProvider()) + assert block["status"] == "error" + assert "sk-abcdefghijkl12345678" not in block["error"] + assert "[redacted]" in block["error"] + dump_json({"ai_analysis": block}) + + +def test_analyze_report_rejects_malformed_provider_json(dense_path): + provider = MockProvider("this is not json") + block = analyze_report(_report(dense_path), provider=provider) + assert block["status"] == "error" + + +def test_provider_requires_https_and_credentials(): + with pytest.raises(AIAnalysisError): + OpenAICompatibleProvider(api_key="", model="m") + with pytest.raises(AIAnalysisError): + OpenAICompatibleProvider(api_key="k", model="") + with pytest.raises(AIAnalysisError): + OpenAICompatibleProvider(api_key="k", model="m", base_url="http://insecure.example/v1") + + +def test_provider_from_env_resolution(): + with pytest.raises(AIAnalysisError): + provider_from_env(env={}) + provider = provider_from_env( + env={"VSAG_AI_API_KEY": "k", "VSAG_AI_MODEL": "m", "VSAG_AI_BASE_URL": "https://example.com/v1"} + ) + assert provider.model == "m" + assert provider.base_url == "https://example.com/v1" + fallback = provider_from_env( + env={"OPENAI_API_KEY": "k2", "VSAG_AI_MODEL": "m2"}, + base_url="https://other.example/v1", + ) + assert fallback.base_url == "https://other.example/v1" + + +def test_explicit_empty_vsag_key_does_not_use_openai_fallback(): + with pytest.raises(AIAnalysisError, match="no API key"): + provider_from_env( + env={ + "VSAG_AI_API_KEY": "", + "OPENAI_API_KEY": "must-not-be-used", + "VSAG_AI_MODEL": "m", + } + ) + + +def test_provider_rejects_redirects_before_forwarding_credentials(): + provider = OpenAICompatibleProvider( + api_key="sk-supersecretvalue123", model="m", base_url="https://api.example.com/v1" + ) + handler = next( + item for item in provider._opener.handlers if isinstance(item, ai._RejectRedirectHandler) + ) + request = ai.urllib.request.Request( + "https://api.example.com/v1/chat/completions", + headers={"Authorization": "Bearer sk-supersecretvalue123"}, + ) + with pytest.raises(AIAnalysisError, match="redirects are disabled"): + handler.redirect_request( + request, None, 302, "Found", {}, "https://attacker.example/collect" + ) + + +def test_openai_request_body_excludes_credentials_and_vectors(monkeypatch, dense_path, dense_matrix): + captured = {} + + class FakeResponse: + def read(self): + return json.dumps({"choices": [{"message": {"content": _canned_response()}}]}).encode() + + def __enter__(self): + return self + + def __exit__(self, *args): + return False + + def fake_open(request, timeout=None): + captured["headers"] = dict(request.headers) + captured["body"] = request.data.decode("utf-8") + captured["url"] = request.full_url + return FakeResponse() + + provider = OpenAICompatibleProvider( + api_key="sk-supersecretvalue123", model="m", base_url="https://api.example.com/v1" + ) + monkeypatch.setattr(provider._opener, "open", fake_open) + report = _report(dense_path) + block = analyze_report(report, provider=provider) + assert block["status"] == "ok" + assert captured["url"].endswith("/chat/completions") + assert captured["headers"].get("Authorization") == "Bearer sk-supersecretvalue123" + assert "sk-supersecretvalue123" not in captured["body"] + for row in dense_matrix[:3]: + assert repr(row.tolist()) not in captured["body"] + # the key must never reach the persisted report either + report["ai_analysis"] = block + assert "sk-supersecretvalue123" not in dump_json(report) + + +@pytest.mark.parametrize("api_key", ["abc+defghijklmnop\n", "abc/defghijkl\n"]) +def test_malformed_configured_keys_are_rejected_without_leaking(dense_path, api_key): + block = analyze_report( + _report(dense_path), + env={"VSAG_AI_API_KEY": api_key, "VSAG_AI_MODEL": "m"}, + ) + assert block["status"] == "error" + assert "visible ASCII" in block["error"] + assert api_key.strip() not in block["error"] + assert json.dumps(api_key)[1:-1] not in block["error"] + + +def test_analyze_report_redacts_configured_key_in_escaped_exception(dense_path): + api_key = "abc/defghijkl\n" + + class LeakyProvider: + name = "leaky" + model = "m" + redaction_secrets = (api_key,) + + def complete(self, prompt): + raise ValueError("invalid Authorization value %r" % ("Bearer " + api_key)) + + block = analyze_report(_report(dense_path), provider=LeakyProvider()) + assert block["status"] == "error" + assert api_key not in block["error"] + assert api_key.strip() not in block["error"] + assert json.dumps(api_key)[1:-1] not in block["error"] + assert "[redacted]" in block["error"] + dump_json({"ai_analysis": block}) + + +@pytest.mark.parametrize("bare_string", [False, True]) +def test_mock_provider_follows_redaction_secrets_contract(dense_path, bare_string): + api_key = "custom-provider-secret" + + class LeakyMockProvider(MockProvider): + def complete(self, prompt): + raise ValueError("request failed with credential " + api_key) + + secrets = api_key if bare_string else (api_key,) + provider = LeakyMockProvider("", secrets=secrets) + block = analyze_report(_report(dense_path), provider=provider) + assert provider.redaction_secrets == (api_key,) + assert block["status"] == "error" + assert api_key not in block["error"] + assert "request failed with credential" in block["error"] + assert "[redacted]" in block["error"] diff --git a/tools/dataset_profiler/tests/test_cli.py b/tools/dataset_profiler/tests/test_cli.py new file mode 100644 index 0000000000..5239156f9f --- /dev/null +++ b/tools/dataset_profiler/tests/test_cli.py @@ -0,0 +1,135 @@ +# Copyright 2024-present the vsag project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""End-to-end tests for the command-line interface.""" + +from __future__ import annotations + +import json + +import pytest + +from dataset_profiler import cli + + +def test_default_outputs_written_next_to_input(dense_path, capsys): + rc = cli.main([str(dense_path), "--sample-size", "16", "--pairwise-sample-size", "8"], env={}) + assert rc == 0 + json_path = dense_path.with_name("dense_toy.profile.json") + html_path = dense_path.with_name("dense_toy.profile.html") + assert json_path.is_file() and html_path.is_file() + report = json.loads(json_path.read_text(encoding="utf-8")) + assert report["schema_version"] == "1.0" + assert "ai_analysis" not in report + assert html_path.read_text(encoding="utf-8").startswith("") + assert "wrote" in capsys.readouterr().err + + +def test_explicit_output_paths_and_stdout(dense_path, tmp_path, capsys): + out_json = tmp_path / "nested" / "report.json" + rc = cli.main( + [str(dense_path), "--out-json", str(out_json), "--sample-size", "8"], env={} + ) + assert rc == 0 + assert out_json.is_file() + assert not (tmp_path / "nested" / "report.html").exists() + + rc = cli.main([str(dense_path), "--out-json", "-", "--sample-size", "8"], env={}) + assert rc == 0 + stdout = capsys.readouterr().out + parsed = json.loads(stdout) + assert parsed["dataset"]["count"] == 60 + + +def test_two_cli_runs_are_byte_identical(dense_path, tmp_path): + a = tmp_path / "a.json" + b = tmp_path / "b.json" + args = ["--out-json", str(a), "--sample-size", "20", "--pairwise-sample-size", "10"] + assert cli.main([str(dense_path)] + args, env={}) == 0 + args_b = ["--out-json", str(b), "--sample-size", "20", "--pairwise-sample-size", "10"] + assert cli.main([str(dense_path)] + args_b, env={}) == 0 + assert a.read_bytes() == b.read_bytes() + + +def test_invalid_dataset_returns_error_exit_code(tmp_path): + missing = tmp_path / "nope.hdf5" + rc = cli.main([str(missing)], env={}) + assert rc == 1 + + +def test_invalid_config_is_rejected(dense_path): + rc = cli.main([str(dense_path), "--clusters", "-1"], env={}) + assert rc == 1 + + +def test_unsupported_metric_choice_rejected(dense_path): + with pytest.raises(SystemExit): + cli.main([str(dense_path), "--metric", "hamming"], env={}) + + +def test_ai_failure_still_writes_reports(dense_path, tmp_path, capsys): + out = tmp_path / "profile" + rc = cli.main( + [ + str(dense_path), + "--ai", + "--sample-size", "8", + "--out-json", str(out / "r.json"), + "--out-html", str(out / "r.html"), + ], + env={}, # no credentials configured + ) + assert rc == 0 + err = capsys.readouterr().err + assert "AI analysis failed" in err + report = json.loads((out / "r.json").read_text(encoding="utf-8")) + assert report["ai_analysis"]["status"] == "error" + assert (out / "r.html").read_text(encoding="utf-8").find("AI analysis unavailable") != -1 + + +def test_ai_enabled_by_env_var_without_flag(dense_path, tmp_path, capsys): + out = tmp_path / "r.json" + rc = cli.main( + [str(dense_path), "--sample-size", "8", "--out-json", str(out)], + env={"VSAG_DATASET_PROFILER_AI": "1"}, + ) + assert rc == 0 + report = json.loads(out.read_text(encoding="utf-8")) + assert report["ai_analysis"]["status"] == "error" # still no credentials + + +def test_sparse_cli_run_produces_html(sparse_path, tmp_path): + out = tmp_path / "s.profile.html" + rc = cli.main( + [ + str(sparse_path), + "--sample-size", "12", + "--pairwise-sample-size", "6", + "--out-json", str(tmp_path / "s.json"), + "--out-html", str(out), + ], + env={} + ) + assert rc == 0 + assert out.is_file() + report = json.loads((tmp_path / "s.json").read_text(encoding="utf-8")) + assert report["dataset"]["vector_type"] == "sparse" + + +def test_help_mentions_key_flags(): + parser = cli.build_parser() + text = parser.format_help() + for flag in ("--seed", "--sample-size", "--pairwise-sample-size", "--ai", "--out-html"): + assert flag in text + assert "omitted when --out-json is given without --out-html" in " ".join(text.split()) diff --git a/tools/dataset_profiler/tests/test_hdf5.py b/tools/dataset_profiler/tests/test_hdf5.py new file mode 100644 index 0000000000..55891129dd --- /dev/null +++ b/tools/dataset_profiler/tests/test_hdf5.py @@ -0,0 +1,302 @@ +# Copyright 2024-present the vsag project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for the HDF5 readers covering both supported layouts.""" + +from __future__ import annotations + +import numpy as np +import pytest + +import dataset_profiler.hdf5 as hdf5_module +from dataset_profiler.hdf5 import DatasetFormatError, HDF5Dataset + + +def test_dense_metadata_and_blocks(tmp_path, dense_matrix): + import h5py + + path = tmp_path / "dense.hdf5" + with h5py.File(str(path), "w") as f: + f.attrs["distance"] = "euclidean" + f.create_dataset("train", data=dense_matrix) + with HDF5Dataset(str(path), split="train") as dataset: + meta = dataset.metadata + assert meta.vector_type == "dense" + assert meta.dtype == "FLOAT32" + assert meta.count == dense_matrix.shape[0] + assert meta.dim == dense_matrix.shape[1] + assert meta.distance == "euclidean" + assert meta.dim_inferred is False + blocks = list(dataset.iter_dense_blocks(block_rows=25)) + assert sum(block.shape[0] for _, block in blocks) == dense_matrix.shape[0] + np.testing.assert_allclose(blocks[0][1], dense_matrix[:25]) + + +def test_dense_row_selection_matches_source(tmp_path, rng): + import h5py + + matrix = rng.integers(-127, 128, size=(50, 12)).astype(np.int8) + path = tmp_path / "dense.hdf5" + with h5py.File(str(path), "w") as f: + f.create_dataset("train", data=matrix) + with HDF5Dataset(str(path)) as dataset: + assert dataset.metadata.dtype == "INT8" + rows = dataset.read_dense_rows([0, 5, 40]) + np.testing.assert_array_equal(rows, matrix[[0, 5, 40]]) + + +def test_unsigned_dense_uses_evaluator_int8_conversion(tmp_path): + import h5py + + matrix = np.array([[0, 127], [128, 255]], dtype=np.uint8) + path = tmp_path / "unsigned_dense.hdf5" + with h5py.File(str(path), "w") as f: + f.create_dataset("train", data=matrix) + with h5py.File(str(path), "r") as f: + # Use HDF5's numeric conversion as the oracle, not a byte reinterpretation. + expected = f["train"].astype(np.int8)[:] + np.testing.assert_array_equal(expected, [[0, 127], [127, 127]]) + with HDF5Dataset(str(path)) as dataset: + assert dataset.metadata.dtype == "INT8" + blocks = list(dataset.iter_dense_blocks()) + assert blocks[0][1].dtype == np.dtype(np.int8) + np.testing.assert_array_equal(blocks[0][1], expected) + np.testing.assert_array_equal(dataset.read_dense_rows([0, 1]), expected) + + +def test_signed_dense_preserves_int8_boundaries(tmp_path): + import h5py + + matrix = np.array([[-128, -1, 0, 127]], dtype=np.int8) + path = tmp_path / "signed_dense.hdf5" + with h5py.File(str(path), "w") as f: + f.create_dataset("train", data=matrix) + with HDF5Dataset(str(path)) as dataset: + assert dataset.metadata.dtype == "INT8" + np.testing.assert_array_equal( + list(dataset.iter_dense_blocks())[0][1], matrix + ) + np.testing.assert_array_equal(dataset.read_dense_rows([0]), matrix) + + +def test_dense_rejects_unsorted_row_indices(tmp_path, dense_matrix): + import h5py + + path = tmp_path / "dense.hdf5" + with h5py.File(str(path), "w") as f: + f.create_dataset("train", data=dense_matrix) + with HDF5Dataset(str(path)) as dataset: + with pytest.raises(ValueError): + dataset.read_dense_rows([5, 1]) + + +def test_unsupported_float64_rejected(tmp_path): + import h5py + + path = tmp_path / "bad_dtype.hdf5" + with h5py.File(str(path), "w") as f: + f.create_dataset("train", data=np.zeros((4, 3), dtype=np.float64)) + with pytest.raises(DatasetFormatError): + HDF5Dataset(str(path)) + + +def test_missing_split_rejected(tmp_path, dense_matrix): + import h5py + + path = tmp_path / "dense.hdf5" + with h5py.File(str(path), "w") as f: + f.create_dataset("train", data=dense_matrix) + with pytest.raises(DatasetFormatError, match="missing required HDF5 dataset"): + HDF5Dataset(str(path), split="test") + + +def test_empty_or_wrong_shape_dense_rejected(tmp_path): + import h5py + + empty = tmp_path / "empty.hdf5" + with h5py.File(str(empty), "w") as f: + f.create_dataset("train", data=np.zeros((0, 4), dtype=np.float32)) + with pytest.raises(DatasetFormatError, match="empty"): + HDF5Dataset(str(empty)) + + flat = tmp_path / "flat.hdf5" + with h5py.File(str(flat), "w") as f: + f.create_dataset("train", data=np.zeros(10, dtype=np.float32)) + with pytest.raises(DatasetFormatError, match="must be 2-D"): + HDF5Dataset(str(flat)) + + +def test_multi_vector_rejected_and_unknown_type_rejected(tmp_path): + import h5py + + mv = tmp_path / "mv.hdf5" + with h5py.File(str(mv), "w") as f: + f.attrs["type"] = "multi_vector" + f.create_dataset("train", data=np.zeros((3, 4), dtype=np.float32)) + with pytest.raises(DatasetFormatError, match="multi_vector"): + HDF5Dataset(str(mv)) + + weird = tmp_path / "weird.hdf5" + with h5py.File(str(weird), "w") as f: + f.attrs["type"] = "ternary" + f.create_dataset("train", data=np.zeros((3, 4), dtype=np.float32)) + with pytest.raises(DatasetFormatError, match="unsupported HDF5 dataset type"): + HDF5Dataset(str(weird)) + + +def test_sparse_with_offsets_round_trip(sparse_path, sparse_records): + with HDF5Dataset(str(sparse_path)) as dataset: + meta = dataset.metadata + assert meta.vector_type == "sparse" + assert meta.count == len(sparse_records) + assert meta.has_offsets is True + records = list(dataset.iter_sparse_records()) + assert len(records) == len(sparse_records) + for position, ((index, ids, values), (ref_ids, ref_values)) in enumerate( + zip(records, sparse_records) + ): + assert index == position + np.testing.assert_array_equal(ids, ref_ids) + np.testing.assert_allclose(values, ref_values) + + +def test_oversized_sparse_offsets_use_bounded_sequential_decoder( + sparse_path, sparse_records, monkeypatch +): + monkeypatch.setattr(hdf5_module, "_OFFSET_INDEX_BUDGET_BYTES", 1) + monkeypatch.setattr(hdf5_module, "_OFFSET_VALIDATION_ROWS", 2) + with HDF5Dataset(str(sparse_path)) as dataset: + assert dataset.metadata.has_offsets is True + assert dataset.metadata.count == len(sparse_records) + assert dataset._offsets is None + assert any("bounded sequential decoder" in w for w in dataset.metadata.warnings) + records = list(dataset.iter_sparse_records(max_chunk_bytes=16)) + assert len(records) == len(sparse_records) + for (_, ids, values), (ref_ids, ref_values) in zip(records, sparse_records): + np.testing.assert_array_equal(ids, ref_ids) + np.testing.assert_allclose(values, ref_values) + + +def test_oversized_sparse_offsets_reject_mismatched_record_boundary( + sparse_path, monkeypatch +): + import h5py + + # Moving only an internal boundary preserves the count, monotonicity, and + # sentinel, so validation must compare it with the decoded record position. + with h5py.File(str(sparse_path), "r+") as handle: + offsets = handle["train_offsets"] + offsets[1] = int(offsets[1]) + 1 + + monkeypatch.setattr(hdf5_module, "_OFFSET_INDEX_BUDGET_BYTES", 1) + monkeypatch.setattr(hdf5_module, "_OFFSET_VALIDATION_ROWS", 2) + with HDF5Dataset(str(sparse_path)) as dataset: + assert dataset._offsets is None + with pytest.raises(DatasetFormatError, match="decoded record 0.*train_offsets"): + next(dataset.iter_sparse_records(max_chunk_bytes=16)) + + +def test_sparse_without_offsets_matches_offsets_version( + sparse_path, sparse_no_offsets_path, sparse_records +): + with HDF5Dataset(str(sparse_path)) as with_offsets: + a = [(ids, values) for _, ids, values in with_offsets.iter_sparse_records()] + with HDF5Dataset(str(sparse_no_offsets_path)) as without_offsets: + b = [(ids, values) for _, ids, values in without_offsets.iter_sparse_records()] + assert len(a) == len(b) == len(sparse_records) + for (ida, va), (idb, vb) in zip(a, b): + np.testing.assert_array_equal(ida, idb) + np.testing.assert_allclose(va, vb) + assert without_offsets.metadata.count is None + assert any("no '/train_offsets'" in w for w in without_offsets.metadata.warnings) + + +def test_sparse_trailing_bytes_rejected(tmp_path, sparse_records): + import h5py + + from conftest import encode_sparse_record + + stream = b"".join(encode_sparse_record(ids, values) for ids, values in sparse_records) + path = tmp_path / "truncated.hdf5" + with h5py.File(str(path), "w") as f: + f.attrs["type"] = "sparse" + f.create_dataset("train", data=np.frombuffer(stream + b"\x01\x02", dtype=np.int8).copy()) + with HDF5Dataset(str(path)) as dataset: + with pytest.raises(DatasetFormatError, match="trailing"): + list(dataset.iter_sparse_records()) + + +def test_sparse_offset_validation_errors(tmp_path, sparse_records, monkeypatch): + import h5py + + from conftest import encode_sparse_record + + monkeypatch.setattr(hdf5_module, "_OFFSET_VALIDATION_ROWS", 2) + + stream = b"".join(encode_sparse_record(ids, values) for ids, values in sparse_records) + good_offsets = [0] + for ids, values in sparse_records: + good_offsets.append(good_offsets[-1] + 4 + 8 * ids.size) + + bad = tmp_path / "bad_offsets.hdf5" + with h5py.File(str(bad), "w") as f: + f.attrs["type"] = "sparse" + f.create_dataset("train", data=np.frombuffer(stream, dtype=np.int8).copy()) + offsets = np.asarray(good_offsets, dtype=np.uint64) + offsets[-1] += 1 # sentinel no longer matches the byte size + f.create_dataset("train_offsets", data=offsets) + with pytest.raises(DatasetFormatError, match="sentinel"): + HDF5Dataset(str(bad)) + + unsorted = tmp_path / "unsorted_offsets.hdf5" + with h5py.File(str(unsorted), "w") as f: + f.attrs["type"] = "sparse" + f.create_dataset("train", data=np.frombuffer(stream, dtype=np.int8).copy()) + offsets = np.asarray(good_offsets, dtype=np.uint64) + offsets[1], offsets[2] = offsets[2], offsets[1] + f.create_dataset("train_offsets", data=offsets) + with pytest.raises(DatasetFormatError, match="non-decreasing"): + HDF5Dataset(str(unsorted)) + + +def test_read_sparse_rows_returns_requested_indices(sparse_path): + with HDF5Dataset(str(sparse_path)) as dataset: + rows = dataset.read_sparse_rows([2, 10, 13]) + assert set(rows) == {2, 10, 13} + + +def test_sparse_without_offsets_handles_records_across_tiny_chunks( + sparse_no_offsets_path, sparse_records +): + with HDF5Dataset(str(sparse_no_offsets_path)) as dataset: + records = list(dataset.iter_sparse_records(max_chunk_bytes=16)) + assert len(records) == len(sparse_records) + for (_, ids, values), (ref_ids, ref_values) in zip(records, sparse_records): + np.testing.assert_array_equal(ids, ref_ids) + np.testing.assert_allclose(values, ref_values) + + +def test_sparse_impossible_length_is_rejected_before_buffering(tmp_path): + import struct + + import h5py + + path = tmp_path / "impossible_length.hdf5" + stream = np.frombuffer(struct.pack("") + assert "alert(1)", + } + text = render_html(report) + assert "" not in text + assert "<script>" in text + + +def test_warnings_are_rendered(sparse_no_offsets_path): + report = profile_dataset(str(sparse_no_offsets_path), sample_size=8, pairwise_sample_size=8) + text = render_html(report) + assert "Warnings" in text + + +def test_page_title_and_header_use_input_file_name(dense_path): + text = render_html(_dense_report(dense_path)) + assert "VSAG dataset profile: dense_toy.hdf5" in text + assert "dense_toy.hdf5 ·" in text + + +def test_overview_shows_effective_metric(dense_path): + # the file attribute is 'l2' (euclidean) but the run used angular + report = profile_dataset( + str(dense_path), metric="angular", sample_size=8, pairwise_sample_size=8 + ) + text = render_html(report) + assert '
    angular
    ' in text + + +def test_large_constant_range_renders_with_representable_padding(tmp_path): + import h5py + import numpy as np + + path = tmp_path / "huge_range.hdf5" + with h5py.File(str(path), "w") as handle: + handle.attrs["distance"] = "euclidean" + handle.create_dataset( + "train", + data=np.array([[1e20, 0.0], [0.0, 2e20], [2e20, 1e20]], dtype=np.float32), + ) + report = profile_dataset( + str(path), sample_size=3, pairwise_sample_size=3, clusters=2 + ) + # per-dimension means are constant at ~1e20, so the chart range must be + # padded by a representable amount instead of +/-1.0 + assert report["metrics"]["dimensions"]["per_dimension"]["mean"]["values"][0] == ( + report["metrics"]["dimensions"]["per_dimension"]["mean"]["values"][1] + ) + text = render_html(report) + assert "Per-dimension distributions" in text + assert re.search(r'(cx|cy)="[^"]*(nan|inf)', text) is None + + +def test_downsampled_points_labelled_with_source_dimension_ranges(tmp_path): + import h5py + import numpy as np + + path = tmp_path / "bucket.hdf5" + with h5py.File(str(path), "w") as handle: + handle.attrs["distance"] = "euclidean" + handle.create_dataset( + "train", + data=np.arange(16, dtype=np.float32).reshape(4, 4), + ) + report = profile_dataset( + str(path), + sample_size=4, + pairwise_sample_size=4, + max_dimension_points=2, + ) + assert report["metrics"]["dimensions"]["per_dimension"]["mean"]["downsampled"] is True + text = render_html(report) + assert "dimensions 0-1" in text + assert "dimensions 2-3" in text + + +def test_downsampled_points_use_exact_nonuniform_source_ranges(tmp_path): + import h5py + import numpy as np + + path = tmp_path / "nonuniform_buckets.hdf5" + with h5py.File(str(path), "w") as handle: + handle.attrs["distance"] = "euclidean" + handle.create_dataset( + "train", data=np.arange(10, dtype=np.float32).reshape(2, 5) + ) + report = profile_dataset( + str(path), sample_size=2, pairwise_sample_size=2, max_dimension_points=2 + ) + text = render_html(report) + assert "dimensions 0-1" in text + assert "dimensions 2-4" in text + assert "dimensions 0-2" not in text + assert "dimensions 3-4" not in text + + +def test_downsampled_points_use_the_exact_linspace_partition(): + data = downsample(np.arange(784, dtype=np.float64), max_points=768) + text = _chart_line(data, "dimension mean", "Coordinate", "Mean", "#2563eb") + # Bucket 240 is where exact integer division and integer linspace disagree. + assert "dimensions 244-245" in text + assert "dimension 245:" not in text + + +def test_sparse_zero_total_includes_implicit_zeros(sparse_path): + report = profile_dataset(str(sparse_path), sample_size=8, pairwise_sample_size=4) + sparsity = report["metrics"]["sparsity"] + invalid_values = report["metrics"]["invalid_values"]["invalid_values"] + expected = sparsity["cells"] - sparsity["non_zero_values"] - invalid_values + assert sparsity["zero_values"] == expected + assert sparsity["explicit_zero_entries"] == 0 + text = render_html(report) + assert "Zero values%d" % expected in text + assert DEFINITIONS["sparsity"] in text + assert "NaN/Inf values are reported separately" in text + + +def test_dense_zero_count_is_not_rendered_as_unavailable(): + text = _sparsity_section( + { + "representation": "dense", + "total_values": 1, + "zero_values": 0, + "non_zero_values": 1, + } + ) + assert "Zero values0" in text + + +def test_bundled_sparse_example_has_consistent_zero_total(): + output_dir = Path(__file__).resolve().parents[1] / "examples" / "output" + report = json.loads( + (output_dir / "toy_sparse.profile.json").read_text(encoding="utf-8") + ) + sparsity = report["metrics"]["sparsity"] + invalid_values = report["metrics"]["invalid_values"]["invalid_values"] + expected = sparsity["cells"] - sparsity["non_zero_values"] - invalid_values + assert sparsity["zero_values"] == expected == 9966 + html = (output_dir / "toy_sparse.profile.html").read_text(encoding="utf-8") + assert "Zero values9966" in html + diff --git a/tools/dataset_profiler/tests/test_profiler.py b/tools/dataset_profiler/tests/test_profiler.py new file mode 100644 index 0000000000..49dcccbf93 --- /dev/null +++ b/tools/dataset_profiler/tests/test_profiler.py @@ -0,0 +1,1015 @@ +# Copyright 2024-present the vsag project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for the deterministic profiling pipeline.""" + +from __future__ import annotations + +import json +import math + +import numpy as np +import pytest + +import dataset_profiler.profiler as profiler_module +from dataset_profiler.profiler import ( + _DENSE_BLOCK_BYTES, + _DENSE_BLOCK_BYTES_PER_VALUE, + _DENSE_BLOCK_MAX_ROWS, + _DimAccumulator, + _dense_block_rows, + _dense_euclidean, + _mean_pairwise_cosine, + _outliers_metric, + profile_dataset, +) +from dataset_profiler.report import DEFINITIONS +from dataset_profiler.sampling import make_rng, sample_indices +from dataset_profiler.stats import dump_json + +EXPECTED_DENSE_METRICS = set(DEFINITIONS) +REQUIRED_TOP_LEVEL_KEYS = { + "schema_version", + "tool", + "input", + "dataset", + "config", + "metrics", + "warnings", +} + + +def _write_dense(path, matrix, metric="euclidean"): + import h5py + + with h5py.File(str(path), "w") as handle: + handle.attrs["distance"] = metric + handle.create_dataset("train", data=np.asarray(matrix, dtype=np.float32)) + + +def _write_sparse(path, records, metric="ip"): + import h5py + + from conftest import encode_sparse_record + + with h5py.File(str(path), "w") as handle: + handle.attrs["type"] = "sparse" + handle.attrs["distance"] = metric + stream = b"".join( + encode_sparse_record(ids, values) for ids, values in records + ) + handle.create_dataset( + "train", data=np.frombuffer(stream, dtype=np.int8).copy() + ) + + +def test_dense_report_contains_all_documented_metrics(dense_path): + report = profile_dataset(str(dense_path), sample_size=32, pairwise_sample_size=16) + assert set(report) == REQUIRED_TOP_LEVEL_KEYS + assert report["schema_version"] == "1.0" + assert report["tool"]["name"] == "vsag-dataset-profiler" + assert set(report["metrics"]) == EXPECTED_DENSE_METRICS + json.loads(dump_json(report)) # must be JSON serializable + for name, metric in report["metrics"].items(): + assert metric["definition"] == DEFINITIONS[name] + assert metric["scope"] in ("full", "sample", "mixed") + + +def test_dense_full_metrics_match_source_fixture(dense_path, dense_matrix): + report = profile_dataset(str(dense_path), sample_size=200, pairwise_sample_size=200) + metrics = report["metrics"] + valid = np.where(np.isfinite(dense_matrix), dense_matrix, 0.0).astype(np.float64) + norms = np.linalg.norm(valid, axis=1) + assert metrics["norms"]["stats"]["count"] == dense_matrix.shape[0] + assert metrics["norms"]["non_finite_norms"] == 0 + assert math.isclose(metrics["norms"]["stats"]["mean"], float(norms.mean()), rel_tol=1e-9) + + zero_values = int(np.count_nonzero((dense_matrix == 0) & np.isfinite(dense_matrix))) + assert metrics["sparsity"]["total_values"] == dense_matrix.size + assert metrics["sparsity"]["zero_values"] == zero_values + assert metrics["invalid_values"]["invalid_values"] == 2 + assert metrics["invalid_values"]["vectors_with_invalid"] == 1 + + # rows 0/2 and 1/5 are exact duplicates plus the duplicated sparse-style pair + exact = metrics["duplicates"]["exact"] + assert exact["duplicate_vectors"] == 2 + assert exact["unique_vectors"] == dense_matrix.shape[0] - 2 + + +def test_small_dataset_sample_size_truncation_warns(tmp_path): + path = tmp_path / "small_sample_warning.hdf5" + _write_dense(path, [[1, 0], [0, 1]]) + report = profile_dataset(str(path), sample_size=8, pairwise_sample_size=2) + assert report["config"]["effective_sample_size"] == 2 + assert ( + "sample_size reduced from 8 to 2 to stay within the dataset size" + in report["warnings"] + ) + assert not any( + "pairwise_sample_size reduced" in warning for warning in report["warnings"] + ) + + +def test_sampled_metrics_expose_invalid_zero_replacement_semantics(tmp_path): + path = tmp_path / "invalid_sample_warning.hdf5" + _write_dense(path, [[np.nan, np.inf], [0, 0], [3, 4]]) + report = profile_dataset(str(path), sample_size=3, pairwise_sample_size=3) + assert any( + "sampled vectors contain NaN/Inf entries" in warning + and "replace them with 0.0" in warning + for warning in report["warnings"] + ) + pairwise = report["metrics"]["pairwise_distances"]["stats"] + assert pairwise["min"] == 0.0 + assert pairwise["max"] == 5.0 + nearest = report["metrics"]["nearest_neighbor_distances"]["stats"] + assert nearest["min"] == 0.0 + assert nearest["max"] == 5.0 + assert report["metrics"]["invalid_values"]["invalid_values"] == 2 + + +def test_pairwise_matrix_is_bounded_by_sample_size(dense_path): + report = profile_dataset(str(dense_path), sample_size=20, pairwise_sample_size=8) + pairwise = report["metrics"]["pairwise_distances"] + assert pairwise["pair_count"] == 8 * 7 // 2 + assert pairwise["sample_size"] == 8 + assert pairwise["matrix_is_bounded"] is True + assert report["config"]["effective_sample_size"] == 20 + assert report["config"]["effective_pairwise_sample_size"] == 8 + + +def test_two_runs_with_same_seed_are_identical(dense_path): + first = profile_dataset(str(dense_path), seed=7, sample_size=24, pairwise_sample_size=12) + second = profile_dataset(str(dense_path), seed=7, sample_size=24, pairwise_sample_size=12) + assert dump_json(first) == dump_json(second) + + +def test_different_seed_changes_sampling_metadata(dense_path): + kwargs = dict(sample_size=20, pairwise_sample_size=20) + first = profile_dataset(str(dense_path), seed=1, **kwargs) + second = profile_dataset(str(dense_path), seed=2, **kwargs) + assert first["metrics"]["pairwise_distances"]["stats"] != second["metrics"]["pairwise_distances"]["stats"] + + +def test_metric_alias_and_file_distance_attribute(dense_path): + # the fixture stores distance="l2" which must resolve to euclidean + report = profile_dataset(str(dense_path), sample_size=8, pairwise_sample_size=8) + assert report["config"]["metric"] == "euclidean" + report = profile_dataset(str(dense_path), metric="cosine", sample_size=8, pairwise_sample_size=8) + assert report["config"]["metric"] == "angular" + + +def test_unsupported_metric_rejected(dense_path): + with pytest.raises(ValueError, match="unsupported distance metric"): + profile_dataset(str(dense_path), metric="hamming") + + +def test_invalid_config_rejected(dense_path): + with pytest.raises(ValueError): + profile_dataset(str(dense_path), sample_size=0) + with pytest.raises(ValueError): + profile_dataset(str(dense_path), pairwise_sample_size=-3) + with pytest.raises(ValueError): + profile_dataset(str(dense_path), clusters=1001) + + +def test_anisotropy_and_clustering_on_dense(dense_path): + report = profile_dataset( + str(dense_path), sample_size=32, pairwise_sample_size=16, clusters=4, seed=5 + ) + aniso = report["metrics"]["anisotropy"] + assert aniso["available"] is True + assert 0.0 <= aniso["top1_eigenvalue_share"] <= 1.0 + assert aniso["effective_rank"] > 1.0 + clustering = report["metrics"]["clustering"] + assert clustering["available"] is True + assert clustering["clustering_available"] is True + assert sum(clustering["cluster_sizes"]) == 32 + assert len(clustering["labels"]) == 32 + assert len(clustering["projection_points"]) == 32 + + +def test_clustering_disabled_by_default(dense_path): + report = profile_dataset(str(dense_path), sample_size=16, pairwise_sample_size=8) + clustering = report["metrics"]["clustering"] + assert clustering["available"] is True + assert clustering["clustering_available"] is False + + +def test_outliers_flag_huge_norm_row(dense_path): + report = profile_dataset(str(dense_path), sample_size=8, pairwise_sample_size=8) + outliers = report["metrics"]["outliers"] + assert outliers["outlier_count"] >= 1 + assert outliers["method"].startswith("modified z-score") + + +def test_sparse_report_metrics_and_determinism(sparse_path, sparse_records): + first = profile_dataset(str(sparse_path), sample_size=16, pairwise_sample_size=8, seed=3) + second = profile_dataset(str(sparse_path), sample_size=16, pairwise_sample_size=8, seed=3) + assert dump_json(first) == dump_json(second) + + metrics = first["metrics"] + assert set(metrics) == EXPECTED_DENSE_METRICS + assert first["dataset"]["vector_type"] == "sparse" + assert first["dataset"]["count"] == len(sparse_records) + assert first["dataset"]["dim_inferred"] is True + assert metrics["sparsity"]["representation"] == "sparse" + assert metrics["sparsity"]["density"] is not None + assert metrics["duplicates"]["exact"]["duplicate_vectors"] >= 1 + assert metrics["pairwise_distances"]["pair_count"] == 28 + + +def test_sparse_without_offsets_matches_offsets_stream( + sparse_path, sparse_no_offsets_path +): + kwargs = dict(seed=11, sample_size=16, pairwise_sample_size=8) + with_offsets = profile_dataset(str(sparse_path), **kwargs) + without = profile_dataset(str(sparse_no_offsets_path), **kwargs) + # full-dataset statistics must agree between the two encodings + for key in ("norms", "sparsity", "dimensions", "duplicates", "invalid_values"): + assert with_offsets["metrics"][key] == without["metrics"][key], key + assert any("no '/train_offsets'" in w for w in without["warnings"]) + + +def test_sparse_non_ip_metric_warns(sparse_path): + report = profile_dataset( + str(sparse_path), metric="euclidean", sample_size=8, pairwise_sample_size=8 + ) + assert report["config"]["metric"] == "euclidean" + assert any("sparse VSAG evaluation supports only" in w for w in report["warnings"]) + + +def test_no_distance_attribute_falls_back_with_warning(tmp_path, dense_matrix): + import h5py + + path = tmp_path / "no_distance.hdf5" + with h5py.File(str(path), "w") as f: + f.create_dataset("train", data=dense_matrix) + report = profile_dataset(str(path), sample_size=8, pairwise_sample_size=8) + assert report["config"]["metric"] == "euclidean" + assert any("no distance metric was provided" in w for w in report["warnings"]) + + +def test_empty_sparse_dataset_rejected(tmp_path): + import h5py + + path = tmp_path / "empty_sparse.hdf5" + with h5py.File(str(path), "w") as f: + f.attrs["type"] = "sparse" + f.create_dataset("train", data=np.zeros(0, dtype=np.int8)) + with pytest.raises(Exception, match="empty"): + profile_dataset(str(path)) + + +def test_missing_file_rejected(tmp_path): + with pytest.raises(Exception, match="does not exist"): + profile_dataset(str(tmp_path / "nope.hdf5")) + + +def test_split_selection(dense_path): + report = profile_dataset(str(dense_path), split="test", sample_size=4, pairwise_sample_size=4) + assert report["dataset"]["count"] == 6 + + +def test_nearest_neighbor_uses_distances_not_coordinates(tmp_path): + path = tmp_path / "nearest.hdf5" + _write_dense(path, [[0, 0], [3, 4], [6, 8]]) + report = profile_dataset( + str(path), metric="euclidean", sample_size=3, pairwise_sample_size=3 + ) + stats = report["metrics"]["nearest_neighbor_distances"]["stats"] + assert stats["count"] == 3 + assert stats["min"] == 5.0 + assert stats["max"] == 5.0 + assert stats["mean"] == 5.0 + + +@pytest.mark.parametrize("vector_type", ["dense", "sparse"]) +def test_pairwise_subset_is_seeded_random_subset(tmp_path, vector_type): + seed = 17 + rng = make_rng(seed) + expected_sample = sample_indices(10, 6, rng) + expected_positions = np.sort(rng.permutation(6)[:2]) + assert expected_positions.tolist() != [0, 1] + expected_rows = expected_sample[expected_positions] + expected_distance = float(abs(2 ** expected_rows[0] - 2 ** expected_rows[1])) + + path = tmp_path / (vector_type + ".hdf5") + values = (2.0 ** np.arange(10)).astype(np.float32) + if vector_type == "dense": + _write_dense(path, values.reshape(-1, 1)) + else: + records = [ + (np.array([0], dtype=np.uint32), np.array([value], dtype=np.float32)) + for value in values + ] + _write_sparse(path, records, metric="euclidean") + report = profile_dataset( + str(path), + metric="euclidean", + seed=seed, + sample_size=6, + pairwise_sample_size=2, + ) + stats = report["metrics"]["pairwise_distances"]["stats"] + assert stats["count"] == 1 + assert stats["min"] == expected_distance + + +def test_sparse_dimensions_include_implicit_zeros(tmp_path): + path = tmp_path / "implicit_zeros.hdf5" + records = [ + (np.array([0], dtype=np.uint32), np.array([2], dtype=np.float32)), + (np.array([1], dtype=np.uint32), np.array([4], dtype=np.float32)), + ] + _write_sparse(path, records) + report = profile_dataset(str(path), sample_size=2, pairwise_sample_size=2) + per_dimension = report["metrics"]["dimensions"]["per_dimension"] + assert per_dimension["mean"]["values"] == [1.0, 2.0] + assert per_dimension["std"]["values"] == [1.0, 2.0] + assert per_dimension["min"]["values"] == [0.0, 0.0] + assert per_dimension["max"]["values"] == [2.0, 4.0] + assert per_dimension["zero_ratio"]["values"] == [0.5, 0.5] + + +def test_sparse_huge_feature_id_skips_only_dimension_arrays(tmp_path): + path = tmp_path / "huge_feature_id.hdf5" + records = [ + (np.array([0], dtype=np.uint32), np.array([1], dtype=np.float32)), + (np.array([2**31], dtype=np.uint32), np.array([2], dtype=np.float32)), + ] + _write_sparse(path, records) + report = profile_dataset(str(path), sample_size=2, pairwise_sample_size=2) + assert report["dataset"]["dim"] == 2**31 + 1 + assert report["metrics"]["dimensions"]["available"] is False + assert "exceeds the per-dimension statistics limit" in report["metrics"]["dimensions"]["reason"] + assert report["metrics"]["norms"]["count"] == 2 + assert report["metrics"]["sparsity"]["cells"] == 2 * (2**31 + 1) + assert any("other metrics are still reported" in item for item in report["warnings"]) + + +def test_zero_mad_and_iqr_still_detect_extreme_norms(): + norms = np.ones(100, dtype=np.float64) + norms[-1] = 100.0 + result = _outliers_metric(norms) + assert result["outlier_count"] == 1 + assert result["method"] == "median fallback (MAD and IQR are zero)" + identical = _outliers_metric(np.ones(100, dtype=np.float64)) + assert identical["outlier_count"] == 0 + assert identical["method"].startswith("all norms identical") + + +def test_ip_near_duplicates_use_nonnegative_duplicate_distance(tmp_path): + path = tmp_path / "ip_duplicates.hdf5" + _write_dense(path, [[2, 0], [2, 0], [100, 0]], metric="ip") + report = profile_dataset( + str(path), metric="ip", sample_size=3, pairwise_sample_size=3 + ) + near = report["metrics"]["duplicates"]["near_duplicate"] + assert near["metric"] == "euclidean" + assert near["pair_count"] == 3 + assert near["near_duplicate_pairs"] == 1 + + +def test_dense_euclidean_avoids_catastrophic_cancellation(tmp_path): + path = tmp_path / "large_offset.hdf5" + _write_dense(path, [[1e10, 0], [1e10, 1], [1e10, 2]]) + report = profile_dataset( + str(path), metric="euclidean", sample_size=3, pairwise_sample_size=3 + ) + pairwise = report["metrics"]["pairwise_distances"]["stats"] + assert pairwise["min"] == 1.0 + assert pairwise["max"] == 2.0 + assert pairwise["mean"] == pytest.approx(4.0 / 3.0) + nearest = report["metrics"]["nearest_neighbor_distances"]["stats"] + assert nearest["min"] == 1.0 + assert nearest["max"] == 1.0 + near = report["metrics"]["duplicates"]["near_duplicate"] + assert near["near_duplicate_pairs"] == 0 + + +def test_sparse_euclidean_avoids_catastrophic_cancellation(tmp_path): + path = tmp_path / "sparse_large_offset.hdf5" + base = 2 ** 33 + records = [ + (np.array([0], dtype=np.uint32), np.array([base], dtype=np.float32)), + (np.array([0], dtype=np.uint32), np.array([base + 2048], dtype=np.float32)), + (np.array([0], dtype=np.uint32), np.array([base + 4096], dtype=np.float32)), + ] + _write_sparse(path, records, metric="euclidean") + report = profile_dataset( + str(path), metric="euclidean", sample_size=3, pairwise_sample_size=3 + ) + pairwise = report["metrics"]["pairwise_distances"]["stats"] + assert pairwise["min"] == 2048.0 + assert pairwise["max"] == 4096.0 + assert pairwise["mean"] == pytest.approx((2048.0 + 2048.0 + 4096.0) / 3.0) + + +def test_dense_block_rows_are_bounded_by_working_bytes(): + assert _dense_block_rows(0) == _DENSE_BLOCK_MAX_ROWS + assert _dense_block_rows(8) == _DENSE_BLOCK_MAX_ROWS + wide = _dense_block_rows(4096) + assert wide < _DENSE_BLOCK_MAX_ROWS + # raw + float64 copy + boolean flags + float64 temporaries stay in budget + assert wide * 4096 * _DENSE_BLOCK_BYTES_PER_VALUE <= _DENSE_BLOCK_BYTES + assert _dense_block_rows(10 ** 9) == 1 + + +def test_mean_pairwise_cosine_matches_explicit_pairs(): + matrix = np.array([[1.0, 0.0], [1.0, 0.0], [-1.0, 0.0], [0.0, 0.0]]) + expected = (1.0 + -1.0 + 0.0 + -1.0 + 0.0 + 0.0) / 6.0 + assert _mean_pairwise_cosine(matrix) == pytest.approx(expected) + assert _mean_pairwise_cosine(np.ones((1, 2))) is None + + +def test_sparse_implicit_zeros_still_exclude_invalid_entries(tmp_path): + path = tmp_path / "implicit_zeros_invalid.hdf5" + records = [ + (np.array([0], dtype=np.uint32), np.array([2], dtype=np.float32)), + (np.array([0], dtype=np.uint32), np.array([np.nan], dtype=np.float32)), + (np.array([1], dtype=np.uint32), np.array([4], dtype=np.float32)), + ] + _write_sparse(path, records) + report = profile_dataset(str(path), sample_size=3, pairwise_sample_size=3) + per_dimension = report["metrics"]["dimensions"]["per_dimension"] + assert per_dimension["mean"]["values"][0] == 1.0 + assert per_dimension["std"]["values"][0] == 1.0 + assert per_dimension["min"]["values"][0] == 0.0 + assert per_dimension["max"]["values"][0] == 2.0 + assert per_dimension["zero_ratio"]["values"][0] == 0.5 + assert report["metrics"]["invalid_values"]["invalid_values"] == 1 + + +def test_sparse_accumulator_rejects_id_driven_oversized_allocation(): + accumulator = _DimAccumulator() + with pytest.raises(ValueError, match="exceeds per-dimension statistics limit"): + accumulator._ensure(2**31 + 1) + assert accumulator.count.size == 0 + + +def test_zero_robust_spread_detects_deviations_on_both_sides(): + norms = np.ones(101, dtype=np.float64) + norms[0] = 0.0 + norms[-1] = 1000.0 + result = _outliers_metric(norms) + assert result["outlier_count"] == 2 + assert result["method"] == "median fallback (MAD and IQR are zero)" + + +def test_sparse_ip_near_duplicates_use_euclidean_distance(tmp_path): + path = tmp_path / "sparse_ip_duplicates.hdf5" + records = [ + (np.array([0], dtype=np.uint32), np.array([0.5], dtype=np.float32)), + (np.array([0], dtype=np.uint32), np.array([0.5], dtype=np.float32)), + (np.array([0], dtype=np.uint32), np.array([2], dtype=np.float32)), + ( + np.array([0, 1], dtype=np.uint32), + np.array([2, 100], dtype=np.float32), + ), + ] + _write_sparse(path, records, metric="ip") + report = profile_dataset(str(path), sample_size=4, pairwise_sample_size=4) + near = report["metrics"]["duplicates"]["near_duplicate"] + assert near["metric"] == "euclidean" + assert near["median_pairwise_distance"] > 0 + assert near["near_duplicate_pairs"] == 1 + + +def test_large_constant_norm_histogram_has_representable_edges(tmp_path): + path = tmp_path / "constant_large_norm.hdf5" + value = float(2**60) + _write_dense(path, [[value, 0]]) + report = profile_dataset(str(path), sample_size=1, pairwise_sample_size=1) + histogram = report["metrics"]["norms"]["histogram"] + assert sum(histogram["counts"]) == 1 + assert histogram["bins"][0] < value < histogram["bins"][-1] + + +def test_identical_vectors_keep_full_norm_and_duplicate_counts(tmp_path): + path = tmp_path / "identical_vectors.hdf5" + _write_dense(path, np.ones((32, 3), dtype=np.float32)) + report = profile_dataset(str(path), sample_size=8, pairwise_sample_size=4) + norms = report["metrics"]["norms"] + exact = report["metrics"]["duplicates"]["exact"] + assert norms["count"] == 32 + assert norms["stats"]["min"] == norms["stats"]["max"] == pytest.approx(math.sqrt(3)) + assert sum(norms["histogram"]["counts"]) == 32 + assert exact["unique_vectors"] == 1 + assert exact["duplicate_vectors"] == 31 + assert exact["max_group_size"] == 32 + + +@pytest.mark.parametrize("vector_type", ["dense", "sparse"]) +def test_dimension_variance_is_stable_at_large_offsets(tmp_path, vector_type): + path = tmp_path / ("stable_variance_" + vector_type + ".hdf5") + values = np.full(256, 2**30, dtype=np.float32) + values[-1] += 128 + if vector_type == "dense": + _write_dense(path, values.reshape(-1, 1)) + else: + records = [ + (np.array([0], dtype=np.uint32), np.array([value], dtype=np.float32)) + for value in values + ] + _write_sparse(path, records) + report = profile_dataset(str(path), sample_size=8, pairwise_sample_size=4) + per_dimension = report["metrics"]["dimensions"]["per_dimension"] + assert per_dimension["mean"]["values"] == [2**30 + 0.5] + assert per_dimension["std"]["values"][0] == pytest.approx(math.sqrt(63.75)) + + +def test_dense_euclidean_skips_direct_fallback_for_self_distances(monkeypatch): + real_einsum = np.einsum + row_wise_calls = 0 + + def tracking_einsum(subscripts, *operands, **kwargs): + nonlocal row_wise_calls + if subscripts == "ij,ij->i": + row_wise_calls += 1 + return real_einsum(subscripts, *operands, **kwargs) + + monkeypatch.setattr(np, "einsum", tracking_einsum) + distances = _dense_euclidean(np.eye(4, dtype=np.float64)) + assert row_wise_calls == 1 # only the initial squared-norm reduction + expected = np.full((4, 4), np.sqrt(2.0)) + np.fill_diagonal(expected, 0.0) + np.testing.assert_allclose(distances, expected) + + +def test_sparse_repeated_feature_ids_are_rejected(tmp_path): + path = tmp_path / "repeated_ids.hdf5" + records = [ + ( + np.array([0, 0], dtype=np.uint32), + np.array([1, 2], dtype=np.float32), + ) + ] + _write_sparse(path, records) + with pytest.raises(ValueError, match="record 0 contains repeated feature IDs"): + profile_dataset(str(path), sample_size=1, pairwise_sample_size=1) + + +def test_sparse_nnz_counts_finite_nonzero_values_and_total_zeros(tmp_path): + path = tmp_path / "sparse_nnz.hdf5" + records = [ + ( + np.array([0, 1], dtype=np.uint32), + np.array([2, 0], dtype=np.float32), + ), + (np.array([1], dtype=np.uint32), np.array([np.nan], dtype=np.float32)), + ] + _write_sparse(path, records) + report = profile_dataset(str(path), sample_size=2, pairwise_sample_size=2) + sparsity = report["metrics"]["sparsity"] + assert sparsity["stored_entries"] == 3 + assert sparsity["non_zero_values"] == 1 + assert sparsity["explicit_zero_entries"] == 1 + assert sparsity["zero_values"] == 2 + assert sparsity["zero_ratio"] == 0.5 + assert sparsity["density"] == 0.25 + assert report["metrics"]["invalid_values"]["invalid_values"] == 1 + assert sparsity["nnz_per_vector"]["min"] == 0 + assert sparsity["nnz_per_vector"]["max"] == 1 + + +@pytest.mark.parametrize("vector_type", ["dense", "sparse"]) +def test_angular_identical_vectors_are_near_duplicates(tmp_path, vector_type): + path = tmp_path / ("angular_duplicates_" + vector_type + ".hdf5") + if vector_type == "dense": + _write_dense(path, [[1, 1], [1, 1], [1, 1]], metric="angular") + else: + records = [ + ( + np.array([0, 1], dtype=np.uint32), + np.array([1, 1], dtype=np.float32), + ) + for _ in range(3) + ] + _write_sparse(path, records, metric="angular") + report = profile_dataset( + str(path), metric="angular", sample_size=3, pairwise_sample_size=3 + ) + pairwise = report["metrics"]["pairwise_distances"]["stats"] + near = report["metrics"]["duplicates"]["near_duplicate"] + assert pairwise["min"] == 0.0 + assert pairwise["max"] == 0.0 + assert near["median_pairwise_distance"] == 0.0 + assert near["threshold"] == 0.0 + assert near["near_duplicate_pairs"] == 3 + + +@pytest.mark.parametrize("vector_type", ["dense", "sparse"]) +def test_angular_positive_scalar_multiples_are_near_duplicates(tmp_path, vector_type): + path = tmp_path / ("angular_scalar_multiples_" + vector_type + ".hdf5") + rows = np.array([[1, 1], [3, 3], [3, 3]], dtype=np.float32) + if vector_type == "dense": + _write_dense(path, rows, metric="angular") + else: + records = [ + (np.array([0, 1], dtype=np.uint32), row) for row in rows + ] + _write_sparse(path, records, metric="angular") + + report = profile_dataset( + str(path), metric="angular", sample_size=3, pairwise_sample_size=3 + ) + pairwise = report["metrics"]["pairwise_distances"]["stats"] + near = report["metrics"]["duplicates"]["near_duplicate"] + assert pairwise["min"] == 0.0 + assert pairwise["max"] == 0.0 + assert near["median_pairwise_distance"] == 0.0 + assert near["threshold"] == 0.0 + assert near["near_duplicate_pairs"] == 3 + + +@pytest.mark.parametrize("vector_type", ["dense", "sparse"]) +def test_angular_preserves_small_positive_distances(tmp_path, vector_type): + path = tmp_path / ("angular_small_distance_" + vector_type + ".hdf5") + rows = np.array([[1.0, 0.0], [1.0, 2**-24]], dtype=np.float32) + if vector_type == "dense": + _write_dense(path, rows, metric="angular") + else: + records = [ + (np.array([0, 1], dtype=np.uint32), row) for row in rows + ] + _write_sparse(path, records, metric="angular") + + report = profile_dataset( + str(path), + metric="angular", + sample_size=2, + pairwise_sample_size=2, + near_duplicate_relative_epsilon=0, + ) + expected = 2**-49 + pairwise = report["metrics"]["pairwise_distances"]["stats"] + near = report["metrics"]["duplicates"]["near_duplicate"] + assert pairwise["min"] == pytest.approx(expected) + assert pairwise["max"] == pytest.approx(expected) + assert near["median_pairwise_distance"] == pytest.approx(expected) + assert near["threshold"] == 0.0 + assert near["near_duplicate_pairs"] == 0 + assert near["near_duplicate_ratio"] == 0.0 + + default = profile_dataset( + str(path), metric="angular", sample_size=2, pairwise_sample_size=2 + ) + default_near = default["metrics"]["duplicates"]["near_duplicate"] + assert default_near["threshold"] == pytest.approx(expected * 1e-3) + assert default_near["near_duplicate_pairs"] == 0 + assert default_near["near_duplicate_ratio"] == 0.0 + + +def test_dense_sample_size_is_capped_by_materialized_bytes(tmp_path, monkeypatch): + monkeypatch.setattr(profiler_module, "_SAMPLE_MEMORY_BUDGET_BYTES", 64) + path = tmp_path / "bounded_dense_sample.hdf5" + _write_dense(path, np.arange(24, dtype=np.float32).reshape(6, 4)) + report = profile_dataset(str(path), sample_size=6, pairwise_sample_size=6) + assert report["config"]["effective_sample_size"] == 2 + assert report["config"]["effective_pairwise_sample_size"] == 2 + assert report["metrics"]["pairwise_distances"]["sample_size"] == 2 + assert report["metrics"]["pairwise_distances"]["pair_count"] == 1 + assert ( + "sample_size reduced from 6 to 2 to stay within the dense sample memory budget" + in report["warnings"] + ) + + monkeypatch.setattr(profiler_module, "_SAMPLE_MEMORY_BUDGET_BYTES", 16) + unavailable = profile_dataset(str(path), sample_size=6, pairwise_sample_size=6) + assert unavailable["config"]["effective_sample_size"] == 0 + assert unavailable["config"]["effective_pairwise_sample_size"] == 0 + assert unavailable["metrics"]["pairwise_distances"]["sample_size"] == 0 + assert unavailable["metrics"]["anisotropy"]["available"] is False + + +def test_sparse_sample_bytes_budget_stops_retention_and_expansion( + tmp_path, monkeypatch +): + # 6 sampled rows x 10 entries retain 128 + 10 * 128 = 1408 bytes each; a + # 3000-byte budget stops after the second row instead of eagerly holding + # all rows and then expanding every one of them into Python dicts. + base_ids = np.arange(10, dtype=np.uint32) + records = [ + ( + base_ids + np.uint32(10 * position), + np.linspace(0.1, 1.0, 10, dtype=np.float32), + ) + for position in range(6) + ] + path = tmp_path / "sparse_sample_bytes_budget.hdf5" + _write_sparse(path, records) + + expanded_rows = [] + real_pairwise = profiler_module._pairwise_sparse + + def tracking_pairwise(pair_records, metric): + expanded_rows.append(len(pair_records)) + return real_pairwise(pair_records, metric) + + monkeypatch.setattr(profiler_module, "_pairwise_sparse", tracking_pairwise) + monkeypatch.setattr(profiler_module, "_SAMPLE_MEMORY_BUDGET_BYTES", 3000) + report = profile_dataset(str(path), sample_size=6, pairwise_sample_size=6) + metrics = report["metrics"] + assert report["config"]["effective_sample_size"] == 0 + assert report["config"]["effective_pairwise_sample_size"] == 0 + assert not any( + "dense sample memory budget" in warning for warning in report["warnings"] + ) + + # (a) the over-budget sample is neither fully retained nor expanded. + assert expanded_rows == [] + assert sum( + "sample memory budget" in warning for warning in report["warnings"] + ) == 1 + row_bytes = profiler_module._sparse_sample_row_bytes(base_ids) + assert row_bytes == 1408 + expected_reason = profiler_module._sparse_sample_budget_reason( + row_bytes, 2 * row_bytes, 2 + ) + assert expected_reason in report["warnings"] + + # (b) sampled metrics report unavailable with a budget reason, not garbage. + for name in ( + "pairwise_distances", + "nearest_neighbor_distances", + "anisotropy", + "clustering", + ): + unavailable = metrics[name] + assert unavailable["available"] is False + assert "sample memory budget" in unavailable["reason"] + near = metrics["duplicates"]["near_duplicate"] + assert near["available"] is False + assert "sample memory budget" in near["reason"] + + # (c) streaming and full-split metrics are all still present. + sparsity = metrics["sparsity"] + assert sparsity["nnz_available"] is True + assert sparsity["nnz_reason"] is None + assert sparsity["stored_entries"] == 60 + assert metrics["norms"]["count"] == 6 + assert metrics["dimensions"]["dim"] == 60 + exact = metrics["duplicates"]["exact"] + assert exact["unique_vectors"] == 6 + assert exact["duplicate_vectors"] == 0 + + # With the default budget the same file still profiles normally. + monkeypatch.setattr( + profiler_module, "_SAMPLE_MEMORY_BUDGET_BYTES", 256 * 1024 * 1024 + ) + unbounded = profile_dataset(str(path), sample_size=6, pairwise_sample_size=6) + unbounded_metrics = unbounded["metrics"] + assert unbounded["config"]["effective_sample_size"] == 6 + assert unbounded["config"]["effective_pairwise_sample_size"] == 6 + assert unbounded_metrics["pairwise_distances"]["pair_count"] == 15 + assert unbounded_metrics["anisotropy"]["available"] is True + assert len(expanded_rows) == 2 + assert expanded_rows == [6, 6] + + +def test_dense_nonzero_count_excludes_invalid_entries(tmp_path): + path = tmp_path / "dense_invalid_sparsity.hdf5" + _write_dense(path, [[np.nan, 0], [np.inf, 2]]) + report = profile_dataset(str(path), sample_size=2, pairwise_sample_size=2) + sparsity = report["metrics"]["sparsity"] + assert sparsity["total_values"] == 4 + assert sparsity["zero_values"] == 1 + assert sparsity["non_zero_values"] == 1 + assert sparsity["zero_ratio"] == 0.25 + assert sparsity["density"] == 0.25 + assert sparsity["all_zero_vectors"] == 1 + assert report["metrics"]["invalid_values"]["invalid_values"] == 2 + + +def test_dense_and_sparse_sparsity_exclude_invalid_entries_consistently(tmp_path): + dense_path = tmp_path / "dense_invalid_consistency.hdf5" + sparse_path = tmp_path / "sparse_invalid_consistency.hdf5" + _write_dense(dense_path, [[np.nan, 0], [np.inf, 2]]) + records = [ + ( + np.array([0, 1], dtype=np.uint32), + np.array([np.nan, 0], dtype=np.float32), + ), + ( + np.array([0, 1], dtype=np.uint32), + np.array([np.inf, 2], dtype=np.float32), + ), + ] + _write_sparse(sparse_path, records) + + reports = [ + profile_dataset(str(path), sample_size=2, pairwise_sample_size=2) + for path in (dense_path, sparse_path) + ] + for report in reports: + sparsity = report["metrics"]["sparsity"] + assert sparsity["zero_values"] == 1 + assert sparsity["non_zero_values"] == 1 + assert sparsity["zero_ratio"] == 0.25 + assert sparsity["density"] == 0.25 + assert report["metrics"]["invalid_values"]["invalid_values"] == 2 + + +@pytest.mark.parametrize("vector_type", ["dense", "sparse"]) +def test_singleton_pairwise_sample_provenance(tmp_path, vector_type): + path = tmp_path / ("singleton_" + vector_type + ".hdf5") + if vector_type == "dense": + _write_dense(path, [[3, 4]]) + else: + records = [(np.array([0], dtype=np.uint32), np.array([1], dtype=np.float32))] + _write_sparse(path, records) + report = profile_dataset(str(path), sample_size=1, pairwise_sample_size=1) + assert report["config"]["effective_pairwise_sample_size"] == 1 + pairwise = report["metrics"]["pairwise_distances"] + assert pairwise["sample_size"] == 1 + assert pairwise["pair_count"] == 0 + near = report["metrics"]["duplicates"]["near_duplicate"] + assert near["available"] is False + assert near["sample_size"] == 1 + assert near["pair_count"] == 0 + assert report["metrics"]["nearest_neighbor_distances"]["sample_size"] == 1 + + +def test_pca_uses_covariance_when_it_is_smaller(monkeypatch): + monkeypatch.setattr(profiler_module, "_PCA_SVD_MAX_DIM", 2) + decomposed_shapes = [] + real_eigh = np.linalg.eigh + + def tracking_eigh(matrix): + decomposed_shapes.append(matrix.shape) + return real_eigh(matrix) + + monkeypatch.setattr(np.linalg, "eigh", tracking_eigh) + matrix = np.arange(20, dtype=np.float64).reshape(5, 4) + result = profiler_module._pca_statistics(matrix) + assert result is not None + assert result[1].shape == (5, 2) + assert decomposed_shapes == [(4, 4)] # not the larger 5-by-5 Gram matrix + + covariance_result = result + monkeypatch.setattr(profiler_module, "_PCA_SVD_MAX_DIM", 4096) + svd_result = profiler_module._pca_statistics(matrix) + assert svd_result is not None + np.testing.assert_allclose( + np.sort(covariance_result[0])[::-1], + np.sort(svd_result[0])[::-1], + rtol=1e-9, + atol=1e-12, + ) + pair_matrix = np.zeros((matrix.shape[0], matrix.shape[0]), dtype=np.float64) + covariance_metric = profiler_module._anisotropy_metric( + matrix, pair_matrix, "euclidean", [], covariance_result + ) + svd_metric = profiler_module._anisotropy_metric( + matrix, pair_matrix, "euclidean", [], svd_result + ) + for key in ( + "explained_variance_ratio", + "top1_eigenvalue_share", + "effective_rank", + ): + np.testing.assert_allclose( + covariance_metric[key], svd_metric[key], rtol=1e-9, atol=1e-12 + ) + + +def test_sampled_metrics_reuse_one_pca_decomposition(tmp_path, monkeypatch): + path = tmp_path / "shared_pca.hdf5" + matrix = np.arange(36, dtype=np.float32).reshape(12, 3) + _write_dense(path, matrix) + real_pca_statistics = profiler_module._pca_statistics + pca_calls = 0 + + def tracking_pca_statistics(sample_matrix): + nonlocal pca_calls + pca_calls += 1 + return real_pca_statistics(sample_matrix) + + monkeypatch.setattr(profiler_module, "_pca_statistics", tracking_pca_statistics) + report = profile_dataset( + str(path), sample_size=12, pairwise_sample_size=12, clusters=0 + ) + assert report["metrics"]["anisotropy"]["available"] is True + assert pca_calls == 1 + + +@pytest.mark.parametrize("value,warning_expected", [(np.nan, True), (np.inf, True), (3.0, False)]) +def test_sparse_sample_invalid_warning_has_positive_and_negative_cases( + tmp_path, value, warning_expected +): + path = tmp_path / ("sparse_sample_warning_%s.hdf5" % value) + records = [(np.array([0], dtype=np.uint32), np.array([value], dtype=np.float32))] + _write_sparse(path, records) + report = profile_dataset(str(path), sample_size=1, pairwise_sample_size=1) + warning_found = any( + "sampled vectors contain NaN/Inf entries" in warning + for warning in report["warnings"] + ) + assert warning_found is warning_expected + + +@pytest.mark.parametrize("vector_type", ["dense", "sparse"]) +def test_full_split_norm_and_duplicate_state_honors_budget( + tmp_path, monkeypatch, vector_type +): + monkeypatch.setattr(profiler_module, "_FULL_SPLIT_STATE_BUDGET_BYTES", 32) + + def unexpected_digest(payload): + raise AssertionError("over-budget duplicate detection must not retain hashes") + + monkeypatch.setattr(profiler_module, "_digest", unexpected_digest) + path = tmp_path / ("bounded_full_state_" + vector_type + ".hdf5") + if vector_type == "dense": + _write_dense(path, np.arange(10, dtype=np.float32).reshape(5, 2)) + else: + records = [ + (np.array([0], dtype=np.uint32), np.array([index + 1], dtype=np.float32)) + for index in range(5) + ] + _write_sparse(path, records) + + real_empty = np.empty + + def reject_full_split_allocations(shape, *args, **kwargs): + if shape in (5, (5,), (5, 16)): + raise AssertionError("over-budget full-split state must not be allocated") + return real_empty(shape, *args, **kwargs) + + monkeypatch.setattr(profiler_module.np, "empty", reject_full_split_allocations) + report = profile_dataset(str(path), sample_size=2, pairwise_sample_size=2) + metrics = report["metrics"] + for name in ("norms", "outliers"): + assert metrics[name]["available"] is False + assert "40 bytes for 5 vectors" in metrics[name]["reason"] + assert "32-byte full-split state budget" in metrics[name]["reason"] + exact = metrics["duplicates"]["exact"] + assert exact["available"] is False + assert "320 bytes for 5 vectors (64 bytes per vector)" in exact["reason"] + assert exact["unique_vectors"] is None + assert exact["duplicate_ratio"] is None + assert metrics["duplicates"]["near_duplicate"]["available"] is True + assert metrics["sparsity"]["count"] == 5 + if vector_type == "sparse": + assert metrics["sparsity"]["non_zero_values"] == 5 + assert metrics["sparsity"]["nnz_available"] is False + assert metrics["sparsity"]["nnz_per_vector"] is None + assert metrics["sparsity"]["nnz_histogram"] is None + assert "40 bytes for 5 vectors" in metrics["sparsity"]["nnz_reason"] + expected_warnings = 3 if vector_type == "sparse" else 2 + assert ( + sum("full-split state budget" in warning for warning in report["warnings"]) + == expected_warnings + ) + + +def test_unsigned_dense_statistics_match_evaluator_int8_values(tmp_path): + import h5py + + matrix = np.array([[0, 127], [128, 255]], dtype=np.uint8) + path = tmp_path / "unsigned_dense_statistics.hdf5" + with h5py.File(str(path), "w") as handle: + handle.attrs["distance"] = "euclidean" + handle.create_dataset("train", data=matrix) + + report = profile_dataset(str(path), sample_size=2, pairwise_sample_size=2) + with h5py.File(str(path), "r") as handle: + expected = handle["train"].astype(np.int8)[:].astype(np.float64) + assert report["dataset"]["dtype"] == "INT8" + np.testing.assert_allclose( + report["metrics"]["norms"]["stats"]["mean"], + np.linalg.norm(expected, axis=1).mean(), + ) + per_dimension = report["metrics"]["dimensions"]["per_dimension"] + assert per_dimension["min"]["values"] == [0.0, 127.0] + assert per_dimension["max"]["values"] == [127.0, 127.0] + assert per_dimension["mean"]["values"] == [63.5, 127.0] + expected_distance = np.linalg.norm(expected[0] - expected[1]) + assert report["metrics"]["pairwise_distances"]["stats"]["min"] == pytest.approx( + expected_distance + ) + + +def test_unsigned_dense_saturated_values_are_evaluator_duplicates(tmp_path): + import h5py + + path = tmp_path / "unsigned_dense_duplicates.hdf5" + with h5py.File(str(path), "w") as handle: + handle.create_dataset("train", data=np.array([[127], [255]], dtype=np.uint8)) + report = profile_dataset(str(path), sample_size=2, pairwise_sample_size=2) + assert report["metrics"]["pairwise_distances"]["stats"]["min"] == 0.0 + exact = report["metrics"]["duplicates"]["exact"] + assert exact["unique_vectors"] == 1 + assert exact["duplicate_vectors"] == 1 + diff --git a/tools/dataset_profiler/tests/test_sampling.py b/tools/dataset_profiler/tests/test_sampling.py new file mode 100644 index 0000000000..205bfaa8de --- /dev/null +++ b/tools/dataset_profiler/tests/test_sampling.py @@ -0,0 +1,74 @@ +# Copyright 2024-present the vsag project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for deterministic sampling helpers.""" + +from __future__ import annotations + +import numpy as np +import pytest + +from dataset_profiler.sampling import make_rng, sample_indices, take_prefix_sample + + +def test_sample_indices_is_deterministic_for_a_seed(): + first = sample_indices(1000, 37, make_rng(42)) + second = sample_indices(1000, 37, make_rng(42)) + assert np.array_equal(first, second) + + +def test_sample_indices_changes_with_seed(): + first = sample_indices(10000, 64, make_rng(1)) + second = sample_indices(10000, 64, make_rng(2)) + assert not np.array_equal(first, second) + + +def test_sample_indices_are_distinct_and_sorted(): + sample = sample_indices(500, 50, make_rng(7)) + assert sample.size == 50 + assert len(set(sample.tolist())) == 50 + assert np.all(sample[1:] > sample[:-1]) + assert int(sample.min()) >= 0 + assert int(sample.max()) < 500 + + +def test_sample_indices_returns_everything_when_k_ge_n(): + sample = sample_indices(10, 10, make_rng(3)) + assert sample.tolist() == list(range(10)) + sample = sample_indices(4, 100, make_rng(3)) + assert sample.tolist() == list(range(4)) + + +def test_sample_indices_rejects_negative_arguments(): + with pytest.raises(ValueError): + sample_indices(-1, 5, make_rng(0)) + with pytest.raises(ValueError): + sample_indices(10, -1, make_rng(0)) + + +def test_take_prefix_sample_bounds_and_validates(): + sample = np.arange(20) + assert take_prefix_sample(sample, 5).tolist() == [0, 1, 2, 3, 4] + assert take_prefix_sample(sample, None).tolist() == list(range(20)) + with pytest.raises(ValueError): + take_prefix_sample(sample, 0) + + +def test_make_rng_does_not_touch_global_state(): + np.random.seed(12345) + rng = make_rng(99) + values = rng.integers(0, 1_000_000, size=4) + np.random.seed(12345) + control = np.random.randint(0, 1_000_000, size=4) + assert not np.array_equal(values, control) diff --git a/tools/dataset_profiler/tests/test_stats.py b/tools/dataset_profiler/tests/test_stats.py new file mode 100644 index 0000000000..5d1518f195 --- /dev/null +++ b/tools/dataset_profiler/tests/test_stats.py @@ -0,0 +1,186 @@ +# Copyright 2024-present the vsag project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for the shared numeric helpers.""" + +from __future__ import annotations + +import json +import math + +import numpy as np +import pytest + +from dataset_profiler.stats import ( + as_finite_float, + downsample, + dump_json, + histogram, + jsonable, + metric_keys, + summarize, + top_indices, +) + + +def test_as_finite_float_maps_non_finite_to_none(): + assert as_finite_float(1.23456789) == 1.23456789 + assert as_finite_float(math.nan) is None + assert as_finite_float(math.inf) is None + assert as_finite_float(None) is None + assert as_finite_float("x") is None + assert as_finite_float(np.float64(2.5)) == 2.5 + + +def test_small_finite_values_survive_serialization(): + value = 5e-12 + assert as_finite_float(value) == value + assert json.loads(dump_json({"value": value}))["value"] == value + + +def test_summarize_reports_percentiles_and_skips_non_finite(): + stats = summarize([1.0, 2.0, 3.0, math.nan, math.inf, 4.0]) + assert stats["count"] == 4 + assert stats["min"] == 1.0 + assert stats["max"] == 4.0 + assert stats["p50"] == 2.5 + assert "p99" in stats + + +def test_summarize_empty_input_is_all_none(): + stats = summarize([math.nan, math.inf]) + assert stats["count"] == 0 + assert stats["mean"] is None + assert stats["p50"] is None + + +def test_numpy_stats_do_not_iterate_arrays_element_by_element(): + class NoIterationArray(np.ndarray): + def __iter__(self): + raise AssertionError("ndarray elements were boxed into a Python list") + + values = np.array([2.0, 2.0, 2.0], dtype=np.float64).view(NoIterationArray) + stats = summarize(values) + hist = histogram(values) + assert stats["count"] == 3 + assert stats["mean"] == 2.0 + assert sum(hist["counts"]) == 3 + + +def test_histogram_is_json_safe(): + hist = histogram([0.0, 0.5, 1.0, 2.0, 2.5], bins=5) + assert hist["bin_count"] == 5 + assert len(hist["bins"]) == 6 + assert sum(hist["counts"]) == 5 + json.dumps(hist) # must not raise + empty = histogram([math.nan]) + assert empty == {"bins": [], "counts": [], "bin_count": 0} + + +@pytest.mark.parametrize( + "value", + [ + float(np.finfo(np.float64).max), + 0.9995 * np.finfo(np.float64).max, + -0.9995 * np.finfo(np.float64).max, + float(np.finfo(np.float64).min), + ], +) +def test_constant_histogram_near_float_limit_has_finite_splittable_edges(value): + result = histogram([value, value]) + edges = np.asarray(result["bins"], dtype=np.float64) + assert result["bin_count"] == 20 + assert edges.size == 21 + assert np.all(np.isfinite(edges)) + assert np.all(np.diff(edges) > 0) + assert sum(result["counts"]) == 2 + + +def test_narrow_nonconstant_histogram_probes_edges_before_counting_once(monkeypatch): + # FLOAT32 rows [1, 0] and [1, 2**-25] have these distinct float64 norms. + # This covers NumPy versions that raise for automatic narrow bins as well + # as versions that return duplicate edges. + values = np.array([1.0, 1.0 + 2**-51], dtype=np.float64) + real_histogram = np.histogram + histogram_calls = [] + + def tracking_histogram(*args, **kwargs): + histogram_calls.append((args, kwargs)) + return real_histogram(*args, **kwargs) + + monkeypatch.setattr(np, "histogram", tracking_histogram) + result = histogram(values) + edges = np.asarray(result["bins"], dtype=np.float64) + assert result["bin_count"] < 20 + assert np.all(np.diff(edges) > 0) + assert sum(result["counts"]) == 2 + + assert len(histogram_calls) == 1 + args, kwargs = histogram_calls[0] + np.testing.assert_array_equal(args[0], values) + assert isinstance(kwargs["bins"], np.ndarray) + np.testing.assert_array_equal(kwargs["bins"], edges) + + +def test_downsample_keeps_small_arrays_and_buckets_large_ones(): + small = downsample(np.arange(10, dtype=float), max_points=16) + assert small["downsampled"] is False + assert small["values"] == [float(v) for v in range(10)] + + big = downsample(np.arange(100, dtype=float), max_points=10) + assert big["downsampled"] is True + assert big["source_length"] == 100 + assert big["bucket_size"] == 10 + assert len(big["values"]) == 10 + + +def test_top_indices_ignores_non_finite(): + values = np.array([1.0, math.nan, 5.0, 3.0, math.inf]) + assert top_indices(values, 2) == [2, 3] + assert top_indices(values, 2, largest=False) == [0, 3] + assert top_indices(np.array([math.nan]), 3) == [] + + +def test_jsonable_converts_numpy_types(): + payload = { + "i": np.int64(3), + "f": np.float32(1.5), + "b": np.True_, + "arr": np.arange(3), + "nested": [{"n": np.int32(1)}], + } + converted = jsonable(payload) + text = json.dumps(converted, allow_nan=False) + assert json.loads(text) == {"i": 3, "f": 1.5, "b": True, "arr": [0, 1, 2], "nested": [{"n": 1}]} + + +def test_dump_json_is_deterministic_and_sorted(): + first = dump_json({"b": 1, "a": [2, 1]}) + second = dump_json({"a": [2, 1], "b": 1}) + assert first == second + assert first.endswith("\n") + assert json.loads(first)["a"] == [2, 1] + + +def test_metric_keys_enumerate_leaf_paths(): + report = { + "metrics": { + "norms": {"stats": {"mean": 1.0, "max": None}, "histogram": {"counts": [1, 2]}}, + "duplicates": {"exact": {"groups": [{"size": 2}]}}, + } + } + keys = metric_keys(report) + assert "metrics.norms.stats.mean" in keys + assert "metrics.norms.histogram.counts" in keys + assert not any(key.startswith("metrics.duplicates.exact.groups") for key in keys)