diff --git a/experiments/nms-20027-painless-flows/.gitignore b/experiments/nms-20027-painless-flows/.gitignore new file mode 100644 index 0000000..88a0d5a --- /dev/null +++ b/experiments/nms-20027-painless-flows/.gitignore @@ -0,0 +1,3 @@ +build/ +results/ +.env diff --git a/experiments/nms-20027-painless-flows/README.md b/experiments/nms-20027-painless-flows/README.md new file mode 100644 index 0000000..b9f241c --- /dev/null +++ b/experiments/nms-20027-painless-flows/README.md @@ -0,0 +1,105 @@ +# NMS-20027: Painless proportional_sum vs. elasticsearch-drift-plugin + +A/B benchmark of [OpenNMS PR #8638](https://github.com/OpenNMS/opennms/pull/8638) (NMS-20027): +does computing flow `proportional_sum` with inline Painless `scripted_metric` scripts regress +netflow query performance vs. the [elasticsearch-drift-plugin](https://github.com/OpenNMS-Plugins/elasticsearch-drift-plugin)? + +Unlike the other directories under `experiments/`, this is **not** an Azure-lab Ansible +experiment — it is a local single-host container experiment (OpenNMS Core + PostgreSQL + +Elasticsearch + nl6 only; no Minion/Sentinel/Kafka). The full Stage-0 plan, design and task +list live in `openspec/changes/nms-20027-painless-flow-benchmark/`. + +## Design in one paragraph + +One build (OpenNMS/opennms @ `03b6b6dd8cfb3d3b90094530d7b2d10439fe0a48`), two variants that +differ in exactly one config line — `proportionalSumStrategy` in +`etc/org.opennms.features.flows.persistence.elastic.cfg`: **B** = `plugin` (drift plugin path), +**C** = `painless` (PR default). Elasticsearch 8.18.2 carries drift plugin `v2.0.7_es-8.18.2` +in *both* variants (idle in C) so the cluster is identical. A 40M-flow corpus is seeded once +via nl6 (send-ledger reconciled) and is read-only during trials. 6 trials × 2 query shapes +(dense whole-window / sparse fine-window) per variant; trial 1 discarded. The Painless path +intentionally fixes NMS-20001, so a B↔C correctness diff of series/totals is mandatory and is +reported next to the latency delta. + +**Scope: relative A/B only.** Generator and SUT are co-located — no absolute flows/sec or QPS +figure from this experiment is valid. + +## Layout + +| Path | Purpose | +|---|---| +| `compose.yml` | The 4-service stack (database, horizon, elasticsearch, nl6) | +| `es/Dockerfile` | ES 8.18.2 + drift plugin `v2.0.7_es-8.18.2` | +| `variants/variant-b-plugin/` | etc-overlay for variant B (`proportionalSumStrategy=plugin`) | +| `variants/variant-c-painless/` | etc-overlay for variant C (`proportionalSumStrategy=painless`) | +| `scenarios/flow-seed.json` | nl6 IPFIX seed scenario: 5000 flows/s × 8000 s = 40M flows | +| `queries/queries.json` | The fixed query set (dense + sparse shapes) | +| `bin/build-sut.sh` | Tasks 1.1–1.4: pinned checkout → assembly tarball → image (skip-rebuild aware) | +| `bin/run-trials.sh` | Task 4.2: trial runner with doc-count guard and raw-response persistence | +| `bin/emit-manifest.sh` | Task 4.6: run-time manifest probes + schema validation | +| `run-manifest.schema.json` | Contract schema (copied from the opennms-benchmark skill) | +| `build/`, `results/` | Generated at run time — gitignored | + +The two variant directories MUST stay byte-identical except the `proportionalSumStrategy` +line. Any additional config (e.g. enabling the telemetryd IPFIX listener) is applied to +*both* directories identically — it is a control and part of `config_delta`. + +## Runbook (maps to `tasks.md`) + +```sh +# 1. Build the shared fixture (~1–1.5 h once; re-runs skip) [tasks 1.1–1.4] +./bin/build-sut.sh + +# 2. Bring up variant B; verify plugin + smoke query [tasks 2.1–2.5] +# (scoped service list: nl6's image is unpinned until step 3, and an +# unpullable image aborts an unscoped `up` entirely) +VARIANT=variant-b-plugin docker compose up -d database elasticsearch horizon +curl -s localhost:9200/_cat/plugins # drift plugin listed +curl -su admin:admin localhost:8980/opennms/rest/info # version matches the SHA build + +# 3. Seed the corpus once (~2.2 h at 5k flows/s); reconcile ledger [tasks 3.1–3.4] +# Pin the nl6 image in compose.yml first, then: docker compose up -d nl6 +# Load scenarios/flow-seed.json into nl6 (REST API / UI — verify flags at nl6.eu). +# Then write build/corpus-identity.json with EXACTLY these keys (the scripts +# read them; the query window comes from here, never from hand-typed env): +# { "doc_count": , +# "window_start_ms": , "window_end_ms": } + +# 4. Trials (window + doc count are read from corpus-identity.json) [tasks 4.1–4.6] +VARIANT=variant-b-plugin ./bin/run-trials.sh +VARIANT=variant-b-plugin ./bin/emit-manifest.sh +VARIANT=variant-c-painless docker compose up -d --force-recreate horizon # flips the one line +VARIANT=variant-c-painless ./bin/run-trials.sh +VARIANT=variant-c-painless ./bin/emit-manifest.sh + +# 5. Stats, correctness diff, report [tasks 5.1–5.5] +# Compute median/p95/min–max over trials 2–6; diff results/variant-b-plugin vs +# results/variant-c-painless; splice runs+manifests JSON into the skill's +# report-template.html (