feat(experiments): NMS-20027 painless-vs-drift flow query benchmark

experiments/nms-20027-painless-flows/.gitignore

@@ -0,0 +1,3 @@
+build/
+results/
+.env

experiments/nms-20027-painless-flows/README.md

@@ -0,0 +1,105 @@
+# NMS-20027: Painless proportional_sum vs. elasticsearch-drift-plugin
+
+A/B benchmark of [OpenNMS PR #8638](https://github.com/OpenNMS/opennms/pull/8638) (NMS-20027):
+does computing flow `proportional_sum` with inline Painless `scripted_metric` scripts regress
+netflow query performance vs. the [elasticsearch-drift-plugin](https://github.com/OpenNMS-Plugins/elasticsearch-drift-plugin)?
+
+Unlike the other directories under `experiments/`, this is **not** an Azure-lab Ansible
+experiment — it is a local single-host container experiment (OpenNMS Core + PostgreSQL +
+Elasticsearch + nl6 only; no Minion/Sentinel/Kafka). The full Stage-0 plan, design and task
+list live in `openspec/changes/nms-20027-painless-flow-benchmark/`.
+
+## Design in one paragraph
+
+One build (OpenNMS/opennms @ `03b6b6dd8cfb3d3b90094530d7b2d10439fe0a48`), two variants that
+differ in exactly one config line — `proportionalSumStrategy` in
+`etc/org.opennms.features.flows.persistence.elastic.cfg`: **B** = `plugin` (drift plugin path),
+**C** = `painless` (PR default). Elasticsearch 8.18.2 carries drift plugin `v2.0.7_es-8.18.2`
+in *both* variants (idle in C) so the cluster is identical. A 40M-flow corpus is seeded once
+via nl6 (send-ledger reconciled) and is read-only during trials. 6 trials × 2 query shapes
+(dense whole-window / sparse fine-window) per variant; trial 1 discarded. The Painless path
+intentionally fixes NMS-20001, so a B↔C correctness diff of series/totals is mandatory and is
+reported next to the latency delta.
+
+**Scope: relative A/B only.** Generator and SUT are co-located — no absolute flows/sec or QPS
+figure from this experiment is valid.
+
+## Layout
+
+| Path | Purpose |
+|---|---|
+| `compose.yml` | The 4-service stack (database, horizon, elasticsearch, nl6) |
+| `es/Dockerfile` | ES 8.18.2 + drift plugin `v2.0.7_es-8.18.2` |
+| `variants/variant-b-plugin/` | etc-overlay for variant B (`proportionalSumStrategy=plugin`) |
+| `variants/variant-c-painless/` | etc-overlay for variant C (`proportionalSumStrategy=painless`) |
+| `scenarios/flow-seed.json` | nl6 IPFIX seed scenario: 5000 flows/s × 8000 s = 40M flows |
+| `queries/queries.json` | The fixed query set (dense + sparse shapes) |
+| `bin/build-sut.sh` | Tasks 1.1–1.4: pinned checkout → assembly tarball → image (skip-rebuild aware) |
+| `bin/run-trials.sh` | Task 4.2: trial runner with doc-count guard and raw-response persistence |
+| `bin/emit-manifest.sh` | Task 4.6: run-time manifest probes + schema validation |
+| `run-manifest.schema.json` | Contract schema (copied from the opennms-benchmark skill) |
+| `build/`, `results/` | Generated at run time — gitignored |
+
+The two variant directories MUST stay byte-identical except the `proportionalSumStrategy`
+line. Any additional config (e.g. enabling the telemetryd IPFIX listener) is applied to
+*both* directories identically — it is a control and part of `config_delta`.
+
+## Runbook (maps to `tasks.md`)
+
+```sh
+# 1. Build the shared fixture (~1–1.5 h once; re-runs skip)          [tasks 1.1–1.4]
+./bin/build-sut.sh
+
+# 2. Bring up variant B; verify plugin + smoke query                 [tasks 2.1–2.5]
+#    (scoped service list: nl6's image is unpinned until step 3, and an
+#     unpullable image aborts an unscoped `up` entirely)
+VARIANT=variant-b-plugin docker compose up -d database elasticsearch horizon
+curl -s localhost:9200/_cat/plugins            # drift plugin listed
+curl -su admin:admin localhost:8980/opennms/rest/info   # version matches the SHA build
+
+# 3. Seed the corpus once (~2.2 h at 5k flows/s); reconcile ledger   [tasks 3.1–3.4]
+#    Pin the nl6 image in compose.yml first, then: docker compose up -d nl6
+#    Load scenarios/flow-seed.json into nl6 (REST API / UI — verify flags at nl6.eu).
+#    Then write build/corpus-identity.json with EXACTLY these keys (the scripts
+#    read them; the query window comes from here, never from hand-typed env):
+#      { "doc_count": <reconciled count>,
+#        "window_start_ms": <T0 epoch ms>, "window_end_ms": <T1 epoch ms> }
+
+# 4. Trials (window + doc count are read from corpus-identity.json)  [tasks 4.1–4.6]
+VARIANT=variant-b-plugin ./bin/run-trials.sh
+VARIANT=variant-b-plugin ./bin/emit-manifest.sh
+VARIANT=variant-c-painless docker compose up -d --force-recreate horizon   # flips the one line
+VARIANT=variant-c-painless ./bin/run-trials.sh
+VARIANT=variant-c-painless ./bin/emit-manifest.sh
+
+# 5. Stats, correctness diff, report                                 [tasks 5.1–5.5]
+#    Compute median/p95/min–max over trials 2–6; diff results/variant-b-plugin vs
+#    results/variant-c-painless; splice runs+manifests JSON into the skill's
+#    report-template.html (<script id="benchmark-data"> is the only edit).
+
+# 6. Teardown                                                        [task 6.1]
+docker compose down -v
+```
+
+**From-scratch reset** (rerun the experiment, or after re-pinning the SHA): the
+named volumes and the host-side identity files have independent lifecycles and
+MUST be cleared together, or the guards report contradictory states
+("already built" vs. "corpus lost"):
+
+```sh
+docker compose down -v && rm -rf build/ results/ .env
+```
+
+## Verify-at-use gates (do not skip)
+
+- **ES compatibility smoke check** (task 2.3): one flow query from the PR build against
+  ES 8.18.2 before seeding. On failure: fall back to ES 7.17.13 + plugin `v2.0.5_es-7.17.13`
+  (edit `es/Dockerfile`), same for both variants.
+- **nl6 image + scenario flags** (task 3.1): pin a real nl6 release in `compose.yml`
+  (`TODO-pin-release` placeholder fails the pull on purpose) and verify the scenario format
+  against nl6 upstream docs; record the image digest as `workload.generator_version`.
+- **Query-shape rendering** (task 4.1): on variant C confirm the dense query renders a
+  `scripted_metric` *with* `nBuckets` and the sparse one *without* (enable Elastic query
+  logging or inspect via ES slow log) before starting timed trials.
+- **Seed rate re-budget** (task 3.2): check the ledger rate 30 min in; below ~5k flows/s,
+  recompute the wall-clock budget before continuing.

experiments/nms-20027-painless-flows/bin/build-sut.sh

@@ -0,0 +1,112 @@
+#!/usr/bin/env bash
+# Copyright 2026 Ronny Trommer <ronny@no42.org>
+# SPDX-License-Identifier: Apache-2.0
+#
+# Tasks 1.1-1.4: build the shared fixture ONCE — pinned checkout -> assembly
+# tarball -> container image. Re-runs skip completed steps (shared-fixture rule:
+# never rebuild an identical SHA). Identities land in build/fixture-identity.json.
+set -euo pipefail
+
+PINNED_SHA=03b6b6dd8cfb3d3b90094530d7b2d10439fe0a48
+IMAGE_TAG="horizon-bench:${PINNED_SHA:0:8}"
+EXP_DIR="$(cd "$(dirname "$0")/.." && pwd)"
+BUILD_DIR="${BUILD_DIR:-$EXP_DIR/build}"
+SRC_DIR="$BUILD_DIR/opennms"
+IDENTITY="$BUILD_DIR/fixture-identity.json"
+
+mkdir -p "$BUILD_DIR"
+
+if [ -f "$IDENTITY" ]; then
+  IDENTITY_SHA="$(jq -r '.git_sha' "$IDENTITY")"
+  if [ "$IDENTITY_SHA" = "$PINNED_SHA" ]; then
+    echo "HORIZON_IMAGE=$(jq -r '.image' "$IDENTITY")" > "$EXP_DIR/.env"
+    echo "fixture-identity.json matches $PINNED_SHA — fixture already built, reusing (task 1.4):"
+    cat "$IDENTITY"
+    exit 0
+  fi
+  echo "fixture-identity.json is for $IDENTITY_SHA, not $PINNED_SHA — rebuilding"
+  rm -f "$IDENTITY"
+fi
+
+# --- 1.1 pinned checkout + toolchain record ---------------------------------
+if [ ! -d "$SRC_DIR/.git" ]; then
+  git clone https://github.com/OpenNMS/opennms "$SRC_DIR"
+fi
+git -C "$SRC_DIR" fetch origin "$PINNED_SHA"
+git -C "$SRC_DIR" checkout --detach "$PINNED_SHA"
+if [ -n "$(git -C "$SRC_DIR" status --porcelain)" ]; then
+  echo "ABORT: source tree has local modifications — a tarball built from it would" >&2
+  echo "carry uncommitted changes while the manifest certifies $PINNED_SHA." >&2
+  exit 1
+fi
+
+JAVA_VERSION="$(java -version 2>&1 | head -1)"
+MVN_VERSION="$( (cd "$SRC_DIR" && ./mvnw --version 2>/dev/null || mvn --version) | head -1)"
+echo "toolchain: $JAVA_VERSION / $MVN_VERSION"
+
+# --- 1.2 assembly tarball ----------------------------------------------------
+# Exact invocation is defined by the checked-out branch (README/.circleci) —
+# historically ./compile.pl then ./assemble.pl. Verify there before trusting this.
+# A tarball is only reused when the marker ties it to the pinned SHA — a
+# leftover from an interrupted or different-SHA build is discarded, never
+# silently baked into the fixture.
+MARKER="$SRC_DIR/opennms-full-assembly/target/.built-from-sha"
+# The assembly emits a core + an optional tarball; opennms-container/core
+# consumes the CORE one. Anything other than exactly one core tarball aborts.
+pick_tarball() {
+  local list count
+  list="$(ls "$SRC_DIR"/opennms-full-assembly/target/*-core.tar.gz 2>/dev/null || true)"
+  count="$(printf '%s' "$list" | grep -c . || true)"
+  if [ "$count" -gt 1 ]; then
+    echo "ABORT: multiple *-core.tar.gz in opennms-full-assembly/target — the" >&2
+    echo "fixture identity would be ambiguous. Remove all but one:" >&2
+    printf '%s\n' "$list" >&2
+    exit 1
+  fi
+  printf '%s' "$list"
+}
+TARBALL="$(pick_tarball)"
+if [ -n "$TARBALL" ] && [ "$(cat "$MARKER" 2>/dev/null || true)" != "$PINNED_SHA" ]; then
+  echo "discarding stale tarball not tied to $PINNED_SHA: $TARBALL"
+  rm -f "$SRC_DIR"/opennms-full-assembly/target/*.tar.gz "$MARKER"
+  TARBALL=""
+fi
+if [ -z "$TARBALL" ]; then
+  # -DskipTests=true: unit tests don't change the artifact, and the shell-test
+  # harness (core/cli run-tests.sh) exits 1 on macOS even when all tests pass.
+  (cd "$SRC_DIR" && ./compile.pl -DskipTests=true \
+    && ./assemble.pl -DskipTests=true -Dopennms.home=/opt/opennms)
+  TARBALL="$(pick_tarball)"
+  [ -n "$TARBALL" ] || { echo "ABORT: assembly produced no tarball" >&2; exit 1; }
+  echo "$PINNED_SHA" > "$MARKER"
+fi
+TARBALL_SHA256="$(shasum -a 256 "$TARBALL" | awk '{print $1}')"
+echo "tarball: $TARBALL ($TARBALL_SHA256)"
+
+# --- 1.3 container image from the tarball ------------------------------------
+# Upstream recipe: `make image` sanity-checks the tarball, unpacks it into
+# tarball-root/, and drives docker buildx with the branch's build args.
+# It requires the DEFAULT buildx builder to be active and tags the result
+# opennms/horizon:<pom-version>; we retag to our pinned fixture tag.
+UPSTREAM_VERSION="$("$SRC_DIR/.circleci/scripts/pom2version.sh" "$SRC_DIR/pom.xml")"
+make -C "$SRC_DIR/opennms-container/core" image
+docker image tag "opennms/horizon:$UPSTREAM_VERSION" "$IMAGE_TAG"
+IMAGE_DIGEST="$(docker inspect --format '{{.Id}}' "$IMAGE_TAG")"
+
+# --- record identities (feeds every run manifest) -----------------------------
+jq -n \
+  --arg sha "$PINNED_SHA" \
+  --arg tarball "$(basename "$TARBALL")" \
+  --arg tarball_sha256 "$TARBALL_SHA256" \
+  --arg image "$IMAGE_TAG" \
+  --arg image_digest "$IMAGE_DIGEST" \
+  --arg java "$JAVA_VERSION" \
+  --arg maven "$MVN_VERSION" \
+  '{git_sha: $sha, tarball: $tarball, tarball_sha256: $tarball_sha256,
+    image: $image, image_digest: $image_digest,
+    toolchain: {java: $java, maven: $maven}}' > "$IDENTITY"
+
+echo "HORIZON_IMAGE=$IMAGE_TAG" > "$EXP_DIR/.env"
+
+echo "fixture built once — identity:"
+cat "$IDENTITY"

experiments/nms-20027-painless-flows/bin/emit-manifest.sh

@@ -0,0 +1,150 @@
+#!/usr/bin/env bash
+# Copyright 2026 Ronny Trommer <ronny@no42.org>
+# SPDX-License-Identifier: Apache-2.0
+#
+# Task 4.6: emit one run manifest per variant from RUN-TIME probes (never
+# backfilled — a run without a run-time manifest is rerun, not patched), then
+# validate against run-manifest.schema.json. IV: sut.config_delta.
+set -euo pipefail
+
+VARIANT="${VARIANT:?set VARIANT=variant-b-plugin or variant-c-painless}"
+BASE_URL="${BASE_URL:-http://localhost:8980/opennms}"
+ES_URL="${ES_URL:-http://localhost:9200}"
+EXP_DIR="$(cd "$(dirname "$0")/.." && pwd)"
+IDENTITY="$EXP_DIR/build/fixture-identity.json"
+CORPUS="$EXP_DIR/build/corpus-identity.json"       # written after seed reconciliation (task 3.4)
+PARAMS="$EXP_DIR/results/$VARIANT/trial-params.json"  # written by run-trials.sh on a valid block
+OUT="$EXP_DIR/results/$VARIANT/run-manifest.json"
+HORIZON_CTR="$(docker compose -f "$EXP_DIR/compose.yml" ps -q horizon)"
+
+[ -f "$IDENTITY" ] || { echo "missing $IDENTITY — run bin/build-sut.sh first" >&2; exit 1; }
+[ -f "$CORPUS" ]   || { echo "missing $CORPUS — seed + reconcile first (tasks 3.x)" >&2; exit 1; }
+[ -f "$PARAMS" ]   || { echo "missing $PARAMS — run bin/run-trials.sh for $VARIANT first" >&2; exit 1; }
+
+# --- run-time probes (measure.md table) --------------------------------------
+VERSION="$(curl -su admin:admin "$BASE_URL/rest/info" | jq -r '.displayVersion // .version')"
+JVM_CMDLINE="$(docker exec "$HORIZON_CTR" sh -c "ps -o args= -p 1 || cat /proc/1/cmdline | tr '\0' ' '")"
+JVM_VERSION="$(docker exec "$HORIZON_CTR" java -version 2>&1 | head -1)"
+# Heap/GC derived from the RUNNING process, never from config files (measure.md).
+# A failed probe ABORTS: a sentinel value would compare equal across variants
+# and pass the comparability gate on a control it never verified.
+JVM_HEAP="$(grep -oE '\-Xms[^ ]+ \-Xmx[^ ]+' <<<"$JVM_CMDLINE" || true)"
+JVM_GC="$(grep -oE '\-XX:\+Use[A-Za-z0-9]+GC' <<<"$JVM_CMDLINE" | head -1 || true)"
+if [ -z "$JVM_HEAP" ] || [ -z "$JVM_GC" ]; then
+  echo "ABORT: heap/GC probe failed on running cmdline: $JVM_CMDLINE" >&2
+  echo "Fix the probe or the SUT and re-emit — never backfill." >&2
+  exit 1
+fi
+# ES + drift plugin are declared controls with a documented fallback pair —
+# record them so a mid-experiment fallback swap cannot pass the gate unnoticed.
+ES_VERSION="$(curl -sf "$ES_URL/" | jq -r '.version.number')"
+ES_PLUGINS="$(curl -sf "$ES_URL/_cat/plugins?h=component,version" | sort | tr '\n' ';')"
+DB_VERSION="$(docker exec "$(docker compose -f "$EXP_DIR/compose.yml" ps -q database)" \
+  psql -U postgres -tAc 'SELECT version()')"
+TSS_BACKEND="$(docker exec "$HORIZON_CTR" sh -c \
+  "grep -h '^org.opennms.timeseries.strategy' /opt/opennms/etc/opennms.properties* 2>/dev/null | tail -1" \
+  || echo 'org.opennms.timeseries.strategy=rrd (default)')"
+# The IV is ONLY the proportionalSumStrategy line; it goes into config_delta,
+# which the gate exempts as the declared IV. Everything ELSE in the overlay is
+# a control, so its hash lives in sut.overlay_sha — a gated comparability path
+# (an exempted config_delta could never catch a stray control-file difference).
+# The hash strips the IV line, so byte-identical controls give the SAME sha on
+# both variants: dotfiles excluded, C collation, filenames included.
+OVERLAY_DIR="$EXP_DIR/variants/$VARIANT"
+STRATEGY="$(grep -o 'proportionalSumStrategy=.*' \
+  "$OVERLAY_DIR/org.opennms.features.flows.persistence.elastic.cfg")"
+OVERLAY_SHA="$(cd "$OVERLAY_DIR" && LC_ALL=C find . -type f -not -name '.*' | LC_ALL=C sort | \
+  while read -r f; do
+    printf '== %s\n' "$f"
+    grep -v '^proportionalSumStrategy=' "$f" || true
+  done | shasum -a 256 | awk '{print $1}')"
+CONFIG_DELTA="overlay $VARIANT: $STRATEGY"
+
+case "$(uname)" in
+  Darwin) CPU="$(sysctl -n machdep.cpu.brand_string) ($(sysctl -n hw.ncpu) cores)"
+          RAM_GB=$(( $(sysctl -n hw.memsize) / 1073741824 ));;
+  *)      CPU="$(lscpu | awk -F': +' '/Model name/{print $2}') ($(nproc) cores)"
+          RAM_GB=$(( $(awk '/MemTotal/{print $2}' /proc/meminfo) / 1048576 ));;
+esac
+
+NL6_VERSION="$(docker inspect --format '{{index .RepoDigests 0}}' \
+  "$(docker compose -f "$EXP_DIR/compose.yml" ps -q nl6 2>/dev/null || true)" 2>/dev/null \
+  || echo 'nl6 not running — copy generator_version from the seed record')"
+
+jq -n \
+  --arg variant "$VARIANT" \
+  --arg version "$VERSION" \
+  --arg jvm_version "$JVM_VERSION" \
+  --arg jvm_cmdline "$JVM_CMDLINE" \
+  --arg jvm_heap "$JVM_HEAP" \
+  --arg jvm_gc "$JVM_GC" \
+  --arg es_version "$ES_VERSION" \
+  --arg es_plugins "$ES_PLUGINS" \
+  --arg db "$DB_VERSION" \
+  --arg tss "$TSS_BACKEND" \
+  --arg config_delta "$CONFIG_DELTA" \
+  --arg overlay_sha "$OVERLAY_SHA" \
+  --arg cpu "$CPU" --argjson ram "$RAM_GB" --arg os "$(uname -srm)" \
+  --arg nl6 "$NL6_VERSION" \
+  --arg started "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
+  --slurpfile fixture "$IDENTITY" \
+  --slurpfile corpus "$CORPUS" \
+  --slurpfile params "$PARAMS" \
+  '{
+    experiment: {
+      question: "Does replacing the elasticsearch-drift-plugin proportional_sum aggregation with inline Painless scripts (PR #8638 / NMS-20027) regress netflow query performance?",
+      independent_variable: "sut.config_delta",
+      variant: $variant
+    },
+    sut: {
+      version_identity: { version: $version, git_sha: $fixture[0].git_sha,
+                          image_digest: $fixture[0].image_digest,
+                          tarball_sha256: $fixture[0].tarball_sha256 },
+      jvm: { version: $jvm_version,
+             heap: $jvm_heap, gc: $jvm_gc, flags: $jvm_cmdline },
+      elasticsearch: { version: $es_version, plugins: $es_plugins },
+      db_version: $db,
+      tss_backend: $tss,
+      config_delta: $config_delta,
+      overlay_sha: $overlay_sha,
+      provisioner: "container-branch"
+    },
+    host: { cpu: $cpu, ram_gb: $ram, disk: "nvme (laptop)", os: $os,
+            generator_colocated: true,
+            hygiene_notes: "laptop scope: governor/turbo not pinned; relative A/B only" },
+    workload: {
+      axis: "rest-ui",
+      generator_scenario: "queries/queries.json (fixed order, trial 1 discarded)",
+      generator_version: "curl (trial runner) / seed: \($nl6)",
+      parameters: $params[0],
+      fixtures: [
+        { name: "sut-image", identity: $fixture[0].image_digest },
+        { name: "assembly-tarball", identity: $fixture[0].tarball_sha256 },
+        { name: "flow-corpus", identity: ($corpus[0] | tostring) }
+      ]
+    },
+    run: { id: "\($variant)-\($started)", started_at: $started },
+    comparability_key: [
+      "sut.version_identity",
+      "sut.jvm.heap",
+      "sut.jvm.gc",
+      "sut.elasticsearch",
+      "sut.db_version",
+      "sut.tss_backend",
+      "sut.config_delta",
+      "sut.overlay_sha",
+      "host.cpu",
+      "host.ram_gb",
+      "host.disk",
+      "host.generator_colocated",
+      "workload.axis",
+      "workload.generator_scenario",
+      "workload.generator_version",
+      "workload.parameters",
+      "workload.fixtures"
+    ]
+  }' > "$OUT"
+
+npx --yes ajv-cli validate --spec=draft2020 \
+  -s "$EXP_DIR/run-manifest.schema.json" -d "$OUT"
+echo "manifest emitted + validated: $OUT"