Skip to content

feat(experiments): NMS-20027 painless-vs-drift flow query benchmark#122

Open
indigo423 wants to merge 7 commits into
mainfrom
feat/nms-20027-painless-flow-benchmark
Open

feat(experiments): NMS-20027 painless-vs-drift flow query benchmark#122
indigo423 wants to merge 7 commits into
mainfrom
feat/nms-20027-painless-flow-benchmark

Conversation

@indigo423

@indigo423 indigo423 commented Jul 21, 2026

Copy link
Copy Markdown
Owner

Summary

Adds experiments/nms-20027-painless-flows/ — a self-contained local container experiment answering: does replacing the elasticsearch-drift-plugin proportional_sum aggregation with inline Painless scripts (OpenNMS/opennms#8638 / NMS-20027) regress netflow query performance?

Unlike the other experiments/ directories this is not an Azure-lab Ansible experiment: it runs OpenNMS Core + PostgreSQL + Elasticsearch + nl6 on a single host, and its results are valid for relative A/B claims only.

Design

  • One build, one variable: both variants are the same assembly build of PR #8638 @ 03b6b6dd, differing only in proportionalSumStrategy=plugin|painless (one line in the etc-overlay; variant dirs verified byte-identical otherwise)
  • Identical ES for both variants: 8.18.2 + drift plugin v2.0.7_es-8.18.2 (idle in the painless variant); ES + plugin identity is probed into every run manifest so a fallback swap can't silently contaminate the comparison
  • 40M-flow corpus seeded once via nl6 with send-ledger reconciliation; read-only during trials (doc-count guard aborts on mutation)
  • 6 trials × 2 query shapes (dense whole-window / sparse fine-window — the PR's two Painless code paths), trial 1 discarded as warmup
  • Mandatory correctness diff: the painless path intentionally fixes NMS-20001, so B↔C series/totals deltas are reported next to the latency delta, never merged into it
  • Run manifests are emitted from run-time probes and validated against run-manifest.schema.json (comparability_key included — verified with ajv)

Test plan

  • bash -n on all scripts
  • docker compose config passes for both variants; fails with an instructive error when HORIZON_IMAGE is unset (single source of truth written by bin/build-sut.sh)
  • Manifest shape validated against the schema with ajv-cli --spec=draft2020 (stubbed probes)
  • Variant overlays byte-identical except the proportionalSumStrategy line
  • Execution (build → seed → trials → report) — follows in a separate run; runbook in the experiment README

🤖 Generated with Claude Code

indigo423 added 7 commits July 22, 2026 00:36
@indigo423
feat(experiments): add NMS-20027 painless-vs-drift flow query benchmark
fe6349c 
Self-contained local container experiment (Core + PostgreSQL + ES + nl6)
comparing OpenNMS PR #8638's Painless scripted_metric proportional_sum
against the elasticsearch-drift-plugin as two config-toggle variants of
one pinned build (03b6b6dd). Includes compose stack, ES 8.18.2 image with
drift plugin v2.0.7, variant overlays, nl6 seed scenario (40M flows),
fixed dense/sparse query set, trial runner with corpus doc-count guard,
and run-manifest emitter validated against the benchmark skill schema.

Results are valid for relative A/B claims only (co-located generator).

Assisted-by: ClaudeCode:claude-fable-5
Signed-off-by: Ronny Trommer <ronny@no42.org>
@indigo423
fix(experiments): harden NMS-20027 benchmark against review findings
3c5a07a 
- persist ES/Postgres data in named volumes so the 40M-flow corpus
  survives container recreation; only `down -v` removes it
- run-trials verifies the corpus doc count against the seeded identity
  (corpus-identity.json), closing the 0==0 trivial-pass hole
- run-trials records actual window/trials as trial-params.json;
  emit-manifest embeds it as workload.parameters so mismatched windows
  fail the comparability gate
- failed heap/GC probes abort manifest emission instead of emitting a
  sentinel that compares equal across variants
- config_delta hashes the whole variant overlay dir, not one file
- build-sut refuses tarballs not tied to the pinned SHA via a marker

Assisted-by: ClaudeCode:claude-fable-5
Signed-off-by: Ronny Trommer <ronny@no42.org>
@indigo423
fix(experiments): close gate-integrity gaps from third review round
44aa4af 
- build-sut reuses the fixture only when fixture-identity.json matches
  PINNED_SHA (a re-pin previously reused the old image silently), aborts
  on a dirty source tree, and refuses ambiguous multi-tarball targets
- controlled overlay content moves out of the gate-exempt config_delta
  (the IV) into a new gated sut.overlay_sha; the hash strips the IV
  line so byte-identical controls compare equal across variants, uses
  C collation, and excludes dotfiles
- run-trials reads the query window from corpus-identity.json (single
  source of truth; hand-typed env windows could miss the seeded range),
  validates the file's contract, deletes any stale trial-params.json at
  block start, and fingerprints queries.json into workload.parameters
- README documents the corpus-identity.json schema and the from-scratch
  reset (down -v + rm build/ results/ .env)

Assisted-by: ClaudeCode:claude-fable-5
Signed-off-by: Ronny Trommer <ronny@no42.org>
@indigo423
fix(experiments): skip tests in fixture build
e675e7a 
The core/cli shell-test harness exits 1 on macOS even when all tests
pass, failing compile.pl after two minutes. Unit tests don't change the
assembly artifact, and both variants share this single build.

Assisted-by: ClaudeCode:claude-fable-5
Signed-off-by: Ronny Trommer <ronny@no42.org>
@indigo423
feat(experiments): pin nl6 seed generator to v0.17.0 by digest
d8c00f7 
Latest upstream release (2026-07-20), multi-arch (amd64+arm64),
verified pullable on GHCR. Replaces the fail-on-purpose placeholder.

Assisted-by: ClaudeCode:claude-fable-5
Signed-off-by: Ronny Trommer <ronny@no42.org>
@indigo423
fix(experiments): select the core assembly tarball deterministically
b6efa5d 
The assembly emits core + optional tarballs; opennms-container/core
consumes the core one. The ambiguity guard now keys on *-core.tar.gz
instead of aborting on the standard two-tarball output.

Assisted-by: ClaudeCode:claude-fable-5
Signed-off-by: Ronny Trommer <ronny@no42.org>
@indigo423
fix(experiments): build the image via upstream 'make image' recipe
e008554 
The core Dockerfile expects the tarball pre-unpacked into tarball-root/
with build args supplied — opennms-container/core's Makefile owns that
(sanity check, unpack, buildx, tags). Retag its opennms/horizon:<pom
version> output to the pinned fixture tag instead of calling docker
build directly.

Assisted-by: ClaudeCode:claude-fable-5
Signed-off-by: Ronny Trommer <ronny@no42.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@indigo423