Skip to content

Add continuous KLL window quantiles with Prometheus and Grafana - #20

Open
zzylol wants to merge 3 commits into
mainfrom
feat/kll-prometheus-grafana
Open

zzylol wants to merge 3 commits into
mainfrom
feat/kll-prometheus-grafana

Conversation

@zzylol

@zzylol zzylol commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Adds a persistent OTAP KLL demo that merges tumbling panes into rolling quantiles and continuously exports p50/p99 to Prometheus, with a provisioned Grafana dashboard.

Before → after

  • Before: the finite demo wrote estimates to console and local files, then exited.
  • After: the finite demo can export its results; the streaming demo runs until Ctrl+C, exports completed-window estimates repeatedly, and displays tumbling and rolling results in Grafana.
  • Windows: retained KLL panes support aligned range queries and configurable rolling lookbacks. Sketches are merged before estimating quantiles; Prometheus stores the resulting gauges.

End-to-end example

From the repository root:

docker compose -f demos/kll-grafana/compose.yaml up -d
cargo build --manifest-path asap-precompute-rs/Cargo.toml \
  --features otap-engine --bin asap-kll-stream-demo

# Run for 30 seconds: 6-second lookback, 1-second panes.
./asap-precompute-rs/target/debug/asap-kll-stream-demo \
  --window-seconds 6 --slide-seconds 1 --run-seconds 30

Open http://localhost:3000/d/asap-kll. Grafana refreshes every five seconds and queries:

request_duration_estimate{quantile=~"0.5|0.99"}
request_duration_rolling_estimate{quantile=~"0.5|0.99"}

Omit --run-seconds for continuous operation. Defaults: 60-second lookback and five-second panes.

Live streaming comparison: tumbling and rolling KLL quantiles

Live screenshot using six-second rolling windows and one-second panes.

Validation

python3 demos/kll-grafana/verify-streaming.py \
  asap-precompute-rs/target/debug/asap-kll-stream-demo

Verified repeated writes, stable series, changing rolling values, and matching Prometheus/Grafana queries. Library, window, runtime, and binary tests passed, along with formatting and Clippy. The finite demo export still returns p50=100 and p99=198; Ctrl+C and export failures were checked.

Limits: pane-aligned ranges, approximate KLL quantiles, in-memory retention, and exactly-once complete upstream panes. See the window guide for configuration and semantics.

@zzylol zzylol changed the title Connect KLL demo estimates to Prometheus and Grafana Add continuous KLL window quantiles with Prometheus and Grafana Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant