Skip to content

fix(telemetry): bound Prometheus distribution buffer (fork pin) - #4663

Closed
robacourt wants to merge 1 commit into
mainfrom
rob/pin-prometheus-core-fork
Closed

fix(telemetry): bound Prometheus distribution buffer (fork pin)#4663
robacourt wants to merge 1 commit into
mainfrom
rob/pin-prometheus-core-fork

Conversation

@robacourt

Copy link
Copy Markdown
Contributor

Summary

Fixes unbounded ETS growth in the Prometheus metrics reporter. When ELECTRIC_PROMETHEUS_PORT is set but the /metrics endpoint is scraped infrequently — or never, as on OpenTelemetry-only deployments — telemetry_metrics_prometheus_core buffers one ETS row per distribution observation and only drains it on scrape. The per-transaction receive_lag distribution dominates, so the prometheus_metrics_dist table grows without bound. A customer hit an 8 GB+ ETS table and eventual OOM this way.

This pins telemetry_metrics_prometheus_core to a fork that bounds the buffer by aggregating automatically — on a size threshold (default 10k buffered samples) and a time fallback (default 60s) — in addition to on scrape, and serializes aggregation in the registry process (also fixing a pre-existing overlapping-scrape race).

Upstream PR: beam-telemetry/telemetry_metrics_prometheus_core#77

Why a git dep is safe here

The telemetry deps (the electric_telemetry path dep and its transitive telemetry_metrics_prometheus_core) are gated behind MIX_TARGET=application in sync-service/mix.exs (telemetry_deps(_) -> [] otherwise). The Hex-published electric package is built with the default target, so this dependency is absent from the published dep tree — the git pin cannot affect mix hex.publish. It only affects the standalone sync-service / Docker / CI build, where mix deps.get fetches git deps fine (git is installed in the builder image). electric-telemetry itself is not published to Hex.

Changes

  • packages/electric-telemetry/mix.exs — pin to the fork branch, with a comment to revert to ~> 1.2 once upstream lands on Hex.
  • packages/electric-telemetry/mix.lock + packages/sync-service/mix.lock — locked to the fork commit (only that one entry changes).
  • Changeset (@core/sync-service patch).

Verified electric-telemetry compiles against the fork. Revert this pin to the Hex release after #77 merges and ships.

🤖 Generated with Claude Code

@codecov

codecov Bot commented Jun 29, 2026

Copy link
Copy Markdown

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
4704 1 4703 66
View the top 2 failed test(s) by shortest run time
Elixir.Electric.ShapeCacheTest::test get_or_create_shape_handle/2 against real db crashes when initial snapshot query fails to return data quickly enough
Stack Traces | 0s run time
30) test get_or_create_shape_handle/2 against real db crashes when initial snapshot query fails to return data quickly enough (Electric.ShapeCacheTest)
     test/electric/shape_cache_test.exs:508
     ** (EXIT from #PID<0.9368.0>) killed
Elixir.Electric.ShapeCacheTest::test start_consumer_for_handle/2 starts a consumer plus dependencies
Stack Traces | 0.0374s run time
1) test start_consumer_for_handle/2 starts a consumer plus dependencies (Electric.ShapeCacheTest)
     test/electric/shape_cache_test.exs:1559
     ** (FunctionClauseError) no function clause matching in Support.TestUtils.wait_until/2

     The following arguments were given to Support.TestUtils.wait_until/2:

         # 1
         1000

         # 2
         #Function<75.111972680/0 in Electric.ShapeCacheTest."test start_consumer_for_handle/2 starts a consumer plus dependencies"/1>

     code: assert wait_until(1000, fn ->
     stacktrace:
       (electric 1.7.4) test/support/test_utils.ex:146: Support.TestUtils.wait_until/2
       test/electric/shape_cache_test.exs:1582: (test)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

When ELECTRIC_PROMETHEUS_PORT is set but the /metrics endpoint is scraped
infrequently or never (e.g. OTel-only deployments), telemetry_metrics_prometheus_core
buffers one ETS row per distribution observation and only drains on scrape, so
the dist table grows without bound (the per-transaction receive_lag metric
dominates). An 8GB+ ETS table and eventual OOM was observed in the field.

Pin telemetry_metrics_prometheus_core to a fork that bounds the buffer by
aggregating automatically on a size threshold (default 10k samples) and a time
fallback (default 60s), in addition to on scrape. Only affects the
MIX_TARGET=application build (the standalone sync-service / Docker image); the
telemetry deps are target-gated out of the Hex package, so this git dep does not
affect publishing of `electric`.

Upstream PR: beam-telemetry/telemetry_metrics_prometheus_core#77
Revert to the Hex release once it lands.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YW7Njz5ZpBDaoGviW1eVR8
@robacourt
robacourt force-pushed the rob/pin-prometheus-core-fork branch from bf688eb to 6987ca4 Compare June 29, 2026 16:54
@robacourt robacourt closed this Jun 30, 2026
@robacourt robacourt reopened this Jun 30, 2026
@robacourt

Copy link
Copy Markdown
Contributor Author

Closed in favour of #4660 . We'll wait for beam-telemetry/telemetry_metrics_prometheus_core#77 to be merged before using it.

@robacourt robacourt closed this Jun 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant