Investigate distribution metric reporting issues (#20)#32
Conversation
Task 2026-03-20--5--distribution-metric-investigation: investigating whether telemetry_metrics.distribution reports sensible results (#20). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests confirm three issues with distribution metric export: 1. Min/max fields are never populated in the exported HistogramDataPoint. The OTLP proto has optional min/max fields but the exporter never sets them. Downstream consumers (e.g. Honeycomb) default these to 0. 2. Values are rounded to integers via round(value) before accumulation, causing precision loss for fractional measurements. 3. Bucket assignment is correct, but percentile estimation by downstream consumers is inherently limited by bucket granularity. A value of 668 in the (500, 750] bucket causes all percentiles to be reported as 500. All 9 tests pass and reproduce the scenarios described in the issue. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f4a5fb6f9e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| assert histogram_data_point.min == nil, | ||
| "Expected min to be nil (not tracked), got: #{inspect(histogram_data_point.min)}" | ||
|
|
||
| assert histogram_data_point.max == nil, | ||
| "Expected max to be nil (not tracked), got: #{inspect(histogram_data_point.max)}" |
There was a problem hiding this comment.
Because this file is a normal mix test test, these assertions turn the current bug into the expected contract: any future fix for issue #20 that starts populating HistogramDataPoint.min/max will fail CI even though the exporter is behaving correctly. If this is meant as one-off investigation code, it should stay out of the regular suite or assert the desired post-fix behavior instead.
Useful? React with 👍 / 👎.
| assert histogram_data_point.sum == 669.0, | ||
| "Expected sum to be 669.0 (rounded from 668.7), got: #{histogram_data_point.sum}" |
There was a problem hiding this comment.
This test currently blesses the precision-loss bug by expecting 668.7 to export as 669.0. Once write_metric/5 is corrected to preserve fractional measurements, the new test suite will reject the fix and keep CI pinned to the broken behavior. The same issue repeats in the multi-value rounding test below.
Useful? React with 👍 / 👎.
2 participants
Summary
Investigation into whether
telemetry_metrics.distributionproduces incorrect results, as reported in #20.Confirmed issues:
Min/max never populated: The
HistogramDataPointproto has optionalminandmaxfields, but the exporter never sets them. Downstream consumers (e.g., Honeycomb) default these to 0, making min/max charts useless.Values rounded to integers:
write_metric/5usesround(value)for the ETS counter update, causing precision loss for fractional measurements. This error accumulates across multiple values.Percentile estimation is bucket-limited (expected behavior, not a bug): Since the exporter sends standard OTLP histogram data (bucket_counts + explicit_bounds), downstream consumers can only estimate percentiles from bucket boundaries. A value of 668 in the (500, 750] bucket causes Honeycomb to report all percentiles as 500.
Root causes in code:
metric_store.ex:107—update_sum_op = {3, round(value)}rounds valuesmetric_store.ex:326-334—convert_data/2constructsHistogramDataPointwithout settingminormaxfieldsmetric_store.ex:102-115—write_metric/5only tracks{count, sum}per bucket in ETS, no min/max trackingTest plan
distribution_investigation_test.exsall pass🤖 Generated with Claude Code