Skip to content

fix: change metrics PK for better memory usage #2545

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kodiakhq merged 7 commits into from
Jul 2, 2026
Merged

fix: change metrics PK for better memory usage #2545

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
093544b
fix: change metrics PK for better memory usage
knudtty Jun 29, 2026
b55c111
fix: change Delta encoding to underlying datatype byte number
knudtty Jun 30, 2026
ff6690f
fix: all MetricName should be LowCardinality
knudtty Jun 30, 2026
39060a7
Merge remote-tracking branch 'origin/main' into aaron/metrics-reduce-…
knudtty Jul 1, 2026
060c03c
fix: edge case for interval range for toStartOf functions in chart
knudtty Jul 1, 2026
a394f9c
add changeset
knudtty Jul 2, 2026
4e5a5f1
Merge branch 'main' into aaron/metrics-reduce-pk-memory
knudtty Jul 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 34 additions & 29 deletions docker/otel-collector/schema/seed/00003_otel_metrics.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,16 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_gauge
`ScopeDroppedAttrCount` UInt32 CODEC(ZSTD(1)),
`ScopeSchemaUrl` String CODEC(ZSTD(1)),
`ServiceName` LowCardinality(String) CODEC(ZSTD(1)),
`MetricName` String CODEC(ZSTD(1)),
`MetricName` LowCardinality(String) CODEC(ZSTD(1)),

@knudtty knudtty Jun 29, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changing to LowCardinality reduces PK memory usage by 2-3x

@iShiBin iShiBin Jun 30, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems to be a good change. I wonder why not apply it to the MetricName to all other metrics tables?

@knudtty knudtty Jun 30, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good callout. That was a mistake.

@iShiBin iShiBin Jun 30, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Glad it was addressed. thanks.

`MetricDescription` String CODEC(ZSTD(1)),
`MetricUnit` String CODEC(ZSTD(1)),
`Attributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
`StartTimeUnix` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
`TimeUnix` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
`StartTimeUnix` DateTime CODEC(Delta, ZSTD(1)),
`TimeUnix` DateTime CODEC(Delta, ZSTD(1)),
`Value` Float64 CODEC(ZSTD(1)),
`Flags` UInt32 CODEC(ZSTD(1)),
`Exemplars.FilteredAttributes` Array(Map(LowCardinality(String), String)) CODEC(ZSTD(1)),
`Exemplars.TimeUnix` Array(DateTime64(9)) CODEC(ZSTD(1)),
`Exemplars.TimeUnix` Array(DateTime) CODEC(ZSTD(1)),
`Exemplars.Value` Array(Float64) CODEC(ZSTD(1)),
`Exemplars.SpanId` Array(String) CODEC(ZSTD(1)),
`Exemplars.TraceId` Array(String) CODEC(ZSTD(1)),
Expand All @@ -27,11 +27,12 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_gauge
INDEX idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_attr_key mapKeys(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1
INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_time_minmax TimeUnix TYPE minmax GRANULARITY 1
Comment on lines -30 to +31

@knudtty knudtty Jun 29, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minmax is an extremely lightweight index and helps efficiently prune granules, even if the PK is not fully qualified for a query to take advantage of TimeUnix

)
ENGINE = MergeTree
PARTITION BY toDate(TimeUnix)
ORDER BY (ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix))
ORDER BY (ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix)

@knudtty knudtty Jun 29, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Time bucketing before the high cardinality Attributes column groups data more efficiently speeding up I/O for queries due to less overall ranges and is very efficient in PK memory

Comment on lines -34 to +35

@knudtty knudtty Jun 29, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hashing the Attributes column massively reduces PK memory

@iShiBin iShiBin Jun 30, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is that? Is it because some attribute key or values are too long?

@knudtty knudtty Jun 30, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, both the length of keys or values as well as the number in a single entry. Additionally, ClickHouse does well with known sized parameters, like the 64 bit hash value

@iShiBin iShiBin Jun 30, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it and thanks.

TTL toDateTime(TimeUnix) + ${TABLES_TTL}
SETTINGS ttl_only_drop_parts = 1;

Expand All @@ -46,16 +47,16 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_sum
`ScopeDroppedAttrCount` UInt32 CODEC(ZSTD(1)),
`ScopeSchemaUrl` String CODEC(ZSTD(1)),
`ServiceName` LowCardinality(String) CODEC(ZSTD(1)),
`MetricName` String CODEC(ZSTD(1)),
`MetricName` LowCardinality(String) CODEC(ZSTD(1)),
`MetricDescription` String CODEC(ZSTD(1)),
`MetricUnit` String CODEC(ZSTD(1)),
`Attributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
`StartTimeUnix` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
`TimeUnix` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
`StartTimeUnix` DateTime CODEC(Delta, ZSTD(1)),
`TimeUnix` DateTime CODEC(Delta, ZSTD(1)),
`Value` Float64 CODEC(ZSTD(1)),
`Flags` UInt32 CODEC(ZSTD(1)),
`Exemplars.FilteredAttributes` Array(Map(LowCardinality(String), String)) CODEC(ZSTD(1)),
`Exemplars.TimeUnix` Array(DateTime64(9)) CODEC(ZSTD(1)),
`Exemplars.TimeUnix` Array(DateTime) CODEC(ZSTD(1)),
`Exemplars.Value` Array(Float64) CODEC(ZSTD(1)),
`Exemplars.SpanId` Array(String) CODEC(ZSTD(1)),
`Exemplars.TraceId` Array(String) CODEC(ZSTD(1)),
Expand All @@ -66,11 +67,12 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_sum
INDEX idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_attr_key mapKeys(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1
INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_time_minmax TimeUnix TYPE minmax GRANULARITY 1
)
ENGINE = MergeTree
PARTITION BY toDate(TimeUnix)
ORDER BY (ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix))
ORDER BY (ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix)
TTL toDateTime(TimeUnix) + ${TABLES_TTL}
SETTINGS ttl_only_drop_parts = 1;
-- +goose StatementEnd
Expand All @@ -86,18 +88,18 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_histogram
`ScopeDroppedAttrCount` UInt32 CODEC(ZSTD(1)),
`ScopeSchemaUrl` String CODEC(ZSTD(1)),
`ServiceName` LowCardinality(String) CODEC(ZSTD(1)),
`MetricName` String CODEC(ZSTD(1)),
`MetricName` LowCardinality(String) CODEC(ZSTD(1)),
`MetricDescription` String CODEC(ZSTD(1)),
`MetricUnit` String CODEC(ZSTD(1)),
`Attributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
`StartTimeUnix` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
`TimeUnix` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
`StartTimeUnix` DateTime CODEC(Delta, ZSTD(1)),
`TimeUnix` DateTime CODEC(Delta, ZSTD(1)),
`Count` UInt64 CODEC(Delta(8), ZSTD(1)),
`Sum` Float64 CODEC(ZSTD(1)),
`BucketCounts` Array(UInt64) CODEC(ZSTD(1)),
`ExplicitBounds` Array(Float64) CODEC(ZSTD(1)),
`Exemplars.FilteredAttributes` Array(Map(LowCardinality(String), String)) CODEC(ZSTD(1)),
`Exemplars.TimeUnix` Array(DateTime64(9)) CODEC(ZSTD(1)),
`Exemplars.TimeUnix` Array(DateTime) CODEC(ZSTD(1)),
`Exemplars.Value` Array(Float64) CODEC(ZSTD(1)),
`Exemplars.SpanId` Array(String) CODEC(ZSTD(1)),
`Exemplars.TraceId` Array(String) CODEC(ZSTD(1)),
Expand All @@ -110,11 +112,12 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_histogram
INDEX idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_attr_key mapKeys(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1
INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_time_minmax TimeUnix TYPE minmax GRANULARITY 1
)
ENGINE = MergeTree
PARTITION BY toDate(TimeUnix)
ORDER BY (ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix))
ORDER BY (ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix)
TTL toDateTime(TimeUnix) + ${TABLES_TTL}
SETTINGS ttl_only_drop_parts = 1;
-- +goose StatementEnd
Expand All @@ -130,12 +133,12 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_exponential_histogram
`ScopeDroppedAttrCount` UInt32 CODEC(ZSTD(1)),
`ScopeSchemaUrl` String CODEC(ZSTD(1)),
`ServiceName` LowCardinality(String) CODEC(ZSTD(1)),
`MetricName` String CODEC(ZSTD(1)),
`MetricName` LowCardinality(String) CODEC(ZSTD(1)),
`MetricDescription` String CODEC(ZSTD(1)),
`MetricUnit` String CODEC(ZSTD(1)),
`Attributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
`StartTimeUnix` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
`TimeUnix` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
`StartTimeUnix` DateTime CODEC(Delta, ZSTD(1)),
`TimeUnix` DateTime CODEC(Delta, ZSTD(1)),
`Count` UInt64 CODEC(Delta(8), ZSTD(1)),
`Sum` Float64 CODEC(ZSTD(1)),
`Scale` Int32 CODEC(ZSTD(1)),
Expand All @@ -145,7 +148,7 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_exponential_histogram
`NegativeOffset` Int32 CODEC(ZSTD(1)),
`NegativeBucketCounts` Array(UInt64) CODEC(ZSTD(1)),
`Exemplars.FilteredAttributes` Array(Map(LowCardinality(String), String)) CODEC(ZSTD(1)),
`Exemplars.TimeUnix` Array(DateTime64(9)) CODEC(ZSTD(1)),
`Exemplars.TimeUnix` Array(DateTime) CODEC(ZSTD(1)),
`Exemplars.Value` Array(Float64) CODEC(ZSTD(1)),
`Exemplars.SpanId` Array(String) CODEC(ZSTD(1)),
`Exemplars.TraceId` Array(String) CODEC(ZSTD(1)),
Expand All @@ -158,11 +161,12 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_exponential_histogram
INDEX idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_attr_key mapKeys(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1
INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_time_minmax TimeUnix TYPE minmax GRANULARITY 1
)
ENGINE = MergeTree
PARTITION BY toDate(TimeUnix)
ORDER BY (ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix))
ORDER BY (ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix)
TTL toDateTime(TimeUnix) + ${TABLES_TTL}
SETTINGS ttl_only_drop_parts = 1;
-- +goose StatementEnd
Expand All @@ -178,12 +182,12 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_summary
`ScopeDroppedAttrCount` UInt32 CODEC(ZSTD(1)),
`ScopeSchemaUrl` String CODEC(ZSTD(1)),
`ServiceName` LowCardinality(String) CODEC(ZSTD(1)),
`MetricName` String CODEC(ZSTD(1)),
`MetricName` LowCardinality(String) CODEC(ZSTD(1)),
`MetricDescription` String CODEC(ZSTD(1)),
`MetricUnit` String CODEC(ZSTD(1)),
`Attributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
`StartTimeUnix` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
`TimeUnix` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
`StartTimeUnix` DateTime CODEC(Delta, ZSTD(1)),
`TimeUnix` DateTime CODEC(Delta, ZSTD(1)),
`Count` UInt64 CODEC(Delta(8), ZSTD(1)),
`Sum` Float64 CODEC(ZSTD(1)),
`ValueAtQuantiles.Quantile` Array(Float64) CODEC(ZSTD(1)),
Expand All @@ -194,11 +198,12 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_summary
INDEX idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_attr_key mapKeys(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1
INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_time_minmax TimeUnix TYPE minmax GRANULARITY 1
)
ENGINE = MergeTree
PARTITION BY toDate(TimeUnix)
ORDER BY (ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix))
ORDER BY (ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix)
TTL toDateTime(TimeUnix) + ${TABLES_TTL}
SETTINGS ttl_only_drop_parts = 1;
-- +goose StatementEnd
Expand Down
11 changes: 11 additions & 0 deletions packages/common-utils/src/__tests__/renderChartConfig.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2095,6 +2095,17 @@ describe('renderChartConfig', () => {
dateRangeEndInclusive: false,
expected: `(toDate(timestamp) >= toDate(fromUnixTimestamp64Milli(${new Date('2025-02-12 03:53:38Z').getTime()})) AND toDate(timestamp) <= toDate(fromUnixTimestamp64Milli(${new Date('2025-02-12 04:08:38Z').getTime()})))AND(timestamp > fromUnixTimestamp64Milli(${new Date('2025-02-12 03:53:38Z').getTime()}) AND timestamp < fromUnixTimestamp64Milli(${new Date('2025-02-12 04:08:38Z').getTime()}))`,
},
{
description:
'wraps includedDataInterval-expanded bound in toStartOf when PK has toStartOfHour(col) — prevents dropping rows whose hour is before the raw expanded start',
timestampValueExpression: 'timestamp, toStartOfHour(timestamp)',
dateRange: [
new Date('2025-02-12 04:00:00Z'),
new Date('2025-02-12 04:20:00Z'),
],
includedDataInterval: '5 minute',
expected: `(timestamp >= toStartOfInterval(fromUnixTimestamp64Milli(${new Date('2025-02-12 04:00:00Z').getTime()}), INTERVAL 5 minute) - INTERVAL 5 minute AND timestamp <= toStartOfInterval(fromUnixTimestamp64Milli(${new Date('2025-02-12 04:20:00Z').getTime()}), INTERVAL 5 minute) + INTERVAL 5 minute)AND(toStartOfHour(timestamp) >= toStartOfHour(toStartOfInterval(fromUnixTimestamp64Milli(${new Date('2025-02-12 04:00:00Z').getTime()}), INTERVAL 5 minute) - INTERVAL 5 minute) AND toStartOfHour(timestamp) <= toStartOfHour(toStartOfInterval(fromUnixTimestamp64Milli(${new Date('2025-02-12 04:20:00Z').getTime()}), INTERVAL 5 minute) + INTERVAL 5 minute))`,
},
];

beforeEach(() => {
Expand Down
20 changes: 12 additions & 8 deletions packages/common-utils/src/core/renderChartConfig.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -896,17 +896,21 @@ export async function timeFilterExpr({
);
}

const startTimeCond = includedDataInterval
const rawStartBound = includedDataInterval
? chSql`toStartOfInterval(fromUnixTimestamp64Milli(${{ Int64: startTime }}), INTERVAL ${includedDataInterval}) - INTERVAL ${includedDataInterval}`
: toStartOf
? chSql`${toStartOf.function}(fromUnixTimestamp64Milli(${{ Int64: startTime }})${toStartOf.formattedRemainingArgs})`
: chSql`fromUnixTimestamp64Milli(${{ Int64: startTime }})`;
: chSql`fromUnixTimestamp64Milli(${{ Int64: startTime }})`;

const endTimeCond = includedDataInterval
const rawEndBound = includedDataInterval
? chSql`toStartOfInterval(fromUnixTimestamp64Milli(${{ Int64: endTime }}), INTERVAL ${includedDataInterval}) + INTERVAL ${includedDataInterval}`
: toStartOf
? chSql`${toStartOf.function}(fromUnixTimestamp64Milli(${{ Int64: endTime }})${toStartOf.formattedRemainingArgs})`
: chSql`fromUnixTimestamp64Milli(${{ Int64: endTime }})`;
: chSql`fromUnixTimestamp64Milli(${{ Int64: endTime }})`;

const startTimeCond = toStartOf
? chSql`${toStartOf.function}(${rawStartBound}${toStartOf.formattedRemainingArgs})`
: rawStartBound;

const endTimeCond = toStartOf
? chSql`${toStartOf.function}(${rawEndBound}${toStartOf.formattedRemainingArgs})`
: rawEndBound;

const isDateType = columnMeta?.type === 'Date' || isToDateExpr;

Expand Down
Loading