Skip to content

Add default Prometheus EMF metric_declaration when target allocator is enabled#319

Closed
musa-asad wants to merge 1 commit into
aws-observability:mainfrom
musa-asad:fix/prometheus-emf-metric-declarations
Closed

Add default Prometheus EMF metric_declaration when target allocator is enabled#319
musa-asad wants to merge 1 commit into
aws-observability:mainfrom
musa-asad:fix/prometheus-emf-metric-declarations

Conversation

@musa-asad

@musa-asad musa-asad commented Jun 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

When the Target Allocator is enabled (agent.prometheus.targetAllocator.enabled=true), the chart renders .spec.prometheus with only scrape_configs and no EMF metric-extraction block. The agent scrapes targets but produces 0 CloudWatch metrics — without metric_declaration, there's no instruction telling the agent which series to export.

This change injects a default emf_processor.metric_declaration block gated behind the TA toggle. Default behavior for non-TA users is unchanged.

Changes

  • templates/_helpers.tpl — render prometheus.emf_processor.metric_declaration when TA enabled and emfProcessor declarations are present
  • values.yaml — add default agent.prometheus.emfProcessor with metricNamespace + metricDeclarations for node-exporter and nginx

Test Output

helm lint:

$ helm lint charts/amazon-cloudwatch-observability --set clusterName=test-cluster --set region=us-west-2
==> Linting charts/amazon-cloudwatch-observability
[INFO] Chart.yaml: icon is recommended

1 chart(s) linted, 0 chart(s) failed

Gating proof (EMF block renders only when TA enabled):

$ helm template ... --set agent.prometheus.targetAllocator.enabled=true | grep -c metric_declaration
1
$ helm template ... --set agent.prometheus.targetAllocator.enabled=false | grep -c metric_declaration
0

CloudWatch metrics before/after:

ContainerInsights/Prometheus metrics (Helm path):   0 → 7
Same default on add-on install path:                0 → 7

OTEL CI flag matrix (no regression):

$ bash charts/amazon-cloudwatch-observability/tests/flag_matrix.sh
[State #1] enabled=false logs=false containerLogs=false  PASS
[State #2] enabled=false logs=false containerLogs=true   PASS
[State #3] enabled=false logs=true  containerLogs=false  PASS
[State #4] enabled=false logs=true  containerLogs=true   PASS
[State #5] enabled=true  logs=false containerLogs=false  PASS
[State #6] enabled=true  logs=false containerLogs=true   PASS
[State #7] enabled=true  logs=true  containerLogs=false  PASS
[State #8] enabled=true  logs=true  containerLogs=true   PASS
All 8 cases passed.

Related

@musa-asad
musa-asad requested review from okankoAMZ and sky333999 June 16, 2026 18:38
@musa-asad musa-asad self-assigned this Jun 16, 2026
@musa-asad

Copy link
Copy Markdown
Collaborator Author

Closing — this adds an opinionated default metric_declaration rather than fixing a bug. Users are expected to provide their own metric_declaration in configuration-values when enabling Prometheus scraping, as documented in the EKS add-on usage guide.

@musa-asad musa-asad closed this Jun 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant