Fix Telegraf options removed in 1.40 on Linux and Windows (fieldpass -> fieldinclude, pid_tag -> tag_with) by suyadav1 · Pull Request #1764 · microsoft/Docker-Provider

suyadav1 · 2026-08-17T22:31:31Z

Problem

ci_prod now ships Telegraf 1.40.1 on both platforms — Linux via kubernetes/linux/setup.sh (#1789) and Windows via kubernetes/windows/setup.ps1 (#1785) — while the shipped configs still use option names that 1.40 removed:

W! DeprecationWarning: Option "fieldpass" of plugin "inputs.disk" deprecated since version 1.29.0 and will be removed in 1.40.0: use 'fieldinclude' instead
W! DeprecationWarning: Option "fieldpass" of plugin "inputs.diskio" ...
W! DeprecationWarning: Option "fieldpass" of plugin "inputs.net" ...
W! DeprecationWarning: Option "ignore_protocol_stats" of plugin "inputs.net" deprecated since version 1.37.0 ...
W! DeprecationWarning: Option "fieldpass" of plugin "inputs.prometheus" ... (x2)

These read as cosmetic, but they are not. Telegraf treats an unrecognized option as a fatal config-load error, not a warning:

E! loading config file failed: plugin inputs.disk: line 451:
   configuration specified the fields ["fieldpass"], but they were not used

So the binary upgrade and the option rename have to ship together. This PR carries no version pins — both upgrades already landed in ci_prod; this is the config-side half that makes them safe.

In the packaged 1.40.1 binary, config check still accepts legacy fieldpass while emitting DeprecationError messages, so its exit code alone does not prove those deprecated names are absent. This PR migrates the names on both platforms and tests the rendering explicitly.

Fix

Migrate every shipped config to the replacement option names:

Removed in 1.40	Replacement
`fieldpass`	`fieldinclude`
`fielddrop`	`fieldexclude`
`pid_tag = true`	`tag_with = ["pid"]`

Covers the Linux daemonset, replicaset, Prometheus sidecar, and the separate ama-logs process-metrics instance; the Windows daemonset and process-metrics configs; and the Prometheus/OSM stanzas generated at runtime by tomlparser-prom-customconfig.rb and tomlparser-osm-config.rb.

pid_tag (inputs.procstat) is not in the warning list above because that config runs as a separate, unguarded Telegraf process whose output isn't in the pod log tail — it is removed in 1.40 all the same, and was caught during validation.

Also drops ignore_protocol_stats from inputs.net. Protocol-stat collection was removed in 1.37.0 and the option has been a no-op since, so behavior already matches the previous true. (That one is scheduled for removal in 1.45, not 1.40.)

Windows is migrated too

This reverses the earlier position in this PR. When it was opened, Windows was pinned to Telegraf 1.24.2, which predates fieldinclude/fieldexclude/tag_with (all added in 1.29.0), so renaming would have broken it immediately and the Windows configs were deliberately left alone.

#1785 has since moved Windows to 1.40.1. The legacy names are now the breaking choice on Windows, not the safe one, so the Windows configs are migrated here along with Linux.

Two consequences:

build/windows/installer/conf/telegraf.conf and telegraf-ama-logs-process-metrics.conf are now part of this change.
tomlparser-prom-customconfig.rb is shared by both platforms. It previously selected option names via is_windows?; that branch is gone — both platforms render fieldinclude/fieldexclude unconditionally, and the timeout_config_key indirection is inlined back to timeout. The remaining is_windows? uses in that file are unrelated (config file paths and the TELEMETRY_CUSTOM_PROM_MONITOR_PODS env var) and are untouched.

kubernetes/windows/setup.ps1 is not modified by this PR; the Windows version pin and its upgrade reminder come from ci_prod.

Startup validation gate

The gate in kubernetes/linux/main.sh was:

/opt/telegraf --non-strict-env-handling --config <conf> --input-filter file -test

--input-filter file restricts loading to the dummy file input, so it returns RC=0 for a config that is fatally invalid — the broken config gets promoted, and the real (backgrounded, unchecked) Telegraf start then dies silently. I verified this against the 1.39.1 binary: the old gate passed a config the agent could not actually load.

Replaced with telegraf config check, which validates every configured plugin and performs no gathering. Note that simply dropping --input-filter file from -test would have been a regression — -test really does scrape kubelet/cadvisor, so it could fail on unready endpoints and silently discard customer Prometheus config.

--non-strict-env-handling must come after the config check subcommand; in the global position it is silently ignored.

Validation

Live A/B against the current production image

The branch image was backdoor-deployed to an AKS cluster alongside the production image (ciprod:3.8.0-ci-prod-09-03-2026-fd42f68c) and compared across 10 Telegraf configuration scenarios — default collection, pod scraping, namespace filtering, custom URLs, field filtering, label selectors, process metrics, and OSM with zero/one/multiple namespaces.

InsightsMetrics and Perf matched per-minute for every scenario; ContainerLogV2 showed no trend change.
Windows Telegraf 1.40.1 was observed running with the migrated configs on both Windows Server 2022 and 2025 nodes, with custom Prometheus scraping enabled — direct confirmation that the Windows rename loads cleanly on the new binary.

Static and unit

59 focused Ruby regression tests and 2,004 assertions passed, including namespaced field filters and escaped selectors. Three new tests cover process-metrics PID tags, modern keys plus escaping in namespace filters, and a guard asserting the shipped templates contain no removed options.
Ruby syntax, Bash syntax, whitespace, and conflict-marker checks passed.
The published Azure Linux Telegraf 1.40.1 binary passed config check for all four shipped Linux templates and for generated daemonset, replicaset, and Prometheus-sidecar configurations. Remaining startup placeholders used test defaults and service-account fixtures in an isolated container with networking disabled.
A deliberately unknown plugin option was rejected by the 1.40.1 config check.

Earlier validation (Telegraf 1.39.1, live cluster)

Config	Deprecation warnings	`config check`
`telegraf.conf` (daemonset)	6 → 0	RC=0
`telegraf-ama-logs-process-metrics.conf`	9 → 0	RC=0
`telegraf-rs.conf` (replicaset)	0 → 0	RC=0
`telegraf-prom-side-car.conf`	0 → 0	RC=0
Generated stanzas (namespaced prom + OSM)	0 → 0	RC=0

Also confirmed:

inputs.net emits an identical field set before/after the rename — no metric drift.
Upstream's own telegraf config migrate independently produces the same mapping for all five affected plugin types (disk, diskio, net, prometheus, procstat). Its output wasn't taken wholesale because it reorders and reformats the entire file; the renames here are surgical.
Key ordering is preserved so plugin-level keys stay above nested tables ([inputs.disk.tagdrop], [inputs.procstat.tags]) — otherwise they'd bind to the nested table, which is what the existing influxdata/telegraf#5615 "ORDER matters" comment guards against.

Not changed

Customer-facing ConfigMap keys — fieldpass/fielddrop under prometheus_data_collection_settings in container-azm-ms-agentconfig.yaml are a public API and stay as-is. The parser reads those keys and maps them to the current Telegraf names when rendering TOML.
TELEMETRY_*_FIELDPASS_LENGTH env var names.
Telegraf version pins on either platform (owned by Upgrade telegraf-agent to 1.40.1 #1789 and Upgrade Windows Telegraf to 1.40.0 #1785, already in ci_prod).
kubernetes/windows/setup.ps1.

azure-pipelines · 2026-08-17T22:31:40Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

The linux agent ships telegraf 1.39.x and logs deprecation warnings for options that telegraf removes in 1.40.0. Telegraf treats an unrecognized option as a fatal config-load error (not a warning), and an empty value such as `fieldpass = []` fails identically, so on 1.40 every linux ama-logs telegraf process would refuse to start -- including on clusters using the default ConfigMap. Migrate the linux configs to the replacement option names: fieldpass -> fieldinclude fielddrop -> fieldexclude pid_tag -> tag_with (inputs.procstat, also removed in 1.40) Covers the daemonset, replicaset, prometheus sidecar and the separate ama-logs process-metrics telegraf instance, plus the prometheus stanzas generated at runtime by tomlparser-prom-customconfig.rb (linux branch) and tomlparser-osm-config.rb. Windows is intentionally left on the legacy option names: it is pinned to telegraf 1.24.2, which predates fieldinclude/fieldexclude/tag_with (added in 1.29.0). tomlparser-prom-customconfig.rb is shared by both platforms, so it now selects the option names via is_windows?, mirroring the existing timeout/response_timeout branch. The customer-facing ConfigMap keys (prometheus_data_collection_settings fieldpass/fielddrop) and the TELEMETRY_* env var names are unchanged; the new names are only used when rendering telegraf TOML. Also drop ignore_protocol_stats from inputs.net. Protocol stat collection was removed in telegraf 1.37.0 and the option is a no-op, so current behavior already matches the previous `true` setting. The startup validation used `--input-filter file -test`, which only loads the dummy file input and so returned success for configs that are fatally invalid, promoting a broken config that then killed telegraf silently at the real (backgrounded, unchecked) start. Replace it with `telegraf config check`, which validates every configured plugin without gathering, so it stays fast and performs no network scrapes. Verified against the telegraf 1.39.1 binary in a live cluster: daemonset config goes from 6 deprecation warnings to 0, process-metrics from 9 to 0, replicaset/sidecar/generated stanzas remain at 0, all with config check RC=0, and inputs.net emits the identical field set. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

suyadav1 · 2026-08-17T23:38:23Z

/azp run

azure-pipelines · 2026-08-17T23:38:38Z

Azure Pipelines: Successfully started running 1 pipeline(s). 2 pipeline(s) were filtered out due to trigger conditions.

github-actions · 2026-08-25T10:44:03Z

This PR is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.

github-actions · 2026-08-31T10:41:04Z

This PR was closed because it has been stalled for 12 days with no activity.

github-actions · 2026-09-06T10:36:19Z

This PR was closed because it has been stalled for 12 days with no activity.

github-actions · 2026-09-16T10:39:19Z

This PR was closed because it has been stalled for 12 days with no activity.

The Telegraf 1.40.1 bump and Windows note removal belong to PR #1764, not this ConfigMap shell-quoting PR. Reverts commit 44237c6. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Resolve the Prometheus parser conflict by retaining ci_prod's TOML escaping and platform-specific field filter names. Update regression coverage and remove the five-line Windows Telegraf pinning note. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Upgrade Windows Telegraf to 1.40.0 Consume the official upstream Windows package with a pinned SHA256 and migrate timeout, field-filter, and procstat PID-tag configuration. Preserve ConfigMap keys and Linux behavior; document the deferred discovery cleanup and deadline gaps. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix Telegraf service startup in Windows containers Host the unchanged upstream executable with the existing win32-service dispatcher instead of relying on Telegraf's session-zero service detection. Preserve both service roles, bound shutdown and log growth, and contain the child in a kill-on-close job. Cover registration and lifecycle behavior without relaxing TLS, authentication, strict config parsing, or protected-memory defaults. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Close Telegraf service startup containment gap Enroll the service host in its kill-on-close job before creating the Telegraf child, retain the non-inheritable handle until host exit, and cover abrupt and normal startup-boundary exits with local Windows process tests. Wire Windows worker tests into the existing Ruby driver and narrowly suppress the public-checksum and test-loopback DevSkim false positives. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08a78bbb-f751-41cb-87b6-96aaf4c1a1ae * Keep existing Telegraf field filters in Windows upgrade Remove the optional fieldinclude/fieldexclude migration and its OS-specific rendering/test helper. Preserve fieldpass/fielddrop and the original timeout variable while retaining the required overall-timeout and procstat PID-tag fixes. Add pinned upstream references for the compatibility rationale. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08a78bbb-f751-41cb-87b6-96aaf4c1a1ae --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08a78bbb-f751-41cb-87b6-96aaf4c1a1ae

Retain the validated Telegraf 1.40.1 configuration and tests. The resulting tree is identical to the previous PR head; this restores the ci_prod parent lost during the earlier pull/rebase. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-09-30T10:42:35Z

This PR was closed because it has been stalled for 12 days with no activity.

) * fix: quote ConfigMap-derived environment assignments * Upgrade Linux Telegraf to 1.40.1 Remove the outdated Windows Telegraf update comment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert misplaced Telegraf update The Telegraf 1.40.1 bump and Windows note removal belong to PR #1764, not this ConfigMap shell-quoting PR. Reverts commit 44237c6. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

azure-pipelines · 2026-10-05T17:34:15Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

…include-migration

suyadav1 · 2026-10-05T18:22:30Z

/azp run

azure-pipelines · 2026-10-05T18:22:46Z

Azure Pipelines: Successfully started running 1 pipeline(s). 2 pipeline(s) were filtered out due to trigger conditions.

zanejohnson-azure · 2026-10-06T20:34:53Z

Windows is pinned to Telegraf 1.24.2 (kubernetes/windows/setup.ps1), which predates fieldinclude/fieldexclude/tag_with — all added in 1.29.0. Renaming there would break it immediately. The Windows configs keep the legacy names. The five-line Windows pinning note has been removed from setup.ps1; its existing upgrade reminder and Windows version remain unchanged.

^ is outdated.

zanejohnson-azure · 2026-10-06T20:37:05Z

PR description says telegraf on windows is 1.24.2.

suyadav1 · 2026-10-06T23:31:00Z

PR description says telegraf on windows is 1.24.2.

Updated the PR descriptioon

zanejohnson-azure · 2026-10-07T05:49:35Z

/azp run

azure-pipelines · 2026-10-07T05:49:52Z

Azure Pipelines: Successfully started running 1 pipeline(s). 2 pipeline(s) were filtered out due to trigger conditions.

suyadav1 requested a review from a team as a code owner August 17, 2026 22:31

suyadav1 force-pushed the fix/telegraf-fieldinclude-migration branch from a688b2a to 2cdc5af Compare August 17, 2026 22:33

github-actions Bot added the no-pr-activity label Aug 25, 2026

github-actions Bot closed this Aug 31, 2026

suyadav1 reopened this Aug 31, 2026

github-actions Bot closed this Sep 6, 2026

suyadav1 reopened this Sep 10, 2026

github-actions Bot closed this Sep 16, 2026

suyadav1 reopened this Sep 24, 2026

suyadav1 and others added 4 commits September 24, 2026 21:56

windows telefraf upgrade 1.40.1 and deprecated field changes

16153dd

github-actions Bot closed this Sep 30, 2026

suyadav1 reopened this Oct 5, 2026

suyadav1 added 2 commits October 5, 2026 17:51

Merge remote-tracking branch 'origin/ci_prod' into fix/telegraf-field…

7ae41c6

…include-migration

tests

8b5c889

github-actions Bot removed the no-pr-activity label Oct 6, 2026

suyadav1 changed the title ~~Fix Telegraf options removed in 1.40 on Linux (fieldpass -> fieldinclude, pid_tag -> tag_with)~~ Fix Telegraf options removed in 1.40 on Linux and Windows (fieldpass -> fieldinclude, pid_tag -> tag_with) Oct 6, 2026

suyadav1 enabled auto-merge (squash) October 7, 2026 01:15

zanejohnson-azure approved these changes Oct 7, 2026

View reviewed changes

suyadav1 merged commit baf496c into ci_prod Oct 7, 2026
20 checks passed

azure-monitor-assistant Bot mentioned this pull request Oct 7, 2026

[Generated by SRE Agent] release: bump VERSION to 3.9.0 and add release notes #1796

Merged

Conversation

suyadav1 commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Windows is migrated too

Startup validation gate

Validation

Live A/B against the current production image

Static and unit

Earlier validation (Telegraf 1.39.1, live cluster)

Not changed

Uh oh!

azure-pipelines Bot commented Aug 17, 2026

Uh oh!

suyadav1 commented Aug 17, 2026

Uh oh!

azure-pipelines Bot commented Aug 17, 2026

Uh oh!

github-actions Bot commented Aug 25, 2026

Uh oh!

github-actions Bot commented Aug 31, 2026

Uh oh!

github-actions Bot commented Sep 6, 2026

Uh oh!

github-actions Bot commented Sep 16, 2026

Uh oh!

github-actions Bot commented Sep 30, 2026

Uh oh!

azure-pipelines Bot commented Oct 5, 2026

Uh oh!

suyadav1 commented Oct 5, 2026

Uh oh!

azure-pipelines Bot commented Oct 5, 2026

Uh oh!

zanejohnson-azure commented Oct 6, 2026

Uh oh!

zanejohnson-azure commented Oct 6, 2026

Uh oh!

suyadav1 commented Oct 6, 2026

Uh oh!

zanejohnson-azure commented Oct 7, 2026

Uh oh!

azure-pipelines Bot commented Oct 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

suyadav1 commented Aug 17, 2026 •

edited

Loading