Skip to content

Update backdoor-deployment skill to use the in-repo chart - #1795

Merged
suyadav1 merged 1 commit into
ci_prodfrom
suyadav/backdoor-skill-inrepo-chart
Oct 6, 2026
Merged

suyadav1 merged 1 commit into
ci_prodfrom
suyadav/backdoor-skill-inrepo-chart

Conversation

@suyadav1

@suyadav1 suyadav1 commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Summary

The backdoor-deployment skill targeted the external sibling chart azuremonitor-containerinsights-for-prod-clusters, which no longer reflects how CI/CD deploys the agent. This retargets it at the in-repo charts/azuremonitor-containerinsights and corrects the mechanics that differ.

Docs-only change — affects .github/skills/backdoor-deployment/SKILL.md only. No product code or pipeline changes.

What was broken

Four things would have caused a silent or hard failure if the skill were run against the in-repo chart as written:

# Issue Fix
1 Chart is template-only — Helm fails with Error: Chart.yaml file is missing Document the envsubst generation step, matching .pipelines/helm-deploy-templates/ama-logs-helm-deploy.yaml
2 Image is assembled as MCR host + imageRepository + : + tag, so the old cidev:<tag> form renders the invalid .../ciprod:cidev:<tag> Bare tags, with ciprod/cidev as a separate imageRepository value
3 An empty tag does not fail — it silently falls back to the hardcoded 3.1.34 in get.addonImageTag, so a run would compare prod against prod and report "no regression" Explicit warning, plus a mandatory read-back of the deployed image before collecting data
4 Cluster coordinates are <your_*> placeholders in values-template.yaml and cannot be parsed from the chart Supplied as --set overrides; workspace resolved from the DCR

Input and discovery changes

  • Required inputs reduced to branch name and cluster resource ID.
  • Production image is read from ReleaseNotes.md rather than a hardcoded tag that goes stale (the previous default, 3.1.35, was already wrong).
  • Workspace is resolved from the cluster's ContainerInsightsExtension DCR → workspaceResourceId → customerId, with a bootstrap path that enables the addon once when no DCR exists yet. This matters because clusters onboarded without an explicit workspace land in a regional default such as DefaultWorkspace-<sub>-<region>, whose name is unrelated to the cluster.

Deployment notes added

  • Full CI/CD override set, including accessTokenSecretName=aad-msi-auth-token — MSI clusters need this in place of the chart's legacy workspace-key default.
  • --server-side / --force-conflicts are Helm 4 flags (the pipeline pins helmVersionToInstall: latest) and fail on Helm 3.
  • Disabling the managed addon is mandatory: installing over it fails with a Helm ownership error. Cleanup must restore the addon to its original state.

Validation

Verified by an actual backdoor deployment to a test cluster, following the updated skill verbatim:

  • helm lint clean; helm template renders 9 objects with no unsubstituted placeholders
  • kubectl apply --dry-run=server validated all 9 objects against a live K8s 1.34.4 API server
  • Install succeeded; 5/5 pods Running, 0 restarts within 90s (2 Linux, 2 Windows, 1 replicaset)
  • Deployed images exact-matched the intended tags — confirming the fallback warning is actionable
  • MSI token adapter acquired tokens successfully using aad-msi-auth-token
  • Sustained ingestion for 5 consecutive minutes: Perf ~1358/min, InsightsMetrics 136/min, ContainerLogV2 ~3.3-4.2k/min
  • Cluster restored to its original managed-addon state afterwards

The Helm ownership error and the DCR-deletion-on-disable behaviour documented here were both observed directly during this validation.

The skill targeted the external sibling chart
`azuremonitor-containerinsights-for-prod-clusters`, which no longer
reflects how CI/CD deploys the agent. Retarget it at the in-repo
`charts/azuremonitor-containerinsights` and correct the mechanics that
differ, verified by a live backdoor deployment.

Chart generation
- The chart is template-only (`Chart-template.yaml` / `values-template.yaml`);
  Helm fails with "Chart.yaml file is missing" until `envsubst` generates
  them. Document the generation step, matching
  `.pipelines/helm-deploy-templates/ama-logs-helm-deploy.yaml`.

Image tags
- Images are assembled as MCR host + `imageRepository` + `:` + tag, so tags
  must be bare and the ciprod/cidev switch is a separate `imageRepository`
  value. The old `cidev:<tag>` form rendered an invalid reference.
- Warn that an empty tag silently falls back to the hardcoded `3.1.34` in
  `get.addonImageTag`, which would otherwise compare prod against prod, and
  require reading the deployed image back before collecting data.

Inputs
- Reduce required inputs to branch name and cluster resource ID.
- Resolve the production image from `ReleaseNotes.md` instead of hardcoding
  a tag that goes stale.
- Resolve the workspace from the cluster's ContainerInsightsExtension DCR,
  including a bootstrap path that enables the addon once when no DCR exists
  yet. Clusters onboarded without an explicit workspace land in a regional
  default whose name is unrelated to the cluster.

Deployment
- Document the full CI/CD override set, including
  `accessTokenSecretName=aad-msi-auth-token`, which MSI clusters need in
  place of the chart's legacy workspace-key default.
- Note that `--server-side` / `--force-conflicts` are Helm 4 flags and fail
  on Helm 3.
- Record that disabling the managed addon is mandatory, since installing
  over it fails with a Helm ownership error, and that cleanup must restore
  the addon to its original state.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4fc0848a-8d3a-4c44-a69e-fd3bfe11ca50
@suyadav1
suyadav1 requested a review from a team as a code owner October 6, 2026 19:52
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@suyadav1

suyadav1 commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
2 pipeline(s) were filtered out due to trigger conditions.

@suyadav1
suyadav1 enabled auto-merge (squash) October 6, 2026 20:07
@suyadav1
suyadav1 merged commit 586b305 into ci_prod Oct 6, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants