From b5975036a1d98aa27c50692a358edccef83e4f9e Mon Sep 17 00:00:00 2001 From: Sunil Yadav Date: Tue, 6 Oct 2026 19:50:39 +0000 Subject: [PATCH] Update backdoor-deployment skill to use the in-repo chart The skill targeted the external sibling chart `azuremonitor-containerinsights-for-prod-clusters`, which no longer reflects how CI/CD deploys the agent. Retarget it at the in-repo `charts/azuremonitor-containerinsights` and correct the mechanics that differ, verified by a live backdoor deployment. Chart generation - The chart is template-only (`Chart-template.yaml` / `values-template.yaml`); Helm fails with "Chart.yaml file is missing" until `envsubst` generates them. Document the generation step, matching `.pipelines/helm-deploy-templates/ama-logs-helm-deploy.yaml`. Image tags - Images are assembled as MCR host + `imageRepository` + `:` + tag, so tags must be bare and the ciprod/cidev switch is a separate `imageRepository` value. The old `cidev:` form rendered an invalid reference. - Warn that an empty tag silently falls back to the hardcoded `3.1.34` in `get.addonImageTag`, which would otherwise compare prod against prod, and require reading the deployed image back before collecting data. Inputs - Reduce required inputs to branch name and cluster resource ID. - Resolve the production image from `ReleaseNotes.md` instead of hardcoding a tag that goes stale. - Resolve the workspace from the cluster's ContainerInsightsExtension DCR, including a bootstrap path that enables the addon once when no DCR exists yet. Clusters onboarded without an explicit workspace land in a regional default whose name is unrelated to the cluster. Deployment - Document the full CI/CD override set, including `accessTokenSecretName=aad-msi-auth-token`, which MSI clusters need in place of the chart's legacy workspace-key default. - Note that `--server-side` / `--force-conflicts` are Helm 4 flags and fail on Helm 3. - Record that disabling the managed addon is mandatory, since installing over it fails with a Helm ownership error, and that cleanup must restore the addon to its original state. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4fc0848a-8d3a-4c44-a69e-fd3bfe11ca50 --- .github/skills/backdoor-deployment/SKILL.md | 317 ++++++++++++++++++-- 1 file changed, 292 insertions(+), 25 deletions(-) diff --git a/.github/skills/backdoor-deployment/SKILL.md b/.github/skills/backdoor-deployment/SKILL.md index e462a8e20..0c5f2e5f8 100644 --- a/.github/skills/backdoor-deployment/SKILL.md +++ b/.github/skills/backdoor-deployment/SKILL.md @@ -1,7 +1,7 @@ --- name: backdoor-deployment description: "Validate a container image change via backdoor deployment. Use when: deploying test image to a cluster, comparing data volume between deployments, comparing resource consumption, backdoor deploy, validate container image, image regression testing, build and deploy branch." -argument-hint: "Provide branch name, current production image, and YAML file path" +argument-hint: "Provide branch name and cluster resource ID (production image, chart path, and workspace are auto-detected)" --- # Backdoor Deployment Automation @@ -10,25 +10,35 @@ Validates a container image change by deploying the current production image, co ## Required Inputs -Check with the user if they want to use the default values or provide new ones. +Only these two come from the user. Everything else is derived — do not ask for it. | Input | Description | Default | |-------|-------------|---------| | **Branch name** | Git branch to build | `suyadav/aiautomation` | -| **Current production image** | Production image tag (e.g. `ciprod:X.Y.Z`) | `ciprod:3.1.35` | -| **YAML file path** | Helm values file for backdoor deployment | `./../azuremonitor-containerinsights/values.yaml` | +| **Cluster resource ID** | Full ARM ID of the target AKS cluster | — ask the user | ## Derived Values -Parse these automatically from the YAML file — do not ask the user. +Resolve all of these automatically. | Value | Source | |-------|--------| -| **Cluster Resource ID** | `OmsAgent.aksResourceID` | -| **Log Analytics Workspace ID** | `OmsAgent.workspaceID` (a GUID used with `az monitor log-analytics query -w`) | +| **Chart path** | Fixed: `charts/azuremonitor-containerinsights` | +| **Current production image** | `ReleaseNotes.md` — see "Determine the current production image" | | **Cluster Name** | Last segment of the cluster resource ID (for `kubectl config use-context`) | | **Subscription ID** | Extracted from the cluster resource ID (`/subscriptions//...`) | | **Resource Group** | Extracted from the cluster resource ID (`/resourceGroups//...`) | +| **Region** | `az aks show -g -n --query location -o tsv` | +| **Kubernetes version** | `kubectl version -o json \| jq -r '.serverVersion.gitVersion' \| sed 's/^v//'` | +| **DCR ID** | The `ContainerInsightsExtension` DCR association on the cluster — see "Resolve the workspace" | +| **Workspace resource ID** | `destinations.logAnalytics[].workspaceResourceId` of that DCR | +| **Workspace GUID** | `customerId` of that workspace (used for `--set OmsAgent.workspaceID` and `az monitor log-analytics query -w`) | + +> **The chart carries no cluster coordinates.** `charts/azuremonitor-containerinsights` is the same +> chart CI/CD deploys, and its `values-template.yaml` ships placeholders (``, +> ``, …) rather than real values, so nothing can be parsed out of it. The cluster +> resource ID comes from the user and the workspace is resolved from the DCR; both are passed as +> `--set` overrides. ## Build Pipeline @@ -44,29 +54,255 @@ Parse these automatically from the YAML file — do not ask the user. - If asked **"what's the next step"**, read `BackdoorDeploymentOutput.md` and suggest the next step. - Before executing any step, verify previous step data exists in `BackdoorDeploymentOutput.md`. If missing, confirm with the user before proceeding. - If the build must be retriggered, **keep the existing production baseline data** — do not re-deploy the production image or re-collect baseline data. -- After the workflow completes, **restore the YAML file** to its original production image values. +- After the workflow completes, **redeploy the production image** so the cluster is left on prod, and + re-enable the managed addon if you disabled it. The generated `Chart.yaml` / `values.yaml` are + gitignored and may be left in place or deleted — either way the tree stays clean. ## Procedures -### Update YAML Image Tags +### Resolve the workspace -1. Only update the image version — do NOT change any other part of the file. -2. Update exactly two fields: `imageTagLinux` and `imageTagWindows`. -3. **Windows naming convention**: prefix `win-` after the image type. Examples: - - `cidev:3.1.27-2-abc123-20250520184627` → `cidev:win-3.1.27-2-abc123-20250520184627` - - `ciprod:3.1.27` → `ciprod:win-3.1.27` +Never ask the user for the workspace. Derive it from the cluster's `ContainerInsightsExtension` DCR +association, which is the workspace the cluster actually ships data to: + +```bash +AKS_RESOURCE_ID="/subscriptions//resourceGroups//providers/Microsoft.ContainerService/managedClusters/" + +DCR_ID=$(az monitor data-collection rule association list --resource "$AKS_RESOURCE_ID" \ + --query "[?name=='ContainerInsightsExtension'].dataCollectionRuleId | [0]" -o tsv) + +WS_RESOURCE_ID=$(az monitor data-collection rule show --ids "$DCR_ID" \ + --query "destinations.logAnalytics[0].workspaceResourceId" -o tsv) + +WS_GUID=$(az monitor log-analytics workspace show --ids "$WS_RESOURCE_ID" --query customerId -o tsv) +``` + +- `WS_GUID` → `--set OmsAgent.workspaceID` and `az monitor log-analytics query -w` +- `WS_RESOURCE_ID` → needed later to re-enable the addon +- `DCR_ID` → needed later to recreate the association + +Resolving from the DCR (rather than guessing a cluster-named workspace) matters: clusters onboarded +without an explicit workspace land in a **regional default** such as +`DefaultWorkspace--` in resource group `DefaultResourceGroup-`, +which bears no relation to the cluster name. + +#### If the DCR does not exist yet + +`DCR_ID` is empty when Container Insights was never enabled on the cluster. Bootstrap it by enabling +the addon once — that provisions the DCR, the association, and a default workspace — then capture +the values and disable it again before deploying: + +```bash +if [ -z "$DCR_ID" ]; then + # omit --workspace-resource-id to let AKS create/attach the regional default workspace, + # or pass one explicitly if the user has a preferred workspace + az aks enable-addons -a monitoring -g -n + + DCR_ID=$(az monitor data-collection rule association list --resource "$AKS_RESOURCE_ID" \ + --query "[?name=='ContainerInsightsExtension'].dataCollectionRuleId | [0]" -o tsv) + WS_RESOURCE_ID=$(az monitor data-collection rule show --ids "$DCR_ID" \ + --query "destinations.logAnalytics[0].workspaceResourceId" -o tsv) + WS_GUID=$(az monitor log-analytics workspace show --ids "$WS_RESOURCE_ID" --query customerId -o tsv) +fi +``` + +Then continue into the normal flow (disable the addon, recreate the association, deploy). Record +whether the addon was **originally** enabled: if it was never enabled, leave it disabled at cleanup +rather than enabling something the user never had. + +This bootstrap also creates the `aad-msi-auth-token` secret that the agent needs for MSI auth, so it +is required even if you only care about the workspace values. + +### Determine the current production image + +Do **not** hardcode a production tag — it goes stale. Read it from `ReleaseNotes.md` in the repo +root, whose newest entry under `## Release History` is the current production release: + +```bash +PROD_TAG=$(grep -m1 -oP 'ciprod:\K[^ ]+(?= \(linux\))' ReleaseNotes.md) +PROD_TAG_WIN=$(grep -m1 -oP 'ciprod:\K[^ ]+(?= \(windows\))' ReleaseNotes.md) +echo "$PROD_TAG / $PROD_TAG_WIN" +# -> 3.8.0-ci-prod-09-03-2026-fd42f68c / win-3.8.0-ci-prod-09-03-2026-fd42f68c +``` + +`grep -m1` takes the **first** match, and release notes are newest-first, so this always yields the +latest release. The entries look like: + +``` +### 09/03/2026 - +##### Version mcr.microsoft.com/azuremonitor/containerinsights/ciprod:3.8.0-ci-prod-09-03-2026-fd42f68c (linux) +##### Version mcr.microsoft.com/azuremonitor/containerinsights/ciprod:win-3.8.0-ci-prod-09-03-2026-fd42f68c (windows) +``` + +These are already **bare tags** in the form this chart expects, and the Windows tag is read directly +rather than derived — use it as-is. + +Confirm your branch is not behind on release notes (`git log -1 --format=%cd -- ReleaseNotes.md`). If +the value disagrees with what the cluster is actually running, prefer the live cluster image and tell +the user: + +```bash +kubectl get ds ama-logs -n kube-system \ + -o jsonpath='{range .spec.template.spec.containers[?(@.name=="ama-logs")]}{.image}{"\n"}{end}' +``` + +### Generate the chart (mandatory first step) + +`charts/azuremonitor-containerinsights` is **template-only**. It ships `Chart-template.yaml` and +`values-template.yaml`; `Chart.yaml` and `values.yaml` are generated at build time and are +**gitignored** (`.gitignore:36-37`), so generating them is safe and leaves the tree clean. + +Running Helm before generating fails with `Error: Chart.yaml file is missing`. + +Generate exactly as CI/CD does (`.pipelines/helm-deploy-templates/ama-logs-helm-deploy.yaml`): + +```bash +cd charts/azuremonitor-containerinsights +export HELM_SEMVER="" # e.g. 3.1.34-17-g67321cf0d-20260323045331 +export IMAGE_TAG="" +export IMAGE_TAG_WINDOWS="" # e.g. win-3.1.34-17-g67321cf0d-20260323045331 +envsubst '${HELM_SEMVER} ${IMAGE_TAG} ${IMAGE_TAG_WINDOWS}' < Chart-template.yaml > Chart.yaml +envsubst '${HELM_SEMVER} ${IMAGE_TAG} ${IMAGE_TAG_WINDOWS}' < values-template.yaml > values.yaml +``` + +`envsubst` only expands `${...}`. The `` / `` style placeholders +are **not** substituted and must be passed as `--set` overrides at deploy time (see below). + +### Image tags for this chart + +The image is assembled as +`mcr.microsoft.com` + `OmsAgent.imageRepository` + `:` + `OmsAgent.imageTagLinux`. + +So the tag is the **bare tag**, and the repo (`ciprod` vs `cidev`) is a **separate** value: + +| Image | `OmsAgent.imageRepository` | `OmsAgent.imageTagLinux` | `OmsAgent.imageTagWindows` | +|-------|---------------------------|--------------------------|-----------------------------| +| production | `/azuremonitor/containerinsights/ciprod` | `$PROD_TAG` from `ReleaseNotes.md` | `$PROD_TAG_WIN` from `ReleaseNotes.md` | +| **test build** | `/azuremonitor/containerinsights/cidev` | `3.1.34-17-g67321cf0d-20260323045331` | `win-3.1.34-17-g67321cf0d-20260323045331` | + +> ⚠️ **Do not use the `cidev:` form here.** That was the convention for the older +> `azuremonitor-containerinsights-for-prod-clusters` chart. Passing `cidev:3.1.34` as the tag renders +> `…/ciprod:cidev:3.1.34`, which is an invalid reference. + +> 🚨 **Silent-fallback footgun — this can invalidate an entire test run.** If `imageTagLinux` / +> `imageTagWindows` are empty or unset, the chart does **not** fail. It silently falls back to the +> hardcoded defaults in `get.addonImageTag` (`templates/_helpers.tpl`): **`3.1.34`** and +> **`win-3.1.34`**. You would then be comparing production against production and conclude "no +> regression". **Always verify the live image after deploying** (see the verification step below). + +**Windows naming convention**: prefix `win-` on the Linux tag — e.g. +`3.8.0-ci-prod-09-03-2026-fd42f68c` → `win-3.8.0-ci-prod-09-03-2026-fd42f68c`. For production tags +read the Windows line from `ReleaseNotes.md` directly instead of deriving it; only derive for CI +build tags, where the build log may list the Linux tag alone. ### Deploy with Helm -Always use `--install` to handle both fresh installs and upgrades: +Deploy with the same override set CI/CD uses. Always use `--install` so the command handles both +fresh installs and upgrades: + +```bash +AKS_RESOURCE_ID="/subscriptions//resourceGroups//providers/Microsoft.ContainerService/managedClusters/" + +helm upgrade --install ama-logs charts/azuremonitor-containerinsights -n kube-system \ + --set global.commonGlobals.CloudEnvironment=azurepubliccloud \ + --set global.commonGlobals.Region= \ + --set global.commonGlobals.Versions.Kubernetes= \ + --set Azure.Cluster.ResourceId="$AKS_RESOURCE_ID" \ + --set OmsAgent.aksResourceID="$AKS_RESOURCE_ID" \ + --set OmsAgent.accessTokenSecretName=aad-msi-auth-token \ + --set OmsAgent.workspaceID="$WS_GUID" \ + --set OmsAgent.imageRepository=/azuremonitor/containerinsights/ \ + --set OmsAgent.imageTagLinux= \ + --set OmsAgent.imageTagWindows= \ + --timeout 10m +``` + +Notes: +- **`accessTokenSecretName=aad-msi-auth-token` is required on MSI-auth clusters.** The chart default + is `ama-logs-secret` (legacy workspace-key auth); leaving the default breaks token acquisition. +- CI/CD additionally passes `--server-side=true --force-conflicts`. Those are **Helm 4** flags — + the pipeline pins `helmVersionToInstall: 'latest'`. On Helm 3 they fail with + `Error: unknown flag: --server-side`, so omit them (check with `helm version --short`). +- The namespace is hardcoded to `kube-system` inside the templates. +- The chart renders a `Secret` named **`ama-logs-secret`**, which is a different object from + `aad-msi-auth-token` — installing does **not** clobber the MSI token secret. +- The chart only **mounts** `container-azm-ms-agentconfig`; it never creates it, so any scenario + ConfigMap you apply for testing is safe from being overwritten. + +### Verify the deployment before collecting data + +Because of the silent-fallback behaviour above, confirm the **live** image is the one you intended: + +```bash +kubectl get ds ama-logs -n kube-system \ + -o jsonpath='{range .spec.template.spec.containers[?(@.name=="ama-logs")]}{.image}{"\n"}{end}' +kubectl get ds ama-logs-windows -n kube-system \ + -o jsonpath='{range .spec.template.spec.containers[?(@.name=="ama-logs-windows")]}{.image}{"\n"}{end}' +``` + +Abort and fix the overrides if either does not match the tag under test. + +### Dry-run first (recommended) + +Validate rendering without touching the cluster: + +```bash +helm lint charts/azuremonitor-containerinsights --set global.commonGlobals.Versions.Kubernetes= +helm template ama-logs charts/azuremonitor-containerinsights -n kube-system \ + | grep -E '^\s+image: ' | sort -u +``` + +A correct render produces 9 objects — ClusterRole, ClusterRoleBinding, 2 ConfigMaps +(`container-azm-ms-aks-k8scluster`, `ama-logs-rs-config`), 2 DaemonSets (Linux + Windows), +1 Deployment (replicaset), Secret, ServiceAccount. + +### AKS managed addon conflicts + +If the target cluster has the **managed Container Insights addon** enabled, it owns the Helm release +`aks-managed-azure-monitor-logs` and reconciles roughly every 20 minutes, reverting any backdoor +install or `kubectl set image`. + +Installing on top of it **fails outright** with an ownership error: + +``` +Error: Unable to continue with install: ServiceAccount "ama-logs" in namespace "kube-system" exists +and cannot be imported into the current release: invalid ownership metadata; annotation validation +error: key "meta.helm.sh/release-name" must equal "ama-logs": current value is +"aks-managed-azure-monitor-logs" +``` + +So disabling the addon is **mandatory**, not optional. You must already have `DCR_ID` and +`WS_RESOURCE_ID` from "Resolve the workspace" — capture them **before** disabling, because the +disable deletes the association: + +```bash +az aks disable-addons -a monitoring -g -n --yes # --yes is required (no TTY prompt) + +# recreate the association that the disable removed +az monitor data-collection rule association create --name ContainerInsightsExtension \ + --rule-id "$DCR_ID" --resource "$AKS_RESOURCE_ID" +``` + +The `aad-msi-auth-token` secret **survives** the disable, so MSI auth keeps working. + +To restore afterwards — only if the addon was enabled when you started — uninstall your release +first, then re-enable with the workspace the DCR pointed at: + ```bash -helm upgrade --install ama-logs -n kube-system +helm uninstall ama-logs -n kube-system +az aks enable-addons -a monitoring -g -n --workspace-resource-id "$WS_RESOURCE_ID" ``` -where `` is the directory containing the YAML (e.g. `./../azuremonitor-containerinsights/`). + +Uninstalling first is required: the addon install hits the same ownership error in reverse if your +release still owns `ama-logs`. + +> Check the exit code of `az` directly. Piping it into `tail`/`grep` masks failures behind the +> pipeline's exit status, so a wrong workspace name can look like success. Verify by re-reading +> `addonProfiles.omsagent.enabled` and the pod list rather than trusting the command's output. ### Collect Table Data -Run Kusto queries via `az monitor log-analytics query -w ` (or the `kusto-mcp` MCP server if available). +Run Kusto queries via `az monitor log-analytics query -w "$WS_GUID"` (or the `kusto-mcp` MCP server if available). Collect aggregated row counts in **1-minute bins** from **(deployment time + 5 min)** to **(deployment time + 10 min)** for these tables: - `ContainerInventory` @@ -121,7 +357,13 @@ Use the ADO API to read the build log directly (no need to download zip files): ``` ##[warning]Linux image built with tag: containerinsightsprod.azurecr.io/public/azuremonitor/containerinsights/cidev:3.1.34-17-g67321cf0d-20260323045331 ``` - Use `grep -o 'cidev:[^ ]*'` or similar to extract the tag. + Extract the **bare tag only** — everything *after* the `cidev:` — because this chart takes the + repository and the tag as separate values: + ```bash + grep -o 'cidev:[^ ]*' | head -1 | cut -d: -f2 + # -> 3.1.34-17-g67321cf0d-20260323045331 + ``` + The `cidev` part becomes `OmsAgent.imageRepository=/azuremonitor/containerinsights/cidev`. 3. **Derive the Windows tag** from the Linux tag using the naming convention (prefix `win-`). Alternatively, find "Docker windows build for multi-arc image" log for a line like: @@ -221,14 +463,28 @@ The workflow has two parallel tracks that converge after the build completes. ### Phase 1: Obtain Build + Deploy Production Image (parallel) -1. **Parse derived values** from the YAML file (see Derived Values table). Save all values to the output file. -2. **Set kubectl context**: `kubectl config use-context `. +1. **Collect inputs and derive values** (see Required Inputs and Derived Values tables). Only the + branch name and cluster resource ID come from the user. Resolve the production image from + `ReleaseNotes.md` (see "Determine the current production image") and the workspace from the DCR + (see "Resolve the workspace", including the bootstrap path if no DCR exists yet). Save all values + to the output file. +2. **Set kubectl context**: `kubectl config use-context `. Then record whether the AKS + **managed monitoring addon** is enabled — cleanup must put it back the way you found it: + ```bash + az aks show -g -n --query "addonProfiles.omsagent.enabled" -o tsv + ``` + If `true`, it will revert your backdoor deployment every ~20 minutes — disable it and recreate the + DCR association (see "AKS managed addon conflicts"). 3. **Check for an existing build** on the branch for the **latest commit** (definition ID 444, org: `github-private`, project: `microsoft`). - If a completed build exists on the latest commit → use it (even if it failed due to Trivy — see "Check Build Failure Reason"). - **IMPORTANT: A build that failed ONLY due to Trivy is still usable.** Do NOT fall back to a previous build. The images are already built and pushed before Trivy runs. Always extract the image tag from the failed build's logs (see "Extract Image Version from Build Logs"). - If no usable build exists → **trigger a new build**. Save the build ID. 4. **If the build is already complete**, skip to Phase 2 after finishing production baseline steps. **If the build is still running**, proceed with steps 5–9 in parallel; periodically check build status during wait times. -5. **Update YAML** with the current production image and **deploy** (see "Update YAML Image Tags" and "Deploy with Helm"). Record the **production deployment time** (UTC). +5. **Generate the chart and deploy the current production image** (see "Generate the chart", "Image + tags for this chart", and "Deploy with Helm") using + `imageRepository=/azuremonitor/containerinsights/ciprod`. **Verify the live image matches the prod + tag** before continuing (see "Verify the deployment before collecting data"). Record the + **production deployment time** (UTC). 6. **Wait 15 minutes**, then verify pods: `kubectl get pods -n kube-system | grep ama-logs`. Confirm all are Running with 0 restarts. Save pod names to the output file. 7. **Collect production baseline data** for all 6 tables (see "Collect Table Data"). Save results to the output file. @@ -236,7 +492,11 @@ The workflow has two parallel tracks that converge after the build completes. 8. **Confirm the build** completed. Check failure reason if needed (see "Check Build Failure Reason"). If it failed for a non-Trivy reason, ask the user whether to retrigger. **If it failed only due to Trivy, treat it as a successful build — the images are valid. Do NOT fall back to a previous build.** 9. **Extract the test image version** from the build logs (see "Extract Image Version from Build Logs"). Save to the output file. -10. **Update YAML** with the test image and **deploy**. Record the **test deployment time** (UTC). +10. **Regenerate the chart with the test tags and deploy** using + `imageRepository=/azuremonitor/containerinsights/cidev` and the bare tags from step 9. + **Verify the live image matches the test tag** before continuing — the chart silently falls back + to `3.1.34` if a tag is empty, which would make the comparison meaningless. Record the **test + deployment time** (UTC). 11. **Wait 15 minutes**, then verify pods are Running. If any pod restarted, get the reason via `kubectl describe pod -n kube-system`. Save pod names to the output file. 12. **Collect test data** for all 6 tables (see "Collect Table Data"). Save results to the output file. @@ -245,5 +505,12 @@ The workflow has two parallel tracks that converge after the build completes. 13. **Compare data volume** between production and test for all tables (see "Compare Data Volume"). If any table shows a difference, **investigate** before reporting (see "Investigate Data Volume Regression"). 14. **Get PodUid** for all pods in both deployments (see "Get PodUid"). 15. **Compare resource consumption** for `memoryWorkingSetBytes` and `cpuUsageNanoCores` (see "Compare Resource Consumption"). If any metric shows a sustained increase, **investigate** before reporting (see "Investigate Resource Consumption Regression"). -16. **Restore YAML** to its original production image values. -17. **Write summary** to the output file: pass/fail for each table and resource check. Include investigation findings for any anomalies — clearly distinguish between code regressions and cluster workload differences. +16. **Restore the cluster**: `helm uninstall ama-logs -n kube-system`, then put the addon back the + way you found it (step 2) — re-enable it with `$WS_RESOURCE_ID` if it was originally enabled, or + leave it disabled if it never was. Confirm 5/5 `ama-logs` pods are Running on the prod image and + that both `ContainerInsightsExtension` and `ContainerInsightsMetricsExtension` DCR associations + exist. Delete any test ConfigMaps you applied. +17. **Write summary** to the output file: pass/fail for each table and resource check. Include the + exact prod and test images **as read back from the live DaemonSets**, not just the intended tags. + Include investigation findings for any anomalies — clearly distinguish between code regressions + and cluster workload differences.