Skip to content

fix: AWS VM counts treat terminated instances as running and misclassify Karpenter nodes#73

Open
amanfredi wants to merge 1 commit into
CrowdStrike:mainfrom
amanfredi:fix/aws-running-count-terminated-karpenter
Open

fix: AWS VM counts treat terminated instances as running and misclassify Karpenter nodes#73
amanfredi wants to merge 1 commit into
CrowdStrike:mainfrom
amanfredi:fix/aws-running-count-terminated-karpenter

Conversation

@amanfredi

Copy link
Copy Markdown

Summary

Two related counting bugs in AWS/aws_cspm_benchmark.py cause large overcounts of running VMs in accounts with high EC2 instance churn (Karpenter / Spot autoscaling):

  1. Terminated instances are counted as running. is_vm_running() returned state != "stopped", which classifies terminated and shutting-down instances as running. DescribeInstances keeps terminated instances visible for up to ~1 hour after termination, so any account that churns instances quickly carries a standing pool of terminated instances that all land in the vms_running column. Fixed by counting only pending and running states as running.

  2. Karpenter-launched EKS nodes are classified as VMs, not Kubernetes nodes. EKS_TAGS only contains managed-nodegroup/eksctl tag keys (eks:cluster-name, aws:eks:cluster-name, eks:nodegroup-name, alpha.eksctl.io/nodegroup-type). Karpenter-launched nodes instead carry eks:eks-cluster-name and karpenter.sh/* tags, so they fall into the VM columns. Fixed by adding eks:eks-cluster-name, karpenter.sh/nodepool (Karpenter v1beta1/v1), and karpenter.sh/provisioner-name (legacy Karpenter) to EKS_TAGS.

The two bugs compound: every recently-terminated Karpenter node is reported as a running VM.

Real-world impact

Observed in an AWS account running EKS with Karpenter (short-lived nodes, high churn). The EC2 console showed ~150 running instances, while the script reported 468 running VMs + 212 running Kubernetes nodes. Comparing old vs. fixed logic against the same live DescribeInstances snapshot:

old logic fixed logic actual state
vms_running 451 1 1 plain VM running
kubenodes_running 119 112 ~113 nodes live (churn between calls)
kubenodes_terminated 0 457 457 terminated Karpenter nodes still visible in the API
vms_terminated 1 1 1 stopped instance

Since these counts feed Falcon sizing/licensing estimates, the overcount is nearly 3x for this account.

Notes

  • With this change the *_terminated columns now include instances in terminated, shutting-down, stopping, and stopped states (previously only stopped), which matches the column label in AWS/README.md.
  • flake8, pylint, and bandit pass with the same configuration as CI (the one pre-existing R1720 pylint hint on main is untouched).

🤖 Generated with Claude Code

…ify Karpenter nodes

is_vm_running() counted every instance whose state was not 'stopped' as
running, including 'terminated' and 'shutting-down'. DescribeInstances
keeps terminated instances visible for up to ~1 hour, so accounts with
high instance churn (e.g. Karpenter or Spot autoscaling) report hundreds
of phantom running VMs. Only 'pending' and 'running' states now count as
running.

Karpenter-launched EKS nodes are tagged with eks:eks-cluster-name and
karpenter.sh/* rather than the managed-nodegroup tags in EKS_TAGS, so
they were counted in the VM columns instead of the Kubernetes node
columns. The Karpenter tag keys are now recognized.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant