From 43f86c8a802480a80ecaba142d5286efecf0464a Mon Sep 17 00:00:00 2001 From: Andrew Mitchell Date: Mon, 2 Mar 2026 22:51:33 +0000 Subject: [PATCH] Pulling in do-not-disrupt grace period --- go.mod | 2 +- go.sum | 4 +- .../content/en/preview/concepts/disruption.md | 57 ++++++++++++++++--- website/content/en/preview/troubleshooting.md | 10 +++- 4 files changed, 61 insertions(+), 12 deletions(-) diff --git a/go.mod b/go.mod index e56e40b37363..2f7df4912d7c 100644 --- a/go.mod +++ b/go.mod @@ -50,7 +50,7 @@ require ( k8s.io/klog/v2 v2.130.1 k8s.io/utils v0.0.0-20251222233032-718f0e51e6d2 sigs.k8s.io/controller-runtime v0.22.4 - sigs.k8s.io/karpenter v1.11.0 + sigs.k8s.io/karpenter v1.11.1-0.20260407205932-501287b36a17 sigs.k8s.io/yaml v1.6.0 ) diff --git a/go.sum b/go.sum index cdc78afd857e..a0147d00a47b 100644 --- a/go.sum +++ b/go.sum @@ -375,8 +375,8 @@ sigs.k8s.io/controller-runtime v0.22.4 h1:GEjV7KV3TY8e+tJ2LCTxUTanW4z/FmNB7l327U sigs.k8s.io/controller-runtime v0.22.4/go.mod h1:+QX1XUpTXN4mLoblf4tqr5CQcyHPAki2HLXqQMY6vh8= sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 h1:IpInykpT6ceI+QxKBbEflcR5EXP7sU1kvOlxwZh5txg= sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730/go.mod h1:mdzfpAEoE6DHQEN0uh9ZbOCuHbLK5wOm7dK4ctXE9Tg= -sigs.k8s.io/karpenter v1.11.0 h1:qDUUVzRSXW70vpV18HHDXVYguEPTgh2ihskfpU1swgc= -sigs.k8s.io/karpenter v1.11.0/go.mod h1:XQtYAxoCysLHjytci7Fx5zw2txgcW2Vxc+qq6DDiFX8= +sigs.k8s.io/karpenter v1.11.1-0.20260407205932-501287b36a17 h1:CNwlJVCbvCnIMj9TbdqTbmkvZubYeCG4bM1rz1sZdKc= +sigs.k8s.io/karpenter v1.11.1-0.20260407205932-501287b36a17/go.mod h1:XQtYAxoCysLHjytci7Fx5zw2txgcW2Vxc+qq6DDiFX8= sigs.k8s.io/randfill v1.0.0 h1:JfjMILfT8A6RbawdsK2JXGBR5AQVfd+9TbzrlneTyrU= sigs.k8s.io/randfill v1.0.0/go.mod h1:XeLlZ/jmk4i1HRopwe7/aU3H5n1zNUcX6TM94b3QxOY= sigs.k8s.io/structured-merge-diff/v6 v6.3.1 h1:JrhdFMqOd/+3ByqlP2I45kTOZmTRLBUm5pvRjeheg7E= diff --git a/website/content/en/preview/concepts/disruption.md b/website/content/en/preview/concepts/disruption.md index d44fc55fa969..37c6fb5d7ff3 100644 --- a/website/content/en/preview/concepts/disruption.md +++ b/website/content/en/preview/concepts/disruption.md @@ -360,20 +360,51 @@ In this scenario, Karpenter cannot voluntary disrupt the node because: As seen in this example, the more PDBs there are affecting a Node, the more difficult it will be for Karpenter to find an opportunity to perform voluntary disruption actions. -Secondly, you can block Karpenter from voluntarily disrupting and draining pods by adding the `karpenter.sh/do-not-disrupt: "true"` annotation to the pod. -You can treat this annotation as a single-pod, permanently blocking PDB. +Secondly, you can block Karpenter from voluntarily disrupting and draining pods by adding the `karpenter.sh/do-not-disrupt` annotation to the pod. +This annotation supports two formats: + +| Format | Example | Behavior | +|--------|---------|----------| +| **Boolean** | `karpenter.sh/do-not-disrupt: "true"` | Provides permanent protection from disruption | +| **Duration** | `karpenter.sh/do-not-disrupt: "30m"` | Provides time-based protection for the specified duration after the pod starts running | + +#### Duration-Based Protection + +When using the duration format, the annotation will be "active" and pods will be protected from disruption for the specified time period after they start running (based on `pod.status.startTime`). +Once the duration expires, the annotation becomes inactive and the pod becomes eligible for disruption. +This is useful for workloads that need protection during startup or critical phases but can be safely disrupted later. + +The duration value must be a valid Go `time.Duration` string. Supported formats include: + +| Duration | Description | +|----------|-------------| +| `"5m"` | 5 minutes | +| `"1h"` | 1 hour | +| `"2h30m"` | 2 hours and 30 minutes | +| `"24h"` | 24 hours | +| `"1h30m45s"` | 1 hour, 30 minutes, and 45 seconds | + +{{% alert title="Note" color="primary" %}} +If an invalid duration is specified, the annotation will be ignored and an event will be emitted on the pod indicating that the duration format is invalid. +{{% /alert %}} + +#### Behavior and Consequences + +You can treat this annotation as a single-pod blocking PDB that is active either permanently (boolean format) or temporarily while the duration hasn't elapsed (duration format). This has the following consequences: -- Nodes with `karpenter.sh/do-not-disrupt` pods will be excluded from [Consolidation]({{}}), and conditionally excluded from [Drift]({{}}). +- Nodes with active `karpenter.sh/do-not-disrupt` pods will be excluded from [Consolidation]({{}}), and conditionally excluded from [Drift]({{}}). - If the Node's owning NodeClaim has a [`terminationGracePeriod`]({{}}) configured, it will still be eligible for disruption via drift. -- Like pods with a blocking PDB, pods with the `karpenter.sh/do-not-disrupt` annotation will **not** be gracefully evicted by the [Termination Controller]({{}}). +- Like pods with a blocking PDB, pods with an active `karpenter.sh/do-not-disrupt` annotation will **not** be gracefully evicted by the [Termination Controller]({{}}). Karpenter will not be able to complete termination of the node until one of the following conditions is met: - - All pods with the `karpenter.sh/do-not-disrupt` annotation are removed. + - All pods with the `karpenter.sh/do-not-disrupt` annotation are removed, or their annotation becomes inactive (duration has elapsed). - All pods with the `karpenter.sh/do-not-disrupt` annotation have entered a [terminal phase](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase) (`Succeeded` or `Failed`). - The owning NodeClaim's [`terminationGracePeriod`]({{}}) has elapsed. -This is useful for pods that you want to run from start to finish without disruption. -Examples of pods that you might want to opt-out of disruption include an interactive game that you don't want to interrupt or a long batch job (such as you might have with machine learning) that would need to start over if it were interrupted. +#### Examples +This is useful for pods that you want to run from start to finish without disruption, or that need protection during critical startup phases. + +**Permanent protection** - useful for interactive games or long-running batch jobs: ```yaml apiVersion: apps/v1 kind: Deployment @@ -384,6 +415,18 @@ spec: karpenter.sh/do-not-disrupt: "true" ``` +**Duration-based protection** - useful for workloads with critical startup phases: +```yaml +apiVersion: apps/v1 +kind: Deployment +spec: + template: + metadata: + annotations: + # Protect for 30 minutes after pod starts running + karpenter.sh/do-not-disrupt: "30m" +``` + {{% alert title="Note" color="primary" %}} The `karpenter.sh/do-not-disrupt` annotation does **not** exclude nodes from the forceful disruption methods: [Expiration]({{}}), [Interruption]({{}}), [Node Repair](), and manual deletion (e.g. `kubectl delete node ...`). While both interruption and node repair have implicit upper-bounds on termination time, expiration and manual termination do not. diff --git a/website/content/en/preview/troubleshooting.md b/website/content/en/preview/troubleshooting.md index 714a032efce8..e73bbcc7fc35 100644 --- a/website/content/en/preview/troubleshooting.md +++ b/website/content/en/preview/troubleshooting.md @@ -482,9 +482,15 @@ Review what [disruptions are](https://kubernetes.io/docs/concepts/workloads/pods #### `karpenter.sh/do-not-disrupt` Annotation -If a pod exists with the annotation `karpenter.sh/do-not-disrupt: true` on a node, and a request is made to delete the node, Karpenter will not drain any pods from that node or otherwise try to delete the node. Nodes that have pods with a `do-not-disrupt` annotation are not considered for consolidation, though their unused capacity is considered for the purposes of running pods from other nodes which can be consolidated. +If a pod exists with an active `karpenter.sh/do-not-disrupt` annotation on a node, and a request is made to delete the node, Karpenter will not drain any pods from that node or otherwise try to delete the node. The annotation is considered "active" when: +- Set to `"true"` (permanent protection) +- Set to a valid duration (e.g., `"30m"`) and the pod has been running for less than that duration -If you want to terminate a node with a `do-not-disrupt` pod, you can simply remove the annotation and the deprovisioning process will continue. +Nodes that have pods with an active `do-not-disrupt` annotation are not considered for consolidation, though their unused capacity is considered for the purposes of running pods from other nodes which can be consolidated. + +If you want to terminate a node with a `do-not-disrupt` pod, you can either remove the annotation from the pod or wait for duration-based protection to expire naturally, and the deprovisioning process will continue. + +For more details on how this annotation works, see [Pod-Level Controls]({{}}) in the Disruption documentation. #### Scheduling Constraints (Consolidation Only)