# Node optimization > The pods that keep your nodes running, a review of each NodePool, Spot interruptions, the cheapest node set your pods fit on, how each pool buys capacity, and the node and volume claim changes LevelFour recommends. Source: https://docs.levelfour.ai/connectors/kubernetes/nodes The **Nodes** and **Autoscaling** tabs of a cluster page look at the nodes themselves: the pods that keep each one running, the NodePool settings that cost money or reliability, Spot interruptions, and how each pool buys its capacity. From the same data, LevelFour recommends NodePool and managed node group changes as pull requests, and smaller PersistentVolumeClaims as manual changes. [Kubernetes recommendations](/connectors/kubernetes/recommendations#nodepools-node-groups-and-claims) covers approving and checking them. ## Agent version Agent 0.3.0 sends the data this page reads: every running and pending pod with its scheduling rules, Karpenter's events on nodes and NodePools, the cluster's StorageClasses and Karpenter's feature gates. [Upgrade the agent](/connectors/kubernetes#upgrade) to 0.3.0 to get them. While a cluster runs an older agent, each part that needs this data reads **Needs agent 0.3.0**. | Feature | Needs agent 0.3.0 | | ----------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | [Pods that block consolidation](#pods-that-block-consolidation) | Yes | | [NodePool review](#nodepool-review) | For the **Node shape off the pods** and **Oversized Spot node kept** findings. The other findings read the NodePool spec that earlier agents send too | | [Spot interruptions](#spot-interruptions) | Yes. The history starts when 0.3.0 does | | [Fitted node set](#fitted-node-set) | Yes | | [Spot, On-Demand and commitment balance](#spot-on-demand-and-commitment-balance) | For **Could move to Spot**, which checks whether each pod can take an interruption. **Could be committed** comes from your bill | | [NodePool and node group recommendations](#nodepool-and-node-group-recommendations) | Yes | | [PersistentVolumeClaim resize](#persistentvolumeclaim-resize) | Yes | > **Note** > > LevelFour reads NodePools from Karpenter's `karpenter.sh/v1` API. A cluster on the older `v1beta1` API reports no NodePools, so the NodePool parts of this page stay empty for it. ## Pods that block consolidation Karpenter and the Cluster Autoscaler remove a node once its pods can run somewhere else. On the **Nodes** tab, **Held by pods** counts the pods that keep their node running, and the **Pods that block consolidation** table lists them a page at a time, with the columns **Node**, **Pool**, **Pod**, **Workload**, **Why it stays** and **Karpenter says**. Each reason comes with a sentence naming what holds the pod, such as the PodDisruptionBudget that allows no disruption. Hover a cell to read it. **Karpenter says** quotes Karpenter's newest `Unconsolidatable` or `DisruptionBlocked` event for the node, with its message on hover. A node with no blocking pod still appears when Karpenter reported one of those events in the last 24 hours. **See details** on a row opens the node drawer on its **Pods** section. There, **Pods that block consolidation** lists every pod on the node with each reason and its sentence in full, and **Karpenter says** lists up to five of Karpenter's events with their message and when each was last seen. The drawer's **Spec** section shows the node's **Taints**, whether it is **Cordoned**, and whether it is **Opted out of disruption**. LevelFour works out the reasons from the agent's hourly report. The rules depend on who removes the node, which the table names under each node: * **Karpenter**, for a node a NodePool launched. It carries the `karpenter.sh/nodepool` label. * **Cluster Autoscaler**, whose rules LevelFour applies to every other node. Both share the reasons in the first table below. Karpenter adds one of its own, and the Cluster Autoscaler four. A reason that says the pod has nowhere else to go counts when no other ready, schedulable node can take the pod and, on a Karpenter node, no NodePool can launch one that would. DaemonSet pods and Pending pods never block a node. ### On every node | Reason | When LevelFour lists it | What to do | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **PodDisruptionBudget** | A PodDisruptionBudget in the pod's namespace selects it and allows no disruption right now (`status.disruptionsAllowed` is 0) | Run more replicas than the budget's `minAvailable`, or set `maxUnavailable` to 1 or more | | **Node affinity** | Its node selector or required node affinity matches no other node, and no NodePool can launch one | Widen the selector, or move the rule to `preferredDuringSchedulingIgnoredDuringExecution`. On Karpenter, let a NodePool launch nodes with the labels the pod asks for | | **Taints** | Every other node it could run on has a taint it does not tolerate, and no NodePool launches a node it tolerates | Add the toleration to the pod, or remove the taint from the nodes or from the NodePool template | | **Pod anti-affinity** | Its required pod anti-affinity on `kubernetes.io/hostname` rules out every other node, because each one already runs a matching pod | Make the anti-affinity preferred instead of required | | **Topology spread** | A spread constraint on `kubernetes.io/hostname` with `whenUnsatisfiable: DoNotSchedule` rules out every other node: moving the pod there would exceed its `maxSkew` | Raise `maxSkew`, or set `whenUnsatisfiable: ScheduleAnyway` | | **Static pod** | The kubelet runs the pod from a file on the node | Nothing. Neither autoscaler waits for a static pod, and it goes away with its node | **Topology spread** follows the scheduler's skew rule. The pod's replacement gets scheduled while its node still stands, so LevelFour counts the matching pods on every node the pod's node selector and affinity allow, tainted or not, with the pod's own node one pod short. A node is ruled out when moving there would leave it more than `maxSkew` pods above the emptiest of those nodes. `matchLabelKeys` add the pod's own label values to the selector. LevelFour does not evaluate a namespace selector, so an anti-affinity term that carries one never makes the list. On a Karpenter node, a NodePool that can launch a node for the pod clears **Pod anti-affinity** and **Topology spread**: Karpenter gives the pod a fresh node. ### On Karpenter nodes | Reason | When LevelFour lists it | What to do | | --------------------------- | ---------------------------------------------------------------------- | ----------------------------------------------------------------- | | **Opted out of disruption** | The pod or its node is annotated `karpenter.sh/do-not-disrupt: "true"` | Remove the annotation from a pod that can restart on another node | The Cluster Autoscaler ignores `karpenter.sh/do-not-disrupt`, so LevelFour lists it on Karpenter nodes alone. On other nodes, `cluster-autoscaler.kubernetes.io/safe-to-evict: "false"` does the same job. ### On nodes outside a NodePool | Reason | When LevelFour lists it | What to do | | ----------------------- | ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | **Local storage** | The pod keeps data on the node: an emptyDir on disk, a hostPath or a local PersistentVolume | If the data can go, annotate the pod `cluster-autoscaler.kubernetes.io/safe-to-evict: "true"`. Otherwise move the data to a PersistentVolumeClaim | | **Not safe to evict** | The pod is annotated `cluster-autoscaler.kubernetes.io/safe-to-evict: "false"` | Remove the annotation from a pod that can restart on another node | | **kube-system, no PDB** | The pod runs in `kube-system`, belongs to no DaemonSet, and no PodDisruptionBudget covers it | Add a PodDisruptionBudget that selects it | | **No controller** | No controller owns the pod, so nothing recreates it on another node. A static pod gets **Static pod** instead | Run it from a Deployment, StatefulSet or Job. If it can go, annotate it `cluster-autoscaler.kubernetes.io/safe-to-evict: "true"` | A pod annotated `cluster-autoscaler.kubernetes.io/safe-to-evict: "true"` never gets **Local storage**, **kube-system, no PDB** or **No controller**. ## NodePool review Once a day, LevelFour reviews every Karpenter NodePool and every managed node group that has nodes. **See details** on a NodePool row of the **Autoscaling** tab, or on a node group row of the **Nodes** tab, opens the **Node pool** drawer. Its rail holds **Settings**, **Findings**, **Fit**, **Balance**, **Spot** and **Nodes**. **Nodes** lists the pool's nodes in the window a page at a time, with the columns **Node**, **Instance type**, **Purchase option**, **Zone**, **Status**, **CPU requested** and **Memory requested**. The pool's saving reads under **Monthly saving** in the **Outcome** of the [Fit](#fitted-node-set) panel. **Published recommendations** lists each recommendation published for the pool, with its status, as a **Save ... a month** link to the **Savings** page. Until LevelFour publishes one, it reads **None published for this pool yet.** **Findings** is a table with the columns **Finding**, **Field**, **Node** and **Impact**. **Impact** reads **Saving**, **Reliability** or **Notice**. Hover a finding to read its detail. Most findings cover Karpenter NodePools. A managed node group gets **Node shape off the pods** and **Spot with no fallback**. | Finding | Field | When LevelFour raises it | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Consolidates only empty nodes** | `spec.disruption.consolidationPolicy` | The policy is `WhenEmpty` and the pool's nodes left capacity unused that your bill charged for. `WhenEmptyOrUnderutilized` lets Karpenter replace underused nodes too | | **Waits long to consolidate** | `spec.disruption.consolidateAfter` | The delay is over `1h`, so Karpenter never gives back capacity freed for less than that | | **Previous generation only** | `spec.template.spec.requirements[...]`, on the instance type, category, family or generation key the NodePool sets | Every family the NodePool allows is three or more generations behind the newest of its kind | | **One pinned instance type** | `spec.template.spec.requirements[node.kubernetes.io/instance-type]`, or the `karpenter.k8s.aws/instance-size` or `karpenter.k8s.aws/instance-cpu` key that pins it | The requirements leave one size to launch, so Karpenter cannot pick a smaller node when the pods need less | | **One instance family** | `spec.template.spec.requirements[karpenter.k8s.aws/instance-family]` | Every node comes from one family. On Spot, that leaves few capacity pools to fall back on when EC2 reclaims one | | **Node shape off the pods** | `spec.template.spec.requirements[karpenter.k8s.aws/instance-family]` or `[karpenter.k8s.aws/instance-category]`. On a node group, `instance_types` | The memory per vCPU the pods ask for is at least double, or at most half, what the nodes offer. The finding names the closest category: `c` at 2 GiB per vCPU, `m` at 4, `r` at 8 | | **Limit nearly reached** | `spec.limits.cpu` or `spec.limits.memory` | The pool's nodes hold 80% or more of the limit, so Karpenter soon stops launching nodes for pending pods | | **Spot with no fallback** | `spec.template.spec.requirements[karpenter.sh/capacity-type]`. On a node group, `instance_types` | The NodePool launches Spot only, so pods wait whenever EC2 has no Spot capacity for its types, and allowing `on-demand` as well lets Karpenter fall back. A node group gets it when Spot would save money but fewer than two instance types of its families hold its pods, so it stays On-Demand | | **Oversized Spot node kept** | `settings.featureGates.spotToSpotConsolidation` in Karpenter's Helm values | A Spot node runs half empty and Karpenter cannot replace it. See [Oversized Spot nodes](#oversized-spot-nodes) | ### Oversized Spot nodes Karpenter never replaces a Spot node with a smaller Spot node while its `SpotToSpotConsolidation` feature gate is off, and its Helm chart ships the gate off. It can still remove a Spot node whose pods fit on the other nodes, but it cannot swap one for a cheaper Spot node. LevelFour raises **Oversized Spot node kept** for a Spot node when all of these hold: * The pods on it request less than half of its allocatable CPU and less than half of its allocatable memory. * It has run longer than the NodePool's `consolidateAfter`. * A smaller type the NodePool allows would hold its pods. * The gate is off in `FEATURE_GATES` on Karpenter's controller, or Karpenter reported `Unconsolidatable` on the node or its NodePool. The finding names the node in **Node**, and its detail names the smaller type. When Karpenter reported the event, the detail quotes it after **Karpenter says**, such as `Unconsolidatable: SpotToSpotConsolidation is disabled, can't replace a spot node with a spot node`. To turn the gate on, set it in Karpenter's Helm values and upgrade the Karpenter release: ```yaml title="karpenter-values.yaml" settings: featureGates: spotToSpotConsolidation: true ``` With the gate on, Karpenter replaces one Spot node with one cheaper Spot node when the NodePool's requirements allow at least 15 instance types priced below the current one. That room to choose keeps Karpenter from stepping down to the cheapest type, which can carry a high interruption rate, and keeps the replacement about as available as the node it replaces. A pool narrowed to a few sizes stays unconsolidatable with the gate on, so widen its requirements to more families, generations or sizes as well. The 15-type rule does not apply when Karpenter replaces several Spot nodes with one. [Spot consolidation](https://karpenter.sh/docs/concepts/disruption/#spot-consolidation) in Karpenter's documentation has the details. The chart passes the gate to the controller in its `FEATURE_GATES` variable, and the agent reports the new value in its next hourly report. To keep the gate off, set `spec.template.spec.expireAfter` on the NodePool instead. The finding's detail names it after **Or on the NodePool:**. Karpenter replaces each node when it reaches that age, and launches the size its pods need at that point. ## Spot interruptions Karpenter writes an event on a node when EC2 warns it will reclaim the instance, and again as Karpenter drains and terminates it. Agent 0.3.0 watches events on nodes and NodePools and sends these reasons: | Series | Karpenter event reasons | | --------------------- | ------------------------------------------------------------------------------------------- | | **Interrupted** | `SpotInterrupted`, `CapacityReservationInstanceInterrupted` | | **Rebalance warning** | `SpotRebalanceRecommendation` | | **Terminating** | `TerminatingOnInterruption`, `InstanceStopping`, `InstanceTerminating`, `InstanceUnhealthy` | The agent also sends Karpenter's disruption events (`Unconsolidatable`, `DisruptionBlocked`, `DisruptionTerminating`), which feed **Karpenter says**. It drops every other event. The node drawer's **Reliability** section has a **Spot interruptions** card: the three series by day, then the events a page at a time, with the columns **Event**, **Message**, **Count** and **Seen**. **Seen** gives the days the event covers, such as `Oct 19` or `Oct 19 to 23`, and its exact first and last times on hover. The **Node pool** drawer's **Spot** panel shows the same card for the whole pool, with a **Raised on** column first that names the node or NodePool behind each event. In the **Nodes** tab's node table, a node that went away after an interruption reads **Interrupted**. > **Note** > > Kubernetes keeps an event for about an hour, so LevelFour has no interruptions from before your agent ran 0.3.0. The history starts at the upgrade. Karpenter writes the interruption events when its interruption handling is on, which needs the SQS queue set in `settings.interruptionQueue` of its Helm values. Without the queue, Karpenter writes none of them and the card stays empty. ## Fitted node set For each NodePool and managed node group, LevelFour works out the cheapest set of nodes its pods fit on. The **Fit** panel of the **Node pool** drawer shows two tables, **Today** and **Fitted**, each with the columns **Instance type**, **Purchase option**, **Nodes** and **Monthly cost**. Under them, **Outcome** gives the **Monthly saving**, or **None: already fits** when no cheaper set holds the pods, the **Pods packed**, how many were **Kept on their node by a blocker**, and what the sets were **Priced from**. What goes into it: * **Pods:** each pod's requests as it runs them. That is the larger of its containers' sum and its largest init container, plus the pod's overhead. * **Pods that stay:** a pod [held by a blocker](#pods-that-block-consolidation) keeps its node, and that node stays in the set. So does a pod held to one named node, or by an affinity term the fit cannot read. * **Where a pod may go:** its node selector and required node affinity on instance labels (type, size, family, category, vCPU, memory, architecture, capacity type) must hold on any node it moves to. A required anti-affinity or spread on `kubernetes.io/hostname` keeps a workload's pods apart, and a hard spread on zones keeps a node in each zone. * **Room for DaemonSets:** every node keeps room for DaemonSet and static pods, as much as the pool's busiest node gives them today. * **Allowed types:** for a NodePool, the types its requirements allow once a size pin is lifted, on the architectures its nodes run today. A managed node group keeps its families, plus the ones **Node shape off the pods** points to, and keeps at least as many nodes as it runs now, because LevelFour cannot read the group's minimum. LevelFour leaves GPU types out. * **No larger nodes:** no type in the fit is larger than the largest node the pool runs today. Per-core pricing makes the biggest node look cheapest, because it carries one set of DaemonSets, and one big node puts more pods at risk when it goes. * **What a node offers:** the allocatable CPU and memory measured on your nodes of that type. For a type your cluster has not run, LevelFour applies the ratio between allocatable and capacity that the same family shows on your nodes, else the reservation the EKS AMI's kubelet makes. Pod slots come from each type's network interface limit. LevelFour sorts the pods from largest to smallest and places each one on the first node with room (first-fit decreasing), for every allowed size. It keeps the cheapest set made of one type, or of one type plus a single node of another for the remainder, at the pool's current mix of Spot and On-Demand. ### How the fit is priced LevelFour prices both node sets from your bill, never from a price list: * **A type your cluster ran:** the hourly rate your [detailed cost data](/connectors/aws/detailed-cost-data) billed for it, Spot and On-Demand apart. * **A type your bill has not seen:** its vCPUs and GiB of memory at the per-core and per-GiB rates your bill gives that capacity type. The rates come from the same family, else from a sibling family on your bill: same architecture, and the same category or generation. **Priced from** then reads **Per-core and per-GiB rates on your bill**, where it would otherwise read **Instance rates on your bill**. A family with neither gets no price. When a node in either set has no price, the fit shows no money. After a change rolls out, [realized savings](/connectors/kubernetes/recommendations#realized-savings) measure it on your bill. ## Spot, On-Demand and commitment balance The **Purchase balance** card on the **Nodes** tab sits beside **Capacity mix** and **Commitment coverage**. It shows node hours that could be bought another way, from the last review's 7 days, added up over the cluster's pools. The **Node pool** drawer's **Balance** panel shows them for one pool. * **Could move to Spot:** the pool's On-Demand node hours, and what running them on Spot would save a month: those hours at your bill's On-Demand rate minus its Spot rate for the same type or family. With no Spot rate on your bill, the card reads **No Spot rate on the bill to price the move**. The pool's interruptions per node a day sit beside it. * **Could be committed:** the pool's steady On-Demand baseline, which is the fewest On-Demand vCPUs it ran in any hour of the window, less the share your Reserved Instances and Savings Plans already cover, with its On-Demand cost from your bill. A pool's pods can take an interruption when each pod outside a DaemonSet has a controller to recreate it, and none is a static pod, opted out of disruption, annotated `safe-to-evict: "false"`, keeping local storage or held by a PodDisruptionBudget at zero. The balance reads **Spot fits** or **Stays off Spot**, with the reason, and names a pod that holds the pool back. **View in Commitments** opens the Savings Plans page of [Commitments](/commitments). ## NodePool and node group recommendations When the fitted set costs less than what a pool spends today, LevelFour recommends one change for the pool, carrying every field it changes and one saving. A pool change covers every workload on the pool, so it does not wait for namespace opt-in. LevelFour reviews each one before it reaches **Savings**, and the drawer's **Published recommendations** lists it from then on. A pool needs agent 0.3.0 and priced nodes to get one. The saving is the pool's monthly spend, projected from its days on your bill, minus what the fitted set costs. It never exceeds the fit's own difference against today's nodes, and it shrinks in proportion when the pool spent less than today's nodes cost. Nodes the pool ran earlier in the month never count as saved. **Karpenter NodePool.** The pull request edits `spec.template.spec.requirements` and `spec.disruption`: * **Size pin:** a requirement that pins one size gains the smaller sizes the fit uses. Every size the pool runs or pins today stays, so no running node drifts out of the NodePool, and Karpenter moves down to smaller nodes where the pods allow. A pinned instance type keeps its family. * **Shape:** when **Node shape off the pods** fires and your bill prices them, the instance family or category requirement gains the finding's category in the pool's own generation and processor, such as `c7g` beside `m7g`. LevelFour adds them only when the fit with them costs less. * **Spot:** an On-Demand-only pool gains `spot` beside `on-demand` in `karpenter.sh/capacity-type`, with On-Demand kept as the fallback, when every pod can take an interruption and your bill has a Spot rate for its hours. The recommendation names the pool's interruption rate. * **Consolidation:** `consolidationPolicy: WhenEmpty` becomes `WhenEmptyOrUnderutilized` when the pool left capacity unused. A `consolidateAfter` longer than `1h` drops to `1h` alongside any other change. On a Spot pool with `SpotToSpotConsolidation` off, the smaller Spot nodes arrive only as Karpenter replaces nodes for another reason (an interruption, `expireAfter` or drift), or once you turn the gate on. The recommendation lists this as a risk and as a step. [Oversized Spot nodes](#oversized-spot-nodes) covers the gate. LevelFour opens the pull request on the source that deploys the NodePool: the Argo CD application or Helm release its annotations name, or the manifest in your repository. **Managed node group.** The pull request edits the Terraform that defines the group: * **`instance_types`** moves to the size the fit picked, within the group's own families or the ones **Node shape off the pods** points to. The agent holds no AWS credentials, so LevelFour cannot read the group's minimum, maximum or desired size. The change keeps at least as many nodes as the group runs today, none larger than its largest. * **`capacity_type`** becomes `SPOT` when every pod can take an interruption, your bill has a Spot rate, and at least two instance types each hold the group's pods. The change lists the two cheapest, and the saving prices the dearer of them, because EC2 picks between them by capacity. With fewer, the group stays On-Demand and gets **Spot with no fallback**. A NodePool near its limit gets **Limit nearly reached** and never a recommendation. ## PersistentVolumeClaim resize An EBS volume never shrinks in place. A smaller claim means a new claim and a copy of the data, which a pull request cannot do, so LevelFour delivers a claim resize as **Manual Setup**: the recommendation lists the steps and you run them. They follow one pattern: create a claim at the new size in the same StorageClass, copy the data across, point the workload at the new claim, and delete the old one. If the StorageClass has `reclaimPolicy: Retain`, deleting the old claim leaves its EBS volume behind, and you pay for it until you delete the volume too. LevelFour recommends a smaller size for a claim when: * its StorageClass uses a CSI driver, such as the EBS CSI driver (`ebs.csi.aws.com`), and sets `allowVolumeExpansion: true`; * the claim is Bound and a workload mounts it, in a namespace [labelled for recommendations](/connectors/kubernetes#choose-namespaces); * the resize saves at least 1 USD a month. The new claim sits closer to its data, and expansion lets you grow it in place if the data outgrows it. In the volume drawer, the **Resize** card links the recommendation, or says why a claim gets none: its StorageClass uses an in-tree provisioner such as `kubernetes.io/aws-ebs`, or does not allow volume expansion. On the **Storage** tab, the **Storage classes** card shows each class's **Provisioner**, **Volume type** and **Volume expansion**. The new size is the most data the claim held over the window, plus 30 days of its growth, times 1.3. LevelFour rounds it up to a whole GiB, and never goes under EBS's smallest volume for the type: 4 GiB for `io1` and `io2`, 125 GiB for `st1` and `sc1`, 1 GiB for the rest. The saving comes from the claim's EBS storage lines on your bill (`EBS:VolumeUsage*`). IOPS and throughput charges stay out of it.