Node optimization

The Nodes and Autoscaling tabs of a cluster page look at the nodes themselves: the pods that keep each one running, the NodePool settings that cost money or reliability, Spot interruptions, and how each pool buys its capacity. From the same data, LevelFour recommends NodePool and managed node group changes as pull requests, and smaller PersistentVolumeClaims as manual changes. Kubernetes recommendations covers approving and checking them.

Agent version

Agent 0.3.0 sends the data this page reads: every running and pending pod with its scheduling rules, Karpenter's events on nodes and NodePools, the cluster's StorageClasses and Karpenter's feature gates. Upgrade the agent to 0.3.0 to get them. While a cluster runs an older agent, each part that needs this data reads Needs agent 0.3.0.

Pods that block consolidation

Karpenter and the Cluster Autoscaler remove a node once its pods can run somewhere else. On the Nodes tab, Held by pods counts the pods that keep their node running, and the Pods that block consolidation table lists them a page at a time, with the columns Node, Pool, Pod, Workload, Why it stays and Karpenter says. Each reason comes with a sentence naming what holds the pod, such as the PodDisruptionBudget that allows no disruption. Hover a cell to read it.

Karpenter says quotes Karpenter's newest Unconsolidatable or DisruptionBlocked event for the node, with its message on hover. A node with no blocking pod still appears when Karpenter reported one of those events in the last 24 hours.

See details on a row opens the node drawer on its Pods section. There, Pods that block consolidation lists every pod on the node with each reason and its sentence in full, and Karpenter says lists up to five of Karpenter's events with their message and when each was last seen. The drawer's Spec section shows the node's Taints, whether it is Cordoned, and whether it is Opted out of disruption.

LevelFour works out the reasons from the agent's hourly report. The rules depend on who removes the node, which the table names under each node:

• Karpenter, for a node a NodePool launched. It carries the karpenter.sh/nodepool label.
• Cluster Autoscaler, whose rules LevelFour applies to every other node.

Both share the reasons in the first table below. Karpenter adds one of its own, and the Cluster Autoscaler four.

A reason that says the pod has nowhere else to go counts when no other ready, schedulable node can take the pod and, on a Karpenter node, no NodePool can launch one that would. DaemonSet pods and Pending pods never block a node.

On every node

Topology spread follows the scheduler's skew rule. The pod's replacement gets scheduled while its node still stands, so LevelFour counts the matching pods on every node the pod's node selector and affinity allow, tainted or not, with the pod's own node one pod short. A node is ruled out when moving there would leave it more than maxSkew pods above the emptiest of those nodes. matchLabelKeys add the pod's own label values to the selector.

LevelFour does not evaluate a namespace selector, so an anti-affinity term that carries one never makes the list.

On a Karpenter node, a NodePool that can launch a node for the pod clears Pod anti-affinity and Topology spread: Karpenter gives the pod a fresh node.

On Karpenter nodes

The Cluster Autoscaler ignores karpenter.sh/do-not-disrupt, so LevelFour lists it on Karpenter nodes alone. On other nodes, cluster-autoscaler.kubernetes.io/safe-to-evict: "false" does the same job.

On nodes outside a NodePool

A pod annotated cluster-autoscaler.kubernetes.io/safe-to-evict: "true" never gets Local storage, kube-system, no PDB or No controller.

NodePool review

Once a day, LevelFour reviews every Karpenter NodePool and every managed node group that has nodes. See details on a NodePool row of the Autoscaling tab, or on a node group row of the Nodes tab, opens the Node pool drawer. Its rail holds Settings, Findings, Fit, Balance, Spot and Nodes. Nodes lists the pool's nodes in the window a page at a time, with the columns Node, Instance type, Purchase option, Zone, Status, CPU requested and Memory requested.

The pool's saving reads under Monthly saving in the Outcome of the Fit panel. Published recommendations lists each recommendation published for the pool, with its status, as a Save ... a month link to the Savings page. Until LevelFour publishes one, it reads None published for this pool yet.

Findings is a table with the columns Finding, Field, Node and Impact. Impact reads Saving, Reliability or Notice. Hover a finding to read its detail. Most findings cover Karpenter NodePools. A managed node group gets Node shape off the pods and Spot with no fallback.

Oversized Spot nodes

Karpenter never replaces a Spot node with a smaller Spot node while its SpotToSpotConsolidation feature gate is off, and its Helm chart ships the gate off. It can still remove a Spot node whose pods fit on the other nodes, but it cannot swap one for a cheaper Spot node.

LevelFour raises Oversized Spot node kept for a Spot node when all of these hold:

• The pods on it request less than half of its allocatable CPU and less than half of its allocatable memory.
• It has run longer than the NodePool's consolidateAfter.
• A smaller type the NodePool allows would hold its pods.
• The gate is off in FEATURE_GATES on Karpenter's controller, or Karpenter reported Unconsolidatable on the node or its NodePool.

The finding names the node in Node, and its detail names the smaller type. When Karpenter reported the event, the detail quotes it after Karpenter says, such as Unconsolidatable: SpotToSpotConsolidation is disabled, can't replace a spot node with a spot node.

To turn the gate on, set it in Karpenter's Helm values and upgrade the Karpenter release:

settings:
  featureGates:
    spotToSpotConsolidation: true

With the gate on, Karpenter replaces one Spot node with one cheaper Spot node when the NodePool's requirements allow at least 15 instance types priced below the current one. That room to choose keeps Karpenter from stepping down to the cheapest type, which can carry a high interruption rate, and keeps the replacement about as available as the node it replaces. A pool narrowed to a few sizes stays unconsolidatable with the gate on, so widen its requirements to more families, generations or sizes as well. The 15-type rule does not apply when Karpenter replaces several Spot nodes with one. Spot consolidation in Karpenter's documentation has the details.

The chart passes the gate to the controller in its FEATURE_GATES variable, and the agent reports the new value in its next hourly report.

To keep the gate off, set spec.template.spec.expireAfter on the NodePool instead. The finding's detail names it after Or on the NodePool:. Karpenter replaces each node when it reaches that age, and launches the size its pods need at that point.

Spot interruptions

Karpenter writes an event on a node when EC2 warns it will reclaim the instance, and again as Karpenter drains and terminates it. Agent 0.3.0 watches events on nodes and NodePools and sends these reasons:

The agent also sends Karpenter's disruption events (Unconsolidatable, DisruptionBlocked, DisruptionTerminating), which feed Karpenter says. It drops every other event.

The node drawer's Reliability section has a Spot interruptions card: the three series by day, then the events a page at a time, with the columns Event, Message, Count and Seen. Seen gives the days the event covers, such as Oct 19 or Oct 19 to 23, and its exact first and last times on hover. The Node pool drawer's Spot panel shows the same card for the whole pool, with a Raised on column first that names the node or NodePool behind each event. In the Nodes tab's node table, a node that went away after an interruption reads Interrupted.

Karpenter writes the interruption events when its interruption handling is on, which needs the SQS queue set in settings.interruptionQueue of its Helm values. Without the queue, Karpenter writes none of them and the card stays empty.

Fitted node set

For each NodePool and managed node group, LevelFour works out the cheapest set of nodes its pods fit on. The Fit panel of the Node pool drawer shows two tables, Today and Fitted, each with the columns Instance type, Purchase option, Nodes and Monthly cost. Under them, Outcome gives the Monthly saving, or None: already fits when no cheaper set holds the pods, the Pods packed, how many were Kept on their node by a blocker, and what the sets were Priced from.

What goes into it:

• Pods: each pod's requests as it runs them. That is the larger of its containers' sum and its largest init container, plus the pod's overhead.
• Pods that stay: a pod held by a blocker keeps its node, and that node stays in the set. So does a pod held to one named node, or by an affinity term the fit cannot read.
• Where a pod may go: its node selector and required node affinity on instance labels (type, size, family, category, vCPU, memory, architecture, capacity type) must hold on any node it moves to. A required anti-affinity or spread on kubernetes.io/hostname keeps a workload's pods apart, and a hard spread on zones keeps a node in each zone.
• Room for DaemonSets: every node keeps room for DaemonSet and static pods, as much as the pool's busiest node gives them today.
• Allowed types: for a NodePool, the types its requirements allow once a size pin is lifted, on the architectures its nodes run today. A managed node group keeps its families, plus the ones Node shape off the pods points to, and keeps at least as many nodes as it runs now, because LevelFour cannot read the group's minimum. LevelFour leaves GPU types out.
• No larger nodes: no type in the fit is larger than the largest node the pool runs today. Per-core pricing makes the biggest node look cheapest, because it carries one set of DaemonSets, and one big node puts more pods at risk when it goes.
• What a node offers: the allocatable CPU and memory measured on your nodes of that type. For a type your cluster has not run, LevelFour applies the ratio between allocatable and capacity that the same family shows on your nodes, else the reservation the EKS AMI's kubelet makes. Pod slots come from each type's network interface limit.

LevelFour sorts the pods from largest to smallest and places each one on the first node with room (first-fit decreasing), for every allowed size. It keeps the cheapest set made of one type, or of one type plus a single node of another for the remainder, at the pool's current mix of Spot and On-Demand.

How the fit is priced

LevelFour prices both node sets from your bill, never from a price list:

• A type your cluster ran: the hourly rate your detailed cost data billed for it, Spot and On-Demand apart.
• A type your bill has not seen: its vCPUs and GiB of memory at the per-core and per-GiB rates your bill gives that capacity type. The rates come from the same family, else from a sibling family on your bill: same architecture, and the same category or generation. Priced from then reads Per-core and per-GiB rates on your bill, where it would otherwise read Instance rates on your bill. A family with neither gets no price.

When a node in either set has no price, the fit shows no money. After a change rolls out, realized savings measure it on your bill.

Spot, On-Demand and commitment balance

The Purchase balance card on the Nodes tab sits beside Capacity mix and Commitment coverage. It shows node hours that could be bought another way, from the last review's 7 days, added up over the cluster's pools. The Node pool drawer's Balance panel shows them for one pool.

• Could move to Spot: the pool's On-Demand node hours, and what running them on Spot would save a month: those hours at your bill's On-Demand rate minus its Spot rate for the same type or family. With no Spot rate on your bill, the card reads No Spot rate on the bill to price the move. The pool's interruptions per node a day sit beside it.
• Could be committed: the pool's steady On-Demand baseline, which is the fewest On-Demand vCPUs it ran in any hour of the window, less the share your Reserved Instances and Savings Plans already cover, with its On-Demand cost from your bill.

A pool's pods can take an interruption when each pod outside a DaemonSet has a controller to recreate it, and none is a static pod, opted out of disruption, annotated safe-to-evict: "false", keeping local storage or held by a PodDisruptionBudget at zero. The balance reads Spot fits or Stays off Spot, with the reason, and names a pod that holds the pool back.

View in Commitments opens the Savings Plans page of Commitments.

NodePool and node group recommendations

When the fitted set costs less than what a pool spends today, LevelFour recommends one change for the pool, carrying every field it changes and one saving. A pool change covers every workload on the pool, so it does not wait for namespace opt-in. LevelFour reviews each one before it reaches Savings, and the drawer's Published recommendations lists it from then on. A pool needs agent 0.3.0 and priced nodes to get one.

The saving is the pool's monthly spend, projected from its days on your bill, minus what the fitted set costs. It never exceeds the fit's own difference against today's nodes, and it shrinks in proportion when the pool spent less than today's nodes cost. Nodes the pool ran earlier in the month never count as saved.

Karpenter NodePool. The pull request edits spec.template.spec.requirements and spec.disruption:

• Size pin: a requirement that pins one size gains the smaller sizes the fit uses. Every size the pool runs or pins today stays, so no running node drifts out of the NodePool, and Karpenter moves down to smaller nodes where the pods allow. A pinned instance type keeps its family.
• Shape: when Node shape off the pods fires and your bill prices them, the instance family or category requirement gains the finding's category in the pool's own generation and processor, such as c7g beside m7g. LevelFour adds them only when the fit with them costs less.
• Spot: an On-Demand-only pool gains spot beside on-demand in karpenter.sh/capacity-type, with On-Demand kept as the fallback, when every pod can take an interruption and your bill has a Spot rate for its hours. The recommendation names the pool's interruption rate.
• Consolidation: consolidationPolicy: WhenEmpty becomes WhenEmptyOrUnderutilized when the pool left capacity unused. A consolidateAfter longer than 1h drops to 1h alongside any other change.

On a Spot pool with SpotToSpotConsolidation off, the smaller Spot nodes arrive only as Karpenter replaces nodes for another reason (an interruption, expireAfter or drift), or once you turn the gate on. The recommendation lists this as a risk and as a step. Oversized Spot nodes covers the gate.

LevelFour opens the pull request on the source that deploys the NodePool: the Argo CD application or Helm release its annotations name, or the manifest in your repository.

Managed node group. The pull request edits the Terraform that defines the group:

• instance_types moves to the size the fit picked, within the group's own families or the ones Node shape off the pods points to. The agent holds no AWS credentials, so LevelFour cannot read the group's minimum, maximum or desired size. The change keeps at least as many nodes as the group runs today, none larger than its largest.
• capacity_type becomes SPOT when every pod can take an interruption, your bill has a Spot rate, and at least two instance types each hold the group's pods. The change lists the two cheapest, and the saving prices the dearer of them, because EC2 picks between them by capacity. With fewer, the group stays On-Demand and gets Spot with no fallback.

A NodePool near its limit gets Limit nearly reached and never a recommendation.

PersistentVolumeClaim resize

An EBS volume never shrinks in place. A smaller claim means a new claim and a copy of the data, which a pull request cannot do, so LevelFour delivers a claim resize as Manual Setup: the recommendation lists the steps and you run them. They follow one pattern: create a claim at the new size in the same StorageClass, copy the data across, point the workload at the new claim, and delete the old one.

If the StorageClass has reclaimPolicy: Retain, deleting the old claim leaves its EBS volume behind, and you pay for it until you delete the volume too.

LevelFour recommends a smaller size for a claim when:

• its StorageClass uses a CSI driver, such as the EBS CSI driver (ebs.csi.aws.com), and sets allowVolumeExpansion: true;
• the claim is Bound and a workload mounts it, in a namespace labelled for recommendations;
• the resize saves at least 1 USD a month.

The new claim sits closer to its data, and expansion lets you grow it in place if the data outgrows it. In the volume drawer, the Resize card links the recommendation, or says why a claim gets none: its StorageClass uses an in-tree provisioner such as kubernetes.io/aws-ebs, or does not allow volume expansion. On the Storage tab, the Storage classes card shows each class's Provisioner, Volume type and Volume expansion.

The new size is the most data the claim held over the window, plus 30 days of its growth, times 1.3. LevelFour rounds it up to a whole GiB, and never goes under EBS's smallest volume for the type: 4 GiB for io1 and io2, 125 GiB for st1 and sc1, 1 GiB for the rest.

The saving comes from the claim's EBS storage lines on your bill (EBS:VolumeUsage*). IOPS and throughput charges stay out of it.