# Kubernetes recommendations

> How LevelFour sizes container requests, works with HPA, KEDA and VPA, checks every change after it merges, and measures what it saved on your bill.

Source: https://docs.levelfour.ai/connectors/kubernetes/recommendations


LevelFour recommends CPU and memory requests for the containers in the namespaces you opted in. Each recommendation arrives on the **Savings** screen like any other, and you approve it as a pull request on your own repository.

## How a request is sized

A recommendation starts from what the container used over the last two weeks, hour by hour, and adds headroom on top. CPU is sized from its p99, memory from its peak, and neither goes below what the container needs while it starts. Containers built on runtimes that hold memory up front, such as the JVM and .NET, get more headroom than others.

### Sizing policies

A sizing policy sets how much room a namespace or a workload keeps. Set it on the namespace page, or on one workload from its drawer. A workload's policy wins over its namespace's.

| Policy                 | CPU                                   | Memory                                 | Use it for                                                      |
| ---------------------- | ------------------------------------- | -------------------------------------- | --------------------------------------------------------------- |
| **Maximum savings**    | p95, with the least headroom          | Peak, with the least headroom          | Batch jobs and workloads that tolerate a little throttling      |
| **Balanced** (default) | p99, headroom from how spiky usage is | Peak, headroom from how spiky usage is | Most services                                                   |
| **Extra headroom**     | Balanced plus 15 points               | Balanced plus 15 points                | Latency-sensitive services and anything that recently regressed |

A change applies from the next scan. A recommendation already published keeps the sizing it had. Admins and members can change a policy; viewers see it.

## HPA, KEDA and VPA

**HPA and KEDA.** When an autoscaler scales on CPU or memory utilization, lowering the request would make it scale earlier. LevelFour changes the target with the request, in the same pull request, so the autoscaler keeps firing at the same real usage. When the new target would land outside 1 to 90 percent, LevelFour skips that resource. A target set as an average value is independent of the request, so the request changes alone.

**VPA.** A VPA in `Auto`, `Recreate`, `Initial` or `InPlaceOrRecreate` mode writes the requests it controls, so LevelFour leaves those alone. Its container policy decides which: a container set to `mode: Off`, or a resource left out of `controlledResources`, still gets a recommendation. A VPA in `Off` mode only recommends, so the workload gets a recommendation, with the VPA's own target shown beside it.

## After you merge

LevelFour watches every merged change in your cluster:

1. **Rolled out.** The cluster runs the new requests.
2. **Verified** or **Regressed.** A day later, LevelFour compares OOM kills, restarts and CPU throttling against the week before. More of any of them is a regression.
3. **Reverted.** After a regression, LevelFour opens a pull request that puts the old values back. Once the cluster runs them, the next recommendation for that container uses **Extra headroom** for 30 days, whatever its policy.

LevelFour marks a change as never applied when its pull request closes unmerged, or when nothing rolls it out within 30 days.

The workload drawer shows each step on its timeline, next to the pull request that opened and merged it.

## Realized savings

A week after a verified rollout, LevelFour compares what the container cost the week before and the week after, from your AWS bill, and records the difference as realized savings on **Savings Audit**. A scale-to-zero or a deleted volume saves what it cost before.

> **Note**
>
> A lower request cuts the container's share of the bill at once. Your node bill drops when nodes consolidate and fewer of them run, so it can trail the realized figure by a few days.

## Signals on the Kubernetes pages

**Suspected memory leak.** Memory climbs in a steady line between restarts, by at least 20 percent over at least a day. LevelFour holds back any memory cut for that container, because its peak only says how far it got before it restarted.

**CPU runaway.** CPU climbs in a steady line with no restart, by at least 50 percent over at least a day.

**GPU.** When Prometheus scrapes NVIDIA's DCGM exporter, each container shows GPU utilization and GPU memory used against what its GPUs hold. [Configuration](/connectors/kubernetes/configuration) has the setup.

**Health score.** The fleet table scores each cluster from 0 to 100. Efficiency, the share of container cost not wasted, counts 40 percent. Reliability, the share of workloads not at risk with a medium risk counting half a high one, counts 40 percent. Hygiene, the share of containers with CPU and memory requests and a memory limit, counts 20 percent. Hover the score to see each part.

**Compare two periods.** Turn on **Compare** next to the date range on a cluster page to see cost, efficiency and waste for two periods side by side.

## Alerts and the weekly summary

[Cost alerts](/cost-alerts) can watch a cluster, a namespace or a workload: its cost or its waste against a budget, or its reliability at a minimum risk, posting in your Slack channels. Switch on the **Kubernetes** group in [notification preferences](/notifications) to get a summary every Monday with each cluster's cost and unused spend, per namespace. It starts off.