Automation

The agent only reads your cluster. Automation is a separate, optional component that changes it, in the namespaces you allow. It installs with the agent's chart from version 0.4.1, runs with its own service account and permissions, and does nothing until an admin turns it on in the dashboard.

It does two things:

In-place resize. After an OOM kill or sustained CPU throttling, it gives the running container more memory or CPU without restarting it.
Auto-merge. It merges the LevelFour pull requests that fall in a class you opted a namespace into, such as fixes for OOM kills. Every other pull request waits for a person.

Where it acts

The component acts in a namespace only when all three of these hold:

Your install allows it. The chart value automation.namespaces lists the namespaces it may touch. The chart grants it permissions in those namespaces and nowhere else, so Kubernetes refuses anything outside the list.
The namespace is opted in to recommendations, with the levelfour.ai/recommendations=enabled label or the switch on the namespace page. Choose namespaces covers both.
An admin turned it on for that namespace on the cluster's Automation tab.

It never acts in kube-system, kube-public, kube-node-lease or the agent's own namespace. To keep LevelFour to one cluster and two of its namespaces, install the component on that cluster only, list the two namespaces, and turn them on in the dashboard.

The Automation tab marks a namespace that is turned on in the dashboard but missing from automation.namespaces, so you can tell why nothing happens there.

Turn it on

Add the component to your install. With Helm:

values.yaml

automation:
  enabled: true
  namespaces:
    - payments
    - checkout

helm upgrade levelfour-agent oci://public.ecr.aws/levelfour/charts/levelfour-agent \
  --version 0.4.1 \
  --namespace levelfour --reuse-values -f values.yaml

The chart refuses automation.enabled: true with an empty automation.namespaces. Setting automation.namespaces to ["*"] grants the same permissions across the whole cluster, so use it only when you mean it. Configure the agent lists every automation value.

Then open the cluster's Automation tab in the dashboard. It shows whether the component is connected and when it last reported. An admin turns automation on for the cluster, then for each namespace.

In-place resize

The component watches the pods in the namespaces where it may act and raises a container's resources on the running pod, through the Kubernetes pods/resize subresource. It needs Kubernetes 1.33 or later.

Trigger	What it raises	When
OOM kill	Memory	Two OOM kills of the same container within an hour, or one OOM kill and the container in `CrashLoopBackOff`
CPU throttling	CPU	LevelFour measured the container throttled at least a quarter of the time at its current limit, in the last three hours

It only ever raises. A smaller request or limit always arrives as a recommendation and a pull request.

How much. Each step adds 25% of memory or 50% of CPU, or the size LevelFour measured as needed, within the ceilings you set in the chart. A container never goes past double its workload's own spec. The component leaves a memory limit alone when it already sits at 1.25 times the container's peak or more: those kills come from a full node.

What it leaves alone. It skips:

BestEffort pods, and pods that set resources at the pod level.
Init containers that are not sidecars.
Jobs, and pods with no Deployment, StatefulSet or DaemonSet behind them.
Resources a Vertical Pod Autoscaler manages.
A resource whose resizePolicy is RestartContainer. Its fix comes as a pull request.

Quality of service. A resize never changes a pod's QoS class. A Guaranteed pod gets its request and limit raised together. A Burstable pod gets its limit raised. When an HPA or KEDA scales the workload on CPU utilization, only the CPU limit rises, so the autoscaler keeps scaling at the same real use.

JVM workloads. The component reads -Xmx and -XX:MaxRAMPercentage from the pod spec. When the heap is a share of the limit, it sizes the raise so the memory outside the heap gets the room it needs. A container that is OOM killed twice after a raise is marked Exhausted and handed to a pull request, with a note to set -Xmx.

Before every change, the component sends it to Kubernetes as a dry run, so the API server checks it against quotas, LimitRanges and the QoS rules. Then it asks LevelFour to approve it, and LevelFour checks the switches, the maintenance window and the limits again. Nothing changes without both.

What the resize reports

Status	Meaning
Applied	The container runs with the new resources
Infeasible	The node can never hold the new size. The component puts the old values back
Deferred, timed out	The node had no room for 10 minutes. The component puts the old values back, and the pull request moves the workload
Rejected	The API server refused the change, for example a ResourceQuota
Restart required	The container's `resizePolicy` needs a restart. The fix waits for its pull request
Exhausted	Two OOM kills after a raise. The fix waits for its pull request
Rollback failed	Putting the old values back was refused, for example a memory limit that Kubernetes 1.33 and 1.34 will not lower. The raise stays, and Stop and revert handles it later

A change LevelFour or the API server refused is not asked for again right away: the component waits for the next hour, the next day or the next maintenance window, depending on the limit it hit.

The pull request follows

A resize changes the running pod, not the Deployment in your repository, so the next rollout brings the old values back. For 72 hours after a resize, new pods of the same workload get the same resize, counted as one action. Meanwhile LevelFour publishes the change as a recommendation and opens its pull request, and auto-merge carries it when the namespace allows its class. The Automation tab flags a resize whose 72 hours ran out before the pull request merged.

Guardrails

Set them on the Automation tab. They apply to the whole cluster.

Guardrail	Default	What it does
Maintenance window	Any time	The days and hours, in UTC, when the component may act
Changes per hour	3	The most resizes in the cluster in one hour
Changes per workload per day	1	The most resizes of one workload in a day
Memory step	25%	How much one resize adds to memory
CPU step	50%	How much one resize adds to CPU

The component checks in with LevelFour every minute. When it hears nothing for five minutes, it stops changing anything until it hears back, reverts included.

Stop and revert

Two buttons on the Automation tab stop it:

Stop all automation refuses every new change at once, and the component confirms the stop within a minute. The tab shows when it confirmed.
Stop and revert also puts back the original spec on every pod the component resized, read from the workload's own template in the cluster. A value the template does not set, such as a limit a LimitRange added, goes back to what the pod had before the resize. A pod whose resources changed since the resize, or that is gone, is left alone. It stays available after automation is switched off.

A revert finishes once every pod is back, left to its next rollout, or left alone. Until then the component retries a pod whose revert failed, for example an eviction a PodDisruptionBudget refused. The Automation tab shows one line per revert with those counts.

On Kubernetes 1.33 and 1.34, a memory limit cannot go back down on a running pod. With automation.revert.allowEviction: true the component evicts the pod, so it restarts at its original size. Without it, the pod keeps the larger limit until its next rollout.

To remove every permission the component holds, set automation.enabled: false and upgrade the chart.

Auto-merge

Auto-merge is set per namespace on the Automation tab, one class at a time. A pull request merges on its own only when every change in it belongs to a class that namespace allows.

Class	Value	What it covers
Reliability fixes	`reliability_fix`	More memory after OOM kills, or more CPU after throttling, on a resource no VPA manages
Small request cuts	`small_request_decrease`	Requests cut by 25% or less, with limits untouched, from at least 14 days of data. Guaranteed pods, workloads that an HPA scales on CPU utilization, and containers that regressed recently never qualify
Autoscaler targets	`autoscaler_target`	An HPA or KEDA utilization target changed together with its request, keeping the scaling point where it was

Replica counts, idle workloads, NodePools, node groups and volume claims never merge on their own.

LevelFour merges a pull request only when all of these hold:

GitHub reports it mergeable, with every check on its last commit passed.
The base branch requires at least one status check. A branch with no required check never gets an automatic merge.
The maintenance window is open.

Your branch protection still applies. If it requires a review, the pull request waits for one. After a merge, LevelFour checks the change like any other. A regression turns that class off for the namespace and opens the revert pull request. After you merge covers the checks.

Who can do what

Action	Who
Turn automation on for the cluster or a namespace, change the guardrails, resume after a stop, allow an auto-merge class	Admins, signed in to the dashboard. API keys and AI assistants cannot
Stop, stop and revert, turn a namespace off	Admins and members
See the switches, the guardrails and every change	Everyone who can see the cluster

Each resize and each automatic merge records the person who allowed it: the admin who turned the namespace on, or who allowed that class. The saving's timeline shows that person under Approved by. When an admin loses that role or leaves the organization, what they turned on pauses until another admin turns it on again.

Where you see each change

The Automation tab lists every change, stop and setting change with its time, its namespace and workload, the values before and after, and who approved it. When a published recommendation covers the container, its Status & Logs step and its timeline show the resize too.

Permissions

The component has its own service account, separate from the agent's. In each namespace in automation.namespaces, a Role and RoleBinding grant:

API group	Resources	Verbs	Why
core	`pods`	`get`, `list`, `watch`	Find OOM kills and the pods to resize
core	`pods/resize`	`patch`	Resize a running container. This is the only write to a workload
`apps`	`replicasets`, `statefulsets`, `daemonsets`	`get`	Read the workload's own spec, which a revert restores
core	`pods/eviction`	`create`	Only with `automation.revert.allowEviction: true`, to revert a memory limit on Kubernetes 1.33 and 1.34

With automation.namespaces: ["*"], the same rules come in a ClusterRole for every namespace instead. A separate ClusterRole grants get on the kube-system namespace only, which names the cluster. The component holds no permission to change Deployments, StatefulSets or any other object, and none outside the listed namespaces. It talks to api.levelfour.ai on port 443, like the agent, with the same token.

On this page