# Automation > The optional component that resizes running pods after an OOM kill or CPU throttling, merges the pull requests you allow, and stops or reverts everything with one button. Source: https://docs.levelfour.ai/connectors/kubernetes/automation The agent only reads your cluster. Automation is a separate, optional component that changes it, in the namespaces you allow. It installs with the agent's chart from version 0.4.1, runs with its own service account and permissions, and does nothing until an admin turns it on in the dashboard. It does two things: * **In-place resize.** After an OOM kill or sustained CPU throttling, it gives the running container more memory or CPU without restarting it. * **Auto-merge.** It merges the LevelFour pull requests that fall in a class you opted a namespace into, such as fixes for OOM kills. Every other pull request waits for a person. ## Where it acts The component acts in a namespace only when all three of these hold: 1. **Your install allows it.** The chart value `automation.namespaces` lists the namespaces it may touch. The chart grants it permissions in those namespaces and nowhere else, so Kubernetes refuses anything outside the list. 2. **The namespace is opted in to recommendations**, with the `levelfour.ai/recommendations=enabled` label or the switch on the namespace page. [Choose namespaces](/connectors/kubernetes#choose-namespaces) covers both. 3. **An admin turned it on** for that namespace on the cluster's **Automation** tab. It never acts in `kube-system`, `kube-public`, `kube-node-lease` or the agent's own namespace. To keep LevelFour to one cluster and two of its namespaces, install the component on that cluster only, list the two namespaces, and turn them on in the dashboard. The **Automation** tab marks a namespace that is turned on in the dashboard but missing from `automation.namespaces`, so you can tell why nothing happens there. ## Turn it on Add the component to your install. With Helm: ```yaml title="values.yaml" automation: enabled: true namespaces: - payments - checkout ``` ```bash helm upgrade levelfour-agent oci://public.ecr.aws/levelfour/charts/levelfour-agent \ --version 0.4.1 \ --namespace levelfour --reuse-values -f values.yaml ``` The chart refuses `automation.enabled: true` with an empty `automation.namespaces`. Setting `automation.namespaces` to `["*"]` grants the same permissions across the whole cluster, so use it only when you mean it. [Configure the agent](/connectors/kubernetes/configuration#chart-values) lists every `automation` value. Then open the cluster's **Automation** tab in the dashboard. It shows whether the component is connected and when it last reported. An admin turns automation on for the cluster, then for each namespace. ## In-place resize The component watches the pods in the namespaces where it may act and raises a container's resources on the running pod, through the Kubernetes `pods/resize` subresource. It needs Kubernetes 1.33 or later. | Trigger | What it raises | When | | ------------------ | -------------- | ----------------------------------------------------------------------------------------------------------------------- | | **OOM kill** | Memory | Two OOM kills of the same container within an hour, or one OOM kill and the container in `CrashLoopBackOff` | | **CPU throttling** | CPU | LevelFour measured the container throttled at least a quarter of the time at its current limit, in the last three hours | It only ever raises. A smaller request or limit always arrives as a recommendation and a pull request. **How much.** Each step adds 25% of memory or 50% of CPU, or the size LevelFour measured as needed, within the ceilings you set in the chart. A container never goes past double its workload's own spec. The component leaves a memory limit alone when it already sits at 1.25 times the container's peak or more: those kills come from a full node. **What it leaves alone.** It skips: * BestEffort pods, and pods that set resources at the pod level. * Init containers that are not sidecars. * Jobs, and pods with no Deployment, StatefulSet or DaemonSet behind them. * Resources a Vertical Pod Autoscaler manages. * A resource whose `resizePolicy` is `RestartContainer`. Its fix comes as a pull request. **Quality of service.** A resize never changes a pod's QoS class. A Guaranteed pod gets its request and limit raised together. A Burstable pod gets its limit raised. When an HPA or KEDA scales the workload on CPU utilization, only the CPU limit rises, so the autoscaler keeps scaling at the same real use. **JVM workloads.** The component reads `-Xmx` and `-XX:MaxRAMPercentage` from the pod spec. When the heap is a share of the limit, it sizes the raise so the memory outside the heap gets the room it needs. A container that is OOM killed twice after a raise is marked **Exhausted** and handed to a pull request, with a note to set `-Xmx`. **Before every change**, the component sends it to Kubernetes as a dry run, so the API server checks it against quotas, LimitRanges and the QoS rules. Then it asks LevelFour to approve it, and LevelFour checks the switches, the maintenance window and the limits again. Nothing changes without both. ### What the resize reports | Status | Meaning | | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Applied** | The container runs with the new resources | | **Infeasible** | The node can never hold the new size. The component puts the old values back | | **Deferred, timed out** | The node had no room for 10 minutes. The component puts the old values back, and the pull request moves the workload | | **Rejected** | The API server refused the change, for example a ResourceQuota | | **Restart required** | The container's `resizePolicy` needs a restart. The fix waits for its pull request | | **Exhausted** | Two OOM kills after a raise. The fix waits for its pull request | | **Rollback failed** | Putting the old values back was refused, for example a memory limit that Kubernetes 1.33 and 1.34 will not lower. The raise stays, and **Stop and revert** handles it later | A change LevelFour or the API server refused is not asked for again right away: the component waits for the next hour, the next day or the next maintenance window, depending on the limit it hit. ### The pull request follows A resize changes the running pod, not the Deployment in your repository, so the next rollout brings the old values back. For 72 hours after a resize, new pods of the same workload get the same resize, counted as one action. Meanwhile LevelFour publishes the change as a recommendation and opens its pull request, and auto-merge carries it when the namespace allows its class. The **Automation** tab flags a resize whose 72 hours ran out before the pull request merged. ## Guardrails Set them on the **Automation** tab. They apply to the whole cluster. | Guardrail | Default | What it does | | -------------------------------- | -------- | ------------------------------------------------------ | | **Maintenance window** | Any time | The days and hours, in UTC, when the component may act | | **Changes per hour** | 3 | The most resizes in the cluster in one hour | | **Changes per workload per day** | 1 | The most resizes of one workload in a day | | **Memory step** | 25% | How much one resize adds to memory | | **CPU step** | 50% | How much one resize adds to CPU | The component checks in with LevelFour every minute. When it hears nothing for five minutes, it stops changing anything until it hears back, reverts included. ## Stop and revert Two buttons on the **Automation** tab stop it: * **Stop all automation** refuses every new change at once, and the component confirms the stop within a minute. The tab shows when it confirmed. * **Stop and revert** also puts back the original spec on every pod the component resized, read from the workload's own template in the cluster. A value the template does not set, such as a limit a LimitRange added, goes back to what the pod had before the resize. A pod whose resources changed since the resize, or that is gone, is left alone. It stays available after automation is switched off. A revert finishes once every pod is back, left to its next rollout, or left alone. Until then the component retries a pod whose revert failed, for example an eviction a PodDisruptionBudget refused. The **Automation** tab shows one line per revert with those counts. On Kubernetes 1.33 and 1.34, a memory limit cannot go back down on a running pod. With `automation.revert.allowEviction: true` the component evicts the pod, so it restarts at its original size. Without it, the pod keeps the larger limit until its next rollout. To remove every permission the component holds, set `automation.enabled: false` and upgrade the chart. ## Auto-merge Auto-merge is set per namespace on the **Automation** tab, one class at a time. A pull request merges on its own only when every change in it belongs to a class that namespace allows. | Class | Value | What it covers | | ---------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Reliability fixes** | `reliability_fix` | More memory after OOM kills, or more CPU after throttling, on a resource no VPA manages | | **Small request cuts** | `small_request_decrease` | Requests cut by 25% or less, with limits untouched, from at least 14 days of data. Guaranteed pods, workloads that an HPA scales on CPU utilization, and containers that regressed recently never qualify | | **Autoscaler targets** | `autoscaler_target` | An HPA or KEDA utilization target changed together with its request, keeping the scaling point where it was | Replica counts, idle workloads, NodePools, node groups and volume claims never merge on their own. LevelFour merges a pull request only when all of these hold: * GitHub reports it mergeable, with every check on its last commit passed. * The base branch requires at least one status check. A branch with no required check never gets an automatic merge. * The maintenance window is open. Your branch protection still applies. If it requires a review, the pull request waits for one. After a merge, LevelFour checks the change like any other. A regression turns that class off for the namespace and opens the revert pull request. [After you merge](/connectors/kubernetes/recommendations#after-you-merge) covers the checks. ## Who can do what | Action | Who | | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------- | | Turn automation on for the cluster or a namespace, change the guardrails, resume after a stop, allow an auto-merge class | Admins, signed in to the dashboard. API keys and AI assistants cannot | | Stop, stop and revert, turn a namespace off | Admins and members | | See the switches, the guardrails and every change | Everyone who can see the cluster | Each resize and each automatic merge records the person who allowed it: the admin who turned the namespace on, or who allowed that class. The saving's timeline shows that person under **Approved by**. When an admin loses that role or leaves the organization, what they turned on pauses until another admin turns it on again. ## Where you see each change The **Automation** tab lists every change, stop and setting change with its time, its namespace and workload, the values before and after, and who approved it. When a published recommendation covers the container, its **Status & Logs** step and its timeline show the resize too. ## Permissions The component has its own service account, separate from the agent's. In each namespace in `automation.namespaces`, a Role and RoleBinding grant: | API group | Resources | Verbs | Why | | --------- | ------------------------------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------- | | core | `pods` | `get`, `list`, `watch` | Find OOM kills and the pods to resize | | core | `pods/resize` | `patch` | Resize a running container. This is the only write to a workload | | `apps` | `replicasets`, `statefulsets`, `daemonsets` | `get` | Read the workload's own spec, which a revert restores | | core | `pods/eviction` | `create` | Only with `automation.revert.allowEviction: true`, to revert a memory limit on Kubernetes 1.33 and 1.34 | With `automation.namespaces: ["*"]`, the same rules come in a ClusterRole for every namespace instead. A separate ClusterRole grants `get` on the `kube-system` namespace only, which names the cluster. The component holds no permission to change Deployments, StatefulSets or any other object, and none outside the listed namespaces. It talks to `api.levelfour.ai` on port 443, like the agent, with the same token.