Connect a cluster

The LevelFour agent runs in your cluster as a single pod. It reads how each workload is configured and how much CPU and memory it uses, and sends that to LevelFour over HTTPS. LevelFour prices it against your cloud bill, or against hourly prices you set for a cluster with no cloud bill, so you see cost per namespace, workload, container and node, and get rightsizing recommendations for the namespaces you choose. The agent changes nothing in your cluster. What the agent reads lists every permission and every field it sends. An optional automation component, which you install separately and turn on per namespace, can resize pods after OOM kills and merge the pull requests you allow.

Prerequisites

A LevelFour admin, to connect the cluster or create an install key on Connect Providers > Kubernetes in the dashboard.
Kubernetes 1.23 or later.
helm 3.8 or later and kubectl access to the cluster, with rights to create a namespace, a ClusterRole and a ClusterRoleBinding. With GitOps, the tool that deploys the chart needs the same rights.
For costs: the cloud account that runs the cluster connected to LevelFour with its billing data, or node prices you set for a cluster with no cloud bill. How clusters are priced lists what each cloud needs. Without it, LevelFour shows configuration and usage with no prices.

The agent runs on Amazon EKS, Google Kubernetes Engine, Azure Kubernetes Service and any other cluster on Kubernetes 1.23 or later. For a cluster that cannot run it, your bill can still show cost per namespace: see Clusters without the agent.

Review the chart before you install

The chart and the agent image are public on Amazon ECR Public. You can read them before you install, with no login and no access to a cluster:

helm show chart  oci://public.ecr.aws/levelfour/charts/levelfour-agent --version 0.6.1
helm show values oci://public.ecr.aws/levelfour/charts/levelfour-agent --version 0.6.1
helm pull oci://public.ecr.aws/levelfour/charts/levelfour-agent --version 0.6.1 --untar
helm template levelfour-agent oci://public.ecr.aws/levelfour/charts/levelfour-agent --version 0.6.1 \
  --namespace levelfour --set clusterName=review --set token=review

helm pull saves the chart's templates in a levelfour-agent folder. helm template prints the six objects the install creates (ServiceAccount, ClusterRole, ClusterRoleBinding, Secret, Deployment and Service) and applies none of them. To check that LevelFour published the chart and the image, verify their signatures. A 403 from public.ecr.aws comes from an old login saved on your machine: see Helm cannot pull the chart.

Pick a way to install

The Connect Kubernetes dialog asks which way you install. Both install the same chart and the same read-only agent.

One cluster, with Helm	Many clusters, with GitOps
You name the cluster in the dashboard, and it shows the Helm command	You create one install key for your organization
The token in the command works for that cluster only	Each cluster that runs the agent with the key registers itself under its `clusterName`
You run the command once per cluster	Argo CD, Flux or Terraform deploys the chart, and your secret manager supplies the key

Install with Helm

Name the cluster

Open Connect Providers > Kubernetes in the dashboard and click Connect cluster (Add a cluster once you have one). In the Connect Kubernetes dialog, choose One cluster, with Helm and name the cluster. A name uses lowercase letters, digits and hyphens, starts and ends with a letter or digit, and has at most 63 characters.

The dialog shows the install command for the cluster. The token in the command (l4k_...) sends data for this one cluster and cannot read anything from your LevelFour organization.

Run the install command

Run the command with your kubectl context set to the cluster:

helm upgrade --install levelfour-agent oci://public.ecr.aws/levelfour/charts/levelfour-agent \
  --version 0.6.1 \
  --namespace levelfour --create-namespace \
  --set token=YOUR_TOKEN \
  --set clusterName=YOUR_CLUSTER_NAME

The command pins chart version 0.6.1 and creates the levelfour namespace. The chart adds a service account, a read-only ClusterRole and its binding, a Secret holding the token, one Deployment and a Service for the agent's metrics port. It adds no CRD, no webhook and no persistent volume.

Wait for the agent

The dashboard turns green when the agent first reports. To check from the cluster side, run the commands in Verify the install.

Choose the namespaces that get recommendations

Label each namespace you want recommendations for, or turn it on from the dashboard. Start with one and add more when you are ready. Choose namespaces covers both ways, the opt-out and the namespaces that never get recommendations.

To change the proxy, Prometheus, image registry or any other setting, see Configure the agent.

Verify the install

kubectl -n levelfour get pods
kubectl -n levelfour logs deploy/levelfour-agent

The pod reaches Running within a minute and reports to LevelFour as soon as it starts. It turns ready once it has read the cluster's objects, which takes longer on a large cluster. Its first log line names the agent version, the cluster name and the address it sends to.

When LevelFour refuses a report, the agent logs the status, the code and LevelFour's message, such as 401 AUTHENTICATION_ERROR: Unknown ingest token. It logs each answer once, and logs again when LevelFour accepts reports. Troubleshooting explains each code.

The cluster turns green on Connect Providers > Kubernetes when its agent first reports.

What you see and when

Configuration and usage: on the Kubernetes page within a few minutes, for every namespace.
Costs: after the next delivery of your billing data, usually within a day, and up to two days for the Google Cloud export. The most recent day stays marked as not yet priced until the bill covers it. On a cluster with no cloud bill, within about 12 hours of setting node prices.
Recommendations: once a labelled namespace has at least 7 days of usage, they start to appear on the Savings page next to your other recommendations, and you approve them the same way. Each one becomes a pull request on the repository that deploys the workload.

Choose namespaces

Recommendations are opt-in per namespace. The agent reads the whole cluster from the start, so costs and usage cover every namespace. LevelFour recommends changes only in the namespaces you opt in, with a label in the cluster or a switch in the dashboard. A configuration object that sets automationMode also opts in the namespaces it covers, unless a label or the switch turned them off. A namespace you opt in later already has its history, so its recommendations arrive without a new wait.

From the dashboard

Turn a namespace on or off with the switch in the cluster's Namespaces tab, or at the top of the namespace's page. Admins and members can change it; viewers see it.

A label in the cluster wins over the switch. A namespace labelled enabled or disabled shows its switch locked, with the label as the reason. Remove the label to decide from the dashboard again.

With a label

kubectl label namespace payments levelfour.ai/recommendations=enabled

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: payments
  labels:
    levelfour.ai/recommendations: enabled

To keep one workload out of an opted-in namespace, set the same key to disabled on the workload, as a label or an annotation:

kubectl -n payments label deployment checkout-api levelfour.ai/recommendations=disabled

To stop new recommendations in a namespace, remove the label:

kubectl label namespace payments levelfour.ai/recommendations-

These namespaces never get recommendations, whatever their labels or switches: kube-system, kube-public, kube-node-lease, and the namespace the agent runs in (levelfour by default).

Upgrade

Each chart version pins its own agent image, so an upgrade changes one thing: the chart version you pinned. Leave image.tag unset and the image moves with the chart.

Agent 0.3.0 adds read access to events and storageclasses to the ClusterRole, as What the agent reads lists, and sends the data Node optimization reads. Agent 0.4.1 adds read access to resourcequotas, limitranges and the EKS Auto Mode nodeclasses, and ships the optional automation component, which stays uninstalled until you set automation.enabled. Agent 0.6.1 sends each container's live requests and limits and each node's manifest, as What the agent reads lists, and configures the automation component with objects in the cluster.

If you turned the automation component on with chart 0.4.x, rename its values when you upgrade to 0.6.1, because the chart now refuses an automation value it does not know: automation.namespaces becomes automation.allowedNamespaces, automation.limits.maxCPU and maxMemory become automation.hardLimits.maximumCores and maximumGiB (a number of cores and of GiB, so "8" and 16Gi become 8 and 16), and automation.revert.allowEviction becomes automation.revert.allowPodEviction.

helm upgrade levelfour-agent oci://public.ecr.aws/levelfour/charts/levelfour-agent \
  --version NEW_VERSION \
  --namespace levelfour --reset-then-reuse-values

--reset-then-reuse-values needs Helm 3.14 or later. It keeps your values, the token among them, and applies the defaults of the new chart. --reuse-values would skip defaults that the new chart adds.

From 0.4.x with the automation component on, both flags bring the old automation keys back, and the chart refuses them. Save your values to a file, rename the keys there, and upgrade from the file with no reuse flag:

helm get values levelfour-agent -n levelfour -o yaml > values.yaml

Rename whichever of the three keys the file has:

values.yaml

automation:
  enabled: true
  allowedNamespaces:
    - payments
  hardLimits:
    maximumCores: 8
    maximumGiB: 16
  revert:
    allowPodEviction: false

helm upgrade levelfour-agent oci://public.ecr.aws/levelfour/charts/levelfour-agent \
  --version 0.6.1 \
  --namespace levelfour -f values.yaml

The file holds every value you set, the token among them, and the new chart adds its own defaults. Delete it once the upgrade is done, because of the token.

Set targetRevision in the Application to the new version and commit it.

Set ref.tag in the OCIRepository to the new version and commit it.

Set version in the helm_release to the new version and apply.

Uninstall and disconnect

Remove the agent

helm uninstall levelfour-agent -n levelfour
kubectl delete namespace levelfour

argocd app delete levelfour-agent --cascade
kubectl delete namespace levelfour

If an app of apps or an ApplicationSet creates the Application, remove it from that source instead, or Argo CD creates it again.

Remove the HelmRelease and the OCIRepository from your repository. When Flux deletes the HelmRelease, it uninstalls the release. Then delete the namespace:

kubectl delete namespace levelfour

Remove the helm_release and apply. Then delete the namespace:

kubectl delete namespace levelfour

This removes everything the chart created, including the ClusterRole and its binding. Deleting the namespace also removes the Secret that holds the token or install key.

Disconnect the cluster

Disconnect the cluster on Connect Providers > Kubernetes. For a cluster installed with Helm, this revokes its token. Disconnecting frees the cluster's name for reuse.

Remove the agent first. An agent that keeps running with an install key logs 409 cluster_disconnected after you disconnect its cluster. The install key keeps working for your other clusters, so revoke it only when no cluster uses it.

Remove the namespace labels

The levelfour.ai/recommendations labels on your namespaces do nothing once the agent is gone. To remove them too:

kubectl label namespace --all levelfour.ai/recommendations-

On this page