Troubleshooting

Start with the agent's logs:

kubectl -n levelfour logs deploy/levelfour-agent

When LevelFour refuses a report, the agent logs the HTTP status, the code and LevelFour's message, for example 401 AUTHENTICATION_ERROR: Unknown ingest token. It logs each answer once, and logs again when LevelFour accepts reports.

The agent keeps running through a refusal. For a 401, 403 or 409, it holds up to 24 hourly reports in memory and retries every 5 minutes. Once you fix the cause, it resends them, and LevelFour accepts reports up to a day old. A fix that changes the chart values restarts the pod, which drops the held reports.

Answers in the agent logs

401 AUTHENTICATION_ERROR

LevelFour does not accept the token or install key the agent sends. The message says which case you hit:

• Unknown ingest token: the token or install key is wrong, or someone revoked the key. With an install key, put a valid key in the levelfour-agent Secret. The agent reads it within about a minute, with no restart (Rotate an install key). With a token from the Helm command, disconnect the cluster if it still shows on Connect Providers > Kubernetes, connect it again and run the new command.
• Cluster disconnected: someone disconnected a cluster installed with the Helm command. Connect it again on Connect Providers > Kubernetes and run the new command.

403 module_not_enabled

Kubernetes is not turned on for your organization, or its trial has ended. Ask your LevelFour contact to turn it on. The agent holds up to 24 hourly reports and resends them when the module comes on, so a module that is off for less than a day loses no data.

409 cluster_name_taken

Another connected cluster already has the clusterName this agent reports. Give this cluster a unique name in its values and sync. If the cluster that holds the name no longer exists, disconnect it on Connect Providers > Kubernetes. Disconnecting frees the name.

409 cluster_has_token

You connected this cluster with the Helm command, and its agent now reports with an install key. Choose one:

• Keep the per-cluster token: run the agent with token again, as the Helm command set it.
• Move the cluster to the install key: disconnect it on Connect Providers > Kubernetes first. Its next report with the key registers it again.

409 cluster_disconnected

An admin disconnected this cluster while its agent ran with this install key. To bring the cluster back, run the agent with a new install key, or connect it on its own with the Helm command. To leave it disconnected, remove the agent.

409 cluster_uid_mismatch

The per-cluster token belongs to a different cluster. A token from the Helm command works for one cluster only. Connect this cluster on its own on Connect Providers > Kubernetes, or use an install key for every cluster.

422 invalid_cluster_name

The agent reports a clusterName that breaks the naming rule: lowercase letters, digits and hyphens, at most 63 characters, starting and ending with a letter or digit. The chart refuses such a name at install, so this answer means the agent runs without the chart's checks. LevelFour drops these reports instead of holding them. Fix the name, and the cluster registers on its next report.

429 install_key_registration_limit

The install key registered 100 new clusters in the last hour, its limit. Do nothing: the agent keeps its reports, retries every 5 minutes and registers once the hour frees a place. Clusters the key already registered keep reporting.

The install fails

Helm, Argo CD, Flux and Terraform all render the chart with its checks, and report the same errors:

• clusterName: Does not match pattern or clusterName: String length must be greater than or equal to 1: set a clusterName that follows the naming rule.
• token is required: set existingSecret.name to the Secret that holds the install key, or token for a Helm install.
• set token or existingSecret.name, not both: remove one of them.
• chart requires kubeVersion: >=1.23.0-0: the cluster runs a Kubernetes version older than 1.23.

The pod stays in ContainerCreating

The chart mounts the Secret that holds the key, and the pod cannot start until the Secret and its key exist. The pod's events name what is missing:

kubectl -n levelfour describe pod -l app.kubernetes.io/name=levelfour-agent
kubectl -n levelfour describe secret levelfour-agent

The Secret must be named levelfour-agent (or the name in existingSecret.name) and hold the key under token (or the name in existingSecret.key). describe secret lists each key and its size, without the value. Kubernetes starts the pod once the Secret is in place.

The agent cannot reach LevelFour

The agent sends data to api.levelfour.ai on port 443. Allow that destination in your firewall, proxy or egress NetworkPolicy. If the cluster reaches the internet through a proxy, set HTTPS_PROXY and NO_PROXY as in Behind a proxy.

Usage is zero for every workload

The agent reads usage from the kubelet on port 10250 of each node. Allow traffic from the agent's pod to the nodes on that port. A security group for pods or a NetworkPolicy can block it. Behind a proxy, check that NO_PROXY covers the node CIDR.

No recommendations

Check, in order:

1. The namespace carries levelfour.ai/recommendations=enabled, and the workload does not carry disabled.
2. The workload has run for at least 7 days since the agent started, or Prometheus history covers that period.
3. The AWS account running the cluster sends detailed cost data. Without prices, LevelFour cannot size a saving.

Costs show no price

Detailed cost data is not turned on for the AWS account that runs the cluster, or AWS has not delivered the most recent day yet.