# Troubleshooting

> What each answer in the LevelFour agent's logs means, and how to fix a cluster that does not install, does not report, reports no usage or gets no recommendations.

Source: https://docs.levelfour.ai/connectors/kubernetes/troubleshooting


Start with the agent's logs:

```bash
kubectl -n levelfour logs deploy/levelfour-agent
```

When LevelFour refuses a report, the agent logs the HTTP status, the code and LevelFour's message, for example `401 AUTHENTICATION_ERROR: Unknown ingest token`. It logs each answer once, and logs again when LevelFour accepts reports.

The agent keeps running through a refusal. For a `401`, `403` or `409`, it holds up to 24 hourly reports in memory and retries every 5 minutes. Once you fix the cause, it resends them, and LevelFour accepts reports up to a day old. A fix that changes the chart values restarts the pod, which drops the held reports.

## Answers in the agent logs

### 401 AUTHENTICATION\_ERROR

LevelFour does not accept the token or install key the agent sends. The message says which case you hit:

* **`Unknown ingest token`:** the token or install key is wrong, or someone revoked the key. With an install key, put a valid key in the `levelfour-agent` Secret. The agent reads it within about a minute, with no restart ([Rotate an install key](/connectors/kubernetes/gitops#rotate-an-install-key)). With a token from the Helm command, disconnect the cluster if it still shows on **Connect Providers > Kubernetes**, connect it again and run the new command.
* **`Cluster disconnected`:** someone disconnected a cluster installed with the Helm command. Connect it again on **Connect Providers > Kubernetes** and run the new command.

### 403 module\_not\_enabled

Kubernetes is not turned on for your organization, or its trial has ended. Ask your LevelFour contact to turn it on. The agent holds up to 24 hourly reports and resends them when the module comes on, so a module that is off for less than a day loses no data.

### 409 cluster\_name\_taken

Another connected cluster already has the `clusterName` this agent reports. Give this cluster a unique name in its values and sync. If the cluster that holds the name no longer exists, disconnect it on **Connect Providers > Kubernetes**. Disconnecting frees the name.

### 409 cluster\_has\_token

You connected this cluster with the Helm command, and its agent now reports with an install key. Choose one:

* Keep the per-cluster token: run the agent with `token` again, as the Helm command set it.
* Move the cluster to the install key: disconnect it on **Connect Providers > Kubernetes** first. Its next report with the key registers it again.

### 409 cluster\_disconnected

An admin disconnected this cluster while its agent ran with this install key. To bring the cluster back, run the agent with a new install key, or connect it on its own with the [Helm command](/connectors/kubernetes#install-with-helm). To leave it disconnected, [remove the agent](/connectors/kubernetes#uninstall-and-disconnect).

### 409 cluster\_uid\_mismatch

The per-cluster token belongs to a different cluster. A token from the Helm command works for one cluster only. Connect this cluster on its own on **Connect Providers > Kubernetes**, or [use an install key](/connectors/kubernetes/gitops) for every cluster.

### 422 invalid\_cluster\_name

The agent reports a `clusterName` that breaks the naming rule: lowercase letters, digits and hyphens, at most 63 characters, starting and ending with a letter or digit. The chart refuses such a name at install, so this answer means the agent runs without the chart's checks. LevelFour drops these reports instead of holding them. Fix the name, and the cluster registers on its next report.

### 429 install\_key\_registration\_limit

The install key registered 100 new clusters in the last hour, its limit. Do nothing: the agent keeps its reports, retries every 5 minutes and registers once the hour frees a place. Clusters the key already registered keep reporting.

## The install fails

Helm, Argo CD, Flux and Terraform all render the chart with its checks, and report the same errors:

* **`clusterName: Does not match pattern`** or **`clusterName: String length must be greater than or equal to 1`:** set a `clusterName` that follows the naming rule.
* **`token is required`:** set `existingSecret.name` to the Secret that holds the install key, or `token` for a Helm install.
* **`set token or existingSecret.name, not both`:** remove one of them.
* **`chart requires kubeVersion: >=1.23.0-0`:** the cluster runs a Kubernetes version older than 1.23.

## The pod stays in ContainerCreating

The chart mounts the Secret that holds the key, and the pod cannot start until the Secret and its key exist. The pod's events name what is missing:

```bash
kubectl -n levelfour describe pod -l app.kubernetes.io/name=levelfour-agent
kubectl -n levelfour describe secret levelfour-agent
```

The Secret must be named `levelfour-agent` (or the name in `existingSecret.name`) and hold the key under `token` (or the name in `existingSecret.key`). `describe secret` lists each key and its size, without the value. Kubernetes starts the pod once the Secret is in place.

## The agent cannot reach LevelFour

The agent sends data to `api.levelfour.ai` on port 443. Allow that destination in your firewall, proxy or egress NetworkPolicy. If the cluster reaches the internet through a proxy, set `HTTPS_PROXY` and `NO_PROXY` as in [Behind a proxy](/connectors/kubernetes/configuration#behind-a-proxy).

## Usage is zero for every workload

The agent reads usage from the kubelet on port 10250 of each node. Allow traffic from the agent's pod to the nodes on that port. A security group for pods or a NetworkPolicy can block it. Behind a proxy, check that `NO_PROXY` covers the node CIDR.

## No recommendations

Check, in order:

1. The namespace carries `levelfour.ai/recommendations=enabled`, and the workload does not carry `disabled`.
2. The workload has run for at least 7 days since the agent started, or Prometheus history covers that period.
3. The AWS account running the cluster sends [detailed cost data](/connectors/aws/detailed-cost-data). Without prices, LevelFour cannot size a saving.

## Costs show no price

Detailed cost data is not turned on for the AWS account that runs the cluster, or AWS has not delivered the most recent day yet.
