Install with GitOps

An install key connects as many clusters as you run, with no trip to the dashboard per cluster. Each cluster that runs the agent with the key registers itself under its clusterName on its first report. Connect a cluster has the prerequisites and the Helm install for a single cluster.

Give each cluster a unique clusterName. An agent that reports a name another connected cluster already has gets 409 cluster_name_taken, and its cluster stays unregistered until you rename it.

One key registers up to 100 new clusters an hour. A larger fleet comes in over the next hours: each agent past the limit gets 429 install_key_registration_limit, keeps its reports and registers on a later retry.

Install

Create an install key

Open Connect Providers > Kubernetes in the dashboard and click Connect cluster (Add a cluster once you have one). In the Connect Kubernetes dialog, choose Many clusters, with GitOps and create an install key. Give it a name you will recognize in the key list.

The dialog shows the key (l4ki_...) once, so put it in your secret manager before you close it. The key can only send cluster data to LevelFour. It reads nothing.

Store the key

The chart reads the key from a Secret you manage, with this name, key and namespace:

Field	Value
Secret name	`levelfour-agent`
Key	`token`
Namespace	`levelfour`

Create the levelfour namespace and the Secret in every cluster that runs the agent. The chart mounts the Secret as a file, and the agent pod stays in ContainerCreating until the Secret exists.

levelfour-agent-secret.yaml

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: levelfour-agent
  namespace: levelfour
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: YOUR_SECRET_STORE
    kind: ClusterSecretStore
  target:
    name: levelfour-agent
  data:
    - secretKey: token
      remoteRef:
        key: YOUR_SECRET_PATH

kubectl create namespace levelfour
kubectl create secret generic levelfour-agent --namespace levelfour \
  --from-literal=token=YOUR_INSTALL_KEY

Sealed Secrets and SOPS work too. Commit the encrypted Secret with the same name, key and namespace.

Point your GitOps tool at the chart

Each cluster needs two values:

values.yaml

clusterName: YOUR_CLUSTER_NAME
existingSecret:
  name: levelfour-agent

A clusterName uses lowercase letters, digits and hyphens, starts and ends with a letter or digit, and has at most 63 characters. The chart refuses any other name. Leave token unset: the chart reads the key from the Secret. existingSecret.key defaults to token, so set it only if your Secret stores the key under another name. Configure the agent lists every other value.

Pin chart version 0.1.0 in the tool that deploys it. Each example installs the release as levelfour-agent in the levelfour namespace, the names the rest of these pages use.

levelfour-agent.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: levelfour-agent
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: public.ecr.aws/levelfour/charts
    chart: levelfour-agent
    targetRevision: 0.1.0
    helm:
      valuesObject:
        clusterName: YOUR_CLUSTER_NAME
        existingSecret:
          name: levelfour-agent
  destination:
    server: https://kubernetes.default.svc
    namespace: levelfour
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Write repoURL without oci:// and name the chart in chart. Argo CD reads a repoURL with no scheme, next to a chart, as a Helm chart in an OCI registry. The registry is public, so Argo CD needs no repository Secret for it. Argo CD 3.1 and later also accept a native OCI source: repoURL: oci://public.ecr.aws/levelfour/charts/levelfour-agent with path: . and no chart. Use one form or the other, never both in one source.

To run the agent on every cluster Argo CD manages, generate one Application per cluster with an ApplicationSet. Each cluster's name in Argo CD becomes its clusterName, so it has to follow the clusterName rule:

levelfour-agent-applicationset.yaml

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: levelfour-agent
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - clusters: {}
  template:
    metadata:
      name: '{{.nameNormalized}}-levelfour-agent'
    spec:
      project: default
      source:
        repoURL: public.ecr.aws/levelfour/charts
        chart: levelfour-agent
        targetRevision: 0.1.0
        helm:
          releaseName: levelfour-agent
          valuesObject:
            clusterName: '{{.name}}'
            existingSecret:
              name: levelfour-agent
      destination:
        server: '{{.server}}'
        namespace: levelfour
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

argocd cluster add names a cluster after its kubeconfig context, which for Amazon EKS is the cluster's ARN. The chart refuses that name. Rename the cluster in Argo CD first:

argocd cluster set CURRENT_NAME --name payments-prod

clusters: {} matches every cluster registered in Argo CD, including the one Argo CD runs in, which it names in-cluster. Two Argo CD instances each have an in-cluster, and the second to report gets 409 cluster_name_taken. Rename that cluster too, or narrow the generator with a label selector. releaseName keeps the Deployment named levelfour-agent in every cluster.

levelfour-agent.yaml

apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: levelfour-agent
  namespace: levelfour
spec:
  interval: 10m
  url: oci://public.ecr.aws/levelfour/charts/levelfour-agent
  ref:
    tag: "0.1.0"
  layerSelector:
    mediaType: "application/vnd.cncf.helm.chart.content.v1.tar+gzip"
    operation: copy
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: levelfour-agent
  namespace: levelfour
spec:
  interval: 10m
  chartRef:
    kind: OCIRepository
    name: levelfour-agent
  values:
    clusterName: YOUR_CLUSTER_NAME
    existingSecret:
      name: levelfour-agent

The OCIRepository v1 API needs Flux 2.6 or later. Both objects live in the levelfour namespace next to the Secret, so Flux names the release levelfour-agent. To keep them in flux-system instead, set targetNamespace: levelfour and releaseName: levelfour-agent on the HelmRelease, because Flux otherwise names the release <targetNamespace>-<name>.

Set clusterName per cluster, for example with a Kustomize overlay or postBuild.substitute on the Flux Kustomization that applies this file. To have Flux check the chart's signature on every pull, see Verify the chart in Flux.

levelfour-agent.tf

resource "helm_release" "levelfour_agent" {
  name             = "levelfour-agent"
  repository       = "oci://public.ecr.aws/levelfour/charts"
  chart            = "levelfour-agent"
  version          = "0.1.0"
  namespace        = "levelfour"
  create_namespace = true

  values = [yamlencode({
    clusterName = var.cluster_name
    existingSecret = {
      name = "levelfour-agent"
    }
  })]
}

The example uses Helm provider 3.x syntax, where set, set_list and set_sensitive take lists of objects and kubernetes in the provider block is an attribute: kubernetes = { ... }.

To pass the key from Terraform instead of a Secret, drop existingSecret and add set_sensitive = [{ name = "token", value = var.levelfour_install_key }]. Terraform then keeps the key in its state.

Check that each cluster reports

Each cluster registers itself on its first report and shows on Connect Providers > Kubernetes under its clusterName. Verify the install has the commands to check the pod and its logs. Then choose the namespaces that get recommendations.

Rotate an install key

Create a new key

Create a second install key on Connect Providers > Kubernetes. Keep the old key until the last step.

Update the Secret

Put the new key in your secret manager, under the same Secret name and key. External Secrets copies it into the Secret at its next refresh, which refreshInterval sets. To replace the Secret with kubectl:

kubectl create secret generic levelfour-agent --namespace levelfour \
  --from-literal=token=YOUR_NEW_INSTALL_KEY \
  --dry-run=client -o yaml | kubectl apply -f -

The agent needs no restart. It reads the key from the mounted file before each request, and the kubelet updates that file after the Secret changes, so the agent sends with the new key within about a minute.

Revoke the old key

Each cluster moves to the new key with its first report on it. Once the old key's Clusters count reaches zero, revoke the old key. A cluster still on the old key gets 401 AUTHENTICATION_ERROR from then on.

Revoke an install key

Connect Providers > Kubernetes lists each install key with its name, key prefix, clusters, last use and creation date, and a button to revoke it. Clusters counts the connected clusters whose last report came with the key.

Revoking stops every cluster reporting with that key. Their agents keep running and log 401 AUTHENTICATION_ERROR: Unknown ingest token until their Secret holds a valid key. To stop one cluster and keep the others, disconnect that cluster instead.

On this page