homelab-cluster

Overview

Six nodes — a mix of NUCs and Pi 5s — running k3s across amd64 and arm64, so every image gets to fail on exactly one architecture before it works on both. Nothing is configured by hand, which is a polite way of saying I no longer remember how any of it was configured. The cluster's entire desired state lives in a Git repo, ArgoCD reconciles against it continuously, and when a node dies I kubectl drain it and go get a Diet Coke — the alternative being an SSH session to a Pi that stopped listening some time ago.

The interesting part isn't the hardware, it's that adding a service is a git push and deleting a directory genuinely deletes the service. No lingering namespace, no orphaned volume, nothing left behind to be discovered eighteen months later by someone reading a kubectl get all -A with growing concern.

GitOps, three layers deep

The bootstrap is a single app-of-apps: one ArgoCD Application pointed at gitops/, which pulls in two sources — the GitOps config itself, and the sealed secrets directory, recursively.

kind: Application
metadata:
  name: gitops
spec:
  sources:
    - repoURL: [email protected]:JustBrenkman/homelab.git
      path: gitops
    - repoURL: [email protected]:JustBrenkman/homelab.git
      path: gitops/secrets
      directory:
        recurse: true
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

That Application's kustomization contains exactly one resource: an ApplicationSet with a Git directory generator.

kind: ApplicationSet
spec:
  generators:
    - git:
        repoURL: [email protected]:JustBrenkman/homelab.git
        directories:
          - path: gitops/apps/*
  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      destination:
        namespace: '{{path.basename}}'
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [CreateNamespace=true]

This is the whole trick. The generator globs gitops/apps/* and templates one ArgoCD Application per directory it finds, named after the folder, deployed into a namespace named after the folder, with the namespace created on demand. There is no central registry of services to update. Creating gitops/apps/whatever/ with a kustomization.yaml is the entire process for adding a service; removing the directory prunes it.

The registry is the part I specifically did not want. Every hand-maintained list of services eventually disagrees with the services, and it never announces this — it waits until you are relying on it.

prune: true and selfHeal: true are both on deliberately. Together they mean the cluster is not merely initialized from Git — it is continuously corrected toward Git. Anything I change with kubectl gets reverted within minutes.

This is occasionally infuriating and always correct. The failure mode it forecloses is the one where a cluster has been quietly diverging from its manifests for two years, nobody knows which of the two is load-bearing, and the only way to find out is to redeploy and see what breaks. The repo is the only honest description of the cluster, enforced by a robot with no interest in my explanations.

What runs on it

That last row is not a joke — the site you are reading is ghcr.io/justbrenkman/justb-dev, deployed by the same ApplicationSet as everything else, reached through a Traefik IngressRoute. The write-up you're reading is served by the cluster it's describing.

Stateful services pin their storage with explicit PersistentVolume / PersistentVolumeClaim pairs rather than relying on dynamic provisioning. On a small cluster with known disks this is easier to reason about and easier to back up — and it means the answer to "where does this data physically live" is a file I can read, not a provisioner's opinion I have to reverse-engineer at an unpleasant hour.

Secrets

Secrets are committed to the repo, which is only sane because they're Sealed Secrets — encrypted against the cluster's controller key, so the ciphertext is public-safe and only this cluster can decrypt it. Registry credentials, MQTT auth, Frigate's broker credentials, and Anvil's API secrets all live in gitops/secrets/ as .sealed.yaml.

It keeps the property that makes the whole setup work: the repo is complete. There's no out-of-band step and no kubectl create secret I have to remember, which matters because I will not remember it. A rebuild from bare metal needs the repo and the controller key, nothing else — no wiki page last edited by someone who has since left, no step 7 that everybody knows about except the person doing the rebuild.