Eleven Minutes, Zero Humans: A Self-Healing Kubernetes Upgrade Pipeline on Kairos
Upgrading a Kubernetes control plane used to mean a sleepless night.
SSH into each node, run the upgrade manually, and watch etcd health continuously, hoping quorum survives every reboot.
This week, I tested whether the pipeline I built actually ends that story.
Why the Management Cluster Matters
This management cluster is where everything starts.
It was bootstrapped with OpenTofu, running three control plane nodes with K3s in HA mode and Cilium as the CNI, all provisioned as code before any workload ran. I treat it as the root of a platform that will keep growing: more tooling, more workloads, more clusters hanging off it over time.
Roots need to be solid before building on top of them. In 2026, with the pace of CVEs across the stack, what I wanted most from this root wasn’t more features. It was a genuinely simple and reliable upgrade process—something I could trust to run without relearning the steps every time a patch shipped. I wanted to use as much Cloud Native Computing Foundation (CNCF) tooling as possible to avoid vendor lock-in and build on my Golden Kubestronaut knowledge.
That’s what the rest of this post covers.

Setting Up the Nodes
Three control plane nodes run K3s in HA mode on Kairos Hadron, an immutable Linux distribution built around A/B partition upgrades and cosign-signed images. Kairos doesn’t patch in place—it writes a new OS image to an inactive partition and reboots into it. Rollback simply means booting the old partition again.
That’s a strong supply-chain story, but it only works if the upgrade actually happens. My previous pipeline version had a subtle but dangerous bug: concurrency: 0 in the upgrade spec, which I assumed meant “one node at a time.” It actually means all nodes simultaneously. Three control plane nodes rebooted at once during a homelab test. etcd quorum survived—pure luck, not design.
I fixed it to concurrency: 1. That fix is the backbone of everything below.
The Full Pipeline, Tool by Tool
Six tools, each with one job:
Gitea — self-hosted git, running on its own Kairos + k3s cluster, with a Gitea Actions runner executing CI. This is where every manifest, policy, and upgrade spec lives.
Renovate — watches quay.io/kairos/hadron on Quay for new tags. When upstream publishes a release, Renovate opens a PR bumping two lines: the image tag and the metadata.name of the upgrade CR.
Kyverno — a ClusterPolicy admission gate that rejects any upgrade CR whose image doesn’t match quay.io/kairos/hadron:*. This blocks misconfiguration and typosquatting before it reaches a node.
Cosign — verifies the image signature against the upstream GitHub Actions OIDC identity. Not just “is this the right tag,” but “did the real CI pipeline actually produce this artifact.”
ArgoCD — GitOps from the Argo Project. Detects the merged PR as drift and applies the new manifest. No human runs kubectl apply.
kairos-operator — the actual upgrade executor. Watches for NodeOpUpgrade custom resources, cordons one node, pulls the image, writes the new A/B slot, reboots, waits for rejoin, and moves to the next node.

What Actually Happened During the Upgrade
Real timestamps from the July 7th upgrade to Hadron v0.4.0:
| Time | Event / Status |
| 11:40:20 | Baseline — 3 nodes Ready, kernel 7.0.10-hadron |
| 11:42:00 | PR #9 merged (Renovate bump, 2-line diff) |
| 11:42:30 | ArgoCD detects OutOfSync |
| 11:42:37 | ArgoCD Synced — old CR deleted, new CR created |
| 11:42:43 | Preflight: image pull starts (324 MB) |
| 11:43:09 | Image pulled in 25.5s |
| 11:43:11 | mgmt-cp-0 cordoned |
| 11:43:33 | Upgrade job completed — A/B slot written, reboot triggered |
| ~11:47 | mgmt-cp-0 Ready — kernel 7.1.0-hadron confirmed |
| ~11:47 | mgmt-cp-1 cordoned, upgraded, rebooted |
| ~11:49 | mgmt-cp-1 Ready — kernel 7.1.0-hadron |
| ~11:49 | mgmt-cp-2 cordoned, upgraded, rebooted |
| 11:51:36 | mgmt-cp-2 Ready — kernel 7.1.0-hadron. All done. |
Total wall-clock time: 11 minutes. Human intervention after merge: zero. etcd quorum broken: never. Workload disruption: none.

The two-line diff that triggered all of this:
--- a/upgrades/mgmt/hadron-upgrade.yaml
+++ b/upgrades/mgmt/hadron-upgrade.yaml
@@ -11,7 +11,7 @@ metadata:
- name: hadron-mgmt-v0-3-0
+ name: hadron-mgmt-v0-4-0
spec:
- image: quay.io/kairos/hadron:v0.3.0-standard-amd64-generic-v4.1.1-k3s-v1.35.5-k3s1
+ image: quay.io/kairos/hadron:v0.4.0-standard-amd64-generic-v4.1.2-k3s-v1.35.5-k3s1
Why the CR Name Has to Change
NodeOpUpgrade is a one-shot custom resource. kairos-operator marks it complete and never reprocesses it. Patching spec.image on an existing CR does nothing—the operator already considers that object finished. Renovate has to bump metadata.name in the same commit, which forces ArgoCD to delete the old CR and create a genuinely new one. That’s the actual trigger.
The Bug I Didn’t Expect
This pipeline wasn’t flawless on the first real run.
Renovate’s custom regex manager used a field called extractVersionTemplate to convert the dash-formatted version in the CR name (v0-3-0) into a comparable SemVer. That field doesn’t exist in Renovate’s custom manager schema. It silently did nothing.
Result: Renovate correctly bumped spec.image to the new tag, but left metadata.name completely unchanged. kairos-operator saw no new CR—because there wasn’t one—and did nothing. The pipeline looked broken, but it was actually just half-executed.
The fix was switching to currentValueTemplate, which properly converts v0-3-0 into v0.3.0 so Renovate’s version comparison logic works correctly, then reconstructs the dash format for the new tag.
This is exactly why the human review step still exists in this pipeline. Not to execute the upgrade—the automation does that. To catch the one place where automation quietly does the wrong thing while looking like it did the right thing.
What’s Next for This Pipeline
The gateway cluster—a separate single-node Kairos deployment running Netbird—is now wired into the same ArgoCD GitOps loop as mgmt. Same pipeline, same automated upgrade path, no separate process to maintain.
Next up: a CI dry-run stage running kairos-agent upgrade –recovery against the new image before the PR ever merges, catching failures before they touch a live node at all.
Do you know the name of the Immutable OS which can operate at scale? It’s Kairos!

