Kubernetes self-healing upgrade pipeline: 11 minutes, zero humans

Share

Eleven Minutes, Zero Humans: A Self-Healing Kubernetes Upgrade Pipeline on Kairos

Upgrading a Kubernetes control plane used to mean a sleepless night.

SSH into each node, run the upgrade manually, and watch etcd health continuously, hoping quorum survives every reboot.

This week, I tested whether the pipeline I built actually ends that story.

Why the Management Cluster Matters

This management cluster is where everything starts.

It was bootstrapped with OpenTofu, running three control plane nodes with K3s in HA mode and Cilium as the CNI, all provisioned as code before any workload ran. I treat it as the root of a platform that will keep growing: more tooling, more workloads, more clusters hanging off it over time.

Roots need to be solid before building on top of them. In 2026, with the pace of CVEs across the stack, what I wanted most from this root wasn’t more features. It was a genuinely simple and reliable upgrade process—something I could trust to run without relearning the steps every time a patch shipped. I wanted to use as much Cloud Native Computing Foundation (CNCF) tooling as possible to avoid vendor lock-in and build on my Golden Kubestronaut knowledge.

That’s what the rest of this post covers.

Graphic: OpenTofu flowchart

Setting Up the Nodes

Three control plane nodes run K3s in HA mode on Kairos Hadron, an immutable Linux distribution built around A/B partition upgrades and cosign-signed images. Kairos doesn’t patch in place—it writes a new OS image to an inactive partition and reboots into it. Rollback simply means booting the old partition again.

That’s a strong supply-chain story, but it only works if the upgrade actually happens. My previous pipeline version had a subtle but dangerous bug: concurrency: 0 in the upgrade spec, which I assumed meant “one node at a time.” It actually means all nodes simultaneously. Three control plane nodes rebooted at once during a homelab test. etcd quorum survived—pure luck, not design.

I fixed it to concurrency: 1. That fix is the backbone of everything below.

The Full Pipeline, Tool by Tool

Six tools, each with one job:

Gitea — self-hosted git, running on its own Kairos + k3s cluster, with a Gitea Actions runner executing CI. This is where every manifest, policy, and upgrade spec lives.

Renovate — watches quay.io/kairos/hadron on Quay for new tags. When upstream publishes a release, Renovate opens a PR bumping two lines: the image tag and the metadata.name of the upgrade CR.

Kyverno — a ClusterPolicy admission gate that rejects any upgrade CR whose image doesn’t match quay.io/kairos/hadron:*. This blocks misconfiguration and typosquatting before it reaches a node.

Cosign — verifies the image signature against the upstream GitHub Actions OIDC identity. Not just “is this the right tag,” but “did the real CI pipeline actually produce this artifact.”

ArgoCD — GitOps from the Argo Project. Detects the merged PR as drift and applies the new manifest. No human runs kubectl apply.

kairos-operator — the actual upgrade executor. Watches for NodeOpUpgrade custom resources, cordons one node, pulls the image, writes the new A/B slot, reboots, waits for rejoin, and moves to the next node.

Graphic: kairos-operator — the actual upgrade executor.

What Actually Happened During the Upgrade

Real timestamps from the July 7th upgrade to Hadron v0.4.0:

Time Event / Status
11:40:20 Baseline — 3 nodes Ready, kernel 7.0.10-hadron
11:42:00 PR #9 merged (Renovate bump, 2-line diff)
11:42:30 ArgoCD detects OutOfSync
11:42:37 ArgoCD Synced — old CR deleted, new CR created
11:42:43 Preflight: image pull starts (324 MB)
11:43:09 Image pulled in 25.5s
11:43:11 mgmt-cp-0 cordoned
11:43:33 Upgrade job completed — A/B slot written, reboot triggered
~11:47 mgmt-cp-0 Ready — kernel 7.1.0-hadron confirmed
~11:47 mgmt-cp-1 cordoned, upgraded, rebooted
~11:49 mgmt-cp-1 Ready — kernel 7.1.0-hadron
~11:49 mgmt-cp-2 cordoned, upgraded, rebooted
11:51:36 mgmt-cp-2 Ready — kernel 7.1.0-hadron. All done.

Total wall-clock time: 11 minutes. Human intervention after merge: zero. etcd quorum broken: never. Workload disruption: none.

Graphic: Total wall-clock time

The two-line diff that triggered all of this:

--- a/upgrades/mgmt/hadron-upgrade.yaml

+++ b/upgrades/mgmt/hadron-upgrade.yaml

@@ -11,7 +11,7 @@ metadata:

-  name: hadron-mgmt-v0-3-0

+  name: hadron-mgmt-v0-4-0

 spec:

-  image: quay.io/kairos/hadron:v0.3.0-standard-amd64-generic-v4.1.1-k3s-v1.35.5-k3s1

+  image: quay.io/kairos/hadron:v0.4.0-standard-amd64-generic-v4.1.2-k3s-v1.35.5-k3s1

Why the CR Name Has to Change

NodeOpUpgrade is a one-shot custom resource. kairos-operator marks it complete and never reprocesses it. Patching spec.image on an existing CR does nothing—the operator already considers that object finished. Renovate has to bump metadata.name in the same commit, which forces ArgoCD to delete the old CR and create a genuinely new one. That’s the actual trigger.

The Bug I Didn’t Expect

This pipeline wasn’t flawless on the first real run.

Renovate’s custom regex manager used a field called extractVersionTemplate to convert the dash-formatted version in the CR name (v0-3-0) into a comparable SemVer. That field doesn’t exist in Renovate’s custom manager schema. It silently did nothing.

Result: Renovate correctly bumped spec.image to the new tag, but left metadata.name completely unchanged. kairos-operator saw no new CR—because there wasn’t one—and did nothing. The pipeline looked broken, but it was actually just half-executed.

The fix was switching to currentValueTemplate, which properly converts v0-3-0 into v0.3.0 so Renovate’s version comparison logic works correctly, then reconstructs the dash format for the new tag.

This is exactly why the human review step still exists in this pipeline. Not to execute the upgrade—the automation does that. To catch the one place where automation quietly does the wrong thing while looking like it did the right thing.

What’s Next for This Pipeline

The gateway cluster—a separate single-node Kairos deployment running Netbird—is now wired into the same ArgoCD GitOps loop as mgmt. Same pipeline, same automated upgrade path, no separate process to maintain.

Next up: a CI dry-run stage running kairos-agent upgrade –recovery against the new image before the PR ever merges, catching failures before they touch a live node at all.

Do you know the name of the Immutable OS which can operate at scale? It’s Kairos!

Read more