Day 34/40 - Step-By-Step Guide To Upgrade a Multi Node Kubernetes Cluster With Kubeadm

Day 34/40 — Cluster Upgrade With Kubeadm

Part of the Tech Tutorials with Piyush 40-day CKA series. This ~55-minute hands-on tutorial covers the complete end-to-end process of upgrading a multi-node Kubernetes cluster using kubeadm — from node maintenance primitives (drain/cordon/uncordon) through version policy to control-plane and worker-node upgrades.

Key Concepts Covered

Node Maintenance: Drain, Cordon, and Uncordon

Before any upgrade, you must safely evacuate workloads from a node. Kubernetes provides three imperative maintenance primitives:

• Drain (kubectl drain <node>) — Evicts all running Pods from the node. For Pods managed by controllers (Deployments, ReplicaSets, DaemonSets), the controller reschedules replacements on other nodes. For standalone Pods not backed by a controller, drain deletes them permanently; the data is lost. Drain also automatically cordons the node (marks it unschedulable) so no new Pods land during maintenance.
• Cordon (kubectl cordon <node>) — Marks a node as Unschedulable. Existing Pods keep running, but the scheduler will not place new Pods on it. Drain implicitly includes cordoning.
• Uncordon (kubectl uncordon <node>) — Removes the unschedulable taint, making the node available for new workloads again. Uncordon does not move already-rescheduled Pods back; it only affects future scheduling decisions.

Important edge case: DaemonSet-managed Pods (e.g., CNI agents, CSI node drivers) resist eviction because they must exist on every node. By default, kubectl drain refuses to proceed if DaemonSet Pods are present. Use --ignore-daemonsets when you know it is safe (e.g., the DaemonSet will remain functional after the node reboots or upgrades).

Kubernetes Versioning and the Version Skew Policy

Kubernetes releases follow semantic versioning: major.minor.patch (e.g., v1.30.2).

• Major — Always 1 since Kubernetes launched. A future v2 would be a monumental breaking change.
• Minor — Incremented every ~3–4 months. This is the version administrators typically upgrade.
• Patch — Frequent bug-fix and security releases. These are safe to apply regularly.

Version Skew Rule (critical for CKA): You cannot skip minor versions during an upgrade. To move from v1.28.x to v1.30.x, you must first upgrade to v1.29.x, then to v1.30.x. The official documentation explicitly prohibits skipping unsupported minor versions. This is a common exam trap.

Support Window: Kubernetes supports only the latest three minor versions. When v1.31 releases, v1.28 moves out of support (no new patches or bug fixes). Organizations running unsupported versions assume security and stability risk.

Component Version Compatibility

Within a cluster, different control-plane components are allowed to differ by at most one minor version relative to the API server:

Component	Maximum Version Skew Relative to API Server
kube-controller-manager, kube-scheduler	API server version or one minor version behind
kubelet	API server version or two minor versions behind
kubectl	API server version or one minor version behind

In practice, kubeadm keeps all components aligned to the target version during an upgrade.

Upgrade Strategies

The video discusses three high-level strategies:

1. All-at-once — Drain every node simultaneously, upgrade all, then uncordon. Fastest, but causes total workload unavailability. Never used in production.
2. Rolling update (one at a time) — Drain → upgrade → uncordon one node, then move to the next. Applications remain available because other nodes serve traffic throughout. This is the standard production approach demonstrated in the video.
3. Blue/Green — Provision an entirely new cluster on the target version, migrate workloads, then decommission the old cluster. Requires double infrastructure but minimizes downtime and rollback risk. Common in managed clouds (EKS, AKS, GKE) and large on-prem environments.

kubeadm Upgrade Steps (Demo Walkthrough)

The video demonstrates the exact sequence on Ubuntu VMs (control plane + two workers):

On the control plane node:

1. Determine the target version: apt-cache madison kubeadm (after updating the apt repository to the new minor version).
2. Plan the upgrade: sudo kubeadm upgrade plan — shows current version, available target, and component-level diff.
3. Apply the control-plane upgrade: sudo kubeadm upgrade apply v1.30.2 — upgrades API server, scheduler, controller-manager, and etcd manifests.
4. Drain the control-plane node: kubectl drain <master-node> --ignore-daemonsets.
5. Upgrade kubelet and kubectl packages: sudo apt-get install kubelet kubectl.
6. Restart kubelet: sudo systemctl restart kubelet.
7. Uncordon: kubectl uncordon <master-node>.

On each worker node:

1. Update the apt repository to the target minor version.
2. Upgrade kubeadm: sudo apt-get install kubeadm.
3. Upgrade the local node config: sudo kubeadm upgrade node.
4. Drain the worker from the control plane: kubectl drain <worker-node> --ignore-daemonsets.
5. Upgrade kubelet and kubectl: sudo apt-get install kubelet kubectl.
6. Restart kubelet: sudo systemctl restart kubelet.
7. Uncordon: kubectl uncordon <worker-node>.

Verification commands shown:

• kubectl get nodes — confirms all nodes report the new version and Ready status.
• kubectl version — confirms client version.
• sudo cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep image — confirms control-plane container images reflect the new version.

Novel Insights

• kubectl drain is actually a composite operation: it evicts Pods and cordons the node. Many operators mistakenly believe they need to run cordon separately after a drain — the video clarifies this redundancy.
• Standalone Pods (not managed by a Deployment, ReplicaSet, Job, or DaemonSet) are deleted forever during a drain. This is a data-loss trap for operators running one-off database Pods or singleton services without controllers.
• The control plane must be upgraded before worker nodes. Upgrading workers first would leave them talking to an older API server, potentially violating the version-skew tolerance and causing kubelet registration failures.
• In a high-availability (HA) setup with multiple control-plane nodes, you upgrade the primary first with kubeadm upgrade apply, then additional control-plane nodes with kubeadm upgrade node (no apply). The video briefly notes this but focuses on the single-control-plane case.

See Also

Wiki Concepts

• Kubernetes Cluster Upgrade — Synthesized upgrade procedures, strategies, and exam patterns
• Node Maintenance: Drain, Cordon, Uncordon — Deep dive into node evacuation and scheduling gates
• Kubernetes Versioning & Version Skew — Release cadence, support lifecycle, and component compatibility matrix
• Kubeadm Cluster Setup — Bootstrap procedure that precedes upgrades; shares certificate and manifest paths
• Kubernetes Architecture — How Static Pods in /etc/kubernetes/manifests enable kubeadm to upgrade control-plane containers
• Kubernetes Taints and Tolerations — NoExecute taint is the mechanism behind drain-induced eviction
• CKA Certification — Exam domain mapping: cluster upgrades sit in Architecture, Installation & Configuration (~25%)
• CKA Study Roadmap — Day-by-day plan; Day 34 maps to cluster upgrades

Related Sources

• CKA Day 27 — Setup a Multi Node Kubernetes Cluster Using Kubeadm — The cluster built in Day 27 is the one upgraded here

Creator / Entity

• Tech Tutorials with Piyush — Day 34 of the 40-day CKA challenge series