Kubernetes Worker Node Troubleshooting

How to diagnose and recover from worker node failures — the “muscle” side of the cluster. A core Troubleshooting topic on the CKA exam and a daily operational reality in production. Synthesized from CKA Day 39 — Troubleshooting Worker Nodes Failures.

The Worker Node Failure Mindset

When a worker node fails, the symptoms ripple outward:

1. Node status changes to NotReady or SchedulingDisabled
2. Pods on the node may continue running briefly but lose cluster connectivity
3. New Pods cannot be scheduled to the node
4. Services stop routing traffic to Pods on the failed node (once kube-proxy or the node itself fails)

Golden Rule: kubectl describe node is your first stop. It shows Conditions, Events, Allocatable resources, and the exact reason the node is unhealthy. Read it before SSH-ing. Source: CKA Day 39

---

The Standard Node Debugging Chain

Step 1: kubectl get nodes -o wide
        └─→ Check STATUS column (Ready, NotReady, SchedulingDisabled)

Step 2: kubectl describe node <name>
        └─→ Read Conditions, Events, Allocatable, and Taints

Step 3: SSH to the node
        └─→ Node-level diagnosis is required for most failures

Step 4: systemctl status kubelet
        └─→ Is the kubelet service running?

Step 5: journalctl -u kubelet -n 100
        └─→ Read the kubelet's story

Step 6: systemctl status containerd
        └─→ Is the container runtime running?

Step 7: Check resource pressure
        └─→ df -h, free -h, ps aux | wc -l

---

Node Conditions: The Health Dashboard

Every node reports five Conditions in its status. These are the primary diagnostic signals:

Condition	Status=True	Status=False	Status=Unknown
Ready	Node healthy, accepting Pods	Node unhealthy	kubelet has not reported status
DiskPressure	Disk is low; evicting Pods	Disk OK	—
MemoryPressure	Memory is low; evicting Pods	Memory OK	—
PIDPressure	Too many processes; evicting Pods	PIDs OK	—
NetworkUnavailable	CNI not configured	Network ready	—

Ready = Unknown means the kubelet has stopped heartbeating. This is the most common node failure mode — usually a kubelet crash, a kubelet certificate expiry, or a network partition between the node and the API server.

---

Failure Mode Catalog

1. kubelet Failure (Most Common)

The kubelet is the node agent. If it stops, the node becomes NotReady within minutes (default heartbeat timeout is ~40 seconds × 5 missed updates = ~5 minutes).

Symptom	Likely Cause	Fix
Node NotReady	kubelet service crashed	systemctl restart kubelet
kubelet won’t start	Invalid /var/lib/kubelet/config.yaml	Validate YAML; regenerate with kubeadm
Certificate expiry	Bootstrap cert expired	kubeadm certs renew all + restart kubelet
Cannot reach API server	Network partition, wrong API endpoint	Check kubelet config server URL
Container runtime unreachable	containerd/CRI-O down	systemctl restart containerd

# Check kubelet status
systemctl status kubelet
 
# Read kubelet logs
journalctl -u kubelet -n 100 --no-pager
 
# Restart kubelet
systemctl restart kubelet
 
# Check kubelet configuration
cat /var/lib/kubelet/config.yaml | grep -E "apiServerURL|clusterDNS|resolvConf"
 
# Check bootstrap certificate
openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates

CKA Note: The exam may give you SSH access to nodes. Know how to restart kubelet and check its logs without relying on kubectl. Source: CKA Day 39

2. Container Runtime Failure

The container runtime (containerd, CRI-O) creates and manages containers. If it fails, the kubelet cannot start any new containers.

Symptom	Likely Cause	Fix
Pods stuck ContainerCreating	containerd service crashed	systemctl restart containerd
Runtime not responding	containerd socket corrupted	Restart containerd; check /run/containerd/containerd.sock
CNI plugin not ready	Missing /etc/cni/net.d/ config	Install CNI or verify CNI DaemonSet
Image pull fails	Registry unreachable	Check network, DNS, and imagePullSecrets

# Check container runtime status
systemctl status containerd
 
# Read runtime logs
journalctl -u containerd -n 100
 
# Restart runtime
systemctl restart containerd
 
# Check CNI configuration on the node
ls -la /etc/cni/net.d/
cat /etc/cni/net.d/10-calico.conflist

3. CNI / Network Plugin Failure

If the CNI plugin is missing or misconfigured, Pods cannot get IP addresses.

Symptom	Likely Cause	Fix
Pods stuck ContainerCreating	CNI config missing	Install CNI plugin
Node NetworkUnavailable	CNI DaemonSet missing on this node	Check CNI pods, may need to recreate
Pods have no IP (kubectl get pod -o wide)	CNI failed to allocate	Check CNI pod logs in kube-system
Cross-node traffic broken	Wrong CNI config or routing	Verify CNI overlay/BGP configuration

# Check CNI pods
kubectl get pods -n kube-system | grep -E "calico|flannel|cilium|weave"
 
# Check CNI pod logs
kubectl logs -n kube-system -l k8s-app=calico-node
 
# Check node CNI config
ls /etc/cni/net.d/
 
# Verify bridge/overlay interfaces on the node
ip addr show
ip route show

CNI Trap: A node that has been drained, upgraded, or rebooted may lose its CNI config. The CNI DaemonSet should recreate it, but if the DaemonSet itself is missing or crashlooping, the node stays NetworkUnavailable. Source: CKA Day 26

4. kube-proxy Failure

kube-proxy manages the iptables/IPVS rules that route Service traffic to backend Pods.

Symptom	Likely Cause	Fix
Services work on some nodes but not others	kube-proxy Pod crashed on one node	Restart kube-proxy Pod or DaemonSet
Connection refused to Service IP	iptables rules missing	Check kube-proxy logs; restart kube-proxy
IPVS table empty	kube-proxy in IPVS mode but table not programmed	Restart kube-proxy; verify mode: ipvs

# Check kube-proxy Pod status
kubectl get pods -n kube-system -l k8s-app=kube-proxy
 
# Read kube-proxy logs
kubectl logs -n kube-system -l k8s-app=kube-proxy
 
# Check iptables rules on the node
iptables -t nat -L KUBE-SERVICES | head -20
 
# Check IPVS virtual services (if using IPVS mode)
ipvsadm -Ln

5. Resource Pressure Evictions

When a node exhausts disk, memory, or PIDs, the kubelet automatically evicts Pods to reclaim resources.

Pressure Type	Trigger	Eviction Order
DiskPressure	Available disk < threshold (default ~15% free)	BestEffort → Burstable → Guaranteed
MemoryPressure	Available memory < threshold	BestEffort → Burstable
PIDPressure	Running processes approaching limit	BestEffort

How kubelet decides what to evict:

1. Sort Pods by QoS class (BestEffort first, Burstable second, Guaranteed last)
2. Within the same QoS class, sort by resource usage vs request (higher overuse = evicted first)
3. If all else is equal, sort by Pod age (newest first)

# Check disk usage
df -h /var/lib/kubelet
df -h /
 
# Check memory
free -h
 
# Check PID count
ps aux | wc -l
cat /proc/sys/kernel/pid_max
 
# Check kubelet eviction configuration
grep -A 5 evictionHard /var/lib/kubelet/config.yaml

QoS Memory: Guaranteed Pods are never evicted for memory pressure (unless they exceed their limits and are OOMKilled). BestEffort Pods are evicted first. This is why production workloads should always declare requests and limits. Source: CKA Day 16

6. Built-in Pressure Taints

When resource pressure crosses critical thresholds, the kubelet automatically applies taints to the node:

Taint	Effect	Applied When
node.kubernetes.io/disk-pressure	NoSchedule + NoExecute	DiskPressure=True
node.kubernetes.io/memory-pressure	NoSchedule + NoExecute	MemoryPressure=True
node.kubernetes.io/pid-pressure	NoSchedule	PIDPressure=True
node.kubernetes.io/not-ready	NoExecute	Ready=False
node.kubernetes.io/unreachable	NoExecute	Node unreachable from API server

These taints are managed by the kubelet and the node controller — you do not apply them manually. They are the mechanism behind automatic eviction. Only Pods with matching tolerations survive.

CKA Exam Trap: The exam may ask why a Pod was evicted. The answer is almost always resource pressure + the node’s automatic taint application. Check kubectl describe node for DiskPressure or MemoryPressure conditions. Source: CKA Day 14

---

The Layered Node Failure Model

Work through the stack from bottom to top when a node is unhealthy:

Layer 1: Hardware / OS      ──► Is the VM/server up? Can you SSH?
Layer 2: Systemd Services     ──► Is kubelet running? Is containerd running?
Layer 3: kubelet              ──► Is it heartbeating? Are certificates valid?
Layer 4: Container Runtime    ──► Can it create containers? Is CNI configured?
Layer 5: CNI / Networking     ──► Do Pods get IPs? Is cross-node traffic working?
Layer 6: kube-proxy           ──► Are Service iptables rules programmed?
Layer 7: Resource Pressure    ──► Is the node evicting Pods?

---

CKA Exam Speed Patterns

# 1. Quick node status check
kubectl get nodes -o wide
 
# 2. Read the node's story
kubectl describe node <name> | grep -A 20 Conditions
kubectl describe node <name> | grep -A 20 Events
 
# 3. SSH and check services
systemctl status kubelet
systemctl status containerd
 
# 4. Read logs
journalctl -u kubelet -n 50
journalctl -u containerd -n 50
 
# 5. Check resource pressure
df -h /var/lib/kubelet
free -h
 
# 6. Check containers on the node
crictl ps -a
 
# 7. Check CNI
ls /etc/cni/net.d/
 
# 8. Check kube-proxy / iptables
iptables -t nat -L KUBE-SERVICES | head -10

Time Management: The exam rewards knowing the exact systemd service names (kubelet, containerd) and the exact config paths (/var/lib/kubelet/config.yaml, /etc/cni/net.d/). Memorize these.

---

Practical Practice

Exam-style hands-on tasks for this topic. Complete each task before reviewing the solution. Time yourself — CKA tasks average 5–7 minutes.

Task 1: Restart a NotReady kubelet A node is NotReady — SSH to the node, check kubelet status with systemctl, and restart it. Requirements: Do not reboot the node. Verification: kubectl get nodes Solution:

ssh <node>
systemctl status kubelet
journalctl -u kubelet -n 50
sudo systemctl restart kubelet
exit
kubectl get nodes

Task 2: Clean Up DiskPressure A node shows DiskPressure taint — clean up unused images and verify with kubectl describe node. Requirements: Free disk space without deleting running Pods. Verification: kubectl describe node <node> | grep -A 5 Conditions Solution:

ssh <node>
df -h /var/lib/kubelet
sudo crictl rmi --prune
sudo journalctl --vacuum-time=1d
exit
kubectl describe node <node> | grep -A 5 Conditions

Task 3: Verify CNI and kube-proxy kube-proxy is failing — check the logs and verify the CNI configuration. Requirements: Check both kube-proxy Pod logs and node CNI config. Verification: kubectl get pods -n kube-system -l k8s-app=kube-proxy Solution:

kubectl logs -n kube-system -l k8s-app=kube-proxy
ssh <node>
ls /etc/cni/net.d/
cat /etc/cni/net.d/10-calico.conflist
systemctl status containerd
exit
kubectl get nodes -o wide

---

Related Pages

• Kubernetes Architecture — worker node component deep-dive: kubelet, kube-proxy, container runtime
• Kubernetes Control Plane Troubleshooting — Day 38: control plane failures vs worker node failures
• Kubernetes Application Troubleshooting — Day 37: app-level debugging on healthy nodes
• Kubernetes Logging and Monitoring — journalctl, crictl logs, and node-level log paths
• Kubernetes Taints and Tolerations — built-in pressure taints and eviction toleration patterns
• Kubernetes Resource Requests and Limits — QoS classes, OOMKilled, and eviction order
• Kubernetes CNI — CNI plugin installation, configuration, and troubleshooting
• Kubernetes Network Policies — network traffic rules that depend on CNI being functional
• Kubernetes DaemonSet — CNI agents and kube-proxy run as DaemonSets on every node
• Node Maintenance — manual node evacuation vs automatic eviction
• Kubeadm Cluster Setup — worker node installation, kubelet configuration, and bootstrap token workflow
• Kubernetes Health Probes — how kubelet probe failures interact with node-level health
• Kubernetes Static Pods — node-local Pod management (used for control plane, not workers)
• Kubernetes Storage — node disk usage and /var/lib/kubelet space management
• CKA Certification — exam domain mapping: Troubleshooting (~30%)
• CKA Study Roadmap — Day 39 in the 40-day plan
• Tech Tutorials with Piyush — course source

---

Tags: kubernetes worker-node troubleshooting kubelet node-failure resource-pressure eviction cni kube-proxy container-runtime cka devops