CKA Day 39 — Troubleshooting Worker Nodes Failures

Day 39 of the 40-day Certified Kubernetes Administrator course by Tech Tutorials with Piyush. This session covers how to diagnose and recover from worker node failures — the second major Troubleshooting topic after control plane failures.

Key Concepts Covered

The Worker Node Failure Chain

When a worker node fails, the symptoms cascade outward:

1. Node status changes to NotReady
2. Pods on that node are marked for eviction (if NoExecute taints are present)
3. New Pods cannot be scheduled to the node
4. Existing Pods may continue running briefly but lose connectivity to Services

The video teaches a systematic node-level debugging workflow:

kubectl get nodes                      ← Identify the failed node
kubectl describe node <name>           ← Read Conditions, Events, and Allocatable
SSH to the node                        ← Node-level diagnosis required
systemctl status kubelet               ← Check if kubelet is running
journalctl -u kubelet -n 100           ← Read kubelet logs
systemctl status containerd            ← Check container runtime
crictl ps                              ← Check running containers on the node

Node Conditions and What They Mean

kubectl describe node shows a set of Conditions that reveal the node’s health:

Condition	True	False	Unknown
Ready	Node healthy, can accept Pods	Node unhealthy or kubelet down	kubelet hasn’t reported status
DiskPressure	Disk is low; node is evicting Pods	Disk is fine	—
MemoryPressure	Memory is low; node is evicting Pods	Memory is fine	—
PIDPressure	Too many PIDs; node is evicting Pods	PID count normal	—
NetworkUnavailable	CNI not configured	Network is ready	—

CKA Insight: If Ready is Unknown, the kubelet has stopped heartbeating to the API server. This is usually a kubelet crash or network partition.

kubelet Failures

The kubelet is the node agent. If it fails, the node becomes NotReady:

• Symptom: Node shows NotReady; kubectl get pods shows Pods on that node as Unknown
• Causes: kubelet service crashed, kubelet configuration error, container runtime unreachable, certificate expiry
• Diagnosis:
  • systemctl status kubelet
  • journalctl -u kubelet -n 100
  • Check /var/lib/kubelet/config.yaml for misconfiguration
  • Check bootstrap certificate: /var/lib/kubelet/pki/

# Restart kubelet
systemctl restart kubelet
 
# Check kubelet configuration
kubeadm kubelet config print-initial --kubernetes-version=v1.30.0
 
# Check bootstrap certificate
cat /var/lib/kubelet/pki/kubelet-client-current.pem | openssl x509 -noout -dates

Container Runtime Failures

The container runtime (containerd, CRI-O) creates and runs containers. If it fails, the kubelet cannot start any new Pods:

• Symptom: Pods stuck ContainerCreating on one node; kubectl describe pod shows “container runtime not ready”
• Causes: containerd service crashed, CNI plugin not installed, overlay filesystem error
• Diagnosis:
  • systemctl status containerd
  • journalctl -u containerd -n 100
  • Check CNI configuration: /etc/cni/net.d/

# Restart container runtime
systemctl restart containerd
# or
systemctl restart crio
 
# Check CNI plugin files
ls /etc/cni/net.d/
cat /etc/cni/net.d/10-calico.conflist

CNI Plugin Failures

If the CNI plugin is missing or misconfigured, Pods get no IP address and stay in ContainerCreating:

• Symptom: Pods on one node stuck ContainerCreating or Pending
• Causes: CNI plugin not installed, wrong CNI config, CNI DaemonSet missing on the node
• Diagnosis:
  • Check CNI pods: kubectl get pods -n kube-system | grep calico (or flannel/cilium)
  • Check node Ready condition for NetworkUnavailable
  • Check CNI config on the node: ls /etc/cni/net.d/

kube-proxy Failures

kube-proxy manages Service-to-Pod routing rules. If it fails, Services may not route traffic to Pods on that node:

• Symptom: Services work on some nodes but not others; intermittent connection failures
• Causes: kube-proxy Pod crashed, iptables rules corrupted, IPVS misconfigured
• Diagnosis:
  • Check kube-proxy Pod: kubectl get pods -n kube-system -l k8s-app=kube-proxy
  • Check kube-proxy logs: kubectl logs -n kube-system -l k8s-app=kube-proxy
  • Check iptables rules: iptables -t nat -L KUBE-SERVICES on the node

Resource Pressure Evictions

When a node runs out of disk, memory, or PIDs, the kubelet begins evicting Pods to reclaim resources:

Pressure Type	Trigger	Eviction Order
DiskPressure	/var/lib/kubelet or root partition > 85%	Best-effort, then Burstable, then Guaranteed
MemoryPressure	Available memory < threshold	Best-effort, then Burstable
PIDPressure	PIDs used > threshold	Best-effort

QoS Class Matters: Guaranteed Pods are evicted last. BestEffort Pods are evicted first. This is why production workloads should always set resource requests and limits. Source: CKA Day 16

# Check disk usage on the node
df -h
 
# Check memory usage
free -h
 
# Check PID count
ps aux | wc -l
cat /proc/sys/kernel/pid_max

Node Eviction and Taints

When resource pressure reaches critical levels, the kubelet applies taints automatically:

• node.kubernetes.io/disk-pressure:NoSchedule — new Pods blocked
• node.kubernetes.io/memory-pressure:NoSchedule — new Pods blocked
• node.kubernetes.io/disk-pressure:NoExecute — existing Pods evicted
• node.kubernetes.io/memory-pressure:NoExecute — existing Pods evicted

These are built-in taints that the kubelet manages automatically. They connect directly to the taints/tolerations concepts from Day 14.

Tools for Worker Node Debugging

# Node status overview
kubectl get nodes -o wide
kubectl describe node <node-name>
 
# SSH to node for local diagnosis
ssh user@worker-node-ip
 
# Check kubelet
systemctl status kubelet
journalctl -u kubelet -n 100
 
# Check container runtime
systemctl status containerd
journalctl -u containerd -n 100
 
# Check running containers
crictl ps -a
 
# Check CNI
ls /etc/cni/net.d/
cat /var/log/containers/*.log
 
# Check iptables rules for Services
iptables -t nat -L KUBE-SERVICES | head -20
 
# Check resource pressure
df -h /var/lib/kubelet
free -h
cat /proc/loadavg

Synthesis

Day 39 completes the Troubleshooting trilogy (Days 37–39): application failures, control plane failures, and worker node failures. The CKA exam expects fluency in SSH-ing to nodes, reading systemd journal logs, inspecting container runtime state, and understanding how kubelet heartbeats translate into Node Ready conditions. Resource pressure evictions tie directly to the QoS classes and resource limits from earlier in the course.

See Also

Wiki Concepts

• Kubernetes Worker Node Troubleshooting — synthesized concept page with full failure catalog
• Kubernetes Control Plane Troubleshooting — Day 38: control plane failures
• Kubernetes Application Troubleshooting — Day 37: app-level failures
• Kubernetes Architecture — worker node components: kubelet, kube-proxy, container runtime
• Kubernetes Logging and Monitoring — journalctl, crictl logs, and node-level log paths
• Kubernetes Taints and Tolerations — built-in pressure taints and eviction behavior
• Kubernetes Resource Requests and Limits — resource pressure thresholds and QoS classes
• Kubernetes CNI — CNI plugin installation and network troubleshooting
• Node Maintenance — manual node evacuation vs automatic eviction
• Kubeadm Cluster Setup — worker node installation and kubelet configuration
• Kubernetes Health Probes — liveness/readiness and node health interaction
• CKA Certification — exam domain mapping

Related Sources

• CKA Day 38 — Troubleshooting Control Plane Failure — preceding day in the series
• CKA Day 37 — Application Failure Troubleshooting — app-level debugging
• CKA Day 16 — Kubernetes Requests and Limits — QoS classes and eviction order
• CKA Day 27 — Setup a Multi Node Kubernetes Cluster Using Kubeadm — worker node installation
• CKA Day 14 — Taints and Tolerations in Kubernetes — pressure taints and eviction

Creator / Entity

• Tech Tutorials with Piyush — CKA course creator