CKA Day 38 — Troubleshooting Control Plane Failure

Day 38 of the 40-day Certified Kubernetes Administrator course by Tech Tutorials with Piyush. This session covers how to diagnose and recover from control plane component failures — a high-stakes Troubleshooting topic and a core CKA exam skill.

Key Concepts Covered

The Control Plane as Static Pods

On a kubeadm cluster, the control plane components (kube-apiserver, kube-scheduler, kube-controller-manager, etcd) run as Static Pods managed by the kubelet. Their manifest files live in /etc/kubernetes/manifests/ on the control plane node. If a control plane component fails:

1. Check the manifest file exists and is valid YAML
2. Check the container is running with crictl ps or docker ps
3. Check the component’s logs on the node filesystem or via docker logs/crictl logs
4. Check journalctl -u kubelet for kubelet-level errors

kube-apiserver Failures

The API server is the front door. If it fails, the cluster is effectively frozen:

• Symptom: kubectl commands hang or return “connection refused”
• Causes: Certificate expiry, misconfigured etcd endpoint, port conflict, manifest YAML error
• Diagnosis:
  • curl -k https://localhost:6443/healthz from the control plane node
  • docker logs or crictl logs on the API server container
  • Check /etc/kubernetes/pki/ certificate expiration dates
  • Verify etcd endpoint in the manifest matches the actual etcd location

etcd Failures

ETCD stores all cluster state. A failed etcd means no state changes can be written:

• Symptom: API server returns 500 errors; kubectl get may still work (reads from cache) but kubectl apply fails
• Causes: Disk full, quorum loss (in HA), certificate expiry, corrupted data directory
• Diagnosis:
  • ETCDCTL_API=3 etcdctl endpoint health --endpoints=https://127.0.0.1:2379 --cacert=... --cert=... --key=...
  • Check disk space on the control plane node: df -h /var/lib/etcd
  • Check etcd container logs
  • Verify etcd data directory permissions

kube-scheduler Failures

If the scheduler fails, new Pods remain Pending forever:

• Symptom: All new Pods stuck in Pending; kubectl describe pod shows no scheduling events
• Causes: Manifest missing, scheduler crash, API server unreachable from scheduler
• Diagnosis:
  • Check if the scheduler Pod is running in kube-system
  • Check scheduler logs for connection errors to the API server
  • Verify the scheduler manifest exists in /etc/kubernetes/manifests/

kube-controller-manager Failures

If the controller manager fails, the cluster loses its automation:

• Symptom: Pods that die are not replaced; Deployments don’t roll out; Services don’t update Endpoints
• Causes: Manifest missing, API server unreachable, certificate issues
• Diagnosis:
  • Check if controller-manager Pod is running in kube-system
  • Check logs for API connection errors
  • Verify leader election (in HA setups) is functioning

Certificate Expiry

Control plane components communicate via mutual TLS. When certificates expire, components can’t talk:

• Check certificate expiry: kubeadm certs check-expiration
• Renew: kubeadm certs renew all
• Restart affected Static Pods after renewal (kubelet auto-restarts when manifest changes)

Tools for Control Plane Debugging

# Check Static Pod containers
crictl ps | grep kube-apiserver
crictl ps | grep etcd
 
# Read component logs from container runtime
crictl logs <container-id>
 
# Check kubelet status and logs
systemctl status kubelet
journalctl -u kubelet -n 100
 
# Check certificate expiry
kubeadm certs check-expiration
 
# Direct API server health check
curl -k https://localhost:6443/healthz
 
# Check all control plane pods
kubectl get pods -n kube-system

Synthesis

Day 38 is the capstone of the Troubleshooting domain. It brings together Static Pods (Day 13), cluster architecture (Day 5), certificate management (Days 20–21), etcd backup/restore (Day 35), and cluster upgrades (Day 34) into a unified control plane failure recovery framework. The CKA exam expects candidates to SSH into control plane nodes, inspect manifest files, read container runtime logs, and use kubeadm certificate commands to restore control plane health.

See Also

Wiki Concepts

• Kubernetes Control Plane Troubleshooting — synthesized concept page with full failure catalog and recovery procedures
• Kubernetes Architecture — control plane component deep-dive and communication flows
• Kubernetes Static Pods — the mechanism that runs control plane components
• Kubernetes ETCD Backup and Restore — etcd failure recovery and disaster recovery
• Kubeadm Cluster Setup — control plane installation and certificate paths
• Kubernetes Cluster Upgrade — upgrades modify Static Pod manifests and can cause control plane issues
• Node Maintenance — draining control plane nodes for maintenance
• Kubernetes Logging and Monitoring — reading component logs via journalctl and crictl
• Kubernetes Application Troubleshooting — Day 37: app-level failures vs control plane failures
• TLS Fundamentals — certificate expiry and mTLS between control plane components
• CKA Certification — exam domain mapping

Related Sources

• CKA Day 37 — Application Failure Troubleshooting — preceding day in the series
• CKA Day 35 — Kubernetes ETCD Backup And Restore Explained — etcd backup/restore prerequisite
• CKA Day 34 — Cluster Upgrade With Kubeadm — upgrade mechanics that affect control plane
• CKA Day 27 — Setup a Multi Node Kubernetes Cluster Using Kubeadm — original control plane installation
• CKA Day 39 — Troubleshooting Worker Nodes Failures — Next day in the series: worker node failure diagnosis

Creator / Entity

• Tech Tutorials with Piyush — CKA course creator