How to diagnose and recover from failures in the Kubernetes control plane — the brain of the cluster. A high-stakes Troubleshooting topic on the CKA exam and the most critical production skill. Synthesized from CKA Day 38 — Troubleshooting Control Plane Failure.
The Control Plane Failure Mindset
When the control plane fails, the entire cluster stops making decisions. Existing Pods may keep running (because kubelets on worker nodes operate independently for a short time), but no new Pods can be scheduled, no crashed Pods can be replaced, and no configuration changes can be applied.
Golden Rule: Control plane components on kubeadm clusters run as Static Pods in /etc/kubernetes/manifests/. The kubelet on the control plane node reads these manifests and creates the containers. If a component fails, your first stop is the manifest file, then the container runtime, then the logs. Source: CKA Day 38
The Five Control Plane Components
Component
Static Pod Manifest
If It Fails
kube-apiserver
/etc/kubernetes/manifests/kube-apiserver.yaml
Cluster is frozen — no reads, no writes, no kubectl
etcd
/etc/kubernetes/manifests/etcd.yaml
API server can’t persist state; cluster becomes read-only or fully unresponsive
kube-scheduler
/etc/kubernetes/manifests/kube-scheduler.yaml
New Pods stay Pending forever; existing Pods keep running
No self-healing, no rollout, no EndpointSlice updates, no token rotation
kubelet (on control plane node)
systemd service (not a Static Pod)
Static Pods are not recreated if they crash; node becomes NotReady
Diagnostic Layer 1: Is the Component Running?
Check Static Pod Containers
# On the control plane node — list running control plane containerscrictl ps | grep -E "kube-apiserver|etcd|scheduler|controller"# OR if using Docker runtime:docker ps | grep -E "kube-apiserver|etcd|scheduler|controller"
If a container is missing, the kubelet may have failed to create it from the manifest. Check the kubelet logs next.
Check kubelet Health
# kubelet runs as a systemd service, not a Static Podsystemctl status kubelet# Read recent kubelet logsjournalctl -u kubelet -n 100 --no-pager# If kubelet is down, nothing on the node workssystemctl start kubelet
Check Static Pod Manifest Files
# List manifest files on the control plane nodels -la /etc/kubernetes/manifests/# Verify each manifest is valid YAML and has the correct fieldscat /etc/kubernetes/manifests/kube-apiserver.yaml
The API server is the front door. If it fails, kubectl and all controllers stop working.
Symptom
Likely Cause
Fix
kubectl hangs or “connection refused”
API server container down or certificate expired
Check container status, then check kubeadm certs check-expiration
API server returns 500s
etcd unreachable or unhealthy
Check etcd container and etcdctl endpoint health
curl -k https://localhost:6443/healthz fails
API server not listening on 6443
Read API server container logs
# Direct health check from control plane nodecurl -k https://localhost:6443/healthz# Should return: ok# Direct health check with certificatescurl --cacert /etc/kubernetes/pki/ca.crt \ --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt \ --key /etc/kubernetes/pki/apiserver-kubelet-client.key \ https://localhost:6443/healthz# API server container logscrictl logs $(crictl ps | grep kube-apiserver | awk '{print $1}')
Certificate Trap: The API server has the most certificates of any component (serving, kubelet client, etcd client, aggregator, front-proxy). If any one expires, the API server may fail to start. Source: CKA Day 21
etcd Failures
ETCD is the consistent data store. Without it, the API server cannot read or write any cluster state.
Symptom
Likely Cause
Fix
API server 500s, kubectl apply fails
etcd down or unreachable
Check etcd container, disk space, certificate expiry
etcdctl endpoint health fails
etcd container crashed or data corruption
Check container logs and data directory permissions
Quorum lost (HA clusters)
Majority of etcd nodes down
Restart failed etcd nodes; if data is lost, restore from snapshot
# Check etcd health (requires mTLS flags)ETCDCTL_API=3 etcdctl endpoint health \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key# Check disk space on etcd data directorydf -h /var/lib/etcd# Check etcd container logscrictl logs $(crictl ps | grep etcd | awk '{print $1}')# Check etcd data directory permissionsls -la /var/lib/etcd
ETCD Disk Warning: etcd is sensitive to disk latency. If the disk where /var/lib/etcd lives is slow or full, etcd may enter read-only mode or crash. The etcd data directory must have low-latency storage (SSD recommended). Source: CKA Day 35
kube-scheduler Failures
If the scheduler fails, no new Pods are assigned to nodes.
Symptom
Likely Cause
Fix
All new Pods Pending indefinitely
Scheduler container down or can’t reach API server
Check scheduler container and manifest
No Events about scheduling
Scheduler is not running
Verify manifest exists and container is up
# Check scheduler containercrictl ps | grep scheduler# Scheduler logscrictl logs $(crictl ps | grep scheduler | awk '{print $1}')# Check if scheduler can reach API server# (Look for connection errors in scheduler logs)
kube-controller-manager Failures
If the controller manager fails, the cluster loses its automation loops.
Symptom
Likely Cause
Fix
Dead Pods not replaced
Replication controller loop stopped
Check controller-manager container
Deployment rollout not progressing
Deployment controller stopped
Check controller-manager logs
Service Endpoints not updating
Endpoints controller stopped
Restart controller-manager or fix API connectivity
Control plane components use mutual TLS for all internal communication. When certificates expire, components cannot authenticate each other and the cluster degrades.
# Check all certificate expiration dateskubeadm certs check-expiration# Output shows:# CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED# admin.conf Jul 15, 2027 10:00 UTC 364d ca no# apiserver Jul 15, 2027 10:00 UTC 364d ca no# apiserver-etcd-client Jul 15, 2027 10:00 UTC 364d etcd-ca no# ...
Renewing Certificates
# Renew all certificates managed by kubeadmkubeadm certs renew all# After renewing, restart the affected Static Pods# The easiest way is to move the manifest out and back in:mkdir -p /tmp/manifestsmv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/manifests/sleep 10mv /tmp/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/# kubelet detects the file change and recreates the Pod automatically
Important:kubeadm certs renew updates the certificate files on disk but does not restart the running containers. The Static Pods must be recreated to load the new certificates. Moving the manifest file out and back is the standard restart technique. Source: CKA Day 21
The Control Plane Recovery Workflow
When you encounter a cluster-wide failure, follow this ordered workflow:
1. kubectl get nodes
└─► If all nodes show NotReady, control plane is down
2. SSH to the control plane node
3. systemctl status kubelet
└─► If kubelet is down, start it: systemctl start kubelet
4. ls /etc/kubernetes/manifests/
└─► Verify all four manifest files exist
5. crictl ps | grep -E "apiserver|etcd|scheduler|controller"
└─► Verify all four containers are running
6. journalctl -u kubelet -n 50
└─► Read kubelet errors about failed container creation
7. Check component logs for the failed container
crictl logs <container-id>
8. kubeadm certs check-expiration
└─► If expired, renew and restart Static Pods
9. curl -k https://localhost:6443/healthz
└─► Verify API server responds
10. kubectl get pods -n kube-system
└─► Verify all control plane Pods are Running
Disk space, data directory corruption, certificate expiry
df -h, etcdctl endpoint health, crictl logs
Scheduler missing
Manifest deleted, kubelet down
ls /etc/kubernetes/manifests/, systemctl status kubelet
Controller-manager missing
Manifest deleted, API server unreachable
crictl ps, read container logs
Node shows NotReady
kubelet down on that node
systemctl status kubelet on the node
kubectl times out
API server down or network partition
curl -k https://<cp-ip>:6443/healthz
CKA Exam Speed Patterns
# Check control plane pod status from inside the clusterkubectl get pods -n kube-system# Check from the control plane node directlycrictl ps -a | grep kube-system# Read any container's logscrictl logs <container-id># Restart a Static Pod by touching its manifesttouch /etc/kubernetes/manifests/kube-apiserver.yaml# Check certificate status quicklykubeadm certs check-expiration | grep -i expired# Verify API server health without kubectlcurl -k https://localhost:6443/healthz
Exam Tip: The CKA exam environment may give you SSH access to nodes. Know how to check component status without relying on kubectl because kubectl itself requires the API server. crictl, journalctl, and curl are your fallback tools.
Practical Practice
Exam-style hands-on tasks for this topic. Complete each task before reviewing the solution. Time yourself — CKA tasks average 5–7 minutes.
Task 1: Fix API Server Invalid Port
The API server is not responding — check the Static Pod manifest in /etc/kubernetes/manifests/ and fix the invalid port.
Requirements: Do not restart the node. Use kubelet manifest detection.
Verification:curl -k https://localhost:6443/healthzSolution:
cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep -i port# edit the manifest to correct the port (e.g., --secure-port=6443)# kubelet auto-restarts the Static Pod after the manifest changecurl -k https://localhost:6443/healthzkubectl get pods -n kube-system
Task 2: Fix etcd Data Directory Permissions
etcd is failing to start — inspect the etcd container logs and fix the data directory permissions.
Requirements: SSH to the control plane node. Do not delete etcd data.
Verification:ETCDCTL_API=3 etcdctl endpoint health ...Solution:
crictl logs $(crictl ps | grep etcd | awk '{print $1}')ls -la /var/lib/etcdsudo chown -R etcd:etcd /var/lib/etcd # or root:root depending on setup# touch /etc/kubernetes/manifests/etcd.yaml to trigger kubelet restartETCDCTL_API=3 etcdctl endpoint health \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key
Task 3: Verify kube-apiserver and Certificateskubectl get nodes hangs — verify kube-apiserver is running and the certificates have not expired.
Requirements: Use node-local tools only (crictl, journalctl, kubeadm).
Verification:kubectl get nodesSolution:
crictl ps | grep kube-apiserverjournalctl -u kubelet -n 50kubeadm certs check-expiration# if expired:kubeadm certs renew all# restart static pod by moving manifest out and back inmv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/sleep 10mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/kubectl get nodes
Related Pages
Kubernetes Architecture — control plane component deep-dive, communication flows, and HA patterns
Kubernetes Static Pods — the mechanism that runs control plane components on kubeadm clusters