Kubernetes Application Troubleshooting

A systematic, layer-by-layer guide to diagnosing why applications fail in Kubernetes — from image pull errors to rollout stalls. This is the single most tested skill in the Troubleshooting domain (~30%) of the CKA exam. Synthesized from CKA Day 37 — Application Failure Troubleshooting.

The Debugging Mindset

Kubernetes failures are rarely mysterious — they are usually the result of a misconfiguration that Kubernetes reports clearly. The key is knowing where to look and in what order. The CKA exam rewards methodical inspection over intuition.

Golden Rule: Events First. kubectl describe pod Events are ordered chronologically and contain human-readable explanations of failures. Read them before diving into logs or interactive shells. Source: CKA Day 37

---

The Standard Debugging Chain

Step 1: kubectl get pods -o wide
        └─→ Check STATUS, RESTARTS, NODE, AGE

Step 2: kubectl describe pod <name>
        └─→ Read Events (bottom of output)

Step 3: kubectl logs <name> [--previous] [-c <container>]
        └─→ See what the app actually output

Step 4: kubectl exec -it <name> -- sh
        └─→ Interactive inspection inside the container

Step 5: kubectl get events --sort-by='.lastTimestamp'
        └─→ Cluster-wide context if the Pod keeps disappearing

---

Failure Mode Catalog

1. Image Pull Failures (ImagePullBackOff)

The kubelet cannot download the container image. The Pod status shows ImagePullBackOff with exponential backoff retries.

Cause	How to Detect	Fix
Wrong image name/tag	kubectl describe pod → “Failed to pull image”	kubectl set image deployment/<name> <container>=<correct>:<tag>
Private registry, no credentials	kubectl describe pod → “unauthorized”	Add imagePullSecrets to Pod or ServiceAccount
Registry unreachable	kubectl describe pod → network timeout	Check DNS, NetworkPolicy, or node internet access

# Fast image correction
kubectl set image deployment/nginx nginx=nginx:1.25-alpine
 
# Verify the pull spec
kubectl get pod <name> -o jsonpath='{.spec.containers[0].image}'

2. Crash Loops (CrashLoopBackOff)

The container starts, crashes, and is restarted repeatedly. The RESTARTS count climbs in kubectl get pod.

Cause	How to Detect	Fix
Missing env var	kubectl logs --previous → “DATABASE_URL not set”	Add env to Pod spec or ConfigMap
Bad command/args	kubectl logs --previous → command not found	Fix command/args in container spec
Missing ConfigMap/Secret	kubectl describe pod → “configmap not found”	Create the referenced object
Aggressive liveness probe	kubectl describe pod → liveness probe failed	Add initialDelaySeconds or a startup probe
OOM on startup	kubectl describe pod → Last State: OOMKilled	Raise memory limit or fix memory leak

# The three crash-loop diagnostics
kubectl get pod <name>                    # Check RESTARTS
kubectl describe pod <name>             # Check Events and Last State
kubectl logs <name> --previous          # Read the crash output

3. Pods Stuck Pending

The scheduler cannot place the Pod on any node. The Pod is accepted by the API server but never starts running.

Cause	How to Detect	Fix
Insufficient CPU/memory	kubectl describe pod → “Insufficient cpu” or “Insufficient memory”	Lower requests or add nodes
No matching node for taints	kubectl describe pod → taint toleration mismatch	Add tolerations or remove taints
PVC not bound	kubectl describe pod → “unbound immediate PersistentVolumeClaim”	Check PVC status and StorageClass
Node selector mismatch	kubectl describe pod → “0/3 nodes are available”	Verify node labels match selector

4. ContainerCannotRun / Volume Mount Failures

The container cannot start because a required volume or mount is missing or misconfigured.

Cause	How to Detect	Fix
ConfigMap/Secret missing	kubectl describe pod → “MountVolume.SetUp failed”	Create the ConfigMap/Secret
Wrong mount path	Container starts but app can’t find files	Verify mountPath matches app expectations
Permission denied	kubectl describe pod → permission errors	Check fsGroup, runAsUser, or volume permissions

5. Service Not Routing Traffic

Pods are Running but clients get connection refused, timeouts, or 502 errors.

Cause	How to Detect	Fix
Selector mismatch	kubectl get endpoints <svc> shows no IPs	Ensure Service selector matches Pod labels exactly
Readiness probe failing	kubectl get endpoints → target IPs missing	Fix readiness probe or application startup
Port mismatch	kubectl describe svc vs kubectl get pod	Ensure targetPort matches container containerPort
NetworkPolicy blocking	Traffic times out	Add ingress rule allowing client to backend
DNS failure	nslookup <svc> fails	Check CoreDNS logs and Service name/namespace

# The service diagnosis chain
kubectl get svc <name> -o yaml | grep selector     # Verify selector
kubectl get endpoints <name>                       # Verify backend IPs
kubectl describe pod <backend-pod>                 # Check readiness state
kubectl exec -it client-pod -- nslookup <svc>      # Verify DNS

6. Deployment Rollout Stuck

A rolling update makes no progress — old Pods keep running, new Pods never become ready.

Cause	How to Detect	Fix
Readiness probe never passes	kubectl rollout status hangs	Fix app or probe configuration
Resource quota exhausted	kubectl describe pod → “Forbidden: exceeded quota”	Raise Namespace ResourceQuota or reduce replicas
ImagePullBackOff on new Pods	New Pods show ImagePullBackOff	Correct image name in Deployment
MaxSurge / MaxUnavailable conflict	No Pods can be created or terminated	Adjust Deployment strategy

# Rollout diagnostics
kubectl rollout status deployment/<name>
kubectl rollout history deployment/<name>
kubectl get rs -l app=<name>                       # See ReplicaSet generations
kubectl get pod -l app=<name>                      # Check new Pod status

7. Init Container Failures (Init:Error / Init:CrashLoopBackOff)

An init container exits non-zero, blocking the main containers from ever starting.

Cause	How to Detect	Fix
Database not ready	kubectl logs <pod> -c <init> → connection refused	Add retry logic or fix database connectivity
Missing permission	Init container can’t write to shared volume	Check volume permissions and securityContext
Wrong command	kubectl logs <pod> -c <init> → command not found	Fix init container command/args

# Init container diagnostics
kubectl get pod <name>                             # Status shows "Init:Error"
kubectl describe pod <name>                      # Which init container failed
kubectl logs <name> -c <init-container>          # Read the failure output

---

Events: The Cluster-Wide Audit Trail

Events are the most underutilized debugging resource. They persist for a limited time (default ~1 hour) and capture everything Kubernetes does:

# All events, newest last
kubectl get events --sort-by='.lastTimestamp'
 
# Events for a specific object
kubectl get events --field-selector involvedObject.name=<pod-name>
 
# Only warnings (failures)
kubectl get events --field-selector type=Warning
 
# Events for a namespace
kubectl get events -n <namespace>

Common Event Messages and Their Meaning:

Event Message	Meaning	Action
Failed to pull image	Image name/tag/registry issue	Correct the image spec
Back-off pulling image	Same as above, after retries	Same fix
FailedMount	Volume could not attach or mount	Check PVC, node, or CSI driver
FailedScheduling	No node matches the Pod	Check requests, taints, selectors
Unhealthy	Liveness probe failed	Fix probe or app
Killing	Pod is being terminated	Normal, or due to eviction/node drain
Created container	Container started successfully	Good sign
Started container	Container process is running	Good sign

---

The Layered Failure Model

When an application is failing, work through the stack from bottom to top:

Layer 1: Node          ──► Is the node Ready? kubelet healthy? (journalctl -u kubelet)
Layer 2: Pod Schedule  ──► Is the Pod Running? Or Pending? (kubectl get pod)
Layer 3: Container     ──► Is the container alive? (kubectl describe → Events)
Layer 4: Application   ──► Is the app logic healthy? (kubectl logs, kubectl exec)
Layer 5: Service       ──► Can clients reach it? (kubectl get endpoints, nslookup)
Layer 6: Ingress       ──► Is external traffic routing? (kubectl get ingress, controller logs)

CKA Exam Strategy: The exam presents broken applications and asks you to fix them. You won’t have time to guess. Use this chain: get → describe → logs → fix → verify.

---

CKA Exam Speed Patterns

# 1. Quick status check
kubectl get pods -o wide -n <namespace>
 
# 2. Read the story in Events
kubectl describe pod <name> | grep -A 20 Events
 
# 3. Crash analysis
kubectl logs <name> --previous
 
# 4. Multi-container crash
kubectl logs <name> -c <container> --previous
 
# 5. Interactive debugging
kubectl exec -it <name> -- env | grep <VAR>
kubectl exec -it <name> -- cat /etc/config/app.yaml
 
# 6. Service routing
kubectl get svc <name> -o yaml | grep -A 3 selector
kubectl get endpoints <name>
 
# 7. Rollout status
kubectl rollout status deployment/<name>
kubectl rollout undo deployment/<name>  # Emergency rollback
 
# 8. Events stream
kubectl get events --sort-by='.lastTimestamp' | tail -20

---

Production Best Practices

Practice	Rationale
Always check Events first	They contain the exact error from the controller or kubelet
Use --previous for crashes	The current container may not have logged the failure
Validate selectors match labels	A single typo (app: ngnix vs app: nginx) breaks Service routing
Test readiness before liveness	An app that is alive but not ready should not receive traffic
Set resource requests AND limits	Missing limits cause OOMKilled; missing requests cause scheduling issues
Use kubectl apply --dry-run=client	Catch YAML errors before they create broken Pods
Monitor Events centrally	Tools like kubectl get events or cluster event exporters surface patterns

---

Practical Practice

Exam-style hands-on tasks for this topic. Complete each task before reviewing the solution. Time yourself — CKA tasks average 5–7 minutes.

Task 1: Fix ImagePullBackOff A Pod web-1 is in ImagePullBackOff — identify the wrong image tag and fix it. Requirements: Use only imperative commands. Do not delete the Pod. Verification: kubectl get pod web-1 Solution:

kubectl describe pod web-1 | grep -i "Failed to pull image"
kubectl set image pod/web-1 web=nginx:1.25-alpine
kubectl get pod web-1

Task 2: Unstick a Deployment Rollout A Deployment rollout is stuck — inspect the ReplicaSet events and fix the resource limit typo (limts → limits). Requirements: Edit the live Deployment. Verify the new ReplicaSet scales up. Verification: kubectl rollout status deployment/<name> Solution:

kubectl get rs -l app=<name>
kubectl describe rs <new-rs> | grep -i "limts"
kubectl edit deployment <name>  # fix limts -> limits
kubectl rollout status deployment/<name>

Task 3: Fix Service targetPort Mismatch A Service returns connection refused — the Pod is listening on port 8080 but the Service targets 80; fix the targetPort. Requirements: Patch the Service without recreating it. Verification: kubectl get svc <svc> -o yaml | grep targetPort Solution:

kubectl get svc <svc> -o yaml | grep targetPort
kubectl patch svc <svc> --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/targetPort", "value":8080}]'
kubectl get endpoints <svc>

Task 4: Diagnose CrashLoopBackOff A Pod is CrashLoopBackOff — read the previous container logs and fix the missing environment variable. Requirements: Add the missing env var to the Pod spec. Use --dry-run=client -o yaml if regenerating YAML. Verification: kubectl get pod <pod> Solution:

kubectl logs <pod> --previous
kubectl get pod <pod> -o yaml > pod.yaml
# edit pod.yaml to add the missing env var
kubectl apply -f pod.yaml --dry-run=client
kubectl delete pod <pod> && kubectl apply -f pod.yaml

---

Related Pages

• Kubernetes Logging and Monitoring — kubectl logs, node logs, and Metrics Server for resource pressure diagnosis
• Kubernetes Health Probes — liveness, readiness, and startup probe misconfigurations are a leading cause of app failures
• Pod Fundamentals — the unit where failures manifest
• Deployment, ReplicaSet & Replication Controller — rollout failures and self-healing behavior
• Kubernetes Services — selector mismatches and Endpoint diagnostics
• Kubernetes Service Types — port mapping and external access issues
• Kubernetes ConfigMaps and Secrets — missing config objects cause startup crashes
• Kubernetes Resource Requests and Limits — OOMKilled and Pending resource failures
• Kubernetes Network Policies — traffic blocked by policies appears as service failure
• Multi-Container Pods — init container failures and kubectl logs -c patterns
• Init Containers — serial setup failures that block main app startup
• Kubernetes Architecture — control plane components that emit events
• Kubernetes Taints and Tolerations — scheduling blocks from node taints
• Kubernetes Node Affinity — placement constraints that can leave Pods Pending
• Kubernetes Namespaces — ResourceQuota limits that block Pod creation
• CKA Certification — exam domain mapping: Troubleshooting (~30%)
• CKA Study Roadmap — Day 37 in the 40-day plan
• Kubernetes Control Plane Troubleshooting — Day 38: control plane failures vs application failures
• Kubernetes Worker Node Troubleshooting — Day 39: worker node failures that can masquerade as application failures
• Tech Tutorials with Piyush — course source

---

Tags: kubernetes troubleshooting application-failure cka devops diagnostics events crashloopbackoff imagepullbackoff rollout