CKA Day 36 — Kubernetes Logging and Monitoring

Day 36 of the 40-day Certified Kubernetes Administrator course by Tech Tutorials with Piyush. This session covers how to access, interpret, and troubleshoot Kubernetes logs at the container, node, and cluster levels, plus resource monitoring via the Metrics Server and kubectl top.

Key Concepts Covered

Container Logs with kubectl logs

Kubernetes captures the stdout and stderr streams of every container and makes them accessible via kubectl logs. This is the first line of defense when a Pod misbehaves:

• kubectl logs <pod> — stream logs from the default container
• kubectl logs <pod> -c <container> — specify a container in a multi-container Pod
• kubectl logs <pod> --previous — retrieve logs from the previous container instance after a crash or restart
• kubectl logs <pod> --tail=50 — show only the last 50 lines
• kubectl logs <pod> -f — follow (tail) logs in real time

CKA Trap: If a Pod has restarted multiple times, kubectl logs only shows the current container. Use --previous to see why the last container exited, which is essential for diagnosing CrashLoopBackOff.

Node-Level Log Locations

Container runtimes (containerd, CRI-O) write logs to the node filesystem. On a standard kubeadm cluster:

• /var/log/containers/ — symlinks to actual log files, organized as <pod-name>_<namespace>_<container-name>-<container-id>.log
• /var/log/pods/ — pod-level log directories
• journalctl -u kubelet — kubelet service logs (critical for NotReady node diagnosis)
• journalctl -u containerd — container runtime logs

The kubelet on each node is responsible for rotating and forwarding these logs. In exam scenarios where kubectl logs is unavailable (e.g., API server down), SSH into the node and read files directly.

Cluster-Level Logging Architecture

While kubectl logs is sufficient for ad-hoc debugging, production clusters require centralized log aggregation:

┌─────────────┐     ┌─────────────┐     ┌─────────────────┐
│   Node 1    │────▶│  Fluentd/   │────▶│  Central Store  │
│  /var/log   │     │  Fluent Bit │     │ (Elasticsearch, │
└─────────────┘     └─────────────┘     │  Loki, S3)      │
┌─────────────┘                         └─────────────────┘
│   Node 2    │
│  /var/log   │
└─────────────┘

• Node agent (DaemonSet): Fluentd, Fluent Bit, or Logstash reads /var/log/containers/ and forwards to a central store
• Sidecar pattern: For applications that don’t write to stdout, a sidecar reads log files from a shared emptyDir or hostPath volume and streams them to the collector
• Central store: Elasticsearch, Loki, CloudWatch Logs, or Google Cloud Logging

Metrics Server and Resource Monitoring

The Metrics Server is a cluster-wide aggregator of resource usage data (CPU and memory). It is not installed by default on all clusters, but it is required for autoscaling and live resource observation:

# Install Metrics Server (common exam prerequisite)
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
 
# View node-level consumption
kubectl top node
 
# View Pod-level consumption
kubectl top pod -n <namespace>
 
# View a specific Pod
kubectl top pod <pod-name> --containers

Command	Purpose
kubectl top node	CPU/memory usage per node
kubectl top pod	CPU/memory usage per Pod
kubectl top pod --containers	Breakdown by container within each Pod

Exam Note: kubectl top returns current usage, not historical. For trends and alerting, you need a full monitoring stack (Prometheus + Grafana), but the CKA exam only tests Metrics Server and kubectl top.

Logging for Troubleshooting

The Troubleshooting domain (~30% of the CKA exam) relies heavily on log analysis:

Symptom	Log Source	Command
Pod CrashLoopBackOff	Previous container logs	kubectl logs <pod> --previous
Pod stuck Pending	Scheduler events	kubectl describe pod <pod>
Service not routing	kube-proxy / CNI	kubectl logs -n kube-system -l k8s-app=kube-proxy
Node NotReady	kubelet / runtime	journalctl -u kubelet on the node
DNS resolution failures	CoreDNS	kubectl logs -n kube-system -l k8s-app=kube-dns
Ingress misrouting	Ingress controller	kubectl logs -n ingress-nginx deployment/ingress-nginx-controller
API server errors	API server audit logs	Node filesystem or audit webhook

Synthesis

Day 36 bridges the gap between individual Pod debugging and cluster-wide observability. The CKA exam expects fluency in kubectl logs variations, awareness of node-level log paths, and the ability to use kubectl top to correlate resource pressure with failures. Centralized logging is not configured on the exam cluster, but understanding the DaemonSet-based node agent pattern is part of the curriculum because it connects logging to workload scheduling concepts from earlier days.

Metrics Server is the simplest metric pipeline, yet it underpins everything from manual troubleshooting (kubectl top node to find the overloaded worker) to automated scaling (HPA and VPA from Days 16–17). Knowing how to install it and interpret its output is a practical exam skill.

See Also

Wiki Concepts

• Kubernetes Logging and Monitoring — synthesized concept page with full command reference and architecture patterns
• Kubernetes Health Probes — log analysis is the natural follow-up to probe debugging
• Pod Fundamentals — kubectl logs operates on the smallest deployable unit
• Kubernetes DaemonSet — log collectors run as node-level DaemonSets
• Sidecar Pattern — log shipping via auxiliary containers
• Multi-Container Pods — kubectl logs -c for container selection
• Kubernetes Resource Requests and Limits — Metrics Server and kubectl top for resource observation
• Horizontal Pod Autoscaler (HPA) — consumes Metrics Server data for scaling decisions
• Kubernetes Architecture — control plane and node component log locations
• Kubernetes Taints and Tolerations — tolerations for log collector DaemonSets on control plane nodes
• Observability Index — broader monitoring, logging, and tracing patterns

Related Sources

• CKA Day 35 — Kubernetes ETCD Backup And Restore Explained — preceding day in the series
• CKA Day 18 — Kubernetes Health Probes Explained — probes generate events that logs capture
• CKA Day 16 — Kubernetes Requests and Limits — resource metrics prerequisite
• CKA Day 17 — Kubernetes Autoscaling Explained — HPA depends on Metrics Server
• CKA Day 37 — Application Failure Troubleshooting — Next day: systematic debugging of app failures

Creator / Entity

• Tech Tutorials with Piyush — CKA course creator and Kubernetes educator