Kubernetes Disaster Recovery

Disaster recovery (DR) for Kubernetes goes beyond backing up ETCD. It encompasses the full stack: cluster state, persistent data, container images, secrets, and operational runbooks. This page synthesizes DR strategies, recovery time objectives (RTO), and production patterns for resilient Kubernetes infrastructure. Contextualized by CKA Day 35 — ETCD Backup and Restore.

The DR Hierarchy

Not all disasters are equal. A sensible DR strategy layers protections from cheapest to most expensive:

Layer	What It Protects	Tool / Pattern	RTO
1. GitOps	Desired state (YAML manifests)	ArgoCD, Flux, Git	Minutes
2. ETCD Backup	Live cluster state (Pods, nodes, events)	etcdctl snapshot save/restore	15–30 min
3. Persistent Volume Snapshots	Application data (databases, files)	CSI snapshots, Velero	15–60 min
4. Cluster Infrastructure	Nodes, network, load balancers	Infrastructure-as-Code (Terraform, Pulumi)	30–120 min
5. Cross-Region Failover	Entire cluster + data	Multi-cluster, DNS failover, replication	5–30 min

Design principle: Protect lower layers before building higher ones. GitOps + ETCD backup handles 90% of real-world failures. Multi-region failover is expensive and only justified for strict SLA requirements. Source: CKA Day 35

ETCD as the Critical Path

ETCD holds the “brain” of the cluster. Without it, even perfectly healthy worker nodes and running containers lose their orchestration context. This makes ETCD backup the first and most important DR primitive:

• Frequency: Hourly for active development clusters; daily for stable production.
• Retention: 24 hourly + 7 daily + 4 weekly snapshots.
• Storage: Immediate upload to encrypted object storage (S3, GCS, Azure Blob).
• Verification: Monthly restore drill on a staging cluster.

See Kubernetes ETCD Backup and Restore for the complete imperative command reference and stacked vs external ETCD nuances. Source: CKA Day 35

Persistent Data Protection

ETCD does not contain application data stored in PersistentVolumes. A database Pod’s data lives on the node’s disk or network storage, independent of ETCD. DR for persistent data requires separate tooling:

Approach	Best For	Tooling
CSI Volume Snapshots	Cloud-native block storage	AWS EBS snapshots, GCP PD snapshots, Azure Disk snapshots
Application-level backup	Databases with complex consistency needs	PostgreSQL pg_dump, MySQL mysqldump, MongoDB mongodump
Cross-cluster replication	Zero-RPO requirements	DRBD, Ceph RBD mirroring, Portworx, Longhorn

Kubernetes-native option: Velero backs up both cluster resources (via ETCD export) and PersistentVolumes (via CSI snapshots or Restic file-level backup) to object storage. It is the closest thing to a unified Kubernetes DR solution. Kubernetes Storage

Cluster Reconstruction Scenarios

Scenario A: ETCD Corruption (Single Control Plane)

1. Stop kubelet (prevents ETCD from restarting repeatedly)
2. Restore ETCD from latest snapshot → new data directory
3. Update etcd.yaml manifest to point to restored directory
4. Start kubelet → ETCD restarts automatically
5. Verify: kubectl get nodes, kubectl get pods -A

Scenario B: Complete Control Plane Loss

1. Provision new control plane node(s)
2. Install kubeadm, containerd, kubelet
3. Restore ETCD snapshot to /var/lib/etcd on the new node
4. Re-create Static Pod manifests for API server, scheduler, controller-manager
5. Re-issue worker join tokens or update worker kubelets to point to new API server

HA mitigation: With 3+ control plane nodes and stacked ETCD, losing one node does not require restore — ETCD Raft tolerates (n-1)/2 failures. Simply replace the failed node and join it as a new control plane. Restore is only needed when quorum is lost (e.g., 2 of 3 nodes fail simultaneously). Kubernetes Architecture

RTO and RPO Definitions

Metric	Definition	Kubernetes Target
RTO (Recovery Time Objective)	How long until the cluster is usable again	15–60 min for single control plane; <5 min for HA with automation
RPO (Recovery Point Objective)	How much data loss is acceptable	1 hour (hourly ETCD snapshots); near-zero with continuous replication

For managed Kubernetes (EKS, GKE, AKS), the cloud provider handles control plane DR. Your responsibility shrinks to persistent volume backups and application-level replication.

CKA Exam Relevance

The CKA exam tests DR at the ETCD layer only. You must know:

1. etcdctl snapshot save with TLS flags.
2. etcdctl snapshot restore to a new data directory.
3. Restarting ETCD after restore (manifest edit for stacked, service restart for external).
4. Verifying cluster recovery with kubectl get.

The exam does not cover multi-region failover, Velero, or CSI snapshots — those are production extensions beyond the certification scope. CKA Certification

Practical Practice

Exam-style hands-on tasks for this topic. Complete each task before reviewing the solution. Time yourself — CKA tasks average 5–7 minutes.

Task 1: Rebuild Control Plane from ETCD Backup Simulate a complete control plane loss and rebuild the cluster using the latest ETCD backup. Requirements: Provision a new VM, install kubeadm/containerd, restore ETCD, recreate manifests. Verification: kubectl get nodes Solution:

# On new control plane node
sudo kubeadm init --pod-network-cidr=10.244.0.0/16
# Stop kubelet, restore ETCD snapshot to /var/lib/etcd
sudo systemctl stop kubelet
ETCDCTL_API=3 etcdctl snapshot restore /backups/latest.db --data-dir=/var/lib/etcd
# Ensure /etc/kubernetes/manifests has API server, scheduler, controller-manager manifests
sudo systemctl start kubelet
kubectl get nodes

Task 2: Verify Persistent Data Survived Verify persistent volume data survived the recovery by checking the application Pods. Requirements: Ensure PVCs and application Pods are running. Check app data. Verification: kubectl get pvc,pod and exec into the app to read data. Solution:

kubectl get pvc
kubectl get pods -l app=data-app
kubectl exec -it <data-pod> -- cat /data/important.txt
# If the PV is on network storage (EBS, NFS), the data outlives the Pod and control plane.

See Also

• Kubernetes ETCD Backup and Restore — Complete imperative command reference for ETCD snapshot and restore
• Kubernetes Architecture — ETCD Raft consensus and control plane HA strategies
• Kubernetes Cluster Upgrade — Pre-upgrade ETCD snapshot is mandatory before any upgrade
• Kubeadm Cluster Setup — Rebuilding a cluster from scratch after total loss
• Kubernetes Storage — CSI snapshots and persistent volume DR patterns
• Kubernetes Static Pods — Control plane bootstrapping and manifest-based recovery
• Kubernetes Control Plane Troubleshooting — Day 38: control plane component failure diagnosis and recovery workflow
• CKA Certification — Exam domain mapping: Architecture, Installation & Configuration (~25%)
• CKA Day 35 — Kubernetes ETCD Backup And Restore Explained

---

Tags: kubernetes disaster-recovery backup etcd resilience cka production devops rto rpo