CKA Day 35: Kubernetes ETCD Backup And Restore Explained

Day 35/40 of the Tech Tutorials with Piyush CKA certification course. This ~34-minute hands-on tutorial covers the complete end-to-end process of backing up and restoring the Kubernetes ETCD datastore — the single source of truth for all cluster state.

Synthesis

ETCD is a distributed, consistent key-value store that acts as the brain of a Kubernetes cluster. Every Pod, Deployment, Service, ConfigMap, Secret, RBAC policy, and Node registration lives in ETCD. If ETCD is lost or corrupted, the entire cluster state disappears — even if the control plane containers are still running. This makes ETCD backup and restore one of the most critical operational skills for a Kubernetes administrator, and it is a guaranteed topic on the CKA exam.

The video explains the ETCD data directory (/var/lib/etcd) and distinguishes between stacked ETCD (running as a Static Pod on the control plane node, managed by kubeadm) and external ETCD (running on separate VMs, common in HA production clusters). For the CKA exam, the stacked model is the default scenario.

The core backup tool is etcdctl with the v3 API. The snapshot save command creates a point-in-time backup file that contains the entire ETCD database. The command requires three TLS parameters — --cacert, --cert, and --key — because ETCD enforces mutual TLS for all client connections, including localhost. These certificates live in /etc/kubernetes/pki/etcd/ on a kubeadm cluster.

Restore is more invasive than backup. The snapshot restore command writes the backed-up data into a new data directory (it does not overwrite the live one in place). After restoring, the ETCD configuration must point to this new directory, and the ETCD process must be restarted. On a kubeadm cluster, this means updating the ETCD Static Pod manifest or restarting the ETCD container so it picks up the restored data.

The video demonstrates a complete disaster-recovery loop: take a snapshot, simulate data loss by deleting a Deployment, stop ETCD, restore from the snapshot, restart ETCD, and verify the deleted Deployment reappears. This proves that the backup truly captures the cluster state and that the restore procedure returns the cluster to the exact moment the snapshot was taken.

A critical nuance covered is that restoring an older ETCD snapshot reverts all cluster state to that point — not just the object you accidentally deleted. Any changes made after the snapshot (new Pods, updated Secrets, node registrations) are lost. This makes backup frequency a business decision: too infrequent and you lose too much state; too frequent and you consume disk space and I/O.

The video also touches on automated backup strategies: CronJobs that run etcdctl snapshot save on a schedule, remote storage of snapshot files (S3, NFS, or mounted PV), and retention policies. For the CKA exam, the focus is on the manual imperative commands, but understanding the automation pattern is essential for production.

Key Commands

# Backup (imperative — must know for CKA)
ETCDCTL_API=3 etcdctl snapshot save snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key
 
# Verify backup integrity
ETCDCTL_API=3 etcdctl snapshot status snapshot.db
 
# Restore (creates a new data directory)
ETCDCTL_API=3 etcdctl snapshot restore snapshot.db \
  --data-dir=/var/lib/etcd-restored
 
# Update ETCD to use restored data (kubeadm stacked scenario)
# Edit /etc/kubernetes/manifests/etcd.yaml to point volume to /var/lib/etcd-restored
# kubelet auto-restarts the Static Pod when the manifest changes

CKA Exam Patterns

• Certificate paths: The exam may place etcd certs in non-standard locations. Always check /etc/kubernetes/pki/etcd/ first, but be prepared to search.
• ETCD endpoint: Usually https://127.0.0.1:2379 on the control plane node. Verify with kubectl get pods -n kube-system to find the ETCD Pod and its node.
• Restore means restart: After snapshot restore, the ETCD process must be restarted. In a Static Pod setup, modifying the manifest triggers kubelet to recreate the container.
• Data directory permissions: ETCD runs as a specific user (often etcd or root depending on setup). Ensure the restored directory has correct ownership and permissions.

See Also

Wiki Concepts

• Kubernetes ETCD Backup and Restore — Deep-dive concept page on backup/restore mechanics, stacked vs external ETCD, and disaster recovery patterns
• Kubernetes Disaster Recovery — Broader disaster recovery strategies including backup automation, retention, and recovery time objectives
• Kubernetes Architecture — ETCD’s role as the cluster’s consistent data store and Raft consensus
• Kubernetes Cluster Upgrade — Upgrades modify control plane Static Pods; always back up ETCD before upgrading
• Kubeadm Cluster Setup — Where ETCD certificates are generated and how the stacked ETCD Static Pod is configured
• Kubernetes Static Pods — ETCD runs as a Static Pod on kubeadm clusters; understanding Static Pod restart behavior is critical for restore
• Node Maintenance — Evacuating a control plane node before maintenance that may affect ETCD
• TLS Fundamentals — ETCD backup/restore commands require mTLS certificate parameters
• CKA Certification — Exam domain mapping: ETCD backup/restore sits in Architecture, Installation & Configuration (~25%)
• CKA Study Roadmap — Day 35 in the 40-day learning plan

Related Sources

• CKA Day 34 — Step-By-Step Guide To Upgrade a Multi Node Kubernetes Cluster With Kubeadm — The cluster upgraded in Day 34 is the same one whose ETCD is backed up here
• CKA Day 27 — Setup a Multi Node Kubernetes Cluster Using Kubeadm — The original cluster installation where ETCD certificates were created
• CKA Day 36 — Kubernetes Logging and Monitoring — Next day in the series: troubleshooting with logs and metrics

Creator / Entity

• Tech Tutorials with Piyush — Day 35 of the 40-day CKA challenge series

---

Tags: cka kubernetes etcd backup restore disaster-recovery cluster-architecture devops