Kubernetes ETCD Backup and Restore

ETCD is the single source of truth for everything in a Kubernetes cluster. This page synthesizes the mechanics of backing up and restoring ETCD — a guaranteed CKA exam task and a critical production operation. Synthesized from CKA Day 35 — Kubernetes ETCD Backup And Restore Explained.

Why ETCD Backup Is Non-Negotiable

Every Kubernetes object — Pods, Deployments, Services, Nodes, ConfigMaps, Secrets, RBAC policies, NetworkPolicies, Events — is stored in ETCD. The API server is the only component that writes to ETCD; everything else reads from it or watches for changes. If ETCD is lost, the cluster becomes a blank slate: the containers may keep running temporarily, but the control plane has no memory of what should exist.

Scenario	Outcome Without Backup
Control plane disk failure	Total cluster state loss; rebuild from scratch
Accidental kubectl delete cascade	No way to recover deleted objects
ETCD data corruption	Cluster enters split-brain or unrecoverable state
Malicious actor wipes ETCD	Complete infrastructure destruction

Production rule: Back up ETCD before every upgrade, every major configuration change, and on a scheduled cadence (hourly for active clusters). Source: CKA Day 35

ETCD Deployment Topologies

Kubernetes supports two ways to run ETCD, and the backup procedure differs slightly for each:

Topology	Where ETCD Runs	Typical Use Case
Stacked	As a Static Pod on control plane nodes (kubeadm default)	Single control plane, CKA exam, small clusters
External	On dedicated VMs separate from control plane	HA production, large clusters, managed services

For the CKA exam, assume stacked ETCD unless explicitly told otherwise. In this topology, ETCD runs as a Pod in the kube-system Namespace, managed by the kubelet reading /etc/kubernetes/manifests/etcd.yaml. Kubernetes Static Pods explains why kubelet, not the API server, manages this Pod.

The Backup Command

The standard tool is etcdctl with API version 3:

ETCDCTL_API=3 etcdctl snapshot save /opt/snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Parameter Breakdown

Flag	Purpose	Typical Value (kubeadm)
--endpoints	ETCD server address	https://127.0.0.1:2379 (localhost on control plane)
--cacert	CA certificate to verify ETCD server	/etc/kubernetes/pki/etcd/ca.crt
--cert	Client certificate for mTLS	/etc/kubernetes/pki/etcd/server.crt
--key	Client private key for mTLS	/etc/kubernetes/pki/etcd/server.key

CKA Exam Trap: The certificate paths may vary. If /etc/kubernetes/pki/etcd/ does not exist, search the node with find / -name "etcd*" -type f. Also verify the exact endpoint with kubectl get pods -n kube-system -l component=etcd -o yaml.

Verify Backup Integrity

Always validate that the snapshot is not corrupt before treating it as a recovery artifact:

ETCDCTL_API=3 etcdctl snapshot status /opt/snapshot.db

This prints the ETCD revision, total keys, and total size — confirming the file is a valid snapshot.

The Restore Command

Restore is not an in-place operation. etcdctl snapshot restore creates a new data directory from the snapshot:

ETCDCTL_API=3 etcdctl snapshot restore /opt/snapshot.db \
  --data-dir=/var/lib/etcd-restored

After restoring, you must point ETCD to this new directory and restart the process:

For Stacked ETCD (kubeadm)

1. Update the Static Pod manifest to mount the restored directory:

   # Edit /etc/kubernetes/manifests/etcd.yaml
   volumes:
     - hostPath:
         path: /var/lib/etcd-restored   # <-- changed from /var/lib/etcd
         type: DirectoryOrCreate
       name: etcd-data

2. Kubelet auto-restarts the ETCD container because the manifest changed.

3. Verify the cluster recovers:

   kubectl get nodes
   kubectl get pods -A

Critical nuance: snapshot restore rewrites the ETCD member ID and cluster ID. In a multi-node ETCD cluster (HA), you must restore to all members or risk split-brain. For single-node ETCD (CKA exam default), this is straightforward. Source: CKA Day 35

Backup Automation Patterns

For production, manual snapshots are insufficient. Common automation strategies:

CronJob-Based Backup

Run a Kubernetes CronJob that mounts the control plane node’s PKI and executes etcdctl snapshot save to a PersistentVolume or S3-compatible store:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-backup
  namespace: kube-system
spec:
  schedule: "0 * * * *"   # Every hour
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: backup
              image: bitnami/etcd:latest
              command:
                - /bin/sh
                - -c
                - |
                  etcdctl snapshot save /backups/etcd-$(date +%Y%m%d-%H%M).db \
                    --endpoints=https://127.0.0.1:2379 \
                    --cacert=/certs/ca.crt \
                    --cert=/certs/server.crt \
                    --key=/certs/server.key
              volumeMounts:
                - name: etcd-certs
                  mountPath: /certs
                - name: backup-storage
                  mountPath: /backups
          volumes:
            - name: etcd-certs
              hostPath:
                path: /etc/kubernetes/pki/etcd
            - name: backup-storage
              persistentVolumeClaim:
                claimName: etcd-backup-pvc
          restartPolicy: OnFailure

Retention and Storage

Concern	Recommendation
Retention	Keep 24 hourly, 7 daily, 4 weekly snapshots; purge old ones automatically
Off-site storage	Sync snapshots to S3, GCS, or Azure Blob immediately after creation
Encryption at rest	Encrypt snapshot files with AES-256 or upload to encrypted object storage
Testing	Perform a quarterly restore drill on a non-production cluster to validate backups

Disaster Recovery Scenarios

Failure	Recovery Approach
Single object deleted	Prefer kubectl apply -f from GitOps/Git; use ETCD restore only if no Git history exists
ETCD data directory corrupted	Stop ETCD, restore from latest snapshot, restart ETCD
Control plane node failure	Rebuild node, join as new control plane (HA), or restore ETCD + re-create Static Pods
Ransomware / malicious wipe	Isolate cluster, restore from off-site backup to new hardware

GitOps vs ETCD Restore: If you manage manifests in Git (e.g., ArgoCD, Flux), most object-level mistakes are recoverable by re-applying YAML. ETCD restore is the nuclear option for when the entire state store is compromised or when Git does not cover dynamically created resources (Events, automatic Service endpoints, node registrations). Kubernetes Disaster Recovery

CKA Exam Patterns

• Certificate discovery: If the exam does not give you cert paths, know the kubeadm default: /etc/kubernetes/pki/etcd/.
• Restore means restart: Many candidates run snapshot restore but forget to restart ETCD. In a stacked setup, edit the manifest; kubelet does the restart.
• Data directory ownership: ETCD may refuse to start if the restored directory has wrong permissions. chown -R etcd:etcd /var/lib/etcd-restored may be necessary.
• Time travel effect: Restoring an older snapshot reverts all cluster state. Be explicit about this in exam explanations.

Practical Practice

Exam-style hands-on tasks for this topic. Complete each task before reviewing the solution. Time yourself — CKA tasks average 5–7 minutes.

Task 1: Take an ETCD Snapshot Take an ETCD snapshot using etcdctl snapshot save with the correct certificates. Requirements: Use the kubeadm default certificate paths. Verification: etcdctl snapshot status /opt/snapshot.db Solution:

ETCDCTL_API=3 etcdctl snapshot save /opt/snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key
ETCDCTL_API=3 etcdctl snapshot status /opt/snapshot.db

Task 2: Restore an ETCD Snapshot Restore an ETCD snapshot to a new data directory and update the Static Pod manifest. Requirements: Do not overwrite /var/lib/etcd in place. Verification: kubectl get nodes Solution:

sudo systemctl stop kubelet
ETCDCTL_API=3 etcdctl snapshot restore /opt/snapshot.db --data-dir=/var/lib/etcd-restored
sudo sed -i 's|/var/lib/etcd|/var/lib/etcd-restored|' /etc/kubernetes/manifests/etcd.yaml
sudo systemctl start kubelet
kubectl get nodes

Task 3: Explain the Time-Travel Effect A restored cluster shows old objects — explain that restore is a “time travel” operation. Requirements: Write a one-sentence explanation in the exam context. Verification: N/A (conceptual) Solution:

# ETCD restore reverts the entire cluster state to the moment the snapshot was taken.
# Any objects created, modified, or deleted after the snapshot will reflect the old state.
# This is why restore is considered a "time travel" operation, not a selective recovery.

See Also

• Kubernetes Disaster Recovery — Broader strategies: backup automation, RTO/RPO, and multi-cluster redundancy
• Kubernetes Architecture — ETCD’s role in the control plane and Raft consensus mechanics
• Kubernetes Cluster Upgrade — Always snapshot ETCD before upgrading; upgrade rewrites Static Pod manifests
• Kubeadm Cluster Setup — Where ETCD certificates are generated and how the stacked topology is configured
• Kubernetes Static Pods — ETCD is a Static Pod on kubeadm clusters; kubelet restart behavior
• Node Maintenance — Safely evacuating a control plane node before ETCD maintenance
• TLS Fundamentals — mTLS certificates used in ETCD backup/restore commands
• CKA Certification — Exam domain: Architecture, Installation & Configuration (~25%)
• CKA Study Roadmap — Day 35 in the 40-day plan
• Kubernetes Control Plane Troubleshooting — Day 38: etcd failure diagnosis and quorum loss recovery
• CKA Day 35 — Kubernetes ETCD Backup And Restore Explained

---

Tags: kubernetes etcd backup restore disaster-recovery cka cluster-architecture devops production