How can I restore etcd cluster from snapshot in docker image on CoreOS?

I have a Kubernetes cluster (v1.5.6) with 3 nodes etcd cluster (etcd version 3.1.5) on vmware. This etcd nodes are running in three docker containers(on three hosts) on coreos on vmware.

I try to backup etcd with the following solution:

docker run --rm --net=host -v /tmp:/etcd_backup -e ETCDCTL_API=3 quay.io/coreos/etcd:v3.1.5 etcdctl --endpoints=[1.1.1.1:2379,2.2.2.2:2379,3.3.3.3:2379] snapshot save etcd_backup/snapshot.db

The backup has been completed succesfully.

I want to create this kubernetes cluster from zero in another vmware environment, but I need to restore etcd from snapshot for this.

So far, I have not found the right solution that works with etcd in docker containers.

I try to restore with the following method, but unfortunately I did not succeed.

First, I created a new etcd node after I run the following command:

docker run --rm --net=host -v /tmp/etcd_bak:/etcd_backup -e ETCDCTL_API=3 registry:5000/quay.io/coreos/etcd:v3.1.5 etcdctl snapshot restore etcd_backup/snapshot.db --name etcd0 --initial-cluster etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380 --initial-cluster-token etcd-cluster-1 --initial-advertise-peer-urls http://etcd0:2380

Result:

2018-06-04 09:25:52.314747 I | etcdserver/membership: added member 7ff5c9c6942f82e [http://etcd0:2380] to cluster 5d1b637f4b7740d5
2018-06-04 09:25:52.314940 I | etcdserver/membership: added member 91b417e7701c2eeb [http://etcd2:2380] to cluster 5d1b637f4b7740d5
2018-06-04 09:25:52.315096 I | etcdserver/membership: added member faeb78734ee4a93d [http://etcd1:2380] to cluster 5d1b637f4b7740d5

Unfortunately, nothing happens.

What is the good solution to restore the etcd backup?

How do I create an empty etcd cluster/node and how should I restore the snapshot?

Solution

according to the Etcd Disaster Recovery document, you need restore all three etcd nodes from snapshot with commands like yours, then run three node with commands like this:

etcd \
  --name m1 \
  --listen-client-urls http://host1:2379 \
  --advertise-client-urls http://host1:2379 \
  --listen-peer-urls http://host1:2380 &

Also, you can extract etcdctl from the image, like this:

docker run --rm -v /opt/bin:/opt/bin registry:5000/quay.io/coreos/etcd:v3.1.5 cp /usr/local/bin/etcdctl /opt/bin

Then use etcdctl to restore snapshot:

# ETCDCTL_API=3 ./etcdctl snapshot restore snapshot.db \
  --name m1 \
  --initial-cluster m1=http://host1:2380,m2=http://host2:2380,m3=http://host3:2380 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-advertise-peer-urls http://host1:2380 \
  --data-dir /var/lib/etcd

This will restore snapshot to the /var/lib/etcd directory. Then start etcd with docker, don't forget mount /var/lib/etcd into your container, and specify --data-dir to it .