Backup and Restore | Cluster Administration

Overview
Prerequisites
Cluster Backup
Cluster Restore for Single-member etcd Clusters
Cluster Restore for Multiple-member etcd Clusters
- Embedded etcd
- Separate etcd
Bringing OpenShift Enterprise Services Back Online
Project Backup
Project Restore
Application Data Backup
Application Data Restore

Overview

In OpenShift Enterprise, you can back up (saving state to separate storage) and restore (recreating state from separate storage) at the cluster level. There is also some preliminary support for per-project backup. The full state of a cluster installation includes:

etcd data on each master
API objects
registry storage
volume storage

This topic does not cover how to back up and restore persistent storage, as those topics are left to the underlying storage provider. However, an example of how to perform a generic backup of application data is provided.

This topic only provides a generic way of backing up applications and the OpenShift Enterprise cluster. It can not take into account custom requirements. Therefore, you should create a full backup and restore procedure. To prevent data loss, necessary precautions should be taken.

Prerequisites

Because the restore procedure involves a complete reinstallation, save all the files used in the initial installation. This may include:
- ~/.config/openshift/installer.cfg.yml (from the Quick Installation method)
- Ansible playbooks and inventory files (from the Advanced Installation method)
- /etc/yum.repos.d/ose.repo (from the Disconnected Installation method)
Backup the procedures for post-installation steps. Some installations may involve steps that are not included in the installer. This may include changes to the services outside of the control of OpenShift Enterprise or the installation of extra services like monitoring agents. Additional configuration that is not supported yet by the advanced installer might also be affected, for example when using multiple authentication providers.
Install packages that provide various utility commands:
```
# yum install etcd
```
If using a container-based installation, pull the etcd image instead:
```
# docker pull rhel7/etcd
```

Note the location of the etcd data directory (or $etcd_DATA_DIR in the following sections), which depends on how etcd is deployed.

Deployment Type	Description	Data Directory
separate etcd	etcd runs as a separate service, either co-located on master nodes or on separate nodes.	/var/lib/etcd
embedded etcd	etcd runs as part of the master service.	*/var/lib/origin/openshift.local.etcd*

Deployment Type

Description

Data Directory

separate etcd

etcd runs as a separate service, either co-located on master nodes or on separate nodes.

/var/lib/etcd

embedded etcd

etcd runs as part of the master service.

/var/lib/origin/openshift.local.etcd

Cluster Backup

Save all the certificates and keys, on each master:

# cd /etc/origin/master
# tar cf /tmp/certs-and-keys-$(hostname).tar *.key *.crt

If etcd is running on more than one host, stop it on each host:
```
# sudo systemctl stop etcd
```
Although this step is not strictly necessary, doing so ensures that the etcd data is fully synchronized.
Create an etcd backup:
```
# etcdctl backup \
    --data-dir $etcd_DATA_DIR \
    --backup-dir $etcd_DATA_DIR.bak
```
If etcd is running on more than one host, the various instances regularly synchronize their data, so creating a backup for one of them is sufficient.

For a container-based installation, you must use docker exec to run etcdctl inside the container.

Copy the db file over to the backup you created:

# cp "$etcd_DATA_DIR"/member/snap/db "$etcd_DATA_DIR.bak"/member/snap/db

Cluster Restore for Single-member etcd Clusters

To restore the cluster:

Reinstall OpenShift Enterprise.

This should be done in the same way that OpenShift Enterprise was previously installed.
Run all necessary post-installation steps.

Restore the certificates and keys, on each master:

# cd /etc/origin/master
# tar xvf /tmp/certs-and-keys-$(hostname).tar

Restore from the etcd backup:

# mv $etcd_DATA_DIR $etcd_DATA_DIR.orig
# cp -Rp $etcd_DATA_DIR.bak $etcd_DATA_DIR
# chcon -R --reference $etcd_DATA_DIR.orig $etcd_DATA_DIR
# chown -R etcd:etcd $etcd_DATA_DIR

Create the new single node cluster using etcd’s --force-new-cluster option. You can do this using the values from /etc/etcd/etcd.conf, or you can temporarily modify the systemd unit file and start the service normally.

To do so, edit the /usr/lib/systemd/system/etcd.service file, and add --force-new-cluster:
```
# sed -i '/ExecStart/s/"$/  --force-new-cluster"/' /usr/lib/systemd/system/etcd.service
# systemctl show etcd.service --property ExecStart --no-pager

ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd --force-new-cluster"
```
Then, restart the etcd service:
```
# systemctl daemon-reload
# systemctl start etcd
```

Verify the etcd service started correctly, then re-edit the /usr/lib/systemd/system/etcd.service file and remove the --force-new-cluster option:

# sed -i '/ExecStart/s/ --force-new-cluster//' /usr/lib/systemd/system/etcd.service
# systemctl show etcd.service --property ExecStart --no-pager

ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd"

Restart the etcd service, then verify the etcd cluster is running correctly and displays OpenShift Enterprise’s configuration:
```
# systemctl daemon-reload
# systemctl restart etcd
```

Cluster Restore for Multiple-member etcd Clusters

When using a separate etcd cluster, you must first restore the etcd backup by creating a new, single node etcd cluster. If you run etcd as a stand-alone service on your master nodes, you can create the single node etcd cluster on a master node. If you use separate etcd with multiple members, you must then also add any additional etcd members to the etcd cluster one by one.

However, the details of the restoration process differ between embedded and external etcd. See the following section and follow the relevant steps before Bringing OpenShift Services Back Online.

Embedded etcd

Restore your etcd backup and configuration:

Run the following on the master with the embedded etcd:

# etcd_DIR=/var/lib/origin/openshift.local.etcd
# mv $etcd_DIR /var/lib/etcd.orig
# cp -Rp /var/lib/origin/etcd-backup-<timestamp>/ $etcd_DIR
# chcon -R --reference /var/lib/etcd.orig/ $etcd_DIR
# chown -R etcd:etcd $etcd_DIR

The $etcd_DIR location differs between external and embedded etcd.

Create the new, single node etcd cluster:

# etcd -data-dir=/var/lib/origin/openshift.local.etcd \
    -force-new-cluster

Verify etcd has started successfully by checking the output from the above command, which should look similar to the following near the end:

[...]
2016-06-24 12:14:45.644073 I | etcdserver: starting server... [version: 2.2.5, cluster version: 2.2]
[...]
2016-06-24 12:14:46.834394 I | etcdserver: published {Name:default ClientURLs:[http://localhost:2379 http://localhost:4001]} to cluster 5580663a6e0002

Shut down the process by running the following from a separate terminal:
```
# pkill etcd
```
Continue to Bringing OpenShift Enterprise Services Back Online.

Separate etcd

Choose a system to be the initial etcd member, and restore its etcd backup and configuration:

Run the following on the etcd host:

# etcd_DIR=/var/lib/etcd/
# mv $etcd_DIR /var/lib/etcd.orig
# cp -Rp /var/lib/origin/etcd-backup-<timestamp>/ $etcd_DIR
# chcon -R --reference /var/lib/etcd.orig/ $etcd_DIR
# chown -R etcd:etcd $etcd_DIR

The $etcd_DIR location differs between external and embedded etcd.

Restore your /etc/etcd/etcd.conf file from backup or .rpmsave.
Create the new single node cluster using etcd’s --force-new-cluster option. You can do this with a long complex command using the values from /etc/etcd/etcd.conf, or you can temporarily modify the systemd unit file and start the service normally.

To do so, edit the /usr/lib/systemd/system/etcd.service file, and add --force-new-cluster:
```
# sed -i '/ExecStart/s/"$/  --force-new-cluster"/' /usr/lib/systemd/system/etcd.service
# systemctl show etcd.service --property ExecStart --no-pager

ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd --force-new-cluster"
```
Then restart the etcd service:
```
# systemctl daemon-reload
# systemctl start etcd
```

Verify the etcd service started correctly, then re-edit the /usr/lib/systemd/system/etcd.service file and remove the --force-new-cluster option:

# sed -i '/ExecStart/s/ --force-new-cluster//' /usr/lib/systemd/system/etcd.service
# systemctl show etcd.service --property ExecStart --no-pager

ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd"

Restart the etcd service, then verify the etcd cluster is running correctly and displays OpenShift Enterprise’s configuration:

# systemctl daemon-reload
# systemctl restart etcd
# etcdctl --cert-file=/etc/etcd/peer.crt \
    --key-file=/etc/etcd/peer.key \
    --ca-file=/etc/etcd/ca.crt \
    --peers="https://172.16.4.18:2379,https://172.16.4.27:2379" \ (1)
    ls /

1	Ensure that you specify the URLs of only active etcd members in the `--peers` parameter value.

If you have additional etcd members to add to your cluster, continue to Adding Additional etcd Members. Otherwise, if you only want a single node separate etcd cluster, continue to Bringing OpenShift Enterprise Services Back Online.

Adding Additional etcd Members

To add additional etcd members to the cluster, you must first adjust the default localhost peer in the peerURLs value for the first member:

Get the member ID for the first member using the member list command:

# etcdctl --cert-file=/etc/etcd/peer.crt \
    --key-file=/etc/etcd/peer.key \
    --ca-file=/etc/etcd/ca.crt \
    --peers="https://172.18.1.18:2379,https://172.18.9.202:2379,https://172.18.0.75:2379" \ (1)
    member list

1	Ensure that you specify the URLs of only active etcd members in the `--peers` parameter value.

Update the value of peerURLs using the etcdctl member update command by passing the member ID obtained from the previous step:

# etcdctl --cert-file=/etc/etcd/peer.crt \
    --key-file=/etc/etcd/peer.key \
    --ca-file=/etc/etcd/ca.crt \
    --peers="https://172.18.1.18:2379,https://172.18.9.202:2379,https://172.18.0.75:2379" \
    member update 511b7fb6cc0001 https://172.18.1.18:2380

Alternatively, you can use curl:

# curl --cacert /etc/etcd/ca.crt \
    --cert /etc/etcd/peer.crt \
    --key /etc/etcd/peer.key \
    https://172.18.1.18:2379/v2/members/511b7fb6cc0001 \
    -XPUT -H "Content-Type: application/json" \
    -d '{"peerURLs":["https://172.18.1.18:2380"]}'

Re-run the member list command and ensure the peer URLs no longer include localhost.

Now, add each additional member to the cluster one at a time.

Each member must be fully added and brought online one at a time. When adding each additional member to the cluster, the peerURLs list must be correct for that point in time, so it will grow by one for each member added. The etcdctl member add command will output the values that need to be set in the etcd.conf file as you add each member, as described in the following instructions.

For each member, add it to the cluster using the values that can be found in that system’s etcd.conf file:

# etcdctl --cert-file=/etc/etcd/peer.crt \
    --key-file=/etc/etcd/peer.key \
    --ca-file=/etc/etcd/ca.crt \
    --peers="https://172.16.4.18:2379,https://172.16.4.27:2379" \
    member add 10.3.9.222 https://172.16.4.27:2380 (1)

Added member named 10.3.9.222 with ID 4e1db163a21d7651 to cluster

etcd_NAME="10.3.9.222"
etcd_INITIAL_CLUSTER="10.3.9.221=https://172.16.4.18:2380,10.3.9.222=https://172.16.4.27:2380"
etcd_INITIAL_CLUSTER_STATE="existing"

1	In this line, `10.3.9.222` is a label for the etcd member. You can specify the host name, IP address, or a simple name.

Using the environment variables provided in the output of the above etcdctl member add command, edit the /etc/etcd/etcd.conf file on the member system itself and ensure these settings match.

Now start etcd on the new member:

# rm -rf /var/lib/etcd/member
# systemctl enable etcd
# systemctl start etcd

Ensure the service starts correctly and the etcd cluster is now healthy:

# etcdctl --cert-file=/etc/etcd/peer.crt \
    --key-file=/etc/etcd/peer.key \
    --ca-file=/etc/etcd/ca.crt \
    --peers="https://172.16.4.18:2379,https://172.16.4.27:2379" \
    member list

51251b34b80001: name=10.3.9.221 peerURLs=https://172.16.4.18:2380 clientURLs=https://172.16.4.18:2379
d266df286a41a8a4: name=10.3.9.222 peerURLs=https://172.16.4.27:2380 clientURLs=https://172.16.4.27:2379

# etcdctl --cert-file=/etc/etcd/peer.crt \
    --key-file=/etc/etcd/peer.key \
    --ca-file=/etc/etcd/ca.crt \
    --peers="https://172.16.4.18:2379,https://172.16.4.27:2379" \
    cluster-health

cluster is healthy
member 51251b34b80001 is healthy
member d266df286a41a8a4 is healthy

Now repeat this process for the next member to add to the cluster.

After all additional etcd members have been added, continue to Bringing OpenShift Enterprise Services Back Online.

Bringing OpenShift Enterprise Services Back Online

On each OpenShift Enterprise master, restore your master and node configuration from backup and enable and restart all relevant services.

On the master in a single master cluster:

# cp /etc/sysconfig/atomic-openshift-master.rpmsave /etc/sysconfig/atomic-openshift-master
# cp /etc/origin/master/master-config.yaml.<timestamp> /etc/origin/master/master-config.yaml
# cp /etc/origin/node/node-config.yaml.<timestamp> /etc/origin/node/node-config.yaml
# systemctl enable atomic-openshift-master
# systemctl enable atomic-openshift-node
# systemctl start atomic-openshift-master
# systemctl start atomic-openshift-node

On each master in a multi-master cluster:

# cp /etc/sysconfig/atomic-openshift-master-api.rpmsave /etc/sysconfig/atomic-openshift-master-api
# cp /etc/sysconfig/atomic-openshift-master-controllers.rpmsave /etc/sysconfig/atomic-openshift-master-controllers
# cp /etc/origin/master/master-config.yaml.<timestamp> /etc/origin/master/master-config.yaml
# cp /etc/origin/node/node-config.yaml.<timestamp> /etc/origin/node/node-config.yaml
# systemctl enable atomic-openshift-master-api
# systemctl enable atomic-openshift-master-controllers
# systemctl enable atomic-openshift-node
# systemctl start atomic-openshift-master-api
# systemctl start atomic-openshift-master-controllers
# systemctl start atomic-openshift-node

On each OpenShift Enterprise node, restore your node-config.yaml file from backup and enable and restart the atomic-openshift-node service:

# cp /etc/origin/node/node-config.yaml.<timestamp> /etc/origin/node/node-config.yaml
# systemctl enable atomic-openshift-node
# systemctl start atomic-openshift-node

Your OpenShift Enterprise cluster should now be back online.

Project Backup

A future release of OpenShift Enterprise will feature specific support for per-project back up and restore.

For now, to back up API objects at the project level, use oc export for each object to be saved. For example, to save the deployment configuration frontend in YAML format:

$ oc export dc frontend -o yaml > dc-frontend.yaml

To back up all of the project (with the exception of cluster objects like namespaces and projects):

$ oc export all -o yaml > project.yaml

Role Bindings

Sometimes custom policy role bindings are used in a project. For example, a project administrator can give another user a certain role in the project and grant that user project access.

These role bindings can be exported:

$ oc get rolebindings -o yaml --export=true > rolebindings.yaml

Service Accounts

If custom service accounts are created in a project, these need to be exported:

$ oc get serviceaccount -o yaml --export=true > serviceaccount.yaml

Secrets

Custom secrets like source control management secrets (SSH Public Keys, Username/Password) should be exported if they are used:

$ oc get secret -o yaml --export=true > secret.yaml

Persistent Volume Claims

If the an application within a project uses a persistent volume through a persistent volume claim (PVC), these should be backed up:

$ oc get pvc -o yaml --export=true > pvc.yaml

Project Restore

To restore a project, recreate the project and recreate all all of the objects that were exported during the backup:

$ oc new-project myproject
$ oc create -f project.yaml
$ oc create -f secret.yaml
$ oc create -f serviceaccount.yaml
$ oc create -f pvc.yaml
$ oc create -f rolebindings.yaml

Some resources can fail to be created (for example, pods and default service accounts).

Application Data Backup

In many cases, application data can be backed up using the oc rsync command, assuming rsync is installed within the container image. The Red Hat rhel7 base image does contain rsync. Therefore, all images that are based on rhel7 contain it as well.

This is a generic backup of application data and does not take into account application-specific backup procedures, for example special export/import procedures for database systems.

Other means of backup may exist depending on the type of the persistent volume (for example, Cinder, NFS, Gluster, or others).

The paths to back up are also application specific. You can determine what path to back up by looking at the mountPath for volumes in the deploymentconfig.

Example of Backing up a Jenkins Deployment’s Application Data

Get the application data mountPath from the deploymentconfig:

$ oc get dc/jenkins -o jsonpath='{ .spec.template.spec.containers[?(@.name=="jenkins")].volumeMounts[?(@.name=="jenkins-data")].mountPath }'
/var/lib/jenkins

Get the name of the pod that is currently running:

$ oc get pod --selector=deploymentconfig=jenkins -o jsonpath='{ .metadata.name }'
jenkins-1-37nux

Use the oc rsync command to copy application data:

$ oc rsync jenkins-1-37nux:/var/lib/jenkins /tmp/

This type of application data backup can only be performed while an application pod is currently running.

Application Data Restore

The process for restoring application data is similar to the application backup procedure using the oc rsync tool. The same restrictions apply and the process of restoring application data requires a persistent volume.

Example of Restoring a Jenkins Deployment’s Application Data

Verify the backup:

$ ls -la /tmp/jenkins-backup/
total 8
drwxrwxr-x.  3 user     user   20 Sep  6 11:14 .
drwxrwxrwt. 17 root     root 4096 Sep  6 11:16 ..
drwxrwsrwx. 12 user     user 4096 Sep  6 11:14 jenkins

Use the oc rsync tool to copy the data into the running pod:
```
$ oc rsync /tmp/jenkins-backup/jenkins jenkins-1-37nux:/var/lib
```
Depending on the application, you may be required to restart the application.
Restart the application with new data (optional):
```
$ oc delete pod jenkins-1-37nux
```
Alternatively, you can scale down the deployment to 0, and then up again:
```
$ oc scale --replicas=0 dc/jenkins
$ oc scale --replicas=1 dc/jenkins
```