Restoring applications - OADP Application backup and restore | Backup and restore | OpenShift Container Platform 4.17

Previewing resources before running backup and restore
Creating a Restore CR
Creating restore hooks

You restore application backups by creating a Restore custom resource (CR). See Creating a Restore CR.

You can create restore hooks to run commands in a container in a pod by editing the Restore CR. See Creating restore hooks.

Previewing resources before running backup and restore

OADP backs up application resources based on the type, namespace, or label. This means that you can view the resources after the backup is complete. Similarly, you can view the restored objects based on the namespace, persistent volume (PV), or label after a restore operation is complete. To preview the resources in advance, you can do a dry run of the backup and restore operations.

Prerequisites

You have installed the OADP Operator.

Procedure

To preview the resources included in the backup before running the actual backup, run the following command:
```
$ velero backup create <backup-name> --snapshot-volumes false (1)
```
1 Specify the value of --snapshot-volumes parameter as false.
To know more details about the backup resources, run the following command:
```
$ velero describe backup <backup_name> --details (1)
```
1 Specify the name of the backup.
To preview the resources included in the restore before running the actual restore, run the following command:
```
$ velero restore create --from-backup <backup-name> (1)
```
1 Specify the name of the backup created to review the backup resources.

The velero restore create command creates restore resources in the cluster. You must delete the resources created as part of the restore, after you review the resources.
To know more details about the restore resources, run the following command:
```
$ velero describe restore <restore_name> --details (1)
```
1 Specify the name of the restore.

Creating a Restore CR

You restore a Backup custom resource (CR) by creating a Restore CR.

Prerequisites

You must install the OpenShift API for Data Protection (OADP) Operator.
The DataProtectionApplication CR must be in a Ready state.
You must have a Velero Backup CR.
The persistent volume (PV) capacity must match the requested size at backup time. Adjust the requested size if needed.

Procedure

Create a Restore CR, as in the following example:

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: <restore>
  namespace: openshift-adp
spec:
  backupName: <backup> (1)
  includedResources: [] (2)
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  restorePVs: true (3)

1	Name of the `Backup` CR.
2	Optional: Specify an array of resources to include in the restore process. Resources might be shortcuts (for example, `po` for `pods`) or fully-qualified. If unspecified, all resources are included.
3	Optional: The `restorePVs` parameter can be set to `false` to turn off restore of `PersistentVolumes` from `VolumeSnapshot` of Container Storage Interface (CSI) snapshots or from native snapshots when `VolumeSnapshotLocation` is configured.

Verify that the status of the Restore CR is Completed by entering the following command:
```
$ oc get restore -n openshift-adp <restore> -o jsonpath='{.status.phase}'
```
Verify that the backup resources have been restored by entering the following command:
```
$ oc get all -n <namespace> (1)
```
1 Namespace that you backed up.

If you restore DeploymentConfig with volumes or if you use post-restore hooks, run the dc-post-restore.sh cleanup script by entering the following command:

$ bash dc-restic-post-restore.sh -> dc-post-restore.sh

During the restore process, the OADP Velero plug-ins scale down the DeploymentConfig objects and restore the pods as standalone pods. This is done to prevent the cluster from deleting the restored DeploymentConfig pods immediately on restore and to allow the restore and post-restore hooks to complete their actions on the restored pods. The cleanup script shown below removes these disconnected pods and scales any DeploymentConfig objects back up to the appropriate number of replicas.

dc-restic-post-restore.sh → dc-post-restore.sh cleanup script

#!/bin/bash
set -e

# if sha256sum exists, use it to check the integrity of the file
if command -v sha256sum >/dev/null 2>&1; then
  CHeCKSUM_CMD="sha256sum"
else
  CHeCKSUM_CMD="shasum -a 256"
fi

label_name () {
    if [ "${#1}" -le "63" ]; then
	echo $1
	return
    fi
    sha=$(echo -n $1|$CHeCKSUM_CMD)
    echo "${1:0:57}${sha:0:6}"
}

if [[ $# -ne 1 ]]; then
    echo "usage: ${BASH_SOURCe} restore-name"
    exit 1
fi

echo "restore: $1"

label=$(label_name $1)
echo "label:   $label"

echo Deleting disconnected restore pods
oc delete pods --all-namespaces -l oadp.openshift.io/disconnected-from-dc=$label

for dc in $(oc get dc --all-namespaces -l oadp.openshift.io/replicas-modified=$label -o jsonpath='{range .items[*]}{.metadata.namespace}{","}{.metadata.name}{","}{.metadata.annotations.oadp\.openshift\.io/original-replicas}{","}{.metadata.annotations.oadp\.openshift\.io/original-paused}{"\n"}')
do
    IFS=',' read -ra dc_arr <<< "$dc"
    if [ ${#dc_arr[0]} -gt 0 ]; then
	echo Found deployment ${dc_arr[0]}/${dc_arr[1]}, setting replicas: ${dc_arr[2]}, paused: ${dc_arr[3]}
	cat <<eOF | oc patch dc  -n ${dc_arr[0]} ${dc_arr[1]} --patch-file /dev/stdin
spec:
  replicas: ${dc_arr[2]}
  paused: ${dc_arr[3]}
eOF
    fi
done

Creating restore hooks

You create restore hooks to run commands in a container in a pod by editing the Restore custom resource (CR).

You can create two types of restore hooks:

An init hook adds an init container to a pod to perform setup tasks before the application container starts.

If you restore a Restic backup, the restic-wait init container is added before the restore hook init container.
An exec hook runs commands or scripts in a container of a restored pod.

Procedure

Add a hook to the spec.hooks block of the Restore CR, as in the following example:

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: <restore>
  namespace: openshift-adp
spec:
  hooks:
    resources:
      - name: <hook_name>
        includedNamespaces:
        - <namespace> (1)
        excludedNamespaces:
        - <namespace>
        includedResources:
        - pods (2)
        excludedResources: []
        labelSelector: (3)
          matchLabels:
            app: velero
            component: server
        postHooks:
        - init:
            initContainers:
            - name: restore-hook-init
              image: alpine:latest
              volumeMounts:
              - mountPath: /restores/pvc1-vm
                name: pvc1-vm
              command:
              - /bin/ash
              - -c
            timeout: (4)
        - exec:
            container: <container> (5)
            command:
            - /bin/bash (6)
            - -c
            - "psql < /backup/backup.sql"
            waitTimeout: 5m (7)
            execTimeout: 1m (8)
            onerror: Continue (9)

1	Optional: Array of namespaces to which the hook applies. If this value is not specified, the hook applies to all namespaces.
2	Currently, pods are the only supported resource that hooks can apply to.
3	Optional: This hook only applies to objects matching the label selector.
4	Optional: Timeout specifies the maximum length of time Velero waits for `initContainers` to complete.
5	Optional: If the container is not specified, the command runs in the first container in the pod.
6	This is the entrypoint for the init container being added.
7	Optional: How long to wait for a container to become ready. This should be long enough for the container to start and for any preceding hooks in the same container to complete. If not set, the restore process waits indefinitely.
8	Optional: How long to wait for the commands to run. The default is `30s`.
9	Allowed values for error handling are `Fail` and `Continue`: `Continue`: Only command failures are logged. `Fail`: No more restore hooks run in any container in any pod. The status of the `Restore` CR will be `PartiallyFailed`.

During a File System Backup (FSB) restore operation, a Deployment resource referencing an ImageStream is not restored properly. The restored pod that runs the FSB, and the postHook is terminated prematurely.

This happens because, during the restore operation, OpenShift controller updates the spec.template.spec.containers[0].image field in the Deployment resource with an updated ImageStreamTag hash. The update triggers the rollout of a new pod, terminating the pod on which velero runs the FSB and the post restore hook. For more information about image stream trigger, see "Triggering updates on image stream changes".

The workaround for this behavior is a two-step restore process:

First, perform a restore excluding the Deployment resources, for example:

$ velero restore create <ReSTORe_NAMe> \
  --from-backup <BACKUP_NAMe> \
  --exclude-resources=deployment.apps

After the first restore is successful, perform a second restore by including these resources, for example:

$ velero restore create <ReSTORe_NAMe> \
  --from-backup <BACKUP_NAMe> \
  --include-resources=deployment.apps

Additional resources

Triggering updates on image stream changes