Restoring applications - OADP Application backup and restore | Backup and restore

Previewing resources before running backup and restore
Creating a Restore CR
Creating restore hooks

Restore application backups by previewing resources before running the restore, creating a Restore custom resource (CR), and configuring restore hooks to run commands in restored pods. This helps you to recover your application data and configuration while controlling the restore process.

Previewing resources before running backup and restore

Preview the backup and restore resources in advance by doing a dry run of the backup and restore operations. This helps you to verify which resources will be included before committing to a full backup or restore.

OADP backs up application resources based on the type, namespace, or label. This means that you can view the resources after the backup is complete. Similarly, you can view the restored objects based on the namespace, persistent volume (PV), or label after a restore operation is complete.

Prerequisites

You have installed the OADP Operator.

Procedure

To preview the resources included in the backup before running the actual backup, run the following command:
```
$ velero backup create <backup-name> --snapshot-volumes false
```
Specify the value of --snapshot-volumes parameter as false.
To know more details about the backup resources, run the following command:
```
$ velero describe backup <backup_name> --details
```
Replace <backup_name> with the name of the backup.
To preview the resources included in the restore before running the actual restore, run the following command:
```
$ velero restore create --from-backup <backup_name>
```
Replace <backup_name> with the name of the backup.

The velero restore create command creates restore resources in the cluster. You must delete the resources created as part of the restore, after you review the resources.
To know more details about the restore resources, run the following command:
```
$ velero describe restore <restore_name> --details
```
Replace <restore_name> with the name of the restore.

Creating a Restore CR

Restore a Backup custom resource (CR) by creating a Restore CR.

When you restore a stateful application that uses the azurefile-csi storage class, the restore operation remains in the Finalizing phase.

Prerequisites

You must install the OpenShift API for Data Protection (OADP) Operator.
The DataProtectionApplication CR must be in a Ready state.
You must have a Velero Backup CR.
The persistent volume (PV) capacity must match the requested size at backup time. Adjust the requested size if needed.

Procedure

Create a Restore CR, as in the following example:
```
apiVersion: velero.io/v1
kind: Restore
metadata:
  name: <restore>
  namespace: openshift-adp
spec:
  backupName: <backup>
  includedResources: []
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  restorePVs: true
```
where:

<backup>

Specifies the name of the Backup CR.

includedResources

Optional: Specifies an array of resources to include in the restore process. Resources might be shortcuts (for example, po for pods) or fully-qualified. If unspecified, all resources are included.

restorePVs: true

Optional: The restorePVs parameter can be set to false to turn off restore of PersistentVolumes from VolumeSnapshot of Container Storage Interface (CSI) snapshots or from native snapshots when VolumeSnapshotLocation is configured.

Verify that the status of the Restore CR is Completed by entering the following command:

$ oc get restores.velero.io -n openshift-adp <restore> -o jsonpath='{.status.phase}'

Verify that the backup resources have been restored by entering the following command:
```
$ oc get all -n <namespace>
```
where:

<namespace>

Specifies the namespace that you backed up.

If you restore deploymentconfig with volumes or if you use post-restore hooks, run the dc-post-restore.sh cleanup script by entering the following command:

$ bash dc-restic-post-restore.sh -> dc-post-restore.sh

During the restore process, the OADP Velero plug-ins scale down the deploymentconfig objects and restore the pods as standalone pods. This is done to prevent the cluster from deleting the restored deploymentconfig pods immediately on restore and to allow the restore and post-restore hooks to complete their actions on the restored pods. The cleanup script shown below removes these disconnected pods and scales any deploymentconfig objects back up to the appropriate number of replicas.

#!/bin/bash
set -e

# if sha256sum exists, use it to check the integrity of the file
if command -v sha256sum >/dev/null 2>&1; then
  CHECKSUM_CMD="sha256sum"
else
  CHECKSUM_CMD="shasum -a 256"
fi

label_name () {
    if [ "${#1}" -le "63" ]; then
	echo $1
	return
    fi
    sha=$(echo -n $1|$CHECKSUM_CMD)
    echo "${1:0:57}${sha:0:6}"
}

if [[ $# -ne 1 ]]; then
    echo "usage: ${BASH_SOURCE} restore-name"
    exit 1
fi

echo "restore: $1"

label=$(label_name $1)
echo "label:   $label"

echo Deleting disconnected restore pods
oc delete pods --all-namespaces -l oadp.openshift.io/disconnected-from-dc=$label

for dc in $(oc get dc --all-namespaces -l oadp.openshift.io/replicas-modified=$label -o jsonpath='{range .items[*]}{.metadata.namespace}{","}{.metadata.name}{","}{.metadata.annotations.oadp\.openshift\.io/original-replicas}{","}{.metadata.annotations.oadp\.openshift\.io/original-paused}{"\n"}')
do
    IFS=',' read -ra dc_arr <<< "$dc"
    if [ ${#dc_arr[0]} -gt 0 ]; then
	echo Found deployment ${dc_arr[0]}/${dc_arr[1]}, setting replicas: ${dc_arr[2]}, paused: ${dc_arr[3]}
	cat <<EOF | oc patch dc  -n ${dc_arr[0]} ${dc_arr[1]} --patch-file /dev/stdin
spec:
  replicas: ${dc_arr[2]}
  paused: ${dc_arr[3]}
EOF
    fi
done

Creating restore hooks

Create restore hooks to run commands in a container in a pod by editing the Restore custom resource (CR).

You can create two types of restore hooks:

An init hook adds an init container to a pod to perform setup tasks before the application container starts.

If you restore a Restic backup, the restic-wait init container is added before the restore hook init container.
An exec hook runs commands or scripts in a container of a restored pod.

Procedure

Add a hook to the spec.hooks block of the Restore CR, as in the following example:

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: <restore>
  namespace: openshift-adp
spec:
  hooks:
    resources:
      - name: <hook_name>
        includedNamespaces:
        - <namespace>
        excludedNamespaces:
        - <namespace>
        includedResources:
        - pods
        excludedResources: []
        labelSelector:
          matchLabels:
            app: velero
            component: server
        postHooks:
        - init:
            initContainers:
            - name: restore-hook-init
              image: alpine:latest
              volumeMounts:
              - mountPath: /restores/pvc1-vm
                name: pvc1-vm
              command:
              - /bin/ash
              - -c
            timeout:
        - exec:
            container: <container>
            command:
            - /bin/bash
            - -c
            - "psql < /backup/backup.sql"
            waitTimeout: 5m
            execTimeout: 1m
            onError: Continue

where:

<namespace>

Optional: Specifies an array of namespaces to which the hook applies. If this value is not specified, the hook applies to all namespaces.

pods

Currently, pods are the only supported resource that hooks can apply to.

labelSelector

Optional: This hook only applies to objects matching the label selector.

timeout

Optional: Specifies the maximum length of time Velero waits for initContainers to complete.

<container>

Optional: Specifies the container in which the command runs. If the container is not specified, the command runs in the first container in the pod.

/bin/bash

Specifies the entrypoint for the init container being added.

waitTimeout: 5m

Optional: Specifies how long to wait for a container to become ready. This should be long enough for the container to start and for any preceding hooks in the same container to complete. If not set, the restore process waits indefinitely.

execTimeout: 1m

Optional: Specifies how long to wait for the commands to run. The default is 30s.

onError: Continue

Specifies the error handling behavior. Allowed values are Fail and Continue:

Continue: Only command failures are logged.
Fail: No more restore hooks run in any container in any pod. The status of the Restore CR will be PartiallyFailed.

During a File System Backup (FSB) restore operation, a Deployment resource referencing an ImageStream is not restored properly. The restored pod that runs the FSB, and the postHook is terminated prematurely.

This happens because, during the restore operation, OpenShift controller updates the spec.template.spec.containers[0].image field in the Deployment resource with an updated ImageStreamTag hash. The update triggers the rollout of a new pod, terminating the pod on which velero runs the FSB and the post restore hook. For more information about image stream trigger, see "Triggering updates on image stream changes".

The workaround for this behavior is a two-step restore process:

First, perform a restore excluding the Deployment resources, for example:

$ velero restore create <RESTORE_NAME> \
  --from-backup <BACKUP_NAME> \
  --exclude-resources=deployment.apps

After the first restore is successful, perform a second restore by including these resources, for example:

$ velero restore create <RESTORE_NAME> \
  --from-backup <BACKUP_NAME> \
  --include-resources=deployment.apps

Additional resources

Triggering updates on image stream changes