This is a cache of https://docs.okd.io/4.15/backup_and_restore/application_backup_and_restore/oadp-3scale/backing-up-and-restoring-3scale-by-using-oadp.html. It is a snapshot of the page at 2025-02-18T21:25:56.892+0000.
Backing up and restoring 3scale by using OADP - OADP Application backup and restore | Backup and restore | OKD 4.15
  1. Documentation
×

Backing up and restoring 3scale by using OADP

With Red Hat 3scale API Management (APIM), you can manage your APIs for internal or external users. Share, secure, distribute, control, and monetize your APIs on an infrastructure platform built with performance, customer control, and future growth in mind. You can deploy 3scale components on-premise, in the cloud, as a managed service, or in any combination based on your requirement.

In this example, the non-service affecting approach is used to back up and restore 3scale on-cluster storage by using the OpenShift API for Data Protection (OADP) Operator. Additionally, ensure that you are restoring 3scale on the same cluster where it was backed up from. If you want to restore 3scale on a different cluster, ensure that both clusters are using the same custom domain.
Prerequisites

Creating the Data Protection Application

You can create a Data Protection Application (DPA) custom resource (CR) for 3scale. For more information on DPA, see "Installing the Data Protection Application".

Procedure

  1. Create a YAML file with the following configuration:

    Example dpa.yaml file
    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: dpa_sample
  namespace: openshift-adp
spec:
  configuration:
    velero:
      defaultPlugins:
        - openshift
        - aws
        - csi
      resourceTimeout: 10m
    nodeAgent:
      enable: true
      uploaderType: kopia
  backupLocations:
    - name: default
      velero:
        provider: aws
        default: true
        objectStorage:
          bucket: <bucket_name> (1)
          prefix: <prefix> (2)
        config:
          region: <region> (3)
          profile: "default"
          s3ForcePathStyle: "true"
          s3Url: <s3_url> (4)
        credential:
          key: cloud
          name: cloud-credentials
    1 Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
    2 Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.
    3 Specify a region for backup storage location.
    4 Specify the URL of the object store that you are using to store backups.

  2. Create the DPA CR by running the following command:

    $ oc create -f dpa.yaml
Next steps

  • Back up the 3scale Operator.

Additional resources

Backing up the 3scale Operator

You can back up the Operator resources, and Secret and APIManager custom resources (CR). For more information, see "Creating a Backup CR".

Prerequisites

  • You created the Data Protection Application (DPA).

Procedure

  1. Back up the Operator resources, such as operatorgroup, namespaces, and subscriptions, by creating a YAML file with the following configuration:

    Example backup.yaml file
    apiVersion: velero.io/v1
kind: Backup
metadata:
  name: operator-install-backup
  namespace: openshift-adp
spec:
  csiSnapshotTimeout: 10m0s
  defaultVolumesToFsBackup: false
  includedNamespaces:
  - threescale   (1)
  includedResources:
  - operatorgroups
  - subscriptions
  - namespaces
  itemOperationTimeout: 1h0m0s
  snapshotMoveData: false
  ttl: 720h0m0s
    1 Namespace where the 3scale Operator is installed.

    You can also back up and restore ReplicationControllers, deployment, and Pod objects to ensure that all manually set environments are backed up and restored. This does not affect the flow of restoration.

  2. Create a backup CR by running the following command:

    $ oc create -f backup.yaml

  3. Back up the Secret CR by creating a YAML file with the following configuration:

    Example backup-secret.yaml file
    apiVersion: velero.io/v1
kind: Backup
metadata:
  name: operator-resources-secrets
  namespace: openshift-adp
spec:
  csiSnapshotTimeout: 10m0s
  defaultVolumesToFsBackup: false
  includedNamespaces:
  - threescale
  includedResources:
  - secrets
  itemOperationTimeout: 1h0m0s
  labelSelector:
    matchLabels:
      app: 3scale-api-management
  snapshotMoveData: false
  snapshotVolumes: false
  ttl: 720h0m0s

  4. Create the Secret CR by running the following command:

    $ oc create -f backup-secret.yaml

  5. Back up the APIManager CR by creating a YAML file with the following configuration:

    Example backup-apimanager.yaml file
    apiVersion: velero.io/v1
kind: Backup
metadata:
  name: operator-resources-apim
  namespace: openshift-adp
spec:
  csiSnapshotTimeout: 10m0s
  defaultVolumesToFsBackup: false
  includedNamespaces:
  - threescale
  includedResources:
  - apimanagers
  itemOperationTimeout: 1h0m0s
  snapshotMoveData: false
  snapshotVolumes: false
  storageLocation: ts-dpa-1
  ttl: 720h0m0s
  volumeSnapshotLocations:
  - ts-dpa-1

  6. Create the APIManager CR by running the following command:

    $ oc create -f backup-apimanager.yaml
Next steps

  • Back up the mysql database.

Additional resources

Backing up the mysql database

You can back up the mysql database by creating and attaching a persistent volume claim (PVC) to include the dumped data in the specified path.

Prerequisites

  • You have backed up the 3scale operator.

Procedure

  1. Create a YAML file with the following configuration for adding an additional PVC:

    Example ts_pvc.yaml file
    kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: example-claim
  namespace: threescale
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: gp3-csi
  volumeMode: Filesystem

  2. Create the additional PVC by running the following command:

    $ oc create -f ts_pvc.yml

  3. Attach the PVC to the system database pod by editing the system database deployment to use the mysql dump:

    $ oc edit deployment system-mysql -n threescale
      volumeMounts:
    - name: example-claim
      mountPath: /var/lib/mysqldump/data
    - name: mysql-storage
      mountPath: /var/lib/mysql/data
    - name: mysql-extra-conf
      mountPath: /etc/my-extra.d
    - name: mysql-main-conf
      mountPath: /etc/my-extra
    ...
      serviceAccount: amp
  volumes:
        - name: example-claim
          persistentVolumeClaim:
            claimName: example-claim (1)
    ...
    1 The PVC that contains the dumped data.

  4. Create a YAML file with following configuration to back up the mysql database:

    Example mysql.yaml file
    apiVersion: velero.io/v1
kind: Backup
metadata:
  name: mysql-backup
  namespace: openshift-adp
spec:
  csiSnapshotTimeout: 10m0s
  defaultVolumesToFsBackup: true
  hooks:
    resources:
    - name: dumpdb
      pre:
      - exec:
          command:
          - /bin/sh
          - -c
          - mysqldump -u $MYSQL_USER --password=$MYSQL_PASSWORD system --no-tablespaces
            > /var/lib/mysqldump/data/dump.sql (1)
          container: system-mysql
          onError: Fail
          timeout: 5m
  includedNamespaces: (2)
  - threescale
  includedResources:
  - deployment
  - pods
  - replicationControllers
  - persistentvolumeclaims
  - persistentvolumes
  itemOperationTimeout: 1h0m0s
  labelSelector:
    matchLabels:
      app: 3scale-api-management
      threescale_component_element: mysql
  snapshotMoveData: false
  ttl: 720h0m0s
    1 A directory where the data is backed up.
    2 Resources to back up.

  5. Back up the mysql database by running the following command:

    $ oc create -f mysql.yaml
Verification

  • Verify that the mysql backup is completed by running the following command:

    $ oc get backups.velero.io mysql-backup
    Example output
    NAME                  STATUS      CREATED   NAMESPACE    POD                    VOLUME          UPLOADER TYPE   STORAGE LOCATION   AGE
mysql-backup-4g7qn   Completed   30s       threescale   system-mysql-2-9pr44   example-claim   kopia           ts-dpa-1           30s
mysql-backup-smh85   Completed   23s       threescale   system-mysql-2-9pr44   mysql-storage   kopia           ts-dpa-1           30s
Next steps

  • Back up the back-end Redis database.

Backing up the back-end Redis database

You can back up the Redis database by adding the required annotations and by listing which resources to back up using the includedResources parameter.

Prerequisites

  • You backed up the 3scale Operator.

  • You backed up the mysql database.

  • The Redis queues have been drained before performing the backup.

Procedure

  1. Edit the annotations on the backend-redis deployment by running the following command:

    $ oc edit deployment backend-redis -n threescale

  2. Add the following annotations:

    annotations:
post.hook.backup.velero.io/command: >-
         ["/bin/bash", "-c", "redis-cli CONFIG SET auto-aof-rewrite-percentage
         100"]
       pre.hook.backup.velero.io/command: >-
         ["/bin/bash", "-c", "redis-cli CONFIG SET auto-aof-rewrite-percentage
         0"]

  3. Create a YAML file with the following configuration to back up the Redis database:

    Example redis-backup.yaml file
    apiVersion: velero.io/v1
kind: Backup
metadata:
  name: redis-backup
  namespace: openshift-adp
spec:
  csiSnapshotTimeout: 10m0s
  defaultVolumesToFsBackup: true
  includedNamespaces:
  - threescale
  includedResources:
  - deployment
  - pods
  - replicationcontrollers
  - persistentvolumes
  - persistentvolumeclaims
  itemOperationTimeout: 1h0m0s
  labelSelector:
    matchLabels:
      app: 3scale-api-management
      threescale_component: backend
      threescale_component_element: redis
  snapshotMoveData: false
  snapshotVolumes: false
  ttl: 720h0m0s

  4. Back up the Redis database by running the following command:

    $ oc get backups.velero.io redis-backup -o yaml
Verification

  • Verify that the Redis backup is completed by running the following command::

    $ oc get backups.velero.io
Next steps

  • Restore the Secrets and APIManager CRs.

Restoring the secrets and APIManager

You can restore the Secrets and APIManager by using the following procedure.

Prerequisites

  • You backed up the 3scale Operator.

  • You backed up mysql and Redis databases.

  • You are restoring the database on the same cluster, where it was backed up.

    If it is on a different cluster, install and configure OADP with nodeAgent enabled on the destination cluster as it was on the source cluster.

Procedure

  1. Delete the 3scale Operator custom resource definitions (CRDs) along with the threescale namespace by running the following command:

    $ oc delete project threescale
    Example output
    "threescale" project deleted successfully

  2. Create a YAML file with the following configuration to restore the 3scale Operator:

    Example restore.yaml file
    apiVersion: velero.io/v1
kind: Restore
metadata:
  name: operator-installation-restore
  namespace: openshift-adp
spec:
  backupName: operator-install-backup
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  - csinodes.storage.k8s.io
  - volumeattachments.storage.k8s.io
  - backuprepositories.velero.io
  itemOperationTimeout: 4h0m0s

  3. Restore the 3scale Operator by running the following command:

    $ oc create -f restore.yaml

  4. Manually create the s3-credentials Secret object by running the following command:

    $ oc apply -f - <<EOF
---
apiVersion: v1
kind: Secret
metadata:
      name: s3-credentials
      namespace: threescale
stringData:
  AWS_ACCESS_KEY_ID: <ID_123456> (1)
  AWS_SECRET_ACCESS_KEY: <ID_98765544> (2)
  AWS_BUCKET: <mybucket.example.com> (3)
  AWS_REGION: <us-east-1> (4)
type: Opaque
EOF
    1 Replace <ID_123456> with your AWS credentials ID.
    2 Replace <ID_98765544> with your AWS credentials KEY.
    3 Replace <mybucket.example.com> with your target bucket name.
    4 Replace <us-east-1> with the AWS region of your bucket.

  5. Scale down the 3scale Operator by running the following command:

    $ oc scale deployment threescale-operator-controller-manager-v2 --replicas=0 -n threescale

  6. Create a YAML file with the following configuration to restore the Secrets:

    Example restore-secret.yaml file
    apiVersion: velero.io/v1
kind: Restore
metadata:
  name: operator-resources-secrets
  namespace: openshift-adp
spec:
  backupName: operator-resources-secrets
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  - csinodes.storage.k8s.io
  - volumeattachments.storage.k8s.io
  - backuprepositories.velero.io
  itemOperationTimeout: 4h0m0s

  7. Restore the Secrets by running the following command:

    $ oc create -f restore-secrets.yaml

  8. Create a YAML file with the following configuration to restore APIManager:

    Example restore-apimanager.yaml file
    apiVersion: velero.io/v1
kind: Restore
metadata:
  name: operator-resources-apim
  namespace: openshift-adp
spec:
  backupName: operator-resources-apim
  excludedResources: (1)
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  - csinodes.storage.k8s.io
  - volumeattachments.storage.k8s.io
  - backuprepositories.velero.io
  itemOperationTimeout: 4h0m0s
    1 The resources that you do not want to restore.

  9. Restore the APIManager by running the following command:

    $ oc create -f restore-apimanager.yaml

  10. Scale up the 3scale Operator by running the following command:

    $ oc scale deployment threescale-operator-controller-manager-v2 --replicas=1 -n threescale
Next steps

  • Restore the mysql database.

Restoring the mysql database

Restoring the mysql database re-creates the following resources:

  • The Pod, ReplicationController, and deployment objects.

  • The additional persistent volumes (PVs) and associated persistent volume claims (PVCs).

  • The mysql dump, which the example-claim PVC contains.

Do not delete the default PV and PVC associated with the database. If you do, your backups are deleted.
Prerequisites

  • You restored the Secret and APIManager custom resources (CR).

Procedure

  1. Scale down the 3scale Operator by running the following command:

    $ oc scale deployment threescale-operator-controller-manager-v2 --replicas=0 -n threescale
    Example output:
    deployment.apps/threescale-operator-controller-manager-v2 scaled

  2. Create the following script to scale down the 3scale operator:

    $ vi ./scaledowndeployment.sh
    Example output:
    for deployment in apicast-production apicast-staging backend-cron backend-listener backend-redis backend-worker system-app system-memcache system-mysql system-redis system-searchd system-sidekiq zync zync-database zync-que; do
    oc scale deployment/$deployment --replicas=0 -n threescale
done

  3. Scale down all the deployment 3scale components by running the following script:

    $ ./scaledowndeployment.sh
    Example output:
    deployment.apps.openshift.io/apicast-production scaled
deployment.apps.openshift.io/apicast-staging scaled
deployment.apps.openshift.io/backend-cron scaled
deployment.apps.openshift.io/backend-listener scaled
deployment.apps.openshift.io/backend-redis scaled
deployment.apps.openshift.io/backend-worker scaled
deployment.apps.openshift.io/system-app scaled
deployment.apps.openshift.io/system-memcache scaled
deployment.apps.openshift.io/system-mysql scaled
deployment.apps.openshift.io/system-redis scaled
deployment.apps.openshift.io/system-searchd scaled
deployment.apps.openshift.io/system-sidekiq scaled
deployment.apps.openshift.io/zync scaled
deployment.apps.openshift.io/zync-database scaled
deployment.apps.openshift.io/zync-que scaled

  4. Delete the system-mysql deployment object by running the following command:

    $ oc delete deployment system-mysql -n threescale
    Example output:
    Warning: apps.openshift.io/v1 deployment is deprecated in v4.14+, unavailable in v4.10000+
deployment.apps.openshift.io "system-mysql" deleted

  5. Create the following YAML file to restore the mysql database:

    Example restore-mysql.yaml file
    apiVersion: velero.io/v1
kind: Restore
metadata:
  name: restore-mysql
  namespace: openshift-adp
spec:
  backupName: mysql-backup
  excludedResources:
    - nodes
    - events
    - events.events.k8s.io
    - backups.velero.io
    - restores.velero.io
    - csinodes.storage.k8s.io
    - volumeattachments.storage.k8s.io
    - backuprepositories.velero.io
    - resticrepositories.velero.io
  hooks:
    resources:
      - name: restoreDB
        postHooks:
          - exec:
              command:
                - /bin/sh
                - '-c'
                - >
                  sleep 30

                  mysql -h 127.0.0.1 -D system -u root
                  --password=$MYSQL_ROOT_PASSWORD <
                  /var/lib/mysqldump/data/dump.sql (1)
              container: system-mysql
              execTimeout: 80s
              onError: Fail
              waitTimeout: 5m
  itemOperationTimeout: 1h0m0s
  restorePVs: true
    1 A path where the data is restored from.

  6. Restore the mysql database by running the following command:

    $ oc create -f restore-mysql.yaml
Verification

  1. Verify that the PodVolumeRestore restore is completed by running the following command:

    $ oc get podvolumerestores.velero.io -n openshift-adp
    Example output:
    NAME                    NAMESPACE    POD                     UPLOADER TYPE   VOLUME                  STATUS      TOTALBYTES   BYTESDONE   AGE
restore-mysql-rbzvm     threescale   system-mysql-2-kjkhl    kopia           mysql-storage           Completed   771879108    771879108   40m
restore-mysql-z7x7l     threescale   system-mysql-2-kjkhl    kopia           example-claim           Completed   380415       380415      40m

  2. Verify that the additional PVC has been restored by running the following command:

    $ oc get pvc -n threescale
    Example output:
    NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
backend-redis-storage   Bound    pvc-3dca410d-3b9f-49d4-aebf-75f47152e09d   1Gi        RWO            gp3-csi        <unset>                 68m
example-claim           Bound    pvc-cbaa49b0-06cd-4b1a-9e90-0ef755c67a54   1Gi        RWO            gp3-csi        <unset>                 57m
mysql-storage           Bound    pvc-4549649f-b9ad-44f7-8f67-dd6b9dbb3896   1Gi        RWO            gp3-csi        <unset>                 68m
system-redis-storage    Bound    pvc-04dadafd-8a3e-4d00-8381-6041800a24fc   1Gi        RWO            gp3-csi        <unset>                 68m
system-searchd          Bound    pvc-afbf606c-d4a8-4041-8ec6-54c5baf1a3b9   1Gi        RWO            gp3-csi        <unset>                 68m
Next steps

  • Restore the back-end Redis database.

Restoring the back-end Redis database

You can restore the back-end Redis database by deleting the deployment and specifying which resources you do not want to restore.

Prerequisites

  • You restored the Secret and APIManager custom resources.

  • You restored the mysql database.

Procedure

  1. Delete the backend-redis deployment by running the following command:

    $ oc delete deployment backend-redis -n threescale
    Example output:
    Warning: apps.openshift.io/v1 deployment is deprecated in v4.14+, unavailable in v4.10000+

deployment.apps.openshift.io "backend-redis" deleted

  2. Create a YAML file with the following configuration to restore the Redis database:

    Example restore-backend.yaml file
    apiVersion: velero.io/v1
kind: Restore
metadata:
  name: restore-backend
  namespace: openshift-adp
spec:
  backupName: redis-backup
  excludedResources:
    - nodes
    - events
    - events.events.k8s.io
    - backups.velero.io
    - restores.velero.io
    - resticrepositories.velero.io
    - csinodes.storage.k8s.io
    - volumeattachments.storage.k8s.io
    - backuprepositories.velero.io
  itemOperationTimeout: 1h0m0s
  restorePVs: true

  3. Restore the Redis database by running the following command:

    $ oc create -f restore-backend.yaml
Verification

  • Verify that the PodVolumeRestore restore is completed by running the following command:

    $ oc get podvolumerestores.velero.io -n openshift-adp
    Example output:
    NAME                    NAMESPACE    POD                     UPLOADER TYPE   VOLUME                  STATUS      TOTALBYTES   BYTESDONE   AGE
restore-backend-jmrwx   threescale   backend-redis-1-bsfmv   kopia           backend-redis-storage   Completed   76123        76123       21m
Next steps

  • Scale the 3scale Operator and deployment.

Scaling up the 3scale Operator and deployment

You can scale up the 3scale Operator and any deployment that was manually scaled down. After a few minutes, 3scale installation should be fully functional, and its state should match the backed-up state.

Prerequisites

  • Ensure that there are no scaled up deployments or no extra pods running. There might be some system-mysql or backend-redis pods running detached from deployments after restoration, which can be removed after the restoration is successful.

Procedure

  1. Scale up the 3scale Operator by running the following command:

    $ oc scale deployment threescale-operator-controller-manager-v2 --replicas=1 -n threescale

  2. Ensure that the 3scale Operator was deployed by running the following command:

    $ oc get deployment -n threescale

  3. Scale up the deployments by executing the following script:

    $ ./scaledeployment.sh

  4. Get the 3scale-admin route to log in to the 3scale UI by running the following command:

    $ oc get routes -n threescale
    Example output
    NAME                         HOST/PORT                                                                   PATH   SERVICES             PORT      TERMINATION     WILDCARD
backend                      backend-3scale.apps.custom-cluster-name.openshift.com                         backend-listener     http      edge/Allow      None
zync-3scale-api-b4l4d        api-3scale-apicast-production.apps.custom-cluster-name.openshift.com          apicast-production   gateway   edge/Redirect   None
zync-3scale-api-b6sns        api-3scale-apicast-staging.apps.custom-cluster-name.openshift.com             apicast-staging      gateway   edge/Redirect   None
zync-3scale-master-7sc4j     master.apps.custom-cluster-name.openshift.com                                 system-master        http      edge/Redirect   None
zync-3scale-provider-7r2nm   3scale-admin.apps.custom-cluster-name.openshift.com                           system-provider      http      edge/Redirect   None
zync-3scale-provider-mjxlb   3scale.apps.custom-cluster-name.openshift.com                                 system-developer     http      edge/Redirect   None

    In this example, 3scale-admin.apps.custom-cluster-name.openshift.com is the 3scale-admin URL.

  5. Use the URL from this output to log in to the 3scale Operator as an administrator. You can verify that the existing data is available before trying to create a backup.