Upgrading using the roxctl CLI | Upgrading | Red Hat Advanced Cluster Security for Kubernetes 3.70

Backing up the Central database
Upgrading the Central cluster
Upgrading all secured clusters
Rolling back Central
- Rolling back Central normally
- Rolling back Central forcefully
Verifying upgrades
Revoking the API token

You can upgrade to the latest version of Red Hat Advanced Cluster Security for Kubernetes from a supported older version.

To upgrade Red Hat Advanced Cluster Security for Kubernetes to the latest version, you must perform the following:

Backup the Central database
Upgrade Central
Upgrade the roxctl CLI
Upgrade Scanner
Verify that all secured clusters are upgraded

Backing up the Central database

You can back up the Central database and use that backup for rolling back from a failed upgrade or data restoration in the case of an infrastructure disaster.

Prerequisites

You must have an API token with read permission for all resources of Red Hat Advanced Cluster Security for Kubernetes. The Analyst system role has read permissions for all resources.
You have installed the roxctl CLI.
You have configured the ROX_API_TOKEN and the ROX_CENTRAL_ADDRESS environment variables.

Procedure

Run the backup command:
- For Red Hat Advanced Cluster Security for Kubernetes 3.0.55 and newer:
  $ roxctl -e "$ROX_CENTRAL_ADDRESS" central backup
- For Red Hat Advanced Cluster Security for Kubernetes 3.0.54 and older:
  $ roxctl -e "$ROX_CENTRAL_ADDRESS" central db backup

Additional resources

Authenticating using the roxctl CLI

Upgrading the Central cluster

After you have backed up the Central database, the next step is to upgrade the Central cluster. This step includes upgrading Central, the roxctl CLI, and the Scanner.

Upgrading Central

You can update Central to the latest version by downloading and deploying the updated images.

Prerequisites

If you deploy images from a private image registry, first push the new image into your private registry, and then replace your image registry for the commands in this section.

Procedure

Run the following commands to upgrade Central:

$ oc -n stackrox patch deploy/central -p '{"spec":{"template":{"spec":{"containers":[{"name":"central","env":[{"name":"ROX_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}]}]}}}}' (1)

1	If you use Kubernetes, enter `kubectl` instead of `oc`.

$ oc -n stackrox patch deployment/scanner -p '{"spec":{"template":{"spec":{"containers":[{"name":"scanner","securityContext":{"runAsUser":65534}}]}}}}' (1)

1	If you use Kubernetes, enter `kubectl` instead of `oc`.

$ oc -n stackrox set image deploy/central central=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:3.70.2 (1)

1	If you use Kubernetes, enter `kubectl` instead of `oc`.

If you are upgrading from Red Hat Advanced Cluster Security for Kubernetes 3.65.0, you must run the following additional command to create the stackrox-central-diagnostics role:
```
$ oc -n stackrox patch role stackrox-central-diagnostics -p '{"rules":[{"apiGroups":["*"],"resources":["deployments","daemonsets","replicasets","configmaps","services"],"verbs":["get","list"]}]}' (1)
```
1 If you use Kubernetes, enter kubectl instead of oc.

If you have not installed Red Hat Advanced Cluster Security for Kubernetes by using helm or Operator, and you want to enable authentication using the OpenShift OAuth server, you must run the following additional command:

$ oc -n stackrox set env deploy/central ROX_ENABLE_OPENSHIFT_AUTH=true

$ oc -n stackrox patch serviceaccount/central -p '
{
"metadata": {
"annotations": {
"serviceaccounts.openshift.io/oauth-redirecturi.main": "sso/providers/openshift/callback",
"serviceaccounts.openshift.io/oauth-redirectreference.main": "{"kind":"OAuthRedirectReference","apiVersion":"v1","reference":{"kind":"Route","name":"central"}}"
}
}
}'

Verification

Check that the new pods have deployed:
```
$ oc get deploy -n stackrox -o wide (1)
```
1 If you use Kubernetes, enter kubectl instead of oc.
```
$ oc get pod -n stackrox --watch (1)
```
1 If you use Kubernetes, enter kubectl instead of oc.

Upgrading the roxctl CLI

To upgrade the roxctl CLI to the latest version you must uninstall the existing version of roxctl CLI and then install the latest version of the roxctl CLI.

Uninstalling the roxctl CLI

You can uninstall the roxctl CLI binary on Linux by using the following procedure.

Procedure

Find and delete the roxctl binary:
```
$ ROXPATH=$(which roxctl) && rm -f $ROXPATH (1)
```
1 Depending on your environment, you might need administrator rights to delete the roxctl binary.

Installing the roxctl CLI on Linux

You can install the roxctl CLI binary on Linux by using the following procedure.

Procedure

Download the latest version of the roxctl CLI:

$ curl -O https://mirror.openshift.com/pub/rhacs/assets/3.70.2/bin/Linux/roxctl

Make the roxctl binary executable:
```
$ chmod +x roxctl
```
Place the roxctl binary in a directory that is on your PATH:

To check your PATH, execute the following command:
```
$ echo $PATH
```

Verification

Verify the roxctl version you have installed:
```
$ roxctl version
```

Installing the roxctl CLI on macOS

You can install the roxctl CLI binary on macOS by using the following procedure.

Procedure

Download the latest version of the roxctl CLI:

$ curl -O https://mirror.openshift.com/pub/rhacs/assets/3.70.2/bin/Darwin/roxctl

Remove all extended attributes from the binary:
```
$ xattr -c roxctl
```
Make the roxctl binary executable:
```
$ chmod +x roxctl
```
Place the roxctl binary in a directory that is on your PATH:

To check your PATH, execute the following command:
```
$ echo $PATH
```

Verification

Verify the roxctl version you have installed:
```
$ roxctl version
```

Installing the roxctl CLI on Windows

You can install the roxctl CLI binary on Windows by using the following procedure.

Procedure

Download the latest version of the roxctl CLI:

$ curl -O https://mirror.openshift.com/pub/rhacs/assets/3.70.2/bin/Windows/roxctl.exe

Verification

Verify the roxctl version you have installed:
```
$ roxctl version
```

After you upgrade the roxctl CLI you can upgrade Scanner.

Upgrading Scanner

You can update Scanner to the latest version by using the roxctl CLI.

Prerequisites

If you deploy images from a private image registry, first push the new image into your private registry, and then replace your image registry for the commands in this section.
If you used Red Hat UBI-based images when you installed Red Hat Advanced Cluster Security for Kubernetes, replace the image names for the commands in this section with the following UBI-based image names:
If you have created custom scanner configurations, you must apply those changes before updating the scanner configuration file:
```
$ roxctl -e "$ROX_CENTRAL_ADDRESS" scanner generate
```
```
$ oc apply -f scanner-bundle/scanner/02-scanner-03-tls-secret.yaml (1)
```
1 If you use Kubernetes, enter kubectl instead of oc.
```
$ oc apply -f scanner-bundle/scanner/02-scanner-04-scanner-config.yaml (1)
```
1 If you use Kubernetes, enter kubectl instead of oc.

Procedure

Update the Scanner image:

$ oc -n stackrox set image deploy/scanner scanner=registry.redhat.io/advanced-cluster-security/rhacs-scanner-rhel8:3.70.2 (1)

1	If you use Kubernetes, enter `kubectl` instead of `oc`.

Update the Scanner database image:

$ oc -n stackrox set image deploy/scanner-db db=registry.redhat.io/advanced-cluster-security/rhacs-scanner-db-rhel8:3.70.2 (1)

1	If you use Kubernetes, enter `kubectl` instead of `oc`.

$ oc -n stackrox set image deploy/scanner-db init-db=registry.redhat.io/advanced-cluster-security/rhacs-scanner-db-rhel8:3.70.2 (1)

1	If you use Kubernetes, enter `kubectl` instead of `oc`.

Verification

Check that the new pods have deployed successfully:
```
$ oc get pod -n stackrox --watch (1)
```
1 If you use Kubernetes, enter kubectl instead of oc.

Verifying the Central cluster upgrade

After you have upgraded both Central and Scanner, verify that the Central cluster upgrade is complete.

Procedure

Check the Central logs:

$ oc logs -n stackrox deploy/central -c central (1)

1	If you use Kubernetes, enter `kubectl` instead of `oc`.

If the upgrade is successful, you will see output similar to the following:

No database restore directory found (this is not an error).
Migrator: 2019/10/25 17:58:54: starting DB compaction
Migrator: 2019/10/25 17:58:54: Free fraction of 0.0391 (40960/1048576) is < 0.7500. Will not compact
badger 2019/10/25 17:58:54 INFO: All 1 tables opened in 2ms
badger 2019/10/25 17:58:55 INFO: Replaying file id: 0 at offset: 846357
badger 2019/10/25 17:58:55 INFO: Replay took: 50.324µs
badger 2019/10/25 17:58:55 DEBUG: Value log discard stats empty
Migrator: 2019/10/25 17:58:55: DB is up to date. Nothing to do here.
badger 2019/10/25 17:58:55 INFO: Got compaction priority: {level:0 score:1.73 dropPrefix:[]}
version: 2019/10/25 17:58:55.189866 ensure.go:49: Info: Version found in the DB was current. We’re good to go!

Upgrading all secured clusters

After upgrading Central services, you must upgrade all secured clusters.

If you are using automatic upgrades:
- Update all your secured clusters by using automatic upgrades.
- Skip the instructions in this section and follow the instructions in the Verify upgrades and Revoking the API token sections.
If you are not using automatic upgrades, you must run the instructions in this section on all secured clusters including the Central cluster.
- To ensure optimal functionality, use the same RHACS version for your secured clusters and the cluster on which Central is installed.

To complete manual upgrades of each secured cluster running Sensor, Collector, and Admission Controller, follow the instructions in this section.

Update readiness probes

If you are upgrading from a version below Red Hat Advanced Cluster Security for Kubernetes 3.65.0, you must run the following additional command to update the readiness probe path. If you are running a higher version than 3.65, skip this step.

Procedure

Update the readiness probe path:

$ oc -n stackrox patch deploy/sensor -p '{"spec":{"template":{"spec":{"containers":[{"name":"sensor","readinessProbe":{"httpGet":{"path":"/ready"}}}]}}}}' (1)

1	If you use Kubernetes, enter `kubectl` instead of `oc`.

Updating OpenShift security context constraints

Depending on the version of Red Hat Advanced Cluster Security for Kubernetes you are upgrading to, you must update certain OpenShift Container Platform security context constraints (SCCs).

Run the commands in this section only if you are using Red Hat Advanced Cluster Security for Kubernetes with OpenShift Container Platform. Otherwise, skip the instructions in this section.

Procedure

Red Hat Advanced Cluster Security for Kubernetes 3.64.0 renames the SCCs. If you are upgrading from a version below Red Hat Advanced Cluster Security for Kubernetes 3.64.0, you must delete and reapply the SCCs, otherwise, skip this step:

Run the following commands to update Central:

$ oc apply -f - <<EOF
kind: SecurityContextConstraints
apiVersion: security.openshift.io/v1
metadata:
  name: stackrox-central
  labels:
    app.kubernetes.io/name: stackrox
  annotations:
    kubernetes.io/description: stackrox-central is the security constraint for the central server
    email: support@stackrox.com
    owner: stackrox
allowHostDirVolumePlugin: false
allowedCapabilities: []
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: false
allowPrivilegedContainer: false
defaultAddCapabilities: []
fsGroup:
  type: MustRunAs
  ranges:
    - max: 4000
      min: 4000
priority: 0
readOnlyRootFilesystem: true
requiredDropCapabilities: []
runAsUser:
  type: MustRunAs
  uid: 4000
seLinuxContext:
  type: MustRunAs
seccompProfiles:
  - '*'
users:
  - system:serviceaccount:stackrox:central
volumes:
  - '*'
EOF

$ oc delete scc central

Run the following commands to update Scanner:

$ oc apply -f - <<EOF
kind: SecurityContextConstraints
apiVersion: security.openshift.io/v1
metadata:
  name: stackrox-scanner
  labels:
    app.kubernetes.io/name: stackrox
  annotations:
    email: support@stackrox.com
    owner: stackrox
    kubernetes.io/description: stackrox-scanner is the security constraint for the Scanner container
priority: 0
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: RunAsAny
seccompProfiles:
  - '*'
users:
  - system:serviceaccount:stackrox:scanner
volumes:
  - '*'
allowHostDirVolumePlugin: false
allowedCapabilities: []
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: false
allowPrivilegedContainer: false
defaultAddCapabilities: []
fsGroup:
  type: RunAsAny
readOnlyRootFilesystem: false
requiredDropCapabilities: []
EOF

$ oc delete scc scanner

Run the following commands on each OpenShift Secured Cluster:

$ oc apply -f - <<EOF
apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: stackrox-admission-control
  labels:
    app.kubernetes.io/name: stackrox
    auto-upgrade.stackrox.io/component: "sensor"
  annotations:
    email: support@stackrox.com
    owner: stackrox
    kubernetes.io/description: stackrox-admission-control is the security constraint for the admission controller
users:
  - system:serviceaccount:stackrox:admission-control
priority: 0
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: RunAsAny
seccompProfiles:
  - '*'
supplementalGroups:
  type: RunAsAny
fsGroup:
  type: RunAsAny
groups: []
readOnlyRootFilesystem: true
allowHostDirVolumePlugin: false
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: false
allowPrivilegedContainer: false
allowedCapabilities: []
defaultAddCapabilities: []
requiredDropCapabilities: []
volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - secret
---
apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: stackrox-collector
  labels:
    app.kubernetes.io/name: stackrox
    auto-upgrade.stackrox.io/component: "sensor"
  annotations:
    email: support@stackrox.com
    owner: stackrox
    kubernetes.io/description: This SCC is based on privileged, hostaccess, and hostmount-anyuid
users:
  - system:serviceaccount:stackrox:collector
allowHostDirVolumePlugin: true
allowPrivilegedContainer: true
fsGroup:
  type: RunAsAny
groups: []
priority: 0
readOnlyRootFilesystem: true
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: RunAsAny
seccompProfiles:
  - '*'
supplementalGroups:
  type: RunAsAny
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: true
allowedCapabilities: []
defaultAddCapabilities: []
requiredDropCapabilities: []
volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - hostPath
  - secret
---
apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: stackrox-sensor
  labels:
    app.kubernetes.io/name: stackrox
    auto-upgrade.stackrox.io/component: "sensor"
  annotations:
    email: support@stackrox.com
    owner: stackrox
    kubernetes.io/description: stackrox-sensor is the security constraint for the sensor
users:
  - system:serviceaccount:stackrox:sensor
  - system:serviceaccount:stackrox:sensor-upgrader
priority: 0
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: RunAsAny
seccompProfiles:
  - '*'
supplementalGroups:
  type: RunAsAny
fsGroup:
  type: RunAsAny
groups: []
readOnlyRootFilesystem: true
allowHostDirVolumePlugin: false
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: true
allowPrivilegedContainer: false
allowedCapabilities: []
defaultAddCapabilities: []
requiredDropCapabilities: []
volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - secret
EOF

$ oc delete scc admission-control collector sensor

Updating other images

You must update the sensor, collector and compliance images on each secured cluster when not using automatic upgrades.

If you are using Kubernetes, use kubectl instead of oc for the commands listed in this procedure.

Procedure

Update the Sensor image:

$ oc -n stackrox set image deploy/sensor sensor=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:3.70.2 (1)

1	If you use Kubernetes, enter `kubectl` instead of `oc`.

Update the Compliance image:

$ oc -n stackrox set image ds/collector compliance=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:3.70.2 (1)

1	If you use Kubernetes, enter `kubectl` instead of `oc`.

Update the Collector image:

$ oc -n stackrox set image ds/collector collector=registry.redhat.io/advanced-cluster-security/rhacs-collector-rhel8:3.70.2 (1)

1	If you use Kubernetes, enter `kubectl` instead of `oc`.

If you are using the collector slim image, run the following command instead:

$ oc -n stackrox set image ds/collector collector=registry.redhat.io/advanced-cluster-security/rhacs-collector-slim-rhel8:{rhacs-version}

Update the admission control image:

$ oc -n stackrox set image deploy/admission-control admission-control=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:3.70.2

Verifying secured cluster upgrade

After you have upgraded secured clusters, verify that the updated pods are working.

Procedure

Check that the new pods have deployed:
```
$ oc get deploy,ds -n stackrox -o wide (1)
```
1 If you use Kubernetes, enter kubectl instead of oc.
```
$ oc get pod -n stackrox --watch (1)
```
1 If you use Kubernetes, enter kubectl instead of oc.

Rolling back Central

You can roll back to a previous version of Central if the upgrade to a new version is unsuccessful.

Rolling back Central normally

You can roll back to a previous version of Central if upgrading Red Hat Advanced Cluster Security for Kubernetes fails.

Prerequisites

You must be using Red Hat Advanced Cluster Security for Kubernetes 3.0.57.0 or higher.
Before you can perform a rollback, you must have free disk space available on your persistent storage. Red Hat Advanced Cluster Security for Kubernetes uses disk space to keep a copy of databases during the upgrade. If the disk space is not enough to store a copy and the upgrade fails, you will not be able to roll back to an earlier version.

Procedure

Run the following command to roll back to a previous version when an upgrade fails (before the Central service starts):
```
$ oc -n stackrox rollout undo deploy/central (1)
```
1 If you use Kubernetes, enter kubectl instead of oc.

Rolling back Central forcefully

You can use forced rollback to roll back to an earlier version of Central (after the Central service starts).

Using forced rollback to switch back to a previous version might result in loss of data and functionality.

Prerequisites

You must be using Red Hat Advanced Cluster Security for Kubernetes 3.0.58.0 or higher.
Before you can perform a rollback, you must have free disk space available on your persistent storage. Red Hat Advanced Cluster Security for Kubernetes uses disk space to keep a copy of databases during the upgrade. If the disk space is not enough to store a copy and the upgrade fails, you will not be able to roll back to an earlier version.

Procedure

Run the following commands to perform a forced rollback:

To forcefully rollback to the previously installed version:
```
$ oc -n stackrox rollout undo deploy/central (1)
```
1 If you use Kubernetes, enter kubectl instead of oc.

To forcefully rollback to a specific version:

Edit Central’s ConfigMap:
```
$ oc -n stackrox edit configmap/central-config (1)
```
1 If you use Kubernetes, enter kubectl instead of oc.

Update the value of the maintenance.forceRollbackVersion key:

data:
  central-config.yaml: |
    maintenance:
      safeMode: false
      compaction:
         enabled: true
         bucketFillFraction: .5
         freeFractionThreshold: 0.75
      forceRollbackVersion: <x.x.x.x> (1)
  ...

1	Specify the version that you want to roll back to.

Update the Central image version:

$ oc -n stackrox \ (1)
  set image deploy/central central=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:<x.x.x.x> (2)

1	If you use Kubernetes, enter `kubectl` instead of `oc`.
2	Specify the version that you want to roll back to. It must be the same version that you specified for the `maintenance.forceRollbackVersion` key in the `central-config` config map.

Verifying upgrades

The updated Sensors and Collectors continue to report the latest data from each secured cluster.

The last time Sensor contacted Central is visible in the RHACS portal.

Procedure

On the RHACS portal, navigate to Platform Configuration → System Health.
Check to ensure that Sensor Upgrade shows clusters up to date with Central.

Revoking the API token

For security reasons, Red Hat recommends that you revoke the API token that you have used to complete Central database backup.

Prerequisites

After the upgrade, you must reload the RHACS portal page and re-accept the certificate to continue using the RHACS portal.

Procedure

On the RHACS portal, navigate to Platform Configuration → Integrations.
Scroll down to the Authentication Tokens category, and click API Token.
Select the checkbox in front of the token name that you want to revoke.
Click Revoke.
On the confirmation dialog box, click Confirm.

Manually upgrading using the roxctl CLI

Backing up the Central database

Upgrading the Central cluster

Upgrading Central

Upgrading the roxctl CLI

Uninstalling the roxctl CLI

Installing the roxctl CLI on Linux

Installing the roxctl CLI on macOS

Installing the roxctl CLI on Windows

Upgrading Scanner

Verifying the Central cluster upgrade

Upgrading all secured clusters

Update readiness probes

Updating OpenShift security context constraints

Updating other images

Verifying secured cluster upgrade

Rolling back Central

Rolling back Central normally

Rolling back Central forcefully

Verifying upgrades

Revoking the API token