Using manual mode with GCP Workload Identity - Managing cloud provider credentials | Authentication and authorization

About manual mode with GCP Workload Identity
Installing an OKD cluster configured for manual mode with GCP Workload Identity
Additional resources

Manual mode with GCP Workload Identity is supported for Google Cloud Platform (GCP).

This credentials strategy is supported for only new OKD clusters and must be configured during installation. You cannot reconfigure an existing cluster that uses a different credentials strategy to use this feature.

About manual mode with GCP Workload Identity

In manual mode with GCP Workload Identity, the individual OKD cluster components can impersonate IAM service accounts using short-term, limited-privilege credentials.

Requests for new and refreshed credentials are automated by using an appropriately configured OpenID Connect (OIDC) identity provider combined with IAM service accounts. Service account tokens that are trusted by GCP are signed by OKD and can be projected into a pod and used for authentication. Tokens are refreshed after one hour.

Detailed authentication flow between GCP and the cluster when using GCP Workload Identity

Figure 1. Workload Identity authentication flow

Using manual mode with GCP Workload Identity changes the content of the GCP credentials that are provided to individual OKD components.

GCP secret format

apiVersion: v1
kind: Secret
metadata:
  namespace: <target_namespace> (1)
  name: <target_secret_name> (2)
data:
  service_account.json: <service_account> (3)

1	The namespace for the component.
2	The name of the component secret.
3	The Base64 encoded service account.

Content of the Base64 encoded service_account.json file using long-lived credentials

{
   "type": "service_account", (1)
   "project_id": "<project_id>",
   "private_key_id": "<private_key_id>",
   "private_key": "<private_key>", (2)
   "client_email": "<client_email_address>",
   "client_id": "<client_id>",
   "auth_uri": "https://accounts.google.com/o/oauth2/auth",
   "token_uri": "https://oauth2.googleapis.com/token",
   "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
   "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/<client_email_address>"
}

1	The credential type is `service_account`.
2	The private RSA key that is used to authenticate to GCP. This key must be kept secure and is not rotated.

Content of the Base64 encoded service_account.json file using GCP Workload Identity

{
   "type": "external_account", (1)
   "audience": "//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/test-pool/providers/test-provider", (2)
   "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
   "token_url": "https://sts.googleapis.com/v1/token",
   "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/<client_email_address>:generateAccessToken", (3)
   "credential_source": {
      "file": "<path_to_token>", (4)
      "format": {
         "type": "text"
      }
   }
}

1	The credential type is `external_account`.
2	The target audience is the GCP Workload Identity provider.
3	The resource URL of the service account that can be impersonated with these credentials.
4	The path to the service account token inside the pod. By convention, this is `/var/run/secrets/openshift/serviceaccount/token` for OKD components.

Installing an OKD cluster configured for manual mode with GCP Workload Identity

To install a cluster that is configured to use the Cloud Credential Operator (CCO) in manual mode with GCP Workload Identity:

Because the cluster is operating in manual mode when using GCP Workload Identity, it is not able to create new credentials for components with the permissions that they require. When upgrading to a different minor version of OKD, there are often new GCP permission requirements. Before upgrading a cluster that is using GCP Workload Identity, the cluster administrator must manually ensure that the GCP permissions are sufficient for existing components and available to any new components.

Additional resources

Configuring the Cloud Credential Operator utility for a cluster update

Configuring the Cloud Credential Operator utility

To create and manage cloud credentials from outside of the cluster when the Cloud Credential Operator (CCO) is operating in manual mode, extract and prepare the CCO utility (ccoctl) binary.

The ccoctl utility is a Linux binary that must run in a Linux environment.

Prerequisites

You have access to an OKD account with cluster administrator access.
You have installed the OpenShift CLI (oc).

Procedure

Obtain the OKD release image by running the following command:

$ RELEASE_IMAGE=$(./openshift-install version | awk '/release image/ {print $3}')

Obtain the CCO container image from the OKD release image by running the following command:
```
$ CCO_IMAGE=$(oc adm release info --image-for='cloud-credential-operator' $RELEASE_IMAGE -a ~/.pull-secret)
```
Ensure that the architecture of the $RELEASE_IMAGE matches the architecture of the environment in which you will use the ccoctl tool.
Extract the ccoctl binary from the CCO container image within the OKD release image by running the following command:
```
$ oc image extract $CCO_IMAGE --file="/usr/bin/ccoctl" -a ~/.pull-secret
```
Change the permissions to make ccoctl executable by running the following command:
```
$ chmod 775 ccoctl
```

Verification

To verify that ccoctl is ready to use, display the help file. Use a relative file name when you run the command, for example:

$ ./ccoctl.rhel9

Example output

OpenShift credentials provisioning tool

Usage:
  ccoctl [command]

Available Commands:
  alibabacloud Manage credentials objects for alibaba cloud
  aws          Manage credentials objects for AWS cloud
  gcp          Manage credentials objects for Google cloud
  help         Help about any command
  ibmcloud     Manage credentials objects for IBM Cloud
  nutanix      Manage credentials objects for Nutanix

Flags:
  -h, --help   help for ccoctl

Use "ccoctl [command] --help" for more information about a command.

Creating GCP resources with the Cloud Credential Operator utility

You can use the ccoctl gcp create-all command to automate the creation of GCP resources.

By default, ccoctl creates objects in the directory in which the commands are run. To create the objects in a different directory, use the --output-dir flag. This procedure uses <path_to_ccoctl_output_dir> to refer to this directory.

Prerequisites

You must have:

Extracted and prepared the ccoctl binary.

Procedure

Set the $RELEASE_IMAGE variable by running the following command:

$ RELEASE_IMAGE=$(./openshift-install version | awk '/release image/ {print $3}')

Extract the list of CredentialsRequest objects from the OKD release image by running the following command:
```
$ oc adm release extract \
  --from=$RELEASE_IMAGE \
  --credentials-requests \
  --cloud=gcp \
  --to=<path_to_directory_with_list_of_credentials_requests>/credrequests (1)
```
1 credrequests is the directory where the list of CredentialsRequest objects is stored. This command creates the directory if it does not exist.

This command can take a few moments to run.

If your cluster uses cluster capabilities to disable one or more optional components, delete the CredentialsRequest custom resources for any disabled components.

Example credrequests directory contents for OKD 4.13 on GCP

0000_26_cloud-controller-manager-operator_16_credentialsrequest-gcp.yaml (1)
0000_30_cluster-api_00_credentials-request.yaml (2)
0000_30_machine-api-operator_00_credentials-request.yaml (3)
0000_50_cloud-credential-operator_05-gcp-ro-credentialsrequest.yaml (4)
0000_50_cluster-image-registry-operator_01-registry-credentials-request-gcs.yaml (5)
0000_50_cluster-ingress-operator_00-ingress-credentials-request.yaml (6)
0000_50_cluster-network-operator_02-cncc-credentials.yaml (7)
0000_50_cluster-storage-operator_03_credentials_request_gcp.yaml (8)

1	The Cloud Controller Manager Operator CR is required.
2	For clusters that use the `TechPreviewNoUpgrade` feature set, the Cluster API Operator CR is required.
3	The Machine API Operator CR is required.
4	The Cloud Credential Operator CR is required.
5	The Image Registry Operator CR is required.
6	The ingress Operator CR is required.
7	The Network Operator CR is required.
8	The Storage Operator CR is an optional component and might be disabled in your cluster.

Use the ccoctl tool to process all CredentialsRequest objects in the credrequests directory:
```
$ ccoctl gcp create-all \
  --name=<name> \
  --region=<gcp_region> \
  --project=<gcp_project_id> \
  --credentials-requests-dir=<path_to_directory_with_list_of_credentials_requests>/credrequests
```
where:
- <name> is the user-defined name for all created GCP resources used for tracking.
- <gcp_region> is the GCP region in which cloud resources will be created.
- <gcp_project_id> is the GCP project ID in which cloud resources will be created.
- <path_to_directory_with_list_of_credentials_requests>/credrequests is the directory containing the files of CredentialsRequest manifests to create GCP service accounts.
If your cluster uses Technology Preview features that are enabled by the TechPreviewNoUpgrade feature set, you must include the --enable-tech-preview parameter.

Verification

To verify that the OKD secrets are created, list the files in the <path_to_ccoctl_output_dir>/manifests directory:

$ ls <path_to_ccoctl_output_dir>/manifests

Example output:

cluster-authentication-02-config.yaml
openshift-cloud-controller-manager-gcp-ccm-cloud-credentials-credentials.yaml
openshift-cloud-credential-operator-cloud-credential-operator-gcp-ro-creds-credentials.yaml
openshift-cloud-network-config-controller-cloud-credentials-credentials.yaml
openshift-cluster-api-capg-manager-bootstrap-credentials-credentials.yaml
openshift-cluster-csi-drivers-gcp-pd-cloud-credentials-credentials.yaml
openshift-image-registry-installer-cloud-credentials-credentials.yaml
openshift-ingress-operator-cloud-credentials-credentials.yaml
openshift-machine-api-gcp-cloud-credentials-credentials.yaml

You can verify that the IAM service accounts are created by querying GCP. For more information, refer to GCP documentation on listing IAM service accounts.

Running the installer

Prerequisites

Configure an account with the cloud platform that hosts your cluster.
Obtain the OKD release image.

Procedure

Change to the directory that contains the installation program and create the install-config.yaml file:
```
$ openshift-install create install-config --dir <installation_directory>
```
where <installation_directory> is the directory in which the installation program creates files.
Edit the install-config.yaml configuration file so that it contains the credentialsMode parameter set to Manual.
Example install-config.yaml configuration file
```
apiVersion: v1
baseDomain: cluster1.example.com
credentialsMode: Manual (1)
compute:
- architecture: amd64
  hyperthreading: Enabled
```
1 This line is added to set the credentialsMode parameter to Manual.
Create the required OKD installation manifests:
```
$ openshift-install create manifests
```
Copy the manifests that ccoctl generated to the manifests directory that the installation program created:
```
$ cp /<path_to_ccoctl_output_dir>/manifests/* ./manifests/
```
Copy the private key that the ccoctl generated in the tls directory to the installation directory:
```
$ cp -a /<path_to_ccoctl_output_dir>/tls .
```
Run the OKD installer:
```
$ ./openshift-install create cluster
```

Verifying the installation

Connect to the OKD cluster.

Verify that the cluster does not have root credentials:

$ oc get secrets -n kube-system gcp-credentials

The output should look similar to:

Error from server (NotFound): secrets "gcp-credentials" not found

Verify that the components are assuming the service accounts that are specified in the secret manifests, instead of using credentials that are created by the CCO:

Example command with the Image Registry Operator

$ oc get secrets -n openshift-image-registry installer-cloud-credentials -o json | jq -r '.data."service_account.json"' | base64 -d

The output should show the role and web identity token that are used by the component and look similar to:

Example output with the Image Registry Operator

{
   "type": "external_account", (1)
   "audience": "//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/test-pool/providers/test-provider",
   "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
   "token_url": "https://sts.googleapis.com/v1/token",
   "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/<client-email-address>:generateAccessToken", (2)
   "credential_source": {
      "file": "/var/run/secrets/openshift/serviceaccount/token",
      "format": {
         "type": "text"
      }
   }
}

1	The credential type is `external_account`.
2	The resource URL of the service account used by the Image Registry Operator.

Additional resources

Preparing to update a cluster with manually maintained credentials