Using manual mode with STS - Managing cloud provider credentials | Authentication and authorization

Installing an OKD cluster configured for manual mode with STS

Manual mode with STS is available as a Technology Preview for Amazon Web Services (AWS).

Support for AWS Secure Token Service (STS) is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.

This credentials strategy is supported for only new OKD clusters and must be configured during installation. You cannot reconfigure an existing cluster that uses a different credentials strategy to use this feature.

In manual mode with STS, the individual OKD cluster components use AWS Secure Token Service (STS) to assign components IAM roles that provide short-term, limited-privilege security credentials. These credentials are associated with IAM roles that are specific to each component that makes AWS API calls.

Requests for new and refreshed credentials are automated by using an appropriately configured AWS IAM OpenID Connect (OIDC) identity provider, combined with AWS IAM roles. OKD signs service account tokens that are trusted by AWS IAM, and can be projected into a pod and used for authentication. Tokens are refreshed after one hour.

Detailed authentication flow between AWS and the cluster when using AWS STS

Figure 1. STS authentication flow

Using manual mode with STS changes the content of the AWS credentials that are provided to individual OKD components.

AWS secret format using long-lived credentials

apiVersion: v1
kind: Secret
metadata:
  namespace: <target-namespace> (1)
  name: <target-secret-name> (2)
data:
  aws_access_key_id: <base64-encoded-access-key-id>
  aws_secret_access_key: <base64-encoded-secret-access-key>

1	The namespace for the component.
2	The name of the component secret.

AWS secret format with STS

apiVersion: v1
kind: Secret
metadata:
  namespace: <target-namespace> (1)
  name: <target-secret-name> (2)
stringData:
  credentials: |-
    [default]
    role_name: <operator-role-name> (3)
    web_identity_token_file: <path-to-token> (4)

1	The namespace for the component.
2	The name of the component secret.
3	The IAM role for the component.
4	The path to the service account token inside the pod. By convention, this is `/var/run/secrets/openshift/serviceaccount/token` for OKD components.

Installing an OKD cluster configured for manual mode with STS

To install a cluster that is configured to use the CCO in manual mode with STS in OKD version 4.7:

Creating AWS resources manually

To install an OKD cluster that is configured to use the CCO in manual mode with STS, you must first manually create the required AWS resources.

Procedure

Generate a private key to sign the ServiceAccount object:
```
$ openssl genrsa -out sa-signer 4096
```

Generate a ServiceAccount object public key:

$ openssl rsa -in sa-signer -pubout -out sa-signer.pub

Create an S3 bucket to hold the OIDC configuration:

$ aws s3api create-bucket --bucket <oidc_bucket_name> --region <aws_region> --create-bucket-configuration LocationConstraint=<aws_region>

If the value of <aws_region> is us-east-1, do not specify the LocationConstraint parameter.

Retain the S3 bucket URL:

OPENID_BUCKET_URL="https://<oidc_bucket_name>.s3.<aws_region>.amazonaws.com"

Build an OIDC configuration:

Create a file named keys.json that contains the following information:
```
{
    "keys": [
        {
            "use": "sig",
            "kty": "RSA",
            "kid": "<public_signing_key_id>",
            "alg": "RS256",
            "n": "<public_signing_key_modulus>",
            "e": "<public_signing_key_exponent>"
        }
    ]
}
```
Where:
- <public_signing_key_id> is generated from the public key with:
  $ openssl rsa -in sa-signer.pub -pubin --outform DER | openssl dgst -binary -sha256 | openssl base64 | tr '/+' '_-' | tr -d '='
  This command converts the public key to DER format, performs a SHA-256 checksum on the binary representation, encodes the data with base64 encoding, and then changes the base64-encoded output to base64URL encoding.
- <public_signing_key_modulus> is generated from the public key with:
  $ openssl rsa -pubin -in sa-signer.pub -modulus -noout | sed -e 's/Modulus=//' | xxd -r -p | base64 -w0 | tr '/+' '_-' | tr -d '='
  This command prints the modulus of the public key, extracts the hex representation of the modulus, converts the ASCII hex to binary, encodes the data with base64 encoding, and then changes the base64-encoded output to base64URL encoding.
- <public_signing_key_exponent> is generated from the public key with:
  $ printf "%016x" $(openssl rsa -pubin -in sa-signer.pub -noout -text | grep Exponent | awk '{ print $2 }') | awk '{ sub(/(00)+/, "", $1); print $1 }' | xxd -r -p | base64 -w0 | tr '/+' '_-' | tr -d '='
  This command extracts the decimal representation of the public key exponent, prints it as hex with a padded 0 if needed, removes leading 00 pairs, converts the ASCII hex to binary, encodes the data with base64 encoding, and then changes the base64-encoded output to use only characters that can be used in a URL.

Create a file named openid-configuration that contains the following information:

{
	"issuer": "$OPENID_BUCKET_URL",
	"jwks_uri": "${OPENID_BUCKET_URL}/keys.json",
    "response_types_supported": [
        "id_token"
    ],
    "subject_types_supported": [
        "public"
    ],
    "id_token_signing_alg_values_supported": [
        "RS256"
    ],
    "claims_supported": [
        "aud",
        "exp",
        "sub",
        "iat",
        "iss",
        "sub"
    ]
}

Upload the OIDC configuration:

$ aws s3api put-object --bucket <oidc_bucket_name> --key keys.json --body ./keys.json

$ aws s3api put-object --bucket <oidc_bucket_name> --key '.well-known/openid-configuration' --body ./openid-configuration

Where <oidc_bucket_name> is the S3 bucket that was created to hold the OIDC configuration.

Allow the AWS IAM OpenID Connect (OIDC) identity provider to read these files:

$ aws s3api put-object-acl --bucket <oidc_bucket_name> --key keys.json --acl public-read

$ aws s3api put-object-acl --bucket <oidc_bucket_name> --key '.well-known/openid-configuration' --acl public-read

Create an AWS IAM OIDC identity provider:

Get the certificate chain from the server that hosts the OIDC configuration:

$ echo | openssl s_client -servername $<oidc_bucket_name>.s3.$<aws_region>.amazonaws.com -connect $<oidc_bucket_name>.s3.$<aws_region>.amazonaws.com:443 -showcerts 2>/dev/null | awk '/BEGIN/,/END/{ if(/BEGIN/){a++}; out="cert"a".pem"; print >out}'

Calculate the fingerprint for the certificate at the root of the chain:
```
$ export BUCKET_FINGERPRINT=$(openssl x509 -in cert<number>.pem -fingerprint -noout | sed -e 's/.*Fingerprint=//' -e 's/://g')
```
Where <number> is the highest number in the files that were saved. For example, if 2 is the highest number in the files that were saved, use cert2.pem.

Create the identity provider:

$ aws iam create-open-id-connect-provider --url $OPENID_BUCKET_URL --thumbprint-list $BUCKET_FINGERPRINT --client-id-list openshift sts.amazonaws.com

Retain the returned ARN of the newly created identity provider. This ARN is later referred to as <aws_iam_openid_arn>.

Generate IAM roles:

Locate all CredentialsRequest CRs in this release image that target the cloud you are deploying on:
```
$ oc adm release extract quay.io/openshift-release-dev/ocp-release:4.<y>.<z>-x86_64 --credentials-requests --cloud=aws
```
Where <y> and <z> are the numbers corresponding to the version of OKD you are installing.

For each CredentialsRequest CR, create an IAM role of type Web identity using the previously created IAM Identity Provider that grants the necessary permissions and establishes a trust relationship that trusts the identity provider previously created.

For example, for the openshift-machine-api-operator CredentialsRequest CR in 0000_30_machine-api-operator_00_credentials-request.yaml, create an IAM role that allows an identity from the created OIDC provider created for the cluster, similar to the following:

{
    "Role": {
        "Path": "/",
        "RoleName": "openshift-machine-api-aws-cloud-credentials",
        "RoleId": "ARSOMEROLEID",
        "Arn": "arn:aws:iam::123456789012:role/openshift-machine-api-aws-cloud-credentials",
        "CreateDate": "2021-01-06T15:54:13Z",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Federated": "<aws_iam_openid_arn>"
                    },
                    "Action": "sts:AssumeRoleWithWebIdentity",
                    "Condition": {
                        "StringEquals": {
                            "<oidc_bucket_name>.s3.<aws_region>.amazonaws.com/$BUCKET_NAME:aud": "openshift"
                        }
                    }
                }
            ]
        },
        "Description": "OpenShift role for openshift-machine-api/aws-cloud-credentials",
        "MaxSessionDuration": 3600,
        "RoleLastUsed": {
            "LastUsedDate": "2021-02-03T02:51:24Z",
            "Region": "<aws_region>"
        }
    }
}

Where <aws_iam_openid_arn> is the returned ARN of the newly created identity provider.

To further restrict the role such that only specific cluster ServiceAccount objects can assume the role, modify the trust relationship of each role by updating the .Role.AssumeRolePolicyDocument.Statement[].Condition field to the specific ServiceAccount objects for each component.

Modify the trust relationship of the cluster-image-registry-operator role to have the following condition:

"Condition": {
  "StringEquals": {
    "<oidc_bucket_name>.s3.<aws_region>.amazonaws.com:sub": [
      "system:serviceaccount:openshift-image-registry:registry",
      "system:serviceaccount:openshift-image-registry:cluster-image-registry-operator"
    ]
  }
}

Modify the trust relationship of the openshift-ingress-operator to have the following condition:

"Condition": {
  "StringEquals": {
    "<oidc_bucket_name>.s3.<aws_region>.amazonaws.com:sub": [
      "system:serviceaccount:openshift-ingress-operator:ingress-operator"
    ]
  }
}

Modify the trust relationship of the openshift-cluster-csi-drivers to have the following condition:

"Condition": {
  "StringEquals": {
    "<oidc_bucket_name>.s3.<aws_region>.amazonaws.com:sub": [
      "system:serviceaccount:openshift-cluster-csi-drivers:aws-ebs-csi-driver-operator",
      "system:serviceaccount:openshift-cluster-csi-drivers:aws-ebs-csi-driver-controller-sa"
    ]
  }
}

Modify the trust relationship of the openshift-machine-api to have the following condition:

"Condition": {
  "StringEquals": {
    "<oidc_bucket_name>.s3.<aws_region>.amazonaws.com:sub": [
      "system:serviceaccount:openshift-machine-api:machine-api-controllers"
    ]
  }
}

For each IAM role, attach an IAM policy to the role that reflects the required permissions from the corresponding CredentialsRequest objects.

For example, for openshift-machine-api, attach an IAM policy similar to the following:

{
    "RoleName": "openshift-machine-api-aws-cloud-credentials",
    "PolicyName": "openshift-machine-api-aws-cloud-credentials",
    "PolicyDocument": {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "ec2:CreateTags",
                    "ec2:DescribeAvailabilityZones",
                    "ec2:DescribeDhcpOptions",
                    "ec2:DescribeImages",
                    "ec2:DescribeInstances",
                    "ec2:DescribeSecurityGroups",
                    "ec2:DescribeSubnets",
                    "ec2:DescribeVpcs",
                    "ec2:RunInstances",
                    "ec2:TerminateInstances",
                    "elasticloadbalancing:DescribeLoadBalancers",
                    "elasticloadbalancing:DescribeTargetGroups",
                    "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
                    "elasticloadbalancing:RegisterTargets",
                    "iam:PassRole",
                    "iam:CreateServiceLinkedRole"
                ],
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "kms:Decrypt",
                    "kms:Encrypt",
                    "kms:GenerateDataKey",
                    "kms:GenerateDataKeyWithoutPlainText",
                    "kms:DescribeKey"
                ],
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "kms:RevokeGrant",
                    "kms:CreateGrant",
                    "kms:ListGrants"
                ],
                "Resource": "*",
                "Condition": {
                    "Bool": {
                        "kms:GrantIsForAWSResource": true
                    }
                }
            }
        ]
    }
}

Prepare to run the OKD installer:

Create the install-config.yaml file:

$ ./openshift-install create install-config

Configure the cluster to install with the CCO in manual mode:
```
$ echo "credentialsMode: Manual" >> install-config.yaml
```
Create install manifests:
```
$ ./openshift-install create manifests
```
Create a tls directory, and copy the private key generated previously there:

The target file name must be ./tls/bound-service-account-signing-key.key.
```
$ mkdir tls ; cp <path_to_service_account_signer> ./tls/bound-service-account-signing-key.key
```

Create a custom Authentication CR with the file name cluster-authentication-02-config.yaml:

$ cat << EOF > manifests/cluster-authentication-02-config.yaml
apiVersion: config.openshift.io/v1
kind: Authentication
metadata:
  name: cluster
spec:
  serviceAccountIssuer: $OPENID_BUCKET_URL
EOF

For each CredentialsRequest CR that is extracted from the release image, create a secret with the target namespace and target name that is indicated in each CredentialsRequest, substituting the AWS IAM role ARN created previously for each component:

Example secret manifest for openshift-machine-api:

$ cat manifests/openshift-machine-api-aws-cloud-credentials-credentials.yaml
apiVersion: v1
stringData:
  credentials: |-
    [default]
    role_arn = arn:aws:iam::123456789012:role/openshift-machine-api-aws-cloud-credentials
    web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
kind: Secret
metadata:
  name: aws-cloud-credentials
  namespace: openshift-machine-api
type: Opaque

Running the installer

Run the OKD installer:
```
$ ./openshift-install create cluster
```

Verifying the installation

Connect to the OKD cluster.

Verify that the cluster does not have root credentials:

$ oc get secrets -n kube-system aws-creds

The output should look similar to:

Error from server (NotFound): secrets "aws-creds" not found

Verify that the components are assuming the IAM roles that are specified in the secret manifests, instead of using credentials that are created by the CCO:
Example command with the Image Registry Operator
```
$ oc get secrets -n openshift-image-registry installer-cloud-credentials -o json | jq -r .data.credentials | base64 --decode
```
The output should show the role and web identity token that are used by the component and look similar to:
Example output with the Image Registry Operator
```
[default]
role_arn = arn:aws:iam::123456789:role/openshift-image-registry-installer-cloud-credentials
web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
```