Using AWS Secrets Manager CSI on ROSA with STS | Tutorials

Prerequisites
Deploying the AWS Secrets and Configuration Provider
Creating a Secret and IAM Access Policies
Create an Application to use this secret
Clean up

The AWS Secrets and Configuration Provider (ASCP) provides a way to expose AWS Secrets as Kubernetes storage volumes. With the ASCP, you can store and manage your secrets in Secrets Manager and then retrieve them through your workloads running on Red Hat OpenShift Service on AWS (ROSA).

edit

Prerequisites

Ensure that you have the following resources and tools before starting this process:

A ROSA cluster deployed with STS
Helm 3
aws CLI
oc CLI
jq CLI

Additional environment requirements

Log in to your ROSA cluster by running the following command:
```
$ oc login --token=<your-token> --server=<your-server-url>
```
You can find your login token by accessing your cluster in pull secret from Red Hat OpenShift Cluster Manager.
Validate that your cluster has STS by running the following command:
```
$ oc get authentication.config.openshift.io cluster -o json \
  | jq .spec.serviceAccountIssuer
```
Example output
```
"https://xxxxx.cloudfront.net/xxxxx"
```
If your output is different, do not proceed. See Red Hat documentation on creating an STS cluster before continuing this process.

Set the SecurityContextConstraints permission to allow the CSI driver to run by running the following command:

$ oc new-project csi-secrets-store
$ oc adm policy add-scc-to-user privileged \
    system:serviceaccount:csi-secrets-store:secrets-store-csi-driver
$ oc adm policy add-scc-to-user privileged \
    system:serviceaccount:csi-secrets-store:csi-secrets-store-provider-aws

Create environment variables to use later in this process by running the following command:

$ export REGION=$(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}")
$ export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster \
   -o jsonpath='{.spec.serviceAccountIssuer}' | sed  's|^https://||')
$ export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text`
$ export AWS_PAGER=""

Deploying the AWS Secrets and Configuration Provider

Use Helm to register the secrets store CSI driver by running the following command:

$ helm repo add secrets-store-csi-driver \
    https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts

Update your Helm repositories by running the following command:
```
$ helm repo update
```

Install the secrets store CSI driver by running the following command:

$ helm upgrade --install -n csi-secrets-store \
    csi-secrets-store-driver secrets-store-csi-driver/secrets-store-csi-driver

Deploy the AWS provider by running the following command:

$ oc -n csi-secrets-store apply -f \
    https://raw.githubusercontent.com/rh-mobb/documentation/main/content/misc/secrets-store-csi/aws-provider-installer.yaml

Check that both Daemonsets are running by running the following command:

$ oc -n csi-secrets-store get ds \
    csi-secrets-store-provider-aws \
    csi-secrets-store-driver-secrets-store-csi-driver

Label the Secrets Store CSI Driver to allow use with the restricted pod security profile by running the following command:

$ oc label csidriver.storage.k8s.io/secrets-store.csi.k8s.io security.openshift.io/csi-ephemeral-volume-profile=restricted

Creating a Secret and IAM Access Policies

Create a secret in Secrets Manager by running the following command:

$ SECRET_ARN=$(aws --region "$REGION" secretsmanager create-secret \
    --name MySecret --secret-string \
    '{"username":"shadowman", "password":"hunter2"}' \
    --query ARN --output text); echo $SECRET_ARN

Create an IAM Access Policy document by running the following command:

$ cat << EOF > policy.json
{
   "Version": "2012-10-17",
   "Statement": [{
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": ["$SECRET_ARN"]
      }]
}
EOF

Create an IAM Access Policy by running the following command:

$ POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn \
--output text iam create-policy \
--policy-name openshift-access-to-mysecret-policy \
--policy-document file://policy.json); echo $POLICY_ARN

Create an IAM Role trust policy document by running the following command:

The trust policy is locked down to the default service account of a namespace you create later in this process.

$ cat <<EOF > trust-policy.json
{
   "Version": "2012-10-17",
   "Statement": [
   {
   "Effect": "Allow",
   "Condition": {
     "StringEquals" : {
       "${OIDC_ENDPOINT}:sub": ["system:serviceaccount:my-application:default"]
      }
    },
    "Principal": {
       "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}"
    },
    "Action": "sts:AssumeRoleWithWebIdentity"
    }
    ]
}
EOF

Create an IAM role by running the following command:

$ ROLE_ARN=$(aws iam create-role --role-name openshift-access-to-mysecret \
--assume-role-policy-document file://trust-policy.json \
--query Role.Arn --output text); echo $ROLE_ARN

Attach the role to the policy by running the following command:

$ aws iam attach-role-policy --role-name openshift-access-to-mysecret \
    --policy-arn $POLICY_ARN

Create an Application to use this secret

Create an OpenShift project by running the following command:
```
$ oc new-project my-application
```

Annotate the default service account to use the STS Role by running the following command:

$ oc annotate -n my-application serviceaccount default \
    eks.amazonaws.com/role-arn=$ROLE_ARN

Create a secret provider class to access our secret by running the following command:

$ cat << EOF | oc apply -f -
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-application-aws-secrets
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "MySecret"
        objectType: "secretsmanager"
EOF

Create a deployment by using our secret in the following command:

$ cat << EOF | oc apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: my-application
  labels:
    app: my-application
spec:
  volumes:
  - name: secrets-store-inline
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "my-application-aws-secrets"
  containers:
  - name: my-application-deployment
    image: k8s.gcr.io/e2e-test-images/busybox:1.29
    command:
      - "/bin/sleep"
      - "10000"
    volumeMounts:
    - name: secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
EOF

Verify the pod has the secret mounted by running the following command:
```
$ oc exec -it my-application -- cat /mnt/secrets-store/MySecret
```

Clean up

Delete the application by running the following command:
```
$ oc delete project my-application
```
Delete the secrets store csi driver by running the following command:
```
$ helm delete -n csi-secrets-store csi-secrets-store-driver
```

Delete the security context constraints by running the following command:

$ oc adm policy remove-scc-from-user privileged \
    system:serviceaccount:csi-secrets-store:secrets-store-csi-driver; oc adm policy remove-scc-from-user privileged \
    system:serviceaccount:csi-secrets-store:csi-secrets-store-provider-aws

Delete the AWS provider by running the following command:

$ oc -n csi-secrets-store delete -f \
https://raw.githubusercontent.com/rh-mobb/documentation/main/content/misc/secrets-store-csi/aws-provider-installer.yaml

Delete AWS Roles and Policies by running the following command:

$ aws iam detach-role-policy --role-name openshift-access-to-mysecret \
    --policy-arn $POLICY_ARN; aws iam delete-role --role-name openshift-access-to-mysecret; aws iam delete-policy --policy-arn $POLICY_ARN

Delete the Secrets Manager secret by running the following command:

$ aws secretsmanager --region $REGION delete-secret --secret-id $SECRET_ARN

Tutorial: Using AWS Secrets Manager CSI on ROSA with STS