Dynamically issuing certificates using the cert-manager Operator on ROSA | Tutorials

Prerequisites
Setting up your environment
Preparing your AWS account
Installing the cert-manager Operator
Creating a custom domain Ingress Controller
Configuring dynamic certificates for custom domain routes
Deploying a sample application
Troubleshooting dynamic certificate provisioning

While wildcard certificates provide simplicity by securing all first-level subdomains of a given domain with a single certificate, other use cases can require the use of individual certificates per domain.

Learn how to use the cert-manager Operator for Red Hat OpenShift and Let’s Encrypt to dynamically issue certificates for routes created using a custom domain.

edit

Prerequisites

A ROSA cluster (HCP or Classic)
A user account with cluster-admin privileges
The OpenShift CLI (oc)
The Amazon Web Services (AWS) CLI (aws)
A unique domain, such as *.apps.example.com
An Amazon Route 53 public hosted zone for the above domain

Setting up your environment

Configure the following environment variables:

$ export DOMAIN=apps.example.com (1)
$ export EMAIL=email@example.com (2)
$ export AWS_PAGER=""
$ export CLUSTER=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}"  | sed 's/-[a-z0-9]\{5\}$//')
$ export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer | sed  's|^https://||')
$ export REGION=$(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}")
$ export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
$ export SCRATCH="/tmp/${CLUSTER}/dynamic-certs"
$ mkdir -p ${SCRATCH}

1	Replace with the custom domain you want to use for the `IngressController`.
2	Replace with the e-mail you want Let’s Encrypt to use to send notifications about your certificates.

Ensure all fields output correctly before moving to the next section:

$ echo "Cluster: ${CLUSTER}, Region: ${REGION}, OIDC Endpoint: ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"

The "Cluster" output from the previous command may be the name of your cluster, the internal ID of your cluster, or the cluster’s domain prefix. If you prefer to use another identifier, you can manually set this value by running the following command:

$ export CLUSTER=my-custom-value

Preparing your AWS account

When cert-manager requests a certificate from Let’s Encrypt (or another ACME certificate issuer), Let’s Encrypt servers validate that you control the domain name in that certificate using challenges. For this tutorial, you are using a dns-01 challenge that proves that you control the dns for your domain name by putting a specific value in a TXT record under that domain name. This is all done automatically by cert-manager. To allow cert-manager permission to modify the Amazon Route 53 public hosted zone for your domain, you need to create an Identity Access Management (IAM) role with specific policy permissions and a trust relationship to allow access to the pod.

The public hosted zone that is used in this tutorial is in the same AWS account as the ROSA cluster. If your public hosted zone is in a different account, a few additional steps for Cross Account Access are required.

Retrieve the Amazon Route 53 public hosted zone ID:

This command looks for a public hosted zone that matches the custom domain you specified earlier as the DOMAIN environment variable. You can manually specify the Amazon Route 53 public hosted zone by running export ZONE_ID=<zone_ID>, replacing <zone_ID> with your specific Amazon Route 53 public hosted zone ID.

$ export ZONE_ID=$(aws route53 list-hosted-zones-by-name --output json \
  --dns-name "${DOMAIN}." --query 'HostedZones[0]'.Id --out text | sed 's/\/hostedzone\///')

Create an AWS IAM policy document for the cert-manager Operator that provides the ability to update only the specified public hosted zone:

$ cat <<EOF > "${SCRATCH}/cert-manager-policy.json"
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "route53:GetChange",
      "Resource": "arn:aws:route53:::change/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:ChangeResourceRecordSets",
        "route53:ListResourceRecordSets"
      ],
      "Resource": "arn:aws:route53:::hostedzone/${ZONE_ID}"
    },
    {
      "Effect": "Allow",
      "Action": "route53:ListHostedZonesByName",
      "Resource": "*"
    }
  ]
}
EOF

Create the IAM policy using the file you created in the previous step:

$ POLICY_ARN=$(aws iam create-policy --policy-name "${CLUSTER}-cert-manager-policy" \
  --policy-document file://${SCRATCH}/cert-manager-policy.json \
  --query 'Policy.Arn' --output text)

Create an AWS IAM trust policy for the cert-manager Operator:

$ cat <<EOF > "${SCRATCH}/trust-policy.json"
{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Condition": {
   "StringEquals" : {
     "${OIDC_ENDPOINT}:sub": "system:serviceaccount:cert-manager:cert-manager"
   }
 },
 "Principal": {
   "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}"
 },
 "Action": "sts:AssumeRoleWithWebIdentity"
 }
 ]
}
EOF

Create an IAM role for the cert-manager Operator using the trust policy you created in the previous step:

$ ROLE_ARN=$(aws iam create-role --role-name "${CLUSTER}-cert-manager-operator" \
   --assume-role-policy-document "file://${SCRATCH}/trust-policy.json" \
   --query Role.Arn --output text)

Attach the permissions policy to the role:

$ aws iam attach-role-policy --role-name "${CLUSTER}-cert-manager-operator" \
  --policy-arn ${POLICY_ARN}

Installing the cert-manager Operator

Create a project to install the cert-manager Operator into:

$ oc new-project cert-manager-operator

Do not attempt to use more than one cert-manager Operator in your cluster. If you have a community cert-manager Operator installed in your cluster, you must uninstall it before installing the cert-manager Operator for Red Hat OpenShift.

Install the cert-manager Operator for Red Hat OpenShift:

$ cat << EOF | oc apply -f -
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: openshift-cert-manager-operator-group
  namespace: cert-manager-operator
spec:
  targetNamespaces:
  - cert-manager-operator
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: openshift-cert-manager-operator
  namespace: cert-manager-operator
spec:
  channel: stable-v1
  installPlanApproval: Automatic
  name: openshift-cert-manager-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF

It takes a few minutes for this Operator to install and complete its set up.

Verify that the cert-manager Operator is running:

$ oc -n cert-manager-operator get pods

Example output

NAME                                                        READY   STATUS    RESTARTS   AGE
cert-manager-operator-controller-manager-84b8799db5-gv8mx   2/2     Running   0          12s

Annotate the service account used by the cert-manager pods with the AWS IAM role you created earlier:
```
$ oc -n cert-manager annotate serviceaccount cert-manager eks.amazonaws.com/role-arn=${ROLE_ARN}
```
Restart the existing cert-manager controller pod by running the following command:
```
$ oc -n cert-manager delete pods -l app.kubernetes.io/name=cert-manager
```

Patch the Operator’s configuration to use external nameservers to prevent dns-01 challenge resolution issues:

$ oc patch certmanager.operator.openshift.io/cluster --type merge \
  -p '{"spec":{"controllerConfig":{"overrideArgs":["--dns01-recursive-nameservers-only","--dns01-recursive-nameservers=1.1.1.1:53"]}}}'

Create a ClusterIssuer resource to use Let’s Encrypt by running the following command:

$ cat << EOF | oc apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ${EMAIL}
    # This key doesn't exist, cert-manager creates it
    privateKeySecretRef:
      name: prod-letsencrypt-issuer-account-key
    solvers:
    - dns01:
        route53:
         hostedZoneID: ${ZONE_ID}
         region: ${REGION}
         secretAccessKeySecretRef:
           name: ''
EOF

Verify the ClusterIssuer resource is ready:

$ oc get clusterissuer.cert-manager.io/letsencrypt-production

Example output

NAME                     READY   AGE
letsencrypt-production   True    47s

Creating a custom domain Ingress Controller

Create and configure a certificate resource to provision a certificate for the custom domain Ingress Controller:

The following example uses a single domain certificate. SAN and wildcard certificates are also supported.

$ cat << EOF | oc apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: custom-domain-ingress-cert
  namespace: openshift-ingress
spec:
  secretName: custom-domain-ingress-cert-tls
  issuerRef:
     name: letsencrypt-production
     kind: ClusterIssuer
  commonName: "${DOMAIN}"
  dnsNames:
  - "${DOMAIN}"
EOF

Verify the certificate has been issued:

It takes a few minutes for this certificate to be issued by Let’s Encrypt. If it takes longer than 5 minutes, run oc -n openshift-ingress describe certificate.cert-manager.io/custom-domain-ingress-cert to see any issues reported by cert-manager.

$ oc -n openshift-ingress get certificate.cert-manager.io/custom-domain-ingress-cert

Example output

NAME                         READY   SECRET                           AGE
custom-domain-ingress-cert   True    custom-domain-ingress-cert-tls   9m53s

Create a new IngressController resource:

$ cat << EOF | oc apply -f -
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: custom-domain-ingress
  namespace: openshift-ingress-operator
spec:
  domain: ${DOMAIN}
  defaultCertificate:
    name: custom-domain-ingress-cert-tls
  endpointPublishingStrategy:
    loadBalancer:
      dnsManagementPolicy: Unmanaged
      providerParameters:
        aws:
          type: NLB
        type: AWS
      scope: External
    type: LoadBalancerService
EOF

This IngressController example will create an internet accessible Network Load Balancer (NLB) in your AWS account. To provision an internal NLB instead, set the .spec.endpointPublishingStrategy.loadBalancer.scope parameter to Internal before creating the IngressController resource.

Verify that your custom domain IngressController has successfully created an external load balancer:

$ oc -n openshift-ingress get service/router-custom-domain-ingress

Example output

NAME                           TYPE           CLUSTER-IP      EXTERNAL-IP                                                                     PORT(S)                      AGE
router-custom-domain-ingress   LoadBalancer   172.30.174.34   a309962c3bd6e42c08cadb9202eca683-1f5bbb64a1f1ec65.elb.us-east-1.amazonaws.com   80:31342/TCP,443:31821/TCP   7m28s

Prepare a document with the necessary dns changes to enable dns resolution for your custom domain Ingress Controller:

$ INGRESS=$(oc -n openshift-ingress get service/router-custom-domain-ingress -ojsonpath="{.status.loadBalancer.ingress[0].hostname}")
$ cat << EOF > "${SCRATCH}/create-cname.json"
{
  "Comment":"Add CNAME to custom domain endpoint",
  "Changes":[{
      "Action":"CREATE",
      "ResourceRecordSet":{
        "Name": "*.${DOMAIN}",
      "Type":"CNAME",
      "TTL":30,
      "ResourceRecords":[{
        "Value": "${INGRESS}"
      }]
    }
  }]
}
EOF

Submit your changes to Amazon Route 53 for propagation:

$ aws route53 change-resource-record-sets \
  --hosted-zone-id ${ZONE_ID} \
  --change-batch file://${SCRATCH}/create-cname.json

While the wildcard CNAME record avoids the need to create a new record for every new application you deploy using the custom domain Ingress Controller, the certificate that each of these applications use is not a wildcard certificate.

Configuring dynamic certificates for custom domain routes

Now you can expose cluster applications on any first-level subdomains of the specified domain, but the connection will not be secured with a TLS certificate that matches the domain of the application. To ensure these cluster applications have valid certificates for each domain name, configure cert-manager to dynamically issue a certificate to every new route created under this domain.

Create the necessary OpenShift resources cert-manager requires to manage certificates for OpenShift routes.

This step creates a new deployment (and therefore a pod) that specifically monitors annotated routes in the cluster. If the issuer-kind and issuer-name annotations are found in a new route, it requests the Issuer (ClusterIssuer in this case) for a new certificate that is unique to this route and which will honor the hostname that was specified while creating the route.

If the cluster does not have access to GitHub, you can save the raw contents locally and run oc apply -f localfilename.yaml -n cert-manager.
```
$ oc -n cert-manager apply -f https://github.com/cert-manager/openshift-routes/releases/latest/download/cert-manager-openshift-routes.yaml
```
The following additional OpenShift resources are also created in this step:
- ClusterRole - grants permissions to watch and update the routes across the cluster
- ServiceAccount - uses permissions to run the newly created pod
- ClusterRoleBinding - binds these two resources

Ensure that the new cert-manager-openshift-routes pod is running successfully:

$ oc -n cert-manager get pods

Example result

NAME                                             READY   STATUS    RESTARTS   AGE
cert-manager-866d8f788c-9kspc                    1/1     Running   0          4h21m
cert-manager-cainjector-6885c585bd-znws8         1/1     Running   0          4h41m
cert-manager-openshift-routes-75b6bb44cd-f8kd5   1/1     Running   0          6s
cert-manager-webhook-8498785dd9-bvfdf            1/1     Running   0          4h41m

Deploying a sample application

Now that dynamic certificates are configured, you can deploy a sample application to confirm that certificates are provisioned and trusted when you expose a new route.

Create a new project for your sample application:
```
$ oc new-project hello-world
```

Deploy a hello world application:

$ oc -n hello-world new-app --image=docker.io/openshift/hello-openshift

Create a route to expose the application from outside the cluster:

$ oc -n hello-world create route edge --service=hello-openshift hello-openshift-tls --hostname hello.${DOMAIN}

Verify the certificate for the route is untrusted:

$ curl -I https://hello.${DOMAIN}

Example output

curl: (60) SSL: no alternative certificate subject name matches target host name 'hello.example.com'
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Annotate the route to trigger cert-manager to provision a certificate for the custom domain:
```
$ oc -n hello-world annotate route hello-openshift-tls cert-manager.io/issuer-kind=ClusterIssuer cert-manager.io/issuer-name=letsencrypt-production
```
It takes 2-3 minutes for the certificate to be created. The renewal of the certificate will automatically be managed by the cert-manager Operator as it approaches expiration.

Verify the certificate for the route is now trusted:

$ curl -I https://hello.${DOMAIN}

Example output

HTTP/2 200
date: Thu, 05 Oct 2023 23:45:33 GMT
content-length: 17
content-type: text/plain; charset=utf-8
set-cookie: 52e4465485b6fb4f8a1b1bed128d0f3b=68676068bb32d24f0f558f094ed8e4d7; path=/; HttpOnly; Secure; SameSite=None
cache-control: private

Troubleshooting dynamic certificate provisioning

The validation process usually takes 2-3 minutes to complete while creating certificates.

If annotating your route does not trigger certificate creation during the certificate create step, run oc describe against each of the certificate,certificaterequest,order, and challenge resources to view the events or reasons that can help identify the cause of the issue.

$ oc get certificate,certificaterequest,order,challenge

For troubleshooting, you can refer to this helpful guide in debugging certificates.

You can also use the cmctl CLI tool for various certificate management activities, such as checking the status of certificates and testing renewals.

Tutorial: Dynamically issuing certificates using the cert-manager Operator on ROSA

Prerequisites

Setting up your environment

Preparing your AWS account

Installing the cert-manager Operator

Creating a custom domain Ingress Controller

Configuring dynamic certificates for custom domain routes

Deploying a sample application

Troubleshooting dynamic certificate provisioning