Serving transport encryption | Serving | Red Hat OpenShift Serverless 1.34

Overview of Serving transport encryption
Choice of a certificate issuer
- Compatible certificate issuers
Setting up OpenShift Serverless transport encryption
Trust configuration
- Trust configuration for OpenShift Serverless Serving components and Knative Services
- Trust configuration on your custom workload
Ensuring seamless CA rotation
Verifying transport encryption is enabled

You can enable OpenShift Serverless Serving transport encryption to allow transporting data over secured and encrypted HTTPS connections using TLS.

OpenShift Serverless Serving transport encryption is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Serving Transport Encryption is only available for Kourier as an ingress layer. For Red Hat OpenShift Service Mesh, use the service mesh mTLS capabilities to ensure encrypted traffic.

Overview of Serving transport encryption

There are three parts to OpenShift Serverless Serving transport encryption:

External domain encryption: Transport encryption on the ingress layer external to the cluster. For example, a cluster-external domain, such as myapp-<namespace>.example.com.
Cluster-local encryption: Transport encryption on the ingress layer internal to the cluster. For example, cluster-local domains, such as myapp.<namespace>.svc.cluster.local.
System-internal encryption: Transport encryption between the ingress-gateway, activator, and queue-proxy Knative internal components.

Control-plane traffic, including Kubernetes PreStopHooks, metadata, and metrics, contains no user data and is not encrypted.

The different parts are independent of each other and can be enabled and disabled individually. They can use the same or different Certificate Authorities (CAs) to sign the necessary certificates.

For diagrams illustrating transport encryption, see OpenShift Serverless Serving Transport Encryption.

External domain encryption

The transport encryption for external domains is handled by the ingress layer of the cluster, which is either the OpenShift Container Platform ingress or Red Hat OpenShift Service Mesh.

Cluster-local encryption

Cluster-local encryption enables transport encryption for cluster-local domains. It has the following properties:

The Certificate Common Name (CN) or Subject Alternative Name (SAN) contains the cluster-local domains of a Knative Service, for example myapp.namespace.svc.cluster.local, myapp.namespace.svc, myapp.namespace.
The cluster-local endpoint of the ingress-controller component uses SNI to select the certificates.
To create the certificates, Knative relies on cert-manager, which needs to be installed and configured for the feature to work. For more information, see cert-manager Operator for Red Hat OpenShift.

The caller must trust the CA that signed the cluster-local certificates. This is out of the scope of OpenShift Serverless.

System-internal encryption

System-internal encryption enables transport encryption for the ingress-gateway, activator, and queue-proxy Knative internal components. These components host TLS endpoints when this configuration is used.

The following prerequisites must be satisfied to use this feature:

For OpenShift Serverless to get the certificates, cert-manager must be installed and configured.
Specific SANs are used to verify each connection. Each component must trust the CA that signed the certificates. To satisfy this requirement, OpenShift Serverless system components consume and trust a bundle of CAs. The CA bundle must be provided by the cluster administrator.

Choice of a certificate issuer

Issuers refer to cert-manager issuers and cluster issuers. They represent certificate authorities (CAs) that can generate signed certificates by honoring certificate signing requests. For more information, see cert-manager documentation on issuers.

Depending on the encryption features that you use, OpenShift Serverless requires your certificate issuer to be able to sign certain certificates. To identify your certificate issuer, refer to the list of cert-manager integrations, which contains examples for the following:

A custom CA stored in a Kubernetes secret
HTTP-01 challenges
DNS-01 challenges
Self-signed issuers

Compatible certificate issuers

Not all issuer types work for each Knative Serving encryption feature.

For cluster-local encryption, the issuer must be able to sign certificates for the following cluster-local domain types:
- myapp.<namespace>
- myapp.<namespace>.svc
- myapp.<namespace>.svc.cluster.local
As the CA usually is not within the cluster, verification using the Automated Certificate Management Environment (ACME) protocol (DNS01/HTTP01) is not possible. You can use an issuer that allows creating these certificates, such as the cert-manager CA issuer.
For system-internal encryption, the issuer must be able to sign certificates with the following Subject Alternative Names (SANs):
- kn-routing
- names of format kn-user-<namespace>, where <namespace> is a namespace where Knative Services are created
- data-plane.knative.dev
Knative requires these SANs to verify connections between the internal components. Because this is not possible using the ACME protocol (DNS01/HTTP01), you must configure an issuer that allows creating these certificates, for example, cert-manager CA issuer.

Setting up OpenShift Serverless transport encryption

Prerequisites

You have access to an OpenShift Container Platform account with cluster administrator access.
Install the {oc-first}.
Install the cert-manager Operator for Red Hat OpenShift.
Install the OpenShift Serverless Operator.

If you install the OpenShift Serverless Operator before installing the cert-manager Operator for Red Hat OpenShift, you must restart the controller and activator deployments in the knative-serving namespace. Failure to restart these deployments prevents Knative from creating the necessary cert-manager resources, which results in pending Knative Services and prevents enabling the Knative Serving cert-manager integration.

Configuring a SelfSigned cluster issuer

The following procedure uses a SelfSigned issuer as the root certificate. For information about the implications and limitations of this method, see the SelfSigned cert-manager documentation.

If you manage your own company-specific Private Key Infrastructure (PKI), use the CA issuer. For more information, see cert-manager documentation on CA issuers.

Procedure

Create a SelfSigned ClusterIssuer custom resource (CR):

Example ClusterIssuer CR

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: knative-serving-selfsigned-issuer
spec:
  selfSigned: {}

Apply the ClusterIssuer CR by running the following command:
```
$ oc apply -f <filename>
```

Create a root certificate that refers to the ClusterIssuer CR:

Example root certificate

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: knative-serving-selfsigned-ca
  namespace: cert-manager (1)
spec:
  secretName: knative-serving-ca (2)

  isCA: true
  commonName: selfsigned-ca
  privateKey:
    algorithm: ECDSA
    size: 256

  issuerRef:
    name: knative-serving-selfsigned-issuer
    kind: ClusterIssuer
    group: cert-manager.io

1	The cert-manager Operator for Red Hat OpenShift namespace, `cert-manager` by default.
2	Secret name later used for the `ClusterIssuer` CR for Knative Serving.

Apply the Certificate CR by running the following command:
```
$ oc apply -f <filename>
```

Creating a ClusterIssuer to be used by Serving

To enable the use of certificates by Serving, you must create a cluster issuer.

Procedure

Create the knative-serving-ca-issuer ClusterIssuer for Serving:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: knative-serving-ca-issuer
spec:
  ca:
    secretName: knative-serving-ca (1)

1	Secret name in the cert-manager Operator for Red Hat OpenShift namespace (`cert-manager` by default) containing the certificate that can be used by OpenShift Serverless Serving components for new certificates.

Apply the ClusterIssuer resource by running the following command:
```
$ oc apply -f <filename>
```

Configuring transport encryption

Configuring transport encryption consists of two parts:

Specifying the ClusterIssuer issuer to use:
- clusterLocalIssuerRef: issuer for cluster-local-domain certificates used for ingress.
- systemInternalIssuerRef: issuer for certificates for system-internal-tls certificates used by Knative internal components.
Specifying transport encryption features to use:
- cluster-local-domain-tls: Enables the transport encryption feature for cluster-local domains
- system-internal-tls: Enables the transport encryption feature for OpenShift Serverless Serving internal components.

Procedure

Enable transport encryption in the KnativeServing resource:

apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
spec:
  ...
  config:
    certmanager:
      clusterLocalIssuerRef: |
        kind: ClusterIssuer
        name: knative-serving-ca-issuer (1)
      systemInternalIssuerRef: |
        kind: ClusterIssuer
        name: knative-serving-ca-issuer (2)
    network:
      cluster-local-domain-tls: Enabled (3)
      system-internal-tls: Enabled (4)

1	Define the cluster issuer for each feature. The same or individual cluster issuers can be used.
2	Define the cluster issuer.
3	Enable the `cluster-local-domain-tls` feature. This and other features can be enabled or disabled individually.
4	Enable the `system-internal-tls` feature.

Apply the KnativeServing resource by running the following command:
```
$ oc apply -f <filename>
```

Optionally, change the defaultCertificate value in the Ingress Controller:

apiVersion: operator.openshift.io/v1
kind: IngressController
 ...
spec:
  defaultCertificate:
    name: ca-ingress-cert

If you changed the defaultCertificate value, you must specify the custom certificate name in the openshift-ingress-default-certificate field in the KnativeServing custom resource.

For example, if the custom certificate name is ca-ingress-cert, add the following configuration:
```
...
spec:
  config:
    network:
      system-internal-tls: Enabled
      openshift-ingress-default-certificate: "ca-ingress-cert"
...
```
If you enabled cluster-local-domain-tls or system-internal-tls, restart the Controller component by running the following command.

When either the cluster-local-domain-tls or the system-internal-tls feature is enabled, you must restart the Controller component to enable the Knative Serving cert-manager integration.
```
$ oc rollout restart deploy/controller -n knative-serving
```
If you enabled system-internal-tls, restart the Activator component by running the following command.

When the system-internal-tls feature is activated, you must restart the Activator component to reconfigure its internal web server, as this is not possible during runtime.
```
$ oc rollout restart deploy/activator -n knative-serving
```

Trust configuration

When you enable any of the transport encryption features, you must make sure that all clients calling trust the Certificate Authority (CA) issuing the certificates used for the transport encryption.

There are multiple places where trust must be ensured:

Cluster external client, such as Browser or other application. This is out of the scope of OpenShift Serverless.
OpenShift Serverless system components, such as Activator, Queue-Proxy, and Ingress-Controller.
Cluster internal client, such as a Knative Service or other workload.

Trust configuration for OpenShift Serverless Serving components and Knative Services

To ensure that OpenShift Serverless Serving components and Knative Services trust the CA that issues certificates, you can create a configmap in the following namespaces with the label networking.knative.dev/trust-bundle: true:

knative-serving: for the system components of OpenShift Serverless Serving.
knative-serving-ingress: for the ingress layer of OpenShift Serverless Serving.
istio-system or your own Service Mesh namespace: when the Service Mesh integration is enabled.

Knative reads all data keys in configmaps with this label, regardless of the name. One key can contain one or multiple CAs or intermediate certificates. If they are valid, they are added to the trust store of the Knative components.

This is an example configmap:

apiVersion: v1
data:
  cacerts.pem: | (1)
    -----BEGIN CERTIFICATE-----
    MIIDDTCCAfWgAwIBAgIQMQuip05h7NLQq2TB+j9ZmTANBgkqhkiG9w0BAQsFADAW
    MRQwEgYDVQQDEwtrbmF0aXZlLmRldjAeFw0yMzExMjIwOTAwNDhaFw0yNDAyMjAw
    OTAwNDhaMBYxFDASBgNVBAMTC2tuYXRpdmUuZGV2MIIBIjANBgkqhkiG9w0BAQEF
    AAOCAQ8AMIIBCgKCAQEA3clC3CV7sy0TpUKNuTku6QmP9z8JUCbLCPCLACCUc1zG
    FEokqOva6TakgvAntXLkB3TEsbdCJlNm6qFbbko6DBfX6rEggqZs40x3/T+KH66u
    4PvMT3fzEtaMJDK/KQOBIvVHrKmPkvccUYK/qWY7rgBjVjjLVSJrCn4dKaEZ2JNr
    Fd0KNnaaW/dP9/FvviLqVJvHnTMHH5qyRRr1kUGTrc8njRKwpHcnUdauiDoWRKxo
    Zlyy+MhQfdbbyapX984WsDjCvrDXzkdGgbRNAf+erl6yUm6pHpQhyFFo/zndx6Uq
    QXA7jYvM2M3qCnXmaFowidoLDsDyhwoxD7WT8zur/QIDAQABo1cwVTAOBgNVHQ8B
    Af8EBAMCAgQwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDwYDVR0TAQH/BAUwAwEB/zAd
    BgNVHQ4EFgQU7p4VuECNOcnrP9ulOjc4J37Q2VUwDQYJKoZIhvcNAQELBQADggEB
    AAv26Vnk+ptQrppouF7yHV8fZbfnehpm07HIZkmnXO2vAP+MZJDNrHjy8JAVzXjt
    +OlzqAL0cRQLsUptB0btoJuw23eq8RXgJo05OLOPQ2iGNbAATQh2kLwBWd/CMg+V
    KJ4EIEpF4dmwOohsNR6xa/JoArIYH0D7gh2CwjrdGZr/tq1eMSL+uZcuX5OiE44A
    2oXF9/jsqerOcH7QUMejSnB8N7X0LmUvH4jAesQgr7jo1JTOBs7GF6wb+U76NzFa
    8ms2iAWhoplQ+EHR52wffWb0k6trXspq4O6v/J+nq9Ky3vC36so+G1ZFkMhCdTVJ
    ZmrBsSMWeT2l07qeei2UFRU=
    -----END CERTIFICATE-----
kind: configmap
metadata:
  labels:
    networking.knative.dev/trust-bundle: "true"
  name: knative-bundle (2)
  namespace: knative-serving

1	Serving components trust all keys containing valid PEM-encoded CA bundles.
2	You can use an arbitrary name.

When a CA bundle configmap is created or updated, the Serving components automatically pick them up and add the CAs or intermediate certificates to their CA trust store. The trust store is refreshed for every new HTTP connection.

Trust configuration on your custom workload

As OpenShift Serverless Serving does not control all workloads and managing trust is highly dependent on your runtime and language, custom workloads are out of the scope of OpenShift Serverless. The following are other options for custom workloads:

Adding a CA bundle to a Container image on build-time. Note that this complicates CA rotation, as you must rebuild and redeploy every application when the CA rotates.
Mounting a CA bundle to the filesystem, such as from a Secret or configmap, and making sure your application uses it to verify TLS connections.
Reading a CA bundle from an environment variable and making sure that your application uses it to verify TLS connections.
Accessing a CA bundle from a secret or configmap using Kubernetes API and making sure your application uses it to verify TLS connections.

Ensuring seamless CA rotation

Ensuring seamless CA rotation is essential to avoid service downtime, or to deal with an emergency.

Procedure

Create a new CA certificate.
Add the public key of the new CA certificate to the CA trust bundles as described in the "Trust configuration for OpenShift Serverless Operator Serving components and Knative Services" section. Retain the public key of the existing CA.
Ensure that all clients have consumed the latest set of CA trust bundles. OpenShift Serverless Serving components will automatically reload the changed CA trust bundles.
If you have custom workload consuming trust bundles, reload or restart them accordingly.
Update the knative-serving-ca-issuer cluster issuer to reference the secret containing the new CA certificate.
Either wait for cert-manager to renew all your certificates or enforce it to renew all the certificates. For more information, see the cert-manager documentation.
Once the CA rotation is fully completed, you can remove the public key of the old CA from the trust bundle configmap. Allow enough time for all components to apply the changes.

Additional resources

cert-manager documentation on reissuance triggered by user actions

Verifying transport encryption is enabled

Procedure

Create a Knative Service:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: test-webapp
  namespace: test-namespace
spec:
  template:
    spec:
      containers:
        - image: docker.io/openshift/hello-openshift
          env:
            - name: RESPONSE
              value: "Hello Serverless!"

Apply the Knative Service YAML by running the following command:
```
$ oc apply -f <filename>
```

Examine status of the Knative Service:

Example command

$ oc get ksvc -n test-namespace -o yaml

Example output

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: test-webapp
  namespace: test-namespace
# spec:
# ...
status:
  address:
    # cluster-local-domain:
    url: https://helloworld.test.svc.cluster.local (1)

1	If you have enabled `cluster-local-domain-tls`, you will see the HTTPS URL.

To verify that system-internal-tls is enabled, check the output of Queue-Proxy logs by running the following command:

Example command

$ oc logs your-pod -n test-namespace -c queue-proxy | grep -E 'certDir|Certificate|tls'

If you see lines similar to the following, system-internal-tls is enabled:

{"severity":"INFO","timestamp":"2024-01-03T07:07:32.892810888Z","logger":"queueproxy","caller":"certificate/watcher.go:62","message":"Starting to watch the following directories for changes{certDir 15 0 /var/lib/knative/certs <nil>} {keyDir 15 0 /var/lib/knative/certs <nil>}","commit":"86420f2-dirty","knative.dev/key":"first/helloworld-00001","knative.dev/pod":"helloworld-00001-deployment-75fbb7d488-qgmxx"}
{"severity":"INFO","timestamp":"2024-01-03T07:07:32.89397512Z","logger":"queueproxy","caller":"certificate/watcher.go:131","message":"Certificate and/or key have changed on disk and were reloaded.","commit":"86420f2-dirty","knative.dev/key":"first/helloworld-00001","knative.dev/pod":"helloworld-00001-deployment-75fbb7d488-qgmxx"}
{"severity":"INFO","timestamp":"2024-01-03T07:07:32.894232939Z","logger":"queueproxy","caller":"sharedmain/main.go:282","message":"Starting tls server admin:8022","commit":"86420f2-dirty","knative.dev/key":"first/helloworld-00001","knative.dev/pod":"helloworld-00001-deployment-75fbb7d488-qgmxx"}
{"severity":"INFO","timestamp":"2024-01-03T07:07:32.894268548Z","logger":"queueproxy","caller":"sharedmain/main.go:282","message":"Starting tls server main:8112","commit":"86420f2-dirty","knative.dev/key":"first/helloworld-00001","knative.dev/pod":"helloworld-00001-deployment-75fbb7d488-qgmxx"}