apiVersion: v1
kind: ConfigMap
metadata:
name: user-ca-bundle
namespace: openshift-config
data:
ca-bundle.crt: |
-----BEGIN CERTIFICATE-----
Custom CA certificate bundle.
-----END CERTIFICATE-----
OpenShift Container Platform monitors certificates for proper validity, for the cluster certificates it issues and manages. The OpenShift Container Platform alerting framework has rules to help identify when a certificate issue is about to occur. These rules consist of the following checks:
API server client certificate expiration is less than five minutes.
The API server is accessible by clients external to the cluster at
api.<cluster_name>.<base_domain>
. You might want clients to access the API
server at a different host name or without the need to distribute the
cluster-managed certificate authority (CA) certificates to the clients. The
administrator must set a custom default certificate to be used by the API server
when serving content.
The user-provided certificates must be provided in a kubernetes.io/tls
type
Secret
in the openshift-config
namespace. Update the API server cluster
configuration, the apiserver/cluster
resource, to enable the use of the
user-provided certificate.
user-provided certificates are managed by the user.
user-provided certificates are managed by the user.
Update the secret containing the user-managed certificate as needed.
Proxy certificates allow users to specify one or more custom certificate authority (CA) certificates used by platform components when making egress connections.
The trustedCA
field of the Proxy object is a reference to a ConfigMap that
contains a user-provided trusted certificate authority (CA) bundle. This bundle
is merged with the Red Hat Enterprise Linux CoreOS (RHCOS) trust bundle and injected into the trust
store of platform components that make egress HTTPS calls. For example,
image-registry-operator
calls an external image registry to download images.
If trustedCA
is not specified, only the RHCOS trust bundle is used for proxied
HTTPS connections. Provide custom CA certificates to the RHCOS trust bundle if
you want to use your own certificate infrastructure.
The trustedCA
field should only be consumed by a proxy validator. The
validator is responsible for reading the certificate bundle from required key
ca-bundle.crt
and copying it to a ConfigMap named trusted-ca-bundle
in the
openshift-config-managed
namespace. The namespace for the ConfigMap referenced
by trustedCA
is openshift-config
:
apiVersion: v1
kind: ConfigMap
metadata:
name: user-ca-bundle
namespace: openshift-config
data:
ca-bundle.crt: |
-----BEGIN CERTIFICATE-----
Custom CA certificate bundle.
-----END CERTIFICATE-----
The additionalTrustBundle
value of the installer configuration is used to
specify any proxy-trusted CA certificates during installation. For example:
$ cat install-config.yaml . . . proxy: httpProxy: http://<HTTP_PROXY> httpsProxy: https://<HTTPS_PROXY> additionalTrustBundle: | -----BEGIN CERTIFICATE----- <MY_HTTPS_PROXY_TRUSTED_CA_CERT> -----END CERTIFICATE----- . . .
The user-provided trust bundle is represented as a ConfigMap. The ConfigMap is
mounted into the file system of platform components that make egress HTTPS
calls. Typically, Operators mount the ConfigMap to
/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
, but this is not required by
the proxy. A proxy can modify or inspect the HTTPS connection. In either case,
the proxy must generate and sign a new certificate for the connection.
Complete proxy support means connecting to the specified proxy and trusting any signatures it has generated. Therefore, it is necessary to let the user specify a trusted root, such that any certificate chain connected to that trusted root is also trusted.
If using the RHCOS trust bundle, place CA certificates in
/etc/pki/ca-trust/source/anchors
.
See Using shared system certificates in the Red Hat Enterprise Linux documentation for more information.
The user sets the expiration term of the user-provided trust bundle.
The default expiration term is defined by the CA certificate itself. It is up to the CA administrator to configure this for the certificate before it can be used by OpenShift Container Platform or RHCOS.
Red Hat does not monitor for when CAs expire. However, due to the long life of CAs, this is generally not an issue. However, you might need to periodically update the trust bundle. |
By default, all platform components that make egress HTTPS calls will use the
RHCOS trust bundle. If trustedCA
is defined, it will also be used.
Any service that is running on the RHCOS node is able to use the trust bundle of the node.
These certificates are managed by the system and not the user.
Updating the user-provided trust bundle consists of either:
updating the PEM-encoded certificates in the ConfigMap referenced by
trustedCA,
or
creating a ConfigMap in the namespace openshift-config
that contains the new
trust bundle and updating trustedCA
to reference the name of the new
ConfigMap.
The mechanism for writing CA certificates to the RHCOS trust bundle is exactly
the same as writing any other file to RHCOS, which is done through the use of
MachineConfigs. When the Machine Config Operator (MCO) applies the new
MachineConfig that contains the new CA certificates, the node is rebooted.
During the next boot, the service coreos-update-ca-trust.service
runs on the
RHCOS nodes, which automatically update the trust bundle with the new CA
certificates. For example:
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: worker
name: 50-examplecorp-ca-cert
spec:
config:
ignition:
version: 2.2.0
storage:
files:
- contents:
source: data:text/plain;charset=utf-8;base64,LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVORENDQXh5Z0F3SUJBZ0lKQU51bkkwRDY2MmNuTUEwR0NTcUdTSWIzRFFFQkN3VUFNSUdsTVFzd0NRWUQKV1FRR0V3SlZVekVYTUJVR0ExVUVDQXdPVG05eWRHZ2dRMkZ5YjJ4cGJtRXhFREFPQmdOVkJBY01CMUpoYkdWcApBMmd4RmpBVUJnTlZCQW9NRFZKbFpDQklZWFFzSUVsdVl5NHhFekFSQmdOVkJBc01DbEpsWkNCSVlYUWdTVlF4Ckh6QVpCZ05WQkFNTUVsSmxaQ0JJWVhRZ1NWUWdVbTl2ZENCRFFURWhNQjhHQ1NxR1NJYjNEUUVKQVJZU2FXNW0KWGpDQnBURUxNQWtHQTFVRUJoTUNWVk14RnpBVkJnTlZCQWdNRGs1dmNuUm9JRU5oY205c2FXNWhNUkF3RGdZRApXUVFIREFkU1lXeGxhV2RvTVJZd0ZBWURWUVFLREExU1pXUWdTR0YwTENCSmJtTXVNUk13RVFZRFZRUUxEQXBTCkFXUWdTR0YwSUVsVU1Sc3dHUVlEVlFRRERCSlNaV1FnU0dGMElFbFVJRkp2YjNRZ1EwRXhJVEFmQmdrcWhraUcKMHcwQkNRRVdFbWx1Wm05elpXTkFjbVZrYUdGMExtTnZiVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUApCRENDQVFvQ2dnRUJBTFF0OU9KUWg2R0M1TFQxZzgwcU5oMHU1MEJRNHNaL3laOGFFVHh0KzVsblBWWDZNSEt6CmQvaTdsRHFUZlRjZkxMMm55VUJkMmZRRGsxQjBmeHJza2hHSUlaM2lmUDFQczRsdFRrdjhoUlNvYjNWdE5xU28KSHhrS2Z2RDJQS2pUUHhEUFdZeXJ1eTlpckxaaW9NZmZpM2kvZ0N1dDBaV3RBeU8zTVZINXFXRi9lbkt3Z1BFUwpZOXBvK1RkQ3ZSQi9SVU9iQmFNNzYxRWNyTFNNMUdxSE51ZVNmcW5obzNBakxRNmRCblBXbG82MzhabTFWZWJLCkNFTHloa0xXTVNGa0t3RG1uZTBqUTAyWTRnMDc1dkNLdkNzQ0F3RUFBYU5qTUdFd0hRWURWUjBPQkJZRUZIN1IKNXlDK1VlaElJUGV1TDhacXczUHpiZ2NaTUI4R0ExVWRJd1FZTUJhQUZIN1I0eUMrVWVoSUlQZXVMOFpxdzNQegpjZ2NaTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3RGdZRFZSMFBBUUgvQkFRREFnR0dNQTBHQ1NxR1NJYjNEUUVCCkR3VUFBNElCQVFCRE52RDJWbTlzQTVBOUFsT0pSOCtlbjVYejloWGN4SkI1cGh4Y1pROGpGb0cwNFZzaHZkMGUKTUVuVXJNY2ZGZ0laNG5qTUtUUUNNNFpGVVBBaWV5THg0ZjUySHVEb3BwM2U1SnlJTWZXK0tGY05JcEt3Q3NhawpwU29LdElVT3NVSks3cUJWWnhjckl5ZVFWMnFjWU9lWmh0UzV3QnFJd09BaEZ3bENFVDdaZTU4UUhtUzQ4c2xqCjVlVGtSaml2QWxFeHJGektjbGpDNGF4S1Fsbk92VkF6eitHbTMyVTB4UEJGNEJ5ZVBWeENKVUh3MVRzeVRtZWwKU3hORXA3eUhvWGN3bitmWG5hK3Q1SldoMWd4VVp0eTMKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
filesystem: root
mode: 0644
path: /etc/pki/ca-trust/source/anchors/examplecorp-ca.crt
The trust store of machines must also support updating the trust store of nodes.
There are no Operators that can auto-renew certificates on the RHCOS nodes.
Red Hat does not monitor for when CAs expire. However, due to the long life of CAs, this is generally not an issue. However, you might need to periodically update the trust bundle. |
service-ca
is an Operator that creates a self-signed CA when an
OpenShift Container Platform cluster is deployed.
A custom expiration term is not supported. The self-signed CA is stored in a
secret with qualified name service-ca/signing-key
in fields tls.crt
(certificate(s)), tls.key
(private key), and ca-bundle.crt
(CA bundle).
Other services can request a service serving certificate by annotating a
service resource with service.beta.openshift.io/serving-cert-secret-name:
<secret name>
. In response, the Operator generates a new certificate, as
tls.crt
, and private key, as tls.key
to the named secret. The certificate
is valid for two years.
Other services can request that the CA bundle for the service CA be injected
into APIService or ConfigMap resources by annotating with
service.beta.openshift.io/inject-cabundle: true
to support validating
certificates generated from the service CA. In response, the Operator writes its
current CA bundle to the CABundle
field of APIService or as service-ca.crt
to a ConfigMap.
As of OpenShift Container Platform 4.3.5, automated rotation is supported and is backported to some 4.2.z and 4.3.z releases. For any release supporting automated rotation, the service CA is valid for 26 months and is automatically refreshed when there is less than 13 months validity left. If necessary, you can manually refresh the service CA.
The service CA expiration of 26 months is longer than the expected upgrade interval for a supported OpenShift Container Platform cluster, such that non-control plane consumers of service CA certificates will be refreshed after CA rotation and prior to the expiration of the pre-rotation CA.
A manually-rotated service CA does not maintain trust with the previous service CA. You might experience a temporary service disruption until the Pods in the cluster are restarted, which ensures that Pods are using service serving certificates issued by the new service CA. |
These certificates are managed by the system and not the user.
Services that use service CA certificates include:
cluster-autoscaler-operator
cluster-monitoring-operator
cluster-authentication-operator
cluster-image-registry-operator
cluster-ingress-operator
cluster-kube-apiserver-operator
cluster-kube-controller-manager-operator
cluster-kube-scheduler-operator
cluster-networking-operator
cluster-openshift-apiserver-operator
cluster-openshift-controller-manager-operator
cluster-samples-operator
cluster-svcat-apiserver-operator
cluster-svcat-controller-manager-operator
machine-config-operator
console-operator
insights-operator
machine-api-operator
operator-lifecycle-manager
This is not a comprehensive list.
Node certificates are signed by the cluster; they come from a certificate authority (CA) that is generated by the bootstrap process. Once the cluster is installed, the node certificates are auto-rotated.
These certificates are managed by the system and not the user.
The kubelet, in OpenShift Container Platform 4 and later, uses the bootstrap certificate
located in /etc/kubernetes/kubeconfig
to initially bootstrap. This is followed
by the
bootstrap
initialization process and
authorization
of the kubelet to create a CSR.
In that process, the kubelet generates a CSR while communicating over the bootstrap channel. The controller manager signs the CSR, resulting in a certificate that the kubelet manages.
These certificates are managed by the system and not the user.
This bootstrap CA is valid for 10 years.
The kubelet-managed certificate is valid for one year and rotates automatically at around the 80 percent mark of that one year.
You cannot customize the bootstrap certificates.
etcd certificates are signed by the etcd-signer; they come from a certificate authority (CA) that is generated by the bootstrap process.
CA certificates:
etcd CA certificate: /etc/ssl/etcd/ca.crt
etcd metric CA certificate: /etc/ssl/etcd/metric-ca.crt
Server certificates: /etc/ssl/etcd/system:etcd-server
Client certificates: <api_server_pod_directory>/secrets/etcd-client/
Peer certificates: /etc/ssl/etcd/system:etcd-peer
Metric certificates: /etc/ssl/etcd/metric-signer
The CA certificates are valid for 10 years. The peer, client, and server certificates are valid for three years.
These certificates are managed by the system and not the user.
etcd certificates are used for encrypted communication between etcd member peers, as well as encrypted client traffic. The following certificates are generated and used by etcd and other processes that communicate with etcd:
Peer certificates: Used for communication between etcd members.
Client certificates: Used for encrypted server-client communication. Client
certificates are currently used by the API server only, and no other service
should connect to etcd directly except for the proxy. Client secrets
(etcd-client
, etcd-metric-client
, etcd-metric-signer
, and etcd-signer
)
are added to the openshift-config
, openshift-monitoring
, and
openshift-kube-apiserver
namespaces.
Server certificates: Used by the etcd server for authenticating client requests.
Metric certificates: All metric consumers connect to proxy with metric-client certificates.
All certificates for OpenShift Lifecycle Manager (OLM) components
(olm-operator
, catalog-operator
, packageserver
, and
marketplace-operator
) are managed by the system.
Operators installed via OLM can have certificates generated for them if they are
providing API services. packageserver
is one example.
Certificates in the openshift-operator-lifecycle-manager
namespace are managed
by OLM with the exception of certificates used by Operators that require a
validating or mutating webhook.
Operators that install validating or mutating webhooks must currently manage those certificates themselves. They do not require the user to manage the certificates.
OLM will not update the certificates of Operators that it manages in proxy environments. These certificates must be managed by the user via the subscription config.
Applications are usually exposed at
<route_name>.apps.<cluster_name>.<base_domain>
. The <cluster_name>
and
<base_domain>
come from the installation config file. <route_name>
is the
host field of the route, if specified, or the route name. For example,
hello-openshift-default.apps.username.devcluster.openshift.com
.
hello-openshift
is the name of the route and the route is in the default
namespace. You might want clients to access the applications without the need to
distribute the cluster-managed CA certificates to the clients. The administrator
must set a custom default certificate when serving application content.
The Ingress Operator generates a default certificate for an Ingress Controller to serve as a placeholder until you configure a custom default certificate. Do not use operator-generated default certificates in production clusters. |
The user-provided certificates must be provided in a kubernetes.io/tls
type
Secret
in the openshift-config
namespace. Update the
ingresscontroller.operator/default
resource in the
openshift-ingress-operator
namespace to enable the use of the user-provided
certificate.
user-provided certificates are managed by the user.
user-provided certificates are managed by the user.
Applications deployed on the cluster use user-provided certificates for default ingress.
Update the secret containing the user-managed certificate as needed.
The Ingress Operator uses certificates for:
Securing access to metrics for Prometheus.
Securing access to routes.
To secure access to Ingress Operator and Ingress Controller metrics, the Ingress
Operator uses service serving certificates. The Operator requests a certificate
from the service-ca
controller for its own metrics, and the service-ca
controller puts the certificate in a secret named metrics-tls
in the
openshift-ingress-operator
namespace. Additionally, the Ingress Operator
requests a certificate for each Ingress Controller, and the service-ca
controller puts the certificate in a secret named router-metrics-certs-<name>
,
where <name>
is the name of the Ingress Controller, in the
openshift-ingress
namespace.
Each Ingress Controller has a default certificate that it uses for secured
routes that do not specify their own certificates. Unless you specify a custom
certificate, the Operator uses a self-signed certificate by default. The
Operator uses its own self-signed signing certificate to sign any default
certificate that it generates. The Operator generates this signing certificate
and puts it in a secret named router-ca
in the openshift-ingress-operator
namespace. When the Operator generates a default certificate, it puts the default
certificate in a secret named router-certs-<name>
(where <name>
is the name
of the Ingress Controller) in the openshift-ingress
namespace.
The Ingress Operator generates a default certificate for an Ingress Controller to serve as a placeholder until you configure a custom default certificate. Do not use Operator-generated default certificates in production clusters. |
An empty defaultCertificate
field causes the Ingress Operator to use its self-signed CA to generate a serving certificate for the specified domain.
The default CA certificate and key generated by the Ingress Operator. Used to sign Operator-generated default serving certificates.
In the default workflow, the wildcard default serving certificate, created by the Ingress Operator and signed using the generated default CA certificate. In the custom workflow, this is the user-provided certificate.
The router deployment. Uses the certificate in secrets/router-certs-default
as its default front-end server certificate.
In the default workflow, the contents of the wildcard default serving certificate (public and private parts) are copied here to enable OAuth integration. In the custom workflow, this is the user-provided certificate.
Transitional resource containing the certificate (public part) of the Operator-generated default CA certificate; read by OAuth and the web console to establish trust. This object will be removed in a future release.
The public (certificate) part of the default serving certificate. Replaces the configmaps/router-ca
resource.
The user updates the cluster proxy configuration with the CA certificate that signed the ingresscontroller
serving certificate. This enables components like auth
, console
, and the registry to trust the serving certificate.
The cluster-wide trusted CA bundle containing the combined Red Hat Enterprise Linux CoreOS (RHCOS) and user-provided CA bundles or an RHCOS-only bundle if a user bundle is not provided.
The custom CA certificate bundle, which instructs other components (for example, auth
and console
) to trust an ingresscontroller
configured with a custom certificate.
The trustedCA
field is used to reference the user-provided CA bundle.
The Cluster Network Operator injects the trusted CA bundle into the proxy-ca
ConfigMap.
OpenShift Container Platform 4.3 and earlier versions use router-ca
.
The expiration terms for the Ingress Operator’s certificates are as follows:
The expiration date for metrics certificates that the service-ca
controller
creates is two years after the date of creation.
The expiration date for the Operator’s signing certificate is two years after the date of creation.
The expiration date for default certificates that the Operator generates is two years after the date of creation.
You cannot specify custom expiration terms on certificates that the Ingress
Operator or service-ca
controller creates.
You cannot specify expiration terms when installing OpenShift Container Platform for
certificates that the Ingress Operator or service-ca
controller creates.
Prometheus uses the certificates that secure metrics.
The Ingress Operator uses its signing certificate to sign default certificates that it generates for Ingress Controllers for which you do not set custom default certificates.
Cluster components that use secured routes may use the default Ingress Controller’s default certificate.
Ingress to the cluster via a secured route uses the default certificate of the Ingress Controller by which the route is accessed unless the route specifies its own certificate.
Ingress certificates are managed by the user. See Replacing the default ingress certificate for more information.
The service-ca
controller automatically rotates the certificates that it
issues. However, it is possible to use oc delete secret <secret>
to
manually rotate service serving certificates.
The Ingress Operator does not rotate its own signing certificate or the default certificates that it generates. Operator-generated default certificates are intended as placeholders for custom default certificates that you configure.
Monitoring components secure their traffic with service CA certificates. These certificates are valid for 2 years and are replaced automatically on rotation of the service CA, which is every 13 months.
If the certificate lives in the openshift-monitoring
or openshift-logging
namespace, it is system managed and rotated automatically.
These certificates are managed by the system and not the user.
Control plane certificates are included in these namespaces:
openshift-config-managed
openshift-kube-apiserver
openshift-kube-apiserver-operator
openshift-kube-controller-manager
openshift-kube-controller-manager-operator
openshift-kube-scheduler
Control plane certificates are managed by the system and rotated automatically.
In the rare case that your control plane certificates expired, see Recovering from expired control plane certificates