OpenShift Container Platform 4.12 release notes

Red Hat Enterprise Linux CoreOS (RHCOS) nodes installed from an OpenShift Container Platform 4.12 boot image now use a platform-specific default console. The default consoles on cloud platforms correspond to the specific system consoles expected by that cloud provider. VMware and OpenStack images now use a primary graphical console and a secondary serial console. Other bare metal installations now use only the graphical console by default, and do not enable a serial console. Installations performed with coreos-installer can override existing defaults and enable the serial console.

Existing nodes are not affected. New nodes on existing clusters are not likely to be affected because they are typically installed from the boot image that was originally used to install the cluster.

For information about how to enable the serial console, see the following documentation:

OpenShift Container Platform now supports configuring Red Hat Enterprise Linux CoreOS (RHCOS) nodes for IBM Secure Execution on IBM Z and LinuxONE (s390x architecture) as a Technology Preview feature. IBM Secure Execution is a hardware enhancement that protects memory boundaries for KVM guests. IBM Secure Execution provides the highest level of isolation and security for cluster workloads, and you can enable it by using an IBM Secure Execution-ready QCOW2 boot image.

To use IBM Secure Execution, you must have host keys for your host machine(s) and they must be specified in your Ignition configuration file. IBM Secure Execution automatically encrypts your boot volumes using LUKS encryption.

For more information, see Installing RHCOS using IBM Secure Execution.

RHCOS now uses Red Hat Enterprise Linux (RHEL) 8.6 packages in OpenShift Container Platform 4.12. This enables you to have the latest fixes, features, and enhancements, as well as the latest hardware support and driver updates. OpenShift Container Platform 4.10 is an Extended Update Support (EUS) release that will continue to use RHEL 8.4 EUS packages for the entirety of its lifecycle.

Assisted Installer SaaS on console.redhat.com supports installation of OpenShift Container Platform on the Nutanix platform with Machine API integration using either the Assisted Installer user interface or the REST API. Integration enables Nutanix Prism users to manage their infrastructure from a single interface, and enables auto-scaling. There are a few additional installation steps to enable Nutanix integration with Assisted Installer SaaS. See the Assisted Installer documentation for details.

Beginning with OpenShift Container Platform 4.12, you can specify either Network Load Balancer (NLB) or Classic as a persistent load balancer type in AWS during installation. Afterwards, if an ingress Controller is deleted, the load balancer type persists with the lbType configured during installation.

For more information, see Installing a cluster on AWS with network customizations.

With this update you can install OpenShift Container Platform to an existing VPC with installer-provisioned infrastructure, extending the worker nodes to Local Zones subnets. The installation program will provision worker nodes on the edge of the AWS network that are specifically designated for user applications by using NoSchedule taints. Applications deployed on the Local Zones locations deliver low latency for end users.

For more information, see Installing a cluster using AWS Local Zones.

OpenShift Container Platform is now available on the GCP Marketplace. Installing an OpenShift Container Platform with a GCP Marketplace image lets you create self-managed cluster deployments that are billed on pay-per-use basis (hourly, per core) through GCP, while still being supported directly by Red Hat.

For more information about installing using installer-provisioned infrastructure, see Using a GCP Marketplace image. For more information about installing a using user-provisioned infrastructure, see Creating additional worker machines in GCP.

The installer now gathers serial console logs from the bootstrap and control plane hosts on GCP and Azure. This log data is added to the standard bootstrap log bundle.

For more information, see Troubleshooting installation issues.

IBM Cloud VPC is now generally available in OpenShift Container Platform 4.12.

For more information about installing a cluster, see Preparing to install on IBM Cloud VPC.

OpenShift Container Platform 4.12 uses Kubernetes 1.25, which removed several deprecated APIs.

A cluster administrator must provide a manual acknowledgment before the cluster can be upgraded from OpenShift Container Platform 4.11 to 4.12. This is to help prevent issues after upgrading to OpenShift Container Platform 4.12, where APIs that have been removed are still in use by workloads, tools, or other components running on or interacting with the cluster. Administrators must evaluate their cluster for any APIs in use that will be removed and migrate the affected components to use the appropriate new API version. After this is done, the administrator can provide the administrator acknowledgment.

All OpenShift Container Platform 4.11 clusters require this administrator acknowledgment before they can be upgraded to OpenShift Container Platform 4.12.

For more information, see Preparing to update to OpenShift Container Platform 4.12.

Beginning with OpenShift Container Platform 4.12, you can enable a feature set as part of the installation process. A feature set is a collection of OpenShift Container Platform features that are not enabled by default.

For more information about enabling a feature set during installation, see Enabling OpenShift Container Platform features using feature gates.

OpenShift Container Platform 4.12 is now supported on ARM architecture-based Azure installer-provisioned infrastructure. AWS Graviton 3 processors are now available for cluster deployments and are also supported on OpenShift Container Platform 4.11. For more information about instance availability and installation documentation, see Supported installation methods for different platforms

Using the oc-mirror CLI plugin to mirror file-based catalog Operator images in OCI format instead of Docker v2 format is now available as a Technology Preview.

For more information, see Mirroring file-based catalog Operator images in OCI format.

In OpenShift Container Platform 4.12, you can install a cluster on GCP into a shared VPC as a Technology Preview. In this installation method, the cluster is configured to use a VPC from a different GCP project. A shared VPC enables an organization to connect resources from multiple projects to a common VPC network. You can communicate within the organization securely and efficiently by using internal IP addresses from that network.

For more information, see Installing a cluster on GCP into a shared VPC.

With this update, in bare-metal installations without a provisioning network, the Ironic API service is accessible through a proxy server. This proxy server provides a consistent IP address for the Ironic API service. If the Metal3 pod that contains metal3-ironic relocates to another pod, the consistent proxy address ensures constant communication with the Ironic API service.

In OpenShift Container Platform 4.12, you can install a cluster on GCP using a virtual machine with a service account attached to it. This allows you to perform an installation without needing to use a service account JSON file.

For more information, see Creating a GCP service account.

In OpenShift Container Platform 4.12, the propagateUserTags parameter is a flag that directs in-cluster Operators to include the specified user tags in the tags of the AWS resources that the Operators create.

For more information, see Optional configuration parameters.

In earlier versions of OpenShift Container Platform, Ironic container images used Red Hat Enterprise Linux (RHEL) 8 as the base image. From OpenShift Container Platform 4.12, Ironic container images use RHEL 9 as the base image. The RHEL 9 base image adds support for CentOS Stream 9, Python 3.8, and Python 3.9 in Ironic components.

For more information about the Ironic provisioning service, see Deploying installer-provisioned clusters on bare metal.

In OpenShift Container Platform 4.12, clusters that run on Red Hat OpenStack Platform (RHOSP) are switched from the legacy OpenStack cloud provider to the external Cloud Controller Manager (CCM). This change follows the move in Kubernetes from in-tree, legacy cloud providers to external cloud providers that are implemented by using the Cloud Controller Manager.

For more information, see The OpenStack Cloud Controller Manager.

In OpenShift Container Platform 4.12, cluster deployments to Red Hat OpenStack Platform (RHOSP) clouds that have distributed compute node (DCN) architecture were validated. A reference architecture for these deployments is forthcoming.

For a brief overview of this type of deployment, see the blog post Deploying Your Cluster at the Edge With OpenStack.

OpenShift Container Platform 4.12 is now supported on the AWS Outposts platform as a Technology Preview. With AWS Outposts you can deploy edge-based worker nodes, while using AWS Regions for the control plane nodes. For more information, see Installing a cluster on AWS with remote workers on AWS Outposts.

The Agent-based installation supports two input modes:

With the preferred mode, you can configure the install-config.yaml file and specify Agent-based specific settings in the agent-config.yaml file. For more information, see About the Agent-based OpenShift Container Platform Installer.

Agent-based OpenShift Container Platform Installer supports OpenShift Container Platform clusters in Federal Information Processing Standards (FIPS) compliant mode. You must set the value of the fips field to True in the install-config.yaml file. For more information, see About FIPS compliance.

You can perform an Agent-based installation in a disconnected environment. To create an image that is used in a disconnected environment, the imageContentSources section in the install-config.yaml file must contain the mirror information or registries.conf file if you are using ZTP manifests. The actual configuration settings to use in these files are supplied by either the oc adm release mirror or oc mirror command. For more information, see Understanding disconnected installation mirroring.

When creating the image set configuration, you can add the graph: true field to build and push the graph-data image to the mirror registry. The graph-data image is required to create OpenShift Update Service (OSUS). The graph: true field also generates the UpdateService custom resource manifest. The oc command-line interface (CLI) can use the UpdateService custom resource manifest to create OSUS.

You can create the agent ISO image with the following IP address configurations:

For more information, see Dual and single IP stack clusters.

You can install the multicluster engine for Kubernetes Operator and deploy a hub cluster with the Agent-based OpenShift Container Platform Installer. For more information, see Preparing an Agent-based installed cluster for the multicluster engine for Kubernetes Operator.

The Agent-based OpenShift Container Platform Installer performs validations on:

For more information, see Installation validations.

With the Agent-based OpenShift Container Platform Installer, you can configure static IP addresses for IPv4, IPv6, or dual-stack (both IPv4 and IPv6) for all the hosts prior to creating the agent ISO image. You can add the static addresses to the hosts section of the agent-config.yaml file or in the NMStateConfig.yaml file if you are using the ZTP manifests. Note that the configuration of the addresses must follow the syntax rules for NMState as described in NMState state examples.

For more information, see About networking.

With the Agent-based OpenShift Container Platform Installer, you can define your installation configurations, generate an ISO for all the nodes, and then have an unattended installation by booting the target systems with the generated ISO. For more information, see Installing a OpenShift Container Platform cluster with the Agent-based OpenShift Container Platform Installer.

You can configure the hostname, network configuration in NMState format, root device hints, and role in an Agent-based installation.

For more information, see About root device hints.

With the Agent-based OpenShift Container Platform Installer, you can deploy to environments where you rely on DHCP to configure networking for all the nodes, as long as you know the IP that at least one of the systems will receive. This IP is required so that all nodes use it as a meeting point. For more information, see DHCP.

You can now install a cluster on Nutanix when the environment has limited access to to the internet, as in the case of a disconnected or restricted network cluster. With this type of installation, you create a registry that mirrors the contents of the OpenShift Container Platform image registry and contains the installation media. You can create this registry on a mirror host, which can access both the internet and your closed network.

For more information, see About disconnected installation mirroring and Installing a cluster on Nutanix in a restricted network.

To install a CSI driver on a cluster running on vSphere, the following requirements must be met:

Components with versions earlier than those above are still supported, but are deprecated. These versions are still fully supported, but version 4.12 of OpenShift Container Platform requires vSphere virtual hardware version 15 or later. For more information, see Deprecated and removed features.

Failing to meet the above requirements prevents OpenShift Container Platform from upgrading to OpenShift Container Platform 4.13 or later.

The following new cluster capabilities have been added:

A new predefined set of cluster capabilities, v4.12, has been added. This includes all capabilities from v4.11, and the new capabilities added with the current release.

For more information, see link: Enabling cluster capabilities.

OpenShift Container Platform 4.12 with multi-architecture compute machines now supports manifest listed images on image streams. For more information about manifest list images, see Configuring multi-architecture compute machines on an OpenShift Container Platform cluster.

On a cluster with multi-architecture compute machines, you can now override the node affinity in the Operator’s Subscription object to schedule pods on nodes with architectures that the Operator supports. For more information, see Using node affinity to control where an Operator is installed.

With this release, there are several updates to the Administrator perspective of the web console.

With this release, there are several updates to the Developer perspective of the web console. You can perform the following actions:

Using Krew to install and manage plugins for the OpenShift CLI (oc) is now available as a Technology Preview.

For more information, see Managing CLI plugins with Krew.

The Security Profiles Operator (SPO) is now available for OpenShift Container Platform 4.12 and later.

The SPO provides a way to define secure computing (seccomp) profiles and SELinux profiles as custom resources, synchronizing profiles to every node in a given namespace.

For more information, see Security Profiles Operator Overview.

Assisted Installer supports installation of OpenShift Container Platform 4.12 and later versions with dual stack networking for the API VIP and ingress VIP on bare metal only. This support introduces two new configuration settings: api_vips and ingress_vips, which can take a list of IP addresses. The legacy settings, api_vip and ingress_vip must also be set in OpenShift Container Platform 4.12; however, since they only take one IP address, you must set the IPv4 address when configuring dual stack networking for the API VIP and ingress VIP with the legacy api_vip and ingress_vip configuration settings.

The API VIP address and the ingress VIP address must be of the primary IP address family when using dual-stack networking. Currently, Red Hat does not support dual-stack VIPs or dual-stack networking with IPv6 as the primary IP address family. However, Red Hat does support dual-stack networking with IPv4 as the primary IP address family. Therefore, you must place the IPv4 entries before the IPv6 entries. See the Assisted Installer documentation for details.

Red Hat OpenShift Networking is an ecosystem of features, plugins, and advanced networking capabilities that extend Kubernetes networking beyond the Kubernetes CNI plugin with the advanced networking-related features that your cluster needs to manage its network traffic for one or multiple hybrid clusters. This ecosystem of networking capabilities integrates ingress, egress, load balancing, high-performance throughput, security, and inter-, and intra-cluster traffic management and provides role-based observability tooling to reduce its natural complexities.

For more information, see About networking.

When installing a new cluster the OVN-Kubernetes network plugin is the default networking plugin. For all prior versions of OpenShift Container Platform, OpenShift SDN remains the default networking plugin.

The OVN-Kubernetes network plugin includes a wider array of features than OpenShift SDN, including:

There are also enormous scale, performance, and stability improvements in OpenShift Container Platform 4.12 compared to prior versions.

If you are using the OpenShift SDN network plugin, note that:

For more information about OVN-Kubernetes, including a feature comparison matrix with OpenShift SDN, see About the OVN-Kubernetes network plugin.

For information on migrating to OVN-Kubernetes from OpenShift SDN, see Migrating from the OpenShift SDN network plugin.

This update introduces a new stateless ingress Node Firewall Operator. You can now configure firewall rules at the node level. For more information, see ingress Node Firewall Operator.

The following metrics are now available for the OVN-Kubernetes network plugin:

The following metric has been removed:

Beginning with OpenShift Container Platform 4.12, the ability to configure multiple vCenter datacenters and multiple vCenter clusters in a single vCenter installation using installer-provisioned infrastructure is now available as a Technology Preview feature. Using vCenter tags, you can use this feature to associate vCenter datacenters and compute clusters with openshift-regions and openshift-zones. These associations define failure domains to enable application workloads to be associated with specific locations and failure domains.

Beginning with OpenShift Container Platform 4.12, you can configure the networking settings such as DNS servers or search domains, VLANs, bridges, and interface bonding using the Kubernetes NMState Operator on your VMware vSphere instance.

For more information, see About the Kubernetes NMState Operator.

Beginning with OpenShift Container Platform 4.12, you can configure the networking settings such as DNS servers or search domains, VLANs, bridges, and interface bonding using the Kubernetes NMState Operator on your OpenStack instance.

For more information, see About the Kubernetes NMState Operator.

In OpenShift Container Platform 4.12, the External DNS Operator modifies the format of the ExternalDNS wildcard TXT records on AzureDNS. The External DNS Operator replaces the asterisk with any in ExternalDNS wildcard TXT records. You must avoid the ExternalDNS wildcard A and CNAME records having any leftmost subdomain because this might cause a conflict.

The upstream version of ExternalDNS for OpenShift Container Platform 4.12 is v0.13.1.

In OpenShift Container Platform 4.12, the Cluster ingress Operator exports a new metric named route_metrics_controller_routes_per_shard. The shard_name label of the metric specifies the name of the shards. This metric gives the total number of routes that are admitted by each shard.

The following metrics are sent through telemetry.

In OpenShift Container Platform 4.12, the AWS Load Balancer controller now implements the Kubernetes ingress specification for multiple matches. If multiple paths within an ingress match a request, the longest matching path takes the precedence. If two paths still match, paths with an exact path type take precedence over a prefix path type.

The AWS Load Balancer Operator sets the EnableIPTargetType feature gate to false. The AWS Load Balancer controller disables the support for services and ingress resources for target-type ip.

The upstream version of aws-load-balancer-controller for an OpenShift Container Platform 4.12 is v2.4.4.

You can now use the OpenShift Container Platform Custom Metrics Autoscaler Operator to dynamically scale the default ingress Controller based on metrics in your deployed cluster, such as the number of worker nodes available. The Custom Metrics Autoscaler is available as a Technology Preview feature.

For more information, see Autoscaling an ingress Controller.

In OpenShift Container Platform 4.12, the default value for the maxConnections setting is now 50000. Previously starting with OpenShift Container Platform 4.11, the default value for the maxConnections setting was 20000.

For more information, see ingress Controller configuration parameters.

You can now configure an ingress Controller to stop automatic DNS management and start manual DNS management. Set the dnsManagementPolicy parameter to specify automatic or manual DNS management.

For more information, see Configuring an ingress Controller to manually manage DNS.

OpenShift Container Platform 4.12 adds support for the following SR-IOV devices:

For more information, see Supported devices.

OpenShift Container Platform 4.12 adds OvS Hardware Offload support for the following devices:

For more information, see Supported devices.

OpenShift Container Platform 4.12 adds support for configuring multi-network policy for SR-IOV devices.

You can now configure multi-network for SR-IOV additional networks. Configuring SR-IOV additional networks is a Technology Preview feature and is only supported with kernel network interface cards (NICs).

For more information, see Configuring multi-network policy.

You can update the ingress Controller to switch between an AWS Classic Load Balancer (CLB) and an AWS Network Load Balancer (NLB) without deleting the ingress Controller.

For more information, see Configuring ingress cluster traffic on AWS.

Pods created with the Single Root I/O Virtualization (SR-IOV) CNI plugin, where the IP address management CNI plugin has assigned IPs, now send IPv6 unsolicited neighbor advertisements and/or IPv4 gratuitous address resolution protocol by default onto the network. This enhancement notifies hosts of the new pod’s MAC address for a particular IP to refresh ARP/NDP caches with the correct information.

For more information, see Supported devices.

You can now configure the time-to-live (TTL) duration of both successful and unsuccessful DNS queries cached by CoreDNS.

For more information, see Tuning the CoreDNS cache.

Previously, the subnet that OVN-Kubernetes uses internally was 100.64.0.0/16 for IPv4 and fd98::/48 for IPv6 and could not be modified. To support instances when these subnets overlap with existing subnets in your infrastructure, you can now change these internal subnets to avoid any overlap.

For more information, see Cluster Network Operator configuration object

RHOSP, paired with OpenShift Container Platform, now supports automatic attachment and detachment of Egress IP addresses. The traffic from one or more pods in any number of namespaces has a consistent source IP address for services outside of the cluster. This support applies to OpenShift SDN and OVN-Kubernetes as default network providers.

If you plan to migrate from the OpenShift SDN network plugin to the OVN-Kubernetes network plugin, your configurations for the following capabilities are automatically converted to work with OVN-Kubernetes:

For more information about how the migration to OVN-Kubernetes works, see Migrating from the OpenShift SDN cluster network provider.

For the OVN-Kubernetes network plugin, egress firewalls support audit logging using the same mechanism that network policy audit logging uses. For more information, see Logging for egress firewall and network policy rules.

With this update, in BGP mode, you can use the node selector to advertise the MetalLB service from a subset of nodes, using a specific pool of IP addresses. This feature was introduced as a Technology Preview feature in OpenShift Container Platform 4.11 and is now generally available in OpenShift Container Platform 4.12 for BGP mode only. L2 mode remains a Technology Preview feature.

For more information, see Advertising an IP address pool from a subset of nodes.

This update provides additional deployment specifications for MetalLB. When you use a custom resource to deploy MetalLB, you can use these additional deployment specifications to manage how MetalLB speaker and controller pods deploy and run in your cluster. For example, you can use MetalLB deployment specifications to manage where MetalLB pods are deployed, define CPU limits for MetalLB pods, and assign runtime classes to MetalLB pods.

For more information about deployment specifications for MetalLB, see Deployment specifications for MetalLB.

Previously, the nodeip-configuration service on a cluster host selected the IP address from the interface that the default route used. If multiple routes were present, the service would select the route with the lowest metric value. As a result, network traffic could be distributed from the incorrect interface.

With OpenShift Container Platform 4.12, a new interface has been added to the nodeip-configuration service, which allows users to create a hint file. The hint file contains a variable, NODEIP_HINT, that overrides the default IP selection logic and selects a specific node IP address from the subnet NODEIP_HINT variable. Using the NODEIP_HINT variable allows users to specify which IP address is used, ensuring that network traffic is distributed from the correct interface.

For more information, see Optional: Overriding the default node IP selection logic.

In OpenShift Container Platform 4.12, CoreDNS uses version 1.10.0, which includes the following changes:

With this update, a cluster administrator can configure the reload interval to force HAProxy to reload its configuration less frequently in response to route and endpoint updates. The default minimum HAProxy reload interval is 5 seconds.

For more information, see Configuring HAProxy reload interval.

The Network Observability Operator releases updates independently from the OpenShift Container Platform minor version release stream. Updates are available through a single, rolling stream which is supported on all currently supported versions of OpenShift Container Platform 4. Information regarding new features, enhancements, and bug fixes for the Network Observability Operator can be found in the Network Observability release notes.

IPv6 for secondary network interfaces is now supported in clusters that run on RHOSP.

For more information,see Enabling IPv6 connectivity to pods on RHOSP.

Resulting from the switch to an external OpenStack cloud provider, UDP is now supported for LoadBalancer services for clusters that run on that platform.

If you configured and deployed your hosting service cluster, you can now deploy the SR-IOV Operator for a hosted cluster. For more information, see Deploying the SR-IOV Operator for hosted control planes.

With this update, in installer-provisioned infrastructure clusters, the ingressVIP and apiVIP configuration settings in the install-config.yaml file are deprecated. Instead, use the ingressVIPs and apiVIPs configuration settings. These settings support dual-stack networking for applications on bare metal that require IPv4 and IPv6 access to the cluster by using the ingress VIP and API VIP services. The ingressVIPs and apiVIPs configuration settings use a list format to specify an IPv4 address, an IPv6 address, or both IP address formats. The order of the list indicates the primary and secondary VIP address for each service. The primary IP address must be from the IPv4 network when using dual stack networking.

With this update, you can switch the BlueField-2 network device from data processing unit (DPU) mode to network interface controller (NIC) mode.

For more information, see Switching Bluefield-2 from DPU to NIC.

Previously, for clusters that use the OVN-Kubernetes network plugin, during cluster installation you could enable hybrid networking so that your cluster supported Windows nodes. Now you can enable hybrid networking after installation. For more information, see Configuring hybrid networking.

The ServiceSpec component in the Network API under the Service object describes the attributes that a user creates on a service. The allocateLoadBalancerNodePorts attribute within the ServiceSpec component is now supported as of OpenShift Container Platform 4.12.28 release. The allocateLoadBalancerNodePorts attribute defines whether the NodePorts will be automatically allocated for services of the LoadBalancer type.

For more information, see Network API ServiceSpec object

OpenShift Container Platform is capable of provisioning persistent volumes (PVs) using the Container Storage Interface (CSI) driver for Google Compute Platform (GCP) Filestore. The GCP Filestore CSI Driver Operator that manages this driver is in Technology Preview.

For more information, see see GCP Filestore CSI Driver Operator.

Starting with OpenShift Container Platform 4.8, automatic migration for in-tree volume plugins to their equivalent Container Storage Interface (CSI) drivers became available as a Technology Preview feature. Support for Amazon Web Services (AWS) Elastic Block Storage (EBS) was provided in this feature in OpenShift Container Platform 4.8, and OpenShift Container Platform 4.12 now supports automatic migration for AWS EBS as generally available. CSI migration for AWS EBS is now enabled by default and requires no action by an administrator.

This feature automatically translates in-tree objects to their counterpart CSI representations and should be completely transparent to users. Translated objects are not stored on disk, and user data is not migrated.

While storage class referencing to the in-tree storage plugin will continue working, it is recommended that you switch the default storage class to the CSI storage class.

For more information, see CSI Automatic Migration.

Starting with OpenShift Container Platform 4.8, automatic migration for in-tree volume plugins to their equivalent Container Storage Interface (CSI) drivers became available as a Technology Preview feature. Support for Google Compute Engine Persistent Disk (GCP PD) was provided in this feature in OpenShift Container Platform 4.9, and OpenShift Container Platform 4.12 now supports automatic migration for GCP PD as generally available. CSI migration for GCP PD is now enabled by default and requires no action by an administrator.

This feature automatically translates in-tree objects to their counterpart CSI representations and should be completely transparent to users. Translated objects are not stored on disk, and user data is not migrated.

While storage class referencing to the in-tree storage plugin will continue working, it is recommended that you switch the default storage class to the CSI storage class.

For more information, see CSI Automatic Migration.

Updates from OpenShift Container Platform 4.12 to 4.13 and from 4.13 to 4.14 are blocked if all of the following conditions are true:

For more information, see CSI Automatic Migration.

This new feature exposes the currently available storage capacity using CSIStorageCapacity objects, and enhances scheduling of pods that use Container Storage Interface (CSI) volumes with late binding. Currently, the only OpenShift Container Platform storage type that supports this features is OpenShift Data Foundation.

OpenShift Container Platform provides the ability to deploy OpenShift Container Platform for vSphere on different zones and regions, which allows you to deploy over multiple compute clusters, thus helping to avoid a single point of failure.

For more information, see vSphere CSI topology.

The local ephemeral storage resource management features is now generally available. With this feature, you can manage local ephemeral storage by specifying requests and limits.

For more information, see Ephemeral storage management.

Volume populators use datasource to enable creating pre-populated volumes.

Volume population is currently enabled, and supported as a Technology Preview feature. However, OpenShift Container Platform does not ship with any volume populators.

For more information, see Volume populators.

For OpenShift Container Platform 4.12, VMWare vSphere Container Storage Interface (CSI) Driver Operator requires the following minimum components installed:

If a third-party CSI driver is present in the cluster, OpenShift Container Platform does not overwrite it. The presence of a third-party CSI driver prevents OpenShift Container Platform from upgrading to OpenShift Container Platform 4.13 or later.

For more information, see VMware vSphere CSI Driver Operator requirements.

OpenShift Container Platform 4.12 supports Azure File Container Storage Interface (CSI) Driver Operator with Network File System (NFS) as generally available.

For more information, see NFS support.

Starting in OpenShift Container Platform 4.12, Operator Lifecycle Manager (OLM) introduces the platform Operator type as a Technology Preview feature. The platform Operator mechanism relies on resources from the RukPak component, also introduced in OpenShift Container Platform 4.12, to source and manage content.

A platform Operator is an OLM-based Operator that can be installed during or after an OpenShift Container Platform cluster’s Day 0 operations and participates in the cluster’s lifecycle. As a cluster administrator, you can use platform Operators to further customize your OpenShift Container Platform installation to meet your requirements and use cases.

For more information about platform Operators, see Managing platform Operators. For more information about RukPak and its resources, see Operator Framework packaging format.

By default, when you install an Operator, OpenShift Container Platform randomly installs the Operator pod to one of your worker nodes.

In OpenShift Container Platform 4.12, you can control where an Operator pod is installed by adding affinity constraints to the Operator’s Subscription object.

For more information, see Controlling where an Operator is installed.

In OpenShift Container Platform 4.12, pod security admission synchronization is enabled by default if an Operator is installed in user-created namespaces that have an openshift- prefix. Synchronization is enabled after a cluster service version (CSV) is created in the namespace. The synchronized label inherits the permissions of the service accounts in the namespace.

For more information, see Security context constraint synchronization with pod security standards.

You can configure the security context of a catalog pod by using the --security-context-config flag on the run bundle and bundle-upgrade subcommands. The flag enables seccomp profiles to comply with pod security admission. The flag accepts the values of restricted and legacy. If you do not specify a value, the seccomp profile defaults to restricted. If your catalog pod cannot run with restricted permissions, set the flag to legacy, as shown in the following example:

You can now check bundle manifests for deprecated APIs removed from Kubernetes 1.25 by using the Operator Framework suite of tests with the bundle validate subcommand.

For example:

If your Operator requests permission to use any of the APIs removed from Kubernetes 1.25, the command displays a warning message.

If any of the API versions removed from Kubernetes 1.25 are included in your Operator’s cluster service version (CSV), the command displays an error message.

See Beta APIs removed from Kubernetes 1.25 and the Operator SDK CLI reference for more information.

OpenShift Container Platform 4.12 introduces control plane machine sets. Control plane machine sets provide management capabilities for control plane machines that are similar to what compute machine sets provide for compute machines. For more information, see Managing control plane machines.

OpenShift Container Platform now supports setting the log level verbosity of the cluster autoscaler by setting the logVerbosity parameter in the ClusterAutoscaler custom resource. For more information, see the ClusterAutoscaler resource definition.

OpenShift Container Platform now supports enabling boot diagnostics on Azure machines that your machine set creates. For more information, see "Enabling Azure boot diagnostics" for compute machines or control plane machines.

Red Hat Enterprise Linux CoreOS (RHCOS) image layering allows you to add new images on top of the base RHCOS image. This layering does not modify the base RHCOS image. Instead, it creates a custom layered image that includes all RHCOS functionality and adds additional functionality to specific nodes in the cluster.

Currently, RHCOS image layering allows you to work with Customer Experience and Engagement (CEE) to obtain and apply Hotfix packages on top of your RHCOS image, based on the Red Hat Hotfix policy. It is planned for future releases that you can use RHCOS image layering to incorporate third-party software packages such as Libreswan or numactl.

For more information, see RHCOS image layering.

OpenShift Container Platform now supports updating the default interface-specific safe sysctls.

You can add or remove sysctls from the predefined list. When you add sysctls, they can be set across all nodes. Updating the interface-specific safe sysctls list is a Technology Preview feature only.

For more information, see Updating the interface-specific safe sysctls list.

Setting a time zone for a cron job schedule is now offered as a Technology Preview. If a time zone is not specified, the Kubernetes controller manager interprets the schedule relative to its local time zone.

For more information, see Creating cron jobs.

OpenShift Container Platform support for Linux Control Group version 2 (cgroup v2) has been promoted to Technology Preview. cgroup v2 is the next version of the kernel control groups. cgroups v2 offers multiple improvements, including a unified hierarchy, safer sub-tree delegation, new features such as Pressure Stall Information, and enhanced resource management and isolation. For more information, see Enabling Linux Control Group version 2 (cgroup v2).

OpenShift Container Platform now supports the crun container runtime in Technology Preview. You can switch between the crun container runtime and the default container runtime as needed by using a ContainerRuntimeConfig custom resource (CR). For more information, see About the container engine and container runtime.

OpenShift Container Platform now supports control plane fencing by the Self Node Remediation Operator. In the event of node failure, you can follow remediation strategies on both worker nodes and control plane nodes. For more information, see the Workload Availability for Red Hat OpenShift documentation.

OpenShift Container Platform now supports control plane fencing on the Node Health Check Operator. In the event of node failure, you can follow remediation strategies on both worker nodes and control plane nodes. For more information, see the Workload Availability for Red Hat OpenShift documentation.

The Node Health Check Operator now also includes a web console plugin for managing Node Health Checks. For more information, see the Workload Availability for Red Hat OpenShift documentation.

For installing or updating to the latest version of the Node Health Check Operator, use the stable subscription channel. For more information, see the Workload Availability for Red Hat OpenShift documentation.

This release includes the following version updates for monitoring stack components and dependencies:

You can now use pod topology spread constraints to control how Prometheus, Thanos Ruler, and Alertmanager pods are spread across a network topology when OpenShift Container Platform pods are deployed in multiple availability zones.

You can now configure an optional kubelet service monitor for Prometheus Adapter (PA) that improves data consistency across multiple autoscaling requests. Enabling this service monitor eliminates the possibility that two queries sent at the same time to PA might yield different results because the underlying PromQL queries executed by PA might be on different Prometheus servers.

With this release, if you configure an Alertmanager secret to hold additional keys and if the Alertmanager configuration references these keys as files (such as templates, TLS certificates, or tokens), your configuration settings must point to these keys by using an absolute path rather than a relative path. These keys are available under the /etc/alertmanager/config directory. In earlier releases of OpenShift Container Platform, you could use relative paths in your configuration to point to these keys because the Alertmanager configuration file was located in the same directory as the keys.

At the cluster level by default, a systemd service sets a Receive Packet Steering (RPS) mask for virtual network interfaces. The RPS mask routes interrupt requests from virtual network interfaces according to the list of reserved CPUs defined in the performance profile. At the container level, a CRI-O hook script also sets an RPS mask for all virtual network devices.

With this update, if you set spec.workloadHints.realTime in the performance profile to False, the system also disables both the systemd service and the CRI-O hook script which set the RPS mask. The system disables these RPS functions because RPS is typically relevant to use cases requiring low-latency, realtime workloads only.

To retain RPS functions even when you set spec.workloadHints.realTime to False, see the RPS Settings section of the Red Hat Knowledgebase solution Performance addons operator advanced configuration.

For more information about configuring workload hints, see Understanding workload hints.

The tuned profile now defines the fs.aio-max-nr sysctl value by default, improving asynchronous I/O performance for default node profiles.

The low latency tuning has been updated to use the latest kernel features and options. The fix for 2117780 introduced a new per-CPU kthread, ktimers. This thread must be pinned to the proper CPU cores. With this update, there is no functional change; the isolation of the workload is the same. For more information, see 2102450.

In OpenShift Container Platform 4.12, by enabling C-states and OS-controlled P-states, you can use different power-saving configurations for critical and non-critical workloads. You can apply the configurations through the new perPodPowerManagement workload hint, and the cpu-c-states.crio.io and cpu-freq-governor.crio.io CRI-O annotations. For more information about the feature, see Power-saving configurations.

In OpenShift Container Platform 4.11, a feature allowing you to manually add worker nodes to single-node OpenShift clusters was introduced. This feature is now also available in GitOps ZTP.

For more information, see Adding worker nodes to single-node OpenShift clusters with GitOps ZTP.

In OpenShift Container Platform 4.12, you can use the factory-precaching-cli tool to pre-cache OpenShift Container Platform and Operator images on a server at the factory, and then you can include the pre-cached server to the site for deployment. For more information about the factory-precaching-cli tool, see Pre-caching images for single-node OpenShift deployments.

In OpenShift Container Platform 4.12, you can use the factory-precaching-cli tool in the GitOps ZTP workflow. For more information, see Pre-caching images for single-node OpenShift deployments.

You can now configure OS-level tuning for nodes in a hosted cluster by using the Node Tuning Operator. To configure node tuning, you can create config maps in the management cluster that contain Tuned objects, and reference those config maps in your node pools. The tuning configuration that is definied in the Tuned objects is applied to the nodes in the node pool. For more information, see Configuring node tuning in a hosted cluster.

The kernel module management (KMM) Operator replaces the Special Resource Operator (SRO). KMM includes the following features for connected environments only:

For hub and spoke deployments in an environment that can access the internet, you can use the kernel module management (KMM) Operator deployed in the hub cluster to manage the deployment of the required kernel modules to one or more managed clusters.

Topology Aware Lifecycle Manager (TALM) now provides more detailed status information and messages, and redesigned conditions. You can use the ClusterLabelSelector field for greater flexibility in selecting clusters for update. You can use timeout settings to determine what happens if an update fails for a cluster, for example, skipping the failing cluster and continuing to upgrade other clusters, or stopping policy remediation for all clusters. For more information see Topology Aware Lifecycle Manager for cluster updates.

Encapsulation is the process of moving all Kubernetes-specific mount points to an alternative namespace to reduce the visibility and performance impact of a large number of mount points in the default namespace. Previously, mount namespace encapsulation has been deployed transparently in OpenShift Container Platform specifically for Distributed Units (DUs) installed using GitOps ZTP. In OpenShift Container Platform v4.12, this functionality is now available as a configurable option.

A standard host operating system uses systemd to constantly scan all mount namespaces: both the standard Linux mounts and the numerous mounts that Kubernetes uses to operate. The current implementation of Kubelet and CRI-O both use the top-level namespace for all container and Kubelet mount points. Encapsulating these container-specific mount points in a private namespace reduces systemd overhead and enhances CPU performance. Encapsulation can also improve security, by storing Kubernetes-specific mount points in a location safe from inspection by unprivileged users.

For more information, see Optimizing CPU usage with mount namespace encapsulation.

You can configure the workload partitioning CPU set in single-node OpenShift clusters that you deploy with GitOps ZTP. To do this, you specify cluster management CPU resources with the cpuset field of the SiteConfig custom resource (CR) and the reserved field of the group PolicyGenTemplate CR. The value that you set for cpuset should match the value set in the cluster PerformanceProfile CR .spec.cpu.reserved field for workload partitioning.

For more information, see Workload partitioning.

Hub template functions are now available for use with GitOps ZTP using Red Hat Advanced Cluster Management (RHACM) and Topology Aware Lifecycle Manager (TALM). Hub-side cluster templates reduce the need to create separate policies for many clusters with similiar configurations but with different values. For more information, see Using hub templates in PolicyGenTemplate CRs.

RHACM uses SiteConfig CRs to generate the Day 1 managed cluster installation CRs for ArgoCD. Each ArgoCD application can manage a maximum of 300 SiteConfig CRs. For more information, see Configuring the hub cluster with ArgoCD.

In GitOps ZTP v4.11+, a default policy compliance evaluation timeout value is available for use in PolicyGenTemplate custom resources (CRs). This value specifies how long the related ConfigurationPolicy CR can be in a state of policy compliance or non-compliance before RHACM re-evaluates the applied cluster policies.

Optionally, you can now override the default evaluation intervals for all policies in PolicyGenTemplate CRs.

For more information, see Configuring policy compliance evaluation timeouts for PolicyGenTemplate CRs.

The Assisted Installer currently supports the following OpenShift Container Platform platforms:

Single-node OpenShift does not support VSphere.

This release supports the use of unauthenticated registries when configuring the hub cluster. Registries that do not require authentication are listed under spec.unauthenticatedRegistries in the AgentServiceConfig resource. Any registry on this list is not required to have an entry in the pull secret used for the spoke cluster installation. assisted-service validates the pull secret by making sure it contains the authentication information for every image registry used for installation.

For more information, see Configuring the hub cluster to use unauthenticated registries.

For disconnected installations using GitOps ZTP, if you are deploying OpenShift Container Platform version 4.11 or earlier to a spoke cluster with converged flow enabled, you must mirror the default Ironic agent image to the local image repository. The default Ironic agent images are the following:

For more information about mirroring images, see Mirroring the OpenShift Container Platform image repository.

OpenShift Container Platform now supports specifying kernel arguments for the Discovery ISO in GitOps ZTP deployments. In both manual and automated GitOps ZTP deployments, the Discovery ISO is part of the OpenShift Container Platform installation process on managed bare-metal hosts. You can now edit the InfraEnv resource to specify kernel arguments for the Discovery ISO. This is useful for cluster installations with specific environmental requirements. For example, you can define the rd.net.timeout.carrier kernel argument to help configure the cluster for static networking.

For more information about how to specify kernel arguments, see Configuring kernel arguments for the Discovery ISO by using GitOps ZTP and Configuring kernel arguments for the Discovery ISO for manual installations by using GitOps ZTP.

With this update, you can create OpenShift Container Platform mixed-architecture clusters, also known as heterogeneous clusters, that feature hosts with both AMD64 and AArch64 CPU architectures. You can deploy a heterogeneous spoke cluster from a hub cluster managed by Red Hat Advanced Cluster Management (RHACM). To create a heterogeneous spoke cluster, add an AArch64 worker node to a deployed AMD64 cluster.

To add an AArch64 worker node to a deployed AMD64 cluster, you can specify the AArch64 architecture, the multi-architecture release image, and the operating system required for the node by using an InfraEnv custom resource (CR). You can then provision the AArch64 worker node to the AMD64 cluster by using the Assisted Installer API and the InfraEnv CR.

HTTP is now the default transport in the PTP and bare-metal events infrastructure. AMQ Interconnect is end of life (EOL) from 30 June 2024.

For more information, see About the PTP fast event notifications framework.

In OpenShift Container Platform 4.12, active Insights recommendations are now presented to the user as alerts. You can view and configure these alerts with Alertmanager.

In OpenShift Container Platform 4.12, the Insights Operator now collects the following metrics:

You can now specify application credentials in the clouds.yaml files of clusters that run on Red Hat OpenStack Platform (RHOSP). Application credentials are an alternative to embedding user account details in configuration files. As an example, see the following section of a clouds.yaml file that includes user account details:

Compare that section to one that uses application credentials:

To use application credentials with your cluster as a RHOSP administrator, create the credentials. Then, use them in a clouds.yaml file when you install a cluster. Alternatively, you can create the clouds.yaml file and rotate it into an existing cluster.

The default version for the hypershift.openshift.io API, which is the API for hosted control planes on OpenShift Container Platform, is now v1beta1. Currently, for an existing cluster, the move from alpha to beta is not supported.

With each major, minor, or patch version release of OpenShift Container Platform, the HyperShift Operator is released. The HyperShift command-line interface (CLI) is released as part of each HyperShift Operator release.

The HostedCluster and NodePool API resources are available in the beta version of the API and follow a similar policy to OpenShift Container Platform and Kubernetes.

If you use hosted control planes on OpenShift Container Platform, you can back up and restore etcd by taking a snapshot of etcd and uploading it to a location where you can retrieve it later, such as an S3 bucket. Later, if needed, you can restore the snapshot. For more information, see Backing up and restoring etcd on a hosted cluster.

In a situation where you need disaster recovery for a hosted cluster, you can recover the hosted cluster to the same region within AWS. For more information, see Disaster recovery for a hosted cluster within an AWS region.

Red Hat Virtualization (RHV) will be deprecated in an upcoming release of OpenShift Container Platform. Support for OpenShift Container Platform on RHV will be removed from a future OpenShift Container Platform release, currently planned as OpenShift Container Platform 4.14.

CoreDNS will stop supporting wildcard DNS queries for names under the cluster.local domain. These queries will resolve in OpenShift Container Platform 4.12 as they do in earlier versions, but support will be removed from a future OpenShift Container Platform release.

In OpenShift Container Platform 4.12, support for RHCOS functionality is deprecated for:

While these hardware models remain fully supported in OpenShift Container Platform 4.12, Red Hat recommends that you use later hardware models.

In OpenShift Container Platform 4.12, support for Kuryr on clusters that run on RHOSP is deprecated. Support will be removed no earlier than OpenShift Container Platform 4.14.

Kubernetes 1.25 removed the following deprecated APIs, so you must migrate manifests and API clients to use the appropriate API version. For more information about migrating removed APIs, see the Kubernetes documentation.

The --registry-config and --to option options for the oc registry login command now stop accepting empty files. These options continue to work with files that do not exist. The ability to write output to - (stdout) is also removed.

Support for using Red Hat Enterprise Linux (RHEL) 7 with the OpenShift CLI (oc) has been removed. If you use the OpenShift CLI (oc) with RHEL, you must use RHEL 8 or later.

The following OpenShift CLI (oc) commands were removed with this release:

The Grafana component is no longer a part of the OpenShift Container Platform 4.12 monitoring stack. As an alternative, go to Observe → Dashboards in the OpenShift Container Platform web console to view monitoring dashboards.

Access to the third-party Prometheus and Grafana user interfaces have been removed from the OpenShift Container Platform 4.12 monitoring stack. As an alternative, click Observe in the OpenShift Container Platform web console to view alerting, metrics, dashboards, and metrics targets for monitoring components.

In OpenShift Container Platform 4.11, support for virtual hardware version 13 is removed. Support for virtual hardware version 13 was deprecated in OpenShift Container Platform 4.9. Red Hat recommends that you use virtual hardware version 15 or later.

In OpenShift Container Platform 4.11, support for snapshot.storage.k8s.io/v1beta1 API endpoint is removed. Support for snapshot.storage.k8s.io/v1beta1 API endpoint was deprecated in OpenShift Container Platform 4.7. Red Hat recommends that you use snapshot.storage.k8s.io/v1. All objects created as v1beta1 are available through the v1 endpoint.

Support for deploying custom schedulers manually has been removed with this release. Use the Secondary Scheduler Operator for Red Hat OpenShift instead to deploy a custom secondary scheduler in OpenShift Container Platform.

Support for deploying single-node OpenShift clusters with OpenShiftSDN has been removed with this release. OVN-Kubernetes is the default networking solution for single-node OpenShift deployments.