OpenShift Container Platform 4.15 release notes

See Identifying pod security violations for information about how to find which workloads are causing pod security violations.

See About pod security admission synchronization to understand when pod security admission label synchronization is performed. Pod security admission labels are not synchronized in certain situations, such as the following situations:

If necessary, you can set a custom admission profile on the namespace or pod by setting the pod-security.kubernetes.io/enforce label.

[WARNING] (50) : [/usr/sbin/haproxy.main()] Cannot raise FD limit to 4000237, limit is 1048576.

[ALERT] (50) : [/usr/sbin/haproxy.main()] FD limit (1048576) too low for maxconn=2000000/maxsock=4000237. Please raise 'ulimit-n' to 4000237 or more to avoid any trouble.

General Availability

Deprecated

Removed

Previously, the termination.log in the kube-apiserver log folder had invalid permissions due to settings in the upstream library. With this release, the upstream library was updated and the terminate.log now has the expected permissions. (OCPBUGS-11856)

Previously, the Cluster Version Operator (CVO) enabled a capability if the existing manifest got the capability annotation after an upgrade. This caused the console to be enabled after upgrading to OpenShift Container Platform 4.14 for users who had previously disabled the console capability. With this release, the unnecessary console capability was removed from the existing manifest and the console capability is no longer implicitly enabled. (OCPBUGS-20331)

Previously, when the openshift-kube-controller-manager namespace was deleted, the following error was logged repeatedly: failed to synchronize namespace. With this release, the error is no longer logged when the openshift-kube-controller-manager namespace is deleted. (OCPBUGS-17458)

Previously, deploying IPv6-only hosts from a dual-stack GitOps ZTP hub prevented the correct callback URL from being passed to the baseboard management controller (BMC). Consequently, an IPv4 URL was passed unconditionally. This issue has been resolved, and the IP version of the URL now depends on the IP version of the BMC address. (OCPBUGS-23759)

Previously, the Bare Metal Operator (BMO) container had a hostPort specified as 60000, but the hostPort was not actually in use despite the specification. As a result, other services could not use port 60000. This fix removes the hostPort specification from the container configuration. Now, port 60000 is available for use by other services. (OCPBUGS-18788)

Previously, the Cluster Baremetal Operator (CBO) failed when it checked the infrastructure platformStatus field and returned nil. With OpenShift Container Platform 4.15, the CBO has been updated so that it checks and returns a blank value when apiServerInternalIPs returns nil, which resolves this issue. (OCPBUGS-17589)

Previously, the inspector.ipxe configuration used the IRONIC_IP variable, which did not account for IPv6 addresses because they have brackets. Consequently, when the user supplied an incorrect boot_mac_address, iPXE fell back to the inspector.ipxe configuration, which supplied a malformed IPv6 host header since it did not contain brackets.

Previously, there was a bug when attempting to deploy OpenShift Container Platform on a new bare metal host using RedFish Virtual Media with Cisco UCS hardware. This bug blocked bare metal hosts from new provisions, because Ironic was unable to find a suitable virtual media device. With this update, Ironic does more checks in all available virtual media devices. As a result, Cisco UCS hardware can now be provisioned when using RedFish Virtual Media. (OCPBUGS-23105)

Previously, when installing OpenShift Container Platform with the bootMode field set to UEFISecureBoot on a node where the secureBoot field was set to disabled, the installation program failed to start. With this update, Ironic has been updated so that you can install OpenShift Container Platform with secureBoot set to enabled. (OCPBUGS-9303)

Previously, timestamps were not preserved when copying contents between containers. With this release, the -p flag is added to the cp command to allow timestamps to be preserved. (OCPBUGS-22497)

Previously, an error in the parsing of taints from the MachineSet spec meant that the autoscaler could not account for any taint set directly on the spec. Consequently, when relying on the MachineSet taints for scaling from zero, the taints from the spec were not considered, which could cause incorrect scaling decisions. With this update, parsing issues within the scale from zero logic have been resolved. As a result, auto scaler can now scale up correctly and identify taints that would prevent workloads from scheduling. (OCPBUGS-27750)

Previously, an Amazon Web Services (AWS) code that provided image credentials was removed from the kubelet in OpenShift Container Platform 4.14. Consequently, pulling images from Amazon Elastic Container Registry (ECR) failed without a specified pull secret, because the kubelet could no longer authenticate itself and pass credentials to the container runtime. With this update, a separate credential provider has been configured, which is now responsible for providing ECR credentials for the kubelet. As a result, the kubelet can now pull private images from ECR. (OCPBUGS-27486)

Previously when deploying a hosted control plane (HCP) KubeVirt cluster the --node-selector command, the node selector was not applied to the kubevirt-cloud-controller-manager pods within the HCP namespace. Consequentially, you could not pin the entire HCP pods to specific nodes. With this update, this issue has been fixed. (OCPBUGS-27071)

Previously, the default virtual machine (VM) type for the Microsoft Azure load balancer was changed from Standard to VMSS. Consequently, the service type load balancer could not attach standard VMs to load balancers. This update reverts these changes to the previous configuration to maintain compatibility with OpenShift Container Platform deployments. As a result, load balancer attachments are now more consistent. (OCPBUGS-26210)

Previously, deployments on RHOSP nodes with additional ports with the enable_port_security field set to false were prevented from creating LoadBalancer services. With this update, this issue is resolved. (OCPBUGS-22246)

Previously worker nodes on Red Hat OpenStack Platform (RHOSP) were named with domain components if the Nova metadata service was unavailable the first time the worker nodes booted. OpenShift Container Platform expects the node names to be the same as the Nova instance. The name discrepancy caused the nodes' certificate request to be rejected and the nodes could not join the cluster. With this update, the worker nodes will wait and retry the metadata service indefinitely on first boot ensuring the nodes are correctly named. (OCPBUGS-22200)

Previously, the cluster autoscaler crashed when used with nodes that have Container Storage Interface (CSI) storage. The issue is resolved in this release. (OCPBUGS-23096)

Previously, in certain proxied environments, the Amazon Web Services (AWS) metadata service might not have been present on initial startup, and might have only been available shortly after startup. The kubelet hostname fetching did not account for this delay and, consequently, the node would fail to boot because it would not have a valid hostname. This update ensures that the hostname fetching script retries on failure for some time. As a result, inaccessibility of the metadata service is tolerated for a short period of time. (OCPBUGS-20369)

In OpenShift Container Platform version 4.14 and later, there is a known issue that causes installation of Microsoft Azure Stack Hub to fail. Microsoft Azure Stack Hub clusters that are upgraded to 4.14 or later might encounter load balancer configuration issues as nodes scale up or down. Installing or upgrading to 4.14 in Microsoft Azure Stack Hub environments is not recommended until this issue is resolved. (OCPBUGS-20213)

Previously, some conditions during the startup process of the Cluster Autoscaler Operator caused a lock that prevented the Operator from successfully starting and marking itself available. As a result, the cluster became degraded. The issue is resolved with this release. (OCPBUGS-18954)

Previously, attempting to perform a Google Cloud Platform XPN internal cluster installation failed when control nodes were added to a second internal instance group. This bug has been fixed. (OCPBUGS-5755)

Previously, the termination handler prematurely exited before marking a node for termination. This condition occurred based on the timing of when the termination signal was received by the controller. With this release, the possibility of early termination is accounted for by introducing an additional check for termination. (OCPBUGS-2117)

Previously, when the Build cluster capability was not enabled, the cluster version Operator (CVO) failed to synchronize the build informer, and did not start successfully. With this release, the CVO successfully starts when the Build capability is not enabled. (OCPBUGS-22956)

Previously, the Cloud Credential Operator utility (ccoctl) created custom GCP roles at the cluster level, so each cluster contributed to the quota limit on the number of allowed custom roles. Because of GCP deletion policies, deleted custom roles continue to contribute to the quota limit for many days after they are deleted. With this release, custom roles are added at the project level instead of the cluster level to reduce the total number of custom roles created. Additionally, an option to clean up custom roles is now available when deleting the GCP resources that the ccoctl utility creates during installation. These changes can help avoid reaching the quota limit on the number of allowed custom roles. (OCPBUGS-28850)

Previously, when the Build cluster capability was not enabled, the Cluster Version Operator (CVO) failed to synchronize the build informer and did not start successfully. With this release, the CVO successfully starts when the Build capability is not enabled. (OCPBUGS-26510)

Previously, buckets created by running the ccoctl azure create command were prohibited from allowing public blob access due to a change in the default behavior of Microsoft Azure buckets. With this release, buckets created by running the ccoctl azure create command are explicitly set to allow public blob access. (OCPBUGS-22369)

Previously, an Azure Managed Identity role was omitted from the Cloud Controller Manager service account. As a result, the Cloud Controller Manager could not manage service type load balancers in environments deployed to existing VNets with a private publishing method. With this release, the missing role was added to the Cloud Credential Operator utility (ccoctl) and Azure Managed Identity installations into an existing VNet with private publishing is possible. (OCPBUGS-21745)

Previously, the Cloud Credential Operator did not support updating the vCenter server value in the root secret vshpere-creds that is stored in the kube-system namespace. As a result, attempting to update this value caused both the old and new values to exist because the component secrets were not synchronized correctly. With this release, the Cloud Credential Operator resets the secret data during synchronization so that updating the vCenter server value is supported. (OCPBUGS-20478)

Previously, the Cloud Credential Operator utility (ccoctl) failed to create AWS Security Token Service (STS) resources in China regions because the China region DNS suffix .amazonaws.com.cn differs from the suffix .amazonaws.com that is used in other regions. With this release, ccoctl can detect the correct DNS suffix and use it to create the required resources. (OCPBUGS-13597)

The Cluster Version Operator (CVO) continually retrieves update recommendations and evaluates known conditional update risks against the current cluster state. Previously, failing risk evaluations blocked the CVO from fetching new update recommendations. When the risk evaluations were failing because the update recommendation service served a poorly-defined update risk, this issue could prevent the CVO from noticing the update recommendation service serving an improved risk declaration. With this release, the CVO continues to poll the update recommendation service regardless of whether update risks are successfully evaluated or not. (OCPBUGS-25949)

Previously, BuildRun logs were not visible in the Logs Tab of the BuildRun due to the recent update in the API version of the specified resources. With this update, the Logs of the TaskRuns were added back into the Logs tab of the BuildRun for both v1alpha1 and v1beta1 versions of the builds Operator. (OCPBUGS-29283)

Previously, the console UI failed when a Task in the Pipeline Builder that was previously installed from the ArtifactHub was selected and an error page displayed. With this update, the console UI no longer expects optional data and the console UI no longer fails. (OCPBUGS-24001)

Previously, the Edit Build and BuildRun options in the Actions menu of the Shipwright Plugin did not allow you to edit in the YAML tab. With this update, you can edit in the YAML tab. (OCPBUGS-23164)

Previously, the console searched only for the file name Dockerfile in a repository to identify the repository suitable for the Container strategy in the Import Flows. Since other containerization tools are available, support for the Containerfile file name is now suitable for the Container strategy. (OCPBUGS-22976)

Previously, when an unauthorized user opened a link to the console that contains path and query parameters, and they were redirected to a login page, the query parameters did not restore after the login was successful. As a result, the user needed to restore the search or click the link to the console again. With this update, the latest version saves and restores the query parameters similar to the path. (OCPBUGS-22199)

Previously, when navigating to the Create Channel page from the Add or Topology view, the default name as Channel is present, but the Create button is disabled with Required showing under the name field. With this update, if the default channel name is added then the Required message will not display when clicking the Create button. (OCPBUGS-19783)

Previously, there were similar options to choose from when using the quick search function. With this update, the Source-to-image option is differentiated from the Samples option in the Topology quick search. (OCPBUGS-18371)

Previously, when {serverless-product-name} Operator was installed and the Knative (Kn) serving instance had not been created, then when navigating to the Global configuration page from Administration → Cluster Settings and clicking Knative-serving a 404 page not found error was displayed. With this update, before adding Knative-serving to the Global configuration, a check is in place to determine if a Knative serving instance is created. (OCPBUGS-18267)

Previously there was an issue with the Edit Knative Service form that prevented users from editing the Knative service they previously created. With this update, you can edit a Knative service that was previously created. (OCPBUGS-6513)

Previously, the cluster-backup.sh script cached the etcdctl binary on the local machine indefinitely, making updates impossible. With this update, the cluster-backup.sh script pulls the latest etcdctl binary each time it is run. (OCPBUGS-19052)

Previously, when using a custom Container Network Interface (CNI) plugin in a hosted cluster, role-based access control (RBAC) rules were configured only when you set the hostedcluster.spec.networking.networkType field to Calico. Role-based access control (RBAC) rules were not configured when you set the hostedcluster.spec.networking.networkType field to Other. With this release, RBAC rules are configured properly, when you set the hostedcluster.spec.networking.networkType field to Other. (OCPBUGS-28235)

Previously, a node port failed to expose properly because the ipFamilyPolicy field was set to SingleStack for the kube-apiserver resource. With this update, if the ipFamilyPolicy is set to PreferredDualStack, node port is exposed properly. (OCPBUGS-23350)

Previously, after configuring the Open Virtual Network (OVN) for a hosted cluster, the cloud-network-config-controller, multus-admission-controller, and`ovnkube-control-plane` resources were missing the hypershift.openshift.io/hosted-control-plane:{hostedcluster resource namespace}-{cluster-name} label. With this update, after configuring the Open Virtual Network (OVN) for a hosted cluster, the cloud-network-config-controller, multus-admission-controller, and`ovnkube-control-plane` resources contain the hypershift.openshift.io/hosted-control-plane:{hostedcluster resource namespace}-{cluster-name} label. (OCPBUGS-19370)

Previously, after creating a hosted cluster, to create a config map, if you used a name other than user-ca-bundle, the deployment if the Control Plane Operator (CPO) failed. With this update, you can use unique names to create a config map. The CPO is deployed successfully. (OCPBUGS-19419)

Previously, hosted clusters with .status.controlPlaneEndpoint.port: 443 would mistakenly expose port 6443 for public and private routers. With this update, hosted clusters with .status.controlPlaneEndpoint.port: 443 only expose the port 443. (OCPBUGS-20161)

Previously, if the Kube API server is exposed by using IPv4 and IPv6, and the IP address is set in the HostedCluster resource, the IPv6 environment did not work properly. With this update, when the Kube API server is exposed by using IPv4 and IPv6, the IPv6 environment works properly. (OCPBUGS-20246)

Previously, if the console Operator and Ingress pods were located on the same node, the console Operator would fail and mark the console cluster Operator as unavailable. With this release, if the console Operator and Ingress pods are located on the same node, the console Operator no longer fails. (OCPBUGS-23300)

Previously, if uninstallation of a hosted cluster is stuck, status of the Control Plane Operator (CPO) was reported incorrectly. With this update, the status of the CPO is reported correctly. (OCPBUGS-26412)

Previously, if you tried to override the OpenShift Container Platform version while the initial upgrade was in progress, a hosted cluster upgrade would fail. With this update, if you override the current upgrade with a new OpenShift Container Platform version, the upgrade completes successfully. (OCPBUGS-18122)

Previously, if you update the pull secret for the hosted control planes, it did not reflect on the worker nodes immediately. With this update, when you change the pull secret, reconciliation is triggered and worker nodes are updated with a new pull secret immediately. (OCPBUGS-19834)

Previously, the Hypershift Operator would report time series for node pools that no longer existed. With this release, the Hypershift Operator reports time series for node pools correctly. (OCPBUGS-20179)

Previously, the --enable-uwm-telemetry-remote-write flag was enabled by default. This setting blocked the telemetry reconciliation. With this update, you can disable the --enable-uwm-telemetry-remote-write flag to allow telemetry reconciliation. (OCPBUGS-26410)

Previously, the control Plane Operator (CPO) failed to update the VPC endpoint service when an IAM role path ARN was provided as the additional allowed principal: arn:aws:iam::${ACCOUNT_ID}:role/${PATH}/name With this update, The CPO updates the VPC endpoint service with the arn:aws:iam::${ACCOUNT_ID}:role/${PATH}/name allowed principal successfully. (OCPBUGS-23511)

Previously, to customize OAuth templates, if you configured the HostedCluster.spec.configuration.oauth field, this setting did not reflect in a hosted cluster. With this update, you can configure the HostedCluster.spec.configuration.oauth field in a hosted cluster successfully. (OCPBUGS-15215)

Previously, when deploying a hosted cluster by using a dual stack networking, by default, the clusterIP field was set to an IPv6 network instead of an IPv4 network. With this update, when deploying a hosted cluster by using a dual stack networking, the clusterIP field is set to IPv4 network by default. (OCPBUGS-16189)

Previously, when deploying a hosted cluster, if you configure the advertiseAddress field in the HostedCluster resource, the hosted cluster deployment would fail. With this release, you can deploy a hosted cluster successfully after configuring the advertiseAddress field in the HostedCluster resource. (OCPBUGS-19746)

Previously, when you set the hostedcluster.spec.networking.networkType field to Calico in a hosted cluster, the Cluster Network Operator did not have enough role-based access control (RBAC) permissions to deploy the network-node-identity resource. With this update, the network-node-identity resource is deployed successfully. (OCPBUGS-23083)

Previously, you could not update the default configuration for audit logs in a hosted cluster. Therefore, components of a hosted cluster could not generate audit logs. With this update, you can generate audit logs for components of a hosted cluster by updating the default configuration. (OCPBUGS-13348)

Previously, the Image Registry pruner relied on a cluster role that was managed by the OpenShift API server. This could cause the pruner job to intermittently fail during an upgrade. Now, the Image Registry Operator is responsible for creating the pruner cluster role, which resolves the issue. (OCPBUGS-18969)

The Image Registry Operator makes API calls to the storage account list endpoint as part of obtaining access keys. In projects with several OpenShift Container Platform clusters, this might lead to API limits being reached. As a result, 429 errors were returned when attempting to create new clusters. With this update, the time between calls has been increased from 5 minutes to 20 minutes, and API limits are no longer reached. (OCPBUGS-18469)

Previously, the default low settings for QPS and Burst caused the image registry to return with a gateway timeout error when API server requests were not returned in an appropriate time. To resolve this issue, users had to restart the image registry. With this update, the default settings for QPS and Burst have been increased, and this issue no longer occurs. (OCPBUGS-18999)

Previously, when creating the deployment resource for the Cluster Image Registry Operator, error handling used a pointer variable without checking if the value was nil first. Consequently, when the pointer value was nil, a panic was reported in the logs. With this update, a nil check was added so that the panic is no longer reported in the logs. (OCPBUGS-18103)

Previously, the OpenShift Container Platform 4.14 release introduced a change that gave users the perception that their images were lost when updating from OpenShift Container Platform version 4.13 to 4.14. A change to the default internal registry caused the registry to use an incorrect path when using the Microsoft Azure object storage. With this release, the correct path is used and a job has been added to the registry operator that moves any blobs pushed to the registry that used the wrong storage path into the correct storage path, which effectively merges the two distinct storage paths into a single path.

Previously, installing a cluster on AWS might fail in some cases due to a validation error. With this update, the installation program produces the necessary cloud configuration object to satisfy the machine config operator. This results in the installation succeeding. (OCPBUGS-12707)

Previously, installing a cluster on GCP using a service account attached to a VM for authentication might fail due to an internal data validation bug. With this release, the installation program has been updated to correctly validate the authentication parameters when using a service account attached to a VM. (OCPBUGS-19376)

Previously, the vSphere connection configuration interface showed the network name instead of the cluster name in the "vCenter cluster" field. With this update, the "vCenter cluster" field has been updated to display the cluster name. (OCPBUGS-23347)

Previously, when you authenticated with the credentialsMode parameter not set to Manual and you used the gcloud cli tool, the installation program retrieved Google Cloud Platform (GCP) credentials from the osServiceAccount.json file. This operation caused the GCP cluster installation to fail. Now, a validation check scans the install-config.yaml file and prompts you with a message if you did not set credentialsMode to Manual. Note that in Manual mode, you must edit the manifests and provide the credentials. (OCPBUGS-17757)

Previously when you attempted to install an OpenShift Container Platform on VMware vSphere by using installer-provisioned infrastructure, a resource pool object would include a double backslash. This format caused the installation program to generate an incorrect path to network resources that in turn caused the installation operation to fail. After the installation program processed this resource pool object, the program outputted a "network not found" error message. Now, the installation program retrieves the cluster object for the purposes of joining the InventoryPath with the network name so that the program specifies the correct path to the resource pool object. (OCPBUGS-23376)

Previously, after installing an Azure Red Hat OpenShift cluster, some cluster Operators were unavailable. This was the result of one of the cluster’s load balancers not being created as part of the installation process. With this update, the load balancer is correctly created. After installing a cluster, all cluster Operators are available. (OCPBUGS-24191)

Previously, if the VMware vSphere cluster contained an ESXi host that was offline, the installation failed with a "panic: runtime error: invalid memory address or nil pointer dereference" message. With this update, the error message states that the ESXi host is unavailable. (OCPBUGS-20350)

Previously, if you only used the default machine configuration to specify existing AWS security groups when installing a cluster on AWS (platform.aws.defaultMachinePlatform.additonalSecurityGroupsIDs), the security groups were not applied to control plane machines. With this update, existing AWS security groups are correctly applied to control planes when they are specified using the default machine configuration. (OCPBUGS-20525)

Previously, installing a cluster on AWS failed when the specified machine instance type (platform.aws.type) did not support the machine architecture that was specified for control plane or compute machines (controlPlane.architecture and compute.architecture). With this update, the installation program now checks to determine if the machine instance type supports the specified architecture and displays an error message if it does not. (OCPBUGS-26051)

Previously, the installation program did not validate some configuration settings before installing the cluster. This behavior occurred when these settings were only specified in the default machine configuration (platform.azure.defaultMachinePlatform). As a result, the installation would succeed even if the following conditions were met:

Previously, when installing an AWS cluster to the Secret Commercial Cloud Services (SC2S) region and specifying existing AWS security groups, the installation failed with an error that stated that the functionality was not available in the region. With this fix, the installation succeeds. (OCPBUGS-18830)

Previously, when you specified Key Management Service (KMS) encryption keys in the kmsKeyARN section of the install-config.yaml configuration file for installing a cluster on Amazon Web Services (AWS), permission roles were not added during the cluster installation operation. With this update, after you specify the keys in the configuration file, an additional set of keys are added to the cluster so that the cluster successfully installs. If you specify the credentialsMode parameter in the configuration file, all KMS encryption keys are ignored. (OCPBUGS-13664)

Previously, Agent-based installations on Oracle® Cloud Infrastructure (OCI) did not show a console displaying installation progress to users, making it more difficult to track installation progress. With this update, Agent-based installations on OCI now display installation progress on the console. (OCPBUGS-19092)

Previously, if static networking was defined in the install-config.yaml or agent-config.yaml files for the Agent-based Installer, and an interface name was over 15 characters long, the network manager did not allow the interface to come up. With this update, interface names longer than 15 characters are truncated and the installation can proceed. (OCPBUGS-18552)

Previously, if the user did not specify the rendezevousIP field in the agent-config.yaml file and hosts were defined in the same file with static network configuration, then the first host was designated as a rendezvous node regardless of its role. This caused the installation to fail. With this update, the Agent-based Installer prioritizes the rendezvous node search by first looking among the hosts with a master role and a static IP defined. If none is found, then a potential candidate is searched for through the hosts that do not have a role defined. Hosts with a static network configuration that are explicitly configured with a worker role are ignored. (OCPBUGS-5471)

Previously, the Agent console application was shown during the boot process of all Agent-based installations, enabling network customizations before proceeding with the installation. Because network configuration is rarely needed during cloud installations, this would unnecessarily slow down installations on Oracle® Cloud Infrastructure (OCI).

Previously, the Agent-based Installer enabled an external Cloud Controller Manager (CCM) by default when the platform was defined as external. This prevented users from disabling the external CCM when performing installations on cloud platforms that do not require one. With this update, users are required to enable an external CCM only when performing an Agent-based installation on Oracle® Cloud Infrastructure (OCI). (OCPBUGS-18455)

Previously, the agent wait-for command failed to record logs in the .openshift_install.log file. With this update, logs are recorded in the .openshift_install.log file when you use the agent wait-for command. (OCPBUGS-5728)

Previously, the assisted-service on the bootstrap machine became unavailable after the bootstrap node rebooted, preventing any communication from the assisted-installer-controller. This stopped the assisted-installer-controller from removing uninitialized taints from worker nodes, causing the cluster installation to hang waiting on cluster Operators.

Previously, the platform type was erroneously required to be lowercase in the AgentClusterInstall cluster manifest used by the Agent-based Installer. With this update, mixed case values are required, but the original lowercase values are now accepted and correctly translated. (OCPBUGS-19444)

Previously, the manila-csi-driver-controller-metrics service had empty endpoints due to an incorrect name for the app selector. With this release the app selector name is changed to openstack-manila-csi and the issue is fixed. (OCPBUGS-9331)

Previously, the assisted installer removed the uninitialized taints for all vSphere nodes which prevented the vSphere CCM from initializing the nodes properly. This caused the vSphere CSI operator to degrade during the initial cluster installation because the node’s provider ID was missing. With this release, the assisted installer checks if vSphere credentials were provided in the install-config.yaml. If credentials were provided, the OpenShift version is greater or equal to 4.15, and the agent installer was used, the assisted-installer and assisted-installer-controller do not remove the uninitialized taints. This means that the node’s providerID and VM’s UUID are properly set and the vSphere CSI operator is installed. (OCPBUGS-29485)

Previously, when the maxSurge field was set for a daemon set and the toleration was updated, pods failed to scale down, which resulted in a failed rollout due to a different set of nodes being used for scheduling. With this release, nodes are properly excluded if scheduling constraints are not met, and rollouts can complete successfully. (OCPBUGS-19452)

Previously, a misspelled environment variable prevented a script from detecting that the node.env file was present. This caused the contents of the node.env file to be overwritten after each boot, and the kubelet hostname could not be changed. With this update, the environment variable spelling is corrected and edits to the node.env file persist across reboots. (OCPBUGS-27307)

Previously, the Machine Config Operator allowed user-provided certificate authority updates to be made without requiring a new machine config to trigger. Because the new write method for these updates was missing a newline character, it caused validation errors for the contents of the CA file on-disk and the Machine Config Daemon became degraded. With this release, the CA file contents are fixed, and updates proceed as expected. (OCPBUGS-25424)

Previously, the Machine Config Operator allowed user-provided certificate authority bundle changes to be applied to the cluster without needing a machine config, to prevent disruption. Because of this, the user-ca bundle was not propagating to applications running on the cluster and required a reboot to see the changes take effect. With this update, the MCO now runs the update-ca-trust command and restarts the CRI-O service so that the new CA properly applies. (OCPBUGS-24035)

Previously, the initial mechanism used by the Machine Config Operator to handle image registry certs would delete and create new config maps rather than patching existing ones. This caused a significant increase in API usage from the MCO. With this update, the mechanism has been updated so that it uses a JSON patch instead, thereby resolving the issue. (OCPBUGS-18800)

Previously, the Machine Config Operator was pulling the baremetalRuntimeCfgImage container image multiple times: the first time to obtain node details and subsequent times to verify that the image is available. This caused issues during certificate rotation in situations where the mirror server or Quay was not available, and subsequent image pulls would fail. However, if the image is already on the nodes due to the first image pull then the nodes should start the kubelet regardless. With this update, the baremetalRuntimeCfgImage image is only pulled one time, thereby resolving the issue. (OCPBUGS-18772)

Previously, the nmstatectl command failed to retrieve the correct permanent MAC address during OpenShift Container Platform updates for some network environments. This caused the interface to be renamed and the bond connection on the node to break during the update. With this release, patches were applied to the nmstate package and MCO to prevent renaming, and updates proceed as expected. (OCPBUGS-17877)

Previously, the Machine Config Operator became the default provider of image registry certificates and the node-ca daemon was removed. This caused issues with the HyperShift Operator, because removing the node-ca daemon also removed the image registry path in the Machine Config Server (MCS), which HyperShift uses to get the Ignition configuration and start the bootstrap process. With this update, a flag containing the MCS image registry data is provided, which Ignition can use during the bootstrap process, thereby resolving the issue. (OCPBUGS-17811)

Previously, older RHCOS boot images contained a race condition between services on boot that prevented the node from running the rhcos-growpart command before it pulled images, preventing the node from starting up. This caused node scaling to sometimes fail on clusters that use old boot images because it was determined there was no room left on the disk. With this update, processes were added to the Machine Config Operator for stricter ordering of services so that nodes boot correctly.

Previously, the Machine Config Operator (MCO) leveraged the oc image extract command to pull images during updates but the ImageContentSourcePolicy (ICSP) object was not respected when pulling those images. With this update, the MCO now uses the podman pull command internally and images are pulled from the location as configured in the ICSP. (OCPBUGS-13044)

Previously, the Expand PVC modal assumed the existing PVC had a spec.resources.requests.storage value that includes a unit. As a result, when the Expand PVC modal was used to expand a PVC that had a requests.storage value without a unit, the console would display an incorrect value in the modal. With this update, the console was updated to handle storage values with and without a unit. (OCPBUGS-27909)

Previously, the console check to determine if a file is binary was not robust enough. As a result, XML files were misidentified as binary and not displaying in the console. With this update, an additional check was added to more precisely check if a file is binary. (OCPBUGS-26591)

Previously, the Node Overview page failed to render when a MachineHealthCheck without spec.unhealthyConditions existed on a cluster. With this update, the Node Overview page was updated to allow for MachineHealthChecks without spec.unhealthyConditions. Now, the Node Overview page renders even if MachineHealthChecks without spec.unhealthyConditions are present on the cluster. (OCPBUGS-25140)

Previously, the console was not up-to-date with the newest matchers key for alert notification receivers, and the alert manager receivers created by the console utilized the older match key. With this update, the console uses matchers instead, and converts any existing match instances to matchers when modifying an existing alert manager receiver. (OCPBUGS-23248)

Previously, impersonation access was incorrectly applied. With this update, the console correctly applies impersonation access. (OCPBUGS-23125)

Previously, when the Advanced Cluster Management for Kubernetes (ACM) and multicluster engine for Kubernetes (MCE) Operators are installed and their plugins are enabled, the YAML code Monaco editor failed to load. With this update, optional resource chaining was added to prevent a failed resource call, and the YAML editor no longer fails to load when the ACM and MCE Operators are installed and their plugins enabled. (OCPBUGS-22778)

Previously, the monitoring-plugin component did not start if IPv6 was disabled for a cluster. This release updates the component to support the following internet protocol configurations in a cluster: IPv4 only, IPv6 only, and both IPv4 and IPv6 simultaneously. This change resolves the issue, and the monitoring-plugin component now starts up if the cluster is configured to support only IPv6. (OCPBUGS-21610)

Previously, instances of Alertmanager for core platform monitoring and for user-defined projects could inadvertently become peered during an upgrade. This issue could occur when multiple Alertmanager instances were deployed in the same cluster. This release fixes the issue by adding a --cluster.label flag to Alertmanager that helps to block any traffic that is not intended for the cluster. (OCPBUGS-18707)

Previously, it was not possible to use text-only email templates in an Alertmanager configuration to send text-only email alerts. With this update, you can configure Alertmanager to send text-only email alerts by setting the html field of the email receiver to an empty string. (OCPBUGS-11713)

Previously, when creating an IngressController with an empty spec, the IngressController’s status showed Invalid. However, the route_controller_metrics_routes_per_shard metric would still get created. When the invalid IngressController was deleted, the route_controller_metrics_routes_per_shard metric would not clear, and it would show information for that metric. With this update, metrics are only created for IngressControllers that are admitted, which resolves this issue. (OCPBUGS-3541)

Previously, timeout values larger than what Go programming language could parse were not properly validated. Consequently, timeout values larger than what HAProxy could parse caused issues with HAProxy. With this update, if the timeout specifies a value larger than what can be parsed, it is capped at the maximum that HAProxy can parse. As a result, issues are no longer caused for HAProxy. (OCPBUGS-6959)

Previously, an external neighbor could have its MAC address changed while the cluster was shutting down or hibernating. Although a Gratuitous Address Resolution Protocol (GARP) should notify other neighbors about this change, the cluster would not process the GARP because it was not running. When the cluster was brought back up, that neighbor might not be reachable from the OVN-Kubernetes cluster network because the stale MAC address was being used. This update enables an aging mechanism so that a neighbor’s MAC address is periodically refreshed every 300 seconds. (OCPBUGS-11710)

Previously, when an IngressController was configured with SSL/TLS, but did not have the clientca-configmap finalizer, the Ingress Operator would try to add the finalizer without checking whether the IngressController was marked for deletion. Consequently, if an IngressController was configured with SSL/TLS and was subsequently deleted, the Operator would correctly remove the finalizer. It would then repeatedly, and erroneously, try and fail to update the IngressController to add the finalizer back, resulting in error messages in the Operator’s logs.

Previously, a race condition occurred between the handling of pods that had been scheduled and the pods that had been completed on a node when OVN-Kubernetes started. This condition often occurred when nodes rebooted. Consequently, the same IP was erroneously assigned to multiple pods. This update fixes the race condition so that the same IP is not assigned to multiple pods in those circumstances. (OCPBUGS-16634)

Previously, there was an error that caused a route to be rejected due to a duplicate host claim. When this occurred, the system would mistakenly select the first route it encountered, which was not always the conflicting route. With this update, all routes for the conflicting host are first retrieved and then sorted based on their submission time. This allows the system to accurately determine and select the newest conflicting route. (OCPBUGS-16707)

Previously, when a new ipspec-host pod was started, it would clear or remove the existing XFRM state. Consequently, it would remove existing north-south traffic policies. This issue has been resolved. (OCPBUGS-19817)

Previously, the ovn-k8s-cni-overlay, topology:layer2 NetworkAttachmentDefinition did not work in a hosted pod when using the Kubevirt provider. Consequently, the pod did not start. This issue has been resolved, and pods can now start with an ovn-k8s-cni-overlay NetworkAttachmentDefinition. (OCPBUGS-22869)

Previously, the Azure upstream DNS did not comply with non-EDNS DNS queries because it returned a payload larger than 512 bytes. Because CoreDNS 1.10.1 no longer uses EDNS for upstream queries and only uses EDNS when the original client query uses EDNS, the combination would result in an overflow servfail error when the upstream returned a payload larger than 512 bytes for non-EDNS queries using CoreDNS 1.10.1. Consequently, upgrading from OpenShift Container Platform 4.12 to 4.13 led to some DNS queries failing that previously worked.

Previously, there was a limitation in private Microsoft Azure clusters where secondary IP addresses designated as egress IP addresses lacked outbound connectivity. This meant that pods associated with these IP addresses were unable to access the internet. However, they could still reach external servers within the infrastructure network, which is the intended use case for egress IP addresses. This update enables egress IP addresses for Microsoft Azure clusters, allowing outbound connectivity to be achieved through outbound rules. (OCPBUGS-5491)

Previously, when using multiple NICS, egress IP addresses were not correctly reassigned to the correct egress node when labeled or unlabeled. This bug has been fixed, and egress IP addresses are now reassigned to the correct egress node. (OCPBUGS-18162)

Previously, a new logic introduced for determining where to run the Keepalived process did not consider the ingress VIP or VIPs. As a result, the Keepalived pods might not have ran on ingress nodes, which could break the cluster. With this fix, the logic now includes the ingress VIP or VIPs, and the Keepalived pods should always be available. (OCPBUGS-18771)

Previously on Hypershift clusters, pods were not always being scheduled on separate zones. With this update, the multus-admission-controller deployment now uses a PodAntiAffinity spec for Hypershift to operate in the proper zone. (OCPBUGS-15220)

Previously, a certificate that existed for 10 minutes was used to implement Multus. With this update, a per node certificate is used for the Multus CNI plugin and the certificate’s existence is increased to a 24 hour duration. (OCPBUGS-19861), (OCPBUGS-19859)

Previously, the spec.desiredState.ovn.bridge-mappings API configuration deleted all the external IDs in Open vSwitch (OVS) local tables on each Kubernetes node. As a result, the OVN chassis configuration was deleted, breaking the default cluster network. With this fix, you can use the ovn.bridge-mappings configuration without affecting the OVS configuration. (OCPBUGS-18869)

Previously, if NMEA sentences were lost on their way to the E810 controller, the T-GM would not be able to synchronize the devices in the network synchronization chain. If these conditions were met, the PTP operator reported an error. With this release, a fix is implemented to report 'FREERUN' in case of a loss of the NMEA string. (OCPBUGS-20514)

Previously, pods assigned an IP from the pool created by the Whereabouts CNI plugin persisted in the ContainerCreating state after a node force reboot. With this release, the Whereabouts CNI plugin issue associated with the IP allocation after a node force reboot is resolved. (OCPBUGS-18893)

Previously, when using the assisted installer, OVN-Kubernetes took a long time to bootstrap. This issue occurred because there were three ovnkube-control-plane nodes. The first two started up normally, but the third delayed the installation time. The issue would only resolve after a timeout expiration; afterwards, installation would continue.

Due to how the Machine Config Operator (MCO) handles machine configurations for worker pools and custom pools, the MCO might apply an incorrect cgroup version argument for custom pools. As a consequence, nodes in the custom pool might feature an incorrect cgroup kernel argument that causes unpredictable behavior. As a workaround, specify cgroup version kernel arguments for worker and control plane pools only.(OCPBUGS-19352)

Previously, CRI-O was not configuring the cgroup hierarchy correctly to account for the unique way that crun creates cgroups. As a consequence, disabling the CPU quota with a PerformanceProfile did not work. With this fix, using a PerformanceProfile to disable CPU quota works as expected. (OCPBUGS-20492)

Previously, because of a default setting (container_use_dri_devices, true), containers were unable to use dri devices. With this fix, containers can use dri devices as expected. (OCPBUGS-24042)

Previously, the kubelet was running with the unconfined_service_t SELinux type. As a consequence, all our plugins failed to deploy due to an Selinux denial. With this fix, the kubelet now runs with the kubelet_exec_t SELinux type. As a result, plugins deploy as expected. (OCPBUGS-20022)

Previously, the CRI-O would automatically remove container images on an upgrade. This caused issues in pre-pulling images. With this release, when OpenShift Container Platform performs a minor upgrade, the container images will not be automatically removed and instead are subject to kubelet’s image garbage collection, which will trigger based on disk usage. (OCPBUGS-25228)

Previously, when adding RHCOS machines to an existing cluster using ansible playbooks, machines were installed with openvswitch version 2.7. With this update, RHCOS machines added to existing clusters using ansible playbooks are installed with openvswitch version 3.1. This openvswitch version increases network performance. (OCPBUGS-18595)

Previously, the Tuned profile reports Degraded condition after applying a PerformanceProfile. The generated Tuned profile was trying to set a sysctl value for the default Receive Packet Steering (RPS) mask when it already configured the same value using an /etc/sysctl.d file. Tuned warns about that and the Node Tuning Operator (NTO) treats that as a degradation with the following message The TuneD daemon issued one or more error message(s) when applying the profile profile. TuneD stderr: net.core.rps_default_mask. With this update, the duplication was solved by not setting the default RPS mask using Tuned. The sysctl.d file was left in place as it applies early during boot. (OCPBUGS-25092)

Previously, the Node Tuning Operator (NTO) did not set the UserAgent and used a default one. With this update, the NTO sets the UserAgent appropriately, which makes debugging the cluster easier. (OCPBUGS-19785)

Previously, when the Node Tuning Operator (NTO) pod restarted while there were a large number of CSVs in the cluster, the NTO pod would fail and entered into CrashBackLoop state. With this update, pagination has been added to the list CSVs requests and this avoids the api-server timeout issue that resulted in the CrashBackLoop state. (OCPBUGS-14241)

Previously, to filter operator packages by channel, for example, mirror.operators.catalog.packages.channels, you had to specify the default channel for the package, even if you did not intend to use the packages from that channel. Based on this information, the resulting catalog is considered invalid if the imageSetConfig does not contain the default channel for the package.

Previously, using eus- channels for mirroring in oc-mirror resulted in failure. This was due to the restriction of eus- channels to mirror only even-numbered releases. With this update, oc-mirror can now effectively use eus- channels for mirroring releases. (OCPBUGS-26065)

Previously, while using oc-mirror for mirroring local OCI operator catalogs from a hidden folder resulted in the following error: error: ".hidden_folder/data/publish/latest/catalog-oci/manifest-list/kubebuilder/kube-rbac-proxy@sha256:<SHASUM>" is not a valid image reference: invalid reference format. With this update, the image references are adjusted in the local OCI catalog to prevent any errors during mirroring. (OCPBUGS-25077)

Previously, the OpenShift Container Platform CLI (oc) version was not printed when running the must-gather tool. With this release, the oc version is now listed in the summary section when running must-gather. (OCPBUGS-24199)

Previously, if you ran a command in oc debug. such as oc debug node/worker — sleep 5; exit 1, without attaching to the terminal, a 0 exit code was always returned regardless of the command’s exit code. With this release, the exit code is now properly returned from the command. (OCPBUGS-20342)

Previously, when mirroring, HTTP401 errors were observed due to expired authentication tokens. These errors occurred during the catalog introspection phase or the image mirroring phase. This issue has been fixed for catalog introspection. Additionally, fixing the Network Time Protocol (NTP) resolves the problem seen during the mirroring phase. For more information, see the article on "Access to the requested resource" error when mirroring images. (OCPBUGS-7465)

After you install an Operator, if the catalog becomes unavailable, the subscription for the Operator is updated with a ResolutionFailed status condition. Before this update, when the catalog became available again, the ResolutionFailed status was not cleared. With this update, this status is now cleared from the subscription after the catalog becomes available, as expected. (OCPBUGS-29116)

With this update, OLM performs a best-effort verification that existing custom resources (CRs) are not invalidated when you install an updated custom resource definition (CRD). (OCPBUGS-18948)

Before this update, the install plan for an Operator displayed duplicate values in the clusterSeviceVersionNames field. This update removes the duplicate values. (OCPBUGS-17408)

Before this update, if you created an Operator group with same name as a previously existing cluster role, Operator Lifecycle Manager (OLM) overwrote the cluster role. With this fix, OLM generates a unique cluster role name for every Operator group by using the following syntax:

olm.og.<operator_group_name>.<admin_edit_or_view>-<hash_value>

Previously, if an Operator installation or upgrade took longer than 10 minutes, the operation could fail with the following error:

Bundle unpacking failed. Reason: DeadlineExceeded, Message: Job was active longer than specified deadline

In Operator Lifecycle Manager (OLM), the Catalog Operator was logging many errors regarding missing OperatorGroup objects in namespaces that had no Operators installed. With this fix, if a namespace has no Subscription objects in it, OLM no longer checks if an OperatorGroup object is present in the namespace. (OCPBUGS-25330)

With the security context constraint (SCC) API, users are able to configure security contexts for scheduling workloads on their cluster. Because parts of core OpenShift Container Platform components run as pods that are scheduled on control plane nodes, it is possible to create a SCC that prevents those core components from being properly scheduled in openshift-* namespaces.

Previously, a race condition between udev events and the creation queues associated with physical devices led to some of the queues being configured with the wrong Receive Packet Steering (RPS) mask when they should be reset to zero. This resulted in the RPS mask being configured on the queues of the physical devices, meaning they were using RPS instead of Receive Side Scaling (RSS), which could impact the performance. With this fix, the event was changed to be triggered per queue creation instead of at device creation. This guarantees that no queue will be missing. The queues of all physical devices are now set up with the correct RPS mask which is empty. (OCPBUGS-18662)

Previously, due to differences in setting up a container’s cgroup hierarchy, containers that use the crun OCI runtime along with a PerformanceProfile configuration encountered performance degradation. With this release, when handling a PerformanceProfile request, CRI-O accounts for the differences in crun and correctly configures the CPU quota to ensure performance. (OCPBUGS-20492)

Previously, LVM Storage did not support disabling over-provisioning, and the minimum value for the thinPoolConfig.overprovisionRatio field in the LVMCluster CR was 2. With this release, you can disable over-provisioning by setting the value of the thinPoolConfig.overprovisionRatio field to 1. (OCPBUGS-24396)

Previously, if the LVMCluster CR was created with an invalid device path in the deviceSelector.optionalPaths field, the LVMCluster CR was in Progressing state. With this release, if the deviceSelector.optionalPaths field contains an invalid device path, LVM Storage updates the LVMCluster CR state to Failed. (OCPBUGS-23995)

Previously, the LVM Storage resource pods were preempted while the cluster was congested. With this release, upon updating OpenShift Container Platform, LVM Storage configures the priorityClassName parameter to ensure proper scheduling and preemption behavior while the cluster is congested. (OCPBUGS-23375)

Previously, upon creating the LVMCluster CR, LVM Storage skipped the counting of volume groups. This resulted in the LVMCluster CR moving to Progressing state even when the volume groups were valid. With this release, upon creating the LVMCluster CR, LVM Storage counts all the volume groups, and updates the LVMCluster CR state to Ready if the volume groups are valid. (OCPBUGS-23191)

Previously, if the default device class was not present on all selected nodes, LVM Storage failed to set up the LVMCluster CR. With this release, LVM Storage detects all the default device classes even if the default device class is present only on one of the selected nodes. With this update, you can define the default device class only on one of the selected nodes. (OCPBUGS-23181)

Previously, upon deleting the worker node in the single-node OpenShift (SNO) and worker node topology, the LVMCluster CR still included the configuration of the deleted worker node. This resulted in the LVMCluster CR remaining in Progressing state. With this release, upon deleting the worker node in the SNO and worker node topology, LVM Storage deletes the worker node configuration in the LVMCluster CR, and updates the LVMCluster CR state to Ready. (OCPBUGS-13558)

Previously, CPU limits for the AWS EFS CSI driver container could cause performance degradation of volumes managed by the AWS EFS CSI Driver Operator. With this release, the CPU limits from the AWS EFS CSI driver container have been removed to help prevent potential performance degradation. (OCPBUGS-28645)

Previously, if you used the performancePlus parameter in the Azure Disk CSI driver and provisioned volumes 512 GiB or smaller, you would receive an error from the driver that you need a disk size of at least 512 GiB. With this release, if you use the performancePlus parameter and provision volumes 512 GiB or smaller, the Azure Disk CSI driver automatically resizes volumes to be 513 GiB. (OCPBUGS-17542)

Technology Preview

General Availability

Not Available

Deprecated