OpenShift Container Platform 4.8 release notes

The OpenShift Docs GitHub repository master branch has been renamed to main.

We have begun to progressively replace the terminology of "master" with "control plane". You will notice throughout the documentation that we use both terms, with "master" in parenthesis. For example "…​ the control plane node (also known as the master node)". In a future release, we will update this to be "the control plane node".

GA: General Availability

TP: Technology Preview

DEP: Deprecated

REM: Removed

Previously, the assisted-service container did not wait for postgres to start up and be ready to accept connections. The assisted-service container attempted to establish a database connection, failed, and the assisted-service container failed and restarted. This issue has been fixed by the assisted-service container attempting to connect to the database for up to 10 seconds. If postgres starts and is ready to accept connection within 10 seconds, the assisted-service container connects without going into an error state. If the assisted-service container is unable to connect to postgres within 10 seconds, it goes into an error state, restarts, and tries again. (BZ#1941859)

Previously, Ironic failed to download an image for installation because Ironic uses HTTPS by default and did not have the correct certificate bundle available. This issue is fixed by setting the image download as Insecure to request a transfer without the certificate. (BZ#1953795)

Previously, when using dual-stack networking, worker node host names sometimes did not match the host name that Ironic inspected before deployment. This caused nodes to need manual approval. This has been fixed. (BZ#1955114)

Previously, in UEFI mode, the ironic-python-agent created a UEFI bootloader entry after downloading the RHCOS image. When using an RHCOS image based on RHEL 8.4, the image could fail to boot using this entry. If the entry installed by Ironic was used when booting the image, the boot could fail and output a BIOS error screen. This is fixed by the ironic-python-agent configuring the boot entry based on a CSV file located in the image, instead of using a fixed boot entry. The image boots properly without error. (BZ#1972213)

Previously, a node sometimes selected the incorrect IP version upon startup (IPv6 instead of IPv4, or vice versa). The node would fail to start because it did not receive an IP address. This is fixed by the Cluster Bare Metal Operator passing the IP option to the downloader (ip=dhcp or ip=dhcp6), so this is set correctly at startup and the node starts as expected. (BZ#1946079)

Previously, the image caching mechanism in Ironic was disabled to enable a direct connection to the HTTP server that hosts the virtualmedial iso to prevent local storage issues. Non-standard compliant HTTP clients and redfish implementations caused failures on BMC connections. This has been fixed by reverting to the default Ironic behavior where the virtualmedia iso is cached and served from the Ironic conductor node. Issues caused by non-standard compliant HTTP clients and redfish implementations have been fixed. (BZ#1962905)

Previously, the machine instance state annotation was not set. Consequently, the STATE column was empty. With this update, the machine instance state annotation is now set, and information in the STATE column automatically populates. (BZ#1857008)

Because newer ipmitool packages default to using cipher suite 17, older hardware that does not support cipher suite 17 fails during deployment. When cipher suite 17 is not supported by the hardware, Ironic now uses cipher suite 3 so that deployments on older hardware using ipmitool should succeed. (BZ#1897415)

Previously, in some cases, adoption occurred before the image cache was populated, which resulted in permanent adoption failure, and no retry was attempted. This caused the control plane bare metal hosts to report adoption failed. With this update, adoption of externally provisioned hosts automatically retries after adoption failure until control plane hosts are correctly adopted. (BZ#1905577)

Previously, the Custom Resource (CR) required the Baseboard Management Controller (BMC) details. However, with assisted installers this information was not provided. This update allows the CR to bypass the BMC details when the operator is not creating the nodes. (BZ#1913112)

When provisioning an image to nodes, qemu-image was restricted to 1G of RAM, which could cause the qemu-img to crash. This fix increases the limit to 2G so that qemu-img now completes provisioning reliably. (BZ#1917482)

Because the redfish/v1/SessionService URL requires authentication, Ironic would generate an authentication error when accessing the site. Because there was no functional problem when this error message was reported by Ironic, it has been removed. (BZ#1924816)

For some drives, the partition, for example /dev/sda1, did not have a read-only file. The base device, for example, /dev/sda had this file, however. Therefore, Ironic could not determine that the partition was read-only, which could cause metadata cleaning to fail on that drive. This update ensures that the partition is detected as read-only, and includes an additional check for the base device. As a result, metadata cleaning is not performed on the read-only partition and metadata cleaning no longer fails. (BZ#1935419)

When a Baremetal IPI was deployed with a proxy configured, the internal machine-os image download was directed through the proxy. This corrupted the image and prevented it from being downloaded. This update fixes the internal image traffic to no_proxy, so that the image download no longer uses a proxy. (BZ#1962592)

Previously, bare metal deployments failed if large packet transfers between Ironic and the RAM disk resulted in connection failures. With this update, Ironic queries the RAM disk for information to work around the connection error, allowing deployments to succeed. (BZ#1957976)

Previously, after CVE-2021-3344 was fixed, builds did not automatically mount entitlement keys on the OpenShift Container Platform node. As a result, when the entitlement certificates were stored on the host or node, the fix prevented entitled builds from working seamlessly. The failure to bring in entitlement certificates stored on the host or node was fixed with BZ#1945692 in 4.7.z and BZ#1946363 in 4.6.z; however, those fixes introduced a benign warning message for builds running on Red Hat Enterprise Linux CoreOS (RHCOS) worker nodes. The current release fixes this issue by allowing builds to automatically mount entitlements only on RHEL worker nodes, and avoiding mount attempts on RHCOS worker nodes. Now, there will not be any benign warnings around entitlement mounts when running builds on RHCOS nodes. (BZ#1951084)

Some users pulling images from Docker Hub can encounter the following error:

container image registry lookup failed...​toomanyrequests: You have reached your pull rate limit

Previously, builds did not perform an error check to see if an image push had failed. As a result, builds always logged the Successfully pushed message. Now, builds check if an error has occurred, and only log the Successfully pushed message after an image push has succeeded. (BZ#1947164)

Previously, the documentation and oc explain help text did not convey that the buildArgs field in the BuildConfig object does not support the valueFrom field of its underlying Kubernetes EnvVar type. As a result, users believed it was supported and tried to use it. The current release updates the documentation and help text, so it is more apparent that the BuildConfig object’s buildArgs field does not support the valueFrom field. (BZ#1956826)

When builds interact with image registries, such as pulling base images, intermittent communications issues can produce build failures. The current release increases the number of retries to these interactions. Now, OpenShift Container Platform builds are more resilient when they encounter intermittent communication issues with image registries. (BZ#1937535)

Previously, Cluster Image Registry Operator considered user_domain_name an immutable field and would not modify it after installation. This resulted in a refusal to accept changes to user_domain_name and resulting credentials. This update marks user_domain_name as mutable and does not store it in the image registry config. This allows user_domain_name and all other auth parameters to be modified after installation. (BZ#1937464)

Previously, a proxy update caused a full cluster configuration update, including API server restart, during continuous integration (CI) runs. Consequently, some clusters in the Machine API Operator would time out because of unexpected API server outages. This update separates proxy tests and adds postconditions so that clusters in the Machine API Operator become stable again during CI runs. (BZ#1913341)

Previously, deleting a machine stuck in Insufficient disk space on datastore took longer than expected because there was no distinction between various vCenter task types. With this update, the machine controller deletion procedure checks the vCenter task type, and no longer blocks the deletion of the machine controller. As a result, the machine controller is deleted quickly. (BZ#1918101)

Previously, scaling from zero annotations requeued even if the instance type was missing. Consequently, there were constant requeue and error space messages in the MachineSet controller logs. With this update, users can set the annotation manually if the instance type is not resolved automatically. As a result, scaling from zero for unknown instance types works if users manually provide the annotation. (BZ#1918910)

Previously, HTTP responses were not closed properly by the Machine API termination handler. Consequently, goroutines were leaked in net.http read and write loops, which led to high memory usage. This update ensures that HTTP responses are always closed properly. As a result, memory usage is now stable. (BZ#1934021)

Previously, multiple client sets created inside of the MachineSet controller caused slow startup times, which resulted in pods failing readiness checks in some large clusters. Consequently, the MachineSet controller would get stuck in an endless loop. This update fixes the MachineSet controller so that it uses a single client. As a result, the MachineSet controller behaves as expected. (BZ#1934216)

Previously, instances took longer to boot when an upgrade was performed by the Machine Config Daemon on the first boot. Consequently, worker nodes were stuck in restart loops, and machine healthchecks (MCHs) removed the worker nodes because they did not properly start. With this update, MHCs no longer remove nodes that have not started correctly. Instead, MHCs only remove nodes when explicitly requested. (BZ#1939054)

Previously, a certificate signing request (CSR) approval was delayed for an unknown reason. Consequently, new machines appearing in the cluster during installation were not approved quickly enough, prolonging cluster installation. To mitigate occasional API server unavailability in early installation phases, this update changes the cache resync period from 10 hours to 10 minutes. As a result, control plane machines are now approved faster so that cluster installation is no longer prolonged. (BZ#1940972)

Previously, the default Google Cloud Platform (GCP) image was out of date and referenced a version from the OpenShift Container Platform 4.6 release that did not support newer Ignition versions. Consequently, new machines in clusters that used the default GCP image were not able to boot OpenShift Container Platform 4.7 and later. With this update, the GCP image is updated to match the release version. As a result, new machines can now boot with the default GCP image. (BZ#1954597)

Previously, due to a strict check of the virtual machine’s (VM) ProvisioningState value, the VM would sometimes fail during an existence check. With this update, the check is more lenient so that only deleted machines go into a Failed phase during an existence check. (BZ#1957349)

Previously, if you deleted a control plane machine using oc delete machine in an AWS cluster, the machine was not removed from the load balancers. As a result, the load balancer continued to serve requests to the removed control plane machine. With this fix, when you remove a control plane machine, the load balancer no longer serves requests to the machine. (BZ#1880757)

Previously, when deleting an unreachable machine, the vSphere Virtual Machine Disk (VMDK) that is created for persistent volumes and attached to the node was incorrectly deleted. As a result, the data on the VMDK was unrecoverable. With this fix, the vSphere cloud provider checks for and detaches these disks from the node if the kubelet is not reachable. As a result, you can delete an unreachable machine without losing the VMDK. (BZ#1883993)

Previously, because a generated list of AWS instance types was out of date, some newer Amazon Web Services (AWS) instance types were not able to scale from zero when using the Cluster Autoscaler Operator and machine sets with zero replicas. The list of AWS instance types is now updated to include newer instance types. With this fix, more instance types are available to the Cluster Autoscaler Operator for scaling from zero replicas. (BZ#1896321)

Previously, pod disruption budgets did not drain pods on an unreachable node due to missing upstream eviction API features. As a result, machines on unreachable nodes could take excessive amounts of time to be removed after being deleted. Now, the grace period timeout is changed to 1 second when deleting machines on unreachable nodes. With this fix, the Machine API can successfully drain and delete nodes that are unreachable. (BZ#1905709)

Previously, the Cloud Credential Operator repeated an unsupported platform type: BareMetal warning on bare metal platforms. With this update, bare metal platforms are no longer treated as unknown platforms. As a result, misleading logging messages are reduced. (BZ#1864116)

Previously, a recurring error message stored in the credentialsRequest custom resources (CRs) of the Cloud Credential Operator led to excessive CPU usage and logging in some error scenarios, such as Amazon Web Services (AWS) rate limiting. This update removes the request ID coming back from the cloud provider so that error messages are stored in conditions where users can more easily find them, and eliminates recurring error messages in the credentialsRequest CR. (BZ#1910396)

Previously, both the Cloud Credential Operator (CCO) and the Cluster Version Operator (CVO) reported if the CCO deployment was unhealthy. This resulted in double reporting if there was an issue. With this release, the CCO no longer reports if its deployment is unhealthy. (BZ#1957424)

Previously, the Cluster Version Operator evaluated both the Available and Degraded parameters when setting the cluster_operator_up metric, which caused the ClusterOperatorDown alert to be displayed for Operators with Available=True or Degraded=True, even though Available=True did not match the alert description of "has not been available". With this fix, the Cluster Version Operator now ignores the Degraded parameter when setting the cluster_operator_up metric. (BZ#1834551)

Previously, when Prometheus was installed on the cluster, important platform topology metrics were not available and a CI error would occur if the installer metric that was generated with the invoker was set to "". The possible race condition in which informers were not synced before metrics were served that was causing the error has now been fixed. (BZ#1871303)

Previously, manifests with multiple tolerations for the same key, such as the Cluster Version Operator’s own deployment), would accept only the last entry read and overwrite prior entries. This caused in-cluster tolerations to diverge from the manifest’s listed tolerations. With this update, the Cluster Version Operator now considers tolerations matching when they are completely equal. This allows the Cluster Version Operator to keep all tolerations present in the manifest for the in-cluster resource. (BZ#1941901)

Previously, the Cluster Version Operator did not reconcile env and envFrom for manifests that did not set those properties. This meant the Cluster Version Operator did not properly manage container environments. This update improves the Cluster Version Operator so that it now clears env and envFrom if they are unset in the manifest. This allows the cluster to automatically recover from invalid cluster-admin changes to these properties. (BZ#1951339)

Previously, manifests with multiple tolerations for the same key, such as the cluster-version-operator deployment object, would accept only the last entry read and overwrite prior entries. This caused in-cluster tolerations to diverge from the manifest’s listed tolerations. With this update, the Cluster Version Operator now considers tolerations to match when they are equal. This allows the Cluster Version Operator to keep all tolerations present in the manifest for the in-cluster resource. (BZ#1941901)

Previously, the Cluster Version Operator reported a ClusterOperatorDegraded alert when ClusterOperator resources were degraded for 10 minutes. This alert was sometimes occurred prematurely during installation as resources were still being created. This update changes the 10-minute period to 30 minutes, providing enough time for the installation to progress without premature ClusterOperatorDegraded alerts. (BZ#1957991)

Previously, when users ran a compliance check, NON-COMPLIANT results were given with no indication of required remediation steps for the user to act upon. This release provides an instructions key that allows users to review the steps needed to verify a rule. This allows users and auditors to verify that an Operator is checking a correct value. (BZ#1919367)

Previously, in a web console form that helped users add boot sources to virtualization templates, an explanatory text gave information only for Fedora, regardless of what operating system the template used. This update adds a fix that provides examples that are specific to the template’s operating system so users have relevant guidance. (BZ#1908169)

Previously, in a web console wizard that helped users create virtual machine templates, imprecise language made it unclear whether an operation applied to the template or to a virtual machine. This fix clarifies the description so users can make an informed decision. (BZ#1922063)

Previously, a vague error message in the web console caused unnecessary confusion for some users who tried to add a network interface to a virtual machine they were creating from a template. This update adds detail to the error message so users can troubleshoot the error more easily. (BZ#1924788)

Previously, when you tried to create a virtual machine from a Red Hat Enterprise Linux (RHEL) 6 template in the web console, a pop-up window gave information about how to define the support level, even through RHEL 6 is not supported. This fix changes the text in this window to make it clear that RHEL 6 is not supported. (BZ#1926776)

Previously, a drop-down list in the web console was obscured by button elements, leaving users unable to select certain operating systems when creating a virtual machine. This fix includes an adjustment to the button elements' z-index value, which corrects the error and lets users select any of the available operating systems. (BZ#1930015)

Previously, if you used the web console’s new virtual machine wizard on a cluster with no defined storage classes, the web console got stuck in an infinite loop and crashed. This fix removes the storage class drop-down list in instances where no storage classes are defined. As a result, the web console does not crash. (BZ#1930064)

Previously, text in a button element did not clearly describe the button’s function, which is to remove a VM template from a list of favorites. This fix updates the text to clarify what the button does. (BZ#1937941)

Previously, for virtual machines that have a RerunOnFailure run strategy, stopping the virtual machine resulted in several UI elements becoming unresponsive, preventing users from reading status information or restarting the virtual machines. This update fixes the unresponsive elements so users can use those features. (BZ#1951209)

Previously, for clusters that were configured to have a separate /var partition, querying the file system only returned the size of disks mounted in the root directory, excluding the size of the /var partition. This fix changes how that query runs, and users can now determine the total size of the file system on a cluster. (BZ#1960612)

Previously, the OpenShift Container Storage Operator displayed an error message when the correct storage class was not available. This update removes the error message and disables the Next button until the correct storage class is available. (BZ#1924641)

Previously, when a user clicked the browser’s back button while creating an internal-attached storage cluster, the installation wizard restarted the process. This update fixes the issue. (BZ#1928008)

When you add a node to local volume discovery, you can now see a list of existing nodes, which reduces unnecessary navigation. (BZ#1947311)

Previously, the Create Storage Cluster wizard let you enable an arbiter zone that had an undefined value. The fix in this update filters out undefined values so arbiter zones can be created only with defined values. (BZ#1926798)

Previously, quick start cards were displayed incorrectly in the web console because of inconsistencies in how product titles were spelled and how the registered trade mark symbol was used. In this update, the product names are spelled correctly, and the registered trademark symbol appears consistently in the first card only. (BZ#1931760)

Previously, BZ#1936587 set the global CoreDNS cache max TTL to 900 seconds. Consequently, NXDOMAIN records received from upstream resolvers were cached for 900 seconds. This update explicitly caches negative DNS response records for a maximum of 30 seconds. As a result, resolving NXDOMAINs are no longer cached for 900 seconds. (BZ#1943578)

The fix for BZ#1953097 enabled the CoreDNS bufsize plugin with a size of 1232 bytes. Some primitive DNS resolvers are not capable of receiving DNS response messages over UDP that are greater than 512 bytes. Consequently, some DNS resolvers, such as Go’s internal DNS library, are unable to receive verbose DNS responses from the DNS Operator. This update sets the CoreDNS bufsize to 512 bytes for all servers. As a result, UDP DNS messages are now properly received. (BZ#1966116)

Previously, cluster upstream resolver returned DNS responses that exceeded 512 bytes via UDP. Consequently, coreDNS returned SERVFAIL or other error messages and forced the client to retry over TCP. This update enabled the coreDNS bufisze plugin with a UDP buffer size of 1232 bytes. (BZ#1949361)

Previously, there was a transport leak in the etcd Operator, which caused memory usage to grow over time. The memory leak has been fixed. (BZ#1925586)

Previously, the etcdInsufficientMembers alert fired incorrectly. With this release, the alert is updated to include the pod label in addition to the instance label, so that the alert only fires when quorum is lost. (BZ#1929944)

Previously, the readiness probe was not reporting the correct readiness due to the introduction of SO_REUSEADDR socket options, which caused the etcd pod to show as ready even though the etcd-quorum-guard failed. The readiness probe checks were updated to account for these options, and the etcd readiness probe now properly reflects the readiness of the operand. (BZ#1946607)

Previously, the spec.loglevel field did not set the log-level flag on the etcd operand, so users could not change the etcd log level. Users can now set the log levels as follows:

Previously, following an etcd process, the next process did not start until the relevant ports were released. With the addition of SO_REUSEADDR to this process, the ports can be reused immediately. For more information, see BZ#1927942.

Previously, the etcd-endpoint’s ConfigMap was left empty if the network.Status.ServiceNetwork field was unpopulated. As a result, the etcd Operator failed to scale up. A new feature in OpenShift Container Platform 4.8 allows the etcd Operator to scale up when the network.Status.ServiceNetwork field is unpopulated. (BZ#1902247)

Previously, the image pruner stopped when it failed to delete an image. As a result, when two image pruners were trying to delete an image concurrently, one of them failed with the not found error. With this update, not found errors are ignored, which allows the image pruner to tolerate concurrent deletions. (BZ#1890828)

Previously, a lack of route status inclusion during the Image Registry Operator status assessment meant that the Image Registry Operator was not degraded, even with routes in the degraded state. With this fix, the Image Registry Operator now fetches all configured routes and evaluates their statuses when assessing its own status. With this update, if any of the routes are degraded, the Image Registry Operator reports itself as degraded with an error message. (BZ#1902076)

Previously, an automatically created Docker config secret did not include credentials for integrated internal registry routes. Because no credentials were present for accessing the registry through any of its routes, pods attempting to reach the registry failed due to lack of authentication. This fix includes all configured registry routes in the default Docker credential secret. Now, pods can reach the integrated registry by any of its routes, as credentials now contain an entry for each route. (BZ#1911470)

Previously the image registry was ignoring cluster wide ImageContentSourcePolicy (ICSP) rules. During pull-through, images mirrors were ignored, which caused pull failures in disconnected clusters. With this update, the registry pulls from mirrors if ICSP rules exist for the target repository. As a result, pulling an image from configured mirrors does not fail. (BZ#1918376)

Previously, the Image Registry Operator did not update the .status.readyReplicas field of the config resource, so its value was always 0. With this fix, the Image Registry Operator writes the number of ready image registry replicas from the deployment into the config. Now, this field shows how many image registry replicas are ready. (BZ#1923811)

Azure recommends users use Storage Accounts v2 instead of v1. Under certain security profiles, administrators can force Azure to not accept the creation of v1 Storage Accounts. Because the image registry depends on v1 Storage Accounts, a cluster install would fail in such environments. With this fix, during cluster bootstrap, the Image Registry Operator now attempts to create and use V2 Storage Accounts. Clusters running on v1 continue using V1 Storage Accounts. Installation succeeds and Image Registry Operator now reports Available. (BZ#1929654)

Previously, performance was sometimes slow when importing multiple images from a stream. With this release, the number of concurrent requests to the image registry is increased from five to 50, resulting in improved performance. (BZ#1954715)

Previously, the Insights Operator did not collect Cluster Version Operator (CVO) pods or events in the openshift-cluster-version namespace. As a result, the Insights Operator did not display information about any problems that the CVO might experience, and users could not get diagnostic information about the CVO. The Insights Operator is now updated to gather the CVO pods and events from the openshift-cluster-operator namespace so that issues with the the CVO are reported by the Insights Operator. (BZ#1942271)

Previously, DNSmasq required specifying the prefix length when an IPv6 network was anything other than /64. Consequently, control plane hosts failed to PXE boot. This update includes the subnet prefix length in the DNSmasq configuration. As a result, control plane hosts will now DHCP and PXE boot on IPv6 networks of any prefix length. (BZ#1927068)

When installing to vSphere, the bootstrap machine sometimes did not correctly update the name servers in the /etc/resolv.conf file. As a result, the bootstrap machine could not access the temporary control plane, and installations failed. This fix includes changes that makes finding the right line to update more reliable. With the bootstrap manager now able to access its temporary control plane, installations are able to succeed. (BZ#1967355)

Previously, the installer did not take into account the region where the bootstrap Ignition config should be located when generating its URL. Consequently, the bootstrap machine could not fetch the config from the provided URL because it was incorrect. This update takes the user’s region into account when generating the URL and selects the correct public endpoint. As a result, the installer always generates correct bootstrap Ignition URLs. (BZ#1934123)

Previously, the default version of Azure’s Minimum TLS was 1.0 when creating a storage account. Consequently, policy checks would fail. This update configures the openshift-installer Azure client to set the Minimum TLS version to 1.2 when creating a storage account. As a result, policy checks now pass. (BZ#1943157)

Previously, private clusters deployed using IPI on Azure had an inbound NSG rule that allowed SSH to the bootstrap node. This allowance could trigger Azure’s security policy. With this update, that NSG rule has been removed. (BZ#1943219)

Previously, the installer did not recognize the ap-northeast-3 AWS region. With this update, the installer allows installs to unknown regions that fit the pattern for known partitions, which allows users to create infrastructure in the ap-northeast-3 AWS region. (BZ#1944268)

Previously, on-premise platforms lacked the capability to create internal load balancers. With this update, a check has been added when users create manfiests to ensure that this strategy is only used on cloud platforms such as AWS, Azure, and GCP. (BZ#1953035)

Previously, when naming Google Cloud Platform resources, a filter prevented certain names using the word Google from being used. This update adds a check in the installer on the cluster name that allows some variations of the word Google to be used when setting a name. (BZ#1955336)

Previously, a bare metal installation with installer-provisioned infrastructure required that the installer process was able to communicate with the provisioning network. Now, the installer process can communicate with the virtual IP for the API server. This change enables cases when the provisioning network is not routable and the installer process is run from a remote location, such as Hive for Red Hat OpenStack Platform (RHOSP) or Red Hat Advanced Cluster Management. You might need to adjust your firewall rules to allow communication with TCP ports 6385 and 5050 on virtual IP for the API server. (BZ#1932799)

Previously, when the installation on Red Hat OpenStack Platform (RHOSP) was provided an ID of a subnet that did not exist in the platform.openstack.machinesSubnet field, the openshift-install command produced a SIGSEGV and backtrace. Now, the openshift-install command is revised so that it produces an error like the following message:

FATAL failed to fetch Metadata: failed to load asset "Install Config": platform.openstack.machinesSubnet: Not found: "<network-ID>"

Previously, the installation on Red Hat OpenStack Platform (RHOSP) failed unless the RHOSP HTTPS certificate was imported to the hosting device. Now, a successful installation occurs when the cacert value in cloud.yaml is set to the RHOSP HTTPS certificate. Importing the certificate to the host is no longer required. (BZ#1786314)

Previously, an installation might fail due to inaccurate external network entries in proxy.config.openshift.io. A validation check now identifies these inaccuracies to enable correction. (BZ#1873649)

Previously vague or confusing Terraform component descriptions are now replaced with clearer information. (BZ#1880758)

A previous change to gophercloud/utils introduced a custom HTTP client that used a self-signed certificate. Because this change removed settings from DefaultTransport, including those for proxy environment variables, this caused failures for installations that used both self-signed certificates and proxies. In this update, the custom HTTP client inherits settings from DefaultTransport, so now OpenShift Container Platform can be installed with self-signed certificates and proxies. (BZ#1925216)

Previously, the installer did not take into account defaultMachineSet values in the install config during its validation which caused the installer to fail. This update applies the default values to the install config and starts validating empty fields. (BZ#1903055)

Previously, soft-anti-affinity required the client to set a minimum Nova microversion. Most versions of Ansible OS server module did not automatically require the client to set a minimum. As a result, the soft-anti-affinity commands were likely to fail. This update fixes the use of the Python OpenStack client to set a Nova microversion when dealing with soft-anti-affinity. (BZ#1910067)

Previously, OpenStack UPI playbooks did not tag all resources that were created. Consequently, the openshift-install destroy command failed to properly identify all of the cluster resources and looped over resource deletion until it reached a timeout, which left resources behind. This update adds missing tag instructions to OpenStack UPI playbooks. (BZ#1916593)

Previously, e2e-gcp-upi did not succeed because of a Python package error which resulted in a failure. With this update, you can set the correct Python version, pip version, and the CLOUDSDK_PYTHON for gsutil to resolve the package error. (BZ#1917931)

Previously, pip version 21 did not support installed Python version 2. As a consequence, this lead to an error resolving all dependent packages required to setup the container. In this update, the pip version is fixed to a value less than 21 to avoid the problem. (BZ#1922235)

Previously, the installer collected information about the cloud twice. Consequently, there were double the number of requests to OpenStack API, which created an additional load on the cloud and increased the installation time. This update fixes the issue by collecting information about the cloud before it checks for quota and then reuses the same information for validations. (BZ#1923038)

Previously, when deploying with IPv6 provision network with subnet other than /64, the DNSmasq required specifying the prefix length. Consequently, the hosts failed to PXE boot when using a non-/64 network. This update includes the prefix length in the DNSmasq configuration. As a result, the hosts succeed on DHCP and PXE boot on IPv6 networks of any prefix length. (BZ#1925291)

Previously, the OpenShift Container Platform installer was not reporting IAM permission issues when removing the Shared Subnet tag even though the logging indicated that they were removed. This update checks the results for untagging and logging errors. The logs now indicate the status of untagging shared resources. (BZ#1926547)

Previously, the Azure clusters were created with the disk type of Premium_LRS and with an instance type that did not support PremiumIO capabilities which caused the cluster to fail. This update checks to see if the instance type picked has the PremiumIO capabilities only if the disk type is Premium_LRS which is the default disk type. The code queries the Azure subscription and region to get the information required and returns an error if the conditions are not met. (BZ#1931115)

Previously, the API VIP could become unavailable on the bootstrap when the API server restarted, which made the provisioning services unavailable and caused provisioning to fail. The provisioning services (Ironic) are now included in the VIP health checks, and the API VIP remains available. (BZ#1949859)

Previously, Google Cloud Platform (GCP) load balancer health checkers left stale conntrack entries on the host, which caused network interruptions to the API server traffic that used the GCP load balancers. The health check traffic no longer loops through the host, so there is no longer network disruption against the API server. (BZ#1925698)

Previously, the drain timeout and pool degrading period was too short and would cause alerts prematurely on a normal cluster that needed more time. With this update, the time needed before a timeout reports a failure is extended. This provides the Cluster Operator with more realistic and useful alerts without prematurely degrading performance of a normal cluster. (BZ#1968019)

Previously, while creating a new virtual machine from VMware vSphere with the OpenShift Installer Provisioned Infrastructure (IPI), the node failed to join the cluster. This occurred when Dynamic Host Configuration Protocol (DHCP), entered a host name in place of the name provided by IDI. This has been resolved. (BZ#1920807)

Previously, installation might fail if the network was enabled before the host name was set. This prevented the node from joining the cluser and forced a five minute delay before another attempt could be made. This is now resolved and the node automatically joins the cluster during the first attempt. (BZ#1899187)

Previously, users were able to delete the core user and related SSH keys, although the keys remained. With this update, users cannot delete the core user. (BZ#1885186)

When upgrading from 4.6 to 4.7, the host name set by the vsphere-hostname service was applied only when a node was installed. If the host name was not statically set prior to upgrading, the host name could be lost. This update removes the condition which allowed the vsphere-hostname service to run only when a node is installed. As a result, vSphere host names are no longer lost when upgrading. (BZ#1942207)

Due to a bug in keepalived 2.0.10, if a liveness probe killed a keepalived container, any virtual IP addresses (VIPs) that were assigned to the system remained and were not cleaned up when keepalived restarted. As a result, multiple nodes could hold the same VIP. Now, the VIPs are removed when keepalived is started. As a result, VIPs are held by a single node. (BZ#1931505)

Previously, rpm-ostree related operations were not handled properly on non-CoreOS nodes such as Red Hat Enterprise Linux CoreOS (RHCOS). As a result, RHEL nodes were degraded when an operation, such as kernel switching, was applied in the pool that contained RHEL nodes. With this update, the Machine Config Daemon logs a message whenever a non-supported operation is performed on non-CoreOS nodes. After logging the message, it returns nil instead of an error. RHEL nodes in the pool now proceed as expected when an unsupported operation is performed by the Machine Config Daemon. (BZ#1952368)

Previously, empty static pod files were being written to the /etc/kubernetes/manifests directory. As a result, the kubelet log was reporting errors that could cause confusion with some users. Empty manifests are now moved to a different location when they are not needed. As a result, the errors do not appear in the kubelet log. (BZ#1927042)

Previously, the Reporting Operator incorrectly handled Report custom resources (CRs) that contained a user-provided retention period when reconciling events. Consequently, an expired Report CR would cause the Reporting Operator to continually loop, as the affected custom resources are requeued indefinitely. This update avoids requeueing expired Report CRs that have specified a retention period. As a result, the Reporting Operator correctly handles events for expired Report CRs. (BZ#1926984)

Previously, the mountstats collector for the node-exporter daemontset caused high memory usage on nodes with NFS mount points. With this update, users can now disable the mountstats collector to reduce memory usage. (BZ#1955467)

Previously, an incorrect keepalived setting sometimes resulted in the VIP ending up on an incorrect system and unable to move back to the correct system. With this update, the incorrect keepalived setting is removed so that the VIP ends up on the correct system. (BZ#1916890)

Due to iptables rewriting rules, clients that used a fixed source port to connect to a service via both the service IP and a pod IP might have encountered problems with port conflicts. With this update, an additional OVS rule is inserted to notice when port conflicts occur and to do an extra SNAT to avoid said conflicts. As a result, there are no longer port conflicts when connecting to a service. (BZ#1910378)

Previously, IP port 9 between control plane nodes and egress-assigned nodes was blocked by the internal firewall. This caused the assignment of IP addresses to egress nodes to fail. This update enables access between control plane and egress nodes via IP port 9. As a result, the assignment of IP addresses to egress nodes is now successfully permitted. (BZ#1942856)

Previously, UDP services traffic could be blocked because of stale connection tracking entries that were no longer valid. This pevented access to a server pod after it was cycled for nodeport service. With this update, the connection tracking entries are purged in the case of nodeport service cycling, which allows new network traffic to reach cycled endpoints. (BZ#1949063)

Previously, OVN-Kubernetes network provider ignored network policies with multiple ipBlocks. Every ipBlock after the first one was ignored, resulting in pods being unable to reach all of the configured IP addresses. The code for generating OVN ACLs from Kubernetes network policies has been corrected. As a result, network policies with multiple ipBlocks now work correctly. (BZ#1953680)

Previously, when using the OVN-Kubernetes cluster network provider, a Kubernetes service without any endpoints erroneously accepted connections. With this update, a load balancer is no longer created for services without endpoints and therefore traffic is no longer accepted. (BZ#1918442)

Previously, the Container Network Interface (CNI) plugin for Multus did not understand IPv6 addresses that started with any number of zeros. With this update, the CNI plugin works with IPv6 that start with values greater than zero. (BZ#1919048)

Previously, a race condition might be triggered if the SR-IOV Network Operator initiated a reboot when a change in the machine config policy also triggered a reboot. If this occurred, the node was left in an indeterminate state. With this update, that situation is avoided. (BZ#1921321)

Previously, when creating a new user-provisioned cluster with the Kuryr cluster network provider, the OpenStack subset used by the cluster nodes might be undetected, which caused the cluster installation to time out. With this update, the subnet is correctly detected and a user-provisioned installation succeeds. (BZ#1927244)

Previously, when upgrading from OpenShift Container Platform 4.6 to OpenShift Container Platform 4.7, the Cluster Network Operator (CNO) incorrectly marked itself as having completed its upgrade to the next version. If the upgrade subsequently failed then the CNO reported itself as degraded, but erroneously as being at version 4.7. With this update, the CNO waits for the cluster network provider images to upgrade successfully before reporting the CNO upgrade as successful. (BZ#1928157)

Previously, when using the OVN-Kubernetes cluster network provider, the endpoint slice controller might not run if the Kubernetes version included a minor version that contained non-numeric characters. With this update, the endpoint slice controller is enabled by default. (BZ#1929314)

When using the Kuryr cluster network provider, Neutron Ports created subsequent to the installation were named with a different pattern than Neutron Ports created during installation. As a result, Neutron Ports created after installation were not added to the default load balancer. With this update, Kuryr detects Neutron Ports created with either naming convention. (BZ#1933269)

Previously, Open Virtual Network (OVN) changed the source IP addresses of hairpin traffic packets to the IP address of the load balancer, which sometimes blocked traffic when a network policy was in use. With this update, Kuryr opens traffic to the IP addresses of all services in a network policy’s namespace, and hairpin traffic flows freely. (BZ#1920532)

Previously, when starting a single-stack IPv6 cluster on nodes with IPv4 address, the kubelet might have used the IPv4 IP instead of the IPv6 IP for the node IP. Consequently, host network pods would have IPv4 IPs rather than IPv6 IPs, which made them unreachable from IPv6-only pods. This update fixes the node-IP-picking code, which results in the kubelet using the IPv6 IPs. (BZ#1939740)

Previously, and for unknown reasons, a kubelet could register the wrong IP address for a node. As a consequence, the node would be in a NotReady state until it was rebooted. Now, the systemd manager configuration is reloaded with the valid IP address as an environment variable, meaning that nodes no longer enter a NotReady state because a kubelet registered the wrong IP address. (BZ#1940939)

Previously, refactoring for a shadowed variable caused a regression related to the use of the checkpoint file, and SR-IOV pod sandboxes would not start. A check for the path of the kubelet socket was not properly accounted for during the refactor. The fix properly restores the check for the kubelet socket path, and now the SR-IOV pod sandboxes are properly created. (BZ#1968625)

Previously, Reliable Autonomic Distributed Object Store (RADOS) Block Devices (RBDs) were visible in unprivileged container pods running lsblk. This has been fixed and RBDs no longer are visible in unprivileged container pods running lsblk. (BZ#1772993).

Previously, during a cluster upgrade, the /etc/hostname file was altered by CRI-O, which caused the nodes to fail and to return when rebooting. This update adds special handling in CRI-O to leave the /etc/hosts file alone during upgrade, which allows upgraded nodes to boot without trouble. (BZ#1921937)

Previously, CRI-O was taking too long to create a pod after the network had been provisioned. This would trigger a bug in the network cleanup code, causing network resources not to properly clean up after network resources have been provisioned. This update changes the code to properly clean up networking resources, even if a command has timed out. This allows the cluster to continue normal network operation even if pod creation was taking too long. (BZ#1957224)

Previously, node reboots using a CNI plugin would not complete successfully. CRI-O was modified to call CNI DEL on all containers that were running before reboot. This update cleans up CNI resources and allows a successful reboot. (BZ#1948137)

Previously, a CNI DEL request would not be recalled if it failed, because CNI cleanup operations would not check for cleanup failure. Now, CRI-O recalls CNI DEL requests until they succeed, correctly cleaning up CNI resources. (BZ#1948047)

Previously, a reboot request to a container or image could cause failure if the reboot occurred while the container or image was being committed to the disk. This caused apparent corruption of storage for the container and created failures to pull the image or recreate containers from the image. This update detects when a node has rebooted and clears the container storage if true. (BZ#1942536)

Previously, runc took on the permissions of the entity that ran it. However, permissions on the workdir are set by the container user. When those permissions differ, container creation errors occured and caused failure of the container startup. This patch updates runc to chdir to the workdir multiple times, in case only one time fails. This ensures that creation of the container will succeed. (BZ#1934177)

Previously, the CRI-O logs did not contain information about the source where images were pulled from. With this fix, the log pull source is added to the info level of the CRI-O logs. (BZ#1881694)

Previously, when pods were created and deleted rapidly, a pod might not have enough time to complete the pod sandbox creation before the pod started deletion. As a result, pod deletion could fail with a `ErrCreatePodSandbox`error. This error is now ignored if a pod is terminating. As a result, pod termination no longer fails if a pod could not complete the pod sandbox creation. (BZ#1908378)

Previously, the Machine Config Operator (MCO) did not accept trace as a valid log level. As a result, the MCO could not provide a method to enable trace-level logging, even though CRI-O supports it. The MCO is now updated to support the trace log level. As a result, users can see a trace log level through the MCO configurations. (BZ#1930620)

Previously, the kubelet tried to get the status of images that are not completely pulled. As a result, crictl reports a error locating item named "manifest" error for these images. CRI-O is now updated to not list images that do not have a manifest. As a result, crictl no longer reports these errors. (BZ#1942608)

Previously, outdated status messages were not removed. Because of this, the Machine Config Operator (MCO) was sometimes unable to locate the proper machine config pool. With this release, a cleanup function is added to limit the number of statuses. As a result, the MCO keeps at most 3 different kubeletConfig status. (BZ#1950133)

Previously, when upgrading from OpenShift Container Platform version 4.6.25, in clusters with more than one kubeletconfig CR or ContainerRuntimeConfig CR, the Machine Config Operator (MCO) could generate duplicate machine configs for the same configuration. Consequently, the upgrade failed because the MCO would use the old controller version (IGNITIONVERSION 3.1.0). This update cleans up outdated duplicate machine configs and allows proper upgrading from version 4.6.25. (BZ#1955517)

Previously, some OAuth server metrics were not initialized properly and did not appear in searches in the Prometheus UI. The missing OAuth server metrics are now initialized properly and appear in the Prometheus UI metrics searches. (BZ#1892642)

Previously, if a custom security context constraint (SCC) contained a combination of the defaultAllowPrivilegeEscalation: false and allowPrivilegedContainer: true fields, the security context mutator mutated the privileged openshift-apiserver and oauth-apiserver pods to a state that failed API validation. The pods failed to start, which sometimes caused an OpenShift API outage. The security context mutator now ignores the defaultAllowPrivilegeEscalation field for containers that are already privileged, and custom SCCs that include those fields do not prevent the pods from starting. (BZ#1934400)

Previously, when running the oc explain command, the resource group name was not automatically detected if it was provided as part of the resource string. If two resources in different groups had the same resource name, the highest priority definition was returned unless the group was stated through the --api-version parameter. Now, if the --api-version parameter is not included, a prefix check is run against the resource string to detect the group name. The explanation returned by the command relates to the matching resource in the stated group. (BZ#1725981)

Previously, the oc image extract command did not extract files from the root directory of an image. The command has been updated and can now be used to extract files from the image root directory. (BZ#1919032)

Previously, the oc apply command would fetch the OpenAPI specification on each invocation. The OpenAPI specification is now cached when the command is first run. The cached OpenAPI specification is reused when the oc apply command is run multiple times and the network load is reduced. (BZ#1921885)

Previously, the authorization header created during image mirroring could exceed the header size limit for some registries. This would cause an error during the mirroring operation. Now, the --skip-multiple-scopes option is set to true for the oc adm catalog mirror command, to help prevent the authorization header from exceeding the header size limits. (BZ#1946839)

Previously, the storageClassName attribute was not added to a PersistentVolumeClaim object when the oc volume set command included the --claim-class option. The value of the --claim-class option was added to the volume.beta.kubernetes.io/storage-class annotation instead. This would cause snapshots for the volume to fail due to a dependency on the storageClassName attribute. Now, the oc volume set command applies the value of the --claim-class option to the storageClassName attribute in the PersistentVolumeClaim object, and volume snapshots can reference the attribute value. (BZ#1954124)

Previously, the output of oc adm top --help stated that the oc adm top command could display CPU, memory, and storage resource usage for pods and nodes. The oc adm top command does not display storage resource usage. Now, the storage reference is not included in the oc adm top --help output. (BZ#1959648)

Previously, CustomResourceDefinition (CRD) objects applied as part of an Operator installation could sometimes satisfy the installation requirements of a newer version of the same Operator. Consequently, during an Operator upgrade, the version being replaced could be prematurely removed. In some cases, the upgrade would stop. With this update, the CRDs that are created or updated as part of the Operator bundle installation are annotated to indicate their bundle of origin. These annotations are used by the ClusterServiceVersion (CSV) object to distinguish between pre-existing CRDs and same-bundle CRDs. As a result, upgrades will not complete until the current version’s CRDs have been applied. (BZ#1947946)

Previously, pods that ran an index referenced by a CatalogSource object did not have readOnlyRootFileSystem: false explicitly set in their securityContext field. Consequently, if a security context constraint (SCC) existed that enforced readOnlyRootFileSystem: true and matched the securityContext of the that pod, it would be assigned to that pod and cause it to fail repeatedly. This update explicitly sets readOnlyRootFileSystem: false in the securityContext field. As a result, pods that are referenced by a CatalogSource object are no longer matched to SCCs that enforce a read-only root file system and no longer fail. (BZ#1961472)

Operator Lifecycle Manager (OLM) previously did not allow skipped versions to be installed if the version was specified in the startingCSV field during initial installations. This caused those skipped versions to be unable to be installed, even if users wanted to install them regardless of reasons why they are skipped. This fix updates OLM to allow users to install skipped versions only during initial installation by using the startingCSV specification in Subscription objects; users still cannot upgrade to skipped versions, as expected. (BZ#1906056)

Because k8s.io/apiserver was not handling context errors for the webhook authorizer, context errors, such as timeouts, caused the authorizer to panic. This fix increments the API server version to include an upstream fix for the issue, and as a result, the authorizer can gracefully handle context errors. (BZ#1913525)

Previously, the oc adm catalog mirror command could not be easily used to mirror Operator catalogs across an airgapped environment. With this enhancement, the contents of a catalog can be mirrored to a file system, placed onto removable media, and then mirrored back from the file system to a registry for usage by an airgapped cluster. (BZ#1919168)

The Catalog Operator previously created bundle unpacking jobs for an install plan without setting a timeout. In the case of a non-existent or deleted bundle image, this caused the job to run forever and the install plan would stay in the Installing phase with no indication of the job’s pod failing to resolve the image. With this fix, the Catalog Operator now sets a default 10m timeout on bundle unpacking jobs, which can be configured by using --bundle-unpack-timeout flag. As a result, bundle unpacking jobs fail after the configured timeout, and the installation also transitions to a Failed phase with the reason visible in status.conditions and status.bundleLookups.conditions properties. (BZ#1921264)

Operators that were installed on clusters prior to OpenShift Container Platform 4.6 were previously not identified as coming from a given Operator package for the purposes of dependency resolution and upgrade selection. This caused existing Operator installations to conflict with the criteria of their own subscription, which blocked upgrades and dependency resolution within the namespace. This fix updates OLM to infer the package name and version for Operators that are referenced by a subscription. As a result, upgrades and dependency resolution proceed as expected. (BZ#1921953)

The Info log level used for transient errors caused verbose OLM Operator logs for the default configuration. This fix changes the transient error log level to debug. As a result, fewer non-critical logs are visible for the debug configuration. (BZ#1925614)

Previously, the spec.config.resources section of a Subscription object was always applied to the installed deployment, even when it was unset or empty. This caused resources defined in the cluster service version (CSV) to be ignored, and only the resources defined in the spec.config.resources section of the Subscription object were used. This fix updates OLM to override deployment-specific resources only when the spec.config.resources section is set to a non-nil or non-empty value. (BZ#1926893)

During dependency and upgrade resolution, subscription uniqueness was previously based on the subscribed package name. If two subscriptions in a namespace subscribe to the same package, they are treated as a single subscription internally, resulting in unexpected behaviors. With this fix, subscriptions are now uniquely identified internally within a namespace by .metadata.name instead of .spec.name. As a result, upgrade and dependency resolution behavior is consistent for namespaces containing multiple Subscription objects with the same .spec.name. (BZ#1932001)

When less than one minute remains before an upcoming catalog update polling attempt, the interval jitter function truncates the resync interval down to zero. This caused the Operator Catalog to enter a hot-loop, wasting CPU cycles. This fix increases precision of the jitter function used to calculate resync delays. As a result, the Catalog Operator remains mostly idle until the next catalog update poll. (BZ#1932182)

During an Operator upgrade, the owner reference of any associated ServiceAccount object was updated to point to the new ClusterServiceVersion (CSV) object instead of the old one. This could cause a race condition to occur between the OLM Operator, which reconciles CSVs, and the Catalog Operator, executes install plans, marking the old CSV as Pending/RequirementsNotMet due to the service account ownership change. This blocked upgrade completion while the new CSV waited indefinitely for the old CSV to indicate a healthy status. With this fix, instead of updating owner references in one step, the second owner is now appended to any existing owners. As a result, the same service account can satisfy requirements for both the old and the new CSV. (BZ#1934080)

Cluster service versions (CSV) previously required associated service accounts to either have no ownerReferences values set or to have an ownerReferences value set to the related CSV. This caused default service account, which is not created as part of Operator installation, to be unsatisfied as a CSV requirement if its metadata.ownerReferences field was not empty. With this fix, CSVs now require associated service accounts to either have no ownerReferences values set to CSVs or to have an ownerReference value set to the related CSV. As a result, service accounts with only non-CSV ownerReferences values can satisfy the requirements of any CSV. (BZ#1935909)

Prior to OpenShift Container Platform 4.5, the default catalogs deployed and managed by the Marketplace Operator in the openshift-marketplace namespace were created by OperatorSource objects, which was the API exposed by the Marketplace Operator. Appropriate metrics and alerting were instrumented to indicate an error encountered by the Operator sources. In OpenShift Container Platform 4.6, the OperatorSource resource was removed after being deprecated for several releases, and the Marketplace Operator instead directly created OLM’s CatalogSource resource. However, the same metrics and alerting instrumentation was not done for catalog sources deployed in the openshift-marketplace namespace. Therefore, errors encountered by the default catalog sources were not highlighted with Prometheus alerting. This fix introduces the new catalogsource_ready metric in OLM, which is used by the Marketplace Operator to fire alerts whenever the metric for a default catalog source indicates that a catalog source is in an unready state. As a result, Prometheus alerts are now provided for unready default catalog sources in the openshift-marketplace namespace. (BZ#1936585)

Previously, when a candidate Operator dependency was available from its default channel and a non-default channel, the Operator Lifecycle Manager (OLM) could generate a subscription that arbitrarily specified either of the two channels. Now, Operator dependencies are satisfied by candidates from the default channel first and then from other channels. (BZ#1945261)

Previously, it was possible that a cluster service version (CSV) was copied as a component of multiple Operators. This could happen when a namespace was added to an Operator group after the Operator was installed. This behavior affected memory use and CPU load. Now, CSVs do not appear in the status.components field of an Operator with a reason of Copied, and performance is not affected. (BZ#1946838)

Previously, some resources were caught in infinite loops because ManagedFields was being processed during reconciliation. This fix updates operator-lib to ignore ManagedFields, resulting in consistently reconciled loops. (BZ#1856714)

A minimum help message was being printed for the Operator SDK because the default plugin was not invoked when --help was passed on the command line interface (CLI). This fix invokes the default plugin and prints a more useful help message when a user runs the operator-sdk init --help command. (BZ#166222)

Previously, if ran with missing optional validators, operator-sdk bundle would fail rather than issue warnings. This has been corrected. (BZ#1921727)

Previously, custom security context constraints (SCCs) could have a higher priority than others in a default set. Consequently, those SSCs were sometimes matched to openshift-apiserver pods, which broke their ability to write in their root file system. This bug also caused an outage of some OpenShift APIs. This fix explicitly mentions in the openshift-apiserver pods that the root file system should be writable. As a result, custom SCCs should not prevent openshift-apiserver pods from running. (BZ#1942725)

Previously, when configuring a container to provide low latency response, dynamic interrupt masks with CRI-O did not match the interrupt mask set by irqbalance system service. Each one set different masks and compromised container latency. This update changes the interrupt mask set by setting CRI-O to match the irqbalance system service. As a result, dynamic interrupt mask handling now works as expected. (BZ#1934630)

Previously, multipath was enabled too late in the boot process. Consequently, Red Hat Enterprise Linux CoreOS (RHCOS) would return I/O errors in some multipath environments. With this update, multipath is now enabled earlier in the boot process. As a result, RHCOS no longer returns I/O errors in some multipath environments. (BZ#1954025)

Previously, a potential race condition could cause a fetch of the rootfs in a Red Hat Enterprise Linux CoreOS (RHCOS) PXE deployment to fail in some environments. With this fix, a connectivity check has been added that retries before an attempt to pull the rootfs so that access to the remote server and rootfs file is verified before continuing to the point where the coreos-livepxe-rootfs script used to sometimes fail. (BZ#1871303)

Previously, user presets for MachineConfig were ignored. This meant users could not change the configuration of kdump.service. Now, the priority level of default presets are lower than user configured defaults, so the user configuration can properly override vendor configuration. (BZ#1969208)

Previously, the coreos-installer would refuse to install onto a disk with a corrupted GUID Partition Table (GPT) because it would try to read the GPT of the target disk before overwriting it with the install image. With this fix, the coreos-installer now successfully installs onto a disk with a corrupt GPT by only reading the GPT of the target disk when it is instructed to preserve existing partitions. (BZ#1914976)

Previously, installation of clusters on unformatted direct-access storage devices (DASD) would result in the creation of disk sectors written incorrectly by coreos-installer. Now coreos-installer correctly formats new, unformatted DASD drives to 4096 byte sectors. This allows coreos-installer to complete the installation of the OS image to the disk drive. (BZ#1905159)

Previously, hardware-assisted zlib decompression on s390x z15 systems caused the mounting of the RHEL rootfs image to fail, which resulted in boot failure for REHL s390x z15 nodes using the RHEL 8.3 kernel. The kernel has now been updated to correctly handle zlib-compressed squashfs files when hardware-assisted zlib compression is available. (BZ#1903383)

Previously, the zipl command configured the disk geometry by assuming a sector size of 512 bytes. As a result, on SCSI disks with 4k sectors, the zipl bootloader configuration contained incorrect offsets and zVM was unable to boot. With this fix, zipl now takes the disk sector size into account so that zVM boots successfully. (BZ#1918723)

Previously, chrony.config might automatically run multiple time and fail each time but the first. This caused issues because chrony.config configuration is set during the initial run and cannot be changed. These errors are now avoided by limiting the configuration setup process to the first time chrony.config runs. (BZ#1924869)

Previously, nodes appeared unhealthy and did not operate as expected during periods of high workloads. This resulted from workloads using memory faster than the memory could be reclaimed. With this update, memory reclamation and out-of-memory situations were addressed and these conditions no longer occur during high workload situations. (BZ#1931467)

Previously, the maximum transmission unit (MTU) specification for a bond interface using kernal arguments did not get assigned properly. This has been corrected. (BZ#1932502)

Previously, the clevis-luks-askpass.path unit was not enabled by default. This caused non-root LUKS Clevis devices to fail to unlock automatically on reboot. This update enables the clevis-luks-askpass.path unit by default and allows non-root LUKS Clevis devices to unlock automatically on reboot. (BZ#1947490)

Previously, systemd was excessively reading mountinfo and over-consuming CPU resources, which caused containers to fail to start. This update enables limits when systemd reads mountinfo, which allows containers to start successfully. (BZ#1957726)

Previously, when the Machine Config Operator (MCO) invoked Ignition at startup to check the Ignition version, Ignition would crash. Consequently, the MCO would fail start. With this update, the MCO no longer queries the Ignition version, and the MCO starts successfully. (BZ#1927731)

Previously, the HAProxyDown alert message was vague. Consequently, end users thought the alert meant that router pods, instead of just HAProxy pods, were unavailable. This update makes the HAProxyDown alert message clearer. (BZ#1941592)

Previously, HAProxy’s helper function template that was responsible for generating a file for whitelist IPs expected a wrong argument type. Consequently, no whitelist ACL was applied for the backend in long IP lists. With this update, argument types of the helper function template are changed so that whitelist ACL is applied to the backend of long IP lists. (BZ#1964486)

Previously when creating an Ingress with a custom domain, the Ingress' status was updated by the OpenShift Container Platform Ingress controller with router canonical host name, and used external-dns to sync with Route 53. The problem was that the canonical router host name did not exist in the DNS and was not created by OpenShift Container Platform. OpenShift Container Platform creates the *.apps.<cluster_name>.<base_domain> DNS record and not the apps.<cluster_name>.<base_domain> DNS record. So the canonical router host name was not right. This fix sets the canonical router host name to router-default.apps.<cluster_name>.<base_domain>. Cluster administrators that have automation that takes the canonical host name and prepends a wildcard or a subdomain should be aware that the canonical Ingress host name is set as <ingress-controller-name>.apps.<cluster_name>.<base_domain>. (BZ#1901648)

Previously, the fix for BZ#1932401 overrode the default Go HTTP client transport. Consequently, cluster-wide proxy settings were not plumbed through to the Ingress Operator pod, which resulted in the failure of canary checks on a cluster with a cluster-wide egress proxy. This update explicitly sets proxy settings in the canary client’s HTTP transport. As a result, canary checks work with all cluster-wide proxies. (BZ#1935528)

Previously, the canary DaemonSet did not specify a node selector, so it used the default node selector for the canary namespace. Consequently, the canary DaemonSet could not schedule to infra nodes and in some cases would throw alerts. This update explicitly schedules the canary DaemonSet to infra nodes and tolerates tainted infra nodes. This allows the canary DaemonSet to safely roll out to worker and infra nodes without issues or alerts. (BZ#1933102)

Previously, when upgrading a cluster from a prior version with an idled workload, the idled workload would not wake on HTTP request once upgraded to OpenShift Container Platform 4.6 or 4.7 due to oc idle feature fixups and reworks. With this update, idling changes are mirrored from endpoints to services on Ingress Operator startup. As a result, unidling workloads after upgrades works as expected. (BZ#1925245)

Previously, exposing the default Ingress Controller through an external load balancer that redirected all HTTP traffic to HTTPS caused Ingress Canary endpoint checks performed by the Ingress Operator to fail, which would ultimately cause the Ingress Operator to become degraded. This fix converts the cleartext canary route to an edge encrypted route. Now the canary route works though HTTPS only load balancers when insecure traffic is redirected by the load balancer. (BZ#1932401)

Previously, the Ingress Operator Canary Check Client sent canary requests over HTTP to load balancers that dropped HTTP traffic. This caused the Ingress Operator to become become degraded after canary checks failed. With this fix, instead of relying on a redirect from the router, the Ingress Operator Canary Check Client sends canary check requests over HTTPS from the start. Now, canary checks work for clusters that expose the default Ingress Controller through a load balancer that drops insecure HTTP traffic. (BZ#1934773)

Previously, the HAProxy template used by openshift-router made repeated calls to a firstMatch() function. That function would parse and recompile a regular expression every time. Parsing and recompiling the regular expression on each call to firstMatch() is expensive, particularly for configurations that have many thousands of routes. With this fix, if the regular expression in the call to firstMatch() has already been seen, then an already compiled version is reused and cached. Now, there is a 60 percent reduction in run time when parsing and evaluating the haproxy-config.template. (BZ#1937972)

Previously, users could name a route with an invalid host name by using an override annotation. This update fixes the issue. (BZ#1925697)

Previously, removing selector from a service exposed via a route resulted in the duplication of endpointslices that would be created for the service’s pods, which would trigger HAProxy reload errors due to duplicate server entries. This update filters out accidental duplicate server lines when writing out the HAProxy config file, so that deleting the selector from a service no longer causes the router to fail. (BZ#1961550)

Previously, the Cluster Samples Operator could make changes to the controller cache for objects it was watching, which caused errors when Kubernetes managed the controller cache. This update fixes how the Cluster Samples Operator uses information in the controller cache. As a result, the Cluster Samples Operator does not cause errors by modifying controller caches. (BZ#1949481)

OpenShift Container Platform 4.8 allows users to run service-ca-operator pods as a non-root user to suit their organization’s needs. When run as a non-root user, the service-ca-operator runs as the following UID and GID:

uid=1001(1001) gid=1001 groups=1001

Previously, metrics for block type PVC filesystems were not being reported when requesting a capacity breakdown. This meant users received an inaccurate reporting of metrics across all their filesystems. With this update, block type PVC are included when requested by Kubelet. This provides an accurate reporting of all filesystem metrics. (BZ#1927359)

Previously, /var/lib/kubelet was mounted twice in the Cinder CSI Node Controller container. This caused the CSI Node Controller to fail to start with an error indicating /var/lib/kubelet/pods is out of space. The fix removes the duplicate mount of /var/lib/kubelet and /var/lib/kubelet/pods, which allows the CSI Node Controller to run successfully. (BZ#1952211)

Previously, during Cinder CSI Driver resizing of a persistent volume (PV), the findmnt command received multiple volume mounts and could not choose the correct one, thereby causing resizing to stop. As a result, users would have to extend the file system manually. With this fix, the command now uses the first mount so that the file system is resized along with the PV. (BZ#1919291)

The Cinder CSI Driver Operator now automatically provisions a default VolumeSnapshotClass object for Cinder CSI when creating a default storage class, rather than having to create the VolumeSnapshotClass object manually. (BZ#1905849)

Previously, the recycler-pod template was incorrectly placed in the kubelet static manifest directory. This incorrect location produced static pod log messages that indicated a recycler static pod start failure. With this update, the misplaced recycler-pod template has been removed from the static pod manifest directory. As a result, the error messages no longer appear. (BZ#1896226)

Previously, the Local Storage Operator (LSO) could claim disks belonging to other provisioners because busy disks were erroneously detected as free. Disks are now checked for bind-mounts so that the LSO cannot claim those disks. (BZ#1929175)

Previously, the Local Storage Operator (LSO) would attempt to create a persistent volume (PV) with an invalid label value, because the device-id contained unsupported characters such as :. This has been corrected by moving the device information from labels to annotations. (BZ#1933630)

Previously, the Local Storage Operator (LSO) was not cleaning up persistent volumes (PVs) since the deleter was not being enqueued correctly. This caused PVs to remain in the released state. PVs are now enqueued correctly so that they delete properly. (BZ#1937145)

Previously, the Fibre Channel volume was incorrectly unmounted from a node when a pod was deleted. This happened when a different pod that used the volume was deleted in the API server when the kubelet on the node was not running. With this update, Fibre Channel volumes correctly unmount when a new kubelet starts. Additionally, the volume cannot be mounted to multiple nodes until the new kubelet fully starts and cofnrims that the volume is unmounted, which ensures that Fibre Channel volumes are uncorrupted. (BZ#1954509)

Previously, when attempting to delete a custom resource within the CNV namespace in the console UI in developer mode, clicking Delete in resulted in the Delete button hanging in a stuck state. Additionally, an error message that appears when performing the same action in the CLI was not displaying. With this update, the error message displays as expected and the Delete button does not stick. (BZ#1939753)

Previously, OperatorHub Provider Type filter property did not clearly show the relationship to CatalogSource. Because of this problem, users could not tell what the filter criteria meant. This patch updates the Provider Type filter to Source. This more clearly shows the relationship between filter and CatalogSource. (BZ#1919406)

Previously, the ResourceListDropdown component in the Resources menu was not internationalized for some languages. With this update, the Resources menu is updated to better the user experience for non-English speakers. (BZ#1921267)

Previously, some menu items, such as Delete Persistent Volume Claim, were not internationalized correctly. Now, more menu items are correctly internationalized. (BZ#1926126)

Previously, some text and warning messages for the Add HorizontalPodAutoscaler page were not internationalized. The text is now internationalized. (BZ#1926131)

Previously, when users created an Operator with the Operator SDK and specified an annotation like xDescriptors={"urn:alm:<…​>:hidden"} to hide a field from the Operator instance creation page, the field might still be visible on the page. Now, the hidden fields are omitted from the Operator instance creation page. (BZ#1966077)

Previously, tables displayed incorrectly on mobile devices. With this update, tables now display correctly. (BZ#1927013)

Previously, launching the OpenShift Container Platform web console may be slow. With this update, the web console launches quicker. (BZ#1927310)

Previously, a lack of internationalized notifications to OpenShift Container Platform administrators detracted from the user experience. Now, internationalization is possible. (BZ#1927898)

Previously, a lack of internationalized duration times on the Cluster Utilization dashboard detracted from the user experience. Now, internationalization is possible. (BZ#1927902)

Previously, errors occurred when Operator Lifecycle Manager (OLM) status descriptors in the OpenShift Container Platform web console were assigned incompatible data types. Validation has been added, eliminating incompatible data types from processing, thus avoiding errors. Logged warnings also identify the incompatible status types. (BZ#1927941)

The following OpenShift Container Platform web console views now support multi-faceted filtering:

The following bug fixes address various translation issues for the OpenShift Container Platform web console:

Previously, the web console relied on hard-coded channel strings to populate the channel modal dropdown. As a result, users could see channel values that may not be correct for their current version. Now, if the Cluster Version Operator does not supply the correct channels for a given version, the channel modal dropdown changes to a text input field and suggests channels and help text for the user. The console no longer relies on hard coded channel string. (BZ#1932281)

Previously, timestamps were not correctly formatted for Chinese or Japanese languages. As a result, timestamps were harder to read, which provided a bad user experience. With this update, default timestamp formats are used for Chinese and Japanese in Moment.js, which provides a better user experience. (BZ#1932453)

Previously, the rowFilters prop in the FilterToolbar component did not accept the null value. So if the rowFilters prop was undefined, the uncaught exception was thrown. Now, when the rowFilters prop is referenced in the FilterToolbar component, the null value is accepted. As a result, the FilterToolbar does not throw exceptions when rowFilters prop is undefined. (BZ#1937018)

Previously, the wrong style of help text was applied to the field level help instances. Now, the correct style of help text is shown for the field level help instances and is consistent across the console. (BZ#1942749).

Previously, the Operator Lifecycle Manamgent (OLM) status conditions descriptors were rendered as normal detail items on the resource details page. As a result, the Conditions table was rendered at half width. With this update, conditions descriptors are rendered as a full-width table below the normal Conditions table on the Operand details page. (BZ#1943238)

Previously, the word "Ingresses" was translated for Chinese users, but the user experience was bad. Now the word "Ingress" is not translated. (BZ#1945816)

Previously, the word "Operators" was translated for Chinese users, but the plural translation resulted in a bad user experience. Now the word "Operators" is not translated. (BZ#1945818)

Previously, an incorrect code was causing the User and Group details to show unrelated subjects. Now, a code has been added to filter by User or Group, so the User and Group details show related subjects. (BZ#1951212)

Previously, the pod Containers text was not internationalized, so there was a poor user experience. Now the pod Containers text has been internationalized, so the user experience is improved. (BZ#1937102)

Previously, the PackageManifest list page items did not link to the details page, so users could not easily drill down into individual PackageManifest items from the list page. Now, each PackageManifest item is linked to the details page that matches the convention of the other list pages. Users can easily access the PackageManifest details page from the list page. (BZ#1938321)

The Completions column of the Jobs table sorted by the number of Desired completions instead of the number of Succeeded completions. The data is presented as # Succeeded of # Desired, so when sorting by that column the results looked confusing because the data was sorted by the second number. The Jobs Completions column now sorts on the # Succeeded for better understanding. (BZ#1902003)

The input labels in the Manage Columns modal were not clickable buttons, so you could not click them to manage the columns. With this bug fix, the labels are now clickable buttons that you can use to manage columns. (BZ#1908343)

CSI provisioners were not listed when creating a storage class on the Google Cloud Platform. With this bug fix, the issue is resolved. (BZ#1910500)

Previously, if the user clicked into a Cluster Role from the User Management → Roles list view, the back link from the details page is Cluster Roles, which provides a generic list view of Cluster Roles. This caused backward web console navigation to redirect to the incorrect page. With this release, the back link directs the user to the Role/Bindings list view from the Cluster Role/RoleBinding details page. This allows the user to correctly navigate backward in the web console. (BZ#1915971)

Previously, Created date time was not displayed in a readable format, which made it difficult to understand and use the time shown in UTC. With this release, the displayed time is reformatted so that UTC is readable and understandable. (BZ#1917241)

Previously, pod requests and limit calculations in the web console were incorrect. This was a result of not excluding completed pods or init containers. With this release, pods that are not needed in the calculation are excluded, which improves the accuracy of the results of the web console calculation for pod requests. (BZ#1918785)

Previously, parsing undefined values resulted in a not a number (NaN) exception and the Chart tooltip showed a box with no values. With this release, a start date is specified when fetching data so that the Chart toolip shows correct values. This change ensures that the results are synced and that undefined values are not parsed. (BZ#1906304)

During a previous bug fix, the download link for pod logs was changed to a standard HTML anchor element with an empty download attribute. Consequently, the download file lost the default file name format. This update adds a file name to the anchor element download attribute so that a default file name, formatted as <pod-name>-<container-name>.log, is used when downloading pod logs. (BZ#1945630)

Previously, when a user had permission to create a resource but not permission to edit it, the web console YAML editor was incorrectly set to read-only mode. The editor content is now editable by users with create access for the resource. (BZ#1824911)

Previously, the web console showed times in the 12-hour format in most places, and the 24-hour format in others. Additionally, the year was not displayed for dates more than one year in the past. With this release, dates and times are formatted consistently and match the user locale and language preference settings, and the year is displayed for dates more than one year in the past. (BZ#1862084)

Previously, the web console was polling the ClusterVersion resource for users who didn’t have the authority to view those events. This would output large numbers of errors in the console pod log. To avoid this, checking the user’s persmissions before polling the resource is required, which eliminates unnecessary errors in the console pod log. (BZ#1848151)

Previously, keyboard users of the YAML editor were unable to exit the editor. The view shortcuts popover outside of the editor was unavailable inside the editor for access by the user. With this update, users can display Accessibility help above the editor using the opt + F1 keystrokes. This change allows keyboard users of the YAML editor to exit the editor using the correct keystrokes. (BZ#1874931)

After the 4.x release of OpenShift Container Platform (OCP), binary secret files uploaded to the OCP 4 web console failed to load. This caused the installation to fail. With OpenShift Container Platform 4.8, this capability has been restored to the OCP 4 web console. Input of the required secret can now be accomplished using binary file format. (BZ#1879638)

Previously, the fix for BZ#1871996 to properly create RoleBinding links consistently resulted in the inability to select the binding type when a namespace was selected. Consequently, users with an active namespace could not create a cluster RoleBinding without changing the active namespace to All namespaces. This update reverts part of the changes for BZ#1871996 so that users can create a cluster role binding regardless of active namespace. (BZ#1927882)

Previously, when the label changed for making a service cluster local on the Developer Console, users were not able to create a Knative service. This update to the Knative service uses the latest supported label for cluster-local in order to enable users to create a Knative service as cluster-local from Developer Console. (BZ#1969951)

Previously, the colors for the Low and Medium severity issues of the Image Manifest Vulnerabilities (IMVs) did not match the color representation shown in the (Quay.io) interface. As a result, when the user changed the severity order of vulnerabilities to High, the IMVs ordered the issues incorrectly. This created confusion when reviewing the IMVs. The current release fixes this issue. (BZ#1942716)

Previously, the Topology view in the Developer perspective did not load if OpenShift namespace templates were not available because the Samples Operator was not installed. This update fixes the issue. (BZ#1949810)

Previously, when you imported a devfile, the web console ignored the build guidance placeholder container which provided the configuration for environment variables, ports, and limits. The new deployment had a second container which could not start because the placeholder image could not be fetched and it missed the required configurations. Now, the build guidance container is dropped from the new deployment, and the container adds the environment variables, ports, and limit configurations. (BZ#1952214)

Previously, when switching to the Developer perspective in another tab and reloading the project details, the routes tied to the perspective were not rendered and resulted in a 404 error. This update loads all inactive routes and switches to the correct perspective. (BZ#1929769)

Previously, when an error occurred due to a user not having the required access for a namespace, the Workload drop-down menu in the Monitoring dashboard page continuously displayed a loading-in-progress icon. The current release fixes this issue. Now, the Monitoring dashboard page displays an error message indicating that a Forbidden error has occurred. (BZ#1930546)

Previously, an API server could fail to create a resource, which would return a 409 status code when there was a conflict updating a resource quota resource. Consequently, the resource would fail to create, and you might have had to retry the API request. With this update, the OpenShift Console web application attempts to retry the request 3 times when receiving a 409 status code, which is often sufficient for completing the request. In the event that a 409 status code continues to occur, an error will be displayed in the console. (BZ#1920699)

Previously, when selecting the YAML tab, the metadata.managedFields section did not collapse immediately. This was due to an issue with the Form or YAML switcher for pages such as Pipeline Builder and Edit HorizontalPodAutoscaler (HPA). As a result, the part of the document where you tried to type collapsed. The metadata.managedFields section remained as is, and the cursor was reset to the starting position to the top left of the YAML editor. The current release fixes this issue. Now, upon loading the YAML, the metadata.managedFields section collapses immediately. (BZ#1932472)

Previously, pipelines created in the Git Import flow for private repositories failed to run. This happened because the pipeline ServiceAccount object did not use secrets created by the Git Import flow for private Git repositories. With this update, you can add a secret name to the annotations of the pipeline ServiceAccount object, and add pipeline-specific annotations to the provided secret. As a result, pipelines for private Git repositories run successfully. (BZ#1970470)

Previously, when users inserted a formatted YAML snippet in the YAML editor, the new selection did not match the new content in the snippet. The indentation was removed, and some random letters were seen in the selection. The current release fixes this issue. Now, the cursor remains in the position where it started and adds the missing indentation for the cursor end position. After inserting the YAML snippet, the new selection matches the new content. (BZ#1952545)

Previously, annotations were passed to the specification of the Knative service as well as to the metadata. As a result, decorators were shown for associated revisions of Knative service in Topology. This release fixes this issue by passing annotations only to the Knative service metadata. Now, decorators are shown only for the Knative service in Topology and not associated revisions. (BZ#1954959)

Previously, if you created a pipeline with parameters that had empty strings, for example, ”, the fields in the OpenShift Container Platform web console would not accept the empty strings. The current release fixes this issue. Now, ” is supported as a valid default property within the parameters section. (BZ#1951043)

Previously, users were not able to create a Knative service as a private service from the Developer perspective. This issue has now been fixed, by updating the label 'networking.knative.dev/visibility': 'cluster-local'. (BZ#1970796)

Previously, the Kamelets of type sink were shown in the catalog for event sources along with the type source. This issue has now been fixed by filtering Kamelets resources to list only the source type. (BZ#1972258)

Previously, the load balancer service became unstable when users would scale additional Windows nodes. With this update, the load balancer service is stabilized, which allows users to add multiple Windows nodes without erratic performance. (BZ#1905950)

Previously, the kube-proxy service crashed unexpectedly after the load balancer was created if it was created after the Windows pods were running. With this update, the kube-proxy service does not crash when recreating the load balancer service. (BZ#1939968)

Previously, empty IP address values in the load balancer’s Ingress broke the data path. As a result, the Windows service was unreachable. With this update, the Windows service is reachable even if the IP address value is empty. (BZ#1952914)

Previously, when users created a Windows pod with a projected volume, the pod would remain stuck in the ContainerCreating phase. With this update, the Windows pod creation successfully proceeds to the Running phase. (BZ#1973580)

TP: Technology Preview

GA: General Availability

-: Not Available

DEP: Deprecated

In OpenShift Container Platform 4.1, anonymous users could access discovery endpoints. Later releases revoked this access to reduce the possible attack surface for security exploits because some discovery endpoints are forwarded to aggregated API servers. However, unauthenticated access is preserved in upgraded clusters so that existing use cases are not broken.

## Snippet to remove unauthenticated group from all the cluster role bindings
$ for clusterrolebinding in cluster-status-binding discovery system:basic-user system:discovery system:openshift:discovery ;
do
### Find the index of unauthenticated group in list of subjects
index=$(oc get clusterrolebinding ${clusterrolebinding} -o json | jq 'select(.subjects!=null) | .subjects | map(.name=="system:unauthenticated") | index(true)');
### Remove the element at index from subjects array
oc patch clusterrolebinding ${clusterrolebinding} --type=json --patch "[{'op': 'remove','path': '/subjects/$index'}]";
done

The oc annotate command does not work for LDAP group names that contain an equal sign (=), because the command uses the equal sign as a delimiter between the annotation name and value. As a workaround, use oc patch or oc edit to add the annotation. (BZ#1917280)

When powering on a virtual machine on vSphere with user-provisioned infrastructure, the process of scaling up a node might not work as expected. A known issue in the hypervisor configuration causes machines to be created within the hypervisor but not powered on. If a node appears to be stuck in the Provisioning state after scaling up a machine set, you can investigate the status of the virtual machine in the vSphere instance itself. Use the VMware commands govc tasks and govc events to determine the status of the virtual machine. Check for a similar error message to the following:

Invalid memory setting: memory reservation (sched.mem.min) should be equal to memsize(8192).

The installation of RHCOS on a RHEL KVM installation on IBM Z fails when using an ECKD type DASD as a VirtIO block device. (BZ#1960485)

An Open Virtual Network (OVN) bug causes persistent connectivity issues with Octavia load balancers. When Octavia load balancers are created, OVN might not plug them into some Neutron subnets. These load balancers might be unreachable for some of the Neutron subnets. This problem affects Neutron subnets, which are created for each OpenShift namespace, at random when Kuryr is configured. As a result, when this problem occurs the load balancer that implements OpenShift Service objects will be unreachable from OpenShift namespaces affected by the issue. Because of this bug, OpenShift Container Platform 4.8 deployments that use Kuryr SDN are not recommended on Red Hat OpenStack Platform (RHOSP) 16.1 with OVN and OVN Octavia configured until the bug is fixed. (BZ#1937392)

The Console Operator does not properly update the Ingress resource with the componentRoutes conditions for either of the console’s routes (console or downloads). (BZ#1954148)

The OVN-Kubernetes network provider does not support the externalTrafficPolicy feature for nodeport- and LoadBalancer-type services. The service.spec.externalTrafficPolicy field determines whether traffic for a service is routed to node-local or cluster-wide endpoints. Currently, such traffic is routed by default to cluster-wide endpoints, and there is no way to limit traffic to node-local endpoints. This will be resolved in a future release. (BZ#1903408)

Currently, a Kubernetes port collision issue can cause a breakdown in pod-to-pod communication, even after pods are redeployed. For detailed information and a workaround, see the Red Hat Knowledge Base solution Port collisions between pod and cluster IPs on OpenShift 4 with OVN-Kubernetes. (BZ#1939676, BZ#1939045)

For clusters that use the OVN-Kubernetes network provider and whose compute nodes run RHEL 7.9, upgrading from OpenShift Container Platform 4.7 to OpenShift Container Platform 4.8 is blocked by BZ#1976232. To upgrade to release 4.8, you must wait for the 4.8 patch that includes the fix for this bug. (BZ#1976232)

For clusters that use the OVN-Kubernetes network provider and upgrade from OpenShift Container Platform 4.7 to OpenShift Container Platform 4.8, a bug in OVN-Kubernetes can sometimes cause the pod IP address to become stale. The bug is a rarely experienced race conditition. As a consequence, during the upgrade to the 4.8 release, nodes fail to drain and some Operators report a status of Degraded. As a workaround, identify the pods that are stuck in the CrashLoopBackOff state and that did not complete the upgrade. Delete each pod with the oc delete <pod-name> command. (BZ#1974403)

The description for the tlsSecurityProfile field of the kubeletconfig resource (for example when using the oc explain command) does not list the correct ciphers for the TLS security profiles. As a workaround, review the list of ciphers in the /etc/kubernetes/kubelet.conf file of an affected node. (BZ#1971899)

When running CNF tests in regular mode on a single node, the logic in place to understand if the cluster is ready is missing details. In particular, creating an SR-IOV network will not create a network attachment definition until at least one minute elapses. All the DPDK tests fail in cascade. Run the CNF tests in regular mode skipping the DPDK feature when running against an installation on a single node, with the -ginkgo.skip parameter. Run CNF tests in Discovery mode to execute tests against an installation on a single node. (BZ#1970409)

Currently, CNF-tests does not support secure boot with MLX NICs for SR-IOV and DPDK tests. You can run the CNF tests skipping the SR-IOV feature when running against a secure boot enabled environment in regular mode, with the -ginkgo.skip parameter. Running in Discovery mode is the recommended way to execute tests against a secure boot enabled environment with MLX cards. This will be resolved in a future release. (BZ#1975708)

When the ArgoCD Operator is subscribed to and an ArgoCD and AppProject are started, launching the example application named guestbook fails because the image does not work in more restrictive OpenShift Container Platform environments. As a temporary workaround, users can ensure the ArgoCD Operator works properly by deploying the following example:

PROJ=younamespace
cat > $PROJ-app.yaml <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: simple-restricted-webserver
  namespace: $PROJ
spec:
  destination:
    namespace: $PROJ
    server: https://kubernetes.default.svc
  project: default
  source:
    path: basic-nginx
    repoURL: 'https://github.com/opdev/argocd-example-restricted-apps.git'
    targetRevision: HEAD
EOF
oc create -f $PROJ-app.yaml

If you have the console open in multiple tabs, some sidebar links in the Developer perspective do not directly link to the project, and there is an unexpected change in the selected project. This will be resolved in a future release. (BZ#1839101)

Creating a passthrough route using Ingress fails when using pathType: Prefix. Instead, you can create a passthrough route by setting pathType to ImplementationSpecific and setting path to '':

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress7
  namespace: test-ingress
  annotations:
    route.openshift.io/termination: passthrough
spec:
  rules:
  - host: <ingress-psql-example-test-ingress.apps>
    http:
      paths:
      - path: ''
        pathType: ImplementationSpecific
        backend:
          service:
            name: <ingress-psql-example>
            port:
              number: 8080

Currently, in the Search page, the Pipelines resources table is not immediately updated after you apply or remove the Name filter. However, if you refresh the page and expand the Pipelines section, the Name filter is applied. The same behavior is seen when you remove the Name filter. This will be resolved in a future release. (BZ#1901207).

Documentation now describes that the ProvisioningNetworkCIDR value in the Provisioning custom resource. This limits the IPv6 provisioning networks to a limit of /64 due to dnsmasq. (BZ#1947293)

To assist with troubleshooting, logs collected on boostratp failures by the installer now include IP addresses and routes for the control plane and bootstrap hosts. (BZ#1956079)

When using a self-signed Amazon Commercial Cloud Services cluster, you cannot pull from or push to an internal image registry. As a workaround, you must set spec.disableRedirect to true in the configs.imageregistry/cluster resource. This lets you pull the image layers from the image registry rather than directly from S3 storage. (BZ#1924568)

Previously, the topology URLs created for deployments using Bitbucket repository in the OpenShift Container Platform web console did not work if they included a branch name that contained a slash character. This was due to an issue with the Bitbucket API (BCLOUD-9969). The current release mitigates this issue. If a branch name contains a slash, the topology URLs point to the default branch page for the repository. This issue will be fixed in a future release of OpenShift Container Platform. (BZ#1969535).

Installing OpenShift Container Platform (OCP) version 4.6 on Red Hat Virtualization (RHV) requires RHV version 4.4. If you are running an earlier version of OCP on RHV 4.3, do not update it to OCP version 4.6. Red Hat has not tested running OCP version 4.6 on RHV version 4.3 and does not support this combination. For additional information about tested integrations, see OpenShift Container Platform 4.x Tested Integrations (for x86_x64).

The operator-sdk pkgman-to-bundle command exits with an error when run with the --build-cmd flag. For more information, see (BZ#1967369).

Currently, the prerequisites in the web console quick start cards appear as a paragraph instead of a list. This will be resolved in a future release. (BZ#1905147)

On an OpenShift Container Platform single node configuration, pod creation times are over two times slower when using the real-time kernel (kernel-rt) than when using the non-real-time kernel. When using kernel-rt, the slower creation times impact the maximum number of supported pods because recovery time is impacted after a node reboots. As a workaround when you are using the kernel-rt, you can improve the impacted recovery time by booting with the rcupdate.rcu_normal_after_boot=0 kernel argument, which requires a real-time kernel version kernel-rt-4.18.0-305.16.1.rt7.88.el8_4 or later. This known issue applies to the OpenShift Container Platform version 4.8.15 and later. (BZ#1975356)

Following an OpenShift Container Platform single node reboot, all pods are restarted which causes significant load and longer than normal pod creation times. This happens because the Container Network Interface (CNI) is not able to process the pod add events quickly enough. The following error message is displayed: timed out waiting for OVS port binding. The OpenShift Container Platform single node instance eventually recovers, though more slowly than expected. This known issue applies to the OpenShift Container Platform version 4.8.15 and later. (BZ#1986216)

Prior to OpenShift Container Platform 4.8 the default load balancing algorithm was leastconn. The default was changed to random in OpenShift Container Platform 4.8.0 for non-passthrough routes. Switching to random is incompatible with environments that need to use long-running websocket connections because it significantly increases memory consumption in those environments. To mitigate this significant memory consumption, the default load balancing algorithm was reverted to leastconn in OpenShift Container Platform 4.8. Once there is a solution that does not incur significant memory usage, the default will be changed to random in a future OpenShift Container Platform release.

$ oc get deployment -n openshift-ingress router-default -o yaml | grep -A 2 ROUTER_LOAD_BALANCE_ALGORITHM
        - name: ROUTER_LOAD_BALANCE_ALGORITHM
          value: leastconn

$ oc annotate -n <NAMESPACE> route/<ROUTE-NAME> "haproxy.router.openshift.io/balance=random"

If the images for RHCOS and the Machine Config Operator (MCO) do not change during an upgrade from an OpenShift Container Platform 4.8.z release to a later 4.8.z release, the upgrade is marked as complete before the control plane nodes have finished upgrading. Following this, the upgrade might fail if you perform operations on your cluster before the upgrade is actually complete. As a workaround, verify that updates are completed on the control plane nodes before performing additional operations on the cluster. You can use the oc get mcp/master command to review the status of the MCO-managed nodes that are available on your cluster for each pool. (BZ#2025396)

After upgrading from a 4.7 OpenShift Container Platform cluster to 4.8, the routing path from the internal network to the external network pod through secondary network interface controllers (NICs) on OpenShift Container Platform nodes is disabled by default. This is due to the shared gateway being the default gateway mode in the Open Virtual Network (OVN) design starting from 4.8. If that routing path is required, as a workaround before or after the upgrade, create a gateway-mode-config config map in the openshift-network-operator namespace to force OVN Gateway mode to local.

$ cat localGW.yml

apiVersion: v1
kind: ConfigMap
metadata:
    name: gateway-mode-config
    namespace: openshift-network-operator
data:
    mode: "local"
immutable: true

$ oc apply -f localGW.yml

configmap/gateway-mode-config created

It is not possible to create a macvlan on the physical function (PF) when a virtual function (VF) already exists. This issue affects the Intel E810 NIC. (BZ#2120585)