OpenShift Container Platform 4.3 release notes

Categorizing Operator tests as required/optional.

Configuring test selection and pass/fail behavior.

Shipping custom tests.

Helm v3 support, starting with Operator SDK 0.14.0.

Role-based access control (RBAC) generation.

Support for Prometheus metrics.

Usage of Red Hat Universal Base Image (UBI).

Molecule-based end to end testing.

OpenAPI spec generation.

Kubernetes 1.14 support.

Removal of dep-based projects. All Go projects are now scaffolded to use Go modules. The operator-sdk new command’s --dep-manager flag has been removed.

Required Go version update from v1.10 to v1.13.

Support for Prometheus metrics.

The Authentication Operator reported a static "available" string as a reason for the unavailability condition, which was unclear. This bug fix implements more precise reasons for unavailability conditions, and as a result, inspecting why the Operator is unavailable is more clear. (BZ#1740357)

The oauth-proxy process was reloading CA certificates for each request and storing them in memory. High memory consumption caused the oauth-proxy container to be killed. With this bug fix, CA certificates are now cached unless they change. As a result, memory consumption for the oauth-proxy process has dropped significantly when multiple requests against it are issued. (BZ#1759169)

Previously, the client CA certificate configured for the RequestHeader identity provider (IdP) was not announced among other certificates during TLS handshake with the OAuth server. When login-proxies tried to connect to the OAuth server, they would not use their client certificate, resulting in their request being unauthenticated, which in turn caused users of the IdP being unable to log in to the cluster. This bug fix adds the configured client CA among the rest in the TLS configuration, and as a result, authentication using the RequestHeader IdP works as expected. (BZ#1764558)

The bootstrap user introduced in OpenShift Container Platform 4.1 internally made CLI log flow always available. The message about how to retrieve an authentication token, which was there in OCP 3.x, no longer appeared for users that tried to log in from CLI in cases where only web console flows were configured. With this bug fix, the bootstrap user identity provider (IdP) is no longer configured when it is disabled by the user. As a result, after the bootstrap IdP is disabled by following the steps from the OCP documentation, the message about how to retrieve an authentication token in web console-only scenarios is now displayed. (BZ#1781083)

Previously, the route to oauth-server did not react to Ingress domain changes, which degraded the Authentication Operator and caused the oauth-server to not authenticate properly. The oauth-server route now updates when an Ingress domain change is detected, allowing authentication to work in this scenario. (BZ#1707905)

Builds started very soon after an imagestream was created might not leverage local pullthrough imagestream tags when specified. The build attempts to pull the image from the external image registry, and if the build is not set up with the authorization and certificates needed for that registry, the build would fail. The build controller is now updated to detect when its imagestream cache is missing the necessary information to allow for local pullthrough imagestream tags and retrieve that information from other means. Builds can now successfully leverage local imagestream tag pullthrough. (BZ#1753731)

The build controller sometimes incorrectly assumed a build was instantiated from the build config endpoint when it was actually instantiated directly from the build endpoint. Therefore, confusing logging about non-existent build configs could appear in the build controller logs if a user instantiated an OpenShift build directly, as opposed to initiating a build request off of the build config API endpoint. The build controller is now updated to better check whether a build was instantiated from the build config endpoint and refrain from logging unnecessary error messages.The build controller logs no longer have these confusing error messages for build instantiated directly versus from the build config endpoint. (BZ#1767218), (BZ#1767219)

Previously, the update protocol Cincinnati, designed to facilitate automatic updates, used tags for payload references. This could yield different results when applying the same release of the same graph at different points. Now the payload reference uses the image SHA instead, if the container registry provides the manifestref. This guarantees the exact release version a cluster is going to use. (BZ#1686589)

Previously, the specified volumeMode was not passed to newly created disks, so PVCs might not bind properly. The volumeMode is now passed properly to the newly created disks. (BZ#1753688)

Previously, the virtual machine detail page did not load properly when accessed directly by the URL. The page now loads properly. (BZ#1731480)

Previously, the kubevirt-storage-class-defaults ConfigMap setting was not reflected properly for VMware VM imports. Because of this, blockMode PVCs could not be used for VMware VM imports. The storage class defaults are now used properly when requesting VMware imported disks. (BZ#1762217)

Previously, the title for the Import VM wizard was incorrect and could be confusing. The wizard now has the correct title of Import Virtual Machine. (BZ#1768442)

Previously, the confirmation buttons for storage and network configuration in the VM migration wizard were located in the wrong place. These confirmation buttons are now located in the correct location. (BZ#1778783)

Previously, the Create Virtual Machine wizard did not prompt for confirmation before creating a VM, which meant the user could unexpectedly create a VM. With this fix, the user must click "Create Virtual Machine" on the review page before a VM is created. (BZ#1674407)

Previously, the Create Virtual Machine wizard had required fields that were not always intuitive when importing a VM. The Create Virtual Machine wizard has been redesigned to work as expected. (BZ#1710939)

Previously, error messages for validating VM names were not helpful. These error messages have been improved to be more descriptive. (BZ#1743938)

Previously, CRI-O was not properly filtering Podman containers during a restore operation. Because Podman containers do not have CRI-O-specific metadata, at startup, CRI-O would interpret the Podman containers it saw as CRI-O containers that were incorrectly created. It would therefore ask the storage library to delete the containers. This bug fix now properly filters Podman containers on CRI-O restore so that they are no longer deleted from storage upon startup. (BZ#1758500)

Etcd would become overloaded with a large number of objects, causing the cluster to go down when etcd failed. Now, the etcd client balancer facilitates peer failovers in the event of a client connection timeout. (BZ#1706103)

Etcd would fail during the upgrade process and result in disaster recovery remediation steps. Now, etcd has been updated to resolve gRPC package to prevent catastrophic cluster failure. (BZ#1733594)

After changing the storage type in the image registry Operator’s configuration, both the previous and new storage types appeared Operator’s status. Because of this behavior, the image registry Operator was not removed after you deleted its configuration. Now only the new storage type is displayed, so the image registry Operator is removed after you change the storage type that the image uses. BZ#1722878)

Because it was possible for older imagestreams to have invalid names, image pruning failed when the specs for the imagestream’s tags were invalid. Now, the image pruner always prune images when the associated imagestream has an invalid name. BZ#1749256)

When the image registry operator’s management state was Removed, it did not report itself as Available or the correct version number. Because of this issue, upgrades failed when the image registry operator was set to Removed. Now when you set the image registry Operator’s status to Removed, it reports itself as Available and at the correct version. You can complete upgrades even if you remove the image registry from the cluster. (BZ#1753778)

It was possible to configure the image registry Operator with an invalid Azure container name, and the image registry did not deploy on Azure because of the invalid name. Now the image registry Operator’s API schema ensures that the Azure container name that you enter conforms to Azure’s API requirements and is valid, which ensures that the Operator can deploy. (BZ#1750675)

An unnecessary service monitoring object was created for each of the following contollers: kube-apiserver, kube-controller-manager, and kube-scheduler. The unused service monitoring object is no longer created. (BZ#1735509)

When a cluster is in a non-upgradeable state because either Technology Preview features or custom features are enabled, no alert was sent. The cluster will now send a TechPreviewNoupgrade alert through Prometheus if an upgrade is attempted on a cluster in a non-upgradeable state. (BZ#1731228)

When defining a StatefulSet resource object, custom labels were not applied when creating PersistentVolumeClaim resource objects from the template specified by volumeClaimTemplates parameter. Custom labels are now applied correctly to PersistentVolumeClaim objects created from the volumeClaimTemplates objects defined by a StatefulSet resource. (BZ#1753467)

Previously, if the lease ConfigMap for the Kubernetes Controller Manager (KCM) was deleted, KCM did not have permission to recreate the ConfigMap and was unable to do so. The KCM can now recreate the lease ConfigMap if it is deleted. (BZ#1780843)

Mismatches between cluster version and ClusterLogging version would cause ClusterLogging to fail to deploy. Now, the kubeversion is verified that it supports the deployed ClusterLogging version. (BZ#1765261)

The data in journald for facility values was not sanitized and values were incorrect, causing fluentd to emit error messages at the wrong level. Now, fluentd logs at the debug level and these errors are reported correctly. (BZ#1753936,BZ#1766187)

The oauth-proxy was misconfigured in a way that users were unable to log in after logging out. Now, the oauth-proxy has been reconfigured so that users can log in again after logging out. (BZ#1725517)

Eventrouter was not able to handle unknown event types, which would result in Eventrouter crashing. Now, Eventrouter properly handles unknown event types. (BZ#1753568)

The Management Console Dashboard Details were unnecessarily watching the Infrastructure resources. As a result, errors regarding early web socket connection terminations were possible. Now, the Details card does not watch Infrastructure resources and only fetches the resource data once. Errors are not reported after implementing this fix. (BZ#1765083)

The console Operator would record an initial empty string value for the console URL before the router had a chance to provide the host name. Now, the Operator waits until the hostname is filled and eliminates the empty string value. (BZ#1768684)

Previously, the containerImage field in the metering-operator CSV bundle referenced an image tag that was not listed in the image-references file that ART uses for substitution purposes. This meant that ART wasn’t able to properly substitute the origin image listed in the containerImage field with the associated image-registry repository and sha256 tag. This bug fix replaces the image tag latest with release-4.3, which is what was defined in the image-references file. As a result, ART is now able to successfully substitute the metering-operator container image. (BZ#1782237)

Previously, Hadoop Dockerfile.rhel copied the gcs-connector JAR file to the wrong location in the container. The path has been corrected to now point to the right location. (BZ#1767629)

Previously, not all related objects were deleted when the CNO was changed, which left stale network-attachment-definitions. The code has been refactored to now do this in a more generic way in OpenShift Container Platform 4.3 so that the related objects are cleaned up properly. (BZ#1755586)

Previously, some updates were dropped which caused events to be missed. Events are no longer dropped. (BZ#1747532)

Previously, in clusters that had high network traffic volumes with packet loss, a once-successful connection to a service could fail with a Connection reset by peer error. As a result, clients had to reconnect and retransmit. An update has been made to iptables rules to process TCP retransmits correctly. Established connections will remain open until they are closed. (BZ#1762298)

Previously, NetworkPolicy rule applications to new namespaces could occur slowly in clusters that had many namespaces, namespace changes, and NetworkPolicies that select namespaces. New namespaces could take significant amounts of time before they could be accessed from other namespaces. Due to an update in Namespace and NetworkPolicy code, NetworkPolicies should be applied promptly to new namespaces. (BZ#1752636)

Previously, SDN pods did not clean up Egress IP addresses when they restarted on a node, resulting in IP address conflicts. SDN pods now clean up stale Egress IP addresses as they start, preventing such conflicts from occuring. (link: BZ#1753216)

Previously, DNS names were queried every time they occured in an EgressNetworkPolicy. Records were queried regardless of whether a particular DNS record had been refreshed by a previous query, resulting in slow network performance. DNS records are now queried based on unique names rather than per each EgressNetworkPolicy. As a result, DNS query performance has been significantly improved. (BZ#1684079)

Route creation between multiple service endpoints was not possible from the console. Now, the GUI has been updated to add or remove up to three alternative service endpoints. (BZ#1725006)

Previously, when containers had a high (or > 1) restart count, the kubelet could inject duplicate container metrics into the metrics stream, causing the /metrics endpoint on the kubelet to throw a 500 error. With this bug fix, only metrics of the most current container (running or stopped) are included. As a result, the /metrics endpoint now allows metrics to flow to Prometheus without causing a 500 error. (BZ#1779285)

Upstream changes were made to the long path names test. Pods with names longer than 255 character were not logged and no warning was issued. Now, the long names test is removed and Pods with names longer than 255 characters will log as expected. (BZ#1711544)

The LocalStorageCapacityIsolation feature was disabled, and users were unable to use the Statefulset.emptyDir.sizeLimit parameter. Now, the LocalStorageCapacityIsolation feature has been enabled and the Statefulset.emptyDir.sizeLimit parameter can be set. (BZ#1758434)

Previously when using server-side print, the wide output option was ignored when used in a watch (oc get clusteroperators -o wide). The operation has been fixed to now properly recognize all the possible options when using server-side print. (BZ#1685189)

The oc explain command links to upstream documentation were out of date. These links have been updated and are now valid. (BZ#1727781)

Full usage menu information was printed along with bad flag error messages, causing the error message to be lost at times. Now, when the oc command --help command is run, the bad flag error is the only information displayed. (BZ#1748777)

The oc status command was not displaying DaemonSets in a consistent format due to missing status code information. Now, the Daemonsets, Deployments, and Deployment Configurations are printed properly in the output of the oc status command. (BZ#1540560)

The commands oc version and oopenshift-install version would show as Dirty due to incorectly set flags. These flags have been updated and the commands no longer display a Dirty GitTreeState or GitVersion. (BZ#1715001)

The oc status command would suggest oc set probe pod to verify pods are still running, including pods that may have been owned by controllers. Now, pods that are owned by controllers are ignored for probe suggestions. (BZ#1712697)

Previously, the oc new-build help command was not properly filtering flags. This caused irrelevant flags to be printed when invoking oc new-build --help. This has been fixed, and now the help command only prints relevant output. (BZ#1737392)

The ClusterResourceQuota in 4.2 and 4.3 were not allowing non-strings as limit values because the OpenAPI schema was wrong. Therefore, integer quota values could not be set in ClusterResourceQuota objects, even though doing so was previously possible in 4.1. The OpenAPI schema for ClusterResourceQuota has been fixed to allow integers so that integers can now be used as quota values in ClusterResourceQuota again. (BZ#1756417)

During upgrades, openshift-apiserver would report degraded. The reason for degradation was MultipleAvailable, but this was not understandable to the user. This bug fix now lists the reason for the degradation, so that no information is hidden from the user. (BZ#1767156)

The console workload shows a restricted access error if the knative serverless TP1 Operator is installed and you are logged in as non-admin user. With this bug fix, the Overview sidebar resources now work as expected for both normal and knative-specific deployments. A non-admin user can now view the workloads. (BZ#1758628)

The topology view data model was originally a subset of the project Workloads page. As more feature were added, the topology view grew to be similar but did not share the same code. As use cases became more complex, certain edge cases were being missed in the new code. In certain situations, the Pod list from the topology view was incorrect. With this bug fix, code logic is now shared between the topology view and project Workloads page. As a result, whether viewing the sidebar Pod list from topology or from the project Workloads list, the Pod details are now identical. (BZ#1760827)

Previously, when the Route object was created, the first port from the list of available ports was set instead of setting the selected port from the target-port dropdown menu. Because of this, the user was unable to select their desired target port. The port selected from the target port dropdown menu is now applied when creating a Route object; if no port is selected, the first port from the list is set. (BZ#1760836)

Previously, certain features, such as the name of the application and the build status, were not rendered in the Topology view on the Edge browser. With this bug fix, the Edge browser renders the application name and the build status as expected. (BZ#1760858)

In the web console Overview, a non-admin user was not able to view workloads when the Knative Operator was installed, even if a deployment that was not a Knative workload was selected. This bug fix adds a check in case there are no configurations found so that the system will not add Knative-specific resources in Overview. This enables a non-admin user to now view the workloads as expected. (BZ#1760810)

Previously, when the Topology context menu was open, the associated node was not easily identifiable. This caused confusion for users because they did not know which node the context menu referred to. Now when right-clicking a node to open the context menu, a visual hover, or drop shadow, is applied to the node for easier identification. (BZ#1776401)

Previously, the Import from Git form in the web console used a regular expression too restrictive to validate the Git URL, which disallowed some valid URLs. The regular expression has been updated to accept all valid Git URLs. (BZ#1766350), (BZ#1771851)

Error messages from the developer console were duplicated. Now, this system has been updated to reflect values from the client side. As a result, error messages are now clear and concise. (BZ#1688613)

Previously, the web console could experience a runtime error when visiting the Resources tab of an OLM operand resource. The web console could also freeze when trying to sort the Resources tab for an OLM operand resource. These issues are now resolved. (BZ#1756319)

Previously, visiting the OpenShift web console pod details page in Microsoft Edge could result in a runtime error, preventing the page from displaying. The issue is now resolved and the pod details page now displays correctly. (BZ#1768654)

Previously, if a dashboard card watched Prometheus results, the dashboard page’s performance could decrease due to an incorrect comparison between old alerts and new alerts. The comparison defect has been fixed. (BZ#1781053)

In previous versions, the documentation link on the Network Policy page was incorrect. It has been replaced with the correct link. (BZ#1692227)

Previously, Prometheus queries contained a range selector, which prevented the chart on the default page of the Prometheus UI from rendering. The queries no longer contain range selectors, so the query now renders properly. (BZ#1746979)

Recycle was the default value for the Persistent Volume Reclaim policy even though that option was deprecated. Persistent Volumes contained deprecated values by default. The default Persistent Volume Reclaim policy is now Retain, so new Persistent Volumes do not contain deprecated values. (BZ#1751647)

Previously, after upgrading your cluster, the web console could use cached CSS stylesheets, which might cause some rendering issues when loading the console. The problem has been fixed, and the web console now correctly uses the correct stylesheets after an upgrade. (BZ#1772687)

Previously, when using the web console in some situations part of the options menu was hidden behind other elements on the page. The options menu no longer appears behind other page elements and will expand in a viewable space on the page to ensure the entire menu is always visible. (BZ#1723254)

Previously, long node names could overflow the table column in the OpenShift console pods table. With this bug fix, they now correctly wrap. (BZ#1713193)

Previously, creating a report query using an example YAML would result in an error. This bug fix adds a new YAML example for report queries that contains all required fields so that an error does not occur. (BZ#1753124)

Previously on the Install Plan Details page, the namespace for associated catalog sources was set incorrectly. This resulted in broken links because the namespace did not exist. This bug fix uses the status.plan field of the InstallPlan resource to associate the catalog source with the correct namespace to build links from. Thus, the catalog source links now work as expected. (BZ#1767072)

Previously, unknown custom resources were automatically split into words to estimate what the user should see. However, some resources were split inappropriately. With this bug fix, custom resources now use the name as defined in the Custom Resource Definition, rather than being split into separate words.(BZ#1722811)

If you have Service Mesh installed, upgrade Service Mesh before upgrading OpenShift Container Platform. For a workaround, see Updating OpenShift Service Mesh from version 1.0.1 to 1.0.2.

Determination of active Pods when a rollout fails can be incorrect in the Topology view. (BZ#1760828)

When a user with limited cluster-wide permissions creates an application using the Container Image option in the Add page, and chooses the Image name from internal registry option, no imagestreams are detected in the project, though an imagestream exists. (BZ#1784264)

The ImageContentSourcePolicy is not supported by the registry at the time of release (BZ#1787112). In disconnected environments, Jenkins can be enabled to pull through by default. Use this command as a workaround to use Jenkins in disconnected environments:

$ oc tag <jenkins_source_image> jenkins:2 --reference-policy=source -n openshift

The OpenShift Cluster Version Operator (CVO) does not correctly mount SSL certificates from the host, which prevents cluster version updates when using MITM proxy checking. (BZ#1773419)

When adding defaultProxy and gitProxy under builds.config.openshift.io， the Jenkins pipeline build cannot retrieve the proxy configuration. (BZ#1753562)

When installing on Red Hat OpenStack Platform 13 or 16, where the OpenStack endpoints are configured with self-signed TLS certificates the installation will fail. (BZ#1786314,BZ#1769879,BZ#1735192)

Installer-provisioned infrastructure installations on OpenStack fail with Security group rule already exists error when OpenStack Neutron is under heavy load. (BZ#1788062)

Clusters will display errors and abnormal states after etcd backup or restore functions are conducted during the etcd encryption migration process. (BZ#1776811)

RHCOS master and worker nodes may go into a NotReady,SchedulingDisabled state while upgrading from 4.2.12 to 4.3.0. (BZ#1786993)

The public cloud access image for RHEL cannot be used directly if you enable FIPS mode. This is caused by public cloud images not allowing kernel integrity checks. To do this, you must upload your own images. (BZ#1788051)

The Operator Lifecycle Manager (OLM) does not work in OpenShift Container Platform when Kuryr SDN is enabled. (BZ#1786217)

The oc adm catalog build and oc adm catalog mirror commands do not work for the restricted cluster. (BZ#1773821)

When upgrading an OpenShift Container Platform cluster from 4.1 to 4.2 to 4.3, there is a possibility that the Node Tuning Operator tuned Pods can get stuck in the ContainerCreating state.

$ oc get pods -n openshift-cluster-node-tuning-operator

$ oc delete daemonset/tuned -n openshift-cluster-node-tuning-operator
$ oc get daemonset/tuned -n openshift-cluster-node-tuning-operator
$ oc get pods -n openshift-cluster-node-tuning-operator

The Node Feature Discovery (NFD) Operator version 4.3 fails to deploy from OpertorHub on the OpenShift Container Platform web console. As a workaround, download the oc client for your operating system, and place the kubeconfig file from the installer in ~/.kube/config. Run these commands to deploy the NFD Operator from the CLI and GitHub:

$ cd $GOPATH/src/openshift
$ git clone https://github.com/openshift/cluster-nfd-operator.git
$ cd cluster-nfd-operator
$ git checkout release-4.3
$ PULLPOLICY=Always make deploy
$ oc get pods -n openshift-nfd

$ oc get pods -n openshift-nfd
NAME READY STATUS RESTARTS AGE
nfd-master-gj4bh 1/1 Running 0 9m46s
nfd-master-hngrm 1/1 Running 0 9m46s
nfd-master-shwg5 1/1 Running 0 9m46s
nfd-operator-b74cbdc66-jsgqq 1/1 Running 0 10m
nfd-worker-87wpn 1/1 Running 2 9m47s
nfd-worker-d7kj8 1/1 Running 1 9m47s
nfd-worker-n4g7g 1/1 Running 1 9m47s

If a cluster-wide egress proxy is configured and then later unset, Pods for applications that have been previously deployed by OLM-managed Operators can enter a CrashLoopBackOff state. This is caused by the deployed Operator still being configured to rely on the proxy.

$ oc get deployments --all-namespaces \
    -l olm.owner,olm.owner!=packageserver (1)

There is an issue with the Machine Config Operator (MCO) supporting Day 2 proxy support, which describes when an existing non-proxied cluster is reconfigured to use a proxy. The MCO should apply newly configured proxy CA certificates in a ConfigMap to the RHCOS trust bundle; this is not working. As a workaround, you must manually add the proxy CA certificate to your trust bundle and then update the trust bundle:

$ cp /opt/registry/certs/<my_root_ca>.crt /etc/pki/ca-trust/source/anchors/
$ update-ca-trust extract
$ oc adm drain <node>
$ systemctl reboot

When upgrading to a new OpenShift Container Platform z-stream release, connectivity to the API server might be interrupted as nodes are upgraded, causing API requests to fail. (BZ#1791162)

When upgrading to a new OpenShift Container Platform z-stream release, connectivity to routers might be interrupted as router Pods are updated. For the duration of the upgrade, some applications might not be consistently reachable. (BZ#1809665)

Git clone operations that go through an HTTPS proxy fail. Non-TLS (HTTP) proxies can be used successfully. (BZ#1750650)

Git clone operations fail in builds running behind a proxy if the source URIs use the git:// or ssh:// scheme. (BZ#1751738)

In OpenShift Container Platform 4.1, anonymous users could access discovery endpoints. Later releases revoked this access to reduce the possible attack surface for security exploits because some discovery endpoints are forwarded to aggregated API servers. However, unauthenticated access is preserved in upgraded clusters so that existing use cases are not broken.

## Snippet to remove unauthenticated group from all the cluster role bindings
$ for clusterrolebinding in cluster-status-binding discovery system:basic-user system:discovery system:openshift:discovery ;
do
### Find the index of unauthenticated group in list of subjects
index=$(oc get clusterrolebinding ${clusterrolebinding} -o json | jq 'select(.subjects!=null) | .subjects | map(.name=="system:unauthenticated") | index(true)');
### Remove the element at index from subjects array
oc patch clusterrolebinding ${clusterrolebinding} --type=json --patch "[{'op': 'remove','path': '/subjects/$index'}]";
done