OpenShift Container Platform 3.10 Release Notes

Migrates RHEL Server hosts to the RPM-based installation method for the kubelet

Migrates the container runtime and RHEL Atomic Hosts to the system container-based installation method for the kubelet only (because the container runtime is part of RHEL Atomic Host)

Some build container environment variables were modified when redacted in the container log. As a result, URL proxy settings (such as HTTP/S proxies) were modified, breaking these settings. A copy of these environment variables are made prior to redaction in the logs. (BZ#11571349)

Streaming of build logs failed due to a server-side timeout waiting for the build pod to start. Therefore, oc start-build could hang if the --wait and --follow flags were set. With this bug fix:

The build watch maintained by the openshift jenkins sync plug-in would no longer function, even while watchers on other API object types still functioned. The finding of a build would then fall upon the background build list thread, which by default runs at 5-minute intervals. This bug fix adds better logging around unexpected closure of the openshift jenkins sync plug-in watches, adds reconnect when those closures occur, and adds the ability for customers to configure the relist interval. Now, customers do not have to wait up to 5 minutes for the pipeline strategy builds to start. (BZ#1554902)

The build controller was susceptible to incorrectly failing builds when time was not synchronized accurately between multiple masters. The controller logic is now improved to not depend on time synchronization. (BZ#1547551)

The webhook payload can contain an empty commit array, which results in an array indexing error when processed by the APIserver. As a result, the API server crashes. Check for an empty array before attempting to index into it. With this bug fix, empty commit payloads are handled without crashing the API server. (BZ#1585663)

An invalid SELinux context for the Docker engine prevented docker exec to work. With this bug fix, the issue is resolved. (BZ#1517212)

Jenkins would fail to parse certificates with the Bag Attributes preceding the BEGIN CERTIFICATE line and fail to start since the openshift jenkins image adds such a certificate to the Kubernetes cloud configuration. With this bug fix, remove the Bag Attributes preceding the BEGIN CERTIFICATE line in the certificates mounted into the pod; generally validate the certificate for proper format. Jenkins can now start when such certificates are introduced. (BZ#1548619)

A new value in the Reference field was not considered as a change. Therefore, the status field was not updated. This bug fix updates detection of changes. You can now set Reference: true and get any image reference in an image stream tag. (BZ#1555149)

Additional certificate name constraints prevented valid certificates from being processed, resulting in an error of "tls: failed to parse certificate from server: x509: unhandled critical extension". As a result, valid certificates were unusable. By moving to newer golang libraries that fixed the constraint. certificates that previously failed can now be used. (BZ#1518583)

Previously, PhantomJS would not install on jenkins-slave-base-rhel7 image. this was because PhantomJS is packaged as tar.bz2 archive and jenkins-slave-base-rhel7 did not contain the bzip2 binary. The OpenShift Container Platform version 3.10, includes newer Jenkins image with bzip2 binary. (BZ#1544693)

Due to a compatibility issue in earlier versions, the networkPluginName entry was listed twice in the node-config.yaml. The duplicate entry is no longer needed and has been removed. (BZ#1567970)

Due to a change in the installer, if using images from a registry other than the default, you need to configure the registry using the oreg_url parameter in the /etc/ansible/hosts file for all components and images. Previously, you needed to configure the oreg_url, openshift_docker_additional_registries, and openshift_docker_insecure_registries parameters. (BZ#1516534)

Environments where the Azure cloud provider is enabled now provision a default storage class for use with Azure storage. (BZ#1537479)

You can now uninstall the Service Catalog using the Ansible Playbook if the openshift-ansible-serivce-broker project is not present. Previously, the uninstall playbook would fail if the project is not present. (BZ#1561485)

Because nfs storage cannot provide the file system capabilities required by OpenShift registry, logging, and metrics components, a check has been added to the installer that will not allow nfs storage for these components. To use nfs storage for these components, you must opt-in by setting the cluster variable openshift_enable_unsupported_configurations to true, otherwise the procedure will fail. The use of nfs storage for registry, metrics, and logging components is only supported for proof of concept environments and not for production environments. (BZ#1416639)

Ansible playbooks were taking too long to execute and could result in certificate errors from hosts that are not relevant to the task being performed. The playbooks have been modified to check only relevant hosts. (BZ#1516526)

Ansible installer playbooks were creating persistent volumes before creating storage classes causing the playbooks to be run twice. The playbooks were changed to create the storage classes before any persistent volumes. (BZ#1564170)

Because the way the OpenShift prefix and version were set for the console, the version reported by the console was different than the version displayed by other components. Control plane upgrade now ensures that the console version matches the version of other control plane components. (BZ#1540427)

Because of the Ansible installation playbooks, you needed to manually configure storage classes after installation in order to create PVCs. You can now configure storage classes at installation time by setting the following parameters in your inventory file:

openshift_storageclass_name=test-1
openshift_storageclass_provisioner=rbd
openshift_storageclass_parameters={'fstype': 'ext4', 'iopsPerGB': '10', 'foo': 'bar'}

The certificate expiration playbook, easy-mode.yaml, was not checking all certificate files for expiration information. As a result, expired files were not being discovered, which could result in errors. The Ansible playbook has been updated. (BZ#1520971)

Previously, dnsmasq was configured to listen on a specific IP address in an effort to avoid binding to 127.0.0.1:53, which is where the node service runs its DNS service. This update configures dnsmasq to bind to all interfaces except the loopback, which ensures that dnsmasq works properly on hosts with multiple interfaces. (BZ#1481366)

In rare cases, the router or registry registryurl variables may need to be set to values other than the first master registry_url value. This fix allows the openshift_hosted_router_registryurl and openshift_hosted_registry_registryurl variables to be set in the inventory. (BZ#1509853)

A recent change in SELinux policy requires that an additional SEBoolean is set when running any pods with systemd which includes CFME. (BZ#1587825)

The kube- and openshift- prefixes are preserved for internal use cases. to avoid name conflict, it is better to use the preserved prefix as default logging project. This fix uses the preserved prefix as the default logging project. This fits the pattern used by other infrastructure applications and allows the EFK stack to participate with other services that assume the infrastructure is deployed to namespaces with a known pattern (for example, openshift-). (BZ#1535300)

A utility logging-dump.sh dumps the ElasticSearch logs as part of useful information for troubleshooting. In OpenShift Container Platform 3.10, the log location of ElasticSearch has been moved from /elasticsearch/logging-es[-ops]/logs to /elasticsearch/persistent/logging-es[-ops]/logs. logging-dump.sh fails to dump ElasticSearch logs with a an error of Unable to get ES logs from pod <ES_POD_NAME>. In addition to /elasticsearch/logging-es[-ops]/logs, check the new path /elasticsearch/persistent/logging-es[-ops]/logs for the logs files. With this bug fix, logging-dump.sh successfully dumps ElasticSearch logs. (BZ#1588416)

Previously, if a pod took more than five minutes to become ready, the web console would warn you, regardless of timeoutSeconds specified in the deployment configuration. For some applications, this period was too short. This fix removes this warning from the web console. (BZ#1550138)

Prior to this release, the copy and paste operation in the web console container terminal did not work properly on Firefox and Internet Explorer. This fix updates xterm.js to v3.1.0. You can now copy and paste from the context menu or using keyboard shortcuts. (BZ#1278733)

When the “No results match” result occurred in the Console or Catalog page, there were two links for clearing the search keys, “Clear Filters” and “Clear All Filters”. With this fix, all occurrences of “Clear Filters” were changed to “Clear All Filters”. Now there is one option to clear filters. (BZ#1549450)

Different BuildConfig Webhook URLs were obtained by the CLI and Web Console. This caused the CLI to use the the correct build.openshift.io API group, while the Web Console did not use an API group. This fix updated the Webhook filter to use the correct build.openshift.io API group for the Web Console, and as a result the the correct URL for the BuildConfig Webhook is provided. (BZ#1559325)

Manually typing a URL with a non-existing image, such as /console/project/pro1/browse/images/non-existent-image, caused the loading screen to freeze even though the process was finished and the alert, “The image stream details could not be loaded”, to be displayed. With this fix, the loaded scope variable is set when the image is or is not loaded and is used in the view to hide the loading screen. As a result, following the attempt to load the image data, the screen will not freeze on loading. (BZ#1550797)

Previously, the web console did not support deploying an application with private repository image on the Deploy Image page. This is fixed and users can now deploy an app with a private repository image. (BZ#1489374)

Previously, DaemonSet nodes were restricted with project default node selector, causing the creation and deletion of DaemonSet pods in a loop on those nodes. This fix patched upstream DaemonSet logic to be aware of project default node selector. As a result, creation and deletion loop of DaemonSet pods on the nodes that got restricted by project default node selector is resolved. (BZ#1501514)

Previously, the client was not able to read full discovery but was stuck on the first aggregated server which was temporarily unavailable. This led to not having the proper information about all the resources that were available. This fix introduced a default timeout for discovery actions. As a result, in case of a failure on an aggregated server the client will continue discovering resources on other servers and allow users to work with the ones that are available. (BZ#1525014)

Previously, when pods that used DeploymentConfigs with the recreate strategy were evicted, a new pod did not come online until the timeout interval elapsed. Now the the recreate strategy creates a new pod even if evicted pods are present. (BZ#1549931)

Previously, the auto_snapshot parameter was set to true in the cassandra.yaml file, and because of changes to Hawkular Metrics introduced in OpenShift Container Platform 3.7, so many snapshots were generated that the disk might fill up. Now auto_snapshot is disabled by default, and snapshots are generated only if you set the openshift_metrics_cassandra_take_snapshot property to true in the Ansible inventory file. (BZ#1567251)

Previously, you could not distribute multiple CA certificates to pods in the cluster. This limitation caused issues with load balancer configurations that required a different CA certificate than the one generated by the installer for the the master node. Now you can define the location of the load balancer certificate in the openshift_additional_ca parameter during installation. The certificate is added to the /etc/origin/master/ca-bundle.crt file, which is made available to pods in the cluster. (BZ#1535585)

In version 3.9, the Prometheus service account did not have the required permissions to access the metrics endpoint of the router, so Prometheus could not obtain the router’s metrics. Now the Prometheus service account has the necessary additional role to access the metrics endpoint and can obtain metrics from the router. (BZ#1565095)

Previously, the service controller sent a request to the cloud provider every time a service was created. This request checked whether the cloud provider had a load balancer for the service, even for non-LoadBalancer services. In clusters where many services were created, the extra requests dominated some cloud provider API usage. The service controller no longer sends this request to the cloud provider when a non-LoadBalancer service is created, which reduces the cloud provider API usage. (BZ#1571940)

Previously, the egress router configuration prevented egress router pods from connecting to the public IP address of the nodes that host them. If an egress pod was configured to use its node as a name server in the /etc/resolv.conf file, DNS resolution failed. Traffic from an egress router pod to its node is now routed via the SDN tunnel instead of through the egress interface. Egress routers can now connect to their node’s IP, and egress router DNS works. (BZ#1552738)

If two nodes swapped IP addresses after you rebooted them, other nodes were sometimes unable to send traffic to pods on one or both of those nodes. Now, the OVS flow correctly manage node IP address reassignment, and pod-to-pod traffic continues even if nodes swap IP addresses. (BZ#1538220)

Previously, changing an EgressIP of a NetNamespace while its existing EgressIP is active, assigned duplicate EgressIPs to the NetNamespaces of the same HostSubnets, resulting in egress IPs to stop working if an egress IP is moved from one project or node to another. Additionally, if the same egress IP is assigned to two different projects, or two different nodes, then it may not work correctly even after the duplicate assignment is removed. The EgressIPs field on a NetNamespace have been fixed to change while the egress IP is active. This results in static per-project egress IPs should work more reliably. (BZ#1551028)

The kube-proxy and kubelet parts of the OpenShift node process were being given different default values for the configuration options describing how to interact with iptables. This resulted in OpenShift periodically add a false iptables rule that would cause some per-project static egress IPs to not be used for some length of time, until the false rule was removed again. While the bogus rule was present, traffic from those projects would use the node IP address of the node hosting the egress IP, rather than the egress IP itself. The inconsistent configuration was resolved, causing the false iptables rule to no longer be added, and projects now consistently use their static egress IPs. (BZ#1552869)

Previously, OpenShift’s default network plug-in did not contain the newest NetworkPolicy features introduced upstream in Kubernetes. These included policies for controlling egress, and policies based on IP addresses rather than pods or namespaces. This meant that in version 3.9, creating a NetworkPolicy with an ipBlock stanza would cause nodes to crash, and creating a NetworkPolicy that contained only "egress" rules would erroneously cause ingress traffic to be blocked. Now, OpenShift Container Platform is aware of the unsupported NetworkPolicy features, though it does not yet implement them, and if a NetworkPolicy contains ipBlock rules, those rules are ignored. This may cause the policy to be treated as "deny all" if the ipBlock rule was the only rule in the policy. If a NetworkPolicy contains only "egress" rules, it is ignored completely and does not affect ingress. (BZ#1583255)

When deleting a pod, some of the IP files were not deleted as intended. This was caused by the garbage collection picking up a dead container. The kubelet keeps the information from at least one container in the case of if a restart is needed. This bug fix ensures that a proper clean up happens only if the network plug-in returns success, but some other error happens after that before the runtime (eg. dockershim or CRI-O) returns to kubelet. (BZ#1532965)

Previously, the dnsmasq service would randomly freeze and would need a manual restart to start the resolution. This caused no logs to be captured for the dnsmasq service on OpenShift Container Platform node hosts. This was caused by the interface connecting with dnsmasq changing between releases, overloading the service. The dns-forward-max and cache-size option limits have been increased to 10000, and the service now works as expected. (BZ#1560489)

The updated egress policy needed to block outgoing traffic, patch OVS flows, and then re-enable traffic. However, the OVS flow generation for DNS names was slow. This resulted in a few seconds of egress traffic downtime. With this bug fix, egress policy handling is updated to pre-populate all new OVS flows before blocking the outgoing traffic. This reduces the downtime during egress policy updates. (BZ#1558484)

Due to incorrect cleanup of the internal state, if you deleted a "static per-project egress IPs" from one project and then tried to reuse that IP for a different project, the OVS rules for the new project would be created incorrectly. The egress IP would not be used for the new project, and might start being used again for some traffic from the old project. The internal state is now cleaned up correctly when removing an egress IP and egress traffic works as expected. (BZ#1543786)

When using per-namespace static egress IPs, all external traffic is routed through the egress IP. External means all traffic that is not directed to another pod, and so includes traffic from the pod to the pod’s node. When pods are told to use the node’s IP address for DNS, and the pod is using a static egress IP, then DNS traffic will be routed to the egress node first, and then back to the original node, which might be configured to not accept DNS requests from other hosts, causing the pod to be unable to resolve DNS. Pod-to-node DNS requests now bypass the egress IP and go directly to the node and DNS works. (BZ#1557924)

Previously, errors and warning messages for the oc describe command were not clear. This issue is fixed now. (BZ#1523778)

Previously, the garbage collector tried to delete images that were in use by stopped containers. Changes are made in the OpenShift Container Platform version 3.10, which prevents garbage collector from attempting to remove images in use by stopped containers. (BZ#1577739)

The cpu-cfs-quota used to get applied, even if the node-config.yaml file had cpu-cfs-quota set to false. This happened because the container cgroup for cfs quota was unbound, but the pod level cgroup was bounded. This issue is now fixed, changes were made so that the pod level cgroups remain unbounded. Now if cpu-cfs-quota is set to false, it ignores any limits from being enforced. (BZ#1581409)

The web console was incorrectly assigning extensions/v1beta1 as the API version when creating HPA resources, regardless of the actual group of the scale target. This issue is fixed. (BZ#1543043)

Previously, the HAProxy config failed to load causing router to not service any routes. This was because the Headless service had service.Spec.ClusterIP=None field set, which was not getting ignored as part of un-idling. This is fixed, the HAProxy config ignore headless services during unidle handling and the router service routes as expected. (BZ#1567532)

Path based routes did not work as expected for mixed TLS scenarios. Splitting up of the route types into separate map files caused this issue. Causing haproxy to match the wrong route. Maps are now merged automatically and they are searched appropriately to correctly match the incoming requests with the corresponding backends. (BZ#1534816)

When upgrading the HAProxy Docker image, no logging of requests occurs by default. If logging was requested using the httplog option, a warning message was shown, as this option is not available on a TCP-only connection. In this situation, HAProxy will fall back to using the tcplog option instead. The warning message is therefore harmless and has been removed. (BZ#1533346)

The type: openshift registry adapter does not support discovery of APB images. This means that users of this registry adapter must manually include a list of images to bootstrap. This enhancement introduces the use of a new registry adapter, type: partner_rhcc, which works with https://registry.connect.redhat.com, and supports image discovery without this manual requirement. (BZ#1576881)

When attempting to deprovision a service instance, an error was occurring during the process, combined with an invalid response body, which was causing the deprovision process to fail. Changes have now been implemented to return the proper response body with the operation key, and to enhance the overall robustness of the deprovisioning workflow, which will increase the likelihood of successful deprovisioning. (BZ#1562732)

According to the Open Source Broker (OSB) API documentation, if a binding exists, a status of 200 OK must be returned from a binding call. An issue had been occurring where an incorrect response code (201) was being returned instead. This issue has been fixed by introducing support for asynchronous bindings. (BZ#1563560)

When a service class is removed from a provisioned service instance in the service broker’s catalog, the service catalog marks the class as removedFromBrokerCatalog: true. This prevents the class from being used in new service plans or instances. An issue was preventing this status from being reset to removedFromBrokerCatalog: false if the service class is re-added to the broker catalog, and was preventing removed classes from being used again later. This issue is now resolved. (BZ#1548122)

The Prometheus console did not previously allow access to service catalog metrics, which were only available using curl in the back end. The service catalog controller now exposes metrics for Prometheus to scrape, which enables monitoring of the service catalog. (BZ#1549021)

The capacity of the local persistent storage volume (PV) was being reported in some cases as different to that reported by the df utility. This was due to a lack of propagation of newly mounted devices to the pods, resulting in an additional PV being created for the configured directory. The capacity of this newly created PV was equal to the root device. This propagation issue has now been fixed. (BZ#1490722)

When the API call quota for in the AWS cloud was reached, certain AWS API calls returned errors. These errors were not correctly handled when detaching AWS persistent Volumes, with some AWS volumes remaining attached to the nodes despite there being no pod using them. The volumes had to be detached manually, otherwise they became stuck forever. This bug fix updates the AWS API call error handling for dynamic volume detaching. As a result, even when the AWS API call quota is reached, the attach/detach controller re-tries to detach the volume until it succeeds, ensuring the volumes that should be detached are actually detached. (BZ#1537236)

Previously, when masterConfig.ImagePolicyConfig.ExternalRegistryHostname was added for the master-config.yaml and the API and controller service was restarted, the API pod would recreate, but the controllers pod would error with CrashLoopBackOff. Having an m3.large instance in AWS resolves the issue. (BZ#1593635)

You can now define a set of hooks to run arbitrary tasks during the node upgrade process. To implement these hooks set openshift_node_upgrade_pre_hook, openshift_node_upgrade_hook, or openshift_node_upgrade_post_hook to the path of the task file you wish to execute. The openshift_node_upgrade_pre_hook hook is executed after draining the node and before it has been upgraded. The openshift_node_upgrade_hook is executed after the node has been drained and packages updated but before it is marked schedulable again. The openshift_node_upgrade_post_hook hook is executed after the node has been marked schedulable immediately before moving on to other nodes. (BZ#1559143)

There is one known Kubelet wedge state that will be fixed in the 1.10 rebase where the Kubelet will display messages like system:anonymous cannot access resource foo. This means that the certificates expired before the kubelet could refresh them. If restarting the kubelet does not fix the issue, delete the contents of /etc/origin/node/certificates/, and then restart the kubelet.

The blue-green node deployment method as documented in Upgrading Clusters should only be used for the initial upgrade path from OpenShift Container Platform 3.9 to 3.10. It will be further updated when the first asynchronous OpenShift Container Platform 3.10.z update is released.

In the GA release of the Downgrading OpenShift documentation, an issue was found with the steps for restoring etcd. The document has since been updated and this is no longer an issue.

OpenShift Container Platform 3.10 adds the ability for multiple instances of an APB to be invoked in the same namespace. This new ability requires relying on a globally unique identifier (GUID) for each instance. While instances deployed by an 3.9 version of an APB lack the GUID, 3.10 APBs require it.