OpenShift Container Platform 3.9 Release Notes

OpenShift Container Platform template provisioning

Offline OpenScapScans

Alert management: You can choose Prometheus (currently in Technology Preview) and use it in CloudForms.

Reporting enhancements

Provider updates

Chargeback enhancements

UX enhancememts

A minimal and secure architecture.

Excellent scale and performance.

The ability to run any Open Container Initiative (OCI) or docker image.

Familiar operational tooling and commands.

kubelet_volume_stats_capacity_bytes

kubelet_volume_stats_inodes

kubelet_volume_stats_inodes_free

kubelet_volume_stats_inodes_used

kubelet_volume_stats_used_bytes

Require as much CPU time as possible.

Are sensitive to processor cache misses.

Are low-latency network applications.

Coordinate with other processes and benefit from sharing a single processor cache.

The event timestamp.

The activity that generated the entry.

The API endpoint that was called.

The HTTP output.

The item changed due to an activity, with details of the change.

The user name of the user that initiated an activity.

The name of the namespace the event occurred in, where possible.

The status of the event, either success or failure.

User login and logout from (including session timeout) the web interface, including unauthorized access attempts.

Account creation, modification, or removal.

Account role or policy assignment or de-assignment.

Scaling of pods.

Creation of new project or application.

Creation of routes and services.

Triggers of builds and/or pipelines.

Addition or removal or claim of persistent volumes.

Restructured playbooks to push all fact-gathering and common dependencies up into the initialization plays so they are only called once rather than each time a role needs access to their computed values.

Refactored playbooks to limit the hosts they touch to only those that are truly relevant to the playbook.

prometheus 2.1.0

Alertmanager 0.14.0

AlertBuffer 0.2

node_exporter 0.15.2

Build pods use multiple containers. Binary builds need to specify which container to stream content into, and for custom builds the name of the container is different from non-custom builds. When streaming binary content into a custom build, the expected container, git-clone, does not exist and the build fails. The logic for streaming binary content into a custom build pod will be changed to reference the correct container name, custom-build. With this bug fix, binary content will successfully stream into the custom build container. (BZ#1560659)

Resource constraints can lead to the readiness probe in the example Jenkins templates readiness probes citing failure prematurely. Jenkins deployments would fail unnecessarily. With this bug fix, the readiness probe was relaxed in the templates. As a result, there is a decrease in unnecessary Jenkins deployment failures due to the aggressive readiness probe. (BZ#1559675)

The master admin.kubeconfig file was added to the oc command to allow the operation to have the proper authorization and access to the necessary resources. (BZ#1561247)

The installer improperly tried to set the SELinux context on a path that may not exist. This task was meant to work around a problem in CRI-O that no longer exists and, as such, that task has been removed. (BZ#1564949)

Service catalog pods had a high log verbosity set by default. Therefore, service catalog pods on the master node produced a large amount of log data. The default log verbosity is now reset to a lower level. (BZ#1564179)

The Elasticsearch server TLS certificate does not have an external host name in the subject alt. name list. Clients accessing Elasticsearch externally cannot turn on the MITM server certificate validation. When configuring Elasticsearch to allow external access, add the external host name in the subject alt. name list. TLS clients can turn on server certificate validation. (BZ#1554878)

The Fluentd plug-in logs the entire error response on failure, which fills up the on-disk logs. The entire response is now only logged when in debug mode and on-disk logs no logger consume the disk. (BZ#1554885)

The default write operation for Fluentd to Elasticsearch is index. Writes can trigger unnecessary delete operations for Elasticsearch, causing extra load that affects performance. Use the create operation. Writes to elasticsearch will only create records or skip updates if the records are duplicates reducing the load on the server. (BZ#1565909)

The curator pod was crash-looping because it was unable to find its entry point script due to a bad merge from origin into downstream dist-git. The pod was not functional and cycled crash-looping. With this bug fix, the code was synced with upstream. (BZ#1572419)

The Fluentd secure-forward plug-in supports the host name placeholder ${hostname} in the configuration file. Although the value is case-sensitive, the upper case string ${HOSTNAME} was set and it failed to pick up the correct hostname of the Fluentd container. The bug is now fixed. (BZ#1553576)

After manually typing a URL with non-existing image, page load messaging would remain on the page, signaling that the page load is ongoing, even though it is done and the The image stream details could not be loaded alert is shown. Set the loaded scope variable when the image is or is not loaded and use it in the view to hide the loading messaging. After the attempt to load the image data, the loading messaging is now hidden, even if the image cannot be loaded. (BZ#1550797)

Previously, the web console would not let you add new keys when editing a ConfigMap that was empty. Clicking Add Item in the editor would have no effect. With this bug fix, you can now correctly add items when editing a ConfigMap that has none. (BZ#1558863)

Restricting DaemonSet nodes with the project’s default node selector resulted in the deletion and creation of DaemonSet pods in a loop on those nodes that were restricted by adding project default node selector. With this bug fix, the upstream DaemonSet logic is now updated to be aware of the project’s default node selector. (BZ#1571093)

The Hawkular Alerts components has been removed from Hawkular Metrics. This change has no functional impact on Hawkular Metrics. (BZ#1543647)

Previously there was incorrect management of OVS flows. If two nodes rebooted and swapped IP addresses when they came back up, then other nodes might not be able to send traffic to pods on one or both of those nodes. The code that manages OVS flows is now more careful to make the correct changes in cases of node IP reassignment. Pod-to-pod traffic should continue to work correctly even after nodes swap IP addresses. (BZ#1570394)

The update Egress policy needed blocking outgoing traffic, patching OVS flows, and then re-enabling traffic. However, the OVS flow generation for DNS names was slow. This resulted in a few seconds of Egress traffic downtime, which may not be acceptable. With this bug fix, update Egress policy handling is updated to pre-populate all new OVS flows before blocking the outgoing traffic. This reduces the downtime during Egress policy updates. (BZ#1571430)

When using per-namespace static egress IPs, all external traffic is routed through the egress IP. External means all traffic, which is not directed to another pod, and so this includes traffic from the pod to the pod’s node. When pods are told to use the node’s IP address for DNS, and the pod is using a static egress IP, then DNS traffic will be routed to the egress node first, and then back to the original node, which might be configured to not accept DNS requests from other hosts, causing the pod to be unable to resolve DNS. Pod-to-node DNS requests now bypass the egress IP and go directly to the node and DNS works. (BZ#1570398)

This bug fix addresses an issue on the node where setting disabling cpu-cfs-quota did not prevent CPU CFS limits from being set on pods when cgroups-per-qos was enabled. (BZ#1558155)

This bug fix addresses an issue where clusters running with OpenStack cloud integration have nodes removed when the corresponding instance is stopped. Node resources whose instances are stopped are no longer removed from the cluster. (BZ#1558422)

Nodes entered an impaired state when a volume is forcefully detached and not rebooted. Any new volume attached to the node is stuck in an attaching state. Any node that has a volume stuck in an attaching state for more than 21 minutes will be tainted and must be removed from cluster, then added back to remove the taint and fix the impaired state of the node. With this bug fix, impaired are removed from scheduling, giving the OpenShift Container Platform administrator the ability to fix the node and bring it back. (BZ#1455680)

Previous releases of OpenShift Container Platform would improperly reconfigure docker to mark the internal registry as insecure when it should not have. This has been fixed in OpenShift Container Platform 3.9 and should no longer happen. (BZ#1502028)

Use CRI-O as an RPM to use CRI-O as the container runtime. To install CRI-O as an RPM, set the following two options:

openshift_use_crio=True
openshift_crio_use_rpm=True

The yedit module now generates unique backup files. Previously, if changes were made to the same resource multiple times, only the latest diff would be saved. (BZ#1555426)

Administrators can now see messages for which we are unable to determine the proper namespace to associate with them. Otherwise, messages appear to be missing and are not viewable for review. A Kibana Index pattern will be created for administrators if it does not exist. (BZ#1519522)

In the absence of inventory values, reuse the values used for the current deployment to preserve tuned values. In the case of Elasticsearch, when a user had done tuning of the cluster but did not propagate those values into variables, upgrading logging would use role default values, which may put the cluster in a bad state and lead to loss of log data. Values are now honored in order for EFK: inventory → existing environment → role defaults. (BZ#1561196)

The number of Kibana index-patterns for cluster administrators is now limited. Previously, the list was unmanageable and unneeded on large clusters with many namespaces. Cluster administrators now only see a limited subset of index-patterns. (BZ#1563230)

Jenkins no_proxy processing could not handle suffixes like ".svc". As a result, communication between a Jenkins Kubernetes agent pod and the Jenkins master would attempt to go through a configured http_proxy and fail. With this bug fix, the OpenShift Container Platform jenkins agent images are updated to automatically include the jenkins master and jnlp hosts in the no_proxy list. The Jenkins limitation for no_proxy processing is now circumvented. (BZ#1578989)

When creating the Elasticsearch server certificate, the external Elasticsearch host names were unconditionally added to the subjectAltName. Installation would fail because only host name components beginning with a letter are allowed in the subjectAltName, so host names like es.0xdeadbeef.com were disallowed and would cause an error. A warning is now issued if the Elasticsearch host name contains a component which does not begin with a letter, and it is not added to the subjectAltName. Logging installation now completes successfully. (BZ#1567767)

The plug-in only caught the KubeException, but not more general exceptions. Therefore, consumers were stuck cycling until the API server could be contacted. Metadata fetch is now more relaxed and gracefully catches the exception, returning no metadata, and subsequently the record is orphaned. (BZ#1560170)

logging-elasticsearch-ops was missing in the delete` configmaps` list in the openshift-ansible delete_logging role. The logging-elasticsearch-ops configmap still exists after running the uninstall Ansible playbook for logging. logging-elasticsearch-ops is added to the delete configmaps list. All of the logging configmaps including logging-elasticsearch-ops are now uninstalled by running the uninstall Ansible playbook for logging. (BZ#1549220)

The Create Project button was incorrectly displayed to users when they had no projects and self-provisioning had been disabled on the projects list page of the web console. The action would always fail, so the button should have been hidden. The bug is now fixed, and Create Project is now correctly hidden in the console when self-provisioning is disabled. (BZ#1577359)

This bug fix addresses an issue pulling images from a private docker hub registry. (BZ#1578088)

This bug fix addresses where cfs_quota might still be set on a pod even when cpu-cfs-quota is set to false on the node. (BZ#1581860)

Users are now allowed to disable JSON payload parsing. Parsing each log message into JSON and attaching it to the final payload is an expensive operation. Fluentd can now be configured to disable parsing of message payloads. This is the initial configuration change to deprecating the feature from the fluent-plugin-kubernetes_metadata_filter. (BZ#1569825)

The webhook payload can contain an empty commit array, which results in an array indexing error when processed by the API server. As a result, the API server crashes. Check for an empty array before attempting to index into it. With this bug fix, empty commit payloads are handled without crashing the API server. (BZ#1586076)

A secret with a wrong password causes pull failures for all images. Any public image from the same registry pull will fail. This bug fix adds retry logic for the 401 error when the password is wrong. Now, if the image is public, the image is pulled and the wrong secret is ignored. (BZ#1506175)

The openshift-jenkins-sync plug-in assumed the Jenkins service and pipeline strategy build were in the same project when constructing the build URL for the OpenShift Container Platform web console. When Jenkins is in one project and the pipeline strategy build is in another project, the view log link in the OpenShift Container Platform web console points to the wrong URL because if cannot find the Jenkins service/route. The openshift-jenkins-sync plug-in now looks for the Jenkins service/route in the namespace it is running in. Also, if the user has explicitly configured the root URL in Jenkins, there is greater precedence. The URL for a given pipeline strategy build in the OpenShift Container Platform web console now renders correctly. (BZ#1542460)

Image validation used to validate an old image object and the image signature import controller would generate such an image. As a result, invalid images were pushed to etcd. With this big fix, validation is changed to validate a new image object and logic to fix some invalid images is now introduced. The controller no longer generates invalid images and it is no longer possible to upload an invalid image object. (BZ#1560311)

The transfer of plug-ins from the RPM installation location to the Jenkins home directory were not occurring properly with the OpenShift Container Platform v2 Jenkins RHEL image when Jenkins was previously deployed on an OpenShift Container Platform pod with a persistent volume. An upgrade of the OpenShift Container Platform v2 Jenkins RHEL image would not result in the deployment having the most recent plug-ins associated with the newer image. TheOpenShift Container Platform v2 Jenkins RHEL image run script is now updated to properly transfer the plug-ins. An upgrade of the OpenShift Container Platform v2 Jenkins RHEL image now results in the deployment having the most recent plug-ins associated with the newer image. (BZ#1550193)

If the Jenkins root URL could not be retrieved from the route from the Jenkins template, then the unusable URL could be used in constructing the various annotations for pipeline builds. The associated annotation links would not render when referenced from the OpenShift Container Platform web console. To help account for those edge cases, the sync plug-in now looks for explicitly configured root URLs in Jenkins. The links associated with the pipeline build annotations now render if the root URL is properly configured. (BZ#1558997)

Allowed registries for import configuration settings were considered only for image imports. You could easily get around the image import validation by editing image streams manually and use any desired image. With this bug fix, image streams are now also validated. You cannot use an external image that does not match an entry in whitelisted registry entries. (BZ#1505315)

In certain cases, an existing etcd installation might not have updated configuration variables, causing services to fail. This bug fix ensures the etcd.conf file is verified during upgrades and that all variables are set as expected. (BZ#1529575)

To enable support for storage devices on Microsoft Azure, the seboolean virt_use_samba is required. (BZ#1537872)

The node configuration file had hardcoded labels in the CRI-O section. Therefore, double labels could occur if labels were set elsewhere in the installer. Remove the unecessary hardcoded labels, eliminatingthe possibility of double labels. (BZ#1553012)

The secure-forward template generated in the configMap does not include the <store> tag, as mentioned in the documentation. The configuration fails when more stores are defined. Add the enclosing <store> tag for the template. Removing the comments provides a syntactically valid configuration. (BZ#1498398)

To label nodes for Fluentd, a scrip was run out of /tmp. When the noexec option was set for /tmp, the playbook failed. Instead of running a script where paused, label with a pause using the shell Ansible task. With this bug fix, you are able to pause and run to completion. (BZ#1588009)

There were changes to the kube-proxy iptables rules in upstream Kubernetes. Network performance and overall system performance was severely impacted on extremely large clusters like OpenShift Online. With this bug fix, there are multiple optimizations of the kube-proxy iptables rule and performance problems are resolved. (BZ#1514174)

A version of the OVS RPM was used that did not have the right SELinux policy. Therefore, OVS failed due to SELinux. Get the correct version of the OVS RPM with the correct rules. With this bug fix, OVS now works. (BZ#1548677)

When using the static per-project egress IPs feature, egress IPs may stop working in some circumstances if an egress IP is moved from one project to another, or from one node to another. Additionally, if the same egress IP is assigned to two different projects, or two different nodes, then it might not work correctly, even after the duplicate assignment is removed. This bug fix resolves the issue and static per-project egress IPs should work more reliably. (BZ#1553294)

OpenShift Container Platform’s default network plug-in has not been updated to implement the new NetworkPolicy features introduced upstream in Kubernetes (policies for controlling egress, and policies based on IP addresses rather than pods or namespaces). Therefore, in OpenShift Container Platform 3.9, creating a NetworkPolicy with an ipBlock section would cause nodes to crash, and creating a NetworkPolicy that contained only egress rules would erroneously cause ingress traffic to be blocked. The code is now aware of the unsupported NetworkPolicy features, though it does not yet implement them. If a NetworkPolicy contains ipBlock rules, those rules are ignored. This may cause the policy to be treated as deny all if the ipBlock rule was the only rule in the policy. If a NetworkPolicy contains only egress rules, it is ignored completely and does not affect ingress. (BZ#1585243)

There was an regression issue in which the docker client in use by the kubelet qualifies image paths without a domain with docker.io client-side, resulting in all unqualified image paths attempting the pull from docker.io and ignoring the domain search list in the docker daemon. With this bug fix, the regression issue is resolved. (BZ#1588768)

Unbinding a template service instance throws an error if the template service instance was deleted. It becomes impossible to unbind a service instance if the template service instance was manually deleted, including if the project containing the TSI was deleted. The template service broker will return success/gone in cases where the unbind refers to a non-existent template service instance. The unbind can now proceed even if the TSI no longer exists. (BZ#1540819)

When deleting a namespace, the objects within the namespace are deleted by the namespace controller, not the user. Service bindings, when deleted, get unbound via an unbind request associated with the user doing the deletion. This leads to an unbind request coming from the namespace controller, which did not have all permissions required to perform an unbind. Change what permissions are required for unbind to align them with the permissions the namespace controller has. The unbind triggered by the namespace controller deleting the binding will succeed. (BZ#1554141)

This bug fix adds a small compatibility check to eliminate a pain point with API endpoints changing from 3.7 to 3.9. (BZ#1554145)

You may now define a set of hooks to run arbitrary tasks during the node upgrade process. To implement these hooks, set openshift_node_upgrade_pre_hook, openshift_node_upgrade_hook, or openshift_node_upgrade_post_hook to the path of the task file you want to execute. The openshift_node_upgrade_pre_hook hook is executed after draining the node and before it is upgraded. The openshift_node_upgrade_hook is executed after the node has been drained and packages updated but before it is marked schedulable again. The openshift_node_upgrade_post_hook hook is executed after the node is marked schedulable immediately before moving on to other nodes. (BZ#1572786)

Improper input validation of the OpenShift Container Platform routing configuration can cause an entire shard to be brought down. A malicious user can use this vulnerability to cause a Denial of Service attack for other users of the router shard. (BZ#1553035)

OpenShift and Atomic Enterprise Ansible deploys a misconfigured etcd file that causes the SSL client certificate authentication to be disabled. Quotations around the values of ETCD_CLIENT_CERT_AUTH and ETCD_PEER_CLIENT_CERT_AUTH in etcd.conf result in etcd being configured to allow remote users to connect without any authentication if they can access the etcd server bound to the network on the master nodes. An attacker could use this flaw to read and modify all the data about the OpenShift Container Platform cluster in the etcd datastore, potentially adding another compute node, or bringing down the entire cluster. (BZ#1557822)

A privilege escalation flaw was found in the source-to-image component of OpenShift Container Platform, which allows the assemble script to run as the root user in a non-privileged container. An attacker can use this flaw to open network connections, and possibly other actions, on the host which are normally only available to a root user. (BZ#1579096) (BZ#1579096)

A new flag is now added to the oc adm drain command to allow you to select nodes by label. There was a need to be able to drain multiple nodes, without having to perform the drain operation on each individual node. The oc adm drain command now supports a --selector flag, which results in all nodes matching a given label being drained. (BZ#1466390)

The link generation code assumed all project logs are written to indices that have a common naming pattern. Therefore, users were linked to non-existent indices. With this bug fix, project logs that will be archived to different indices are annotated with the required information to properly build the link. Users are now routed using a link that will query the data store correctly and return data. (BZ#1523047)

The service catalog pods were previously labeled with node selectors that were limited to the first master. Now, the pods are properly labeled with a label that applies to all masters, which enables service catalog pods to deploy to all masters properly. (BZ#1554623)

When adding IP route and host name packages to the openshift-ansible image, Ansible uses common user-space utilities for determining default facts. The ansible_default_ipv4.address fact is populated using utilities from the iproute package, and this fact is used for populating the OpenShift Container Platform IP in roles/openshift_facts/library/openshift_facts.py. (BZ#1558689)

Due to corrupt file chunk buffers, Fluentd was blocked processing messages until the buffer was removed. This bug fix introduces a handler to remove corrupt messages from the processing work flow. Corrupt messages are now sent to a dead letter queue while continuing to process other messages and the pipleline is no longer blocked. (BZ#1562004)

Curator checked the readiness of Elasticsearch at startup. If Elasticsearch was not ready after one minute, Curator gave up. This was repeated 5 times with exponential backoff via pod restart policy with default backofflimit=5. Curator could not be deployed without Elasticsearch. Now, Curator checks for Elasticsearch readiness indefinitely before each run. Curator and Elasticsearch can be deployed independently. (BZ#1564350)

Headless service had service.Spec.ClusterIP=None set and this was not ignored as part of unidling. This generates an incorrect endpoint ID in the HAProxy configuration and the configuration will fail to load. This leads to the router not servicing any routes. Headless services are now ignored during unidle handling and, therefore, there is now no problem with HAProxy configuration loading. The router will service routes as expected. (BZ#1571479)

The incoming data had a field record["event"] that is a String value and not the hash value expected by the transform_eventrouter code. This causes the code to throw an error and fluentd to emit an error like:

error_class=NoMethodError error="undefined method `key?' for \"request\":String"

Recently, cloudResourceSyncManager was implemented, which continuously fetched node addresses from cloud providers. The kubelet then received node addresses from the cloudResourceSyncManager. At the time of node registration or kubelet start, the kubelet fetches node addresses in a blocking loop from cloudResourceSyncManager. The issue was that cloudResourceSyncManager was not started before the kubelet had started fetching node addresses from it for the first time. Due to this, the kubelet got stuck in the blocking loop and never returned. It caused node failures at the network level, and no node could be registered. Also, as the kubelet was blocked early, the cloudResourceSyncManager never got a chance to start. The cloudResourceSyncManager is now started early in the kubelet startup process so that the kubelet does not get blocked on it and cloudResourceSyncManager is always started. (BZ#1601813)

Ansible 2.6.0 will not evaluate undefined variables with |bool as false. You must define a | default(false) for logging_elasticsearch_rollout_override. Specify a default on the when condition. The playbook executes successfully. (BZ#1602015)

This feature allows the number of indices and replicas to be configured using environment variables. The logs collected for infra services consumes a large portion of the available disk space. Spreading the data across available nodes by modifying the replica and shard settings allow Elasticsearch to better support these large amounts of data. This feature results in improved performance in Elasticsearch when there are large amounts of data from infra services. (BZ#1553257)

There were several problems related to updates. Spec changes for instances were blocked, even if there was not an ongoing operation; deleting a service instance that was updated to an invalid service plan would cause a crash; instances were not updated properly if a previous update had failed. (BZ#1507595)

The wizard was setting the valid state prior to retrieving data. The primary button was set to Bind due to prematurely setting the wizard state to valid. Do not set the wizard state to valid until the data is retrieved and you determined that more steps are necessary. At start up, the wizard is now not in the valid state. (BZ#1520828)

The playbook for the openshift-metrics component was not verifying the return code of some commands. Some incorrect parameters could be used to set up metrics. The playbook succeeded, but metrics were incorrectly set up. With this bug fix, the playbook verifies exit code for oc apply commands. Installation fails when incorrect commands are used. (BZ#1534538)

During upgrade, all masters were restarted simultaneously. In a multi-master environment, leader election could have failed A short pause is now added between master restarts and leader election no longer fails. (BZ#1540054)

Unicode characters in pull specs for images were treated incorrectly. Installation stopped with a cryptic message. Unicode handling in image paths is now fixed and a correct error message is displayed. (BZ#1544648)

The default node selector for Elasticsearch was not set. Installation with default settings and enabled logging failed. Correct handling of a default node selector is now implemented. The logging installation completes without a node selector set. (BZ#1547210)

There were bugs in the per-project static IP code. If you removed the static IP from a project and then re-added it, it would not always work correctly. With this bug fix, removing and re-adding static egress IPs now works. (BZ#1547899)

Patch files for logging ConfigMaps were incorrectly generated. There were failures when applying the file because reference lines were not there. This bug fix changes how white-listed lines were handled. They are prevented from ending up in patch files that were generated, but the patch is still allowed to be applied after. Logging ConfigMaps are now correctly patched based on changes from current deployments. (BZ#1552744)

When fluentd is configured as the combination of collectors and MUX, event logs from the event were supposed to be processed by MUX, not by the collector for the both MUX_CLIENT_MODE maximal and minimal. This occurred because if an event log is formatted in the collector (the event record is put under the Kubernetes key), the log is forwarded to MUX and passed to the k8s-meta plug-in there and the existing Kubernetes record is overwritten. It removed the event information from the log. + Fix 1: To avoid the replacement, if the log is from event router, the tag is rewritten to ${tag}.raw in input-post-forward-mux.conf, which treats the log in the MUX_CLIENT_MODE=minimal way. + Fix 2: There was another bug in Ansible in that the environment variable TRANSFORM_EVENTS was not set in MUX even if openshift_logging_install_eventrouter was set to true. + With these two fixes, the event logs are correctly logged when MUX is configured with MUX_CLIENT_MODE=maximal as well as minimal. (BZ#1554293)

The ability to skip disabling swap by use of openshift_disable_swap=False is removed from OpenShift Container Platform 3.9. This feature was undocumented and should not be used. (BZ#1557200)

The kube-proxy and kubelet parts of the OpenShift Container Platform node process were being given different default values for the config options describing how to interact with iptables. OpenShift Container Platform would periodically add a bogus iptables rule that would cause some per-project static egress IPs to not be used for some length of time, until the bogus rule was removed again. While the bogus rule was present, traffic from those projects would use the node IP address of the node hosting the egress IP, rather than the egress IP itself. The inconsistent configuration was resolved, causing the bogus iptables rule to no longer be added. Projects now consistently use their static egress IPs. (BZ#1560584)

A node selector for TSB was not documented. Therefore, users were not aware that template_service_broker_selector and openshift_hosted_infra_selector could be changed. With this bug fix, hosts.example now lists openshift_hosted_infra_selector and template_service_broker_selector. TSB and the infra node selector is now more visible. (BZ#1569165)

Kibana index pattern seeding was modified to reduce the number of entries in the drop-list that were visible to cluster admins. Cluster admins navigating to Kibana for logging from the web console were no longer sent to the correct URL. Take into consideration a user’s cluster-admin role when creating the web console link to pod logs in Kibana. With this bug fix, the link is now created correctly. (BZ#1598305)

The unstaller attempted to deploy node_exporter during a Prometheus uninstall. Uninstall failed due to node_exporter not being able to deploy to the Prometheus namespace. This bug fix adds a check to deploy node_exporter if Prometheus is installed and present. Now, Prometheus uninstall completes successfully without attempting to install node_exporter into the non-existent namespace. (BZ#1598642)

In certain environments, the image stream content changes rapidly enough to prevent replacement with new content due to the object being modified. The upgrade would fail. The image stream update now retries three times. With this bug fix, the upgrade should be successful. (BZ#1623642)

When wildcard routes are enabled and namespace ownership checks are disabled, non-wildcard routes get removed and immediately re-added on the resync interval boundaries and this causes a brief route outage and results in intermittent errors on a route. Do not remove and re-add the routes on resync interval in the specific case when wildcard routes enabled and namespace ownership checks are disabled. With this bug fix, non-wildcard routes continue to serve without any intermittent errors. (BZ#1624078)

Javascript runtime errors are caused by accessing an undefined key on the secrets object. Use the lodash _.get() method, which will not error out if accessing an undefined key on the secrets object. The secrets drop-down is now a visible event if there is a secret without any data. (BZ#1633285)

openshift_facts uses default images to determine the version in a containerized installation. When custom images and the internal registry is used, the default images cannot be pulled. Therefore, openshift.common.version is left empty and the upgrade fails when OpenShift Container Platform is not installed. If there is a custom_image name, pass it to openshift_facts to use. With this bug fix, openshift.common.version is set in containerized installations with the registry mirror and installation succeeds. (BZ#1634004)

Systemd has a default service timeout of 90 seconds and, in some cases, ovsdb-server does not start up in that time. Therefore, openvswitch, which has a dependency on ovsdb-server, cannot start and installation fails. Increase systemd timeout for ovsdb-server to five minutes. As a result, ovsdb-server does not timeout and openvswitch starts successfully. (BZ#1636234)

The OSB Client Library used by the Service Catalog controller pod was not closing and freeing TCP connections used to communicate with brokers. Over a period of time, many TCP connections would remain open and, eventually, the communication between the Service Catalog controller and brokers would fail. Additionally, the pod would become unresponsive. Use a new version of the OSB Client Library, which contains a fix to close connections when finishing a HTTP request and free idle connections. (BZ#1638726)

When using docker with log-driver journald, the setting in /etc/sysconfig/docker has changed to use --log-driver journald instead of --log-driver=journald. Fluentd cannot detect that journald is being used, so assumes json-file, and cannot read any kubernetes metadata because it does notlook for the journald CONTAINER_NAME field. This results in lots of Fluentd errors. Change the way Fluentd detects the docker log driver so that it looks for --log-driver journald in addition to `--log-driver=journald. Fluentd can now detect the docker log driver, and can correctly process Kubernetes container logs. (BZ#1638900)

The master API sent an error without details, but the registry was not ready for this and the registry panicked. Check if the details field is available. The registry sends a proper error to the client and logs the full error from the master API. (BZ#1639839)

Running oc logs $fluentd_pod suggests that you run oc exec <pod_name> /opt/app-root/src/utils/logs, which includes the non-existing utility logs path. Therefore, oc exec <pod_name> /opt/app-root/src/utils/logs fails with no such file or directory. This bug fix updates the suggested command line to oc exec <pod_name> — logs since the utility logs is now in the PATH and there is no need to specify the full path. As a result, oc logs $fluentd_pod suggests the correct command line to show the fluentd logs. (BZ#1643340)

There was inconsistent user usage when creating and removing temporary directories. Removing the directories can fail if the user does not have proper access. With this bug fix update, tasks use same become:false settings and the removal does not fail when using different user. (BZ#1643732)

Unnecessarily short timeout resulted in a failure to reuse artifacts from a previous build when incremental builds were selected with s2i. This could occur when the size of the artifacts being reused was particularly large or the host system was running particularly slowly. Invalid artifacts could be used in a subsequent build, or artifacts would be recreated instead of reused resulting in performance degradation. Timeout has been increased to a sufficiently large value as to avoid this problem. Artifact reuse should no longer timeout. (BZ#1644343)

Prometheus was automatically discovering two separate scrape endpoints for the Kubernetes router. One of the endpoints was not able to be scraped due to a missing auth config. This appeared as a failing scrape in the Prometheus status. This bug fix removed the duplicate prometheus scrape endpoint for the Kubernetes router. Now, Prometheus only scrapes a single, working endpoint for the Kubernetes router. (BZ#1644544)

Egress IP-related iptables rules were not recreated if they were deleted. If a user restarted firewalld or iptables.service on a node that hosted egress IPs, then those egress IPs would stop working. Traffic that should have used the egress IP would use the node’s normal IP instead. Egress IP iptables rules are now recreated if they are removed and egress IPs work reliably. (BZ#1653382)

Previously, the metrics playbooks attempted to verify the availability of the python-passlib library in a manner that was unpredictable. This library is also a dependency of the openshift-ansible packaging, so there is no need for this additional check and it is now removed. (BZ#1654887)

master nodes are now schedulable. Web console pods are now restricted to only run on masters. Therefore, master nodes are no longer marked as non-schedulable. (BZ#1535673)

The APB tool now works with Minishift. (BZ#1536687)

There is a regression error when running on OpenStack. The function normalize_openstack_facts pulled in a commit that broke how the inventory file is parsed when hosted on OpenStack. When using OpenStack without a floating IP for your cluster, you will see an error. To address this issue, you must manually patch the /usr/share/ansible/openshift-ansible/playbooks/init/cluster_facts.yml file with:

--- /usr/share/ansible/openshift-ansible/roles/openshift_facts/library/openshift_facts.py       2019-01-14 20:09:27.723830675 +0000
+++ /usr/share/ansible/openshift-ansible/roles/openshift_facts/library/openshift_facts.py.fixed 2019-01-14 17:08:33.433853271 +0000
@@ -337,12 +337,12 @@
     for f_var, h_var, ip_var in [('hostname', 'hostname', 'local-ipv4'),
                                  ('public_hostname', 'public-hostname', 'public-ipv4')]:
         try:
-            if socket.gethostbyname(metadata['ec2_compat'][h_var]) == metadata['ec2_compat'][ip_var].split(',')[0]:
+            if socket.gethostbyname(metadata['ec2_compat'][h_var]) == metadata['ec2_compat'][ip_var]:
                 facts['network'][f_var] = metadata['ec2_compat'][h_var]
             else:
-                facts['network'][f_var] = metadata['ec2_compat'][ip_var].split(',')[0]
+                facts['network'][f_var] = metadata['ec2_compat'][ip_var]
         except socket.gaierror:
-            facts['network'][f_var] = metadata['ec2_compat'][ip_var].split(',')[0]
+            facts['network'][f_var] = metadata['ec2_compat'][ip_var]

     return facts