$ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets
Updating the operating system (OS) on a host, by either upgrading across major
releases or updating the system software for a minor release, can impact the
OKD software running on those machines. In particular, these updates
can affect the iptables rules or ovs flows that OKD requires to
operate.
To safely upgrade the OS on a host:
Drain the node in preparation for maintenance:
$ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets
In order to protect sensitive packages that do not need to be updated, apply the exclude rules to the host:
# atomic-openshift-docker-excluder exclude # atomic-openshift-excluder exclude
Update the host packages and reboot the host. A reboot ensures that the host is
running the newest versions and means that the docker and OKD
processes have been restarted, which forces them to check that all of the
rules in other services are correct.
# yum update # reboot
However, instead of rebooting a node host, you can restart the services that are
affected or preserve the iptables state. Both processes are described in the
OKD
iptables topic. The ovs flow rules do not need to be saved, but restarting
the OKD node software fixes the flow rules.
Configure the host to be schedulable again:
$ oc adm uncordon <node_name>
If using OpenShift Container Storage, upgrade the OKD nodes running OpenShift Container Storage one at a time.
Run oc get daemonset -n <project_name> to verify the label found under
NODe-SeLeCTOR. The default value is glusterfs=storage-host. To determine what
the pod is, run oc get pods -n <project_name> --selectors=glusterfs=.
Remove the daemonset label from the node:
$ oc label node <node_name> <daemonset_label> -n <project_name>
This will cause the OpenShift Container Storage pod to terminate on that node.
To overwrite the existing label, use the --overwrite flag.
To run the upgrade playbook on the single node where you terminated OpenShift
Container Storage , use -e openshift_upgrade_nodes_label="type=upgrade".
When the upgrade completes, relabel the node with the daemonset label:
$ oc label node <node_name> <daemonset_label> -n <project_name>
Wait for the OpenShift Container Storage pod to respawn and appear.
oc rsh into the gluster pod to check the volume heal:
$ oc rsh <pod_name> $ for vol in `gluster volume list`; do gluster volume heal $vol info; done $ exit
ensure all of the volumes are healed and there are no outstanding tasks. The
heal info command lists all pending entries for a given volume’s heal process.
A volume is considered healed when Number of entries for that volume is 0.
Use gluster volume status <volume_name> for additional details about the
volume. The Online state should be marked Y for all bricks.