Deleting a machine

Deleting a specific machine
Lifecycle hooks for the machine deletion phase
Additional resources

You can delete a specific machine.

Deleting a specific machine

You can delete a specific machine.

Do not delete a control plane machine unless your cluster uses a control plane machine set.

Prerequisites

Install an OpenShift Container Platform cluster.
Install the OpenShift CLI (oc).
Log in to oc as a user with cluster-admin permission.

Procedure

View the machines that are in the cluster by running the following command:
```
$ oc get machine -n openshift-machine-api
```
The command output contains a list of machines in the <clusterid>-<role>-<cloud_region> format.
Identify the machine that you want to delete.

Delete the machine by running the following command:

$ oc delete machine <machine> -n openshift-machine-api

By default, the machine controller tries to drain the node that is backed by the machine until it succeeds. In some situations, such as with a misconfigured pod disruption budget, the drain operation might not be able to succeed. If the drain operation fails, the machine controller cannot proceed removing the machine.

You can skip draining the node by annotating machine.openshift.io/exclude-node-draining in a specific machine.

If the machine that you delete belongs to a machine set, a new machine is immediately created to satisfy the specified number of replicas.

Lifecycle hooks for the machine deletion phase

Machine lifecycle hooks are points in the reconciliation lifecycle of a machine where the normal lifecycle process can be interrupted. In the machine Deleting phase, these interruptions provide the opportunity for components to modify the machine deletion process.

Terminology and definitions

To understand the behavior of lifecycle hooks for the machine deletion phase, you must understand the following concepts:

Reconciliation

Reconciliation is the process by which a controller attempts to make the real state of the cluster and the objects that it comprises match the requirements in an object specification.

Machine controller

The machine controller manages the reconciliation lifecycle for a machine. For machines on cloud platforms, the machine controller is the combination of an OpenShift Container Platform controller and a platform-specific actuator from the cloud provider.

In the context of machine deletion, the machine controller performs the following actions:

Drain the node that is backed by the machine.
Delete the machine instance from the cloud provider.
Delete the Node object.

Lifecycle hook

A lifecycle hook is a defined point in the reconciliation lifecycle of an object where the normal lifecycle process can be interrupted. Components can use a lifecycle hook to inject changes into the process to accomplish a desired outcome.

There are two lifecycle hooks in the machine Deleting phase:

preDrain lifecycle hooks must be resolved before the node that is backed by the machine can be drained.
preTerminate lifecycle hooks must be resolved before the instance can be removed from the infrastructure provider.

Hook-implementing controller

A hook-implementing controller is a controller, other than the machine controller, that can interact with a lifecycle hook. A hook-implementing controller can do one or more of the following actions:

Add a lifecycle hook.
Respond to a lifecycle hook.
Remove a lifecycle hook.

Each lifecycle hook has a single hook-implementing controller, but a hook-implementing controller can manage one or more hooks.

Machine deletion processing order

In OpenShift Container Platform 4.14, there are two lifecycle hooks for the machine deletion phase: preDrain and preTerminate. When all hooks for a given lifecycle point are removed, reconciliation continues as normal.

The sequence of events in the machine `Deleting` phase.

Figure 1. Machine deletion flow

The machine Deleting phase proceeds in the following order:

An existing machine is slated for deletion for one of the following reasons:
- A user with cluster-admin permissions uses the oc delete machine command.
- The machine gets a machine.openshift.io/delete-machine annotation.
- The machine set that manages the machine marks it for deletion to reduce the replica count as part of reconciliation.
- The cluster autoscaler identifies a node that is unnecessary to meet the deployment needs of the cluster.
- A machine health check is configured to replace an unhealthy machine.
The machine enters the Deleting phase, in which it is marked for deletion but is still present in the API.
If a preDrain lifecycle hook exists, the hook-implementing controller that manages it does a specified action.

Until all preDrain lifecycle hooks are satisfied, the machine status condition Drainable is set to False.
There are no unresolved preDrain lifecycle hooks and the machine status condition Drainable is set to True.
The machine controller attempts to drain the node that is backed by the machine.
- If draining fails, Drained is set to False and the machine controller attempts to drain the node again.
- If draining succeeds, Drained is set to True.
The machine status condition Drained is set to True.
If a preTerminate lifecycle hook exists, the hook-implementing controller that manages it does a specified action.

Until all preTerminate lifecycle hooks are satisfied, the machine status condition Terminable is set to False.
There are no unresolved preTerminate lifecycle hooks and the machine status condition Terminable is set to True.
The machine controller removes the instance from the infrastructure provider.
The machine controller deletes the Node object.

Deletion lifecycle hook configuration

The following YAML snippets demonstrate the format and placement of deletion lifecycle hook configurations within a machine set:

YAML snippet demonstrating a preDrain lifecycle hook

apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
  ...
spec:
  lifecycleHooks:
    preDrain:
    - name: <hook_name> (1)
      owner: <hook_owner> (2)
  ...

1	The name of the `preDrain` lifecycle hook.
2	The hook-implementing controller that manages the `preDrain` lifecycle hook.

YAML snippet demonstrating a preTerminate lifecycle hook

apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
  ...
spec:
  lifecycleHooks:
    preTerminate:
    - name: <hook_name> (1)
      owner: <hook_owner> (2)
  ...

1	The name of the `preTerminate` lifecycle hook.
2	The hook-implementing controller that manages the `preTerminate` lifecycle hook.

Example lifecycle hook configuration

The following example demonstrates the implementation of multiple fictional lifecycle hooks that interrupt the machine deletion process:

Example configuration for lifecycle hooks

apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
  ...
spec:
  lifecycleHooks:
    preDrain: (1)
    - name: MigrateImportantApp
      owner: my-app-migration-controller
    preTerminate: (2)
    - name: BackupFileSystem
      owner: my-backup-controller
    - name: CloudProviderSpecialCase
      owner: my-custom-storage-detach-controller (3)
    - name: WaitForStorageDetach
      owner: my-custom-storage-detach-controller
  ...

1	A `preDrain` lifecycle hook stanza that contains a single lifecycle hook.
2	A `preTerminate` lifecycle hook stanza that contains three lifecycle hooks.
3	A hook-implementing controller that manages two `preTerminate` lifecycle hooks: `CloudProviderSpecialCase` and `WaitForStorageDetach`.

Machine deletion lifecycle hook examples for Operator developers

Operators can use lifecycle hooks for the machine deletion phase to modify the machine deletion process. The following examples demonstrate possible ways that an Operator can use this functionality.

Example use cases for `preDrain` lifecycle hooks

Proactively replacing machines: An Operator can use a preDrain lifecycle hook to ensure that a replacement machine is successfully created and joined to the cluster before removing the instance of a deleted machine. This can mitigate the impact of disruptions during machine replacement or of replacement instances that do not initialize promptly.
Implementing custom draining logic: An Operator can use a preDrain lifecycle hook to replace the machine controller draining logic with a different draining controller. By replacing the draining logic, the Operator would have more flexibility and control over the lifecycle of the workloads on each node.

For example, the machine controller drain libraries do not support ordering, but a custom drain provider could provide this functionality. By using a custom drain provider, an Operator could prioritize moving mission-critical applications before draining the node to ensure that service interruptions are minimized in cases where cluster capacity is limited.

Example use cases for `preTerminate` lifecycle hooks

Verifying storage detachment: An Operator can use a preTerminate lifecycle hook to ensure that storage that is attached to a machine is detached before the machine is removed from the infrastructure provider.
Improving log reliability: After a node is drained, the log exporter daemon requires some time to synchronize logs to the centralized logging system.

A logging Operator can use a preTerminate lifecycle hook to add a delay between when the node drains and when the machine is removed from the infrastructure provider. This delay would provide time for the Operator to ensure that the main workloads are removed and no longer adding to the log backlog. When no new data is being added to the log backlog, the log exporter can catch up on the synchronization process, thus ensuring that all application logs are captured.

Quorum protection with machine lifecycle hooks

For OpenShift Container Platform clusters that use the Machine API Operator, the etcd Operator uses lifecycle hooks for the machine deletion phase to implement a quorum protection mechanism.

By using a preDrain lifecycle hook, the etcd Operator can control when the pods on a control plane machine are drained and removed. To protect etcd quorum, the etcd Operator prevents the removal of an etcd member until it migrates that member onto a new node within the cluster.

This mechanism allows the etcd Operator precise control over the members of the etcd quorum and allows the Machine API Operator to safely create and remove control plane machines without specific operational knowledge of the etcd cluster.

Control plane deletion with quorum protection processing order

When a control plane machine is replaced on a cluster that uses a control plane machine set, the cluster temporarily has four control plane machines. When the fourth control plane node joins the cluster, the etcd Operator starts a new etcd member on the replacement node. When the etcd Operator observes that the old control plane machine is marked for deletion, it stops the etcd member on the old node and promotes the replacement etcd member to join the quorum of the cluster.

The control plane machine Deleting phase proceeds in the following order:

A control plane machine is slated for deletion.
The control plane machine enters the Deleting phase.
To satisfy the preDrain lifecycle hook, the etcd Operator takes the following actions:
1. The etcd Operator waits until a fourth control plane machine is added to the cluster as an etcd member. This new etcd member has a state of Running but not ready until it receives the full database update from the etcd leader.
2. When the new etcd member receives the full database update, the etcd Operator promotes the new etcd member to a voting member and removes the old etcd member from the cluster.
After this transition is complete, it is safe for the old etcd pod and its data to be removed, so the preDrain lifecycle hook is removed.
The control plane machine status condition Drainable is set to True.
The machine controller attempts to drain the node that is backed by the control plane machine.
- If draining fails, Drained is set to False and the machine controller attempts to drain the node again.
- If draining succeeds, Drained is set to True.
The control plane machine status condition Drained is set to True.
If no other Operators have added a preTerminate lifecycle hook, the control plane machine status condition Terminable is set to True.
The machine controller removes the instance from the infrastructure provider.
The machine controller deletes the Node object.

YAML snippet demonstrating the etcd quorum protection preDrain lifecycle hook

apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
  ...
spec:
  lifecycleHooks:
    preDrain:
    - name: etcdQuorumOperator (1)
      owner: clusteroperator/etcd (2)
  ...

1	The name of the `preDrain` lifecycle hook.
2	The hook-implementing controller that manages the `preDrain` lifecycle hook.