This is a cache of https://docs.okd.io/latest/nodes/nodes/nodes-update-boot-images.html. It is a snapshot of the page at 2025-09-23T19:04:27.574+0000.
Boot image management - Working with nodes | Nodes | OKD 4
  1. Documentation
×

Boot image management

For Google Cloud Platform (GCP) and Amazon Web Services (AWS) clusters, by default the Machine Config Operator (MCO) manages and updates the boot image that is used to scale up your nodes. This means that by default, the MCO updates the boot images whenever you upgrade your cluster.

For VMware vSphere, you can enable boot image management as a Technology Preview feature.

For all other platforms, the MCO does not update the boot image with each cluster update.

Boot image management on vSphere is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
Additional resources

About boot image management

By default, for Google Cloud Platform (GCP) and Amazon Web Services (AWS) clusters, the Machine Config Operator (MCO) updates the boot image in the machine sets in your cluster whenever you update your cluster.

For VMware vSphere, you can enable boot image management as a Technology Preview feature. For information on how to enable this feature, see "Enabling boot image management".

Boot image management on vSphere is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

You can disable the boot image management feature, if needed. When the feature is disabled, the boot image no longer updates with the cluster. For example, with the feature disabled, if your cluster was originally created with OKD 4.16, the boot image that the MCO would use to create nodes is the same 4.16 version, even if your cluster is at a later version.

However, using an older boot image could cause the following issues:

  • Extra time to start nodes

  • Certificate expiration issues

  • Version skew issues

For information on how to disable this feature, see "Disabling boot image management". If you disable this feature, you can re-enable the feature at any time. For information, see "Enabling boot image management".

The ability to configure boot image management is available for only GCP and AWS clusters. It is not supported for clusters managed by the Cluster CAPI Operator.

How the cluster behaves after disabling or re-enabling the feature, depends upon when you made the change, including the following scenarios:

  • If you disable the feature before updating to a new OKD version:

    • The boot image version used by the machine sets remains the same OKD version as when the feature was disabled.

    • When you scale up nodes, the new nodes use that same OKD version.

  • If you disable the feature after updating to a new OKD version:

    • The boot image version used by the machine sets is updated to match the updated OKD version.

    • When you scale up nodes, the new nodes use the updated OKD version.

    • If you update to a later OKD version, the boot image version in the machine sets remains at the current version and is not updated with the cluster.

  • If you enable the feature after disabling:

    • The boot image version used by the machine sets is updated to the current OKD version, if different.

    • When you scale up nodes, the new nodes use the current OKD version in the cluster.

Because a boot image is used only when a node is scaled up, this feature has no effect on existing nodes.

To view the current boot image used in your cluster, use one of the following methods, based on your platform:

  • For GCP and AWS, you can examine a machine set.

    The location and format of the boot image within the machine set differs, based on the platform. However, the boot image is always listed in the spec.template.spec.providerSpec. parameter.
    Example GCP machine set with the boot image reference
    apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  name: ci-ln-hmy310k-72292-5f87z-worker-a
  namespace: openshift-machine-api
spec:
# ...
  template:
# ...
    spec:
# ...
      providerSpec:
# ...
        value:
          disks:
          - autoDelete: true
            boot: true
            image: projects/rhcos-cloud/global/images/rhcos-412-85-202203181601-0-gcp-x86-64 (1)
# ...
    1 This boot image is the same as the originally-installed OKD version, in this example OKD 4.12, regardless of the current version of the cluster. The way that the boot image is represented in the machine set depends on the platform, as the structure of the providerSpec field differs from platform to platform.
    Example AWS machine set with the boot image reference
    apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  name: ci-ln-hmy310k-72292-5f87z-worker-a
  namespace: openshift-machine-api
spec:
# ...
  template:
# ...
    spec:
# ...
      providerSpec:
         value:
          ami:
            id: ami-0e8fd9094e487d1ff
# ...

  • For VMware vSphere, examine an affected node by opening an oc debug session to the node and using the rpm-ostree status command.

    sh-5.1# rpm-ostree status
    Example AWS node with the boot image reference
    State: idle
Deployments:
* ostree-unverified-registry:quay.io/my-registry/...
                   Digest: sha256:...

If any of the machine sets for which you want to enable boot image management use a *-user-data secret that is based on Ignition version 2.2.0, the Machine Config Operator converts the Ignition version to 3.4.0 when you enable the feature. OKD versions 4.5 and lower use Ignition version 2.2.0. If this conversion fails, the MCO or your cluster could degrade. An error message that includes err: converting ignition stub failed: failed to parse Ignition config is added to the output of the oc get ClusterOperator machine-config command. You can use the following general steps to correct the problem:

  1. Disable the boot image management feature. For information, see "Disabling boot image management".

  2. Manually update the *-user-data secret to use Ignition version to 3.2.0.

  3. Enable the boot image management feature. For information, see "Enabling boot image management".
Additional resources

Disabling boot image management

By default, for Google Cloud Platform (GCP) and Amazon Web Services (AWS) clusters, the Machine Config Operator (MCO) manages and updates the boot image in the machine sets in your cluster whenever you update your cluster. For VMware vSphere, you can enable boot image management as a Technology Preview feature.

You can disable the boot image management feature for your cluster by editing the MachineConfiguration object. When disabled, the Machine Config Operator (MCO) no longer manages the boot image in your cluster and no longer updates the boot image with each cluster update.

Disabling this feature does not rollback the nodes or machine sets to the originally-installed boot image. The machine sets retain the boot image version that was present when the feature was disabled and is not updated if the cluster is upgraded to a new OKD version in the future. This feature has no effect on existing nodes.

After disabling the feature, you can re-enable the feature at any time. For more information, see "Enabling updated boot images".

Procedure

  1. Edit the MachineConfiguration object to disable the boot image management feature for some or all of your machine sets:

    $ oc edit MachineConfiguration cluster

    • Optional: Disable the feature for all machine sets:

      apiVersion: operator.openshift.io/v1
kind: MachineConfiguration
metadata:
  name: cluster
  namespace: openshift-machine-config-operator
spec:
# ...
  managedBootImages: (1)
    machineManagers:
    - apiGroup: machine.openshift.io (2)
      resource: machinesets (3)
      selection:
        mode: None (4)
      1 Configures the boot image management feature.
      2 Specifies an API group. This must be machine.openshift.io.
      3 Specifies the resource within the specified API group to apply the change. This must be machinesets.
      4 Specifies that the feature is disabled for all machine sets in the cluster.
Verification

  • When the affected nodes return to the READY state, view the current state of the boot image management feature by viewing the machine configuration object:

    $ oc get machineconfiguration cluster -o yaml
    Example machine set with the boot image reference
    kind: MachineConfiguration
metadata:
  name: cluster
# ...
status:
  conditions:
  - lastTransitionTime: "2025-05-01T20:11:49Z"
    message: Reconciled 2 of 4 MAPI MachineSets | Reconciled 0 of 0 CAPI MachineSets
      | Reconciled 0 of 0 CAPI MachineDeployments
    reason: BootImageUpdateConfigurationUpdated
    status: "True"
    type: BootImageUpdateProgressing
  - lastTransitionTime: "2025-05-01T19:30:13Z"
    message: 0 Degraded MAPI MachineSets | 0 Degraded CAPI MachineSets | 0 CAPI MachineDeployments
    reason: BootImageUpdateConfigurationUpdated
    status: "False"
    type: BootImageUpdateDegraded
  managedBootImagesStatus:
    machineManagers:
    - apiGroup: machine.openshift.io
      resource: machinesets
      selection:
        mode: All

  • When the affected nodes return to the READY state, check the current boot image by using one of the following methods:

    • For GCP and AWS, get the boot image version by running the following command. The location and format of the boot image within the machine set differs, based on the platform. However, the boot image is always listed in the spec.template.spec.providerSpec. parameter.

      $ oc get machinesets <machineset_name> -n openshift-machine-api -o yaml
      Example machine set with the boot image reference
      apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: ci-ln-77hmkpt-72292-d4pxp
    update-boot-image: "true"
  name: ci-ln-77hmkpt-72292-d4pxp-worker-a
  namespace: openshift-machine-api
spec:
# ...
  template:
# ...
    spec:
# ...
      providerSpec:
# ...
        value:
          disks:
          - autoDelete: true
            boot: true
            image: projects/rhcos-cloud/global/images/<boot_image> (1)
# ...
      1 This boot image is the same as the current OKD version.

    • For VMware vSphere, get the boot image version from an affected node:

      1. Open an oc debug session to the node by running a command similar to the following:

        $ oc debug node/<node_name>

      2. Set /host as the root directory within the debug shell by running the following command:

        sh-5.1# chroot /host

      3. Run the rpm-ostree status command to view that the custom layered image is in use:

        sh-5.1# rpm-ostree status
        Example output
        State: idle
Deployments:
* ostree-unverified-registry:quay.io/my-registry/...
                   Digest: sha256:...

Enabling boot image management

By default, for Google Cloud Platform (GCP) and Amazon Web Services (AWS) clusters, the Machine Config Operator (MCO) updates the boot image in the machine sets in your cluster whenever you update your cluster.

If you disabled the boot image management feature, so that the boot images are not updated, you can re-enable the feature by editing the MachineConfiguration object.

For VMware vSphere, you can enable boot image management as a Technology Preview feature.

Boot image management on vSphere is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Enabling the feature updates the boot image to the current OKD version. If the cluster is again updated to a new OKD version in the future, the boot image is updated again. New nodes created after enabling the feature use the updated boot image. This feature has no effect on existing nodes.

Prerequisites

  • For vSphere, enable the TechPreviewNoupgrade feature set on the cluster. For more information, see "Enabling features using feature gates".

    Enabling the TechPreviewNoupgrade feature set cannot be undone and prevents minor version updates. These feature sets are not recommended on production clusters.

    Wait until the managedBootImagesStatus stanza displays in the MachineConfiguration object.

    apiVersion: operator.openshift.io/v1
kind: MachineConfiguration
metadata:
  name: cluster
# ...
status:
# ...
  managedBootImagesStatus:
    machineManagers:
    - apiGroup: machine.openshift.io
      resource: machinesets
      selection:
        mode: None
Procedure

  1. Edit the MachineConfiguration object, named cluster, to enable the boot image management feature for some or all of your machine sets:

    $ oc edit MachineConfiguration cluster

    • Optional: Enable the boot image management feature for all machine sets:

      apiVersion: operator.openshift.io/v1
kind: MachineConfiguration
metadata:
  name: cluster
  namespace: openshift-machine-config-operator
spec:
# ...
  managedBootImages: (1)
    machineManagers:
    - apiGroup: machine.openshift.io (2)
      resource: machinesets (3)
      selection:
        mode: All (4)
      1 Configures the boot image management feature.
      2 Specifies the API group. This must be machine.openshift.io.
      3 Specifies the resource within the specified API group to apply the change. This must be machinesets.
      4 Specifies that the feature is enabled for all machine sets in the cluster.

    • Optional: Enable the boot image management feature for specific machine sets:

      apiVersion: operator.openshift.io/v1
kind: MachineConfiguration
metadata:
  name: cluster
  namespace: openshift-machine-config-operator
spec:
# ...
  managedBootImages: (1)
    machineManagers:
    - apiGroup: machine.openshift.io (2)
      resource: machinesets (3)
      selection:
        mode: Partial (4)
        partial:
          machineResourceSelector:
            matchLabels:
              region: "east"
      1 Configures the boot image update feature.
      2 Specifies the API group. This must be machine.openshift.io.
      3 Specifies the resource within the specified API group to apply the change. This must be machinesets.
      4 Specifies that the feature is enabled for machine sets with the specified label.

      If an appropriate label is not present on the machine set, add a key-value pair by running a command similar to following:

      $ oc label machineset.machine ci-ln-hmy310k-72292-5f87z-worker-a region="east" -n openshift-machine-api
Verification

  • When the affected nodes return to the READY state, view the current state of the boot image management feature by viewing the machine configuration object:

    $ oc get machineconfiguration cluster -o yaml
    Example machine set with the boot image reference
    kind: MachineConfiguration
metadata:
  name: cluster
# ...
status:
  conditions:
  - lastTransitionTime: "2025-05-01T20:11:49Z"
    message: Reconciled 2 of 4 MAPI MachineSets | Reconciled 0 of 0 CAPI MachineSets
      | Reconciled 0 of 0 CAPI MachineDeployments
    reason: BootImageUpdateConfigurationUpdated
    status: "True"
    type: BootImageUpdateProgressing
  - lastTransitionTime: "2025-05-01T19:30:13Z"
    message: 0 Degraded MAPI MachineSets | 0 Degraded CAPI MachineSets | 0 CAPI MachineDeployments
    reason: BootImageUpdateConfigurationUpdated
    status: "False"
    type: BootImageUpdateDegraded
  managedBootImagesStatus:
    machineManagers:
    - apiGroup: machine.openshift.io
      resource: machinesets
      selection:
        mode: All

  • When the affected nodes return to the READY state, check the current boot image by using one of the following methods:

    • For GCP and AWS, get the boot image version by running the following command. The location and format of the boot image within the machine set differs, based on the platform. However, the boot image is always listed in the spec.template.spec.providerSpec. parameter.

      $ oc get machinesets <machineset_name> -n openshift-machine-api -o yaml
      Example machine set with the boot image reference
      apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: ci-ln-77hmkpt-72292-d4pxp
    update-boot-image: "true"
  name: ci-ln-77hmkpt-72292-d4pxp-worker-a
  namespace: openshift-machine-api
spec:
# ...
  template:
# ...
    spec:
# ...
      providerSpec:
# ...
        value:
          disks:
          - autoDelete: true
            boot: true
            image: projects/rhcos-cloud/global/images/<boot_image> (1)
# ...
      1 This boot image is the same as the current OKD version.

    • For VMware vSphere, get the boot image version from an affected node:

      1. Open an oc debug session to the node by running a command similar to the following:

        $ oc debug node/<node_name>

      2. Set /host as the root directory within the debug shell by running the following command:

        sh-5.1# chroot /host

      3. Run the rpm-ostree status command to view that the custom layered image is in use:

        sh-5.1# rpm-ostree status
        Example output
        State: idle
Deployments:
* ostree-unverified-registry:quay.io/my-registry/...
                   Digest: sha256:...