Additional CRI-O storage locations for faster container startup

About additional storage locations for CRI-O
Configuring additional storage locations for CRI-O
Additional resources

To reduce application startup time, make your applications run more efficiently, and configure lazy pulling, you can configure additional storage locations for the CRI-O container engine.

Fields in the ContainerRuntimeConfig custom resource (CR) let you specify where CRI-O stores and resolves container image layers, complete container images, and OCI artifacts.

Using additional CRI-O storage locations is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

About additional storage locations for CRI-O

To reduce application startup time and make your applications run more efficiently, you can configure additional storage locations for the CRI-O container engine.

By using storage locations for the CRI-O container engine other than the default gives you control over where CRI-O stores and retrieves OCI artifacts, complete container images, and container image layers. Using additional storage locations for these CRI-O objects can reduce application startup time and make your applications run more efficiently through dedicated solid-state drive (SSD) storage, shared image caches, or lazy pulling.

By default, CRI-O stores all container data under a single root directory, /var/lib/containers/storage. This works well for typical workloads, but can create problems in clusters that use large images or artifacts, such as artificial intelligence and machine learning (AI/ML) workloads.

For example, large OCI artifacts, such as machine learning models, are stored in the default location, consuming space and preventing the use of faster dedicated storage. By configuring the additionalArtifactStores field, you can store large AI/ML models on high-performance solid-state drives (SSD) separate from the root file system. As a result, your workloads can experience faster start times and your clusters can use storage more efficiently.

Also, you could use the additionalImageStores field to mount an NFS share with prepopulated images across all worker nodes. Nodes read from the shared cache instead of pulling from an external registry. This is useful in disconnected environments or when many nodes run the same workloads.

With the additionalLayerStores field, you could enable lazy pulling through a third-party storage plugin, such as stargz-store. With lazy pulling, containers start after downloading only the required file chunks. The remaining data is fetched during runtime.

After you configure any of these new storage locations, the Machine Config Operator (MCO) reboots the affected nodes with the new configuration. After the reboot, CRI-O begins resolving storage from the additional locations.

Additional storage for OCI artifacts

Use the additionalArtifactStores field in a container runtime config to specify read-only locations where CRI-O resolves OCI artifacts, such as machine learning models pulled as OCI volume images. CRI-O checks these locations in order before falling back to the default storage location. CRI-O requires an existing, prepopulated artifacts/ subdirectory within each configured path. For example, if the path is /mnt/ssd-artifacts, place the artifacts in the /mnt/ssd-artifacts/artifacts/ directory.

The following example container runtime config configures storage for OCI artifacts.

apiVersion: machineconfiguration.openshift.io/v1
kind: ContainerRuntimeConfig
metadata:
  name: ssd-artifact-stores
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ""
  containerRuntimeConfig:
    additionalArtifactStores:
      - path: /mnt/ssd-artifacts
      - path: /mnt/nfs-shared-artifacts

When you create the container runtime config, the Machine Config Operator (MCO) writes the configuration to the /etc/crio/crio.conf.d/01-ctrcfg-additionalArtifactStores file on the target nodes.

Additional storage for container images

Use the additionalImageStores field to specify read-only container image caches on shared or high-performance storage. When CRI-O needs an image, it checks the additional image stores first. If the image exists there, no registry pull happens.

The following example container runtime config configures storage for container images.

apiVersion: machineconfiguration.openshift.io/v1
kind: ContainerRuntimeConfig
metadata:
  name: shared-image-cache
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ""
  containerRuntimeConfig:
    additionalImageStores:
      - path: /mnt/nfs-image-cache
      - path: /mnt/ssd-images

When you create the container runtime config, the Machine Config Operator (MCO) writes the configuration to the /etc/containers/storage.conf file on the target nodes.

Additional container image layers for lazy pulling

Use the additionalLayerStores field to enable lazy pulling through a third-party storage plugin.

Note that CRI-O falls back to a standard image pull in the following cases:

The registry does not support HTTP range requests.
The image is in standard OCI format, not a lazy-pull-compatible format such as eStargz or Nydus.
The storage plugin is not running.

The following example container runtime config configures container image layers for lazy pulling.

apiVersion: machineconfiguration.openshift.io/v1
kind: ContainerRuntimeConfig
metadata:
  name: lazy-pulling
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ""
  containerRuntimeConfig:
    additionalLayerStores:
      - path: /var/lib/stargz-store

When you create the container runtime config, the Machine Config Operator (MCO) writes the configuration to the /etc/containers/storage.conf file on the target nodes.

Configuring additional storage locations for CRI-O

To reduce application startup time and make your applications run more efficiently, you can configure additional storage locations for the CRI-O container engine to store OCI objects by using the ContainerRuntimeConfig custom resource (CR).

Use the additionalArtifactStores, additionalImageStores, and additionalLayerStores fields in a ContainerRuntimeConfig to specify read-only locations where CRI-O stores and resolves OCI artifacts, container images, or container image layers. CRI-O checks these locations in order before falling back to the default storage location.

When using multiple ContainerRuntimeConfig resources, merge all additional storage configurations into a single ContainerRuntimeConfig for each machine config pool. Multiple ContainerRuntimeConfig resources affecting the same configuration file might result in only a subset of the changes taking effect.

Prerequisites

You enabled the required Technology Preview features for your cluster by adding the TechPreviewNoUpgrade feature set to the FeatureGate CR named cluster. For information about enabling Feature Gates, see "Enabling features using feature gates".

Enabling the TechPreviewNoUpgrade feature set on your cluster cannot be undone and prevents minor version updates. This feature set allows you to enable these Technology Preview features on test clusters, where you can fully test them. Do not enable this feature set on production clusters.

If you are configuring the additionalImageStores or additionalLayerStores field, the target storage paths must exist and be accessible on the nodes and the container image or layers must be present in the directory. For network storage, ensure the paths are mounted before applying the configuration.
If you are configuring the additionalLayerStores field, you must meet the following additional prerequisites:
- A supported storage plugin binary must be installed on each node, such as Stargz Store or Nydus Storage Plugin. See "Stargz Store plugin" or "Nydus Storage Plugin" for more information. You must have installed the plugin by using one of the following methods:
  - Use a daemon set to run the plugin as a privileged container.
  - Use a machine config to install the binary and configure it as a systemd service.
  - Use Image mode for OpenShift to install the plugin in a custom FCOS image.
- You converted the container images to a lazy-pull-compatible format, such as eStargz or Nydus. See "eStargz format" or "Nydus format" for more information.
- Your container registry must support HTTP range requests.

Procedure

Create a YAML file for the ContainerRuntimeConfig CR similar to the following example:
```
apiVersion: machineconfiguration.openshift.io/v1
kind: ContainerRuntimeConfig
metadata:
  name: crio-additional-stores
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ""
  containerRuntimeConfig:
    additionalArtifactStores:
      - path: /mnt/ssd-artifacts
      - path: /mnt/nfs-shared-artifacts
    additionalImageStores:
      - path: /mnt/nfs-image-cache
      - path: /mnt/ssd-images
    additionalLayerStores:
      - path: /var/lib/stargz-store
```
where:

spec.machineConfigPoolSelector.matchLabels

Specifies a label associated with the nodes that you want to update.

spec.containerRuntimeConfig.additionalArtifactStores.path

Optional: Specifies the path to the directory that contains OCI artifacts. CRI-O searches for content in an artifacts/ subdirectory within this path. You can specify up to 10 directories.

spec.containerRuntimeConfig.additionalImageStores.path

Optional: Specifies the path to an NFS share or other location that contains pre-populated container images. You can specify up to 10 directories.

spec.containerRuntimeConfig.additionalLayerStores.path

Optional: Specifies the path to the directory that contains lazy-pull-compatible-formatted container image layers. You can specify up to 5 directories.

The specified path must meet the following criteria:
- Contains between 1 and 256 characters
- Is an absolute path, starting with the / character
- Contains only alphanumeric characters: a-z, A-Z, 0-9, /, ., _, and -
- Cannot contain consecutive forward slashes
You can configure any combination of these three additional CRI-O storage locations.

For a layer store, the MCO automatically appends the :ref suffix to the path when writing to the storage.conf file. This suffix switches the container storage library from storing actual image layers (blobs) to storing references (pointers) to where those layers can be found, which is required for the lazy-pulling plugins. You do not need to include the suffix in the ContainerRuntimeConfig path.

If a path does not exist or is inaccessible at runtime, CRI-O generates a warning and continues with the remaining stores. The default storage location is always used as a fallback.
Create the ContainerRuntimeConfig CR by running the following command:
```
$ oc create -f <container_runtime_config>.yaml
```
Replace <container_runtime_config> with the name of the YAML file.

After you configure any of these new storage locations, the Machine Config Operator (MCO) reboots the affected nodes with the new configuration.

Verification

After the nodes have returned to the Ready status, check that the new stores have been added to the node configuration:
1. Start a debug pod by running the following command:
  $ oc debug node/<node_name>
  where <node_name> specifies the name of one of the nodes in the affected machine config pool.
2. Set /host as the root directory within the debug shell by running the following command:
  sh-5.1# chroot /host
  - For an artifact store, review the contents of the /etc/crio/crio.conf.d/01-ctrcfg-additionalArtifactStores file by running the following command:
    
    sh-5.1# cat /etc/crio/crio.conf.d/01-ctrcfg-additionalArtifactStores
    
    Example output
    
    [crio] [crio.runtime] additional_artifact_stores = ["/mnt/ssd-artifacts", "/mnt/nfs-shared-artifacts"]
  - For an image store, review the contents of the /etc/containers/storage.conf file by running the following command:
    
    sh-5.1# cat /etc/containers/storage.conf
    
    Example output
    
    [storage] [storage.options] additionalimagestores = ["/mnt/nfs-image-cache", "/mnt/ssd-images"]
  - For a layer store, review the contents of the /etc/containers/storage.conf file by running the following command:
    
    sh-5.1# cat /etc/containers/storage.conf
    
    Example output
    
    [storage] [storage.options] additionallayerstores = ["/var/lib/stargz-store:ref"]

About additional storage locations for CRI-O

Configuring additional storage locations for CRI-O

Additional resources