Creating images

Learning container best practices
- General container image guidelines
- OKD-specific guidelines
Including metadata in images
- Defining image metadata
Creating images from source code with source-to-image
- Understanding the source-to-image build process
- How to write source-to-image scripts
About testing source-to-image images

Learn how to create your own container images, based on pre-built images that are ready to help you. The process includes learning best practices for writing images, defining metadata for images, testing images, and using a custom builder workflow to create images to use with OKD. After you create an image, you can push it to the OpenShift image registry.

Learning container best practices

When creating container images to run on OKD there are a number of best practices to consider as an image author to ensure a good experience for consumers of those images. Because images are intended to be immutable and used as-is, the following guidelines help ensure that your images are highly consumable and easy to use on OKD.

General container image guidelines

The following guidelines apply when creating a container image in general, and are independent of whether the images are used on OKD.

Reuse images

Wherever possible, base your image on an appropriate upstream image using the FROM statement. This ensures your image can easily pick up security fixes from an upstream image when it is updated, rather than you having to update your dependencies directly.

In addition, use tags in the FROM instruction, for example, rhel:rhel7, to make it clear to users exactly which version of an image your image is based on. Using a tag other than latest ensures your image is not subjected to breaking changes that might go into the latest version of an upstream image.

Maintain compatibility within tags

When tagging your own images, try to maintain backwards compatibility within a tag. For example, if you provide an image named image and it currently includes version 1.0, you might provide a tag of image:v1. When you update the image, as long as it continues to be compatible with the original image, you can continue to tag the new image image:v1, and downstream consumers of this tag are able to get updates without being broken.

If you later release an incompatible update, then switch to a new tag, for example image:v2. This allows downstream consumers to move up to the new version at will, but not be inadvertently broken by the new incompatible image. Any downstream consumer using image:latest takes on the risk of any incompatible changes being introduced.

Avoid multiple processes

Do not start multiple services, such as a database and SSHD, inside one container. This is not necessary because containers are lightweight and can be easily linked together for orchestrating multiple processes. OKD allows you to easily colocate and co-manage related images by grouping them into a single pod.

This colocation ensures the containers share a network namespace and storage for communication. Updates are also less disruptive as each image can be updated less frequently and independently. Signal handling flows are also clearer with a single process as you do not have to manage routing signals to spawned processes.

Use `exec` in wrapper scripts

Many images use wrapper scripts to do some setup before starting a process for the software being run. If your image uses such a script, that script uses exec so that the script’s process is replaced by your software. If you do not use exec, then signals sent by your container runtime go to your wrapper script instead of your software’s process. This is not what you want.

If you have a wrapper script that starts a process for some server. You start your container, for example, using podman run -i, which runs the wrapper script, which in turn starts your process. If you want to close your container with CTRL+C. If your wrapper script used exec to start the server process, podman sends SIGINT to the server process, and everything works as you expect. If you did not use exec in your wrapper script, podman sends SIGINT to the process for the wrapper script and your process keeps running like nothing happened.

Also note that your process runs as PID 1 when running in a container. This means that if your main process terminates, the entire container is stopped, canceling any child processes you launched from your PID 1 process.

Clean temporary files

Remove all temporary files you create during the build process. This also includes any files added with the ADD command. For example, run the yum clean command after performing yum install operations.

You can prevent the yum cache from ending up in an image layer by creating your RUN statement as follows:

RUN yum -y install mypackage && yum -y install myotherpackage && yum clean all -y

Note that if you instead write:

RUN yum -y install mypackage
RUN yum -y install myotherpackage && yum clean all -y

Then the first yum invocation leaves extra files in that layer, and these files cannot be removed when the yum clean operation is run later. The extra files are not visible in the final image, but they are present in the underlying layers.

The current container build process does not allow a command run in a later layer to shrink the space used by the image when something was removed in an earlier layer. However, this may change in the future. This means that if you perform an rm command in a later layer, although the files are hidden it does not reduce the overall size of the image to be downloaded. Therefore, as with the yum clean example, it is best to remove files in the same command that created them, where possible, so they do not end up written to a layer.

In addition, performing multiple commands in a single RUN statement reduces the number of layers in your image, which improves download and extraction time.

Place instructions in the proper order

The container builder reads the Dockerfile and runs the instructions from top to bottom. Every instruction that is successfully executed creates a layer which can be reused the next time this or another image is built. It is very important to place instructions that rarely change at the top of your Dockerfile. Doing so ensures the next builds of the same image are very fast because the cache is not invalidated by upper layer changes.

For example, if you are working on a Dockerfile that contains an ADD command to install a file you are iterating on, and a RUN command to yum install a package, it is best to put the ADD command last:

FROM foo
RUN yum -y install mypackage && yum clean all -y
ADD myfile /test/myfile

This way each time you edit myfile and rerun podman build or docker build, the system reuses the cached layer for the yum command and only generates the new layer for the ADD operation.

If instead you wrote the Dockerfile as:

FROM foo
ADD myfile /test/myfile
RUN yum -y install mypackage && yum clean all -y

Then each time you changed myfile and reran podman build or docker build, the ADD operation would invalidate the RUN layer cache, so the yum operation must be rerun as well.

Mark important ports

The EXPOSE instruction makes a port in the container available to the host system and other containers. While it is possible to specify that a port should be exposed with a podman run invocation, using the EXPOSE instruction in a Dockerfile makes it easier for both humans and software to use your image by explicitly declaring the ports your software needs to run:

Exposed ports show up under podman ps associated with containers created from your image.
Exposed ports are present in the metadata for your image returned by podman inspect.
Exposed ports are linked when you link one container to another.

Set environment variables

It is good practice to set environment variables with the ENV instruction. One example is to set the version of your project. This makes it easy for people to find the version without looking at the Dockerfile. Another example is advertising a path on the system that could be used by another process, such as JAVA_HOME.

Avoid default passwords

Avoid setting default passwords. Many people extend the image and forget to remove or change the default password. This can lead to security issues if a user in production is assigned a well-known password. Passwords are configurable using an environment variable instead.

If you do choose to set a default password, ensure that an appropriate warning message is displayed when the container is started. The message should inform the user of the value of the default password and explain how to change it, such as what environment variable to set.

Avoid sshd

It is best to avoid running sshd in your image. You can use the podman exec or docker exec command to access containers that are running on the local host. Alternatively, you can use the oc exec command or the oc rsh command to access containers that are running on the OKD cluster. Installing and running sshd in your image opens up additional vectors for attack and requirements for security patching.

Use volumes for persistent data

Images use a volume for persistent data. This way OKD mounts the network storage to the node running the container, and if the container moves to a new node the storage is reattached to that node. By using the volume for all persistent storage needs, the content is preserved even if the container is restarted or moved. If your image writes data to arbitrary locations within the container, that content could not be preserved.

All data that needs to be preserved even after the container is destroyed must be written to a volume. Container engines support a readonly flag for containers, which can be used to strictly enforce good practices about not writing data to ephemeral storage in a container. Designing your image around that capability now makes it easier to take advantage of it later.

Explicitly defining volumes in your Dockerfile makes it easy for consumers of the image to understand what volumes they must define when running your image.

See the Kubernetes documentation for more information on how volumes are used in OKD.

Even with persistent volumes, each instance of your image has its own volume, and the filesystem is not shared between instances. This means the volume cannot be used to share state in a cluster.

OKD-specific guidelines

The following are guidelines that apply when creating container images specifically for use on OKD.

Enable images for source-to-image (S2I)

For images that are intended to run application code provided by a third party, such as a Ruby image designed to run Ruby code provided by a developer, you can enable your image to work with the Source-to-Image (S2I) build tool. S2I is a framework that makes it easy to write images that take application source code as an input and produce a new image that runs the assembled application as output.

Support arbitrary user ids

By default, OKD runs containers using an arbitrarily assigned user ID. This provides additional security against processes escaping the container due to a container engine vulnerability and thereby achieving escalated permissions on the host node.

For an image to support running as an arbitrary user, directories and files that are written to by processes in the image must be owned by the root group and be read/writable by that group. Files to be executed must also have group execute permissions.

Adding the following to your Dockerfile sets the directory and file permissions to allow users in the root group to access them in the built image:

RUN chgrp -R 0 /some/directory && \
    chmod -R g=u /some/directory

Because the container user is always a member of the root group, the container user can read and write these files.

Care must be taken when altering the directories and file permissions of the sensitive areas of a container. If applied to sensitive areas, such as the /etc/passwd file, such changes can allow the modification of these files by unintended users, potentially exposing the container or host. CRI-O supports the insertion of arbitrary user IDs into a container’s /etc/passwd file. As such, changing permissions is never required.

Additionally, the /etc/passwd file should not exist in any container image. If it does, the CRI-O container runtime will fail to inject a random UID into the /etc/passwd file. In such cases, the container might face challenges in resolving the active UID. Failing to meet this requirement could impact the functionality of certain containerized applications.

In addition, the processes running in the container must not listen on privileged ports, ports below 1024, since they are not running as a privileged user.

If your S2I image does not include a user declaration with a numeric user, your builds fail by default. To allow images that use either named users or the root 0 user to build in OKD, you can add the project’s builder service account, system:serviceaccount:<your-project>:builder, to the anyuid security context constraint (SCC). Alternatively, you can allow all images to run as any user.

Use services for inter-image communication

For cases where your image needs to communicate with a service provided by another image, such as a web front end image that needs to access a database image to store and retrieve data, your image consumes an OKD service. Services provide a static endpoint for access which does not change as containers are stopped, started, or moved. In addition, services provide load balancing for requests.

Provide common libraries

For images that are intended to run application code provided by a third party, ensure that your image contains commonly used libraries for your platform. In particular, provide database drivers for common databases used with your platform. For example, provide JDBC drivers for MySQL and PostgreSQL if you are creating a Java framework image. Doing so prevents the need for common dependencies to be downloaded during application assembly time, speeding up application image builds. It also simplifies the work required by application developers to ensure all of their dependencies are met.

Use environment variables for configuration

users of your image are able to configure it without having to create a downstream image based on your image. This means that the runtime configuration is handled using environment variables. For a simple configuration, the running process can consume the environment variables directly. For a more complicated configuration or for runtimes which do not support this, configure the runtime by defining a template configuration file that is processed during startup. During this processing, values supplied using environment variables can be substituted into the configuration file or used to make decisions about what options to set in the configuration file.

It is also possible and recommended to pass secrets such as certificates and keys into the container using environment variables. This ensures that the secret values do not end up committed in an image and leaked into a container image registry.

Providing environment variables allows consumers of your image to customize behavior, such as database settings, passwords, and performance tuning, without having to introduce a new layer on top of your image. Instead, they can simply define environment variable values when defining a pod and change those settings without rebuilding the image.

For extremely complex scenarios, configuration can also be supplied using volumes that would be mounted into the container at runtime. However, if you elect to do it this way you must ensure that your image provides clear error messages on startup when the necessary volume or configuration is not present.

This topic is related to the Using Services for Inter-image Communication topic in that configuration like datasources are defined in terms of environment variables that provide the service endpoint information. This allows an application to dynamically consume a datasource service that is defined in the OKD environment without modifying the application image.

In addition, tuning is done by inspecting the cgroups settings for the container. This allows the image to tune itself to the available memory, CPU, and other resources. For example, Java-based images tune their heap based on the cgroup maximum memory parameter to ensure they do not exceed the limits and get an out-of-memory error.

Set image metadata

Defining image metadata helps OKD better consume your container images, allowing OKD to create a better experience for developers using your image. For example, you can add metadata to provide helpful descriptions of your image, or offer suggestions on other images that are needed.

Clustering

You must fully understand what it means to run multiple instances of your image. In the simplest case, the load balancing function of a service handles routing traffic to all instances of your image. However, many frameworks must share information to perform leader election or failover state; for example, in session replication.

Consider how your instances accomplish this communication when running in OKD. Although pods can communicate directly with each other, their IP addresses change anytime the pod starts, stops, or is moved. Therefore, it is important for your clustering scheme to be dynamic.

Logging

It is best to send all logging to standard out. OKD collects standard out from containers and sends it to the centralized logging service where it can be viewed. If you must separate log content, prefix the output with an appropriate keyword, which makes it possible to filter the messages.

If your image logs to a file, users must use manual operations to enter the running container and retrieve or view the log file.

Liveness and readiness probes

Document example liveness and readiness probes that can be used with your image. These probes allow users to deploy your image with confidence that traffic is not be routed to the container until it is prepared to handle it, and that the container is restarted if the process gets into an unhealthy state.

Templates

Consider providing an example template with your image. A template gives users an easy way to quickly get your image deployed with a working configuration. Your template must include the liveness and readiness probes you documented with the image, for completeness.

Including metadata in images

This topic only defines the metadata needed by the current set of use cases. Additional metadata or use cases may be added in the future.

Defining image metadata

You can use the LABEL instruction in a Dockerfile to define image metadata. Labels are similar to environment variables in that they are key value pairs attached to an image or a container. Labels are different from environment variable in that they are not visible to the running application and they can also be used for fast look-up of images and containers.

Docker documentation for more information on the LABEL instruction.

The label names are typically namespaced. The namespace is set accordingly to reflect the project that is going to pick up the labels and use them. For OKD the namespace is set to io.openshift and for Kubernetes the namespace is io.k8s.

See the Docker custom metadata documentation for details about the format.

Table 1. Supported Metadata
Variable	Description
`io.openshift.tags`	This label contains a list of tags represented as a list of comma-separated string values. The tags are the way to categorize the container images into broad areas of functionality. Tags help UI and generation tools to suggest relevant container images during the application creation process. LABEL io.openshift.tags mongodb,mongodb24,nosql
`io.openshift.wants`	Specifies a list of tags that the generation tools and the UI uses to provide relevant suggestions if you do not have the container images with specified tags already. For example, if the container image wants `mysql` and `redis` and you do not have the container image with `redis` tag, then UI can suggest you to add this image into your deployment. LABEL io.openshift.wants mongodb,redis
`io.k8s.description`	This label can be used to give the container image consumers more detailed information about the service or functionality this image provides. The UI can then use this description together with the container image name to provide more human friendly information to end users. LABEL io.k8s.description The MySQL 5.5 Server with master-slave replication support
`io.openshift.non-scalable`	An image can use this variable to suggest that it does not support scaling. The UI then communicates this to consumers of that image. Being not-scalable means that the value of `replicas` should initially not be set higher than `1`. LABEL io.openshift.non-scalable true
`io.openshift.min-memory` and `io.openshift.min-cpu`	This label suggests how much resources the container image needs to work properly. The UI can warn the user that deploying this container image may exceed their user quota. The values must be compatible with Kubernetes quantity. LABEL io.openshift.min-memory 16Gi LABEL io.openshift.min-cpu 4

Creating images from source code with source-to-image

Source-to-image (S2I) is a framework that makes it easy to write images that take application source code as an input and produce a new image that runs the assembled application as output.

The main advantage of using S2I for building reproducible container images is the ease of use for developers. As a builder image author, you must understand two basic concepts in order for your images to provide the best S2I performance, the build process and S2I scripts.

Understanding the source-to-image build process

The build process consists of the following three fundamental elements, which are combined into a final container image:

Sources
Source-to-image (S2I) scripts
Builder image

S2I generates a Dockerfile with the builder image as the first FROM instruction. The Dockerfile generated by S2I is then passed to Buildah.

How to write source-to-image scripts

You can write source-to-image (S2I) scripts in any programming language, as long as the scripts are executable inside the builder image. S2I supports multiple options providing assemble/run/save-artifacts scripts. All of these locations are checked on each build in the following order:

A script specified in the build configuration.
A script found in the application source .s2i/bin directory.
A script found at the default image URL with the io.openshift.s2i.scripts-url label.

Both the io.openshift.s2i.scripts-url label specified in the image and the script specified in a build configuration can take one of the following forms:

image:///path_to_scripts_dir: absolute path inside the image to a directory where the S2I scripts are located.
file:///path_to_scripts_dir: relative or absolute path to a directory on the host where the S2I scripts are located.
http(s)://path_to_scripts_dir: URL to a directory where the S2I scripts are located.

Script Description

assemble

The assemble script builds the application artifacts from a source and places them into appropriate directories inside the image. This script is required. The workflow for this script is:

Optional: Restore build artifacts. If you want to support incremental builds, make sure to define save-artifacts as well.
Place the application source in the desired location.
Build the application artifacts.
Install the artifacts into locations appropriate for them to run.

run

The run script executes your application. This script is required.

save-artifacts

The save-artifacts script gathers all dependencies that can speed up the build processes that follow. This script is optional. For example:

For Ruby, gems installed by Bundler.
For Java, .m2 contents.

These dependencies are gathered into a tar file and streamed to the standard output.

usage

The usage script allows you to inform the user how to properly use your image. This script is optional.

test/run

The test/run script allows you to create a process to check if the image is working correctly. This script is optional. The proposed flow of that process is:

Build the image.
Run the image to verify the usage script.
Run s2i build to verify the assemble script.
Optional: Run s2i build again to verify the save-artifacts and assemble scripts save and restore artifacts functionality.
Run the image to verify the test application is working.

The suggested location to put the test application built by your test/run script is the test/test-app directory in your image repository.

Example S2I scripts

The following example S2I scripts are written in Bash. Each example assumes its tar contents are unpacked into the /tmp/s2i directory.

assemble script:

#!/bin/bash

# restore build artifacts
if [ "$(ls /tmp/s2i/artifacts/ 2>/dev/null)" ]; then
    mv /tmp/s2i/artifacts/* $HOME/.
fi

# move the application source
mv /tmp/s2i/src $HOME/src

# build application artifacts
pushd ${HOME}
make all

# install the artifacts
make install
popd

run script:

#!/bin/bash

# run the application
/opt/application/run.sh

save-artifacts script:

#!/bin/bash

pushd ${HOME}
if [ -d deps ]; then
    # all deps contents to tar stream
    tar cf - deps
fi
popd

usage script:

#!/bin/bash

# inform the user how to use the image
cat <<EOF
This is a S2I sample builder image, to use it, install
https://github.com/openshift/source-to-image
EOF

Additional resources

S2I Image Creation Tutorial

About testing source-to-image images

As an Source-to-Image (S2I) builder image author, you can test your S2I image locally and use the OKD build system for automated testing and continuous integration.

S2I requires the assemble and run scripts to be present to successfully run the S2I build. Providing the save-artifacts script reuses the build artifacts, and providing the usage script ensures that usage information is printed to console when someone runs the container image outside of the S2I.

The goal of testing an S2I image is to make sure that all of these described commands work properly, even if the base container image has changed or the tooling used by the commands was updated.

Understanding testing requirements

The standard location for the test script is test/run. This script is invoked by the OKD S2I image builder and it could be a simple Bash script or a static Go binary.

The test/run script performs the S2I build, so you must have the S2I binary available in your $PATH. If required, follow the installation instructions in the S2I README.

S2I combines the application source code and builder image, so to test it you need a sample application source to verify that the source successfully transforms into a runnable container image. The sample application should be simple, but it should exercise the crucial steps of assemble and run scripts.

Generating scripts and tools

The S2I tooling comes with powerful generation tools to speed up the process of creating a new S2I image. The s2i create command produces all the necessary S2I scripts and testing tools along with the Makefile:

$ s2i create <image_name> <destination_directory>

The generated test/run script must be adjusted to be useful, but it provides a good starting point to begin developing.

The test/run script produced by the s2i create command requires that the sample application sources are inside the test/test-app directory.

Testing locally

The easiest way to run the S2I image tests locally is to use the generated Makefile.

If you did not use the s2i create command, you can copy the following Makefile template and replace the IMAGE_NAME parameter with your image name.

Sample Makefile

IMAGE_NAME = openshift/ruby-20-centos7
CONTAINER_ENGINE := $(shell command -v podman 2> /dev/null | echo docker)

build:
	${CONTAINER_ENGINE} build -t $(IMAGE_NAME) .

.PHONY: test
test:
	${CONTAINER_ENGINE} build -t $(IMAGE_NAME)-candidate .
	IMAGE_NAME=$(IMAGE_NAME)-candidate test/run

Basic testing workflow

The test script assumes you have already built the image you want to test. If required, first build the S2I image. Run one of the following commands:

If you use Podman, run the following command:
```
$ podman build -t <builder_image_name>
```
If you use Docker, run the following command:
```
$ docker build -t <builder_image_name>
```

The following steps describe the default workflow to test S2I image builders:

Verify the usage script is working:
- If you use Podman, run the following command:
  $ podman run <builder_image_name> .
- If you use Docker, run the following command:
  $ docker run <builder_image_name> .

Build the image:

$ s2i build file:///path-to-sample-app _<BUILDER_IMAGE_NAME>_ _<OUTPUT_APPLICATION_IMAGE_NAME>_

Optional: if you support save-artifacts, run step 2 once again to verify that saving and restoring artifacts works properly.
Run the container:
- If you use Podman, run the following command:
  $ podman run <output_application_image_name>
- If you use Docker, run the following command:
  $ docker run <output_application_image_name>
Verify the container is running and the application is responding.

Running these steps is generally enough to tell if the builder image is working as expected.

Using OKD for building the image

Once you have a Dockerfile and the other artifacts that make up your new S2I builder image, you can put them in a git repository and use OKD to build and push the image. Define a Docker build that points to your repository.

If your OKD instance is hosted on a public IP address, the build can be triggered each time you push into your S2I builder image GitHub repository.

You can also use the ImageChangeTrigger to trigger a rebuild of your applications that are based on the S2I builder image you updated.