$ REG_CREDS=${XDG_RUNTIME_DIR}/containers/auth.json
This guide describes how to work with custom catalogs packaged using either the Package Manifest Format or Bundle Format on Operator Lifecycle Manager (OLM) in OpenShift Container Platform.
Operator Lifecycle Manager (OLM) always installs Operators from the latest version of an Operator catalog. For OpenShift Container Platform 4.5, Red Hat-provided Operators are distributed via Quay App Registry catalogs from quay.io.
Catalog | Description |
---|---|
|
Public catalog for Red Hat products packaged and shipped by Red Hat. Supported by Red Hat. |
|
Public catalog for products from leading independent software vendors (ISVs). Red Hat partners with ISVs to package and ship. Supported by the ISV. |
|
Public catalog for software maintained by relevant representatives in the operator-framework/community-operators GitHub repository. No official support. |
As catalogs are updated, the latest versions of Operators change, and older versions may be removed or altered. This behavior can cause problems maintaining reproducible installs over time. In addition, when OLM runs on an OpenShift Container Platform cluster in a restricted network environment, it is unable to access the catalogs from quay.io directly.
Using the oc adm catalog build
command, cluster administrators can create an Operator catalog image. An Operator catalog image is:
a point-in-time export of an App Registry type catalog’s content.
the result of converting an App Registry catalog to a container image type catalog.
an immutable artifact.
Creating an Operator catalog image provides a simple way to use this content without incurring the aforementioned issues.
Cluster administrators can build a custom Operator catalog image based on the Package Manifest Format to be used by Operator Lifecycle Manager (OLM). The catalog image can be pushed to a container image registry that supports Docker v2-2. For a cluster on a restricted network, this registry can be a registry that the cluster has network access to, such as a mirror registry created during a restricted network cluster installation.
The internal registry of the OpenShift Container Platform cluster cannot be used as the target registry because it does not support pushing without a tag, which is required during the mirroring process. |
For this example, the procedure assumes use of a mirror registry that has access to both your network and the Internet.
Only the Linux version of the |
Workstation with unrestricted network access
oc
version 4.3.5+ Linux client
podman
version 1.4.4+
Access to mirror registry that supports Docker v2-2
If you are working with private registries, set the REG_CREDS
environment variable to the file path of your registry credentials for use in later steps. For example, for the podman
cli:
$ REG_CREDS=${XDG_RUNTIME_DIR}/containers/auth.json
If you are working with private namespaces that your quay.io account has access to, you must set a Quay authentication token. Set the AUTH_TOKEN
environment variable for use with the --auth-token
flag by making a request against the login API using your quay.io credentials:
$ AUTH_TOKEN=$(curl -sH "Content-Type: application/json" \
-XPOST https://quay.io/cnr/api/v1/users/login -d '
{
"user": {
"username": "'"<quay_username>"'",
"password": "'"<quay_password>"'"
}
}' | jq -r '.token')
On the workstation with unrestricted network access, authenticate with the target mirror registry:
$ podman login <registry_host_name>
Also authenticate with registry.redhat.io
so that the base image can be pulled
during the build:
$ podman login registry.redhat.io
Build a catalog image based on the redhat-operators
catalog from Quay.io, tagging and pushing it to your mirror registry:
$ oc adm catalog build \
--appregistry-org redhat-operators \(1)
--from=registry.redhat.io/openshift4/ose-operator-registry:v4.5 \(2)
--filter-by-os="linux/amd64" \(3)
--to=<registry_host_name>:<port>/olm/redhat-operators:v1 \(4)
[-a ${REG_CREDS}] \(5)
[--insecure] \(6)
[--auth-token "${AUTH_TOKEN}"] (7)
1 | Organization (namespace) to pull from an App Registry instance. |
2 | Set --from to the ose-operator-registry base image using the tag that
matches the target OpenShift Container Platform cluster major and minor version. |
3 | Set --filter-by-os to the operating system and architecture to use for the
base image, which must match the target OpenShift Container Platform cluster. Valid values
are linux/amd64 , linux/ppc64le , and linux/s390x . |
4 | Name your catalog image and include a tag, for example, v1 . |
5 | Optional: If required, specify the location of your registry credentials file. |
6 | Optional: If you do not want to configure trust for the target registry, add the --insecure flag. |
7 | Optional: If other application registry catalogs are used that are not public, specify a Quay authentication token. |
INFO[0013] loading Bundles dir=/var/folders/st/9cskxqs53ll3wdn434vw4cd80000gn/T/300666084/manifests-829192605
...
Pushed sha256:f73d42950021f9240389f99ddc5b0c7f1b533c054ba344654ff1edaf6bf827e3 to example_registry:5000/olm/redhat-operators:v1
Sometimes invalid manifests are accidentally introduced catalogs provided by Red Hat; when this happens, you might see some errors:
...
INFO[0014] directory dir=/var/folders/st/9cskxqs53ll3wdn434vw4cd80000gn/T/300666084/manifests-829192605 file=4.2 load=package
W1114 19:42:37.876180 34665 builder.go:141] error building database: error loading package into db: fuse-camel-k-operator.v7.5.0 specifies replacement that couldn't be found
Uploading ... 244.9kB/s
These errors are usually non-fatal, and if the Operator package mentioned does not contain an Operator you plan to install or a dependency of one, then they can be ignored.
Cluster administrators can mirror their catalog’s content into a registry and
use a CatalogSource to load the content onto an OpenShift Container Platform cluster. For this
example, the procedure uses a custom redhat-operators
catalog image previously
built and pushed to a supported registry.
Workstation with unrestricted network access
A custom Operator catalog image pushed to a supported registry
oc
version 4.3.5+
podman
version 1.4.4+
Access to mirror registry that supports Docker v2-2
If you are working with private registries, set the REG_CREDS
environment variable to the file path of your registry credentials for use in later steps. For example, for the podman
cli:
$ REG_CREDS=${XDG_RUNTIME_DIR}/containers/auth.json
The oc adm catalog mirror
command extracts the contents of your custom Operator catalog image to generate the manifests required for mirroring. You can choose to either:
Allow the default behavior of the command to automatically mirror all of the image content to your mirror registry after generating manifests, or
Add the --manifests-only
flag to only generate the manifests required for mirroring, but do not actually mirror the image content to a registry yet. This can be useful for reviewing what will be mirrored, and it allows you to make any changes to the mapping list if you only require a subset of the content. You can then use that file with the oc image mirror
command to mirror the modified list of images in a later step.
On your workstation with unrestricted network access, run the following command:
$ oc adm catalog mirror \
<registry_host_name>:<port>/olm/redhat-operators:v1 \(1)
<registry_host_name>:<port> \
[-a ${REG_CREDS}] \(2)
[--insecure] \(3)
--filter-by-os='.*' \(4)
[--manifests-only] (5)
1 | Specify your Operator catalog image. |
2 | Optional: If required, specify the location of your registry credentials file. |
3 | Optional: If you do not want to configure trust for the target registry, add the --insecure flag. |
4 | This flag is currently required due to a known issue with multiple architecture support. |
5 | Optional: Only generate the manifests required for mirroring and do not actually mirror the image content to a registry. |
If the |
using database path mapping: /:/tmp/190214037
wrote database to /tmp/190214037
using database at: /tmp/190214037/bundles.db (1)
...
1 | Temporary database generated by the command. |
After running the command, a <image_name>-manifests/
directory is created in the current directory and generates the following files:
The imageContentSourcePolicy.yaml
file defines an ImageContentSourcePolicy
object that can configure nodes to translate between the image references stored in Operator manifests and the mirrored registry.
The mapping.txt
file contains all of the source images and where to map them in the target registry. This file is compatible with the oc image mirror
command and can be used to further customize the mirroring configuration.
If you used the --manifests-only
flag in the previous step and want to mirror only a subset of the content:
Modify the list of images in your mapping.txt
file to your specifications. If you are unsure of the exact names and versions of the subset of images you want to mirror, use the following steps to find them:
Run the sqlite3
tool against the temporary database that was generated by the oc adm catalog mirror
command to retrieve a list of images matching a general search query. The output helps inform how you will later edit your mapping.txt
file.
For example, to retrieve a list of images that are similar to the string clusterlogging.4.3
:
$ echo "select * from related_image \
where operatorbundle_name like 'clusterlogging.4.3%';" \
| sqlite3 -line /tmp/190214037/bundles.db (1)
1 | Refer to the previous output of the oc adm catalog mirror command to find the path of the database file. |
image = registry.redhat.io/openshift4/ose-logging-kibana5@sha256:aa4a8b2a00836d0e28aa6497ad90a3c116f135f382d8211e3c55f34fb36dfe61
operatorbundle_name = clusterlogging.4.3.33-202008111029.p0
image = registry.redhat.io/openshift4/ose-oauth-proxy@sha256:6b4db07f6e6c962fc96473d86c44532c93b146bbefe311d0c348117bf759c506
operatorbundle_name = clusterlogging.4.3.33-202008111029.p0
...
Use the results from the previous step to edit the mapping.txt
file to only include the subset of images you want to mirror.
For example, you can use the image
values from the previous example output to find that the following matching lines exist in your mapping.txt
file:
mapping.txt
registry.redhat.io/openshift4/ose-logging-kibana5@sha256:aa4a8b2a00836d0e28aa6497ad90a3c116f135f382d8211e3c55f34fb36dfe61=<registry_host_name>:<port>/openshift4-ose-logging-kibana5:a767c8f0
registry.redhat.io/openshift4/ose-oauth-proxy@sha256:6b4db07f6e6c962fc96473d86c44532c93b146bbefe311d0c348117bf759c506=<registry_host_name>:<port>/openshift4-ose-oauth-proxy:3754ea2b
In this example, if you only want to mirror these images, you would then remove all other entries in the mapping.txt
file and leave only the above two lines.
Still on your workstation with unrestricted network access, use your modified mapping.txt
file to mirror the images to your registry using the oc image mirror
command:
$ oc image mirror \
[-a ${REG_CREDS}] \
--filter-by-os='.*' \
-f ./redhat-operators-manifests/mapping.txt
If the |
Apply the ImageContentSourcePolicy
object:
$ oc apply -f ./redhat-operators-manifests/imageContentSourcePolicy.yaml
Create a CatalogSource
object that references your catalog image.
Modify the following to your specifications and save it as a catalogsource.yaml
file:
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
name: my-operator-catalog
namespace: openshift-marketplace
spec:
sourceType: grpc
image: <registry_host_name>:<port>/olm/redhat-operators:v1 (1)
displayName: My Operator Catalog
publisher: grpc
1 | Specify your custom Operator catalog image. |
Use the file to create the CatalogSource
object:
$ oc create -f catalogsource.yaml
Verify the following resources are created successfully.
Check the pods:
$ oc get pods -n openshift-marketplace
NAME READY STATUS RESTARTS AGE
my-operator-catalog-6njx6 1/1 Running 0 28s
marketplace-operator-d9f549946-96sgr 1/1 Running 0 26h
Check the catalog source:
$ oc get catalogsource -n openshift-marketplace
NAME DISPLAY TYPE PUBLISHER AGE
my-operator-catalog My Operator Catalog grpc 5s
Check the package manifest:
$ oc get packagemanifest -n openshift-marketplace
NAME CATALOG AGE
etcd My Operator Catalog 34s
You can now install the Operators from the OperatorHub page on your restricted network OpenShift Container Platform cluster web console.
After a cluster administrator has configured OperatorHub to use custom Operator catalog images, administrators can keep their OpenShift Container Platform cluster up to date with the latest Operators by capturing updates made to App Registry catalogs provided by Red Hat. This is done by building and pushing a new Operator catalog image, then replacing the existing spec.image
parameter in the CatalogSource
object with the new image digest.
For this example, the procedure assumes a custom redhat-operators
catalog image is already configured for use with OperatorHub.
Only the Linux version of the |
Workstation with unrestricted network access
oc
version 4.3.5+ Linux client
podman
version 1.4.4+
Access to mirror registry that supports Docker v2-2
OperatorHub configured to use custom catalog images
If you are working with private registries, set the REG_CREDS
environment variable to the file path of your registry credentials for use in later steps. For example, for the podman
cli:
$ REG_CREDS=${XDG_RUNTIME_DIR}/containers/auth.json
If you are working with private namespaces that your quay.io account has access to, you must set a Quay authentication token. Set the AUTH_TOKEN
environment variable for use with the --auth-token
flag by making a request against the login API using your quay.io credentials:
$ AUTH_TOKEN=$(curl -sH "Content-Type: application/json" \
-XPOST https://quay.io/cnr/api/v1/users/login -d '
{
"user": {
"username": "'"<quay_username>"'",
"password": "'"<quay_password>"'"
}
}' | jq -r '.token')
On the workstation with unrestricted network access, authenticate with the target mirror registry:
$ podman login <registry_host_name>
Also authenticate with registry.redhat.io
so that the base image can be pulled
during the build:
$ podman login registry.redhat.io
Build a new catalog image based on the redhat-operators
catalog from Quay.io, tagging and pushing it to your mirror registry:
$ oc adm catalog build \
--appregistry-org redhat-operators \(1)
--from=registry.redhat.io/openshift4/ose-operator-registry:v4.5 \(2)
--filter-by-os="linux/amd64" \(3)
--to=<registry_host_name>:<port>/olm/redhat-operators:v2 \(4)
[-a ${REG_CREDS}] \(5)
[--insecure] \(6)
[--auth-token "${AUTH_TOKEN}"] (7)
1 | Organization (namespace) to pull from an App Registry instance. |
2 | Set --from to the ose-operator-registry base image using the tag that matches the target OpenShift Container Platform cluster major and minor version. |
3 | Set --filter-by-os to the operating system and architecture to use for the base image, which must match the target OpenShift Container Platform cluster. Valid values are linux/amd64 , linux/ppc64le , and linux/s390x . |
4 | Name your catalog image and include a tag, for example, v2 because it is the updated catalog. |
5 | Optional: If required, specify the location of your registry credentials file. |
6 | Optional: If you do not want to configure trust for the target registry, add the --insecure flag. |
7 | Optional: If other application registry catalogs are used that are not public, specify a Quay authentication token. |
INFO[0013] loading Bundles dir=/var/folders/st/9cskxqs53ll3wdn434vw4cd80000gn/T/300666084/manifests-829192605
...
Pushed sha256:f73d42950021f9240389f99ddc5b0c7f1b533c054ba344654ff1edaf6bf827e3 to example_registry:5000/olm/redhat-operators:v2
Mirror the contents of your catalog to your target registry. The following oc adm catalog mirror
command extracts the contents of your custom Operator catalog image to generate the manifests required for mirroring and mirrors the images to your registry:
$ oc adm catalog mirror \
<registry_host_name>:<port>/olm/redhat-operators:v2 \(1)
<registry_host_name>:<port> \
[-a ${REG_CREDS}] \(2)
[--insecure] \(3)
--filter-by-os='.*' (4)
1 | Specify your new Operator catalog image. |
2 | Optional: If required, specify the location of your registry credentials file. |
3 | Optional: If you do not want to configure trust for the target registry, add the --insecure flag. |
4 | This flag is currently required due to a known issue with multiple architecture support. If the --filter-by-os flag remains unset or set to any value other than .* , the command filters out different architectures, which changes the digest of the manifest list, also known as a multi-arch image. The incorrect digest causes deployments of those images and Operators on disconnected clusters to fail. For more information, see BZ#1890951. |
Apply the newly generated manifests:
$ oc apply -f ./redhat-operators-manifests
It is possible that you do not need to apply the |
Update your CatalogSource
object that references your catalog image.
If you have your original catalogsource.yaml
file for this CatalogSource
object:
Edit your catalogsource.yaml
file to reference your new catalog image in the spec.image
field:
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
name: my-operator-catalog
namespace: openshift-marketplace
spec:
sourceType: grpc
image: <registry_host_name>:<port>/olm/redhat-operators:v2 (1)
displayName: My Operator Catalog
publisher: grpc
1 | Specify your new Operator catalog image. |
Use the updated file to replace the CatalogSource
object:
$ oc replace -f catalogsource.yaml
Alternatively, edit the catalog source using the following command and reference your new catalog image in the spec.image
parameter:
$ oc edit catalogsource <catalog_source_name> -n openshift-marketplace
Updated Operators should now be available from the OperatorHub page on your OpenShift Container Platform cluster.
You can validate Operator catalog image content by running it as a container and querying its gRPC API. To further test the image, you can then resolve an OLM subscription by referencing the image in a CatalogSource
object. For this example, the procedure uses a custom redhat-operators
catalog image previously built and pushed to a supported registry.
A custom Operator catalog image pushed to a supported registry
podman
version 1.4.4+
oc
version 4.3.5+
Access to mirror registry that supports Docker v2-2
Pull the Operator catalog image:
$ podman pull <registry_host_name>:<port>/olm/redhat-operators:v1
Run the image:
$ podman run -p 50051:50051 \
-it <registry_host_name>:<port>/olm/redhat-operators:v1
Query the running image for available packages using grpcurl
:
$ grpcurl -plaintext localhost:50051 api.Registry/ListPackages
{
"name": "3scale-operator"
}
{
"name": "amq-broker"
}
{
"name": "amq-online"
}
Get the latest Operator bundle in a channel:
$ grpcurl -plaintext -d '{"pkgName":"kiali-ossm","channelName":"stable"}' localhost:50051 api.Registry/GetBundleForChannel
{
"csvName": "kiali-operator.v1.0.7",
"packageName": "kiali-ossm",
"channelName": "stable",
...
Get the digest of the image:
$ podman inspect \
--format='{{index .RepoDigests 0}}' \
<registry_host_name>:<port>/olm/redhat-operators:v1
example_registry:5000/olm/redhat-operators@sha256:f73d42950021f9240389f99ddc5b0c7f1b533c054ba344654ff1edaf6bf827e3
Assuming an Operator group exists in namespace my-ns
that supports your Operator and its dependencies, create a CatalogSource
object using the image digest. For example:
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
name: custom-redhat-operators
namespace: my-ns
spec:
sourceType: grpc
image: example_registry:5000/olm/redhat-operators@sha256:f73d42950021f9240389f99ddc5b0c7f1b533c054ba344654ff1edaf6bf827e3
displayName: Red Hat Operators
Create a subscription that resolves the latest available servicemeshoperator
and its dependencies from your catalog image:
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: servicemeshoperator
namespace: my-ns
spec:
source: custom-redhat-operators
sourceNamespace: my-ns
name: servicemeshoperator
channel: "1.0"
opm
cliThe new opm
cli tool is introduced alongside the new Bundle Format. This tool
allows you to create and maintain catalogs of Operators from a list of
bundles, called an index, that are equivalent to a "repository". The result is
a container image, called an index image, which can be stored in a container
registry and then installed on a cluster.
An index contains a database of pointers to Operator manifest content that can be queried via an included API that is served when the container image is run. On OpenShift Container Platform, OLM can use the index image as a catalog by referencing it in a CatalogSource, which polls the image at regular intervals to enable frequent updates to installed Operators on the cluster.
To create a bundle image using the Operator SDK, see Working with bundle images.
opm
You can install the opm
cli tool on your workstation.
podman
version 1.4.4+
Set the REG_CREDS
environment variable to the file path of your registry credentials for use in later steps. For example, for the podman
cli:
$ REG_CREDS=${XDG_RUNTIME_DIR}/containers/auth.json
Authenticate with registry.redhat.io
:
$ podman login registry.redhat.io
Extract the opm
binary from the Operator Registry image and copy it to your local file system:
$ oc image extract registry.redhat.io/openshift4/ose-operator-registry:v4.5 \
-a ${REG_CREDS} \(1)
--path /usr/bin/opm:. \
--confirm
1 | Specify the location of your registry credentials file. |
Make the binary executable:
$ chmod +x ./opm
Place the file anywhere in your PATH
, such as /usr/local/bin/
.
$ sudo mv ./opm /usr/local/bin/
Verify that the client was installed correctly:
$ opm version
Version: version.Version{OpmVersion:"1.12.3", GitCommit:"", BuildDate:"2020-07-01T23:18:58Z", GoOs:"linux", GoArch:"amd64"}
You can create an index image using the opm
cli.
opm
version 1.12.3+
podman
version 1.4.4+
A bundle image built and pushed to a registry.
Start a new index:
$ opm index add \
--bundles quay.io/<namespace>/test-operator:v0.1.0 \(1)
--tag quay.io/<namespace>/test-catalog:latest \(2)
[--binary-image <registry_base_image>] (3)
1 | Comma-separated list of bundle images to add to the index. |
2 | The image tag that you want the index image to have. |
3 | Optional: An alternative registry base image to use for serving the catalog. |
Push the index image to a registry:
$ podman push quay.io/<namespace>/test-catalog:latest
You can create a catalog from an index image and apply it to an OpenShift Container Platform cluster.
An index image built and pushed to a registry.
Apply a CatalogSource
object to your cluster that references your index image:
$ cat <<EOF | oc apply -f -
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
name: test-catalog
namespace: openshift-marketplace
spec:
sourceType: grpc
image: quay.io/<namespace>/test-catalog:latest (1)
displayName: Test Catalog
updateStrategy:
registryPoll: (2)
interval: 30m
EOF
1 | Specify your index image. |
2 | Catalog sources can automatically check for new versions to keep up to date. |
Verify using the OpenShift Container Platform web console or cli that the catalog loaded successfully and that packages are available. For example, using the cli:
Check the pods:
$ oc get pods -n openshift-marketplace
Check the catalog source:
$ oc get catalogsource -n openshift-marketplace
Check the package manifest:
$ oc get packagemanifests -n openshift-marketplace
You can update an existing index image using the opm
cli.
opm
version 1.12.3+
podman
version 1.4.4+
An index image built and pushed to a registry.
A CatalogSource
object created and applied to a cluster.
Update the existing index:
$ opm index add \
--bundles quay.io/<namespace>/another-operator:v1 \(1)
--from-index quay.io/<namespace>/test-catalog:latest \(2)
--tag quay.io/<namespace>/test-catalog:latest (3)
1 | The additional bundle image to add to the index. |
2 | The existing index that was previously pushed. |
3 | The image tag that you want the updated index image to have. |
Push the updated index image:
$ podman push quay.io/<namespace>/test-catalog:latest
After Operator Lifecycle Manager (OLM) polls the index image at its regular interval, verify that the new packages are successfully added:
$ oc get packagemanifests -n openshift-marketplace