Configure or Deploy a Docker Registry

Overview

OpenShift can build Docker images from your source code, deploy them, and manage their lifecycle. To enable this, OpenShift provides an internal, integrated Docker registry that can be deployed in your OpenShift environment to locally manage images.

Deploying the Registry

Starting in OpenShift Enterprise 3.2, quick installations automatically handle the initial deployment of the Docker registry and the OpenShift Enterprise router. However, you may need to manually create the registry if:

You did an advanced install and did not include the openshift_registry_selector variable.

Or,
For some reason it was not automatically deployed during a quick installation.

Or,
You deleted the registry and need to deploy it again.

To deploy the integrated Docker registry, use the oadm registry command as a user with cluster administrator privileges. For example:

$ oadm registry --config=/etc/origin/master/admin.kubeconfig \(1)
    --service-account=registry \(2)
    --images='registry.access.redhat.com/openshift3/ose-${component}:${version}' \(3)
    --selector='region=infra' (4)

During advanced installation, the openshift_registry_selector and openshift_hosted_router_selector Ansible settings are set to region=infra by default. The default router and registry will only be automatically deployed if a node exists that matches the region=infra label. <1> --config is the path to the CLI configuration file for the cluster administrator. <2> --service-account is the service account used to run the registry’s pod. <3> Required to pull the correct image for OpenShift Enterprise. <4> Optionally, you can specify the node location where you want to install the registry by specifying the corresponding node label.

This creates a service and a deployment configuration, both called docker-registry. Once deployed successfully, a pod is created with a name similar to docker-registry-1-cpty9.

Use --selector to deploy the registry to any node(s) that match a specified node label:

$ oadm registry <registry_name> --replicas=<number> --selector=<label> \
    --service-account=registry

For example, if you want to create a registry named registry and have it placed on a node labeled with region=infra:

$ oadm registry registry --replicas=1 --selector='region=infra' \
  --service-account=registry

To see a full list of options that you can specify when creating the registry:

$ oadm registry --help

Registry Compute Resources

By default, the registry is created with no settings for compute resource requests or limits. For production, it is highly recommended that the deployment configuration for the registry be updated to set resource requests and limits for the registry pod. Otherwise, the registry pod will be considered a BestEffort pod.

See Compute Resources for more information on configuring requests and limits.

Storage for the Registry

The registry stores Docker images and metadata. If you simply deploy a pod with the registry, it uses an ephemeral volume that is destroyed if the pod exits. Any images anyone has built or pushed into the registry would disappear.

Production Use

For production use, attach a remote volume or define and use the persistent storage method of your choice.

For example, to use an existing persistent volume claim:

$ oc volume deploymentconfigs/docker-registry --add --name=registry-storage -t pvc \
     --claim-name=<pvc_name> --overwrite

Or, to attach an existing NFS volume to the registry:

$ oc volume deploymentconfigs/docker-registry \
     --add --overwrite --name=registry-storage --mount-path=/registry \
     --source='{"nfs": { "server": "<fqdn>", "path": "/path/to/export"}}'

See Known Issues if using a scaled registry with a shared NFS volume.

Use Amazon S3 as a Storage Back-end

There is also an option to use Amazon Simple Storage Service storage with the internal Docker registry. It is a secure cloud storage manageable through AWS Management Console. To use it, the registry’s configuration file must be manually edited and mounted to the registry pod. However, before you start with the configuration, look at upstream’s recommended steps.

Take a default YAML configuration file as a base and replace the filesystem entry in the storage section with s3 entry such as below. The resulting storage section may look like this:

storage:
  cache:
    layerinfo: inmemory
  delete:
    enabled: true
  s3:
    accesskey: awsaccesskey
    secretkey: awssecretkey
    region: us-west-1
    regionendpoint: http://myobjects.local
    bucket: bucketname
    encrypt: true
    keyid: mykeyid
    secure: true
    v4auth: false
    chunksize: 5242880
    rootdirectory: /s3/object/name/prefix

All of the s3 configuration options are documented in upstream’s driver reference documentation.

Overriding the registry configuration will take you through the additional steps on mounting the configuration file into pod.

When the registry runs on the S3 storage back-end, there are reported issues.

Non-Production Use

For non-production use, you can use the --mount-host=<path> option to specify a directory for the registry to use for persistent storage. The registry volume is then created as a host-mount at the specified <path>.

The --mount-host option mounts a directory from the node on which the registry container lives. If you scale up the docker-registry deployment configuration, it is possible that your registry pods and containers will run on different nodes, which can result in two or more registry containers, each with its own local storage. This will lead to unpredictable behavior, as subsequent requests to pull the same image repeatedly may not always succeed, depending on which container the request ultimately goes to.

The --mount-host option requires that the registry container run in privileged mode. This is automatically enabled when you specify --mount-host. However, not all pods are allowed to run privileged containers by default. If you still want to use this option, create the registry and specify that it use the registry service account that was created during installation:

$ oadm registry --service-account=registry \
    --config=/etc/origin/master/admin.kubeconfig \
    --images='registry.access.redhat.com/openshift3/ose-${component}:${version}' \
    --mount-host=<path>

The Docker registry pod runs as user 1001. This user must be able to write to the host directory. You may need to change directory ownership to user ID 1001 with this command:

$ sudo chown 1001:root <path>

Maintaining the Registry IP Address

OpenShift Enterprise refers to the integrated registry by its service IP address, so if you decide to delete and recreate the docker-registry service, you can ensure a completely transparent transition by arranging to re-use the old IP address in the new service. If a new IP address cannot be avoided, you can minimize cluster disruption by rebooting only the masters.

Re-using the Address: To re-use the IP address, you must save the IP address of the old docker-registry service prior to deleting it, and arrange to replace the newly assigned IP address with the saved one in the new docker-registry service.

Make a note of the ClusterIP for the service:

$ oc get svc/docker-registry -o yaml | grep clusterIP:

Delete the service:

$ oc delete svc/docker-registry dc/docker-registry

Create the registry definition in registry.yaml, replacing <options> with, for example, those used in step 3 of the instructions in the Non-Production Use section:
```
$ oadm registry <options> -o yaml > registry.yaml
```
Edit registry.yaml, find the Service there, and change its ClusterIP to the address noted in step 1.
Create the registry using the modified registry.yaml:
```
$ oc create -f registry.yaml
```

Rebooting the Masters

If you are unable to re-use the IP address, any operation that uses a pull specification that includes the old IP address will fail. To minimize cluster disruption, you must reboot the masters:

# systemctl restart atomic-openshift-master

This ensures that the old registry URL, which includes the old IP address, is cleared from the cache.

We recommend against rebooting the entire cluster because that incurs unnecessary downtime for pods and does not actually clear the cache.

Viewing Logs

To view the logs for the Docker registry, use the oc logs command with the deployment config:

$ oc logs dc/docker-registry
2015-05-01T19:48:36.300593110Z time="2015-05-01T19:48:36Z" level=info msg="version=v2.0.0+unknown"
2015-05-01T19:48:36.303294724Z time="2015-05-01T19:48:36Z" level=info msg="redis not configured" instance.id=9ed6c43d-23ee-453f-9a4b-031fea646002
2015-05-01T19:48:36.303422845Z time="2015-05-01T19:48:36Z" level=info msg="using inmemory layerinfo cache" instance.id=9ed6c43d-23ee-453f-9a4b-031fea646002
2015-05-01T19:48:36.303433991Z time="2015-05-01T19:48:36Z" level=info msg="Using OpenShift Auth handler"
2015-05-01T19:48:36.303439084Z time="2015-05-01T19:48:36Z" level=info msg="listening on :5000" instance.id=9ed6c43d-23ee-453f-9a4b-031fea646002

File Storage

Tag and image metadata is stored in OpenShift Enterprise, but the registry stores layer and signature data in a volume that is mounted into the registry container at /registry. As oc exec does not work on privileged containers, to view a registry’s contents you must manually SSH into the node housing the registry pod’s container, then run docker exec on the container itself:

List the current pods to find the pod name of your Docker registry:
```
# oc get pods
```
Then, use oc describe to find the host name for the node running the container:
```
# oc describe pod <pod_name>
```
Log into the desired node:
```
# ssh node.example.com
```
List the running containers on the node host and identify the container ID for the Docker registry:
```
# docker ps | grep ose-docker-registry
```

List the registry contents using the docker exec command:

# docker exec -it 4c01db0b339c find /registry
/registry/docker
/registry/docker/registry
/registry/docker/registry/v2
/registry/docker/registry/v2/blobs (1)
/registry/docker/registry/v2/blobs/sha256
/registry/docker/registry/v2/blobs/sha256/ed
/registry/docker/registry/v2/blobs/sha256/ed/ede17b139a271d6b1331ca3d83c648c24f92cece5f89d95ac6c34ce751111810
/registry/docker/registry/v2/blobs/sha256/ed/ede17b139a271d6b1331ca3d83c648c24f92cece5f89d95ac6c34ce751111810/data (2)
/registry/docker/registry/v2/blobs/sha256/a3
/registry/docker/registry/v2/blobs/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
/registry/docker/registry/v2/blobs/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4/data
/registry/docker/registry/v2/blobs/sha256/f7
/registry/docker/registry/v2/blobs/sha256/f7/f72a00a23f01987b42cb26f259582bb33502bdb0fcf5011e03c60577c4284845
/registry/docker/registry/v2/blobs/sha256/f7/f72a00a23f01987b42cb26f259582bb33502bdb0fcf5011e03c60577c4284845/data
/registry/docker/registry/v2/repositories (3)
/registry/docker/registry/v2/repositories/p1
/registry/docker/registry/v2/repositories/p1/pause (4)
/registry/docker/registry/v2/repositories/p1/pause/_manifests
/registry/docker/registry/v2/repositories/p1/pause/_manifests/revisions
/registry/docker/registry/v2/repositories/p1/pause/_manifests/revisions/sha256
/registry/docker/registry/v2/repositories/p1/pause/_manifests/revisions/sha256/e9a2ac6418981897b399d3709f1b4a6d2723cd38a4909215ce2752a5c068b1cf
/registry/docker/registry/v2/repositories/p1/pause/_manifests/revisions/sha256/e9a2ac6418981897b399d3709f1b4a6d2723cd38a4909215ce2752a5c068b1cf/signatures (5)
/registry/docker/registry/v2/repositories/p1/pause/_manifests/revisions/sha256/e9a2ac6418981897b399d3709f1b4a6d2723cd38a4909215ce2752a5c068b1cf/signatures/sha256
/registry/docker/registry/v2/repositories/p1/pause/_manifests/revisions/sha256/e9a2ac6418981897b399d3709f1b4a6d2723cd38a4909215ce2752a5c068b1cf/signatures/sha256/ede17b139a271d6b1331ca3d83c648c24f92cece5f89d95ac6c34ce751111810
/registry/docker/registry/v2/repositories/p1/pause/_manifests/revisions/sha256/e9a2ac6418981897b399d3709f1b4a6d2723cd38a4909215ce2752a5c068b1cf/signatures/sha256/ede17b139a271d6b1331ca3d83c648c24f92cece5f89d95ac6c34ce751111810/link (6)
/registry/docker/registry/v2/repositories/p1/pause/_uploads (7)
/registry/docker/registry/v2/repositories/p1/pause/_layers (8)
/registry/docker/registry/v2/repositories/p1/pause/_layers/sha256
/registry/docker/registry/v2/repositories/p1/pause/_layers/sha256/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
/registry/docker/registry/v2/repositories/p1/pause/_layers/sha256/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4/link (9)
/registry/docker/registry/v2/repositories/p1/pause/_layers/sha256/f72a00a23f01987b42cb26f259582bb33502bdb0fcf5011e03c60577c4284845
/registry/docker/registry/v2/repositories/p1/pause/_layers/sha256/f72a00a23f01987b42cb26f259582bb33502bdb0fcf5011e03c60577c4284845/link

1	This directory stores all layers and signatures as blobs.
2	This file contains the blob’s contents.
3	This directory stores all the image repositories.
4	This directory is for a single image repository p1/pause.
5	This directory contains signatures for a particular image manifest revision.
6	This file contains a reference back to a blob (which contains the signature data).
7	This directory contains any layers that are currently being uploaded and staged for the given repository.
8	This directory contains links to all the layers this repository references.
9	This file contains a reference to a specific layer that has been linked into this repository via an image.

Accessing the Registry Directly

For advanced usage, you can access the registry directly to invoke docker commands. This allows you to push images to or pull them from the integrated registry directly using operations like docker push or docker pull. To do so, you must be logged in to the registry using the docker login command. The operations you can perform depend on your user permissions, as described in the following sections.

User Prerequisites

To access the registry directly, the user that you use must satisfy the following, depending on your intended usage:

For any direct access, you must have a regular user, if one does not already exist, for your preferred identity provider. A regular user can generate an access token required for logging in to the registry. System users, such as system:admin, cannot obtain access tokens and, therefore, cannot access the registry directly.

For example, if you are using HTPASSWD authentication, you can create one using the following command:
```
# htpasswd /etc/origin/openshift-htpasswd <user_name>
```
The user must have the system:registry role. To add this role:
```
# oadm policy add-role-to-user system:registry <user_name>
```
Have the admin role for the project associated with the Docker operation. For example, if accessing images in the global openshift project:
```
 $ oadm policy add-role-to-user admin <user_name> -n openshift
```
For writing or pushing images, for example when using the docker push command, the user must have the system:image-builder role. To add this role:
```
$ oadm policy add-role-to-user system:image-builder <user_name>
```

For more information on user permissions, see Managing Role Bindings.

Logging in to the Registry

Ensure your user satisfies the prerequisites for accessing the registry directly.

To log in to the registry directly:

Ensure you are logged in to OpenShift Enterprise as a regular user:
```
$ oc login
```
Get your access token:
```
$ oc whoami -t
```

$ docker login -u <username> -e <any_email_address> \
    -p <token_value> <registry_ip>:<port>

Pushing and Pulling Images

After logging in to the registry, you can perform docker pull and docker push operations against your registry.

You can pull arbitrary images, but if you have the system:registry role added, you can only push images to the registry in your project.

In the following examples, we use:

Component

Value

<registry_ip>

172.30.124.220

<port>

5000

<project>

openshift

<image>

busybox

<tag>

omitted (defaults to latest)

Pull an arbitrary image:
```
$ docker pull docker.io/busybox
```

Tag the new image with the form <registry_ip>:<port>/<project>/<image>. The project name must appear in this pull specification for OpenShift Enterprise to correctly place and later access the image in the registry.

$ docker tag docker.io/busybox 172.30.124.220:5000/openshift/busybox

Your regular user must have the system:image-builder role for the specified project, which allows the user to write or push an image. Otherwise, the docker push in the next step will fail. To test, you can create a new project to push the busybox image.

Push the newly-tagged image to your registry:

$ docker push 172.30.124.220:5000/openshift/busybox
...
cf2616975b4a: Image successfully pushed
Digest: sha256:3662dd821983bc4326bee12caec61367e7fb6f6a3ee547cbaff98f77403cab55

Securing the Registry

Optionally, you can secure the registry so that it serves traffic via TLS:

Deploy the registry.

Fetch the service IP and port of the registry:

$ oc get svc/docker-registry
NAME              LABELS                                    SELECTOR                  IP(S)            PORT(S)
docker-registry   docker-registry=default                   docker-registry=default   172.30.124.220   5000/TCP

You can use an existing server certificate, or create a key and server certificate valid for specified IPs and host names, signed by a specified CA. To create a server certificate for the registry service IP and the docker-registry.default.svc.cluster.local host name:

$ oadm ca create-server-cert \
    --signer-cert=/etc/origin/master/ca.crt \
    --signer-key=/etc/origin/master/ca.key \
    --signer-serial=/etc/origin/master/ca.serial.txt \
    --hostnames='docker-registry.default.svc.cluster.local,172.30.124.220' \
    --cert=/etc/secrets/registry.crt \
    --key=/etc/secrets/registry.key

Create the secret for the registry certificates:

$ oc secrets new registry-secret \
    /etc/secrets/registry.crt \
    /etc/secrets/registry.key

Add the secret to the registry pod’s service accounts (including the default service account):

$ oc secrets add serviceaccounts/registry secrets/registry-secret
$ oc secrets add serviceaccounts/default  secrets/registry-secret

Add the secret volume to the registry deployment configuration:

$ oc volume dc/docker-registry --add --type=secret \
    --secret-name=registry-secret -m /etc/secrets

Enable TLS by adding the following environment variables to the registry deployment configuration:
```
$ oc env dc/docker-registry \
    REGISTRY_HTTP_TLS_CERTIFICATE=/etc/secrets/registry.crt \
    REGISTRY_HTTP_TLS_KEY=/etc/secrets/registry.key
```
See more details on overriding registry options.

Update the scheme used for the registry’s liveness probe from HTTP to HTTPS:

$ oc patch dc/docker-registry -p '{"spec": {"template": {"spec": {"containers":[{
    "name":"registry",
    "livenessProbe":  {"httpGet": {"scheme":"HTTPS"}}
  }]}}}}'

If your registry was initially deployed on OpenShift Enterprise 3.2 or later, update the scheme used for the registry’s readiness probe from HTTP to HTTPS:

$ oc patch dc/docker-registry -p '{"spec": {"template": {"spec": {"containers":[{
    "name":"registry",
    "readinessProbe":  {"httpGet": {"scheme":"HTTPS"}}
  }]}}}}'

Validate the registry is running in TLS mode. Wait until the latest docker-registry deployment completes and verify the Docker logs for the registry container. You should find an entry for listening on :5000, tls.
```
$ oc logs dc/docker-registry | grep tls
time="2015-05-27T05:05:53Z" level=info msg="listening on :5000, tls" instance.id=deeba528-c478-41f5-b751-dc48e4935fc2
```

Copy the CA certificate to the Docker certificates directory. This must be done on all nodes in the cluster:

$ dcertsdir=/etc/docker/certs.d
$ destdir_addr=$dcertsdir/172.30.124.220:5000
$ destdir_name=$dcertsdir/docker-registry.default.svc.cluster.local:5000

$ sudo mkdir -p $destdir_addr $destdir_name
$ sudo cp ca.crt $destdir_addr    (1)
$ sudo cp ca.crt $destdir_name

1	The *ca.crt* file is a copy of */etc/origin/master/ca.crt* on the master.

Remove the --insecure-registry option only for this particular registry in the /etc/sysconfig/docker file. Then, reload the daemon and restart the docker service to reflect this configuration change:
```
$ sudo systemctl daemon-reload
$ sudo systemctl restart docker
```

Validate the docker client connection. Running docker push to the registry or docker pull from the registry should succeed. Make sure you have logged into the registry.

$ docker tag|push <registry/image> <internal_registry/project/image>

For example:

$ docker pull busybox
$ docker tag docker.io/busybox 172.30.124.220:5000/openshift/busybox
$ docker push 172.30.124.220:5000/openshift/busybox
...
cf2616975b4a: Image successfully pushed
Digest: sha256:3662dd821983bc4326bee12caec61367e7fb6f6a3ee547cbaff98f77403cab55

Advanced: Overriding the Registry Configuration

You can override the integrated registry’s default configuration, found by default at /config.yml in a running registry’s container, with your own custom configuration.

Upstream configuration options in this file may also be overridden using environment variables. However, the middleware section may not be overridden using environment variables. Learn how to override specific configuration options.

Deploying Updated Configuration

To enable managing the registry configuration file directly, it is recommended that the configuration file be mounted as a secret volume:

Deploy the registry.

Edit the registry configuration file locally as needed. The initial YAML file deployed on the registry is provided below. Review supported options.

Registry configuration file

version: 0.1
log:
  level: debug
http:
  addr: :5000
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /registry
  delete:
    enabled: true
auth:
  openshift:
    realm: openshift
middleware:
  repository:
    - name: openshift
      options:
        pullthrough: true

Create a new secret called registry-config from your custom registry configuration file you edited locally:
```
$ oc secrets new registry-config config.yml=</path/to/custom/registry/config.yml>
```
Add the registry-config secret as a volume to the registry’s deployment configuration to mount the custom configuration file at /etc/docker/registry/:
```
$ oc volume dc/docker-registry --add --type=secret \
    --secret-name=registry-config -m /etc/docker/registry/
```
Update the registry to reference the configuration path from the previous step by adding the following environment variable to the registry’s deployment configuration:
```
$ oc env dc/docker-registry \
    REGISTRY_CONFIGURATION_PATH=/etc/docker/registry/config.yml
```

This may be performed as an iterative process to achieve the desired configuration. For example, during troubleshooting, the configuration may be temporarily updated to put it in debug mode.

To update an existing configuration:

This procedure will overwrite the currently deployed registry configuration.

Edit the local registry configuration file, config.yml.
Delete the registry-config secret:
```
$ oc delete secret registry-config
```

Recreate the secret to reference the updated configuration file:

$ oc secrets new registry-config config.yml=</path/to/custom/registry/config.yml>

Redeploy the registry to read the updated configuration:
```
$ oc deploy docker-registry --latest
```

Maintain configuration files in a source control repository.

Registry Configuration Reference

There are many configuration options available in the upstream docker distribution library. Not all configuration options are supported or enabled. Use this section as a reference.

Log

Upstream options are supported.

log:
  level: debug
  formatter: text
  fields:
    service: registry
    environment: staging

Hooks

Mail hooks are not supported.

Storage

The following storage drivers are supported:

Filesystem
S3. Learn more about CloudFront configuration.
OpenStack Swift
Google Cloud Storage (GCS), starting in OpenShift Enterprise 3.2.1.13.

General registry storage configuration options are supported.

General Storage Configuration Options

storage:
  delete:
    enabled: true (1)
  redirect:
    disable: false
  cache:
    blobdescriptor: inmemory
  maintenance:
    uploadpurging:
      enabled: true
      age: 168h
      interval: 24h
      dryrun: false
    readonly:
      enabled: false

1	This entry is mandatory for image pruning to work properly.

Auth

Auth options should not be altered. The openshift extension is the only supported option.

auth:
  openshift:
    realm: openshift

Middleware

The repository middleware extension allows to configure OpenShift Enterprise middleware responsible for interaction with OpenShift Enterprise and image proxying.

The repository middleware extension should not be altered except for the options section to disable pull-through cache.

middleware:
  repository:
    - name: openshift (1)
      options:
        pullthrough: true (2)

1	These entries are mandatory. Their presence ensures required components get loaded. These values shouldn’t be changed.
2	Let the registry act as a proxy for remote blobs. See below for more details.

CloudFront Middleware

The CloudFront middleware extension can be added to support AWS, CloudFront CDN storage provider. CloudFront middleware speeds up distribution of image content internationally. The blobs are distributed to several edge locations around the world. The client is always directed to the edge with the lowest latency.

The CloudFront middleware extension can be only used with S3 storage. It is utilized only during blob serving. Therefore, only blob downloads can be speeded up, not uploads.

The following is an example of minimal configuration of S3 storage driver with a CloudFront middleware:

version: 0.1
log:
  level: debug
http:
  addr: :5000
storage:
  cache:
    blobdescriptor: inmemory
  delete:
    enabled: true
  s3: (1)
    accesskey: BJKMSZBRESWJQXRWMAEQ
    secretkey: 5ah5I91SNXbeoUXXDasFtadRqOdy62JzlnOW1goS
    region: us-east-1
    bucket: docker.myregistry.com
auth:
  openshift:
    realm: openshift
middleware:
  registry:
    - name: openshift
  repository:
    - name: openshift
   storage:
    - name: cloudfront (2)
      options:
        baseurl: https://jrpbyn0k5k88bi.cloudfront.net/ (3)
        privatekey: /etc/docker/cloudfront-ABCEDFGHIJKLMNOPQRST.pem (4)
        keypairid: ABCEDFGHIJKLMNOPQRST (5)
    - name: openshift

1	The S3 storage must be configured the same way regardless of CloudFront middleware.
2	The CloudFront storage middleware needs to be listed before OpenShift middleware.
3	The CloudFront base URL. In the AWS management console, this is listed as Domain Name of CloudFront distribution.
4	The location of your AWS private key on the filesystem. This must be not confused with Amazon EC2 key pair. Please refer to AWS documentation on creating CloudFront key pairs for your trusted signers. The file needs to be mounted as a secret secret into the registry pod.
5	The ID of your Cloudfront key pair.

Overriding Middleware Configuration Options

The middleware section cannot be overridden using environment variables. There are a few exceptions, however. For example:

middleware:
  repository:
    - name: openshift
      options:
        acceptschema2: false (1)
        enforcequota: false (2)
        projectcachettl: 1m (3)
        blobrepositorycachettl: 10m (4)

1	A configuration option that can be overridden by the boolean environment variable `REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ACCEPTSCHEMA2`, which allows for the ability to accept manifest schema v2 on manifest put requests.
2	A configuration option that can be overridden by the boolean environment variable `REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA`, which allows the ability to turn quota enforcement on or off. By default, quota enforcement is off. It overrides OpenShift Enterprise middleware configuration option. Recognized values are true and false.
3	A configuration option that can be overridden by the environment variable `REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_PROJECTCACHETTL`, specifying an eviction timeout for project quota objects. It takes a valid time duration string (for example, 2m). If empty, you get the default timeout. If zero (0m), caching is disabled.
4	A configuration option that can be overriden by the environment variable `REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_BLOBREPOSITORYCACHETTL`, specifying an eviction timeout for associations between blob and containing repository. The format of the value is the same as in `projectcachettl` case.

Image Pullthrough

If enabled, the registry will attempt to fetch requested blob from a remote registry unless the blob exists locally. The remote candidates are calculated from DockerImage entries stored in status of the image stream, a client pulls from. All the unique remote registry references in such entries will be tried in turn until the blob is found. The blob, served this way, will not be stored in the registry.

This feature is on by default. However, it can be disabled using a configuration option.

Manifest schema v2 support

Each image has a manifest describing its blobs, instructions for running it and additional metadata. The manifest is versioned which have different structure and fields as it evolves over time. The same image can be represented by multiple manifest versions. Each version will have different digest though.

The registry currently supports manifest v2 schema 1 (schema1). The manifest v2 schema 2 (schema2) is not yet supported.

You should be wary of compatibility issues with various Docker clients:

Docker clients of version 1.9 or older support only schema1. Any manifest this client pulls or pushes will be of this legacy schema.
Docker clients of version 1.10 support both schema1 and schema2. And by default, they will push the latter to the registry if it supports newer schema. Which means only schema1 will be pushed to the internal Docker registry.

Reporting

Reporting is unsupported.

HTTP

Upstream options are supported. Learn how to alter these settings via environment variables. Only the tls section should be altered. For example:

http:
  addr: :5000
  tls:
    certificate: /etc/secrets/registry.crt
    key: /etc/secrets/registry.key

Notifications

Upstream options are supported. The REST API Reference provides more comprehensive integration options.

Example:

notifications:
  endpoints:
    - name: registry
      disabled: false
      url: https://url:port/path
      headers:
        Accept:
          - text/plain
      timeout: 500
      threshold: 5
      backoff: 1000

Redis

Redis is not supported.

Health

Upstream options are supported. The registry deployment configuration provides an integrated health check at /healthz.

Proxy

Proxy configuration should not be enabled. This functionality is provided by the OpenShift Enterprise repository middleware extension, pullthrough: true.

Whitelisting Docker Registries

You can specify a whitelist of docker registries, allowing you to curate a set of images and templates that are available for download by OpenShift Enterprise users. This curated set can be placed in one or more docker registries, and then added to the whitelist. When using a whitelist, only the specified registries are accessible within OpenShift Enterprise, and all other registries are denied access by default.

To configure a whitelist:

Edit the /etc/sysconfig/docker file to block all registries:
```
BLOCK_REGISTRY='--block-registry=all'
```
You may need to uncomment the BLOCK_REGISTRY line.
In the same file, add registries to which you want to allow access:
```
ADD_REGISTRY='--add-registry=<registry1> --add-registry=<registry2>'
```
Allowing Access to Registries
```
ADD_REGISTRY='--add-registry=registry.access.redhat.com'
```
This example would restrict access to images available on the Red Hat Customer Portal.

Once the whitelist is configured, if a user tries to pull from a docker registry that is not on the whitelist, they will receive an error message stating that this registry is not allowed.

Exposing the Registry

To expose your internal registry externally, it is recommended that you run a secure registry. To expose the registry you must first have deployed a router.

Deploy the registry.
Secure the registry.
Deploy a router.

Create a passthrough route via the oc create route passthrough command, specifying the registry as the route’s service. By default, the name of the created route is the same as the service name.

For example:

$ oc get svc
NAME              CLUSTER_IP       EXTERNAL_IP   PORT(S)                 SELECTOR                  AGE
docker-registry   172.30.69.167    <none>        5000/TCP                docker-registry=default   4h
kubernetes        172.30.0.1       <none>        443/TCP,53/UDP,53/TCP   <none>                    4h
router            172.30.172.132   <none>        80/TCP                  router=router             4h

$ oc create route passthrough    \
    --service=docker-registry    \(1)
    --hostname=<host>
route "docker-registry" created     (2)

1	Specify the registry as the route’s service.
2	The route name is identical to the service name.

$ oc get route/docker-registry -o yaml
apiVersion: v1
kind: route
metadata:
  name: docker-registry
spec:
  host: <host> (1)
  to:
    kind: Service
    name: docker-registry (2)
  tls:
    termination: passthrough (3)

1	The host for your route. You must be able to resolve this name externally via DNS to the router’s IP address.
2	The service name for your registry.
3	Specify this route as a passthrough route.

Passthrough is currently the only type of route supported for exposing the secure registry.

Next, you must trust the certificates being used for the registry on your host system. The certificates referenced were created when you secured your registry.
```
$ sudo mkdir -p /etc/docker/certs.d/<host>
$ sudo cp <ca certificate file> /etc/docker/certs.d/<host>
$ sudo systemctl restart docker
```

Log in to the registry using the information from securing the registry. However, this time point to the host name used in the route rather than your service IP. You should now be able to tag and push images using the route host.

$ oc get imagestreams -n test
NAME      DOCKER REPO   TAGS      UPDATED

$ docker pull busybox
$ docker tag busybox <host>/test/busybox
$ docker push <host>/test/busybox
The push refers to a repository [<host>/test/busybox] (len: 1)
8c2e06607696: Image already exists
6ce2e90b0bc7: Image successfully pushed
cf2616975b4a: Image successfully pushed
Digest: sha256:6c7e676d76921031532d7d9c0394d0da7c2906f4cb4c049904c4031147d8ca31

$ docker pull <host>/test/busybox
latest: Pulling from <host>/test/busybox
cf2616975b4a: Already exists
6ce2e90b0bc7: Already exists
8c2e06607696: Already exists
Digest: sha256:6c7e676d76921031532d7d9c0394d0da7c2906f4cb4c049904c4031147d8ca31
Status: Image is up to date for <host>/test/busybox:latest

$ oc get imagestreams -n test
NAME      DOCKER REPO                       TAGS      UPDATED
busybox   172.30.11.215:5000/test/busybox   latest    2 seconds ago

Your image streams will have the IP address and port of the registry service, not the route name and port. See oc get imagestreams for details.

In the <host>/test/busybox example above, test refers to the project name.

Known Issues

The following are the known issues when deploying or using the integrated registry.

Image Push Errors with Scaled Registry Using Shared NFS Volume

When using a scaled registry with a shared NFS volume, you may see one of the following errors during the push of an image:

digest invalid: provided digest did not match uploaded content
blob upload unknown
blob upload invalid

These errors are returned by an internal registry service when Docker attempts to push the image. Its cause originates in the synchronization of file attributes across nodes. Factors such as NFS client side caching, network latency, and layer size can all contribute to potential errors that might occur when pushing an image using the default round-robin load balancing configuration.

You can perform the following steps to minimize the probability of such a failure:

Ensure that the sessionAffinity of your docker-registry service is set to ClientIP:
```
$ oc get svc/docker-registry --template='{{.spec.sessionAffinity}}'
```
This should return ClientIP, which is the default in recent OpenShift Enterprise versions. If not, change it:
```
$ oc get -o yaml svc/docker-registry | \
      sed 's/$sessionAffinity:\s*$.*/\1ClientIP/' | \
      oc replace -f -
```
Ensure that the NFS export line of your registry volume on your NFS server has the no_wdelay options listed. See Export Settings in the Persistent Storage Using NFS topic for details.

Pull of Internally Managed Image Fails with `not found` Error

This error occurs when the pulled image is pushed to an image stream different from the one it is being pulled from. This is caused by re-tagging a built image into an arbitrary image stream:

$ oc tag srcimagestream:latest anyproject/pullimagestream:latest

And subsequently pulling from it, using an image reference such as:

internal.registry.url:5000/anyproject/pullimagestream:latest

During a manual Docker pull, this will produce a similar error:

Error: image anyproject/pullimagestream:latest not found

To prevent this, avoid the tagging of internally managed images completely, or re-push the built image to the desired namespace manually.

Image Push fails with `500 Internal Server Error` on S3 storage

There are problems reported happening when the registry runs on S3 storage back-end. Pushing to a Docker registry occasionally fails with the following error:

Received unexpected HTTP status: 500 Internal Server Error

To debug this, you need to view the registry logs. In there, look for similar error messages occurring at the time of the failed push:

time="2016-03-30T15:01:21.22287816-04:00" level=error msg="unknown error completing upload: driver.Error{DriverName:\"s3\", Enclosed:(*url.Error)(0xc20901cea0)}" http.request.method=PUT
...
time="2016-03-30T15:01:21.493067808-04:00" level=error msg="response completed with error" err.code=UNKNOWN err.detail="s3: Put https://s3.amazonaws.com/oso-tsi-docker/registry/docker/registry/v2/blobs/sha256/ab/abe5af443833d60cf672e2ac57589410dddec060ed725d3e676f1865af63d2e2/data: EOF" err.message="unknown error" http.request.method=PUT
...
time="2016-04-02T07:01:46.056520049-04:00" level=error msg="error putting into main store: s3: The request signature we calculated does not match the signature you provided. Check your key and signing method." http.request.method=PUT
atest

If you see such errors, contact your Amazon S3 support. There may be a problem in your region or with your particular bucket.

Build Fails with `error: build error: Failed to push image: EOF`

Check your registry log. If you see similar error message to the one below:

time="2016-08-10T07:29:06.882023903Z" level=panic msg="Configuration error: OpenShift registry middleware not activated" 2016-08-10 07:29:06.882174 I | http: panic serving 10.131.0.1:34558: &{0xc820010680 map[] 2016-08-10 07:29:06.882023903 +0000 UTC panic Configuration error: OpenShift registry middleware not activated}

It means that your custom configuration file lacks mandatory entries in the middleware section. Add them, re-deploy the registry, and restart your builds.

Image Pruning Fails

If you encounter the following error when pruning images:

BLOB sha256:49638d540b2b62f3b01c388e9d8134c55493b1fa659ed84e97cb59b87a6b8e6c error deleting blob

And your registry log contains the following information:

error deleting blob \"sha256:49638d540b2b62f3b01c388e9d8134c55493b1fa659ed84e97cb59b87a6b8e6c\": operation unsupported

It means that your custom configuration file lacks mandatory entries in the storage section, namely storage:delete:enabled set to true. Add them, re-deploy the registry, and repeat your image pruning operation.

What’s Next?

After you have a registry deployed, you can:

Configure authentication; by default, authentication is set to Deny All.
Deploy a router.

Overview

Deploying the Registry

Registry Compute Resources

Storage for the Registry

Production Use

Use Amazon S3 as a Storage Back-end

Non-Production Use

Maintaining the Registry IP Address

Viewing Logs

File Storage

Accessing the Registry Directly

User Prerequisites

Logging in to the Registry

Pushing and Pulling Images

Securing the Registry

Advanced: Overriding the Registry Configuration

Deploying Updated Configuration

Registry Configuration Reference

Log

Hooks

Storage

Auth

Middleware

CloudFront Middleware

Overriding Middleware Configuration Options

Image Pullthrough

Manifest schema v2 support

Reporting

HTTP

Notifications

Redis

Health

Proxy

Whitelisting Docker Registries

Exposing the Registry

Known Issues

Image Push Errors with Scaled Registry Using Shared NFS Volume

Pull of Internally Managed Image Fails with not found Error

Image Push fails with 500 Internal Server Error on S3 storage

Build Fails with error: build error: Failed to push image: EOF

Image Pruning Fails

What’s Next?

Pull of Internally Managed Image Fails with `not found` Error

Image Push fails with `500 Internal Server Error` on S3 storage

Build Fails with `error: build error: Failed to push image: EOF`