OpenShift Ansible Broker Configuration | Installation and Configuration

Overview

When the OpenShift Ansible broker (OAB) is deployed in a cluster, its behavior is largely dictated by the broker’s configuration file loaded on startup. The broker’s configuration is stored as a configmap object in the broker’s namespace (openshift-ansible-service-broker by default).

Example OpenShift Ansible Broker Configuration File

registry: (1)
  - type: dockerhub
    name: docker
    url: https://registry.hub.docker.com
    org: <dockerhub_org>
    fail_on_error: false
  - type: rhcc
    name: rhcc
    url: https://registry.access.redhat.com
    fail_on_error: true
    white_list:
      - "^foo.*-apb$"
      - ".*-apb$"
    black_list:
      - "bar.*-apb$"
      - "^my-apb$"
  - type: local_openshift
    name: lo
    namespaces:
      - openshift
    white_list:
      - ".*-apb$"
dao: (2)
  etcd_host: localhost
  etcd_port: 2379
log: (3)
  logfile: /var/log/ansible-service-broker/asb.log
  stdout: true
  level: debug
  color: true
openshift: (4)
  host: ""
  ca_file: ""
  bearer_token_file: ""
  image_pull_policy: IfNotPresent
  sandbox_role: "edit"
  keep_namespace: false
  keep_namespace_on_error: true
broker: (5)
  bootstrap_on_startup: true
  dev_broker: true
  launch_apb_on_bind: false
  recovery: true
  output_request: true
  ssl_cert_key: /path/to/key
  ssl_cert: /path/to/cert
  refresh_interval: "600s"
  auth:
    - type: basic
      enabled: true
secrets: (6)
  - title: Database credentials
    secret: db_creds
    apb_name: dh-rhscl-postgresql-apb

1	See Registry Configuration for details.
2	See DAO Configuration for details.
3	See Log Configuration for details.
4	See OpenShift Configuration for details.
5	See Broker Configuration for details.
6	See Secrets Configuration for details.

Modifying the OpenShift Ansible Broker Configuration

To modify the OAB’s default broker configuration after it has been deployed:

Edit the broker-config configmap object in the OAB’s namespace as a user with cluster-admin privileges:
```
$ oc edit configmap broker-config -n openshift-ansible-service-broker
```
After saving any updates, redeploy the OAB’s deployment configuration for the changes to take effect:
```
$ oc rollout latest dc/asb -n openshift-ansible-service-broker
```

Registry Configuration

The registry section allows you to define the registries that the broker should look at for APBs.

Table 1. `registry` Section Configuration Options
Field	Description	Required
`name`	The name of the registry. Used by the broker to identify APBs from this registry.	Y
`user`	The user name for authenticating to the registry. Not used when `auth_type` is set to `secret` or `file`.	N
`pass`	The password for authenticating to the registry. Not used when `auth_type` is set to `secret` or `file`.	N
`auth_type`	How the broker should read the registry credentials if they are not defined in the broker configuration via `user` and `pass`. Can be `secret` (uses a secret in the broker namespace) or `file` (uses a mounted file).	N
`auth_name`	Name of the secret or file storing the registry credentials that should be read. Used when `auth_type` is set to `secret`.	N, only required when `auth_type` is set to `secret` or `file`.
`org`	The namespace or organization that the image is contained in.	N
`type`	The type of registry. Available adapters are `mock`, `rhcc`, `openshift`, `dockerhub`, and `local_openshift`.	Y
`namespaces`	The list of namespaces to configure the `local_openshift` registry type with. By default, a user should use `openshift`.	N
`url`	The URL that is used to retrieve image information. Used extensively for RHCC while the `dockerhub` type uses hard-coded URLs.	N
`fail_on_error`	Should this registry fail, the bootstrap request if it fails. Will stop the execution of other registries loading.	N
`white_list`	The list of regular expressions used to define which image names should be allowed through. Must have a white list to allow APBs to be added to the catalog. The most permissive regular expression that you can use is `.*-apb$` if you would want to retrieve all APBs in a registry. See APB Filtering for more details.	N
`black_list`	The list of regular expressions used to define which images names should never be allowed through. See APB Filtering for more details.	N
`images`	The list of images to be used with an OpenShift Container Registry.	N

Production or Development

A production broker configuration is designed to be pointed at a trusted container distribution registry, such as the Red Hat Container Catalog (RHCC):

registry:
  - name: rhcc
    type: rhcc
    url: https://registry.access.redhat.com
    tag: v3.9
    white_list:
      - ".*-apb$"
  - type: local_openshift
    name: localregistry
    namespaces:
      - openshift
    white_list: []

However, a development broker configuration is primarily used by developers working on the broker. To enable developer settings, set the registry name to dev and the dev_broker field in the broker section to true:

registry:
  name: dev

broker:
  dev_broker: true

Storing Registry Credentials

The broker configuration determines how the broker should read any registry credentials. They can be read from the user and pass values in the registry section, for example:

registry:
  - name: isv
    type: openshift
    url: https://registry.connect.redhat.com
    user: <user>
    pass: <password>

If you want to ensure these credentials are not publicly accessible, the auth_type field in the registry section can be set to the secret or file type. The secret type configures a registry to use a secret from the broker’s namespace, while the file type configures a registry to use a secret that has been mounted as a volume.

To use the secret or file type:

The associated secret should have the values username and password defined. When using a secret, you must ensure that the openshift-ansible-service-broker namespace exists, as this is where the secret will be read from.

For example, create a reg-creds.yaml file:
```
$ cat reg-creds.yaml
---
username: <username>
password: <password>
```

Create a secret from this file in the openshift-ansible-service-broker namespace:

$ oc create secret generic \
    registry-credentials-secret \
    --from-file reg-creds.yaml \
    -n openshift-ansible-service-broker

Choose whether you want to use the secret or file type:

To use the secret type:

In the broker configuration, set auth_type to secret and auth_name to the name of the secret:

registry:
  - name: isv
    type: openshift
    url: https://registry.connect.redhat.com
    auth_type: secret
    auth_name: registry-credentials-secret

Set the namespace where the secret is located:

openshift:
  namespace: openshift-ansible-service-broker

To use the file type:

Edit the asb deployment configuration to mount your file into /tmp/registry-credentials/reg-creds.yaml:

$ oc edit dc/asb -n openshift-ansible-service-broker

In the containers.volumeMounts section, add:

volumeMounts:
  - mountPath: /tmp/registry-credentials
    name: reg-auth

In the volumes section, add:

    volumes:
      - name: reg-auth
        secret:
          defaultMode: 420
          secretName: registry-credentials-secret

In the broker configuration, set auth_type to file and auth_type to the location of the file:

registry:
  - name: isv
    type: openshift
    url: https://registry.connect.redhat.com
    auth_type: file
    auth_name: /tmp/registry-credentials/reg-creds.yaml

Mock Registry

A mock registry is useful for reading local APB specs. Instead of going out to a registry to search for image specs, this uses a list of local specs. Set the name of the registry to mock to use the mock registry.

registry:
  - name: mock
    type: mock

Dockerhub Registry

The dockerhub type allows you to load APBs from a specific organization in the DockerHub. For example, the ansibleplaybookbundle organization.

registry:
  - name: dockerhub
    type: dockerhub
    org: ansibleplaybookbundle
    user: <user>
    pass: <password>
    white_list:
      - ".*-apb$"

APB Filtering

APBs can be filtered out by their image name using a combination of the white_list or black_list parameters, set on a registry basis inside the broker’s configuration.

Both are optional lists of regular expressions that will be run over the total set of discovered APBs for a given registry to determine matches.

Table 2. APB Filter Behavior
Present	Allowed	Blocked
Only whitelist	Matches a regex in list.	Any APB that does not match.
Only blacklist	All APBs that do not match.	APBs that match a regex in list.
Both present	Matches regex in whitelist but not in blacklist.	APBs that match a regex in blacklist.
None	No APBs from the registry.	All APBs from that registry.

For example:

Whitelist Only

white_list:
  - "foo.*-apb$"
  - "^my-apb$"

Anything matching on foo.*-apb$ and only my-apb will be allowed through in this case. All other APBs will be rejected.

Blacklist Only

black_list:
  - "bar.*-apb$"
  - "^foobar-apb$"

Anything matching on bar.*-apb$ and only foobar-apb will be blocked in this case. All other APBs will be allowed through.

Whitelist and Blacklist

white_list:
  - "foo.*-apb$"
  - "^my-apb$"
black_list:
  - "^foo-rootkit-apb$"

Here, foo-rootkit-apb is specifically blocked by the blacklist despite its match in the whitelist because the whitelist match is overridden.

Otherwise, only those matching on foo.*-apb$ and my-apb will be allowed through.

Example Broker Configuration registry Section:

registry:
  - type: dockerhub
    name: dockerhub
    url: https://registry.hub.docker.com
    user: <user>
    pass: <password>
    org: <org>
    white_list:
      - "foo.*-apb$"
      - "^my-apb$"
    black_list:
      - "bar.*-apb$"
      - "^foobar-apb$"

Local OpenShift Container Registry

Using the local_openshift type will allow you to load APBs from the OpenShift Container Registry that is internal to the OKD cluster. You can configure the namespaces in which you want to look for published APBs.

registry:
  - type: local_openshift
    name: lo
    namespaces:
      - openshift
    white_list:
      - ".*-apb$"

Red Hat Container Catalog Registry

Using the rhcc type will allow you to load APBs that are published to the Red Hat Container Catalog (RHCC) registry.

registry:
  - name: rhcc
    type: rhcc
    url: https://registry.access.redhat.com
    white_list:
      - ".*-apb$"

ISV Registry

Using the openshift type allows you to load APBs that are published to the ISV container registry at registry.connect.redhat.com.

registry:
  - name: isv
    type: openshift
    user: <user> (1)
    pass: <password>
    url: https://registry.connect.redhat.com
    images: (2)
      - <image_1>
      - <image_2>
    white_list:
      - ".*-apb$"

1	See Storing Registry Credentials for other authentication options.
2	Because the `openshift` type currently cannot search the configured registry, it is required that you configure the broker with a list of images you would like to source from for when the broker bootstraps. The image names must be the fully qualified name without the registry URL.

Multiple Registries

You can use more than one registry to separate APBs into logical organizations and be able to manage them from the same broker. The registries must have a unique, non-empty name. If there is no unique name, the service broker will fail to start with an error message alerting you to the problem.

registry:
  - name: dockerhub
    type: dockerhub
    org: ansibleplaybookbundle
    user: <user>
    pass: <password>
    white_list:
      - ".*-apb$"
  - name: rhcc
    type: rhcc
    url: <rhcc_url>
    white_list:
      - ".*-apb$"

DAO Configuration

Field Description Required

Field	Description	Required
`etcd_host`	The URL of the etcd host.	Y
`etcd_port`	The port to use when communicating with `etcd_host`.	Y

etcd_host

The URL of the etcd host.

etcd_port

The port to use when communicating with etcd_host.

Log Configuration

Field Description Required

Field	Description	Required
`logfile`	Where to write the broker’s logs.	Y
`stdout`	Write logs to stdout.	Y
`level`	Level of the log output.	Y
`color`	Color the logs.	Y

logfile

Where to write the broker’s logs.

stdout

Write logs to stdout.

level

Level of the log output.

color

Color the logs.

OpenShift Configuration

Field Description Required

Field	Description	Required
`host`	OKD host.	N
`ca_file`	Location of the certificate authority file.	N
`bearer_token_file`	Location of bearer token to be used.	N
`image_pull_policy`	When to pull the image.	Y
`namespace`	The namespace that the broker has been deployed to. Important for things like passing parameter values via secret.	Y
`sandbox_role`	Role to give to an APB sandbox environment.	Y
`keep_namespace`	Always keep namespace after an APB execution.	N
`keep_namespace_on_error`	Keep namespace after an APB execution has an error.	N

host

OKD host.

ca_file

Location of the certificate authority file.

bearer_token_file

Location of bearer token to be used.

image_pull_policy

When to pull the image.

namespace

The namespace that the broker has been deployed to. Important for things like passing parameter values via secret.

sandbox_role

Role to give to an APB sandbox environment.

keep_namespace

Always keep namespace after an APB execution.

keep_namespace_on_error

Keep namespace after an APB execution has an error.

Broker Configuration

The broker section tells the broker what functionality should be enabled and disabled. It will also tell the broker where to find files on disk that will enable the full functionality.

Field Description Default Value Required

Field	Description	Default Value	Required
`dev_broker`	Allow development routes to be accessible.	`false`	N
`launch_apb_on_bind`	Allow bind to be a no-op.	`false`	N
`bootstrap_on_startup`	Allow the broker attempt to bootstrap itself on start up. Will retrieve the APBs from configured registries.	`false`	N
`recovery`	Allow the broker to attempt to recover itself by dealing with pending jobs noted in etcd.	`false`	N
`output_request`	Allow the broker to output the requests to the log file as they come in for easier debugging.	`false`	N
`ssl_cert_key`	Tells the broker where to find the TLS key file. If not set, the API server will attempt to create one.	`""`	N
`ssl_cert`	Tells the broker where to find the TLS *.crt* file. If not set, the API server will attempt to create one.	`""`	N
`refresh_interval`	The interval to query registries for new image specs.	`"600s"`	N
`auto_escalate`	Allows the broker to escalate the permissions of a user while running the APB.	`false`	N
`cluster_url`	Sets the prefix for the URL that the broker is expecting.	`ansible-service-broker`	N

dev_broker

Allow development routes to be accessible.

false

launch_apb_on_bind

Allow bind to be a no-op.

false

bootstrap_on_startup

Allow the broker attempt to bootstrap itself on start up. Will retrieve the APBs from configured registries.

false

recovery

Allow the broker to attempt to recover itself by dealing with pending jobs noted in etcd.

false

output_request

Allow the broker to output the requests to the log file as they come in for easier debugging.

false

ssl_cert_key

Tells the broker where to find the TLS key file. If not set, the API server will attempt to create one.

""

ssl_cert

Tells the broker where to find the TLS .crt file. If not set, the API server will attempt to create one.

""

refresh_interval

The interval to query registries for new image specs.

"600s"

auto_escalate

Allows the broker to escalate the permissions of a user while running the APB.

false

cluster_url

Sets the prefix for the URL that the broker is expecting.

ansible-service-broker

Async bind and unbind is an experimental feature and is not supported or enabled by default. With the absence of async bind, setting launch_apb_on_bind to true can cause the bind action to timeout and will span a retry. The broker will handle this with "409 Conflicts" because it is the same bind request with different parameters.

Secrets Configuration

The secrets section creates associations between secrets in the broker’s namespace and APBs the broker runs. The broker uses these rules to mount secrets into running APBs, allowing the user to use secrets to pass parameters without exposing them to the catalog or users.

The section is a list where each entry has the following structure:

Field Description Required

Field	Description	Required
`title`	The title of the rule. This is just for display and output purposes.	Y
`apb_name`	The name of the APB to associate with the specified secret. This is the fully qualified name (`<registry_name>-<image_name>`).	Y
`secret`	The name of the secret to pull parameters from.	Y

title

The title of the rule. This is just for display and output purposes.

apb_name

The name of the APB to associate with the specified secret. This is the fully qualified name (<registry_name>-<image_name>).

secret

The name of the secret to pull parameters from.

You can download and use the create_broker_secret.py file to create and format this configuration section.

secrets:
- title: Database credentials
  secret: db_creds
  apb_name: dh-rhscl-postgresql-apb

Running Behind a Proxy

When running the OAB inside of a proxied OKD cluster, it is important to understand its core concepts and consider them within the context of a proxy used for external network access.

As an overview, the broker itself runs as a pod within the cluster. It has a requirement for external network access depending on how its registries have been configured.

Registry Adapter Whitelists

The broker’s configured registry adapters must be able to communicate with their external registries in order to bootstrap successfully and load remote APB manifests. These requests can be made via the proxy, however, the proxy must ensure that the required remote hosts are accessible.

Example required whitelisted hosts:

Registry Adapter Type Whitelisted Hosts

Registry Adapter Type	Whitelisted Hosts
`rhcc`	`registry.access.redhat.com`, `access.redhat.com`
`dockerhub`	`docker.io`

rhcc

registry.access.redhat.com, access.redhat.com

dockerhub

docker.io

Configuring the Broker Behind a Proxy Using Ansible

If during initial installation you configure your OKD cluster to run behind a proxy (see Configuring Global Proxy Options), when the OAB is deployed it will:

inherit those cluster-wide proxy settings automatically and
generate the required NO_PROXY list, including the cidr fields and serviceNetworkCIDR,

and no further configuration is needed.

Configuring the Broker Behind a Proxy Manually

If your cluster’s global proxy options were not configured during initial installation or prior to the broker being deployed, or if you have modified the global proxy settings, you must manually configure the broker for external access via proxy:

Before attempting to run the OAB behind a proxy, review Working with HTTP Proxies and ensure your cluster is configured accordingly to run behind a proxy.

In particular, the cluster must be configured to not proxy internal cluster requests. This is typically configured with a NO_PROXY setting of:

.cluster.local,.svc,<serviceNetworkCIDR_value>,<master_IP>,<master_domain>,.default

in addition to any other desired NO_PROXY settings. See Configuring NO_PROXY for more details.

Brokers deploying unversioned, or v1 APBs must also add 172.30.0.1 to their NO_PROXY list. APBs prior to v2 extracted their credentials from running APB pods via an exec HTTP request, rather than a secret exchange. Unless you are running a broker with experimental proxy support in a cluster prior to OKD 3.9, you probably do not have to worry about this.

Edit the broker’s DeploymentConfig as a user with cluster-admin privileges:
```
$ oc edit dc/asb -n openshift-ansible-service-broker
```
Set the following environment variables:
- HTTP_PROXY
- HTTPS_PROXY
- NO_PROXY
See Setting Proxy Environment Variables in Pods for more information.
After saving any updates, redeploy the OAB’s deployment configuration for the changes to take effect:
```
$ oc rollout latest dc/asb -n openshift-ansible-service-broker
```

Setting Proxy Environment Variables in Pods

It is common that APB pods themselves may require external access via proxy as well. If the broker recognizes it has a proxy configuration, it will transparently apply these environment variables to the APB pods that it spawns. As long as the modules used within the APB respect proxy configuration via environment variable, the APB will also use these settings to perform its work.

Finally, it is possible the services spawned by the APB may also require external network access via proxy. The APB must be authored to set these environment variables explicitly if recognizes them in its own execution environment, or the cluster operator must manually modify the required services to inject them into their environments.