Master and Node Configuration

Overview
Prerequisites
Configuring Masters and Nodes
- Making Configuration Changes Using Ansible
- Making Manual Configuration Changes
Configuration Options
- Master Configuration File Options
- Node Configuration File Options

Overview

The master and node configuration files determine the make-up of your OpenShift Container Platform cluster, and define a range of options. These include overriding the default plug-ins, connecting to etcd, automatically creating service accounts, building image names, customizing project requests, configuring volume plug-ins, and much more.

This topic covers the many options available for customizing your OpenShift Container Platform masters and nodes, and shows you how to make changes to the configuration after installation.

The /etc/origin/master/master-config.yaml and /etc/origin/node/node-config.yaml files define a wide range of options that can be configured on the OpenShift master and nodes. These options include overriding the default plug-ins, connecting to etcd, automatically creating service accounts, building image names, customizing project requests, configuring volume plug-ins, and much more.

Prerequisites

For testing environments deployed via the quick install, one master should be sufficient. The quick installation method should not be used for production environments.

Production environments should be installed using the advanced install. In production environments, it is a good idea to use multiple masters for the purposes of high availability (HA). A cluster architecture of three masters is recommended, and HAproxy is the recommended solution for this.

If etcd is being installed on the master hosts, you must configure your cluster to use at least three masters. It cannot use only two masters, because etcd would not be able to decide which one is authoritative. The only way to successfully run only two masters is if you install etcd on hosts other than the masters.

Configuring Masters and Nodes

The method you use to configure your master and node configuration files must match the method that was used to install your OpenShift Container Platform cluster. If you followed the:

Advanced installation method using Ansible, then make your configuration changes in the Ansible playbook.
Quick installation method, then make your changes manually in the configuration files themselves.

Making Configuration Changes Using Ansible

For this section, familiarity with Ansible is assumed.

Only a portion of the available host configuration options are exposed to Ansible. After an OpenShift Container Platform install, Ansible creates an inventory file with some substituted values. Modifying this inventory file and re-running the Ansible installer playbook is how you customize your OpenShift Container Platform cluster.

While OpenShift supports using Ansible as the advanced install method, using an Ansible playbook and inventory file, you can also use other management tools, such as Puppet, Chef, Salt).

Use Case: Configure the cluster to use HTPasswd authentication

This use case assumes you have already set up SSH keys to all the nodes referenced in the playbook.
The htpasswd utility is in the httpd-tools package:
```
# yum install httpd-tools
```

To modify the Ansible inventory and make configuration changes:

Open the ./hosts inventory file:

Example 1. Sample inventory file:

[OSEv3:children]
masters
nodes

[OSEv3:vars]
ansible_ssh_user=cloud-user
ansible_become=true
openshift_deployment_type=openshift-enterprise

[masters]
ec2-52-6-179-239.compute-1.amazonaws.com  openshift_ip=172.17.3.88 openshift_public_ip=52-6-179-239 openshift_hostname=master.example.com  openshift_public_hostname=ose3-master.public.example.com containerized=True
[nodes]
ec2-52-6-179-239.compute-1.amazonaws.com  openshift_ip=172.17.3.88 openshift_public_ip=52-6-179-239 openshift_hostname=master.example.com  openshift_public_hostname=ose3-master.public.example.com containerized=True openshift_schedulable=False
ec2-52-95-5-36.compute-1.amazonaws.com  openshift_ip=172.17.3.89 openshift_public_ip=52.3.5.36 openshift_hostname=node.example.com openshift_public_hostname=ose3-node.public.example.com containerized=True

Add the following new variables to the [OSEv3:vars] section of the file:

# htpasswd auth
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
# Defining htpasswd users
openshift_master_htpasswd_users={'<name>': '<hashed-password>', '<name>': '<hashed-password>'}
# or
#openshift_master_htpasswd_file=<path/to/local/pre-generated/htpasswdfile>

For HTPasswd authentication, you can use either the openshift_master_htpasswd_users variable to create the specified user(s) and password(s) or the openshift_master_htpasswd_file variable to specify a pre-generated flat file (the htpasswd file) with the users and passwords already created.

Because OpenShift Container Platform requires a hashed password to configure HTPasswd authentication, you can use the htpasswd command, as shown in the following section, to generate the hashed password(s) for your user(s) or to create the flat file with the users and associated hashed passwords.

The following example changes the authentication method from the default deny all setting to htpasswd and use the specified file to generate user IDs and passwords for the jsmith and bloblaw users.

# htpasswd auth
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
# Defining htpasswd users
openshift_master_htpasswd_users={'jsmith': '$apr1$wIwXkFLI$bAygtKGmPOqaJftB', 'bloblaw': '7IRJ$2ODmeLoxf4I6sUEKfiA$2aDJqLJe'}
# or
#openshift_master_htpasswd_file=<path/to/local/pre-generated/htpasswdfile>

Re-run the ansible playbook for these modifications to take effect:
```
$ ansible-playbook -b -i ./hosts ~/src/openshift-ansible/playbooks/byo/config.yml
```
The playbook updates the configuration, and restarts the OpenShift master service to apply the changes.

You have now modified the master and node configuration files using Ansible, but this is just a simple use case. From here you can see which master and node configuration options are exposed to Ansible and customize your own Ansible inventory.

Using the `htpasswd` commmand

To configure the OpenShift Container Platform cluster to use HTPasswd authentication, you need at least one user with a hashed password to include in the inventory file.

You can:

Generate the username and password to add directly to the ./hosts inventory file.
Create a flat file to pass the credentials to the ./hosts inventory file.

To create a user and hashed password:

Run the following command to add the specified user:
```
$ htpasswd -n <user_name>
```
You can include the -b option to supply the password on the command line:

$ htpasswd -nb <user_name> <password>
Enter and confirm a clear-text password for the user.

For example:
```
$ htpasswd -n myuser
New password:
Re-type new password:
myuser:$apr1$vdW.cI3j$WSKIOzUPs6Q
```
The command generates a hashed version of the password.

You can then use the hashed password when configuring HTPasswd authentication. The hashed password is the string after the :. In the above example,you would enter:

openshift_master_htpasswd_users={'myuser': '$apr1$wIwXkFLI$bAygtISk2eKGmqaJftB'}

To create a flat file with a user name and hashed password:

Execute the following command:
```
$ htpasswd -c </path/to/users.htpasswd> <user_name>
```
You can include the -b option to supply the password on the command line:

$ htpasswd -c -b <user_name> <password>
Enter and confirm a clear-text password for the user.

For example:
```
htpasswd -c users.htpasswd user1
New password:
Re-type new password:
Adding password for user user1
```
The command generates a file that includes the user name and a hashed version of the user’s password.

You can then use the password file when configuring HTPasswd authentication.

For more information on the htpasswd command, see HTPasswd Identity Provider.

Making Manual Configuration Changes

After installing OpenShift Container Platform using the quick install, you can make modifications to the master and node configuration files to customize your cluster.

Use Case: Configure the cluster to use HTPasswd authentication

To manually modify a configuration file:

Open the configuration file you want to modify, which in this case is the /etc/origin/master/master-config.yaml file:

Add the following new variables to the identityProviders stanza of the file:

oauthConfig:
  ...
  identityProviders:
  - name: my_htpasswd_provider
    challenge: true
    login: true
    mappingMethod: claim
    provider:
      apiVersion: v1
      kind: HTPasswdPasswordIdentityProvider
      file: /path/to/users.htpasswd

Save your changes and close the file.

Restart the master for the changes to take effect:

$ systemctl restart atomic-openshift-master-api atomic-openshift-master-controllers

You have now manually modified the master and node configuration files, but this is just a simple use case. From here you can see all the master and node configuration options, and further customize your own cluster by making further modifications.

Configuration Options

Master Configuration File Options

The table below contains the options available for configuring your OpenShift Container Platform master-config.yaml file. Use this table as a reference, and then follow the section on making manual configuration changes and substitute in whatever values you want to change.

Table 1. Master Configuration File Options
Option	Description
`servingInfo`	Describes how to start serving. For example: servingInfo: bindAddress: 0.0.0.0:8443 bindNetwork: tcp4 certFile: master.server.crt clientCA: ca.crt keyFile: master.server.key maxRequestsInFlight: 500 requestTimeoutSeconds: 3600
`corsAllowedOrigins`	Specifies the host name to use to access the API server from a web application.
`apiLevels`	A list of API levels that should be enabled on startup; for example, `v1beta3` and `v1`.
`apiServerArguments`	Contains key value pairs that match the API server’s command-line arguments and are passed directly to the Kubernetes API server. These are not migrated, but if you reference a value that does not exist, then the server will not start. apiServerArguments: event-ttl: - "15m"
`assetConfig`	If present, then the asset server starts based on the defined parameters. For example: assetConfig: logoutURL: "" masterPublicURL: https://master.ose32.example.com:8443 publicURL: https://master.ose32.example.com:8443/console/ servingInfo: bindAddress: 0.0.0.0:8443 bindNetwork: tcp4 certFile: master.server.crt clientCA: "" keyFile: master.server.key maxRequestsInFlight: 0 requestTimeoutSeconds: 0
`controllers`	A list of the controllers that should be started. If set to `none`, then no controllers will start automatically. The default value is `` which will start all controllers. When using ``, you may exclude controllers by prepending a `-` in front of the controller name. No other values are recognized at this time.
`pauseControllers`	When set to `true`, this instructs the master to not automatically start controllers, but instead to wait until a notification to the server is received before launching them.
`controllerLeaseTTL`	Enables controller election, instructing the master to attempt to acquire a lease before controllers start, and renewing it within a number of seconds defined by this value. Setting this value as a non-negative forces `pauseControllers=true`. The value default is off (`0`, or omitted) and controller election can be disabled with `-1`.
`admissionConfig`	Contains admission control plug-in configuration. OpenShift has a configurable list of admission controller plug-ins that are triggered whenever API objects are created or modified. This option allows you to override the default list of plug-ins; for example, disabling some plug-ins, adding others, changing the ordering, and specifying configuration. Both the list of plug-ins and their configuration can be controlled from Ansible.
`disabledFeatures`	Lists features that should not be started. This is defined as `omitempty` because it is unlikely that you would want to manually disable features.
`etcdStorageConfig`	Contains information about how API resources are stored in etcd. These values are only relevant when etcd is the backing store for the cluster.
`etcdClientInfo`	Contains information about how to connect to etcd. Specifies if etcd is run as embedded or non-embedded, and the hosts. The rest of the configuration is handled by the Ansible inventory. For example: etcdClientInfo: ca: ca.crt certFile: master.etcd-client.crt keyFile: master.etcd-client.key urls: - https://m1.aos.example.com:4001
`kubernetesMasterConfig`	Contains information about how to connect to kubelet’s KubernetesMasterConfig. If present, then start the kubernetes master in this process.
`etcdConfig`	If present, then etcd starts based on the defined parameters. For example: etcdConfig: address: master.ose32.example.com:4001 peerAddress: master.ose32.example.com:7001 peerServingInfo: bindAddress: 0.0.0.0:7001 certFile: etcd.server.crt clientCA: ca.crt keyFile: etcd.server.key servingInfo: bindAddress: 0.0.0.0:4001 certFile: etcd.server.crt clientCA: ca.crt keyFile: etcd.server.key storageDirectory: /var/lib/origin/openshift.local.etcd
`oauthConfig`	If present, then the /oauth endpoint starts based on the defined parameters. For example: oauthConfig: assetPublicURL: https://master.ose32.example.com:8443/console/ grantConfig: method: auto identityProviders: - challenge: true login: true mappingMethod: claim name: htpasswd_all provider: apiVersion: v1 kind: HTPasswdPasswordIdentityProvider file: /etc/origin/openshift-passwd masterCA: ca.crt masterPublicURL: https://master.ose32.example.com:8443 masterURL: https://master.ose32.example.com:8443 sessionConfig: sessionMaxAgeSeconds: 3600 sessionName: ssn sessionSecretsFile: /etc/origin/master/session-secrets.yaml tokenConfig: accessTokenMaxAgeSeconds: 86400 authorizeTokenMaxAgeSeconds: 500
`assetConfig`	If present, then the asset server starts based on the defined parameters. For example: assetConfig: logoutURL: "" masterPublicURL: https://master.ose32.example.com:8443 publicURL: https://master.ose32.example.com:8443/console/ servingInfo: bindAddress: 0.0.0.0:8443 bindNetwork: tcp4 certFile: master.server.crt clientCA: "" keyFile: master.server.key maxRequestsInFlight: 0 requestTimeoutSeconds: 0
`dnsConfig`	If present, then start the DNS server based on the defined parameters. For example: dnsConfig: bindAddress: 0.0.0.0:8053 bindNetwork: tcp4
`serviceAccountConfig`	Holds options related to service accounts: `LimitSecretReferences` (boolean): Controls whether or not to allow a service account to reference any secret in a namespace without explicitly referencing them. `ManagedNames` (string): A list of service account names that will be auto-created in every namespace. If no names are specified, then the `ServiceAccountsController` will not be started. `MasterCA` (string): The certificate authority for verifying the TLS connection back to the master. The service account controller will automatically inject the contents of this file into pods so that they can verify connections to the master. `PrivateKeyFile` (string): Contains a PEM-encoded private RSA key, used to sign service account tokens. If no private key is specified, then the service account `TokensController` will not be started. `PublicKeyFiles` (string): A list of files, each containing a PEM-encoded public RSA key. If any file contains a private key, then OpenShift uses the public portion of the key. The list of public keys is used to verify service account tokens; each key is tried in order until either the list is exhausted or verification succeeds. If no keys are specified, then service account authentication will not be available.
`masterClients`	Holds all the client connection information for controllers and other system components: `OpenShiftLoopbackKubeConfig` (string): the .kubeconfig filename for system components to loopback to this master. `ExternalKubernetesKubeConfig` (string): the .kubeconfig filename for proxying to Kubernetes.
`imageConfig`	Holds options that describe how to build image names for system components: `Format` (string): Describes how to determine image names for system components `Latest` (boolean): Defines whether to attempt to use the latest system component images or the latest release.
`imagePolicyConfig`	Controls limits and behavior for importing images: `MaxImagesBulkImportedPerRepository` (integer): Controls the number of images that are imported when a user does a bulk import of a Docker repository. This number is set low to prevent users from importing large numbers of images accidentally. This can be set to `-1` for no limit. `DisableScheduledImport` (boolean): Allows scheduled background import of images to be disabled. `ScheduledImageImportMinimumIntervalSeconds` (integer): The minimum number of seconds that can elapse between when image streams scheduled for background import are checked against the upstream repository. The default value is `900` (15 minutes). `MaxScheduledImageImportsPerMinute` (integer): The maximum number of image streams that can be imported in the background, per minute. The default value is `60`. This can be set to `-1` for unlimited imports. This can be controlled with the Ansible inventory.
`policyConfig`	Holds information about where to locate critical pieces of bootstrapping policy. This is controlled by Ansible, so you may not need to modify this: `BootstrapPolicyFile` (string): Points to a template that contains roles and rolebindings that will be created if no policy object exists in the master namespace. `OpenShiftSharedResourcesNamespace` (string): The namespace where shared OpenShift resources are located, such as shared templates. `OpenShiftInfrastructureNamespace` (string): The namespace where OpenShift infrastructure resources are located, such as controller service accounts.
`projectConfig`	Holds information about project creation and defaults: `DefaultNodeSelector` (string): Holds the default project node label selector. `ProjectRequestMessage` (string): The string presented to a user if they are unable to request a project via the projectrequest API endpoint. `ProjectRequestTemplate` (string): The template to use for creating projects in response to projectrequest. It is in the format `<namespace>/<template>`. It is optional, and if it is not specified, a default template is used. `SecurityAllocator`: Controls the automatic allocation of UIDs and MCS labels to a project. If nil, allocation is disabled: `mcsAllocatorRange` (string): Defines the range of MCS categories that will be assigned to namespaces. The format is `<prefix>/<numberOfLabels>[,<maxCategory>]`. The default is `s0/2` and will allocate from c0 → c1023, which means a total of 535k labels are available. If this value is changed after startup, new projects may receive labels that are already allocated to other projects. The prefix may be any valid SELinux set of terms (including user, role, and type). However, leaving the prefix at its default allows the server to set them automatically. For example, `s0:/2` would allocate labels from s0:c0,c0 to s0:c511,c511 whereas `s0:/2,512` would allocate labels from s0:c0,c0,c0 to s0:c511,c511,511. `mcsLabelsPerProject` (integer): Defines the number of labels to reserve per project. The default is `5` to match the default UID and MCS ranges. `uidAllocatorRange` (string): Defines the total set of Unix user IDs (UIDs) automatically allocated to projects, and the size of the block each namespace gets. For example, `1000-1999/10` would allocate ten UIDs per namespace, and would be able to allocate up to 100 blocks before running out of space. The default is to allocate from 1 billion to 2 billion in 10k blocks, which is the expected size of ranges for container images when user namespaces are started.
`routingConfig`	Holds information about routing and route generation: `Subdomain` (string): The suffix appended to $service.$namespace. to form the default route hostname. Can be controlled via Ansible with `openshift_master_default_subdomain`. Example: routingConfig: subdomain: ""
`networkConfig`	To be passed to the compiled-in-network plug-in. Many of the options here can be controlled in the Ansible inventory. `NetworkPluginName` (string) `ClusterNetworkCIDR` (string) `HostSubnetLength` (unsigned integer) `ServiceNetworkCIDR` (string) `ExternalIPNetworkCIDRs` (string array): Controls which values are acceptable for the service external IP field. If empty, no external IP may be set. It can contain a list of CIDRs which are checked for access. If a CIDR is prefixed with `!`, then IPs in that CIDR are rejected. Rejections are applied first, then the IP is checked against one of the allowed CIDRs. For security purposes, you should ensure this range does not overlap with your nodes, pods, or service CIDRs. For Example: networkConfig: clusterNetworkCIDR: 10.3.0.0/16 hostSubnetLength: 8 networkPluginName: example/openshift-ovs-subnet # serviceNetworkCIDR must match kubernetesMasterConfig.servicesSubnet serviceNetworkCIDR: 179.29.0.0/16
`volumeConfig`	Contains options for configuring volume plug-ins in the master node: `DynamicProvisioningEnabled` (boolean): Default value is `true`, and toggles dynamic provisioning off when `false`.

Node Configuration File Options

The table below contains the options available for configuring your OpenShift Container Platform node-config.yaml file. Use this table as a reference, and then follow the section on making manual configuration changes and substitute in whatever values you want to change.

Table 2. Node Configuration File Options
Option	Description
`nodeName`	The value of the `nodeName` (string) is used to identify this particular node in the cluster. If possible, this should be your fully qualified hostname. If you are describing a set of static nodes to the master, then this value must match one of the values in the list.
`nodeIP`	A node may have multiple IPs. This specifies the IP to use for pod traffic routing. If left unspecified, a network look-up is performed on the nodeName, and the first non-loopback address is used.
`servingInfo`	Describes how to start serving.
`masterKubeConfig`	The filename for the .kubeconfig file that describes how to connect this node to the master.
`dnsDomain`	Holds the domain suffix.
`dnsIP`	(string) Contains the IP. Can be controlled with `openshift_dns_ip` in the Ansible inventory.
`dockerConfig`	Holds Docker-related configuration options.
`imageConfig`	Holds options that describe how to build image names for system components.
`iptablesSyncPeriod`	(string) How often iptables rules are refreshed. This can be controlled with `openshift_node_iptables_sync_period` from the Ansible inventory. If the `dnsIP` parameter is omitted, the value defaults to the kubernetes service IP, which is the first nameserver in the pod’s */etc/resolv.conf* file.
`kubeletArguments,omitempty`	Key-value pairs that are passed directly to the Kubelet that matches the Kubelet’s command line arguments. These are not migrated or validated, so if you use them, then they may become invalid. Use caution, because these values override other settings in the node configuration that may cause invalid configurations.
`masterKubeConfig`	The filename for the .kubeconfig file that describes how to connect this node to the master.
`networkPluginName,omitempty`	Deprecated and maintained for backward compatibility, use `NetworkConfig.NetworkPluginName` instead.
`networkConfig`	Provides network options for the node: `NetworkPluginName` (string): Specifies the networking plug-in. `MTU` (unsigned integer): Maximum transmission unit for the network packets.
`volumeDirectory`	The directory that volumes will be stored under.
`imageConfig`	Holds options that describe how to build image names for system components.
`allowDisabledDocker`	If this is set to `true`, then the Kubelet will ignore errors from Docker. This means that a node can start on a machine that does not have Docker started.
`podManifestConfig`	Holds the configuration for enabling the Kubelet to create pods based from a manifest file or files placed locally on the node.
`authConfig`	Holds authn/authz configuration options.
`dockerConfig`	Holds Docker-related configuration options.
`kubeletArguments,omitempty`	Key-value pairs that are passed directly to the Kubelet that matches the Kubelet’s command line arguments. These are not migrated or validated, so if you use them, then they may become invalid. Use caution, because these values override other settings in the node configuration that may cause invalid configurations.
`proxyArguments,omitempty`	`ProxyArguments` are key-value pairs that are passed directly to the Proxy that matches the Proxy’s command-line arguments. These are not migrated or validated, so if you use them they may become invalid. Use caution, because these values override other settings in the node configuration that may cause invalid configurations.
`iptablesSyncPeriod`	(string) How often iptables rules are refreshed. This can be controlled with `openshift_node_iptables_sync_period` from the Ansible inventory.
`volumeConfig`	Contains options for configuring volumes on the node. It can be used to apply a filesystem quota if the underlying volume directory is on XFS with grpquota enabled.