Configuring for Google Compute Engine | Configuring Clusters

Before you begin
- Configuring authorization for Google Cloud Platform
- Google Compute Engine objects
Configuring OpenShift Container Platform for GCE
Using the GCP external load balancer as a service

You can configure OpenShift Container Platform to access an existing Google Compute Engine (GCE) infrastructure, including using GCE volumes as persistent storage for application data.

Before you begin

Configuring authorization for Google Cloud Platform

Roles

Configuring GCP for OpenShift Container Platform requires the following GCP role:

roles/owner

Needed for creating service accounts, cloud storage, instances, images, templates, Cloud DNS entries, and to deploy load balancers and health checks.

delete permissions might also be required if the user is expected to redeploy the environment during testing phases.

You can also create a service account to avoid using personal users when deploying GCP objects.

See the Understanding roles section of the GCP documentation for more information, including steps for how to configure roles.

Scopes and service accounts

GCP uses scopes to determine if an authenticated identity is authorized to perform operations within a resource. For example, if application A with a read-only scope access token can only read, while application B with a read-write scope access token can read and modify data.

The scopes are defined at the GCP API level as https://www.googleapis.com/auth/compute.readonly.

You can specify scopes using the --scopes=[SCOPE,…] option when creating instances, or you can use the --no-scopes option to create the instance without scopes if you don’t want the instance accessing the GCP API.

See the Scopes section of the GCP documentation for more information.

All GCP projects include a default [PROJECT_NUMBER]-compute@developer.gserviceaccount.com service account with project editor permissions.

By default, a newly created instance is automatically enabled to run as the default service account with the following access scopes:

You can specify another service account with the --service-account=SERVICE_ACCOUNT option when creating the instance, or explicitly disabling service accounts for the instance using the --no-service-account option using the gcloud CLI.

See the Creating a new service account section of the GCP documentation for more information.

Google Compute Engine objects

Integrating OpenShift Container Platform with Google Compute Engine (GCE) requires the following components or services.

A GCP project: A GCP project is the base level organizing entity that forms the basis for creating, enabling, and using all GCP services. This includes managing APIs, enabling billing, adding and removing collaborators, and managing permissions.

See the project resource section in the GCP documentation for more information.

Project IDs are unique identifiers, and project IDs must be unique across all of Google Cloud Engine. This means you cannot use myproject as a project ID if someone else has created a project with that ID before.

Billing: You cannot create new resources unless billing is attached to an account. The new project can be linked to an existing project or new information can be entered.

See Create, Modify, or Close Your Billing Account in the GCP documentation for more information.

Cloud identity and access management: Deploying OpenShift Container Platform requires the proper permissions. A user must be able to create service accounts, cloud storage, instances, images, templates, Cloud DNS entries, and deploy load balancers and health checks. Delete permissions are also helpful in order to be able to redeploy the environment while testing.

You can create service accounts with specific permissions, then use them to deploy infrastructure components instead of regular users. You can also create roles to limit access to different users or service accounts.

GCP instances use service accounts to allow applications to call GCP APIs. For example, OpenShift Container Platform node hosts can call the GCP disk API to provide a persistent volume to an application.

Access control to the various infrastructure, service resources, and fine-grained roles are available using the IAM service. For more information, see the Access cloud overview section of the GCP documentation.

SSH keys: GCP injects SSH public keys as authorized keys so you can log in using SSH in the created instances. You can configure the SSH keys per instance or per project.

You can use existing SSH keys. GCP metadata can help with storing the SSH keys that are injected at boot time in the instances to allow SSH access.

See the Metadata section of the GCP documentation for more information.

GCP regions and zones: GCP has a global infrastructure that covers regions and availability zones. While deploying OpenShift Container Platform in GCP on different zones can help avoid single-point-of-failures, there are some caveats regarding storage.

GCP disks are created within a zone. Therefore, if a OpenShift Container Platform node host goes down in zone "A" and the pods move to zone "B", the persistent storage cannot be attached to those pods because the disks are in a different zone.

Deploying a single zone of multizone OpenShift Container Platform environment is an important decision to make before installing OpenShift Container Platform. If deploying a multizone environment, the recommended setup is to use three different zones in a single region.

See the GCP documentation on regions and zones and the Kubernetes documentation on multiple zones for more information.

External IP address: So that GCP instances can communicate with the Internet, you must attach an external IP address to the instance. Also, an external IP address is required to communicate with instances deployed in GCP from outside the Virtual Private Cloud (VPC) Network.

Requiring an External IP address for internet access is a limitation of the provider. You can configure firewall rules to block incoming external traffic in instances if not needed.

See the GCP documentation on external IP address for more information.

Cloud DNS: GCP cloud DNS is a DNS service used to publish domain names to the global DNS using GCP DNS servers.

The public cloud DNS zone requires a domain name that you purchased either through Google’s "Domains" service or through a third-party provider. When you create the zone, you must add the name servers provided by Google to the registrar.

See the GCP documentation on Cloud DNS for more information.

GCP VPC networks have an internal DNS service that automatically resolves internal host names.

The internal fully qualified domain name (FQDN) for an instance follows the [HOST_NAME].c.[PROJECT_ID].internal format.

See the GCP documentation on Internal DNS for more information.

Load balancing: The GCP load balancing service enables the distribution of traffic across multiple instances in the GCP cloud.

There are five types of Load Balancing:

HTTPS and TCP proxy load balancing are the only options for using HTTPS health checks for master nodes, which checks the status of /healthz.

Because HTTPS load balancing requires a custom certificate, this implementation uses TCP Proxy load balancing to simplify the process.

See the GCP documentation on Load balancing for more information.

Instances sizes: A successful OpenShift Container Platform environment requires some minimum hardware requirements:

Table 1. Instances sizes
Role	Size
Master	`n1-standard-8`
Node	`n1-standard-4`

GCP allows you to create custom instance sizes to fit different requirements. See Creating an Instance with a Custom Machine Type for more information, or see Machine types and OpenShift Container Platform Minimum Hardware Requirements for more information about instance sizes.

Storage Options

By default, each GCP instance has a small root persistent disk that contains the operating system. When applications running on the instance require more storage space, you can add additional storage options to the instance:

Standard persistent disks
SSD persistent disks
Local SSDs
Cloud storage buckets

For more information, see the GCP documentation on Storage options.

Configuring OpenShift Container Platform for GCE

You can configure OpenShift Container Platform for GCE in two ways:

Using Ansible
Manually by modifying the master-config.yaml file

Option 1: Configuring OpenShift Container Platform for GCP using Ansible

You can configure OpenShift Container Platform for Google Compute Platform (GCP) by modifying the Ansible inventory file at installation time or after installation.

Procedure

At minimum, you must define the openshift_cloudprovider_kind, openshift_gcp_project and openshift_gcp_prefix parameters, as well as the optional openshift_gcp_multizone for multizone deployments and openshift_gcp_network_name if you are not using the default network name.

Add the following section to the Ansible inventory file at installation to configure your OpenShift Container Platform environment for GCP:

[OSEv3:vars]
openshift_cloudprovider_kind=gce
openshift_gcp_project=<projectid> (1)
openshift_gcp_prefix=<uid> (2)
openshift_gcp_multizone=False (3)
openshift_gcp_network_name=<network name> (4)

1	Provide the GCP project ID where the existing instances are running. This ID is generated when you create the project in the Google Cloud Platform Console.
2	Provide a unique string to identify each OpenShift Container Platform cluster. This must be unique across GCP.
3	Optionally, set to `True` to trigger multizone deployments on GCP. Set to `False` by default.
4	Optionally, provide the network name if not using `default` network.

Installing with Ansible also creates and configures the following files to fit your GCP environment:

/etc/origin/cloudprovider/gce.conf
/etc/origin/master/master-config.yaml
/etc/origin/node/node-config.yaml

If you are running load balancer services using GCP, the Compute Engine VM node instances require the ocp suffix. For example, if the value of the openshift_gcp_prefix parameter is set to mycluster, you must tag the nodes with myclusterocp. See Adding and Removing Network Tags for more information on how to add network tags to Compute Engine VM instances.
Optionally, you can configure multizone support.

The cluster installation process configures single-zone support by default, but you can configure for multiple zones to avoid single-point-of-failures.

Because GCP disks are created within a zone, deploying OpenShift Container Platform in GCP on different zones can cause problems with storage. If an OpenShift Container Platform node host goes down in zone "A" and the pods move to zone "B", the persistent storage cannot be attached to those pods because the disks are now in a different zone. See Multiple zone limitations in the Kubernetes documentation for more information.

To enable multizone support using the Ansible inventory file, add the following parameter:
```
[OSEv3:vars]
openshift_gcp_multizone=true
```
To return to single-zone support, set the openshift_gcp_multizone value to false and rerun the Ansible inventory file.

Option 2: Manually configuring OpenShift Container Platform for GCE

Manually configuring master hosts for GCE

Perform the following procedure on all master hosts.

Procedure

Add the GCE parameters to the apiServerArguments and controllerArguments sections of the master configuration file at /etc/origin/master/master-config.yaml by default:

apiServerArguments:
  cloud-provider:
    - "gce"
  cloud-config:
    - "/etc/origin/cloudprovider/gce.conf"
controllerArguments:
  cloud-provider:
    - "gce"
  cloud-config:
    - "/etc/origin/cloudprovider/gce.conf"

When you configure OpenShift Container Platform for GCP using Ansible, the /etc/origin/cloudprovider/gce.conf file is created automatically. Because you are manually configuring OpenShift Container Platform for GCP, you must create the file and enter the following:

[Global]
project-id = <project-id> (1)
network-name = <network-name> (2)
node-tags = <node-tags> (3)
node-instance-prefix = <instance-prefix> (4)
multizone = true (5)

1	Provide the GCP project ID where the existing instances are running.
2	Provide the network name if not using the default.
3	Provide the tag for the GCP nodes. Must contain `ocp` as a suffix. For example, if the value of the `node-instance-prefix` parameter is set to `mycluster`, the nodes must be tagged with `myclusterocp`.
4	Provide a unique string to identify your OpenShift Container Platform cluster.
5	Set to `true` to trigger multizone deployments on GCP. Set to `False` by default.

The cluster installation process configures single-zone support by default.

Deploying OpenShift Container Platform in GCP on different zones can be helpful to avoid single-point-of-failures, but can cause problems with storage. This is because GCP disks are created within a zone. If an OpenShift Container Platform node host goes down in zone "A" and the pods should be moved to zone "B", the persistent storage cannot be attached to those pods, because the disks are now in a different zone. See Multiple zone limitations in the Kubernetes documentation for more information.

For running load balancer services using GCP, the Compute Engine VM node instances require the ocp suffix: <openshift_gcp_prefix>ocp. For example, if the value of the openshift_gcp_prefix parameter is set to mycluster, you must tag the nodes with myclusterocp. See Adding and Removing Network Tags for more information on how to add network tags to Compute Engine VM instances.

Restart the OpenShift Container Platform host services:

# master-restart api
# master-restart controllers
# systemctl restart atomic-openshift-node

To return to single-zone support, set the multizone value to false and restart the master and node host services.

Manually configuring node hosts for GCE

Perform the following on all node hosts.

Procedure

Edit the appropriate node configuration map and update the contents of the kubeletArguments section:
```
kubeletArguments:
  cloud-provider:
    - "gce"
  cloud-config:
    - "/etc/origin/cloudprovider/gce.conf"
```
The nodeName must match the instance name in GCP in order for the cloud provider integration to work properly. The name must also be RFC1123 compliant.
Restart the OpenShift Container Platform services on all nodes.
```
# systemctl restart atomic-openshift-node
```

Configuring the OpenShift Container Platform registry for GCP

Google Cloud Platform (GCP) provides object cloud storage that OpenShift Container Platform can use to store container images using the OpenShift Container Platform container registry.

For more information, see Cloud Storage in the GCP documentation.

Prerequisites

You must create the bucket to host the registry images before the installation. The following commands create a regional bucket using the configured service account:

gsutil mb -c regional -l <region> gs://ocp-registry-bucket
cat <<EOF > labels.json
{
  "ocp-cluster": "mycluster"
}
EOF
gsutil label set labels.json gs://ocp-registry-bucket
rm -f labels.json

A bucket’s data is automatically encrypted using a Google-managed key by default. To specify a different key to encrypt the data, see the Data Encryption Options available in GCP.

See the Creating storage buckets documentation for more information.

Procedure

To configure the Ansible inventory file for the registry to use a Google Cloud Storage (GCS) bucket:

[OSEv3:vars]
# GCP Provider Configuration
openshift_hosted_registry_storage_provider=gcs
openshift_hosted_registry_storage_kind=object
openshift_hosted_registry_replicas=1 (1)
openshift_hosted_registry_storage_gcs_bucket=<bucket_name> (2)
openshift_hosted_registry_storage_gcs_keyfile=<bucket_keyfile> (3)
openshift_hosted_registry_storage_gcs_rootdirectory=<registry_directory> (4)

1	The number of replicas to configure.
2	The bucket name to for registry storage.
3	The path on the installer host where the bucket’s keyfile is located if you use a custom key file to encrypt the data.
4	Directory used to store the data. `/registry` by default

For more information, see Cloud Storage in the GCP documentation.

Manually configuring OpenShift Container Platform registry for GCP

To use GCP object storage, edit the registry’s configuration file and mount to the registry pod.

See the Google Cloud Storage Driver documentation for more information about storage driver configuration files.

Procedure

Export the current /etc/registry/config.yml file:

$ oc get secret registry-config \
    -o jsonpath='{.data.config\.yml}' -n default | base64 -d \
  >> config.yml.old

Create a new configuration file from the old /etc/registry/config.yml file:
```
$ cp config.yml.old config.yml
```
Edit the file to include the GCP parameters. Specify the bucket and keyfile in the storage section of a registry’s configuration file:
```
storage:
  delete:
    enabled: true
  cache:
    blobdescriptor: inmemory
  gcs:
    bucket: ocp-registry (1)
    keyfile: mykeyfile (2)
```
1 Replace with the GCP bucket name.

2 A private service account key file in JSON format. If using the Google Application Default Credentials, do not specify the keyfile parameter.

Delete the registry-config secret:

$ oc delete secret registry-config -n default

Recreate the secret to reference the updated configuration file:

$ oc create secret generic registry-config \
    --from-file=config.yml -n default

Redeploy the registry to read the updated configuration:
```
$ oc rollout latest docker-registry -n default
```

Verify the registry is using GCP object storage

To verify if the registry is using GCP bucket storage:

Procedure

After a successful registry deployment using GCP storage, the registry deploymentconfig does not show any information if the registry is using an emptydir instead of GCP bucket storage:
```
$ oc describe dc docker-registry -n default
...
Mounts:
  ...
  /registry from registry-storage (rw)
Volumes:
registry-storage:
Type:       EmptyDir (1)
...
```
1 The temporary directory that shares a pod’s lifetime.

Check if the /registry mountpoint is empty. This is the volume GCP storage will use:

$ oc exec \
    $(oc get pod -l deploymentconfig=docker-registry \
    -o=jsonpath='{.items[0].metadata.name}')  -i -t -- ls -l /registry
total 0

If it is empty, it is because the GCP bucket configuration is performed in the registry-config secret:

$ oc describe secret registry-config
Name:         registry-config
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
config.yml:  398 bytes

The installer creates a config.yml file with the desired configuration using the extended registry capabilities as seen in Storage in the installation documentation. To view the configuration file, including the storage section where the storage bucket configuration is stored:

$ oc exec \
    $(oc get pod -l deploymentconfig=docker-registry \
      -o=jsonpath='{.items[0].metadata.name}') \
  cat /etc/registry/config.yml

version: 0.1
log:
  level: debug
http:
  addr: :5000
storage:
  delete:
    enabled: true
  cache:
    blobdescriptor: inmemory
  gcs:
    bucket: ocp-registry
auth:
  openshift:
    realm: openshift
middleware:
  registry:
  - name: openshift
  repository:
  - name: openshift
    options:
      pullthrough: True
      acceptschema2: True
      enforcequota: False
  storage:
  - name: openshift

Or you can view the secret:

$ oc get secret registry-config -o jsonpath='{.data.config\.yml}' | base64 -d
version: 0.1
log:
  level: debug
http:
  addr: :5000
storage:
  delete:
    enabled: true
  cache:
    blobdescriptor: inmemory
  gcs:
    bucket: ocp-registry
auth:
  openshift:
    realm: openshift
middleware:
  registry:
  - name: openshift
  repository:
  - name: openshift
    options:
      pullthrough: True
      acceptschema2: True
      enforcequota: False
  storage:
  - name: openshift

You can verify that any image push was successful by viewing Storage in the GCP console, then clicking Browser and selecting the bucket, or by running the gsutil command:

$ gsutil ls gs://ocp-registry/
gs://ocp-registry/docker/

$ gsutil du gs://ocp-registry/
7660385     gs://ocp-registry/docker/registry/v2/blobs/sha256/03/033565e6892e5cc6dd03187d00a4575720a928db111274e0fbf31b410a093c10/data
7660385     gs://ocp-registry/docker/registry/v2/blobs/sha256/03/033565e6892e5cc6dd03187d00a4575720a928db111274e0fbf31b410a093c10/
7660385     gs://ocp-registry/docker/registry/v2/blobs/sha256/03/
...

If using an emptyDir volume, the /registry mountpoint looks similar to the following:

$ oc exec \
    $(oc get pod -l deploymentconfig=docker-registry \
    -o=jsonpath='{.items[0].metadata.name}')  -i -t -- df -h /registry
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc         30G  226M   30G   1% /registry


$ oc exec \
    $(oc get pod -l deploymentconfig=docker-registry \
    -o=jsonpath='{.items[0].metadata.name}')  -i -t -- ls -l /registry
total 0
drwxr-sr-x. 3 1000000000 1000000000 22 Jun 19 12:24 docker

Configuring OpenShift Container Platform to use GCP storage

OpenShift Container Platform can use GCP storage using persistent volumes mechanisms. OpenShift Container Platform creates the disk in GCP and attaches the disk to the correct instance.

GCP disks are ReadWriteOnce access mode, which means the volume can be mounted as read-write by a single node. See the Access modes section of the Architecture guide for more information.

Procedure

OpenShift Container Platform creates the following storageclass when you use the gce-pd provisioner and if you use the openshift_cloudprovider_kind=gce and openshift_gcp_* variables in the Ansible inventory. Otherwise, if you configured OpenShift Container Platform without using Ansible and the storageclass has not been created at installation time, you can create it manually:
```
$ oc get --export storageclass standard -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 annotations:
   storageclass.kubernetes.io/is-default-class: "true"
 creationTimestamp: null
 name: standard
 selfLink: /apis/storage.k8s.io/v1/storageclasses/standard
parameters:
 type: pd-standard
provisioner: kubernetes.io/gce-pd
reclaimPolicy: Delete
```
After you request a PV and using the storageclass shown in the previous step, OpenShift Container Platform creates disks in the GCP infrastructure. To verify that the disks were created:
```
$ gcloud compute disks list | grep kubernetes
kubernetes-dynamic-pvc-10ded514-7625-11e8-8c52-42010af00003  us-west1-b  10       pd-standard  READY
```

About Red Hat OpenShift Container Storage

Red Hat OpenShift Container Storage (RHOCS) is a provider of agnostic persistent storage for OpenShift Container Platform either in-house or in hybrid clouds. As a Red Hat storage solution, RHOCS is completely integrated with OpenShift Container Platform for deployment, management, and monitoring regardless if it is installed on OpenShift Container Platform (converged) or with OpenShift Container Platform (independent). OpenShift Container Storage is not limited to a single availability zone or node, which makes it likely to survive an outage. You can find complete instructions for using RHOCS in the RHOCS 3.10 Deployment Guide.

Using the GCP external load balancer as a service

You can configure OpenShift Container Platform to use the GCP load balancer by exposing services externally using a LoadBalancer service. OpenShift Container Platform creates the load balancer in GCP and creates the necessary firewall rules.

Procedure

Create a new application:
```
$ oc new-app openshift/hello-openshift
```

Expose the load balancer service:

$ oc expose dc hello-openshift --name='hello-openshift-external' --type='LoadBalancer'

This command creates a LoadBalancer service similar to the following example:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: hello-openshift
  name: hello-openshift-external
spec:
  externalTrafficPolicy: Cluster
  ports:
  - name: port-1
    nodePort: 30714
    port: 8080
    protocol: TCP
    targetPort: 8080
  - name: port-2
    nodePort: 30122
    port: 8888
    protocol: TCP
    targetPort: 8888
  selector:
    app: hello-openshift
    deploymentconfig: hello-openshift
  sessionAffinity: None
  type: LoadBalancer

To verify that the service has been created:

$ oc get svc
NAME                       TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                         AGE
hello-openshift            ClusterIP      172.30.62.10     <none>          8080/TCP,8888/TCP               20m
hello-openshift-external   LoadBalancer   172.30.147.214   35.230.97.224   8080:31521/TCP,8888:30843/TCP   19m

The LoadBalancer type and External IP values indicate that the service is using GCP load balancers to expose the application.

OpenShift Container Platform creates the required objects in the GCP infrastructure such as:

Firewall rules:

$ gcloud compute firewall-rules list | grep k8s
k8s-4612931a3a47c204-node-http-hc        my-net  INGRESS    1000      tcp:10256
k8s-fw-a1a8afaa7762811e88c5242010af0000  my-net  INGRESS    1000      tcp:8080,tcp:8888

These firewall rules are applied to instances tagged with <openshift_gcp_prefix>ocp. For example, if the value of the openshift_gcp_prefix parameter is set to mycluster, you must tag the nodes with myclusterocp. See Adding and Removing Network Tags for more information on how to add network tags to Compute Engine VM instances.

Health checks:

$ gcloud compute http-health-checks list | grep k8s
k8s-4612931a3a47c204-node        10256  /healthz

A load balancer:

$ gcloud compute target-pools list | grep k8s
a1a8afaa7762811e88c5242010af0000  us-west1  NONE                      k8s-4612931a3a47c204-node
$ gcloud compute forwarding-rules list | grep a1a8afaa7762811e88c5242010af0000
a1a8afaa7762811e88c5242010af0000  us-west1  35.230.97.224  TCP          us-west1/targetPools/a1a8afaa7762811e88c5242010af0000

To verify that the load balancer is properly configured, run the following command from an external host:

$ curl 35.230.97.224:8080
Hello OpenShift!

1	Replace with the GCP bucket name.
2	A private service account key file in JSON format. If using the Google Application Default Credentials, do not specify the `keyfile` parameter.