us-east-1
In OKD 4, you can install a cluster on VMware vSphere infrastructure in a restricted network by creating an internal mirror of the installation release content.
Support for multiple vCenters is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope. |
You have completed the tasks in Preparing to install a cluster using installer-provisioned infrastructure.
You reviewed your VMware platform licenses. Red Hat does not place any restrictions on your VMware licenses, but some VMware infrastructure components require licensing.
You reviewed details about the OKD installation and update processes.
You read the documentation on selecting a cluster installation method and preparing it for users.
You created a registry on your mirror host and obtained the imageContentSources
data for your version of OKD.
Because the installation media is on the mirror host, you can use that computer to complete all installation steps. |
You provisioned persistent storage for your cluster. To deploy a private image registry, your storage must provide the ReadWriteMany access mode.
The OKD installer requires access to port 443 on the vCenter and ESXi hosts. You verified that port 443 is accessible.
If you use a firewall, you confirmed with the administrator that port 443 is accessible. Control plane nodes must be able to reach vCenter and ESXi hosts on port 443 for the installation to succeed.
If you use a firewall and plan to use the Telemetry service, you configured the firewall to allow the sites that your cluster requires access to.
If you are configuring a proxy, be sure to also review this site list. |
In OKD 4, you can perform an installation that does not require an active connection to the internet to obtain software components. Restricted network installations can be completed using installer-provisioned infrastructure or user-provisioned infrastructure, depending on the cloud platform to which you are installing the cluster.
If you choose to perform a restricted network installation on a cloud platform, you still require access to its cloud APIs. Some cloud functions, like Amazon Web Service’s Route 53 DNS and IAM services, require internet access. Depending on your network, you might require less internet access for an installation on bare metal hardware, Nutanix, or on VMware vSphere.
To complete a restricted network installation, you must create a registry that mirrors the contents of the OpenShift image registry and contains the installation media. You can create this registry on a mirror host, which can access both the internet and your closed network, or by using other methods that meet your restrictions.
Clusters in restricted networks have the following additional limitations and restrictions:
The ClusterVersion
status includes an Unable to retrieve available updates
error.
By default, you cannot use the contents of the Developer Catalog because you cannot access the required image stream tags.
Download the Fedora CoreOS (FCOS) image to install OKD on a restricted network VMware vSphere environment.
Obtain the OKD installation program. For a restricted network installation, the program is on your mirror registry host.
Log in to the Red Hat Customer Portal’s Product Downloads page.
Under Version, select the most recent release of OKD 4 for RHEL 8.
The FCOS images might not change with every release of OKD. You must download images with the highest version that is less than or equal to the OKD version that you install. Use the image versions that match your OKD version if they are available. |
Download the Fedora CoreOS (FCOS) - vSphere image.
Upload the image you downloaded to a location that is accessible from the bastion server.
The image is now available for a restricted installation. Note the image name or location for use in OKD deployment.
You can deploy an OKD cluster to multiple vSphere data centers. Each data center can run multiple clusters. This configuration reduces the risk of a hardware failure or network outage that can cause your cluster to fail. To enable regions and zones, you must define multiple failure domains for your OKD cluster.
The VMware vSphere region and zone enablement feature requires the vSphere Container Storage Interface (CSI) driver as the default storage driver in the cluster. As a result, the feature is only available on a newly installed cluster. For a cluster that was upgraded from a previous release, you must enable CSI automatic migration for the cluster. You can then configure multiple regions and zones for the upgraded cluster. |
The default installation configuration deploys a cluster to a single vSphere data center. If you want to deploy a cluster to multiple vSphere data centers, you must create an installation configuration file that enables the region and zone feature.
The default install-config.yaml
file includes vcenters
and failureDomains
fields, where you can specify multiple vSphere data centers and clusters for your OKD cluster. You can leave these fields blank if you want to install an OKD cluster in a vSphere environment that consists of single data center.
The following list describes terms associated with defining zones and regions for your cluster:
Failure domain: Establishes the relationships between a region and zone. You define a failure domain by using vCenter objects, such as a datastore
object. A failure domain defines the vCenter location for OKD cluster nodes.
Region: Specifies a vCenter data center. You define a region by using a tag from the openshift-region
tag category.
Zone: Specifies a vCenter cluster. You define a zone by using a tag from the openshift-zone
tag category.
If you plan on specifying more than one failure domain in your |
You must create a vCenter tag for each vCenter data center, which represents a region. Additionally, you must create a vCenter tag for each cluster than runs in a data center, which represents a zone. After you create the tags, you must attach each tag to their respective data centers and clusters.
The following table outlines an example of the relationship among regions, zones, and tags for a configuration with multiple vSphere data centers running in a single VMware vCenter.
Data center (region) | Cluster (zone) | Tags |
---|---|---|
us-east |
us-east-1 |
us-east-1a |
us-east-1b |
||
us-east-2 |
us-east-2a |
|
us-east-2b |
||
us-west |
us-west-1 |
us-west-1a |
us-west-1b |
||
us-west-2 |
us-west-2a |
|
us-west-2b |
You can customize the OKD cluster you install on VMware vSphere.
You have the OKD installation program and the pull secret for your cluster. For a restricted network installation, these files are on your mirror host.
You have the imageContentSources
values that were generated during mirror registry creation.
You have obtained the contents of the certificate for your mirror registry.
You have retrieved a Fedora CoreOS (FCOS) image and uploaded it to an accessible location.
Create the install-config.yaml
file.
Change to the directory that contains the installation program and run the following command:
$ ./openshift-install create install-config --dir <installation_directory> (1)
1 | For <installation_directory> , specify the directory name to store the
files that the installation program creates. |
When specifying the directory:
Verify that the directory has the execute
permission. This permission is required to run Terraform binaries under the installation directory.
Use an empty directory. Some installation assets, such as bootstrap X.509 certificates, have short expiration intervals, therefore you must not reuse an installation directory. If you want to reuse individual files from another cluster installation, you can copy them into your directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OKD version.
At the prompts, provide the configuration details for your cloud:
Optional: Select an SSH key to use to access your cluster machines.
For production OKD clusters on which you want to perform installation debugging or disaster recovery, specify an SSH key that your |
Select vsphere as the platform to target.
Specify the name of your vCenter instance.
Specify the user name and password for the vCenter account that has the required permissions to create the cluster.
The installation program connects to your vCenter instance.
Select the data center in your vCenter instance to connect to.
After you create the installation configuration file, you can modify the file to create a multiple vSphere data center environment. This means that you can deploy an OKD cluster to multiple vSphere data centers. For more information about creating this environment, see the section named VMware vSphere region and zone enablement. |
Select the default vCenter datastore to use.
You can specify the path of any datastore that exists in a datastore cluster. By default, Storage Distributed Resource Scheduler (SDRS), which uses Storage vMotion, is automatically enabled for a datastore cluster. Red Hat does not support Storage vMotion, so you must disable Storage DRS to avoid data loss issues for your OKD cluster. You cannot specify more than one datastore path. If you must specify VMs across multiple datastores, use a |
Select the vCenter cluster to install the OKD cluster in. The installation program uses the root resource pool of the vSphere cluster as the default resource pool.
Select the network in the vCenter instance that contains the virtual IP addresses and DNS records that you configured.
Enter the virtual IP address that you configured for control plane API access.
Enter the virtual IP address that you configured for cluster ingress.
Enter the base domain. This base domain must be the same one that you used in the DNS records that you configured.
Enter a descriptive name for your cluster.
The cluster name you enter must match the cluster name you specified when configuring the DNS records.
In the install-config.yaml
file, set the value of platform.vsphere.clusterOSImage
to the image location or name. For example:
platform:
vsphere:
clusterOSImage: http://mirror.example.com/images/rhcos-43.81.201912131630.0-vmware.x86_64.ova?sha256=ffebbd68e8a1f2a245ca19522c16c86f67f9ac8e4e0c1f0a812b068b16f7265d
Edit the install-config.yaml
file to give the additional information that is required for an installation in a restricted network.
Update the pullSecret
value to contain the authentication information for
your registry:
pullSecret: '{"auths":{"<mirror_host_name>:5000": {"auth": "<credentials>","email": "you@example.com"}}}'
For <mirror_host_name>
, specify the registry domain name
that you specified in the certificate for your mirror registry, and for
<credentials>
, specify the base64-encoded user name and password for
your mirror registry.
Add the additionalTrustBundle
parameter and value.
additionalTrustBundle: |
-----BEGIN certificate-----
ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ
-----END certificate-----
The value must be the contents of the certificate file that you used for your mirror registry. The certificate file can be an existing, trusted certificate authority, or the self-signed certificate that you generated for the mirror registry.
Add the image content resources, which resemble the following YAML excerpt:
imageContentSources:
- mirrors:
- <mirror_host_name>:5000/<repo_name>/release
source: quay.io/openshift-release-dev/ocp-release
- mirrors:
- <mirror_host_name>:5000/<repo_name>/release
source: registry.redhat.io/ocp/release
For these values, use the imageContentSources
that you recorded during mirror registry creation.
Optional: Set the publishing strategy to Internal
:
publish: Internal
By setting this option, you create an internal Ingress Controller and a private load balancer.
Make any other modifications to the install-config.yaml
file that you require.
For more information about the parameters, see "Installation configuration parameters".
Back up the install-config.yaml
file so that you can use
it to install multiple clusters.
The |
You can customize the install-config.yaml
file to specify more details about
your OKD cluster’s platform or modify the values of the required
parameters.
apiVersion: v1
baseDomain: example.com (1)
compute: (2)
- architecture: amd64
name: <worker_node>
platform: {}
replicas: 3
controlPlane: (2)
architecture: amd64
name: <parent_node>
platform: {}
replicas: 3
metadata:
creationTimestamp: null
name: test (3)
platform:
vsphere: (4)
apiVIPs:
- 10.0.0.1
failureDomains: (5)
- name: <failure_domain_name>
region: <default_region_name>
server: <fully_qualified_domain_name>
topology:
computeCluster: "/<data_center>/host/<cluster>"
datacenter: <data_center>
datastore: "/<data_center>/datastore/<datastore>" (6)
networks:
- <VM_Network_name>
resourcePool: "/<data_center>/host/<cluster>/Resources/<resourcePool>" (7)
folder: "/<data_center_name>/vm/<folder_name>/<subfolder_name>"
tagIDs: (8)
- <tag_id> (9)
zone: <default_zone_name>
ingressVIPs:
- 10.0.0.2
vcenters:
- datacenters:
- <data_center>
password: <password>
port: 443
server: <fully_qualified_domain_name>
user: administrator@vsphere.local
diskType: thin (10)
clusterOSImage: http://mirror.example.com/images/rhcos-47.83.202103221318-0-vmware.x86_64.ova (11)
pullSecret: '{"auths":{"<local_registry>": {"auth": "<credentials>","email": "you@example.com"}}}' (12)
sshKey: 'ssh-ed25519 AAAA...'
additionalTrustBundle: | (13)
-----BEGIN certificate-----
ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ
-----END certificate-----
imageContentSources: (14)
- mirrors:
- <mirror_host_name>:<mirror_port>/<repo_name>/release
source: <source_image_1>
- mirrors:
- <mirror_host_name>:<mirror_port>/<repo_name>/release-images
source: <source_image_2>
1 | The base domain of the cluster. All DNS records must be sub-domains of this base and include the cluster name. | ||
2 | The controlPlane section is a single mapping, but the compute section is a sequence of mappings. To meet the requirements of the different data structures, the first line of the compute section must begin with a hyphen, - , and the first line of the controlPlane section must not. Only one control plane pool is used. |
||
3 | The cluster name that you specified in your DNS records. | ||
4 | Optional: Provides additional configuration for the machine pool parameters for the compute and control plane machines. | ||
5 | Establishes the relationships between a region and zone. You define a failure domain by using vCenter objects, such as a datastore object. A failure domain defines the vCenter location for OKD cluster nodes. |
||
6 | The path to the vSphere datastore that holds virtual machine files, templates, and ISO images.
|
||
7 | Optional: Provides an existing resource pool for machine creation. If you do not specify a value, the installation program uses the root resource pool of the vSphere cluster. | ||
8 | Optional: Each VM created by OKD is assigned a unique tag that is specific to the cluster. The assigned tag enables the installation program to identify and remove the associated VMs when a cluster is decommissioned. You can list up to ten additional tag IDs to be attached to the VMs provisioned by the installation program. | ||
9 | The ID of the tag to be associated by the installation program. For example, urn:vmomi:InventoryServiceTag:208e713c-cae3-4b7f-918e-4051ca7d1f97:GLOBAL . For more information about determining the tag ID, see the vSphere Tags and Attributes documentation. |
||
10 | The vSphere disk provisioning method. | ||
11 | The location of the Fedora CoreOS (FCOS) image that is accessible from the bastion server. | ||
12 | For <local_registry> , specify the registry domain name, and optionally the
port, that your mirror registry uses to serve content. For example
registry.example.com or registry.example.com:5000 . For <credentials> ,
specify the base64-encoded user name and password for your mirror registry. |
||
13 | Provide the contents of the certificate file that you used for your mirror registry. | ||
14 | Provide the imageContentSources section from the output of the command to mirror the repository. |
In OKD 4.12 and later, the |
Production environments can deny direct access to the internet and instead have
an HTTP or HTTPS proxy available. You can configure a new OKD
cluster to use a proxy by configuring the proxy settings in the
install-config.yaml
file.
You have an existing install-config.yaml
file.
You reviewed the sites that your cluster requires access to and determined whether any of them need to bypass the proxy. By default, all cluster egress traffic is proxied, including calls to hosting cloud provider APIs. You added sites to the Proxy
object’s spec.noProxy
field to bypass the proxy if necessary.
The For installations on Amazon Web Services (AWS), Google Cloud Platform (GCP), Microsoft Azure, and OpenStack, the |
Edit your install-config.yaml
file and add the proxy settings. For example:
apiVersion: v1
baseDomain: my.domain.com
proxy:
httpProxy: http://<username>:<pswd>@<ip>:<port> (1)
httpsProxy: https://<username>:<pswd>@<ip>:<port> (2)
noProxy: example.com (3)
additionalTrustBundle: | (4)
-----BEGIN certificate-----
<MY_TRUSTED_CA_CERT>
-----END certificate-----
additionalTrustBundlePolicy: <policy_to_add_additionalTrustBundle> (5)
1 | A proxy URL to use for creating HTTP connections outside the cluster. The
URL scheme must be http . |
2 | A proxy URL to use for creating HTTPS connections outside the cluster. |
3 | A comma-separated list of destination domain names, IP addresses, or other network CIDRs to exclude from proxying. Preface a domain with . to match subdomains only. For example, .y.com matches x.y.com , but not y.com . Use * to bypass the proxy for all destinations.
You must include vCenter’s IP address and the IP range that you use for its machines. |
4 | If provided, the installation program generates a config map that is named user-ca-bundle in
the openshift-config namespace to hold the additional CA
certificates. If you provide additionalTrustBundle and at least one proxy setting, the Proxy object is configured to reference the user-ca-bundle config map in the trustedCA field. The Cluster Network
Operator then creates a trusted-ca-bundle config map that merges the contents specified for the trustedCA parameter
with the FCOS trust bundle. The additionalTrustBundle field is required unless
the proxy’s identity certificate is signed by an authority from the FCOS trust
bundle. |
5 | Optional: The policy to determine the configuration of the Proxy object to reference the user-ca-bundle config map in the trustedCA field. The allowed values are Proxyonly and Always . Use Proxyonly to reference the user-ca-bundle config map only when http/https proxy is configured. Use Always to always reference the user-ca-bundle config map. The default value is Proxyonly . |
The installation program does not support the proxy |
If the installer times out, restart and then complete the deployment by using the
|
Save the file and reference it when installing OKD.
The installation program creates a cluster-wide proxy that is named cluster
that uses the proxy
settings in the provided install-config.yaml
file. If no proxy settings are
provided, a cluster
Proxy
object is still created, but it will have a nil
spec
.
Only the |
You can modify the default installation configuration file, so that you can deploy an OKD cluster to multiple vSphere data centers.
The default install-config.yaml
file configuration from the previous release of OKD is deprecated. You can continue to use the deprecated default configuration, but the openshift-installer
will prompt you with a warning message that indicates the use of deprecated fields in the configuration file.
The example uses the |
You have an existing install-config.yaml
installation configuration file.
You must specify at least one failure domain for your OKD cluster, so that you can provision data center objects for your VMware vCenter server. Consider specifying multiple failure domains if you need to provision virtual machine nodes in different data centers, clusters, datastores, and other components. To enable regions and zones, you must define multiple failure domains for your OKD cluster. |
Enter the following govc
command-line tool commands to create the openshift-region
and openshift-zone
vCenter tag categories:
If you specify different names for the |
$ govc tags.category.create -d "OpenShift region" openshift-region
$ govc tags.category.create -d "OpenShift zone" openshift-zone
To create a region tag for each region vSphere data center where you want to deploy your cluster, enter the following command in your terminal:
$ govc tags.create -c <region_tag_category> <region_tag>
To create a zone tag for each vSphere cluster where you want to deploy your cluster, enter the following command:
$ govc tags.create -c <zone_tag_category> <zone_tag>
Attach region tags to each vCenter data center object by entering the following command:
$ govc tags.attach -c <region_tag_category> <region_tag_1> /<data_center_1>
Attach the zone tags to each vCenter data center object by entering the following command:
$ govc tags.attach -c <zone_tag_category> <zone_tag_1> /<data_center_1>/host/vcs-mdcnc-workload-1
Change to the directory that contains the installation program and initialize the cluster deployment according to your chosen installation requirements.
install-config.yaml
file with multiple data centers defined in a vSphere center---
compute:
---
vsphere:
zones:
- "<machine_pool_zone_1>"
- "<machine_pool_zone_2>"
---
controlPlane:
---
vsphere:
zones:
- "<machine_pool_zone_1>"
- "<machine_pool_zone_2>"
---
platform:
vsphere:
vcenters:
---
datacenters:
- <data_center_1_name>
- <data_center_2_name>
failureDomains:
- name: <machine_pool_zone_1>
region: <region_tag_1>
zone: <zone_tag_1>
server: <fully_qualified_domain_name>
topology:
datacenter: <data_center_1>
computeCluster: "/<data_center_1>/host/<cluster1>"
networks:
- <VM_Network1_name>
datastore: "/<data_center_1>/datastore/<datastore1>"
resourcePool: "/<data_center_1>/host/<cluster1>/Resources/<resourcePool1>"
folder: "/<data_center_1>/vm/<folder1>"
- name: <machine_pool_zone_2>
region: <region_tag_2>
zone: <zone_tag_2>
server: <fully_qualified_domain_name>
topology:
datacenter: <data_center_2>
computeCluster: "/<data_center_2>/host/<cluster2>"
networks:
- <VM_Network2_name>
datastore: "/<data_center_2>/datastore/<datastore2>"
resourcePool: "/<data_center_2>/host/<cluster2>/Resources/<resourcePool2>"
folder: "/<data_center_2>/vm/<folder2>"
---
You can configure an OKD cluster to use a user-managed load balancer in place of the default load balancer.
Configuring a user-managed load balancer depends on your vendor’s load balancer. The information and examples in this section are for guideline purposes only. Consult the vendor documentation for more specific information about the vendor’s load balancer. |
Red Hat supports the following services for a user-managed load balancer:
Ingress Controller
OpenShift API
OpenShift MachineConfig API
You can choose whether you want to configure one or all of these services for a user-managed load balancer. Configuring only the Ingress Controller service is a common configuration option. To better understand each service, view the following diagrams:
The following configuration options are supported for user-managed load balancers:
Use a node selector to map the Ingress Controller to a specific set of nodes. You must assign a static IP address to each node in this set, or configure each node to receive the same IP address from the Dynamic Host Configuration Protocol (DHCP). Infrastructure nodes commonly receive this type of configuration.
Target all IP addresses on a subnet. This configuration can reduce maintenance overhead, because you can create and destroy nodes within those networks without reconfiguring the load balancer targets. If you deploy your ingress pods by using a machine set on a smaller network, such as a /27
or /28
, you can simplify your load balancer targets.
You can list all IP addresses that exist in a network by checking the machine config pool’s resources. |
Before you configure a user-managed load balancer for your OKD cluster, consider the following information:
For a front-end IP address, you can use the same IP address for the front-end IP address, the Ingress Controller’s load balancer, and API load balancer. Check the vendor’s documentation for this capability.
For a back-end IP address, ensure that an IP address for an OKD control plane node does not change during the lifetime of the user-managed load balancer. You can achieve this by completing one of the following actions:
Assign a static IP address to each control plane node.
Configure each node to receive the same IP address from the DHCP every time the node requests a DHCP lease. Depending on the vendor, the DHCP lease might be in the form of an IP reservation or a static DHCP assignment.
Manually define each node that runs the Ingress Controller in the user-managed load balancer for the Ingress Controller back-end service. For example, if the Ingress Controller moves to an undefined node, a connection outage can occur.
You can configure an OKD cluster to use a user-managed load balancer in place of the default load balancer.
Before you configure a user-managed load balancer, ensure that you read the "Services for a user-managed load balancer" section. |
Read the following prerequisites that apply to the service that you want to configure for your user-managed load balancer.
MetalLB, which runs on a cluster, functions as a user-managed load balancer. |
You defined a front-end IP address.
TCP ports 6443 and 22623 are exposed on the front-end IP address of your load balancer. Check the following items:
Port 6443 provides access to the OpenShift API service.
Port 22623 can provide ignition startup configurations to nodes.
The front-end IP address and port 6443 are reachable by all users of your system with a location external to your OKD cluster.
The front-end IP address and port 22623 are reachable only by OKD nodes.
The load balancer backend can communicate with OKD control plane nodes on port 6443 and 22623.
You defined a front-end IP address.
TCP ports 443 and 80 are exposed on the front-end IP address of your load balancer.
The front-end IP address, port 80 and port 443 are be reachable by all users of your system with a location external to your OKD cluster.
The front-end IP address, port 80 and port 443 are reachable to all nodes that operate in your OKD cluster.
The load balancer backend can communicate with OKD nodes that run the Ingress Controller on ports 80, 443, and 1936.
You can configure most load balancers by setting health check URLs that determine if a service is available or unavailable. OKD provides these health checks for the OpenShift API, Machine Configuration API, and Ingress Controller backend services.
The following examples show health check specifications for the previously listed backend services:
Path: HTTPS:6443/readyz
Healthy threshold: 2
Unhealthy threshold: 2
Timeout: 10
Interval: 10
Path: HTTPS:22623/healthz
Healthy threshold: 2
Unhealthy threshold: 2
Timeout: 10
Interval: 10
Path: HTTP:1936/healthz/ready
Healthy threshold: 2
Unhealthy threshold: 2
Timeout: 5
Interval: 10
Configure the HAProxy Ingress Controller, so that you can enable access to the cluster from your load balancer on ports 6443, 22623, 443, and 80. Depending on your needs, you can specify the IP address of a single subnet or IP addresses from multiple subnets in your HAProxy configuration.
# ...
listen my-cluster-api-6443
bind 192.168.1.100:6443
mode tcp
balance roundrobin
option httpchk
http-check connect
http-check send meth GET uri /readyz
http-check expect status 200
server my-cluster-master-2 192.168.1.101:6443 check inter 10s rise 2 fall 2
server my-cluster-master-0 192.168.1.102:6443 check inter 10s rise 2 fall 2
server my-cluster-master-1 192.168.1.103:6443 check inter 10s rise 2 fall 2
listen my-cluster-machine-config-api-22623
bind 192.168.1.100:22623
mode tcp
balance roundrobin
option httpchk
http-check connect
http-check send meth GET uri /healthz
http-check expect status 200
server my-cluster-master-2 192.168.1.101:22623 check inter 10s rise 2 fall 2
server my-cluster-master-0 192.168.1.102:22623 check inter 10s rise 2 fall 2
server my-cluster-master-1 192.168.1.103:22623 check inter 10s rise 2 fall 2
listen my-cluster-apps-443
bind 192.168.1.100:443
mode tcp
balance roundrobin
option httpchk
http-check connect
http-check send meth GET uri /healthz/ready
http-check expect status 200
server my-cluster-worker-0 192.168.1.111:443 check port 1936 inter 10s rise 2 fall 2
server my-cluster-worker-1 192.168.1.112:443 check port 1936 inter 10s rise 2 fall 2
server my-cluster-worker-2 192.168.1.113:443 check port 1936 inter 10s rise 2 fall 2
listen my-cluster-apps-80
bind 192.168.1.100:80
mode tcp
balance roundrobin
option httpchk
http-check connect
http-check send meth GET uri /healthz/ready
http-check expect status 200
server my-cluster-worker-0 192.168.1.111:80 check port 1936 inter 10s rise 2 fall 2
server my-cluster-worker-1 192.168.1.112:80 check port 1936 inter 10s rise 2 fall 2
server my-cluster-worker-2 192.168.1.113:80 check port 1936 inter 10s rise 2 fall 2
# ...
# ...
listen api-server-6443
bind *:6443
mode tcp
server master-00 192.168.83.89:6443 check inter 1s
server master-01 192.168.84.90:6443 check inter 1s
server master-02 192.168.85.99:6443 check inter 1s
server bootstrap 192.168.80.89:6443 check inter 1s
listen machine-config-server-22623
bind *:22623
mode tcp
server master-00 192.168.83.89:22623 check inter 1s
server master-01 192.168.84.90:22623 check inter 1s
server master-02 192.168.85.99:22623 check inter 1s
server bootstrap 192.168.80.89:22623 check inter 1s
listen ingress-router-80
bind *:80
mode tcp
balance source
server worker-00 192.168.83.100:80 check inter 1s
server worker-01 192.168.83.101:80 check inter 1s
listen ingress-router-443
bind *:443
mode tcp
balance source
server worker-00 192.168.83.100:443 check inter 1s
server worker-01 192.168.83.101:443 check inter 1s
listen ironic-api-6385
bind *:6385
mode tcp
balance source
server master-00 192.168.83.89:6385 check inter 1s
server master-01 192.168.84.90:6385 check inter 1s
server master-02 192.168.85.99:6385 check inter 1s
server bootstrap 192.168.80.89:6385 check inter 1s
listen inspector-api-5050
bind *:5050
mode tcp
balance source
server master-00 192.168.83.89:5050 check inter 1s
server master-01 192.168.84.90:5050 check inter 1s
server master-02 192.168.85.99:5050 check inter 1s
server bootstrap 192.168.80.89:5050 check inter 1s
# ...
Use the curl
CLI command to verify that the user-managed load balancer and its resources are operational:
Verify that the cluster machine configuration API is accessible to the Kubernetes API server resource, by running the following command and observing the response:
$ curl https://<loadbalancer_ip_address>:6443/version --insecure
If the configuration is correct, you receive a JSON object in response:
{
"major": "1",
"minor": "11+",
"gitVersion": "v1.11.0+ad103ed",
"gitCommit": "ad103ed",
"gitTreeState": "clean",
"buildDate": "2019-01-09T06:44:10Z",
"goVersion": "go1.10.3",
"compiler": "gc",
"platform": "linux/amd64"
}
Verify that the cluster machine configuration API is accessible to the Machine config server resource, by running the following command and observing the output:
$ curl -v https://<loadbalancer_ip_address>:22623/healthz --insecure
If the configuration is correct, the output from the command shows the following response:
HTTP/1.1 200 OK
Content-Length: 0
Verify that the controller is accessible to the Ingress Controller resource on port 80, by running the following command and observing the output:
$ curl -I -L -H "Host: console-openshift-console.apps.<cluster_name>.<base_domain>" http://<load_balancer_front_end_IP_address>
If the configuration is correct, the output from the command shows the following response:
HTTP/1.1 302 Found
content-length: 0
location: https://console-openshift-console.apps.ocp4.private.opequon.net/
cache-control: no-cache
Verify that the controller is accessible to the Ingress Controller resource on port 443, by running the following command and observing the output:
$ curl -I -L --insecure --resolve console-openshift-console.apps.<cluster_name>.<base_domain>:443:<Load Balancer Front End IP Address> https://console-openshift-console.apps.<cluster_name>.<base_domain>
If the configuration is correct, the output from the command shows the following response:
HTTP/1.1 200 OK
referrer-policy: strict-origin-when-cross-origin
set-cookie: csrf-token=UlYWOyQ62LWjw2h003xtYSKlh1a0Py2hhctw0WmV2YEdhJjFyQwWcGBsja261dGLgaYO0nxzVErhiXt6QepA7g==; Path=/; Secure; SameSite=Lax
x-content-type-options: nosniff
x-dns-prefetch-control: off
x-frame-options: DENY
x-xss-protection: 1; mode=block
date: Wed, 04 Oct 2023 16:29:38 GMT
content-type: text/html; charset=utf-8
set-cookie: 1e2670d92730b515ce3a1bb65da45062=1bf5e9573c9a2760c964ed1659cc1673; path=/; HttpOnly; Secure; SameSite=None
cache-control: private
Configure the DNS records for your cluster to target the front-end IP addresses of the user-managed load balancer. You must update records to your DNS server for the cluster API and applications over the load balancer.
<load_balancer_ip_address> A api.<cluster_name>.<base_domain>
A record pointing to Load Balancer Front End
<load_balancer_ip_address> A apps.<cluster_name>.<base_domain>
A record pointing to Load Balancer Front End
DNS propagation might take some time for each DNS record to become available. Ensure that each DNS record propagates before validating each record. |
For your OKD cluster to use the user-managed load balancer, you must specify the following configuration in your cluster’s install-config.yaml
file:
# ...
platform:
vsphere:
loadBalancer:
type: UserManaged (1)
apiVIPs:
- <api_ip> (2)
ingressVIPs:
- <ingress_ip> (3)
# ...
1 | Set UserManaged for the type parameter to specify a user-managed load balancer for your cluster. The parameter defaults to OpenShiftManagedDefault , which denotes the default internal load balancer. For services defined in an openshift-kni-infra namespace, a user-managed load balancer can deploy the coredns service to pods in your cluster but ignores keepalived and haproxy services. |
2 | Required parameter when you specify a user-managed load balancer. Specify the user-managed load balancer’s public IP address, so that the Kubernetes API can communicate with the user-managed load balancer. |
3 | Required parameter when you specify a user-managed load balancer. Specify the user-managed load balancer’s public IP address, so that the user-managed load balancer can manage ingress traffic for your cluster. |
Use the curl
CLI command to verify that the user-managed load balancer and DNS record configuration are operational:
Verify that you can access the cluster API, by running the following command and observing the output:
$ curl https://api.<cluster_name>.<base_domain>:6443/version --insecure
If the configuration is correct, you receive a JSON object in response:
{
"major": "1",
"minor": "11+",
"gitVersion": "v1.11.0+ad103ed",
"gitCommit": "ad103ed",
"gitTreeState": "clean",
"buildDate": "2019-01-09T06:44:10Z",
"goVersion": "go1.10.3",
"compiler": "gc",
"platform": "linux/amd64"
}
Verify that you can access the cluster machine configuration, by running the following command and observing the output:
$ curl -v https://api.<cluster_name>.<base_domain>:22623/healthz --insecure
If the configuration is correct, the output from the command shows the following response:
HTTP/1.1 200 OK
Content-Length: 0
Verify that you can access each cluster application on port, by running the following command and observing the output:
$ curl http://console-openshift-console.apps.<cluster_name>.<base_domain> -I -L --insecure
If the configuration is correct, the output from the command shows the following response:
HTTP/1.1 302 Found
content-length: 0
location: https://console-openshift-console.apps.<cluster-name>.<base domain>/
cache-control: no-cacheHTTP/1.1 200 OK
referrer-policy: strict-origin-when-cross-origin
set-cookie: csrf-token=39HoZgztDnzjJkq/JuLJMeoKNXlfiVv2YgZc09c3TBOBU4NI6kDXaJH1LdicNhN1UsQWzon4Dor9GWGfopaTEQ==; Path=/; Secure
x-content-type-options: nosniff
x-dns-prefetch-control: off
x-frame-options: DENY
x-xss-protection: 1; mode=block
date: Tue, 17 Nov 2020 08:42:10 GMT
content-type: text/html; charset=utf-8
set-cookie: 1e2670d92730b515ce3a1bb65da45062=9b714eb87e93cf34853e87a92d6894be; path=/; HttpOnly; Secure; SameSite=None
cache-control: private
Verify that you can access each cluster application on port 443, by running the following command and observing the output:
$ curl https://console-openshift-console.apps.<cluster_name>.<base_domain> -I -L --insecure
If the configuration is correct, the output from the command shows the following response:
HTTP/1.1 200 OK
referrer-policy: strict-origin-when-cross-origin
set-cookie: csrf-token=UlYWOyQ62LWjw2h003xtYSKlh1a0Py2hhctw0WmV2YEdhJjFyQwWcGBsja261dGLgaYO0nxzVErhiXt6QepA7g==; Path=/; Secure; SameSite=Lax
x-content-type-options: nosniff
x-dns-prefetch-control: off
x-frame-options: DENY
x-xss-protection: 1; mode=block
date: Wed, 04 Oct 2023 16:29:38 GMT
content-type: text/html; charset=utf-8
set-cookie: 1e2670d92730b515ce3a1bb65da45062=1bf5e9573c9a2760c964ed1659cc1673; path=/; HttpOnly; Secure; SameSite=None
cache-control: private
You can install OKD on a compatible cloud platform.
You can run the |
You have the OKD installation program and the pull secret for your cluster.
You have verified that the cloud provider account on your host has the correct permissions to deploy the cluster. An account with incorrect permissions causes the installation process to fail with an error message that displays the missing permissions.
Optional: Before you create the cluster, configure an external load balancer in place of the default load balancer.
You do not need to specify API and Ingress static addresses for your installation program. If you choose this configuration, you must take additional actions to define network targets that accept an IP address from each referenced vSphere subnet. See the section "Configuring a user-managed load balancer". |
Change to the directory that contains the installation program and initialize the cluster deployment:
$ ./openshift-install create cluster --dir <installation_directory> \ (1)
--log-level=info (2)
1 | For <installation_directory> , specify the
location of your customized ./install-config.yaml file. |
2 | To view different installation details, specify warn , debug , or
error instead of info . |
When the cluster deployment completes successfully:
The terminal displays directions for accessing your cluster, including a link to the web console and credentials for the kubeadmin
user.
Credential information also outputs to <installation_directory>/.openshift_install.log
.
Do not delete the installation program or the files that the installation program creates. Both are required to delete the cluster. |
...
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/home/myuser/install_dir/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.mycluster.example.com
INFO Login to the console with user: "kubeadmin", and password: "password"
INFO Time elapsed: 36m22s
|
You can log in to your cluster as a default system user by exporting the cluster kubeconfig
file.
The kubeconfig
file contains information about the cluster that is used by the CLI to connect a client to the correct cluster and API server.
The file is specific to a cluster and is created during OKD installation.
You deployed an OKD cluster.
You installed the oc
CLI.
Export the kubeadmin
credentials:
$ export KUBECONFIG=<installation_directory>/auth/kubeconfig (1)
1 | For <installation_directory> , specify the path to the directory that you stored
the installation files in. |
Verify you can run oc
commands successfully using the exported configuration:
$ oc whoami
system:admin
Operator catalogs that source content provided by Red Hat and community projects are configured for OperatorHub by default during an OKD installation. In a restricted network environment, you must disable the default catalogs as a cluster administrator.
Disable the sources for the default catalogs by adding disableAllDefaultSources: true
to the OperatorHub
object:
$ oc patch OperatorHub cluster --type json \
-p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'
Alternatively, you can use the web console to manage catalog sources. From the Administration → Cluster Settings → Configuration → OperatorHub page, click the Sources tab, where you can create, update, delete, disable, and enable individual sources. |
After you install the cluster, you must create storage for the Registry Operator.
On platforms that do not provide shareable object storage, the OpenShift Image Registry Operator bootstraps itself as Removed
. This allows openshift-installer
to complete installations on these platform types.
After installation, you must edit the Image Registry Operator configuration to switch the managementState
from Removed
to Managed
. When this has completed, you must configure storage.
The Image Registry Operator is not initially available for platforms that do not provide default storage. After installation, you must configure your registry to use storage so that the Registry Operator is made available.
Instructions are shown for configuring a persistent volume, which is required for production clusters. Where applicable, instructions are shown for configuring an empty directory as the storage location, which is available for only non-production clusters.
Additional instructions are provided for allowing the image registry to use block storage types by using the Recreate
rollout strategy during upgrades.
As a cluster administrator, following installation you must configure your registry to use storage.
Cluster administrator permissions.
A cluster on VMware vSphere.
Persistent storage provisioned for your cluster, such as Red Hat OpenShift Data Foundation.
OKD supports |
Must have "100Gi" capacity.
Testing shows issues with using the NFS server on RHEL as storage backend for core services. This includes the OpenShift Container Registry and Quay, Prometheus for monitoring storage, and Elasticsearch for logging storage. Therefore, using RHEL NFS to back PVs used by core services is not recommended. Other NFS implementations on the marketplace might not have these issues. Contact the individual NFS implementation vendor for more information on any testing that was possibly completed against these OKD core components. |
To configure your registry to use storage, change the spec.storage.pvc
in the configs.imageregistry/cluster
resource.
When you use shared storage, review your security settings to prevent outside access. |
Verify that you do not have a registry pod:
$ oc get pod -n openshift-image-registry -l docker-registry=default
No resourses found in openshift-image-registry namespace
If you do have a registry pod in your output, you do not need to continue with this procedure. |
Check the registry configuration:
$ oc edit configs.imageregistry.operator.openshift.io
storage:
pvc:
claim: (1)
1 | Leave the claim field blank to allow the automatic creation of an image-registry-storage persistent volume claim (PVC). The PVC is generated based on the default storage class. However, be aware that the default storage class might provide ReadWriteOnce (RWO) volumes, such as a RADOS Block Device (RBD), which can cause issues when you replicate to more than one replica. |
Check the clusteroperator
status:
$ oc get clusteroperator image-registry
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
image-registry 4.7 True False False 6h50m
See About remote health monitoring for more information about the Telemetry service
If necessary, you can opt out of remote health reporting.
If necessary, see Registering your disconnected cluster.