Manually installing a single-node OpenShift cluster with ZTP - Clusters at the network far edge | Scalability and performance

Generating ZTP installation and configuration CRs manually
Creating the managed bare-metal host secrets
Installing a single managed cluster
Monitoring the managed cluster installation status
Troubleshooting the managed cluster
RHACM generated cluster installation CRs reference

You can deploy a managed single-node OpenShift cluster by using Red Hat Advanced Cluster Management (RHACM) and the assisted service.

If you are creating multiple managed clusters, use the SiteConfig method described in Deploying far edge sites with ZTP.

The target bare-metal host must meet the networking, firmware, and hardware requirements listed in Recommended cluster configuration for vDU application workloads.

Generating ZTP installation and configuration CRs manually

Use the generator entrypoint for the ztp-site-generate container to generate the site installation and configuration custom resource (CRs) for a cluster based on SiteConfig and PolicyGenTemplate CRs.

Prerequisites

You have installed the OpenShift CLI (oc).
You have logged in to the hub cluster as a user with cluster-admin privileges.

Procedure

Create an output folder by running the following command:
```
$ mkdir -p ./out
```

Export the argocd directory from the ztp-site-generate container image:

$ podman run --log-driver=none --rm registry.redhat.io/openshift4/ztp-site-generate-rhel8:v4.11 extract /home/ztp --tar | tar x -C ./out

The ./out directory has the reference PolicyGenTemplate and SiteConfig CRs in the out/argocd/example/ folder.

Example output

out
 └── argocd
      └── example
           ├── policygentemplates
           │     ├── common-ranGen.yaml
           │     ├── example-sno-site.yaml
           │     ├── group-du-sno-ranGen.yaml
           │     ├── group-du-sno-validator-ranGen.yaml
           │     ├── kustomization.yaml
           │     └── ns.yaml
           └── siteconfig
                  ├── example-sno.yaml
                  ├── KlusterletAddonConfigOverride.yaml
                  └── kustomization.yaml

Create an output folder for the site installation CRs:
```
$ mkdir -p ./site-install
```

Modify the example SiteConfig CR for the cluster type that you want to install. Copy example-sno.yaml to site-1-sno.yaml and modify the CR to match the details of the site and bare-metal host that you want to install, for example:

Example single-node OpenShift cluster SiteConfig CR

apiVersion: ran.openshift.io/v1
kind: SiteConfig
metadata:
  name: "<site_name>"
  namespace: "<site_name>"
spec:
  baseDomain: "example.com"
  pullsecretRef:
    name: "assisted-deployment-pull-secret" (1)
  clusterImageSetNameRef: "openshift-4.11" (2)
  sshPublicKey: "ssh-rsa AAAA..." (3)
  clusters:
  - clusterName: "<site_name>"
    networkType: "OVNKubernetes"
    clusterLabels: (4)
      common: true
      group-du-sno: ""
      sites : "<site_name>"
    clusterNetwork:
      - cidr: 1001:1::/48
        hostPrefix: 64
    machineNetwork:
      - cidr: 1111:2222:3333:4444::/64
    serviceNetwork:
      - 1001:2::/112
    additionalNTPSources:
      - 1111:2222:3333:4444::2
    #crTemplates:
    #  KlusterletAddonConfig: "KlusterletAddonConfigOverride.yaml" (5)
    nodes:
      - hostName: "example-node.example.com" (6)
        role: "master"
        bmcAddress: idrac-virtualmedia://<out_of_band_ip>/<system_id>/ (7)
        bmcCredentialsName:
          name: "bmh-secret" (8)
        bootMACAddress: "AA:BB:CC:DD:EE:11"
        bootMode: "UEFI" (9)
        rootDeviceHints:
          wwn: "0x11111000000asd123"
        cpuset: "0-1,52-53"  (10)
        nodeNetwork: (11)
          interfaces:
            - name: eno1
              macAddress: "AA:BB:CC:DD:EE:11"
          config:
            interfaces:
              - name: eno1
                type: ethernet
                state: up
                ipv4:
                  enabled: false
                ipv6: (12)
                  enabled: true
                  address:
                  - ip: 1111:2222:3333:4444::aaaa:1
                    prefix-length: 64
            dns-resolver:
              config:
                search:
                - example.com
                server:
                - 1111:2222:3333:4444::2
            routes:
              config:
              - destination: ::/0
                next-hop-interface: eno1
                next-hop-address: 1111:2222:3333:4444::1
                table-id: 254

1	Create the `assisted-deployment-pull-secret` CR with the same namespace as the `SiteConfig` CR.
2	`clusterImageSetNameRef` defines an image set available on the hub cluster. To see the list of supported versions on your hub cluster, run `oc get clusterimagesets`.
3	Configure the SSH public key used to access the cluster.
4	Cluster labels must correspond to the `bindingRules` field in the `PolicyGenTemplate` CRs that you define. For example, `policygentemplates/common-ranGen.yaml` applies to all clusters with `common: true` set, `policygentemplates/group-du-sno-ranGen.yaml` applies to all clusters with `group-du-sno: ""` set.
5	Optional. The CR specifed under `KlusterletAddonConfig` is used to override the default `KlusterletAddonConfig` that is created for the cluster.
6	For single-node deployments, define a single host. For three-node deployments, define three hosts. For standard deployments, define three hosts with `role: master` and two or more hosts defined with `role: worker`.
7	BMC address that you use to access the host. Applies to all cluster types.
8	Name of the `bmh-secret` CR that you separately create with the host BMC credentials. When creating the `bmh-secret` CR, use the same namespace as the `SiteConfig` CR that provisions the host.
9	Configures the boot mode for the host. The default value is `UEFI`. Use `UEFISecureBoot` to enable secure boot on the host.
10	`cpuset` must match the value set in the cluster `PerformanceProfile` CR `spec.cpu.reserved` field for workload partitioning.
11	Specifies the network settings for the node.
12	Configures the IPv6 address for the host. For single-node OpenShift clusters with static IP addresses, the node-specific API and Ingress IPs should be the same.

Generate the day-0 installation CRs by processing the modified SiteConfig CR site-1-sno.yaml by running the following command:

$ podman run -it --rm -v `pwd`/out/argocd/example/siteconfig:/resources:Z -v `pwd`/site-install:/output:Z,U registry.redhat.io/openshift4/ztp-site-generate-rhel8:v4.11.1 generator install site-1-sno.yaml /output

Example output

site-install
└── site-1-sno
    ├── site-1_agentclusterinstall_example-sno.yaml
    ├── site-1-sno_baremetalhost_example-node1.example.com.yaml
    ├── site-1-sno_clusterdeployment_example-sno.yaml
    ├── site-1-sno_configmap_example-sno.yaml
    ├── site-1-sno_infraenv_example-sno.yaml
    ├── site-1-sno_klusterletaddonconfig_example-sno.yaml
    ├── site-1-sno_machineconfig_02-master-workload-partitioning.yaml
    ├── site-1-sno_machineconfig_predefined-extra-manifests-master.yaml
    ├── site-1-sno_machineconfig_predefined-extra-manifests-worker.yaml
    ├── site-1-sno_managedcluster_example-sno.yaml
    ├── site-1-sno_namespace_example-sno.yaml
    └── site-1-sno_nmstateconfig_example-node1.example.com.yaml

Optional: Generate just the day-0 MachineConfig installation CRs for a particular cluster type by processing the reference SiteConfig CR with the -E option. For example, run the following commands:

Create an output folder for the MachineConfig CRs:
```
$ mkdir -p ./site-machineconfig
```

Generate the MachineConfig installation CRs:

$ podman run -it --rm -v `pwd`/out/argocd/example/siteconfig:/resources:Z -v `pwd`/site-machineconfig:/output:Z,U registry.redhat.io/openshift4/ztp-site-generate-rhel8:v4.11.1 generator install -E site-1-sno.yaml /output

Example output

site-machineconfig
└── site-1-sno
    ├── site-1-sno_machineconfig_02-master-workload-partitioning.yaml
    ├── site-1-sno_machineconfig_predefined-extra-manifests-master.yaml
    └── site-1-sno_machineconfig_predefined-extra-manifests-worker.yaml

Generate and export the day-2 configuration CRs using the reference PolicyGenTemplate CRs from the previous step. Run the following commands:

Create an output folder for the day-2 CRs:
```
$ mkdir -p ./ref
```

Generate and export the day-2 configuration CRs:

$ podman run -it --rm -v `pwd`/out/argocd/example/policygentemplates:/resources:Z -v `pwd`/ref:/output:Z,U registry.redhat.io/openshift4/ztp-site-generate-rhel8:v4.11.1 generator config -N . /output

The command generates example group and site-specific PolicyGenTemplate CRs for single-node OpenShift, three-node clusters, and standard clusters in the ./ref folder.

Example output

ref
 └── customResource
      ├── common
      ├── example-multinode-site
      ├── example-sno
      ├── group-du-3node
      ├── group-du-3node-validator
      │    └── Multiple-validatorCRs
      ├── group-du-sno
      ├── group-du-sno-validator
      ├── group-du-standard
      └── group-du-standard-validator
           └── Multiple-validatorCRs

Use the generated CRs as the basis for the CRs that you use to install the cluster. You apply the installation CRs to the hub cluster as described in "Installing a single managed cluster". The configuration CRs can be applied to the cluster after cluster installation is complete.

Additional resources

Creating the managed bare-metal host secrets

Add the required secret custom resources (CRs) for the managed bare-metal host to the hub cluster. You need a secret for the ZTP pipeline to access the Baseboard Management Controller (BMC) and a secret for the assisted installer service to pull cluster installation images from the registry.

The secrets are referenced from the SiteConfig CR by name. The namespace must match the SiteConfig namespace.

Procedure

Create a YAML secret file containing credentials for the host Baseboard Management Controller (BMC) and a pull secret required for installing OpenShift and all add-on cluster Operators:

Save the following YAML as the file example-sno-secret.yaml:

apiVersion: v1
kind: secret
metadata:
  name: example-sno-bmc-secret
  namespace: example-sno (1)
data: (2)
  password: <base64_password>
  username: <base64_username>
type: Opaque
---
apiVersion: v1
kind: secret
metadata:
  name: pull-secret
  namespace: example-sno  (3)
data:
  .dockerconfigjson: <pull_secret> (4)
type: kubernetes.io/dockerconfigjson

1	Must match the namespace configured in the related `SiteConfig` CR
2	Base64-encoded values for `password` and `username`
3	Must match the namespace configured in the related `SiteConfig` CR
4	Base64-encoded pull secret

Add the relative path to example-sno-secret.yaml to the kustomization.yaml file that you use to install the cluster.

Installing a single managed cluster

You can manually deploy a single managed cluster using the assisted service and Red Hat Advanced Cluster Management (RHACM).

Prerequisites

You have installed the OpenShift CLI (oc).
You have logged in to the hub cluster as a user with cluster-admin privileges.
You have created the baseboard management controller (BMC) secret and the image pull-secret secret custom resources (CRs). See "Creating the managed bare-metal host secrets" for details.
Your target bare-metal host meets the networking and hardware requirements for managed clusters.

Procedure

Create a ClusterImageSet for each specific cluster version to be deployed, for example clusterImageSet-4.11.yaml. A ClusterImageSet has the following format:

apiVersion: hive.openshift.io/v1
kind: ClusterImageSet
metadata:
  name: openshift-4.11.0 (1)
spec:
   releaseImage: quay.io/openshift-release-dev/ocp-release:4.11.0-x86_64 (2)

1	The descriptive version that you want to deploy.
2	Specifies the `releaseImage` to deploy and determines the operating system image version. The discovery ISO is based on the image version as set by `releaseImage`, or the latest version if the exact version is unavailable.

Apply the clusterImageSet CR:
```
$ oc apply -f clusterImageSet-4.11.yaml
```

Create the Namespace CR in the cluster-namespace.yaml file:

apiVersion: v1
kind: Namespace
metadata:
     name: <cluster_name> (1)
     labels:
        name: <cluster_name> (1)

1	The name of the managed cluster to provision.

Apply the Namespace CR by running the following command:
```
$ oc apply -f cluster-namespace.yaml
```
Apply the generated day-0 CRs that you extracted from the ztp-site-generate container and customized to meet your requirements:
```
$ oc apply -R ./site-install/site-sno-1
```

Additional resources

Connectivity prerequisites for managed cluster networks

Monitoring the managed cluster installation status

Ensure that cluster provisioning was successful by checking the cluster status.

Prerequisites

All of the custom resources have been configured and provisioned, and the Agent custom resource is created on the hub for the managed cluster.

Procedure

Check the status of the managed cluster:
```
$ oc get managedcluster
```
True indicates the managed cluster is ready.
Check the agent status:
```
$ oc get agent -n <cluster_name>
```
Use the describe command to provide an in-depth description of the agent’s condition. Statuses to be aware of include BackendError, InputError, ValidationsFailing, InstallationFailed, and AgentIsConnected. These statuses are relevant to the Agent and AgentClusterInstall custom resources.
```
$ oc describe agent -n <cluster_name>
```

Check the cluster provisioning status:

$ oc get agentclusterinstall -n <cluster_name>

Use the describe command to provide an in-depth description of the cluster provisioning status:
```
$ oc describe agentclusterinstall -n <cluster_name>
```
Check the status of the managed cluster’s add-on services:
```
$ oc get managedclusteraddon -n <cluster_name>
```

Retrieve the authentication information of the kubeconfig file for the managed cluster:

$ oc get secret -n <cluster_name> <cluster_name>-admin-kubeconfig -o jsonpath={.data.kubeconfig} | base64 -d > <directory>/<cluster_name>-kubeconfig

Troubleshooting the managed cluster

Use this procedure to diagnose any installation issues that might occur with the managed cluster.

Procedure

Check the status of the managed cluster:
```
$ oc get managedcluster
```
Example output
```
NAME            HUB ACCEPTED   MANAGED CLUSTER URLS   JOINED   AVAILABLE   AGE
SNO-cluster     true                                   True     True      2d19h
```
If the status in the AVAILABLE column is True, the managed cluster is being managed by the hub.

If the status in the AVAILABLE column is Unknown, the managed cluster is not being managed by the hub. Use the following steps to continue checking to get more information.

Check the AgentClusterInstall install status:

$ oc get clusterdeployment -n <cluster_name>

Example output

NAME        PLATFORM            REGION   CLUSTERTYPE   INSTALLED    INFRAID    VERSION  POWERSTATE AGE
Sno0026    agent-baremetal                               false                          Initialized
2d14h

If the status in the INSTALLED column is false, the installation was unsuccessful.

If the installation failed, enter the following command to review the status of the AgentClusterInstall resource:
```
$ oc describe agentclusterinstall -n <cluster_name> <cluster_name>
```
Resolve the errors and reset the cluster:
1. Remove the cluster’s managed cluster resource:
  $ oc delete managedcluster <cluster_name>
2. Remove the cluster’s namespace:
  $ oc delete namespace <cluster_name>
  This deletes all of the namespace-scoped custom resources created for this cluster. You must wait for the ManagedCluster CR deletion to complete before proceeding.
3. Recreate the custom resources for the managed cluster.

RHACM generated cluster installation CRs reference

Red Hat Advanced Cluster Management (RHACM) supports deploying OpenShift Container Platform on single-node clusters, three-node clusters, and standard clusters with a specific set of installation custom resources (CRs) that you generate using SiteConfig CRs for each site.

Every managed cluster has its own namespace, and all of the installation CRs except for ManagedCluster and ClusterImageSet are under that namespace. ManagedCluster and ClusterImageSet are cluster-scoped, not namespace-scoped. The namespace and the CR names match the cluster name.

The following table lists the installation CRs that are automatically applied by the RHACM assisted service when it installs clusters using the SiteConfig CRs that you configure.

Table 1. Cluster installation CRs generated by RHACM
CR	Description	Usage
`BareMetalHost`	Contains the connection information for the Baseboard Management Controller (BMC) of the target bare-metal host.	Provides access to the BMC to load and boot the discovery image on the target server by using the Redfish protocol.
`InfraEnv`	Contains information for installing OpenShift Container Platform on the target bare-metal host.	Used with `ClusterDeployment` to generate the discovery ISO for the managed cluster.
`AgentClusterInstall`	Specifies details of the managed cluster configuration such as networking and the number of control plane nodes. Displays the cluster `kubeconfig` and credentials when the installation is complete.	Specifies the managed cluster configuration information and provides status during the installation of the cluster.
`ClusterDeployment`	References the `AgentClusterInstall` CR to use.	Used with `InfraEnv` to generate the discovery ISO for the managed cluster.
`NMStateConfig`	Provides network configuration information such as `MAC` address to `IP` mapping, DNS server, default route, and other network settings. This is not needed if DHCP is used.	Sets up a static IP address for the managed cluster’s Kube API server.
`Agent`	Contains hardware information about the target bare-metal host.	Created automatically on the hub when the target machine’s discovery image boots.
`ManagedCluster`	When a cluster is managed by the hub, it must be imported and known. This Kubernetes object provides that interface.	The hub uses this resource to manage and show the status of managed clusters.
`KlusterletAddonConfig`	Contains the list of services provided by the hub to be deployed to the `ManagedCluster` resource.	Tells the hub which addon services to deploy to the `ManagedCluster` resource.
`Namespace`	Logical space for `ManagedCluster` resources existing on the hub. Unique per site.	Propagates resources to the `ManagedCluster`.
`secret`	Two CRs are created: `BMC secret` and `Image Pull secret`.	`BMC secret` authenticates into the target bare-metal host using its username and password. `Image Pull secret` contains authentication information for the OpenShift Container Platform image installed on the target bare-metal host.
`ClusterImageSet`	Contains OpenShift Container Platform image information such as the repository and image name.	Passed into resources to provide OpenShift Container Platform images.