Advanced managed cluster configuration with SiteConfig resources

Create a set of extra manifest CRs that the GitOps ZTP pipeline uses to customize the cluster installs.

In your custom /siteconfig directory, create a subdirectory /custom-manifest for your extra manifests. The following example illustrates a sample /siteconfig with /custom-manifest folder:

siteconfig
├── site1-sno-du.yaml
├── site2-standard-du.yaml
├── extra-manifest/
└── custom-manifest
    └── 01-example-machine-config.yaml

Add your custom extra manifest CRs to the siteconfig/custom-manifest directory.

In your SiteConfig CR, enter the directory name in the extraManifests.searchPaths field, for example:

clusters:
- clusterName: "example-sno"
  networkType: "OVNKubernetes"
  extraManifests:
    searchPaths:
      - extra-manifest/ (1)
      - custom-manifest/ (2)

Save the SiteConfig, /extra-manifest, and /custom-manifest CRs, and push them to the site configuration repo.

To prevent the GitOps ZTP pipeline from applying the 03-sctp-machine-config-worker.yaml CR file, apply the following YAML in the SiteConfig CR:

apiVersion: ran.openshift.io/v1
kind: SiteConfig
metadata:
  name: "site1-sno-du"
  namespace: "site1-sno-du"
spec:
  baseDomain: "example.com"
  pullsecretRef:
    name: "assisted-deployment-pull-secret"
  clusterImageSetNameRef: "openshift-4.16"
  sshPublicKey: "<ssh_public_key>"
  clusters:
- clusterName: "site1-sno-du"
  extraManifests:
    filter:
      exclude:
        - 03-sctp-machine-config-worker.yaml

Save the SiteConfig CR and push the changes to the site configuration repository.

Optional: To prevent the GitOps ZTP pipeline from applying all the /source-crs/extra-manifest CRs during cluster installation, apply the following YAML in the SiteConfig CR:

- clusterName: "site1-sno-du"
  extraManifests:
    filter:
      inclusionDefault: exclude

Optional: To exclude all the /source-crs/extra-manifest RAN CRs and instead include a custom CR file during installation, edit the custom SiteConfig CR to set the custom manifests folder and the include file, for example:

clusters:
- clusterName: "site1-sno-du"
  extraManifestPath: "<custom_manifest_folder>" (1)
  extraManifests:
    filter:
      inclusionDefault: exclude  (2)
      include:
        - custom-sctp-machine-config-worker.yaml

siteconfig
  ├── site1-sno-du.yaml
  └── user-custom-manifest
        └── custom-sctp-machine-config-worker.yaml

Update the SiteConfig CR to include the bmac.agent-install.openshift.io/remove-agent-and-node-on-delete=true annotation and push the changes to the Git repository:

apiVersion: ran.openshift.io/v1
kind: SiteConfig
metadata:
  name: "cnfdf20"
  namespace: "cnfdf20"
spec:
  clusters:
    nodes:
    - hostname: node6
      role: "worker"
      crAnnotations:
        add:
          BareMetalHost:
            bmac.agent-install.openshift.io/remove-agent-and-node-on-delete: true
# ...

Verify that the BareMetalHost object is annotated by running the following command:

oc get bmh -n <managed-cluster-namespace> <bmh-object> -ojsonpath='{.metadata}' | jq -r '.annotations["bmac.agent-install.openshift.io/remove-agent-and-node-on-delete"]'

true

Suppress the generation of the BareMetalHost CR by updating the SiteConfig CR to include the crSuppression.BareMetalHost annotation:

apiVersion: ran.openshift.io/v1
kind: SiteConfig
metadata:
  name: "cnfdf20"
  namespace: "cnfdf20"
spec:
  clusters:
  - nodes:
    - hostName: node6
      role: "worker"
      crSuppression:
      - BareMetalHost
# ...

Push the changes to the Git repository and wait for deprovisioning to start. The status of the BareMetalHost CR should change to deprovisioning. Wait for the BareMetalHost to finish deprovisioning, and be fully deleted.

Verify that the BareMetalHost and Agent CRs for the worker node have been deleted from the hub cluster by running the following commands:

$ oc get bmh -n <cluster-ns>

$ oc get agent -n <cluster-ns>

Verify that the node record has been deleted from the spoke cluster by running the following command:

$ oc get nodes