Deploying hosted control planes on Azure - Deploying hosted control planes | Hosted control planes

Overview of hosted control planes on Azure
Setting up Azure resources
Deleting an Azure hosted cluster and its resources

With hosted control planes on Microsoft Azure, you can reduce the cost associated with dedicated control-plane node VMs for each cluster. You can provision compute nodes as Azure Virtual Machine Scale sets for dynamic scaling, and ensure credential isolation with per-cluster Azure service principals.

Hosted control planes on Azure is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Overview of hosted control planes on Azure

You can deploy and manage hosted clusters on an OpenShift management cluster that runs in Microsoft Azure. With a self-managed deployment model, you manage your own OpenShift cluster.

Architecture of hosted control planes on Azure

At a high level, the architecture of hosted control planes on Azure consists of three layers:

Management cluster: An Azure OpenShift cluster that hosts the HyperShift Operator and the control planes for your hosted clusters.
Control plane: Kubernetes control-plane components that run as pods on the management cluster.
Data plane: Compute nodes that run as Azure virtual machines (VMs) in your Azure subscription.

The architecture uses Azure Workload Identity for secure, credential-free authentication between OKD components and Azure services. As a result, you do not need to manage long-lived service principal credentials, and you get better security through federated identity credentials.

Deployment workflow for hosted control planes on Azure

Deploying self-managed hosted control planes on Azure involves a three-phase process in the following order:

Phase 1: Set up Azure Workload Identity

This phase establishes secure authentication infrastructure so that OKD components can access Azure services. During this phase, the security infrastructure that your hosted clusters require is created:

Managed identities for each OKD component, including the image registry, ingress, CSI drivers, cloud provider, network operator, and so on
OIDC issuer in Azure Blob Storage for service account token validation
Federated credentials that establish trust relationships between Azure Entra ID and OKD service accounts

Phase 2: Set up the management cluster

In this phase, you prepare your OKD management cluster on Azure to host and manage hosted clusters. During this phase, you install the following components:

Azure DNS zones, for private and public-private topologies
External DNS, for private and public-private topologies
HyperShift Operator

Phase 3: Create hosted clusters

In this phase, you create and configure hosted clusters. This phase involves the following steps:

Setting up infrastructure
Creating a hosted cluster
Integrating workload identity
Configuring private endpoint access (optional)

Additional resources

Azure Workload Identity documentation

Azure infrastructure and credentials

Before you get started with hosted control planes on Microsoft Azure, get familiar with the required Azure resources and the necessary permissions.

Summary of requirements

You need the following resources, tools, access, and permissions to set up hosted control planes on Azure:

Azure resources

An OKD management cluster on Azure.
An Azure subscription with contributor and user-access administrator permissions.
(Optional) A parent DNS zone in Azure for delegating cluster DNS records. This DNS zone is required only if you plan to use external DNS.

Tools and access

Azure command-line interface (CLI), az, configured with your subscription
OpenShift CLI (oc)
hosted control planes CLI, hcp
jq command-line JSON processor
Cloud Credential Operator utility (ccoctl)
A valid OKD pull secret

Permissions

Subscription-level contributor and user access administrator roles
Microsoft Graph api permissions for creating service principals

DNS management options

You can set up hosted control planes on Azure with or without external DNS. If you plan to use a private or a public-private topology, external DNS is required. See the following table for more information:

Table 1. DNS approaches for hosted control planes on Azure
	With external DNS	Without external DNS
Best for	Production, multi-cluster	Development, testing
api server DNS	Custom (`api-cluster.example.com`)	Azure Load Balancer (`abc123.region.cloudapp.azure.com`)
Setup complexity	Low, requires DNS zones and service principals	None
Management	Fully automatic	Manual or Azure-provided

Resource group strategy

Cluster-specific resource groups are created and deleted with each hosted cluster. These resources include managed resource groups for cluster infrastructure. If you use custom networking, these resources also include Virtual Network (VNet) resource groups and Network Security Group (NSG) resource groups.

Security considerations for hosted control planes on Azure

Hosted control planes on Azure employs the following security best practices:

Workload identity federation, which eliminates long-lived credentials by using OIDC-based authentication.
Least-privilege access, where each component has its own managed identity with minimal required permissions.
Network isolation, where you can use custom VNets and NSGs to implement network segmentation and security policies.
Federated credentials, where trust relationships are scoped to specific service accounts, preventing unauthorized access.
Private connectivity, available as an option through Azure Private Link, which provides private api server access to ensure that control-plane traffic never traverses the public internet. Private connectivity is available for private and public-private topologies only.

Setting up Azure resources

Before you can create management and hosted clusters for your deployment of hosted control planes on Azure, you need to set up an OIDC issuer, Workload Identities, and your Azure infrastructure.

Setting up an OIDC issuer

To prepare to deploy hosted control planes on Azure, you need to set up Azure Workload Identities and an OIDC issuer for hosted clusters.

Prerequisites

The Azure command-line interface (CLI) is installed and configured.
The jq command-line JSON processor is installed.
The Cloud Credential Operator utility (ccoctl) is installed. For more information, see "How to obtain the ccoctl tool for OpenShift 4".
The appropriate Azure permissions are set.

Procedure

Set your environment variables as shown in the following example:

PERSISTENT_RG_NAME="os4-common"
LOCATION="eastus"
AZURE_CREDS="/path/to/azure-creds.json"
SUBSCRIPTION_ID="my-subscription-id"

Create a persistent resource group by entering the following command:

$ az group create --name $PERSISTENT_RG_NAME --location $LOCATION

Configure an OIDC issuer URL by using the Cloud Credential Operator tool to complete the following steps:

Set the OIDC issuer variables as shown in the following example:

OIDC_STORAGE_ACCOUNT_NAME="yourstorageaccount"
TENANT_ID="your-tenant-id"

Create an RSA key pair and save the private and public key by entering the following command:
```
$ ccoctl azure create-key-pair
```

Set variables for the token issuer key paths as shown in the following example:

SA_TOKEN_ISSUER_PRIVATE_KEY_PATH="/path/to/serviceaccount-signer.private"
SA_TOKEN_ISSUER_PUBLIC_KEY_PATH="/path/to/serviceaccount-signer.public"

Create an OIDC issuer by entering the following command:

$ ccoctl azure create-oidc-issuer \
    --oidc-resource-group-name ${PERSISTENT_RG_NAME} \
    --tenant-id ${TENANT_ID} \
    --region ${LOCATION} \
    --name ${OIDC_STORAGE_ACCOUNT_NAME} \
    --subscription-id ${SUBSCRIPTION_ID} \
    --public-key-file ${SA_TOKEN_ISSUER_PUBLIC_KEY_PATH}

Set the OIDC issuer URL as shown in the following example:

OIDC_ISSUER_URL="https://${OIDC_STORAGE_ACCOUNT_NAME}.blob.core.windows.net/${OIDC_STORAGE_ACCOUNT_NAME}"

Verification

Try to access the OIDC issuer by entering the following command:

$ curl -s "${OIDC_ISSUER_URL}/.well-known/openid-configuration" | jq .

Additional resources

How to obtain the ccoctl tool for OpenShift 4 (Red Hat Knowledgebase article)

Creating Azure Workload Identities

To ensure control over Identity and Access Management (IAM) resources in your Azure deployment, create Workload Identities separately from your infrastructure.

Workload Identities authenticate hosted cluster components to Azure services by using OIDC federation. You must create identities separately and then consume them during infrastructure or cluster creation.

Prerequisites

You have an Azure credentials file in the following format:

Example file

{
  "subscriptionId": "your-subscription-id",
  "tenantId": "your-tenant-id",
  "clientId": "your-client-id",
  "clientSecret": "your-client-secret"
}

You have a resource group to create the managed identities in.
You have an OIDC issuer URL for Workload Identity federation. For more information, see "Setting up an OIDC issuer".

Procedure

Set environment variables as shown in the following example:

CLUSTER_NAME="my-self-managed-cluster"
INFRA_ID="${CLUSTER_NAME}-$(openssl rand -hex 4)"

On the hosted control planes command-line interface, hcp, enter the following command:
```
$ hcp create iam azure \
    --name <my_cluster_name> \
    --infra-id <infra_id> \
    --azure-creds <azure_credentials_file> \
    --resource-group-name <resource_group> \
    --oidc-issuer-url <oidc_issuer_url> \
    --output-file <workload_identities_file> \
    --location <my_region> \
    --cloud <my_cloud_environment>
```
where:

<my_cluster_name>

Specifies the name of the cluster you intend to create.

<infra_id>

Specifies the unique identifier for naming Azure resources. Typically, this identifier is the cluster name with a suffix.

<azure_credentials_file>

Specifies the Azure credentials file with permission to create managed identities and federated credentials.

<resource_group>

Specifies the name of the resource group where you intend to create identities.

<oidc_issuer_url>

Specifies the URL of the OIDC identity provider for Workload Identity federation.

<workload_identities_file>

Specifies the output file path, such as my-cluster-name-iam-output.json.

You can also add these optional flags to the hcp create iam azure command:

<my_region>

Specifies the Azure region for the managed identities. The default value is eastus.

<my_cloud_environment>

Specifies the Azure cloud environment. The default value is AzurePublicCloud.

Verification

Review the output file, which looks like the following example:

Example output

{
  "disk": {
    "tenantID": "...",
    "clientID": "...",
    "resourceID": "/subscriptions/.../providers/Microsoft.ManagedIdentity/userAssignedIdentities/my-cluster-abc123-disk"
  },
  "file": {
    "tenantID": "...",
    "clientID": "...",
    "resourceID": "..."
  },
  "imageRegistry": { ... },
  "ingress": { ... },
  "cloudProvider": { ... },
  "nodePoolManagement": { ... },
  "network": { ... },
  "controlPlaneOperator": { ... }
}

The output includes 8 user-assigned identities, one per cluster component, along with federated credentials for each identity:

Disk CSI driver
File CSI driver
Image registry
Ingress Operator
Cloud provider
Node pool management
Network Operator
Control Plane Operator

Creating Azure infrastructure

Create an Azure infrastructure separately so that when you create a hosted cluster on Azure, you can use pre-existing infrastructure.

Prerequisites

You have an Azure credentials file with the following format:

{
  "subscriptionId": "<my_subscription_id>",
  "tenantId": "<my_tenant_id>",
  "clientId": "<my_client_id>",
  "clientSecret": "<my_client_secret>"
}

You have an existing public DNS zone in your Azure subscription for your base domain.
You created Workload Identities. For more information, see "Creating Azure Workload Identities".

Procedure

To create the infrastructure with a new virtual network, subnet, and network security group, enter the following command:
```
$ hcp create infra azure \
  --name <my_cluster_name> \
  --infra-id <infra_id> \
  --azure-creds <azure_credentials_file> \
  --base-domain <base_domain> \
  --location <location> \
  --workload-identities-file <workload_identities_file> \
  --assign-identity-roles \
  --dns-zone-rg-name <dns_zone_rg> \
  --output-file <output_infra_file>
```
where:
- --name specifies the name of the hosted cluster you intend to create.
- --infra-id specifies a unique name that identifies your infrastructure. This value is used to name and tag Azure resources. Typically, it is the name of your cluster with a suffix appended to it.
- --azure-creds specifies an Azure credentials file that has permission to create infrastructure resources, such as virtual networks, subnets, and load balancers.
- --base-domain specifies the base domain for the ingress of your hosted cluster. The base domain must correspond to an existing public DNS zone in your Azure subscription.
- --location specifies the Azure region where you want to create the infrastructure, such as eastus or westus2.
- --workload-identities-file specifies the path to the JSON file that contains the Workload Identity configuration.
- --assign-identity-roles specifies that automatic RBAC role assignment is enabled for Workload Identities.
- --dns-zone-rg-name specifies the name of the resource group that contains your public DNS zone.
- --output-file specifies the file where the details of the infrastructure are stored in YAML format.
To create the infrastructure with an existing virtual network, subnet, and network security group, enter the following command:
```
$ hcp create infra azure \
  --name <my_cluster_name> \
  --infra-id <infra_id> \
  --azure-creds <azure_credentials_file> \
  --base-domain <base_domain> \
  --location <location> \
  --workload-identities-file <workload_identities_file> \
  --vnet-id /subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.Network/virtualNetworks/<vnet_name> \
  --subnet-id /subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.Network/virtualNetworks/<vnet_name>/subnets/<subnet_name> \
  --network-security-group-id /subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.Network/networkSecurityGroups/<network_security_group_name> \
  --assign-identity-roles \
  --dns-zone-rg-name <dns_zone_rg> \
  --output-file <output_infra_file>
```
where:
- --name specifies the name of the hosted cluster you intend to create.
- --infra-id specifies a unique name that identifies your infrastructure. This value is used to name and tag Azure resources. Typically, it is the name of your cluster with a suffix appended to it.
- --azure-creds specifies an Azure credentials file that has permission to create infrastructure resources, such as virtual networks, subnets, and load balancers.
- --base-domain specifies the base domain for the ingress of your hosted cluster. The base domain must correspond to an existing public DNS zone in your Azure subscription.
- --location specifies the Azure region where you want to create the infrastructure, such as eastus or westus2.
- --workload-identities-file specifies the path to the JSON file that contains the Workload Identity configuration.
- --vnet-id specifies your existing virtual network ID, which includes your subscription ID and your virtual network name.
- --subnet-id specifies the ARM resource ID of your subnet, which includes your subscription ID, virtual network name, and subnet name.
- --network-security-group-id specifies your existing network security group ID, which includes your subscription ID and your network security group name.
- --assign-identity-roles specifies that automatic RBAC role assignment is enabled for Workload Identities.
- --dns-zone-rg-name specifies the name of the resource group that contains your public DNS zone.
- --output-file specifies the file where the details of the infrastructure are stored in YAML format.

Additional resources

Creating Azure Workload Identities

Deleting an Azure hosted cluster and its resources

If you no longer need a hosted cluster, you can remove it and its infrastructure. Any related Workload Identities and OIDC issuers that were created during setup can be reused for other clusters or deleted separately if you no longer need them.

Deleting a hosted cluster on Azure

If you are no longer using a hosted cluster on Azure, you can delete it.

Procedure

To delete a hosted cluster, enter the following command:
```
$ hcp destroy cluster azure \
  --name $CLUSTER_NAME \
  --azure-creds $AZURE_CREDS \
  --dns-zone-rg-name $PERSISTENT_RG_NAME \
  --preserve-resource-group
```
- --name specifies your hosted cluster name.
- --azure-creds specifies an Azure credentials file that has permission to create infrastructure resources, such as virtual networks, subnets, and load balancers.
- --dns-zone-rg-name specifies the name of the resource group that contains your DNS zone.
- --preserve-resource-group specifies that the infrastructure will be preserved. If you do not want to preserve the infrastructure, do not include this flag.

Deleting Azure infrastructure

If you have Azure infrastructure without a hosted cluster, you can remove the infrastructure if you are not using it.

For example, this scenario can happen if you created the infrastructure standalone but never created a hosted cluster. Or, you might have manually deleted the hosted cluster or management cluster, but the infrastructure resources still exist.

You can delete the entire infrastructure, or delete cluster-specific resources but preserve the main resource group. Preserving the main resource group is helpful when you have other resources in the same resource group that you want to keep.

If you have a hosted cluster and want to delete infrastructure while you delete the hosted cluster, follow the steps in "Deleting a hosted cluster on Azure", but omit the --preserve-resource-group flag.

Procedure

To delete the infrastructure, enter one of the following commands:
- To delete the infrastructure, including the resource group, enter the following command:
  $ hcp destroy infra azure \ --name <my_cluster_name> \ --infra-id <infra_id> \ --azure-creds <azure_credentials_file>
  - --name specifies your hosted cluster name.
  - --infra-id specifies a unique name that identifies your infrastructure. This value is used to name and tag Azure resources. Typically, it is the name of your cluster with a suffix appended to it.
  - --azure-creds specifies an Azure credentials file that has permission to create infrastructure resources, such as virtual networks, subnets, and load balancers.
- To preserve the resource group but delete only cluster-specific resources, enter the following command:
  $ hcp destroy infra azure \ --name <my_cluster_name> \ --infra-id <infra_id> \ --azure-creds <azure_credentials_file> \ --preserve-resource-group
  - --name specifies your hosted cluster name.
  - --infra-id specifies a unique name that identifies your infrastructure. This value is used to name and tag Azure resources. Typically, it is the name of your cluster with a suffix appended to it.
  - --azure-creds specifies an Azure credentials file that has permission to create infrastructure resources, such as virtual networks, subnets, and load balancers.
  - --preserve-resource-group specifies that you want to preserve the resource group.

Deleting Azure Workload Identities

As part of the process to delete a hosted cluster on Azure, you need to delete the Workload Identities.

To delete the Workload Identities, you enter a command on the hosted control planes command-line interface, hcp. The command uses the file that was generated when you created the Workload Identities to identify the identities to delete. Both the managed identities and their federated credentials are removed.

Prerequisites

If you created infrastructure by using the Workload Identities, delete the infrastructure before you delete the identities.

Procedure

Enter the following command:
```
$ hcp destroy iam azure \
    --azure-creds <azure_credentials_file> \
    --workload-identities-file <workload_identities_file> \
    --resource-group-name <resource_group> \
    --name <my_cluster_name> \
    --infra-id <infra_id> \
    --dns-zone-rg-name <dns_zone_rg> \
    --cloud <my_cloud_environment>
```
- <azure_credentials_file> specifies the Azure credentials file with permission to create managed identities and federated credentials.
- <workload_identities_file> specifies the path to the Workload Identities JSON file, such as my-cluster-name-iam-output.json.
- <resource_group> specifies the name of the resource group where you created identities.
- <my_cluster_name> specifies the name of your hosted cluster.
- <infra_id> specifies the unique identifier for naming Azure resources. Typically, this identifier is the cluster name with a suffix.
- <dns_zone_rg> specifies the DNS zone resource group.
- <my_cloud_environment> specifies the Azure cloud environment. Setting the --cloud flag is optional. The default value is AzurePublicCloud.