$ oc edit <resource_type> <resource_name> -n kubevirt-hyperconverged
You can configure node placement rules to specify where OKD Virtualization Operators, workloads, and controllers are deployed. Custom placement rules allow you to isolate virtual machine (VM) traffic or dedicate specialized compute resources to critical workloads.
|
You can configure node placement rules for some components after installing OKD Virtualization, but virtual machines cannot be present if you want to configure node placement rules for workloads. |
You can use node placement rules to deploy virtual machines only on nodes intended for virtualization workloads, to deploy Operators only on infrastructure nodes, or to maintain separation between workloads.
Depending on the object, you can use one or more of the following rule types:
nodeSelectorAllows pods to be scheduled on nodes that are labeled with the key-value pair or pairs that you specify in this field. The node must have labels that exactly match all listed pairs.
affinityEnables you to use more expressive syntax to set rules that match nodes with pods. Affinity also allows for more nuance in how the rules are applied. For example, you can specify that a rule is a preference, not a requirement. If a rule is a preference, pods are still scheduled when the rule is not satisfied.
tolerationsAllows pods to be scheduled on nodes that have matching taints. If a taint is applied to a node, that node only accepts pods that tolerate the taint.
You can apply node placement rules to ensure that virtualization components run on the most suitable nodes for your workload requirements. You can apply node placement rules by editing OKD Virtualization objects.
You have installed the OpenShift CLI (oc).
You are logged in with cluster administrator permissions.
Edit the object in your default editor by running the following command:
$ oc edit <resource_type> <resource_name> -n kubevirt-hyperconverged
Save the file to apply the changes.
Examples of node placement rules for different objects that determine where OKD Virtualization components are deployed.
To specify the nodes where OLM deploys the OKD Virtualization Operators, edit the Subscription object during OKD Virtualization installation.
Currently, you cannot configure node placement rules for the Subscription object by using the web console.
The Subscription object does not support the affinity node placement rule.
Example Subscription object with nodeSelector rule:
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: hco-operatorhub
namespace: kubevirt-hyperconverged
spec:
source: community-operators
sourceNamespace: openshift-marketplace
name: community-kubevirt-hyperconverged
startingCSV: kubevirt-hyperconverged-operator.v4.22.0
channel: "stable"
config:
nodeSelector:
example.io/example-infra-key: example-infra-value
OLM deploys the OKD Virtualization Operators on nodes labeled example.io/example-infra-key = example-infra-value.
The following pods are affected by the Subscription object node placement rules. These pods are created when the OKD Virtualization Operator is first installed, even without an HyperConverged custom resource (CR):
aaq-operator
cdi-operator
cluster-network-addons-operator
hco-operator
hco-webhook
hostpath-provisioner-operator
hyperconverged-cluster-cli-download
kubevirt-migration-operator
ssp-operator
virt-operator
Example Subscription object with tolerations rule:
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: hco-operatorhub
namespace: kubevirt-hyperconverged
spec:
source: community-operators
sourceNamespace: openshift-marketplace
name: community-kubevirt-hyperconverged
startingCSV: kubevirt-hyperconverged-operator.v4.22.0
channel: "stable"
config:
tolerations:
- key: "key"
operator: "Equal"
value: "virtualization"
effect: "NoSchedule"
OLM deploys OKD Virtualization Operators on nodes labeled key = virtualization:NoSchedule taint. Only pods with the matching tolerations are scheduled on these nodes.
To specify the nodes where OKD Virtualization deploys its components, you can edit the nodePlacement object in the HyperConverged custom resource (CR) file that you create during OKD Virtualization installation.
Example HyperConverged object with nodeSelector rule:
apiVersion: hco.kubevirt.io/v1
kind: HyperConverged
metadata:
name: kubevirt-hyperconverged
namespace: kubevirt-hyperconverged
spec:
deployment:
nodePlacements:
infra:
nodeSelector:
example.io/example-infra-key: example-infra-value
workload:
nodeSelector:
example.io/example-workloads-key: example-workloads-value
Infrastructure resources are placed on nodes labeled example.io/example-infra-key = example-infra-value.
Workloads are placed on nodes labeled example.io/example-workloads-key = example-workloads-value.
Example HyperConverged object with affinity rule:
apiVersion: hco.kubevirt.io/v1
kind: HyperConverged
metadata:
name: kubevirt-hyperconverged
namespace: kubevirt-hyperconverged
spec:
deployment:
nodePlacements:
infra:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: example.io/example-infra-key
operator: In
values:
- example-infra-value
workload:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: example.io/example-workloads-key
operator: In
values:
- example-workloads-value
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: example.io/num-cpus
operator: Gt
values:
- 8
Infrastructure resources are placed on nodes labeled example.io/example-infra-key = example-value.
Workloads are placed on nodes labeled example.io/example-workloads-key = example-workloads-value.
Nodes that have more than eight CPUs are preferred for workloads, but if they are not available, pods are still scheduled.
Example HyperConverged object with tolerations rule:
apiVersion: hco.kubevirt.io/v1
kind: HyperConverged
metadata:
name: kubevirt-hyperconverged
namespace: kubevirt-hyperconverged
spec:
deployment:
nodePlacements:
workload:
tolerations:
- key: "key"
operator: "Equal"
value: "virtualization"
effect: "NoSchedule"
Nodes reserved for OKD Virtualization components are labeled with the key = virtualization:NoSchedule taint. Only pods with matching tolerations are scheduled on reserved nodes.
The following pods are affected by the HyperConverged object node placement rules. These pods are created when an HyperConverged CR is deployed:
cdi-apiserver
cdi-deployment
cdi-uploadproxy
kubemacpool-cert-manager
kubemacpool-mac-controller-manager
kubevirt-apiserver-proxy
kubevirt-console-plugin
kubevirt-ipam-controller-manager
kubevirt-migration-controller
virt-api
virt-controller
virt-exportproxy
virt-template-validator
You can edit the HostPathProvisioner object directly or by using the web console.
|
You must schedule the hostpath provisioner (HPP) and the OKD Virtualization components on the same nodes. Otherwise, virtualization pods that use the hostpath provisioner cannot run. You cannot run virtual machines. |
After you deploy a virtual machine (VM) with the HPP storage class, you can remove the hostpath provisioner pod from the same node by using the node selector. However, you must first revert that change, at least for that specific node, and wait for the pod to run before trying to delete the VM.
You can configure node placement rules by specifying nodeSelector, affinity, or tolerations for the spec.workload field of the HostPathProvisioner object that you create when you install the hostpath provisioner.
Example HostPathProvisioner object with nodeSelector rule:
apiVersion: hostpathprovisioner.kubevirt.io/v1beta1
kind: HostPathProvisioner
metadata:
name: hostpath-provisioner
spec:
imagePullPolicy: IfNotPresent
pathConfig:
path: "</path/to/backing/directory>"
useNamingPrefix: false
workload:
nodeSelector:
example.io/example-workloads-key: example-workloads-value
Workloads are placed on nodes labeled example.io/example-workloads-key = example-workloads-value.
On IBM Z® (s390x) clusters, OKD Virtualization only supports VMs on bare metal LPAR nodes. Without a nodeSelector rule, the scheduler can place VM workloads on incompatible z/VM or KVM guest nodes, causing VM pods to enter a crash loop.
By patching the HyperConverged custom resource (CR), you can restrict both OKD Virtualization infrastructure components and VM workloads to bare metal LPAR nodes in a single declarative configuration. The Node Feature Discovery (NFD) Operator automatically labels bare metal LPAR nodes with feature.node.kubernetes.io/cpu-model.hypervisor: PR_SM by reading /proc/cpuinfo at node boot. You can use this label as a reliable discriminator for LPAR-only placement.
Your cluster has bare metal LPAR compute nodes on IBM Z® or IBM® LinuxONE hardware.
You have installed the Node Feature Discovery (NFD) Operator and an active NodeFeatureDiscovery instance is running.
You have installed OKD Virtualization.
You have installed the OpenShift CLI (oc).
You are logged in with cluster administrator permissions.
Verify that the NFD Operator has labeled the LPAR nodes by running the following command:
$ oc get nodes -L feature.node.kubernetes.io/cpu-model.hypervisor
worker-lpar PR_SM
worker-zvm z_VM_7.4.0
worker-kvm KVM_Linux
Nodes labeled PR_SM are valid VM targets. Nodes labeled KVM_Linux or z_VM_7.4.0 are not supported VM targets.
Patch the HyperConverged CR to restrict OKD Virtualization infrastructure components and VM workloads to PR_SM nodes by running the following command:
$ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv \
--type=merge -p '
spec:
infra:
nodePlacement:
nodeSelector:
feature.node.kubernetes.io/cpu-model.hypervisor: "PR_SM"
workloads:
nodePlacement:
nodeSelector:
feature.node.kubernetes.io/cpu-model.hypervisor: "PR_SM"
'
where:
spec.infra.nodePlacementSpecifies the node placement for OKD Virtualization infrastructure components, including virt-handler. Setting this field prevents virt-handler rollout failures on non-LPAR worker nodes.
spec.workloads.nodePlacementSpecifies the default scheduling placement for all VM pods (virt-launcher) across the cluster, ensuring that VMs start only on PR_SM LPAR nodes without requiring per-VM nodeSelector definitions.
|
In environments where |
$ oc adm taint nodes <non_lpar_node> kubevirt.io/no-virt=true:NoSchedule
Verify that virt-handler pods are running only on PR_SM LPAR nodes by running the following command:
$ oc get pods -n openshift-cnv -l kubevirt.io=virt-handler -o wide
Confirm that no virt-handler pods are running on nodes labeled KVM_Linux or z_VM_7.4.0.
Verify that the HyperConverged Operator is healthy by running the following command:
$ oc get hyperconverged kubevirt-hyperconverged -n openshift-cnv
The output must show Available: True and Degraded: False.
Optional: Deploy a test VM without a nodeSelector to confirm cluster-wide placement by running the following command:
$ oc apply -f - <<EOF
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: test-vm-placement
namespace: default
spec:
runStrategy: Always
template:
spec:
domain:
devices:
disks:
- disk:
bus: virtio
name: containerdisk
resources:
requests:
memory: 512Mi
volumes:
- containerDisk:
image: quay.io/kubevirt/cirros-container-disk-demo:latest
name: containerdisk
EOF
Verify that the test VM runs on a PR_SM LPAR node by running the following command:
$ NODE=$(oc get pod -n default -l vm.kubevirt.io/name=test-vm-placement \
-o jsonpath='{.items[0].spec.nodeName}') && \
echo "Scheduled node: $NODE" && \
oc get node "$NODE" -L feature.node.kubernetes.io/cpu-model.hypervisor
The node must report PR_SM in the CPU-MODEL.HYPERVISOR column.
Delete the test VM by running the following command:
$ oc delete vm test-vm-placement -n default