The telco core reference design specifications (RDS) configures an OKD cluster running on commodity hardware to host telco core workloads.
The telco core reference design specifications (RDS) describes a platform that supports large-scale telco applications, including control plane functions such as signaling and aggregation. It also includes some centralized data plane functions, such as user plane functions (UPF). These functions generally require scalability, complex networking support, resilient software-defined storage, and support performance requirements that are less stringent and constrained than far-edge deployments such as RAN.
The telco core cluster use model is designed for clusters that run on commodity hardware. Telco core clusters support large scale telco applications including control plane functions such as signaling, aggregation, and session border controller (SBC); and centralized data plane functions such as 5G user plane functions (UPF). Telco core cluster functions require scalability, complex networking support, resilient software-defined storage, and support performance requirements that are less stringent and constrained than far-edge RAN deployments.
Networking requirements for telco core functions vary widely across a range of networking features and performance points. IPv6 is a requirement and dual-stack is common. Some functions need maximum throughput and transaction rate and require support for user-plane DPDK networking. Other functions use more typical cloud-native patterns and can rely on OVN-Kubernetes, kernel networking, and load balancing.
Telco core clusters are configured as standard with three control plane and two or more worker nodes configured with the stock (non-RT) kernel.
In support of workloads with varying networking and performance requirements, you can segment worker nodes by using MachineConfigPool
custom resources (CR), for example, for non-user data plane or high-throughput use cases.
In support of required telco operational features, core clusters have a standard set of Day 2 OLM-managed Operators installed.
The telco core and telco RAN reference design specifications (RDS) capture the recommended, tested, and supported configurations to get reliable and repeatable performance for clusters running the telco core and telco RAN profiles.
Each RDS includes the released features and supported configurations that are engineered and validated for clusters to run the individual profiles. The configurations provide a baseline OKD installation that meets feature and KPI targets. Each RDS also describes expected variations for each individual configuration. Validation of each RDS includes many long duration and at-scale tests.
The validated reference configurations are updated for each major Y-stream release of OKD. Z-stream patch releases are periodically re-tested against the reference configurations. |
Deviating from the validated telco core and telco RAN DU reference design specifications (RDS) can have significant impact beyond the specific component or feature that you change. Deviations require analysis and engineering in the context of the complete solution.
All deviations from the RDS should be analyzed and documented with clear action tracking information. Due diligence is expected from partners to understand how to bring deviations into line with the reference design. This might require partners to provide additional resources to engage with Red Hat to work towards enabling their use case to achieve a best in class outcome with the platform. This is critical for the supportability of the solution and ensuring alignment across Red Hat and with partners. |
Deviation from the RDS can have some or all of the following consequences:
It can take longer to resolve issues.
There is a risk of missing project service-level agreements (SLAs), project deadlines, end provider performance requirements, and so on.
Unapproved deviations may require escalation at executive levels.
Red Hat prioritizes the servicing of requests for deviations based on partner engagement priorities. |
The following configurations and use models are applicable to all telco core use cases. The telco core use cases build on this common baseline of features.
Telco core clusters conform to the following requirements:
High availability control plane (three or more control plane nodes)
Non-schedulable control plane nodes
Multiple machine config pools
Telco core use cases require persistent storage as provided by Red Hat OpenShift Data Foundation.
Telco core cluster networking conforms to the following requirements:
Dual stack IPv4/IPv6 (IPv4 primary).
Fully disconnected – clusters do not have access to public networking at any point in their lifecycle.
Supports multiple networks. Segmented networking provides isolation between operations, administration and maintenance (OAM), signaling, and storage traffic.
Cluster network type is OVN-Kubernetes as required for IPv6 support.
Telco core clusters have multiple layers of networking supported by underlying RHCOS, SR-IOV Network Operator, Load Balancer and other components. These layers include the following:
Cluster networking layer. The cluster network configuration is defined and applied through the installation configuration. Update the configuration during Day 2 operations with the NMState Operator. Use the initial configuration to establish the following:
Host interface configuration.
Active/active bonding (LACP).
Secondary/additional network layer.
Configure the OKD CNI through network additionalNetwork
or NetworkAttachmentDefinition
CRs.
Use the initial configuration to configure MACVLAN virtual network interfaces.
Application workload layer. User plane networking runs in cloud-native network functions (CNFs).
Telco CNFs can use Service Mesh. All telco core clusters require a Service Mesh implementation. The choice of implementation and configuration is outside the scope of this specification.
Cluster workloads are detailed in "Application workloads".
Worker nodes should run on either of the following CPUs:
Intel 3rd Generation Xeon (IceLake) CPUs or better when supported by OKD, or CPUs with the silicon security bug (Spectre and similar) mitigations turned off. Skylake and older CPUs can experience 40% transaction performance drops when Spectre and similar mitigations are enabled.
AMD EPYC Zen 4 CPUs (Genoa, Bergamo, or newer) or better when supported by OKD.
Currently, per-pod power management is not available for AMD CPUs. |
IRQ balancing is enabled on worker nodes.
The PerformanceProfile
CR sets globallyDisableIrqLoadBalancing
to false.
Guaranteed QoS pods are annotated to ensure isolation as described in "CPU partitioning and performance tuning".
All cluster nodes should have the following features:
Have Hyper-Threading enabled
Have x86_64 CPU architecture
Have the stock (non-realtime) kernel enabled
Are not configured for workload partitioning
The balance between power management and maximum performance varies between machine config pools in the cluster. The following configurations should be consistent for all nodes in a machine config pools group.
Cluster scaling. See "Scalability" for more information.
Clusters should be able to scale to at least 120 nodes.
CPU partitioning is configured using a PerformanceProfile
CR and is applied to nodes on a per MachineConfigPool
basis.
See "CPU partitioning and performance tuning" for additional considerations.
CPU requirements for OKD depend on the configured feature set and application workload characteristics. For a cluster configured according to the reference configuration running a simulated workload of 3000 pods as created by the kube-burner node-density test, the following CPU requirements are validated:
The minimum number of reserved CPUs for control plane and worker nodes is 2 CPUs (4 hyper-threads) per NUMA node.
The NICs used for non-DPDK network traffic should be configured to use at least 16 RX/TX queues.
Nodes with large numbers of pods or other resources might require additional reserved CPUs. The remaining CPUs are available for user workloads.
Variations in OKD configuration, workload size, and workload characteristics require additional analysis to determine the effect on the number of required CPUs for the OpenShift platform. |
Application workloads running on telco core clusters can include a mix of high performance cloud-native network functions (CNFs) and traditional best-effort or burstable pod workloads.
Guaranteed QoS scheduling is available to pods that require exclusive or dedicated use of CPUs due to performance or security requirements. Typically, pods that run high performance or latency sensitive CNFs by using user plane networking (for example, DPDK) require exclusive use of dedicated whole CPUs achieved through node tuning and guaranteed QoS scheduling. When creating pod configurations that require exclusive CPUs, be aware of the potential implications of hyper-threaded systems. Pods should request multiples of 2 CPUs when the entire core (2 hyper-threads) must be allocated to the pod.
Pods running network functions that do not require high throughput or low latency networking should be scheduled with best-effort or burstable QoS pods and do not require dedicated or isolated CPU cores.
Use the following information to plan telco core workloads and cluster resources:
CNF applications should conform to the latest version of Red Hat Best Practices for Kubernetes.
Use a mix of best-effort and burstable QoS pods as required by your applications.
Use guaranteed QoS pods with proper configuration of reserved or isolated CPUs in the PerformanceProfile
CR that configures the node.
Guaranteed QoS Pods must include annotations for fully isolating CPUs.
Best effort and burstable pods are not guaranteed exclusive CPU use. Workloads can be preempted by other workloads, operating system daemons, or kernel tasks.
Use exec probes sparingly and only when no other suitable option is available.
Do not use exec probes if a CNF uses CPU pinning.
Use other probe implementations, for example, httpGet
or tcpSocket
.
When you need to use exec probes, limit the exec probe frequency and quantity. The maximum number of exec probes must be kept below 10, and the frequency must not be set to less than 10 seconds.
You can use startup probes, because they do not use significant resources at steady-state operation. This limitation on exec probes applies primarily to liveness and readiness probes. Exec probes cause much higher CPU usage on management cores compared to other probe types because they require process forking.
Signaling workloads typically use SCTP, REST, gRPC or similar TCP or UDP protocols. Signaling workloads support hundreds of thousands of transactions per second (TPS) by using a secondary multus CNI configured as MACVLAN or SR-IOV interface. These workloads can run in pods with either guaranteed or burstable QoS.
The following sections describe the various OKD components and configurations that you use to configure and deploy clusters to run telco core workloads.
No reference design updates in this release
CPU partitioning improves performance and reduces latency by separating sensitive workloads from general-purpose tasks, interrupts, and driver work queues. The CPUs allocated to those auxiliary processes are referred to as reserved in the following sections. In a system with Hyper-Threading enabled, a CPU is one hyper-thread.
The operating system needs a certain amount of CPU to perform all the support tasks, including kernel networking.
A system with just user plane networking applications (DPDK) needs at least one core (2 hyper-threads when enabled) reserved for the operating system and the infrastructure components.
In a system with Hyper-Threading enabled, core sibling threads must always be in the same pool of CPUs.
The set of reserved and isolated cores must include all CPU cores.
Core 0 of each NUMA node must be included in the reserved CPU set.
Low latency workloads require special configuration to avoid being affected by interrupts, kernel scheduler, or other parts of the platform. For more information, see "Creating a performance profile".
The minimum reserved capacity (systemReserved
) required can be found by following the guidance in the Which amount of CPU and memory are recommended to reserve for the system in OpenShift 4 nodes? Knowledgebase article.
The actual required reserved CPU capacity depends on the cluster configuration and workload attributes.
The reserved CPU value must be rounded up to a full core (2 hyper-threads) alignment.
Changes to CPU partitioning cause the nodes contained in the relevant machine config pool to be drained and rebooted.
The reserved CPUs reduce the pod density, because the reserved CPUs are removed from the allocatable capacity of the OKD node.
The real-time workload hint should be enabled for real-time capable workloads.
Applying the real time workloadHint
setting results in the nohz_full
kernel command line parameter being applied to improve performance of high performance applications.
When you apply the workloadHint
setting, any isolated or burstable pods that do not have the cpu-quota.crio.io: "disable"
annotation and a proper runtimeClassName
value, are subject to CRI-O rate limiting.
When you set the workloadHint
parameter, be aware of the tradeoff between increased performance and the potential impact of CRI-O rate limiting.
Ensure that required pods are correctly annotated.
Hardware without IRQ affinity support affects isolated CPUs. All server hardware must support IRQ affinity to ensure that pods with guaranteed CPU QoS can fully use allocated CPUs.
OVS dynamically manages its cpuset
entry to adapt to network traffic needs.
You do not need to reserve an additional CPU for handling high network throughput on the primary CNI.
If workloads running on the cluster use kernel level networking, the RX/TX queue count for the participating NICs should be set to 16 or 32 queues if the hardware permits it. Be aware of the default queue count. With no configuration, the default queue count is one RX/TX queue per online CPU; which can result in too many interrupts being allocated.
Some drivers do not deallocate the interrupts even after reducing the queue count. |
If workloads running on the cluster require cgroup v1, you can configure nodes to use cgroup v1 as part of the initial cluster deployment. See "Enabling Linux control group version 1 (cgroup v1)" and Red Hat Enterprise Linux 9 changes in the context of Red Hat OpenShift workloads.
Support for cgroup v1 is planned for removal in OKD 4.19. Clusters running cgroup v1 must transition to cgroup v2. |
Telco core cloud-native functions (CNFs) typically require a service mesh implementation. Specific service mesh features and performance requirements are dependent on the application. The selection of service mesh implementation and configuration is outside the scope of this documentation. You must account for the impact of service mesh on cluster resource usage and performance, including additional latency introduced in pod networking, in your implementation.
The following diagram describes the telco core reference design networking configuration.
Support for disabling vendor plugins in the SR-IOV Operator
Extended telco core RDS validation with MetalLB and EgressIP telco QE validation
FRR-K8s is now available under the Cluster Network Operator.
If you have custom |
The cluster is configured for dual-stack IP (IPv4 and IPv6).
The validated physical network configuration consists of two dual-port NICs. One NIC is shared among the primary CNI (OVN-Kubernetes) and IPVLAN and MACVLAN traffic, while the second one is dedicated to SR-IOV VF-based pod traffic.
A Linux bonding interface (bond0
) is created in active-active IEEE 802.3ad LACP mode with the two NIC ports attached.
The top-of-rack networking equipment must support and be configured for multi-chassis link aggregation (mLAG) technology.
VLAN interfaces are created on top of bond0
, including for the primary CNI.
Bond and VLAN interfaces are created at cluster install time during the network configuration stage of the installation.
Except for the vlan0
VLAN used by the primary CNI, all other VLANs can be created during Day 2 activities with the Kubernetes NMstate Operator.
MACVLAN and IPVLAN interfaces are created with their corresponding CNIs. They do not share the same base interface. For more information, see "Cluster Network Operator".
SR-IOV VFs are managed by the SR-IOV Network Operator.
To ensure consistent source IP addresses for pods behind a LoadBalancer Service, configure an EgressIP
CR and specify the podSelector
parameter.
You can implement service traffic separation by doing the following:
Configure VLAN interfaces and specific kernel IP routes on the nodes using NodeNetworkConfigurationPolicy
CRs.
Create a MetalLB BGPPeer
CR for each VLAN to establish peering with the remote BGP router.
Define a MetalLB BGPAdvertisement
CR to specify which IP address pools should be advertised to a selected list of BGPPeer
resources.
The following diagram illustrates how specific service IP addresses are advertised to the outside via specific VLAN interfaces.
Services routes are defined in BGPAdvertisement
CRs and configured with values for IPAddressPool1
and BGPPeer1
fields.
No reference design updates in this release
The Cluster Network Operator (CNO) deploys and manages the cluster network components including the default OVN-Kubernetes network plugin during cluster installation. The CNO allows for configuring primary interface MTU settings, OVN gateway modes to use node routing tables for pod egress, and additional secondary networks such as MACVLAN.
In support of network traffic separation, multiple network interfaces are configured through the CNO.
Traffic steering to these interfaces is configured through static routes applied by using the NMState Operator.
To ensure that pod traffic is properly routed, OVN-K is configured with the routingViaHost
option enabled.
This setting uses the kernel routing table and the applied static routes rather than OVN for pod egress traffic.
The Whereabouts CNI plugin is used to provide dynamic IPv4 and IPv6 addressing for additional pod network interfaces without the use of a DHCP server.
OVN-Kubernetes is required for IPv6 support.
Large MTU cluster support requires connected network equipment to be set to the same or larger value. MTU size up to 8900 is supported.
MACVLAN and IPVLAN cannot co-locate on the same main interface due to their reliance on the same underlying kernel mechanism, specifically the rx_handler
.
This handler allows a third-party module to process incoming packets before the host processes them, and only one such handler can be registered per network interface.
Since both MACVLAN and IPVLAN need to register their own rx_handler
to function, they conflict and cannot coexist on the same interface.
Review the source code for more details:
Alternative NIC configurations include splitting the shared NIC into multiple NICs or using a single dual-port NIC, though they have not been tested and validated.
Clusters with single-stack IP configuration are not validated.
The reachabilityTotalTimeoutSeconds
parameter in the Network
CR configures the EgressIP
node reachability check total timeout in seconds.
The recommended value is 1
second.
Pod egress traffic is handled by kernel routing table using the routingViaHost
option.
Appropriate static routes must be configured in the host.
FRR-K8s is now available under the Cluster Network Operator.
If you have custom |
MetalLB is a load-balancer implementation for bare metal Kubernetes clusters that uses standard routing protocols. It enables a Kubernetes service to get an external IP address which is also added to the host network for the cluster. The MetalLB Operator deploys and manages the lifecycle of a MetalLB instance in a cluster. Some use cases might require features not available in MetalLB, such as stateful load balancing. Where necessary, you can use an external third party load balancer. Selection and configuration of an external load balancer is outside the scope of this specification. When an external third-party load balancer is used, the integration effort must include enough analysis to ensure all performance and resource utilization requirements are met.
Stateful load balancing is not supported by MetalLB. An alternate load balancer implementation must be used if this is a requirement for workload CNFs.
You must ensure that the external IP address is routable from clients to the host network for the cluster.
MetalLB is used in BGP mode only for telco core use models.
For telco core use models, MetalLB is supported only with the OVN-Kubernetes network provider used in local gateway mode.
See routingViaHost
in "Cluster Network Operator".
BGP configuration in MetalLB is expected to vary depending on the requirements of the network and peers.
You can configure address pools with variations in addresses, aggregation length, auto assignment, and so on.
MetalLB uses BGP for announcing routes only.
Only the transmitInterval
and minimumTtl
parameters are relevant in this mode.
Other parameters in the BFD profile should remain close to the defaults as shorter values can lead to false negatives and affect performance.
You can now create virtual functions for Mellanox NICs with the SR-IOV Network Operator when secure boot is enabled in the cluster host. Before you can create the virtual functions, you must first skip the firmware configuration for the Mellanox NIC and manually allocate the number of virtual functions in the firmware before switching the system to secure boot.
SR-IOV enables physical functions (PFs) to be divided into multiple virtual functions (VFs). VFs can then be assigned to multiple pods to achieve higher throughput performance while keeping the pods isolated. The SR-IOV Network Operator provisions and manages SR-IOV CNI, network device plugin, and other components of the SR-IOV stack.
Only certain network interfaces are supported. See "Supported devices" for more information.
Enabling SR-IOV and IOMMU: the SR-IOV Network Operator automatically enables IOMMU on the kernel command line.
SR-IOV VFs do not receive link state updates from the PF. If a link down detection is required, it must be done at the protocol level.
MultiNetworkPolicy
CRs can be applied to netdevice
networks only.
This is because the implementation uses iptables, which cannot manage vfio interfaces.
SR-IOV interfaces in vfio
mode are typically used to enable additional secondary networks for applications that require high throughput or low latency.
The SriovOperatorConfig
CR must be explicitly created.
This CR is included in the reference configuration policies, which causes it to be created during initial deployment.
NICs that do not support firmware updates with UEFI secure boot or kernel lockdown must be preconfigured with sufficient virtual functions (VFs) enabled to support the number of VFs required by the application workload. For Mellanox NICs, you must disable the Mellanox vendor plugin in the SR-IOV Network Operator. See "Configuring an SR-IOV network device" for more information.
To change the MTU value of a VF after the pod has started, do not configure the SriovNetworkNodePolicy
MTU field.
Instead, use the Kubernetes NMState Operator to set the MTU of the related PF.
No reference design updates in this release
The Kubernetes NMState Operator provides a Kubernetes API for performing state-driven network configuration across cluster nodes. It enables network interface configurations, static IPs and DNS, VLANs, trunks, bonding, static routes, MTU, and enabling promiscuous mode on the secondary interfaces. The cluster nodes periodically report on the state of each node’s network interfaces to the API server.
Not applicable
Initial networking configuration is applied using NMStateConfig
content in the installation CRs.
The NMState Operator is used only when required for network updates.
When SR-IOV virtual functions are used for host networking, the NMState Operator (via nodeNetworkConfigurationPolicy
CRs) is used to configure VF interfaces, such as VLANs and MTU.
No reference design updates in this release
The Cluster Logging Operator enables collection and shipping of logs off the node for remote archival and analysis. The reference configuration uses Kafka to ship audit and infrastructure logs to a remote archive.
Not applicable
The impact of cluster CPU use is based on the number or size of logs generated and the amount of log filtering configured.
The reference configuration does not include shipping of application logs. The inclusion of application logs in the configuration requires you to evaluate the application logging rate and have sufficient additional CPU resources allocated to the reserved set.
No reference design updates in this release
Use the Performance Profile to configure clusters with high power mode, low power mode, or mixed mode. The choice of power mode depends on the characteristics of the workloads running on the cluster, particularly how sensitive they are to latency. Configure the maximum latency for a low-latency pod by using the per-pod power management C-states feature.
Power configuration relies on appropriate BIOS configuration, for example, enabling C-states and P-states. Configuration varies between hardware vendors.
Latency: To ensure that latency-sensitive workloads meet requirements, you require a high-power or a per-pod power management configuration. Per-pod power management is only available for Guaranteed QoS pods with dedicated pinned CPUs.
No reference design updates in this release
Cloud native storage services can be provided by Red Hat OpenShift Data Foundation or other third-party solutions.
OpenShift Data Foundation is a Ceph-based software-defined storage solution for containers. It provides block storage, file system storage, and on-premise object storage, which can be dynamically provisioned for both persistent and non-persistent data requirements. Telco core applications require persistent storage.
All storage data might not be encrypted in flight. To reduce risk, isolate the storage network from other cluster networks. The storage network must not be reachable, or routable, from other cluster networks. Only nodes directly attached to the storage network should be allowed to gain access to it. |
No reference design updates in this release
Red Hat OpenShift Data Foundation is a software-defined storage service for containers. For telco core clusters, storage support is provided by OpenShift Data Foundation storage services running externally to the application workload cluster. OpenShift Data Foundation supports separation of storage traffic using secondary CNI networks.
In an IPv4/IPv6 dual-stack networking environment, OpenShift Data Foundation uses IPv4 addressing. For more information, see Network requirements.
OpenShift Data Foundation network traffic should be isolated from other traffic on a dedicated network, for example, by using VLAN isolation.
You can use other storage solutions to provide persistent storage for telco core clusters. The configuration and integration of these solutions is outside the scope of the reference design specifications (RDS).
Integration of the storage solution into the telco core cluster must include proper sizing and performance analysis to ensure the storage meets overall performance and resource usage requirements.
The following sections describe the various OKD components and configurations that you use to configure the hub cluster with Red Hat Advanced Cluster Management (RHACM).
No reference design updates in this release
Red Hat Advanced Cluster Management (RHACM) provides Multi Cluster Engine (MCE) installation and ongoing GitOps ZTP lifecycle management for deployed clusters.
You manage cluster configuration and upgrades declaratively by applying Policy
custom resources (CRs) to clusters during maintenance windows.
You apply policies with the RHACM policy controller as managed by Topology Aware Lifecycle Manager. Configuration, upgrades, and cluster status are managed through the policy controller.
When installing managed clusters, RHACM applies labels and initial ignition configuration to individual nodes in support of custom disk partitioning, allocation of roles, and allocation to machine config pools.
You define these configurations with SiteConfig
or ClusterInstance
CRs.
Hub cluster sizing is discussed in Sizing your cluster.
RHACM scaling limits are described in Performance and Scalability.
When managing multiple clusters with unique content per installation, site, or deployment, using RHACM hub templating is strongly recommended. RHACM hub templating allows you to apply a consistent set of policies to clusters while providing for unique values per installation.
No reference design updates in this release.
Topology Aware Lifecycle Manager is an Operator which runs only on the hub cluster. TALM manages how changes including cluster and Operator upgrades, configurations, and so on, are rolled out to managed clusters in the network. TALM has the following core features:
Provides sequenced updates of cluster configurations and upgrades (OKD and Operators) as defined by cluster policies.
Provides for deferred application of cluster updates.
Supports progressive rollout of policy updates to sets of clusters in user configurable batches.
Allows for per-cluster actions by adding ztp-done
or similar user-defined labels to clusters.
Supports concurrent cluster deployments in batches of 400.
Only policies with the ran.openshift.io/ztp-deploy-wave
annotation are applied by TALM during initial cluster installation.
Any policy can be remediated by TALM under control of a user created ClusterGroupUpgrade
CR.
No reference design updates in this release
The GitOps Operator provides a GitOps driven infrastructure for managing cluster deployment and configuration. Cluster definitions and configuration are maintained in a Git repository.
ZTP plugins provide support for generating Installation
CRs from SiteConfig
CRs and automatically wrapping configuration CRs in policies based on RHACM PolicyGenerator
CRs.
The SiteConfig Operator provides improved support for generation of Installation
CRs from ClusterInstance
CRs.
Where possible, use |
You should structure the Git repository according to release version, with all necessary artifacts (SiteConfig
, ClusterInstance
, PolicyGenerator
, and PolicyGenTemplate
, and supporting reference CRs) included.
This enables deploying and managing multiple versions of the OpenShift platform and configuration versions to clusters simultaneously and through upgrades.
The recommended Git structure keeps reference CRs in a directory separate from customer or partner provided content. This means that you can import reference updates by simply overwriting existing content. Customer or partner-supplied CRs can be provided in a parallel directory to the reference CRs for easy inclusion in the generated configuration policies.
Each ArgoCD application supports up to 300 nodes. Multiple ArgoCD applications can be used to achieve the maximum number of clusters supported by a single hub cluster.
The SiteConfig
CR must use the extraManifests.searchPaths
field to reference the reference manifests.
Since OKD 4.15, the |
Set the MachineConfigPool
(mcp
) CR paused
field to true during a cluster upgrade maintenance window and set the maxUnavailable
field to the maximum tolerable value.
This prevents multiple cluster node reboots during upgrade, which results in a shorter overall upgrade.
When you unpause the mcp
CR, all the configuration changes are applied with a single reboot.
During installation, custom |
To avoid confusion or unintentional overwriting when updating content, you should use unique and distinguishable names for custom CRs in the reference-crs/
directory under core-overlay and extra manifests in Git.
The SiteConfig
CR allows multiple extra-manifest paths.
When file names overlap in multiple directory paths, the last file found in the directory order list takes precedence.
No reference design updates in this release
The Cluster Monitoring Operator (CMO) is included by default in OKD and provides monitoring (metrics, dashboards, and alerting) for the platform components and optionally user projects. You can customize the default log retention period, custom alert rules, and so on. The default handling of pod CPU and memory metrics, based on upstream Kubernetes and cAdvisor, makes a tradeoff favoring stale data over metric accuracy. This leads to spikes in reporting, which can create false alerts, depending on the user-specified thresholds. OKD supports an opt-in Dedicated Service Monitor feature that creates an additional set of pod CPU and memory metrics that do not suffer from this behavior. For more information, see Dedicated Service Monitors - Questions and Answers (Red Hat Knowledgebase).
In addition to the default configuration, the following metrics are expected to be configured for telco core clusters:
Pod CPU and memory metrics and alerts for user workloads
You must enable the Dedicated Service Monitor feature to represent pod metrics accurately.
The Prometheus retention period is specified by the user. The value used is a tradeoff between operational requirements for maintaining historical data on the cluster against CPU and storage resources. Longer retention periods increase the need for storage and require additional CPU to manage data indexing.
No reference design updates in this release
The scheduler is a cluster-wide component responsible for selecting the correct node for a given workload. It is a core part of the platform and does not require any specific configuration in the common deployment scenarios. However, a few specific use cases are described in the following section.
NUMA-aware scheduling can be enabled through the NUMA Resources Operator. For more information, see "Scheduling NUMA-aware workloads".
The default scheduler does not understand the NUMA locality of workloads.
It only knows about the sum of all free resources on a worker node.
This might cause workloads to be rejected when scheduled to a node with the topology manager policy set to single-numa-node
or restricted
.
For more information, see "Topology Manager policies".
For example, consider a pod requesting 6 CPUs that is scheduled to an empty node that has 4 CPUs per NUMA node. The total allocatable capacity of the node is 8 CPUs. The scheduler places the pod on the empty node. The node local admission fails, as there are only 4 CPUs available in each of the NUMA nodes.
All clusters with multi-NUMA nodes are required to use the NUMA Resources Operator.
See "Installing the NUMA Resources Operator" for more information.
Use the machineConfigPoolSelector
field in the KubeletConfig
CR to select all nodes where NUMA aligned scheduling is required.
All machine config pools must have consistent hardware configuration. For example, all nodes are expected to have the same NUMA zone count.
Pods might require annotations for correct scheduling and isolation. For more information about annotations, see "CPU partitioning and performance tuning".
You can configure SR-IOV virtual function NUMA affinity to be ignored during scheduling by using the excludeTopology
field in SriovNetworkNodePolicy
CR.
No reference design updates in this release
Analyze additional kernel modules to determine impact on CPU load, system performance, and ability to meet KPIs.
Feature | Description |
---|---|
Additional kernel modules |
Install the following kernel modules by using
|
Container mount namespace hiding |
Reduce the frequency of kubelet housekeeping and eviction monitoring to reduce CPU usage. Creates a container mount namespace, visible to kubelet/CRI-O, to reduce system mount scanning overhead. |
Kdump enable |
Optional configuration (enabled by default) |
No reference design updates in this release
Enabling secure boot is the recommended configuration.
When secure boot is enabled, only signed kernel modules are loaded by the kernel. Out-of-tree drivers are not supported. |
No reference design updates in this release
Telco core clusters are expected to be installed in networks without direct access to the internet. All container images needed to install, configure, and operate the cluster must be available in a disconnected registry. This includes OKD images, Day 2 OLM Operator images, and application workload images. The use of a disconnected environment provides multiple benefits, including:
Security - limiting access to the cluster
Curated content – the registry is populated based on curated and approved updates for clusters
A unique name is required for all custom CatalogSource
resources.
Do not reuse the default catalog names.
A valid time source must be configured as part of cluster installation
No reference design updates in this release
Telco core clusters can be installed by using the Agent-based Installer. This method allows you to install OpenShift on bare-metal servers without requiring additional servers or VMs for managing the installation. The Agent-based Installer can be run on any system (for example, from a laptop) to generate an ISO installation image. The ISO is used as the installation media for the cluster supervisor nodes. Installation progress can be monitored using the ABI tool from any system with network connectivity to the supervisor node’s API interfaces.
ABI supports the following:
Installation from declarative CRs
Installation in disconnected environments
Installation with no additional supporting install or bastion servers required to complete the installation
Disconnected installation requires a registry that is reachable from the installed host, with all required content mirrored in that registry.
Networking configuration should be applied as NMState configuration during installation. Day 2 networking configuration using the NMState Operator is not supported.
Telco customers are security conscious and require clusters to be hardened against multiple attack vectors. In OKD, there is no single component or feature responsible for securing a cluster. Use the following security-oriented features and configurations to secure your clusters:
SecurityContextConstraints (scc): All workload pods should be run with restricted-v2
or restricted
scc.
Seccomp: All pods should run with the RuntimeDefault
(or stronger) seccomp profile.
Rootless DPDK pods: Many user-plane networking (DPDK) CNFs require pods to run with root privileges. With this feature, a conformant DPDK pod can run without requiring root privileges. Rootless DPDK pods create a tap device in a rootless pod that injects traffic from a DPDK application to the kernel.
Storage: The storage network should be isolated and non-routable to other cluster networks. See the "Storage" section for additional details.
Refer to Custom nftable firewall rules in OpenShift for a supported method of implementing custom nftables firewall rules in OpenShift cluster nodes. This article is intended for cluster administrators who are responsible for managing network security policies in OpenShift environments. It is crucial to carefully consider the operational implications before deploying this method, including:
Early application: The rules are applied at boot time, before the network is fully operational. Ensure the rules don’t inadvertently block essential services required during the boot process.
Risk of misconfiguration: Errors in your custom rules can lead to unintended consequences, potentially leading to performance impact or blocking legitimate traffic or isolating nodes. Thoroughly test your rules in a non-production environment before deploying them to your main cluster.
External endpoints: OpenShift requires access to external endpoints to function. For more information about the firewall allowlist, see "Configuring your firewall for OKD". Ensure that cluster nodes are permitted access to those endpoints.
Node reboot: Unless node disruption policies are configured, applying the MachineConfig
CR with the required firewall settings causes a node reboot.
Be aware of this impact and schedule a maintenance window accordingly.
For more information, see "Using node disruption policies to minimize disruption from machine config changes".
Node disruption policies are available in OKD 4.17 and later. |
Network flow matrix: For more information about managing ingress traffic, see "OKD network flow matrix". You can restrict ingress traffic to essential flows to improve network security. The matrix provides insights into base cluster services but excludes traffic generated by Day-2 Operators.
Cluster version updates and upgrades: Exercise caution when updating or upgrading OpenShift clusters. Recent changes to the platform’s firewall requirements might require adjustments to network port permissions. Although the documentation provides guidelines, note that these requirements can evolve over time. To minimize disruptions, you should test any updates or upgrades in a staging environment before applying them in production. This helps you to identify and address potential compatibility issues related to firewall configuration changes.
Rootless DPDK pods requires the following additional configuration:
Configure the container_t
SELinux context for the tap plugin.
Enable the container_use_devices
SELinux boolean for the cluster host.
For rootless DPDK pod support, enable the SELinux container_use_devices
boolean on the host to allow the tap device to be created.
This introduces an acceptable security risk.
Use the following custom resources (CRs) to configure and deploy OKD clusters with the telco core profile. Use the CRs to form the common baseline used in all the specific use models unless otherwise indicated.
You can extract the complete set of custom resources (CRs) for the telco core profile from the telco-core-rds-rhel9
container image.
The container image has both the required CRs, and the optional CRs, for the telco core profile.
You have installed podman
.
Extract the content from the telco-core-rds-rhel9
container image by running the following commands:
$ mkdir -p ./out
$ podman run -it registry.redhat.io/openshift4/openshift-telco-core-rds-rhel9:v4.18 | base64 -d | tar xv -C out
The out
directory has the following directory structure. You can view the telco core CRs in the out/telco-core-rds/
directory.
out/
└── telco-core-rds
├── configuration
│ └── reference-crs
│ ├── optional
│ │ ├── logging
│ │ ├── networking
│ │ │ └── multus
│ │ │ └── tap_cni
│ │ ├── other
│ │ └── tuning
│ └── required
│ ├── networking
│ │ ├── metallb
│ │ ├── multinetworkpolicy
│ │ └── sriov
│ ├── other
│ ├── performance
│ ├── scheduling
│ └── storage
│ └── odf-external
└── install
You have access to the cluster as a user with the cluster-admin
role.
You have credentials to access the registry.redhat.io
container image registry.
You installed the cluster-compare
plugin.
Login to the container image registry with your credentials by running the following command:
$ podman login registry.redhat.io
Component | Reference CR | Description | Optional |
---|---|---|---|
Additional kernel modules |
|
Optional. Configures the kernel modules for control plane nodes. |
No |
Additional kernel modules |
|
Optional. Loads the SCTP kernel module in worker nodes. |
No |
Additional kernel modules |
|
Optional. Configures kernel modules for worker nodes. |
No |
Container mount namespace hiding |
|
Configures a mount namespace for sharing container-specific mounts between kubelet and CRI-O on control plane nodes. |
No |
Container mount namespace hiding |
|
Configures a mount namespace for sharing container-specific mounts between kubelet and CRI-O on worker nodes. |
No |
Kdump enable |
|
Configures kdump crash reporting on master nodes. |
No |
Kdump enable |
|
Configures kdump crash reporting on worker nodes. |
No |
Component | Reference CR | Description | Optional |
---|---|---|---|
System reserved capacity |
|
Optional. Configures kubelet, enabling auto-sizing reserved resources for the control plane node pool. |
No |
Component | Reference CR | Description | Optional |
---|---|---|---|
Baseline |
|
Configures the default cluster network, specifying OVN Kubernetes settings like routing via the host. It also allows the definition of additional networks, including custom CNI configurations, and enables the use of MultiNetworkPolicy CRs for network policies across multiple networks. |
No |
Baseline |
|
Optional. Defines a NetworkAttachmentDefinition resource specifying network configuration details such as node selector and CNI configuration. |
Yes |
Load Balancer |
|
Configures MetalLB to manage a pool of IP addresses with auto-assign enabled for dynamic allocation of IPs from the specified range. |
No |
Load Balancer |
|
Configures bidirectional forwarding detection (BFD) with customized intervals, detection multiplier, and modes for quicker network fault detection and load balancing failover. |
No |
Load Balancer |
|
Defines a BGP advertisement resource for MetalLB, specifying how an IP address pool is advertised to BGP peers. This enables fine-grained control over traffic routing and announcements. |
No |
Load Balancer |
|
Defines a BGP peer in MetalLB, representing a BGP neighbor for dynamic routing. |
No |
Load Balancer |
|
Defines a MetalLB community, which groups one or more BGP communities under a named resource. Communities can be applied to BGP advertisements to control routing policies and change traffic routing. |
No |
Load Balancer |
|
Defines the MetalLB resource in the cluster. |
No |
Load Balancer |
|
Defines the metallb-system namespace in the cluster. |
No |
Load Balancer |
|
Defines the Operator group for the MetalLB Operator. |
No |
Load Balancer |
|
Creates a subscription resource for the metallb Operator with manual approval for install plans. |
No |
Multus - Tap CNI for rootless DPDK pods |
|
Configures a MachineConfig resource which sets an SELinux boolean for the tap CNI plugin on worker nodes. |
Yes |
NMState Operator |
|
Defines an NMState resource that is used by the NMState Operator to manage node network configurations. |
No |
NMState Operator |
|
Creates the NMState Operator namespace. |
No |
NMState Operator |
|
Creates the Operator group in the openshift-nmstate namespace, allowing the NMState Operator to watch and manage resources. |
No |
NMState Operator |
|
Creates a subscription for the NMState Operator, managed through OLM. |
No |
SR-IOV Network Operator |
|
Defines an SR-IOV network specifying network capabilities, IP address management (ipam), and the associated network namespace and resource. |
No |
SR-IOV Network Operator |
|
Configures network policies for SR-IOV devices on specific nodes, including customization of device selection, VF allocation (numVfs), node-specific settings (nodeSelector), and priorities. |
No |
SR-IOV Network Operator |
|
Configures various settings for the SR-IOV Operator, including enabling the injector and Operator webhook, disabling pod draining, and defining the node selector for the configuration daemon. |
No |
SR-IOV Network Operator |
|
Creates a subscription for the SR-IOV Network Operator, managed through OLM. |
No |
SR-IOV Network Operator |
|
Creates the SR-IOV Network Operator subscription namespace. |
No |
SR-IOV Network Operator |
|
Creates the Operator group for the SR-IOV Network Operator, allowing it to watch and manage resources in the target namespace. |
No |
Component | Reference CR | Description | Optional |
---|---|---|---|
NUMA-aware scheduler |
|
Enables the NUMA Resources Operator, aligning workloads with specific NUMA node configurations. Required for clusters with multi-NUMA nodes. |
No |
NUMA-aware scheduler |
|
Creates a subscription for the NUMA Resources Operator, managed through OLM. Required for clusters with multi-NUMA nodes. |
No |
NUMA-aware scheduler |
|
Creates the NUMA Resources Operator subscription namespace. Required for clusters with multi-NUMA nodes. |
No |
NUMA-aware scheduler |
|
Creates the Operator group in the numaresources-operator namespace, allowing the NUMA Resources Operator to watch and manage resources. Required for clusters with multi-NUMA nodes. |
No |
NUMA-aware scheduler |
|
Configures a topology-aware scheduler in the cluster that can handle NUMA aware scheduling of pods across nodes. |
No |
NUMA-aware scheduler |
|
Configures control plane nodes as non-schedulable for workloads. |
No |
Component | Reference CR | Description | Optional |
---|---|---|---|
External ODF configuration |
|
Defines a Secret resource containing base64-encoded configuration data for an external Ceph cluster in the openshift-storage namespace. |
No |
External ODF configuration |
|
Defines an OpenShift Container Storage (OCS) storage resource which configures the cluster to use an external storage back end. |
No |
External ODF configuration |
|
Creates the monitored openshift-storage namespace for the OpenShift Data Foundation Operator. |
No |
External ODF configuration |
|
Creates the Operator group in the openshift-storage namespace, allowing the OpenShift Data Foundation Operator to watch and manage resources. |
No |
External ODF configuration |
|
Creates the subscription for the OpenShift Data Foundation Operator in the openshift-storage namespace. |
No |
The Red Hat telco core 4 solution has been validated using the following Red Hat software products for OKD clusters.
Component | Software version |
---|---|
Red Hat Advanced Cluster Management (RHACM) |
2.121 |
Cluster Logging Operator |
6.12 |
OpenShift Data Foundation |
4.18 |
SR-IOV Network Operator |
4.18 |
MetalLB |
4.18 |
NMState Operator |
4.18 |
NUMA-aware scheduler |
4.18 |
[1] This table will be updated when the aligned RHACM version 2.13 is released.
[2] This table will be updated when the aligned Cluster Logging Operator 6.2 is released.