This is a cache of https://docs.okd.io/latest/scalability_and_performance/telco_reference_design_specs/telco-core-rds.html. It is a snapshot of the page at 2025-02-25T19:13:27.138+0000.
Telco core reference design specifications - Telco reference designs | Scalability and performance | OKD 4
  1. Documentation
×

Telco core reference design specifications

Telco core RDS 4 use model overview

The telco core reference design specifications (RDS) describes a platform that supports large-scale telco applications, including control plane functions such as signaling and aggregation. It also includes some centralized data plane functions, such as user plane functions (UPF). These functions generally require scalability, complex networking support, resilient software-defined storage, and support performance requirements that are less stringent and constrained than far-edge deployments such as RAN.

About the telco core cluster use model

The telco core cluster use model is designed for clusters that run on commodity hardware. Telco core clusters support large scale telco applications including control plane functions such as signaling, aggregation, and session border controller (SBC); and centralized data plane functions such as 5G user plane functions (UPF). Telco core cluster functions require scalability, complex networking support, resilient software-defined storage, and support performance requirements that are less stringent and constrained than far-edge RAN deployments.

5G core cluster showing a service-based architecture with overlaid networking topology
Figure 1. Telco core RDS cluster service-based architecture and networking topology

Networking requirements for telco core functions vary widely across a range of networking features and performance points. IPv6 is a requirement and dual-stack is common. Some functions need maximum throughput and transaction rate and require support for user-plane DPDK networking. Other functions use more typical cloud-native patterns and can rely on OVN-Kubernetes, kernel networking, and load balancing.

Telco core clusters are configured as standard with three control plane and two or more worker nodes configured with the stock (non-RT) kernel. In support of workloads with varying networking and performance requirements, you can segment worker nodes by using MachineConfigPool custom resources (CR), for example, for non-user data plane or high-throughput use cases. In support of required telco operational features, core clusters have a standard set of Day 2 OLM-managed Operators installed.

Reference design scope

The telco core and telco RAN reference design specifications (RDS) capture the recommended, tested, and supported configurations to get reliable and repeatable performance for clusters running the telco core and telco RAN profiles.

Each RDS includes the released features and supported configurations that are engineered and validated for clusters to run the individual profiles. The configurations provide a baseline OKD installation that meets feature and KPI targets. Each RDS also describes expected variations for each individual configuration. Validation of each RDS includes many long duration and at-scale tests.

The validated reference configurations are updated for each major Y-stream release of OKD. Z-stream patch releases are periodically re-tested against the reference configurations.

Deviations from the reference design

Deviating from the validated telco core and telco RAN DU reference design specifications (RDS) can have significant impact beyond the specific component or feature that you change. Deviations require analysis and engineering in the context of the complete solution.

All deviations from the RDS should be analyzed and documented with clear action tracking information. Due diligence is expected from partners to understand how to bring deviations into line with the reference design. This might require partners to provide additional resources to engage with Red Hat to work towards enabling their use case to achieve a best in class outcome with the platform. This is critical for the supportability of the solution and ensuring alignment across Red Hat and with partners.

Deviation from the RDS can have some or all of the following consequences:

  • It can take longer to resolve issues.

  • There is a risk of missing project service-level agreements (SLAs), project deadlines, end provider performance requirements, and so on.

  • Unapproved deviations may require escalation at executive levels.

    Red Hat prioritizes the servicing of requests for deviations based on partner engagement priorities.

Telco core common baseline model

The following configurations and use models are applicable to all telco core use cases. The telco core use cases build on this common baseline of features.

Cluster topology

Telco core clusters conform to the following requirements:

  • High availability control plane (three or more control plane nodes)

  • Non-schedulable control plane nodes

  • Multiple machine config pools

Storage

Telco core use cases require persistent storage as provided by Red Hat OpenShift Data Foundation.

Networking

Telco core cluster networking conforms to the following requirements:

  • Dual stack IPv4/IPv6 (IPv4 primary).

  • Fully disconnected – clusters do not have access to public networking at any point in their lifecycle.

  • Supports multiple networks. Segmented networking provides isolation between operations, administration and maintenance (OAM), signaling, and storage traffic.

  • Cluster network type is OVN-Kubernetes as required for IPv6 support.

  • Telco core clusters have multiple layers of networking supported by underlying RHCOS, SR-IOV Network Operator, Load Balancer and other components. These layers include the following:

    • Cluster networking layer. The cluster network configuration is defined and applied through the installation configuration. Update the configuration during Day 2 operations with the NMState Operator. Use the initial configuration to establish the following:

      • Host interface configuration.

      • Active/active bonding (LACP).

    • Secondary/additional network layer. Configure the OKD CNI through network additionalNetwork or NetworkAttachmentDefinition CRs. Use the initial configuration to configure MACVLAN virtual network interfaces.

    • Application workload layer. User plane networking runs in cloud-native network functions (CNFs).

Service Mesh

Telco CNFs can use Service Mesh. All telco core clusters require a Service Mesh implementation. The choice of implementation and configuration is outside the scope of this specification.

Telco core cluster common use model engineering considerations

  • Cluster workloads are detailed in "Application workloads".

  • Worker nodes should run on either of the following CPUs:

    • Intel 3rd Generation Xeon (IceLake) CPUs or better when supported by OKD, or CPUs with the silicon security bug (Spectre and similar) mitigations turned off. Skylake and older CPUs can experience 40% transaction performance drops when Spectre and similar mitigations are enabled.

    • AMD EPYC Zen 4 CPUs (Genoa, Bergamo, or newer) or better when supported by OKD.

      Currently, per-pod power management is not available for AMD CPUs.

    • IRQ balancing is enabled on worker nodes. The PerformanceProfile CR sets globallyDisableIrqLoadBalancing to false. Guaranteed QoS pods are annotated to ensure isolation as described in "CPU partitioning and performance tuning".

  • All cluster nodes should have the following features:

    • Have Hyper-Threading enabled

    • Have x86_64 CPU architecture

    • Have the stock (non-realtime) kernel enabled

    • Are not configured for workload partitioning

  • The balance between power management and maximum performance varies between machine config pools in the cluster. The following configurations should be consistent for all nodes in a machine config pools group.

    • Cluster scaling. See "Scalability" for more information.

    • Clusters should be able to scale to at least 120 nodes.

  • CPU partitioning is configured using a PerformanceProfile CR and is applied to nodes on a per MachineConfigPool basis. See "CPU partitioning and performance tuning" for additional considerations.

  • CPU requirements for OKD depend on the configured feature set and application workload characteristics. For a cluster configured according to the reference configuration running a simulated workload of 3000 pods as created by the kube-burner node-density test, the following CPU requirements are validated:

    • The minimum number of reserved CPUs for control plane and worker nodes is 2 CPUs (4 hyper-threads) per NUMA node.

    • The NICs used for non-DPDK network traffic should be configured to use at least 16 RX/TX queues.

    • Nodes with large numbers of pods or other resources might require additional reserved CPUs. The remaining CPUs are available for user workloads.

    Variations in OKD configuration, workload size, and workload characteristics require additional analysis to determine the effect on the number of required CPUs for the OpenShift platform.

Application workloads

Application workloads running on telco core clusters can include a mix of high performance cloud-native network functions (CNFs) and traditional best-effort or burstable pod workloads.

Guaranteed QoS scheduling is available to pods that require exclusive or dedicated use of CPUs due to performance or security requirements. Typically, pods that run high performance or latency sensitive CNFs by using user plane networking (for example, DPDK) require exclusive use of dedicated whole CPUs achieved through node tuning and guaranteed QoS scheduling. When creating pod configurations that require exclusive CPUs, be aware of the potential implications of hyper-threaded systems. Pods should request multiples of 2 CPUs when the entire core (2 hyper-threads) must be allocated to the pod.

Pods running network functions that do not require high throughput or low latency networking should be scheduled with best-effort or burstable QoS pods and do not require dedicated or isolated CPU cores.

Engineering considerations

Use the following information to plan telco core workloads and cluster resources:

  • CNF applications should conform to the latest version of Red Hat Best Practices for Kubernetes.

  • Use a mix of best-effort and burstable QoS pods as required by your applications.

    • Use guaranteed QoS pods with proper configuration of reserved or isolated CPUs in the PerformanceProfile CR that configures the node.

    • Guaranteed QoS Pods must include annotations for fully isolating CPUs.

    • Best effort and burstable pods are not guaranteed exclusive CPU use. Workloads can be preempted by other workloads, operating system daemons, or kernel tasks.

  • Use exec probes sparingly and only when no other suitable option is available.

    • Do not use exec probes if a CNF uses CPU pinning. Use other probe implementations, for example, httpGet or tcpSocket.

    • When you need to use exec probes, limit the exec probe frequency and quantity. The maximum number of exec probes must be kept below 10, and the frequency must not be set to less than 10 seconds.

    • You can use startup probes, because they do not use significant resources at steady-state operation. This limitation on exec probes applies primarily to liveness and readiness probes. Exec probes cause much higher CPU usage on management cores compared to other probe types because they require process forking.

Signaling workloads

Signaling workloads typically use SCTP, REST, gRPC or similar TCP or UDP protocols. Signaling workloads support hundreds of thousands of transactions per second (TPS) by using a secondary multus CNI configured as MACVLAN or SR-IOV interface. These workloads can run in pods with either guaranteed or burstable QoS.

Telco core RDS components

The following sections describe the various OKD components and configurations that you use to configure and deploy clusters to run telco core workloads.

CPU partitioning and performance tuning

New in this release

  • No reference design updates in this release

Description

CPU partitioning improves performance and reduces latency by separating sensitive workloads from general-purpose tasks, interrupts, and driver work queues. The CPUs allocated to those auxiliary processes are referred to as reserved in the following sections. In a system with Hyper-Threading enabled, a CPU is one hyper-thread.

Limits and requirements

  • The operating system needs a certain amount of CPU to perform all the support tasks, including kernel networking.

    • A system with just user plane networking applications (DPDK) needs at least one core (2 hyper-threads when enabled) reserved for the operating system and the infrastructure components.

  • In a system with Hyper-Threading enabled, core sibling threads must always be in the same pool of CPUs.

  • The set of reserved and isolated cores must include all CPU cores.

  • Core 0 of each NUMA node must be included in the reserved CPU set.

  • Low latency workloads require special configuration to avoid being affected by interrupts, kernel scheduler, or other parts of the platform. For more information, see "Creating a performance profile".

Engineering considerations

  • The minimum reserved capacity (systemReserved) required can be found by following the guidance in the Which amount of CPU and memory are recommended to reserve for the system in OpenShift 4 nodes? Knowledgebase article.

  • The actual required reserved CPU capacity depends on the cluster configuration and workload attributes.

  • The reserved CPU value must be rounded up to a full core (2 hyper-threads) alignment.

  • Changes to CPU partitioning cause the nodes contained in the relevant machine config pool to be drained and rebooted.

  • The reserved CPUs reduce the pod density, because the reserved CPUs are removed from the allocatable capacity of the OKD node.

  • The real-time workload hint should be enabled for real-time capable workloads.

    • Applying the real time workloadHint setting results in the nohz_full kernel command line parameter being applied to improve performance of high performance applications. When you apply the workloadHint setting, any isolated or burstable pods that do not have the cpu-quota.crio.io: "disable" annotation and a proper runtimeClassName value, are subject to CRI-O rate limiting. When you set the workloadHint parameter, be aware of the tradeoff between increased performance and the potential impact of CRI-O rate limiting. Ensure that required pods are correctly annotated.

  • Hardware without IRQ affinity support affects isolated CPUs. All server hardware must support IRQ affinity to ensure that pods with guaranteed CPU QoS can fully use allocated CPUs.

  • OVS dynamically manages its cpuset entry to adapt to network traffic needs. You do not need to reserve an additional CPU for handling high network throughput on the primary CNI.

  • If workloads running on the cluster use kernel level networking, the RX/TX queue count for the participating NICs should be set to 16 or 32 queues if the hardware permits it. Be aware of the default queue count. With no configuration, the default queue count is one RX/TX queue per online CPU; which can result in too many interrupts being allocated.

    Some drivers do not deallocate the interrupts even after reducing the queue count.

  • If workloads running on the cluster require cgroup v1, you can configure nodes to use cgroup v1 as part of the initial cluster deployment. See "Enabling Linux control group version 1 (cgroup v1)" and Red Hat Enterprise Linux 9 changes in the context of Red Hat OpenShift workloads.

    Support for cgroup v1 is planned for removal in OKD 4.19. Clusters running cgroup v1 must transition to cgroup v2.
Additional resources

Service mesh

Description

Telco core cloud-native functions (CNFs) typically require a service mesh implementation. Specific service mesh features and performance requirements are dependent on the application. The selection of service mesh implementation and configuration is outside the scope of this documentation. You must account for the impact of service mesh on cluster resource usage and performance, including additional latency introduced in pod networking, in your implementation.

Additional resources

Networking

The following diagram describes the telco core reference design networking configuration.

Overview of the telco core reference design networking configuration
Figure 2. Telco core reference design networking configuration
New in this release

  • Support for disabling vendor plugins in the SR-IOV Operator

  • Extended telco core RDS validation with MetalLB and EgressIP telco QE validation

  • FRR-K8s is now available under the Cluster Network Operator.

    If you have custom FRRConfiguration CRs in the metallb-system namespace, you must move them under the openshift-network-operator namespace.
Description

  • The cluster is configured for dual-stack IP (IPv4 and IPv6).

  • The validated physical network configuration consists of two dual-port NICs. One NIC is shared among the primary CNI (OVN-Kubernetes) and IPVLAN and MACVLAN traffic, while the second one is dedicated to SR-IOV VF-based pod traffic.

  • A Linux bonding interface (bond0) is created in active-active IEEE 802.3ad LACP mode with the two NIC ports attached. The top-of-rack networking equipment must support and be configured for multi-chassis link aggregation (mLAG) technology.

  • VLAN interfaces are created on top of bond0, including for the primary CNI.

  • Bond and VLAN interfaces are created at cluster install time during the network configuration stage of the installation. Except for the vlan0 VLAN used by the primary CNI, all other VLANs can be created during Day 2 activities with the Kubernetes NMstate Operator.

  • MACVLAN and IPVLAN interfaces are created with their corresponding CNIs. They do not share the same base interface. For more information, see "Cluster Network Operator".

  • SR-IOV VFs are managed by the SR-IOV Network Operator.

  • To ensure consistent source IP addresses for pods behind a LoadBalancer Service, configure an EgressIP CR and specify the podSelector parameter.

  • You can implement service traffic separation by doing the following:

    1. Configure VLAN interfaces and specific kernel IP routes on the nodes using NodeNetworkConfigurationPolicy CRs.

    2. Create a MetalLB BGPPeer CR for each VLAN to establish peering with the remote BGP router.

    3. Define a MetalLB BGPAdvertisement CR to specify which IP address pools should be advertised to a selected list of BGPPeer resources.

      The following diagram illustrates how specific service IP addresses are advertised to the outside via specific VLAN interfaces. Services routes are defined in BGPAdvertisement CRs and configured with values for IPAddressPool1 and BGPPeer1 fields.

Telco core reference design MetalLB service separation
Figure 3. Telco core reference design MetalLB service separation
Additional resources

Cluster Network Operator

New in this release

  • No reference design updates in this release

Description

The Cluster Network Operator (CNO) deploys and manages the cluster network components including the default OVN-Kubernetes network plugin during cluster installation. The CNO allows for configuring primary interface MTU settings, OVN gateway modes to use node routing tables for pod egress, and additional secondary networks such as MACVLAN.

In support of network traffic separation, multiple network interfaces are configured through the CNO. Traffic steering to these interfaces is configured through static routes applied by using the NMState Operator. To ensure that pod traffic is properly routed, OVN-K is configured with the routingViaHost option enabled. This setting uses the kernel routing table and the applied static routes rather than OVN for pod egress traffic.

The Whereabouts CNI plugin is used to provide dynamic IPv4 and IPv6 addressing for additional pod network interfaces without the use of a DHCP server.

Limits and requirements

  • OVN-Kubernetes is required for IPv6 support.

  • Large MTU cluster support requires connected network equipment to be set to the same or larger value. MTU size up to 8900 is supported.

  • MACVLAN and IPVLAN cannot co-locate on the same main interface due to their reliance on the same underlying kernel mechanism, specifically the rx_handler. This handler allows a third-party module to process incoming packets before the host processes them, and only one such handler can be registered per network interface. Since both MACVLAN and IPVLAN need to register their own rx_handler to function, they conflict and cannot coexist on the same interface. Review the source code for more details:

  • Alternative NIC configurations include splitting the shared NIC into multiple NICs or using a single dual-port NIC, though they have not been tested and validated.

  • Clusters with single-stack IP configuration are not validated.

  • The reachabilityTotalTimeoutSeconds parameter in the Network CR configures the EgressIP node reachability check total timeout in seconds. The recommended value is 1 second.

Engineering considerations

  • Pod egress traffic is handled by kernel routing table using the routingViaHost option. Appropriate static routes must be configured in the host.

Additional resources

Load balancer

New in this release

  • FRR-K8s is now available under the Cluster Network Operator.

    If you have custom FRRConfiguration CRs in the metallb-system namespace, you must move them under the openshift-network-operator namespace.
Description

MetalLB is a load-balancer implementation for bare metal Kubernetes clusters that uses standard routing protocols. It enables a Kubernetes service to get an external IP address which is also added to the host network for the cluster. The MetalLB Operator deploys and manages the lifecycle of a MetalLB instance in a cluster. Some use cases might require features not available in MetalLB, such as stateful load balancing. Where necessary, you can use an external third party load balancer. Selection and configuration of an external load balancer is outside the scope of this specification. When an external third-party load balancer is used, the integration effort must include enough analysis to ensure all performance and resource utilization requirements are met.

Limits and requirements

  • Stateful load balancing is not supported by MetalLB. An alternate load balancer implementation must be used if this is a requirement for workload CNFs.

  • You must ensure that the external IP address is routable from clients to the host network for the cluster.

Engineering considerations

  • MetalLB is used in BGP mode only for telco core use models.

  • For telco core use models, MetalLB is supported only with the OVN-Kubernetes network provider used in local gateway mode. See routingViaHost in "Cluster Network Operator".

  • BGP configuration in MetalLB is expected to vary depending on the requirements of the network and peers.

    • You can configure address pools with variations in addresses, aggregation length, auto assignment, and so on.

    • MetalLB uses BGP for announcing routes only. Only the transmitInterval and minimumTtl parameters are relevant in this mode. Other parameters in the BFD profile should remain close to the defaults as shorter values can lead to false negatives and affect performance.

Additional resources

SR-IOV

New in this release

  • You can now create virtual functions for Mellanox NICs with the SR-IOV Network Operator when secure boot is enabled in the cluster host. Before you can create the virtual functions, you must first skip the firmware configuration for the Mellanox NIC and manually allocate the number of virtual functions in the firmware before switching the system to secure boot.

Description

SR-IOV enables physical functions (PFs) to be divided into multiple virtual functions (VFs). VFs can then be assigned to multiple pods to achieve higher throughput performance while keeping the pods isolated. The SR-IOV Network Operator provisions and manages SR-IOV CNI, network device plugin, and other components of the SR-IOV stack.

Limits and requirements

  • Only certain network interfaces are supported. See "Supported devices" for more information.

  • Enabling SR-IOV and IOMMU: the SR-IOV Network Operator automatically enables IOMMU on the kernel command line.

  • SR-IOV VFs do not receive link state updates from the PF. If a link down detection is required, it must be done at the protocol level.

  • MultiNetworkPolicy CRs can be applied to netdevice networks only. This is because the implementation uses iptables, which cannot manage vfio interfaces.

Engineering considerations

  • SR-IOV interfaces in vfio mode are typically used to enable additional secondary networks for applications that require high throughput or low latency.

  • The SriovOperatorConfig CR must be explicitly created. This CR is included in the reference configuration policies, which causes it to be created during initial deployment.

  • NICs that do not support firmware updates with UEFI secure boot or kernel lockdown must be preconfigured with sufficient virtual functions (VFs) enabled to support the number of VFs required by the application workload. For Mellanox NICs, you must disable the Mellanox vendor plugin in the SR-IOV Network Operator. See "Configuring an SR-IOV network device" for more information.

  • To change the MTU value of a VF after the pod has started, do not configure the SriovNetworkNodePolicy MTU field. Instead, use the Kubernetes NMState Operator to set the MTU of the related PF.

Additional resources

NMState Operator

New in this release

  • No reference design updates in this release

Description

The Kubernetes NMState Operator provides a Kubernetes API for performing state-driven network configuration across cluster nodes. It enables network interface configurations, static IPs and DNS, VLANs, trunks, bonding, static routes, MTU, and enabling promiscuous mode on the secondary interfaces. The cluster nodes periodically report on the state of each node’s network interfaces to the API server.

Limits and requirements

Not applicable

Engineering considerations

  • Initial networking configuration is applied using NMStateConfig content in the installation CRs. The NMState Operator is used only when required for network updates.

  • When SR-IOV virtual functions are used for host networking, the NMState Operator (via nodeNetworkConfigurationPolicy CRs) is used to configure VF interfaces, such as VLANs and MTU.

Additional resources

Logging

New in this release

  • No reference design updates in this release

Description

The Cluster Logging Operator enables collection and shipping of logs off the node for remote archival and analysis. The reference configuration uses Kafka to ship audit and infrastructure logs to a remote archive.

Limits and requirements

Not applicable

Engineering considerations

  • The impact of cluster CPU use is based on the number or size of logs generated and the amount of log filtering configured.

  • The reference configuration does not include shipping of application logs. The inclusion of application logs in the configuration requires you to evaluate the application logging rate and have sufficient additional CPU resources allocated to the reserved set.

Additional resources

Power Management

New in this release

  • No reference design updates in this release

Description

Use the Performance Profile to configure clusters with high power mode, low power mode, or mixed mode. The choice of power mode depends on the characteristics of the workloads running on the cluster, particularly how sensitive they are to latency. Configure the maximum latency for a low-latency pod by using the per-pod power management C-states feature.

Limits and requirements

  • Power configuration relies on appropriate BIOS configuration, for example, enabling C-states and P-states. Configuration varies between hardware vendors.

Engineering considerations

  • Latency: To ensure that latency-sensitive workloads meet requirements, you require a high-power or a per-pod power management configuration. Per-pod power management is only available for Guaranteed QoS pods with dedicated pinned CPUs.

Additional resources

Storage

New in this release

  • No reference design updates in this release

Description

Cloud native storage services can be provided by Red Hat OpenShift Data Foundation or other third-party solutions.

OpenShift Data Foundation is a Ceph-based software-defined storage solution for containers. It provides block storage, file system storage, and on-premise object storage, which can be dynamically provisioned for both persistent and non-persistent data requirements. Telco core applications require persistent storage.

All storage data might not be encrypted in flight. To reduce risk, isolate the storage network from other cluster networks. The storage network must not be reachable, or routable, from other cluster networks. Only nodes directly attached to the storage network should be allowed to gain access to it.
Additional resources

Red Hat OpenShift Data Foundation

New in this release

  • No reference design updates in this release

Description

Red Hat OpenShift Data Foundation is a software-defined storage service for containers. For telco core clusters, storage support is provided by OpenShift Data Foundation storage services running externally to the application workload cluster. OpenShift Data Foundation supports separation of storage traffic using secondary CNI networks.

Limits and requirements

  • In an IPv4/IPv6 dual-stack networking environment, OpenShift Data Foundation uses IPv4 addressing. For more information, see Network requirements.

Engineering considerations

  • OpenShift Data Foundation network traffic should be isolated from other traffic on a dedicated network, for example, by using VLAN isolation.

Additional storage solutions

You can use other storage solutions to provide persistent storage for telco core clusters. The configuration and integration of these solutions is outside the scope of the reference design specifications (RDS).

Integration of the storage solution into the telco core cluster must include proper sizing and performance analysis to ensure the storage meets overall performance and resource usage requirements.

Telco core deployment components

The following sections describe the various OKD components and configurations that you use to configure the hub cluster with Red Hat Advanced Cluster Management (RHACM).

Red Hat Advanced Cluster Management

New in this release

  • No reference design updates in this release

Description

Red Hat Advanced Cluster Management (RHACM) provides Multi Cluster Engine (MCE) installation and ongoing GitOps ZTP lifecycle management for deployed clusters. You manage cluster configuration and upgrades declaratively by applying Policy custom resources (CRs) to clusters during maintenance windows.

You apply policies with the RHACM policy controller as managed by Topology Aware Lifecycle Manager. Configuration, upgrades, and cluster status are managed through the policy controller.

When installing managed clusters, RHACM applies labels and initial ignition configuration to individual nodes in support of custom disk partitioning, allocation of roles, and allocation to machine config pools. You define these configurations with SiteConfig or ClusterInstance CRs.

Limits and requirements
Engineering considerations

  • When managing multiple clusters with unique content per installation, site, or deployment, using RHACM hub templating is strongly recommended. RHACM hub templating allows you to apply a consistent set of policies to clusters while providing for unique values per installation.

Additional resources

Topology Aware Lifecycle Manager

New in this release

No reference design updates in this release.

Description

Topology Aware Lifecycle Manager is an Operator which runs only on the hub cluster. TALM manages how changes including cluster and Operator upgrades, configurations, and so on, are rolled out to managed clusters in the network. TALM has the following core features:

  • Provides sequenced updates of cluster configurations and upgrades (OKD and Operators) as defined by cluster policies.

  • Provides for deferred application of cluster updates.

  • Supports progressive rollout of policy updates to sets of clusters in user configurable batches.

  • Allows for per-cluster actions by adding ztp-done or similar user-defined labels to clusters.

Limits and requirements

  • Supports concurrent cluster deployments in batches of 400.

Engineering considerations

  • Only policies with the ran.openshift.io/ztp-deploy-wave annotation are applied by TALM during initial cluster installation.

  • Any policy can be remediated by TALM under control of a user created ClusterGroupUpgrade CR.

Additional resources

GitOps Operator and GitOps ZTP plugins

New in this release

  • No reference design updates in this release

Description

The GitOps Operator provides a GitOps driven infrastructure for managing cluster deployment and configuration. Cluster definitions and configuration are maintained in a Git repository.

ZTP plugins provide support for generating Installation CRs from SiteConfig CRs and automatically wrapping configuration CRs in policies based on RHACM PolicyGenerator CRs.

The SiteConfig Operator provides improved support for generation of Installation CRs from ClusterInstance CRs.

Where possible, use ClusterInstance CRs for cluster installation instead of the SiteConfig with GitOps ZTP plugin method.

You should structure the Git repository according to release version, with all necessary artifacts (SiteConfig, ClusterInstance, PolicyGenerator, and PolicyGenTemplate, and supporting reference CRs) included. This enables deploying and managing multiple versions of the OpenShift platform and configuration versions to clusters simultaneously and through upgrades.

The recommended Git structure keeps reference CRs in a directory separate from customer or partner provided content. This means that you can import reference updates by simply overwriting existing content. Customer or partner-supplied CRs can be provided in a parallel directory to the reference CRs for easy inclusion in the generated configuration policies.

Limits and requirements

  • Each ArgoCD application supports up to 300 nodes. Multiple ArgoCD applications can be used to achieve the maximum number of clusters supported by a single hub cluster.

  • The SiteConfig CR must use the extraManifests.searchPaths field to reference the reference manifests.

    Since OKD 4.15, the spec.extraManifestPath field is deprecated.
Engineering considerations

  • Set the MachineConfigPool (mcp) CR paused field to true during a cluster upgrade maintenance window and set the maxUnavailable field to the maximum tolerable value. This prevents multiple cluster node reboots during upgrade, which results in a shorter overall upgrade. When you unpause the mcp CR, all the configuration changes are applied with a single reboot.

    During installation, custom mcp CRs can be paused along with setting maxUnavailable to 100% to improve installation times.

  • To avoid confusion or unintentional overwriting when updating content, you should use unique and distinguishable names for custom CRs in the reference-crs/ directory under core-overlay and extra manifests in Git.

  • The SiteConfig CR allows multiple extra-manifest paths. When file names overlap in multiple directory paths, the last file found in the directory order list takes precedence.

Additional resources

Monitoring

New in this release

  • No reference design updates in this release

Description

The Cluster Monitoring Operator (CMO) is included by default in OKD and provides monitoring (metrics, dashboards, and alerting) for the platform components and optionally user projects. You can customize the default log retention period, custom alert rules, and so on. The default handling of pod CPU and memory metrics, based on upstream Kubernetes and cAdvisor, makes a tradeoff favoring stale data over metric accuracy. This leads to spikes in reporting, which can create false alerts, depending on the user-specified thresholds. OKD supports an opt-in Dedicated Service Monitor feature that creates an additional set of pod CPU and memory metrics that do not suffer from this behavior. For more information, see Dedicated Service Monitors - Questions and Answers (Red Hat Knowledgebase).

In addition to the default configuration, the following metrics are expected to be configured for telco core clusters:

  • Pod CPU and memory metrics and alerts for user workloads

Limits and requirements

  • You must enable the Dedicated Service Monitor feature to represent pod metrics accurately.

Engineering considerations

  • The Prometheus retention period is specified by the user. The value used is a tradeoff between operational requirements for maintaining historical data on the cluster against CPU and storage resources. Longer retention periods increase the need for storage and require additional CPU to manage data indexing.

Additional resources

Scheduling

New in this release

  • No reference design updates in this release

Description

The scheduler is a cluster-wide component responsible for selecting the correct node for a given workload. It is a core part of the platform and does not require any specific configuration in the common deployment scenarios. However, a few specific use cases are described in the following section.

NUMA-aware scheduling can be enabled through the NUMA Resources Operator. For more information, see "Scheduling NUMA-aware workloads".

Limits and requirements

  • The default scheduler does not understand the NUMA locality of workloads. It only knows about the sum of all free resources on a worker node. This might cause workloads to be rejected when scheduled to a node with the topology manager policy set to single-numa-node or restricted. For more information, see "Topology Manager policies".

    • For example, consider a pod requesting 6 CPUs that is scheduled to an empty node that has 4 CPUs per NUMA node. The total allocatable capacity of the node is 8 CPUs. The scheduler places the pod on the empty node. The node local admission fails, as there are only 4 CPUs available in each of the NUMA nodes.

  • All clusters with multi-NUMA nodes are required to use the NUMA Resources Operator. See "Installing the NUMA Resources Operator" for more information. Use the machineConfigPoolSelector field in the KubeletConfig CR to select all nodes where NUMA aligned scheduling is required.

  • All machine config pools must have consistent hardware configuration. For example, all nodes are expected to have the same NUMA zone count.

Engineering considerations

  • Pods might require annotations for correct scheduling and isolation. For more information about annotations, see "CPU partitioning and performance tuning".

  • You can configure SR-IOV virtual function NUMA affinity to be ignored during scheduling by using the excludeTopology field in SriovNetworkNodePolicy CR.

Additional resources

Topology Manager policies

Node Configuration

New in this release

  • No reference design updates in this release

Limits and requirements

  • Analyze additional kernel modules to determine impact on CPU load, system performance, and ability to meet KPIs.

    Table 1. Additional kernel modules
    Feature Description

    Additional kernel modules

    Install the following kernel modules by using MachineConfig CRs to provide extended kernel functionality to CNFs.

    • sctp

    • ip_gre

    • ip6_tables

    • ip6t_REJECT

    • ip6table_filter

    • ip6table_mangle

    • iptable_filter

    • iptable_mangle

    • iptable_nat

    • xt_multiport

    • xt_owner

    • xt_REDIRECT

    • xt_statistic

    • xt_TCPMSS

    Container mount namespace hiding

    Reduce the frequency of kubelet housekeeping and eviction monitoring to reduce CPU usage. Creates a container mount namespace, visible to kubelet/CRI-O, to reduce system mount scanning overhead.

    Kdump enable

    Optional configuration (enabled by default)
Additional resources

Host firmware and boot loader configuration

New in this release

  • No reference design updates in this release

Engineering considerations

  • Enabling secure boot is the recommended configuration.

    When secure boot is enabled, only signed kernel modules are loaded by the kernel. Out-of-tree drivers are not supported.

Disconnected environment

New in this release

  • No reference design updates in this release

Descrption

Telco core clusters are expected to be installed in networks without direct access to the internet. All container images needed to install, configure, and operate the cluster must be available in a disconnected registry. This includes OKD images, Day 2 OLM Operator images, and application workload images. The use of a disconnected environment provides multiple benefits, including:

  • Security - limiting access to the cluster

  • Curated content – the registry is populated based on curated and approved updates for clusters

Limits and requirements

  • A unique name is required for all custom CatalogSource resources. Do not reuse the default catalog names.

Engineering considerations

  • A valid time source must be configured as part of cluster installation

Additional resources

Agent-based Installer

New in this release

  • No reference design updates in this release

Description

Telco core clusters can be installed by using the Agent-based Installer. This method allows you to install OpenShift on bare-metal servers without requiring additional servers or VMs for managing the installation. The Agent-based Installer can be run on any system (for example, from a laptop) to generate an ISO installation image. The ISO is used as the installation media for the cluster supervisor nodes. Installation progress can be monitored using the ABI tool from any system with network connectivity to the supervisor node’s API interfaces.

ABI supports the following:

  • Installation from declarative CRs

  • Installation in disconnected environments

  • Installation with no additional supporting install or bastion servers required to complete the installation

Limits and requirements

  • Disconnected installation requires a registry that is reachable from the installed host, with all required content mirrored in that registry.

Engineering considerations

  • Networking configuration should be applied as NMState configuration during installation. Day 2 networking configuration using the NMState Operator is not supported.

Additional resources

Security

New in this release
Description

Telco customers are security conscious and require clusters to be hardened against multiple attack vectors. In OKD, there is no single component or feature responsible for securing a cluster. Use the following security-oriented features and configurations to secure your clusters:

  • SecurityContextConstraints (SCC): All workload pods should be run with restricted-v2 or restricted SCC.

  • Seccomp: All pods should run with the RuntimeDefault (or stronger) seccomp profile.

  • Rootless DPDK pods: Many user-plane networking (DPDK) CNFs require pods to run with root privileges. With this feature, a conformant DPDK pod can run without requiring root privileges. Rootless DPDK pods create a tap device in a rootless pod that injects traffic from a DPDK application to the kernel.

  • Storage: The storage network should be isolated and non-routable to other cluster networks. See the "Storage" section for additional details.

Refer to Custom nftable firewall rules in OpenShift for a supported method of implementing custom nftables firewall rules in OpenShift cluster nodes. This article is intended for cluster administrators who are responsible for managing network security policies in OpenShift environments. It is crucial to carefully consider the operational implications before deploying this method, including:

  • Early application: The rules are applied at boot time, before the network is fully operational. Ensure the rules don’t inadvertently block essential services required during the boot process.

  • Risk of misconfiguration: Errors in your custom rules can lead to unintended consequences, potentially leading to performance impact or blocking legitimate traffic or isolating nodes. Thoroughly test your rules in a non-production environment before deploying them to your main cluster.

  • External endpoints: OpenShift requires access to external endpoints to function. For more information about the firewall allowlist, see "Configuring your firewall for OKD". Ensure that cluster nodes are permitted access to those endpoints.

  • Node reboot: Unless node disruption policies are configured, applying the MachineConfig CR with the required firewall settings causes a node reboot. Be aware of this impact and schedule a maintenance window accordingly. For more information, see "Using node disruption policies to minimize disruption from machine config changes".

    Node disruption policies are available in OKD 4.17 and later.

  • Network flow matrix: For more information about managing ingress traffic, see "OKD network flow matrix". You can restrict ingress traffic to essential flows to improve network security. The matrix provides insights into base cluster services but excludes traffic generated by Day-2 Operators.

  • Cluster version updates and upgrades: Exercise caution when updating or upgrading OpenShift clusters. Recent changes to the platform’s firewall requirements might require adjustments to network port permissions. Although the documentation provides guidelines, note that these requirements can evolve over time. To minimize disruptions, you should test any updates or upgrades in a staging environment before applying them in production. This helps you to identify and address potential compatibility issues related to firewall configuration changes.

Limits and requirements

  • Rootless DPDK pods requires the following additional configuration:

    • Configure the container_t SELinux context for the tap plugin.

    • Enable the container_use_devices SELinux boolean for the cluster host.

Engineering considerations

  • For rootless DPDK pod support, enable the SELinux container_use_devices boolean on the host to allow the tap device to be created. This introduces an acceptable security risk.

Additional resources

Scalability

New in this release

  • No reference design updates in this release

Description

Scaling of workloads is described in "Application workloads".

Limits and requirements

  • Cluster can scale to at least 120 nodes.

Telco core reference configuration CRs

Use the following custom resources (CRs) to configure and deploy OKD clusters with the telco core profile. Use the CRs to form the common baseline used in all the specific use models unless otherwise indicated.

Extracting the telco core reference design configuration CRs

You can extract the complete set of custom resources (CRs) for the telco core profile from the telco-core-rds-rhel9 container image. The container image has both the required CRs, and the optional CRs, for the telco core profile.

Prerequisites

  • You have installed podman.

Procedure

  • Extract the content from the telco-core-rds-rhel9 container image by running the following commands:

    $ mkdir -p ./out
    $ podman run -it registry.redhat.io/openshift4/openshift-telco-core-rds-rhel9:v4.18 | base64 -d | tar xv -C out
Verification

  • The out directory has the following directory structure. You can view the telco core CRs in the out/telco-core-rds/ directory.

    Example output
    out/
└── telco-core-rds
    ├── configuration
    │   └── reference-crs
    │       ├── optional
    │       │   ├── logging
    │       │   ├── networking
    │       │   │   └── multus
    │       │   │       └── tap_cni
    │       │   ├── other
    │       │   └── tuning
    │       └── required
    │           ├── networking
    │           │   ├── metallb
    │           │   ├── multinetworkpolicy
    │           │   └── sriov
    │           ├── other
    │           ├── performance
    │           ├── scheduling
    │           └── storage
    │               └── odf-external
    └── install
Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.

  • You have credentials to access the registry.redhat.io container image registry.

  • You installed the cluster-compare plugin.

Procedure

  1. Login to the container image registry with your credentials by running the following command:

    $ podman login registry.redhat.io
Additional resources

Node configuration reference CRs

Table 2. Node configuration CRs
Component Reference CR Description Optional

Additional kernel modules

control-plane-load-kernel-modules.yaml

Optional. Configures the kernel modules for control plane nodes.

No

Additional kernel modules

sctp_module_mc.yaml

Optional. Loads the SCTP kernel module in worker nodes.

No

Additional kernel modules

worker-load-kernel-modules.yaml

Optional. Configures kernel modules for worker nodes.

No

Container mount namespace hiding

mount_namespace_config_master.yaml

Configures a mount namespace for sharing container-specific mounts between kubelet and CRI-O on control plane nodes.

No

Container mount namespace hiding

mount_namespace_config_worker.yaml

Configures a mount namespace for sharing container-specific mounts between kubelet and CRI-O on worker nodes.

No

Kdump enable

kdump-master.yaml

Configures kdump crash reporting on master nodes.

No

Kdump enable

kdump-worker.yaml

Configures kdump crash reporting on worker nodes.

No

Resource tuning reference CRs

Table 3. Resource tuning CRs
Component Reference CR Description Optional

System reserved capacity

control-plane-system-reserved.yaml

Optional. Configures kubelet, enabling auto-sizing reserved resources for the control plane node pool.

No

Networking reference CRs

Table 4. Networking CRs
Component Reference CR Description Optional

Baseline

Network.yaml

Configures the default cluster network, specifying OVN Kubernetes settings like routing via the host. It also allows the definition of additional networks, including custom CNI configurations, and enables the use of MultiNetworkPolicy CRs for network policies across multiple networks.

No

Baseline

networkAttachmentDefinition.yaml

Optional. Defines a NetworkAttachmentDefinition resource specifying network configuration details such as node selector and CNI configuration.

Yes

Load Balancer

addr-pool.yaml

Configures MetalLB to manage a pool of IP addresses with auto-assign enabled for dynamic allocation of IPs from the specified range.

No

Load Balancer

bfd-profile.yaml

Configures bidirectional forwarding detection (BFD) with customized intervals, detection multiplier, and modes for quicker network fault detection and load balancing failover.

No

Load Balancer

bgp-advr.yaml

Defines a BGP advertisement resource for MetalLB, specifying how an IP address pool is advertised to BGP peers. This enables fine-grained control over traffic routing and announcements.

No

Load Balancer

bgp-peer.yaml

Defines a BGP peer in MetalLB, representing a BGP neighbor for dynamic routing.

No

Load Balancer

community.yaml

Defines a MetalLB community, which groups one or more BGP communities under a named resource. Communities can be applied to BGP advertisements to control routing policies and change traffic routing.

No

Load Balancer

metallb.yaml

Defines the MetalLB resource in the cluster.

No

Load Balancer

metallbNS.yaml

Defines the metallb-system namespace in the cluster.

No

Load Balancer

metallbOperGroup.yaml

Defines the Operator group for the MetalLB Operator.

No

Load Balancer

metallbSubscription.yaml

Creates a subscription resource for the metallb Operator with manual approval for install plans.

No

Multus - Tap CNI for rootless DPDK pods

mc_rootless_pods_selinux.yaml

Configures a MachineConfig resource which sets an SELinux boolean for the tap CNI plugin on worker nodes.

Yes

NMState Operator

NMState.yaml

Defines an NMState resource that is used by the NMState Operator to manage node network configurations.

No

NMState Operator

NMStateNS.yaml

Creates the NMState Operator namespace.

No

NMState Operator

NMStateOperGroup.yaml

Creates the Operator group in the openshift-nmstate namespace, allowing the NMState Operator to watch and manage resources.

No

NMState Operator

NMStateSubscription.yaml

Creates a subscription for the NMState Operator, managed through OLM.

No

SR-IOV Network Operator

sriovNetwork.yaml

Defines an SR-IOV network specifying network capabilities, IP address management (ipam), and the associated network namespace and resource.

No

SR-IOV Network Operator

sriovNetworkNodePolicy.yaml

Configures network policies for SR-IOV devices on specific nodes, including customization of device selection, VF allocation (numVfs), node-specific settings (nodeSelector), and priorities.

No

SR-IOV Network Operator

SriovOperatorConfig.yaml

Configures various settings for the SR-IOV Operator, including enabling the injector and Operator webhook, disabling pod draining, and defining the node selector for the configuration daemon.

No

SR-IOV Network Operator

SriovSubscription.yaml

Creates a subscription for the SR-IOV Network Operator, managed through OLM.

No

SR-IOV Network Operator

SriovSubscriptionNS.yaml

Creates the SR-IOV Network Operator subscription namespace.

No

SR-IOV Network Operator

SriovSubscriptionOperGroup.yaml

Creates the Operator group for the SR-IOV Network Operator, allowing it to watch and manage resources in the target namespace.

No

Scheduling reference CRs

Table 5. Scheduling CRs
Component Reference CR Description Optional

NUMA-aware scheduler

nrop.yaml

Enables the NUMA Resources Operator, aligning workloads with specific NUMA node configurations. Required for clusters with multi-NUMA nodes.

No

NUMA-aware scheduler

NROPSubscription.yaml

Creates a subscription for the NUMA Resources Operator, managed through OLM. Required for clusters with multi-NUMA nodes.

No

NUMA-aware scheduler

NROPSubscriptionNS.yaml

Creates the NUMA Resources Operator subscription namespace. Required for clusters with multi-NUMA nodes.

No

NUMA-aware scheduler

NROPSubscriptionOperGroup.yaml

Creates the Operator group in the numaresources-operator namespace, allowing the NUMA Resources Operator to watch and manage resources. Required for clusters with multi-NUMA nodes.

No

NUMA-aware scheduler

sched.yaml

Configures a topology-aware scheduler in the cluster that can handle NUMA aware scheduling of pods across nodes.

No

NUMA-aware scheduler

Scheduler.yaml

Configures control plane nodes as non-schedulable for workloads.

No

Storage reference CRs

Table 6. Storage CRs
Component Reference CR Description Optional

External ODF configuration

01-rook-ceph-external-cluster-details.secret.yaml

Defines a Secret resource containing base64-encoded configuration data for an external Ceph cluster in the openshift-storage namespace.

No

External ODF configuration

02-ocs-external-storagecluster.yaml

Defines an OpenShift Container Storage (OCS) storage resource which configures the cluster to use an external storage back end.

No

External ODF configuration

odfNS.yaml

Creates the monitored openshift-storage namespace for the OpenShift Data Foundation Operator.

No

External ODF configuration

odfOperGroup.yaml

Creates the Operator group in the openshift-storage namespace, allowing the OpenShift Data Foundation Operator to watch and manage resources.

No

External ODF configuration

odfSubscription.yaml

Creates the subscription for the OpenShift Data Foundation Operator in the openshift-storage namespace.

No

Telco core reference configuration software specifications

The Red Hat telco core 4 solution has been validated using the following Red Hat software products for OKD clusters.

Table 7. Telco core cluster validated software components
Component Software version

Red Hat Advanced Cluster Management (RHACM)

2.121

Cluster Logging Operator

6.12

OpenShift Data Foundation

4.18

SR-IOV Network Operator

4.18

MetalLB

4.18

NMState Operator

4.18

NUMA-aware scheduler

4.18

[1] This table will be updated when the aligned RHACM version 2.13 is released.

[2] This table will be updated when the aligned Cluster Logging Operator 6.2 is released.