Telco core reference design components

No reference design updates in this release

The operating system needs a certain amount of CPU to perform all the support tasks including kernel networking.

A system with Hyper-Threading enabled must always put all core sibling threads to the same pool of CPUs.

The set of reserved and isolated cores must include all CPU cores.

Core 0 of each NUMA node must be included in the reserved CPU set.

Isolated cores might be impacted by interrupts. The following annotations must be attached to the pod if guaranteed QoS pods require full use of the CPU:

cpu-load-balancing.crio.io: "disable"
cpu-quota.crio.io: "disable"
irq-load-balancing.crio.io: "disable"

When per-pod power management is enabled with PerformanceProfile.workloadHints.perPodPowerManagement the following annotations must also be attached to the pod if guaranteed QoS pods require full use of the CPU:

cpu-c-states.crio.io: "disable"
cpu-freq-governor.crio.io: "performance"

The minimum reserved capacity (systemReserved) required can be found by following the guidance in "Which amount of CPU and memory are recommended to reserve for the system in OpenShift 4 nodes?"

The actual required reserved CPU capacity depends on the cluster configuration and workload attributes.

This reserved CPU value must be rounded up to a full core (2 hyper-thread) alignment.

Changes to the CPU partitioning will drain and reboot the nodes in the MCP.

The reserved CPUs reduce the pod density, as the reserved CPUs are removed from the allocatable capacity of the OpenShift node.

The real-time workload hint should be enabled if the workload is real-time capable.

Hardware without Interrupt Request (IRQ) affinity support will impact isolated CPUs. To ensure that pods with guaranteed CPU QoS have full use of allocated CPU, all hardware in the server must support IRQ affinity.

OVS dynamically manages its cpuset configuration to adapt to network traffic needs. You do not need to reserve additional CPUs for handling high network throughput on the primary CNI.

If workloads running on the cluster require cgroups v1, you can configure nodes to use cgroups v1 as part of the initial cluster deployment. For more information, see "Enabling Linux cgroup v1 during installation".

Telco core validation is now extended with bonding, MACVLAN, IPVLAN and SR-IOV networking scenarios.

The cluster is configured in dual-stack IP configuration (IPv4 and IPv6).

The validated physical network configuration consists of two dual-port NICs. One NIC is shared among the primary CNI (OVN-Kubernetes) and IPVLAN and MACVLAN traffic, the second NIC is dedicated to SR-IOV VF-based Pod traffic.

A Linux bonding interface (bond0) is created in an active-active LACP IEEE 802.3ad configuration with the two NIC ports attached.

VLAN interfaces are created on top of bond0, including for the primary CNI.

Bond and VLAN interfaces are created at install time during network configuration. Apart from the VLAN (VLAN0) used by the primary CNI, the other VLANS can be created on Day 2 using the Kubernetes NMState Operator.

MACVLAN and IPVLAN interfaces are created with their corresponding CNIs. They do not share the same base interface.

SR-IOV VFs are managed by the SR-IOV Network Operator. The following diagram provides an overview of SR-IOV NIC sharing:

Figure 1. SR-IOV NIC sharing

No reference design updates in this release

OVN-Kubernetes is required for IPv6 support.

Large MTU cluster support requires connected network equipment to be set to the same or larger value.

MACVLAN and IPVLAN cannot co-locate on the same main interface due to their reliance on the same underlying kernel mechanism, specifically the rx_handler. This handler allows a third-party module to process incoming packets before the host processes them, and only one such handler can be registered per network interface. Since both MACVLAN and IPVLAN need to register their own rx_handler to function, they conflict and cannot coexist on the same interface. See ipvlan/ipvlan_main.c#L82 and net/macvlan.c#L1260 for details.

Alternative NIC configurations include splitting the shared NIC into multiple NICs or using a single dual-port NIC.

Single-stack IP cluster not validated.

Pod egress traffic is handled by kernel routing table with the routingViaHost option. Appropriate static routes must be configured in the host.

In OpenShift Container Platform 4.17, frr-k8s is now the default and fully supported Border Gateway Protocol (BGP) backend. The deprecated frr BGP mode is still available. You should upgrade clusters to use the frr-k8s backend.

Stateful load balancing is not supported by MetalLB. An alternate load balancer implementation must be used if this is a requirement for workload CNFs.

The networking infrastructure must ensure that the external IP address is routable from clients to the host network for the cluster.

MetalLB is used in BGP mode only for core use case models.

For core use models, MetalLB is supported with only the OVN-Kubernetes network provider used in local gateway mode. See routingViaHost in the "Cluster Network Operator" section.

BGP configuration in MetalLB varies depending on the requirements of the network and peers.

Address pools can be configured as needed, allowing variation in addresses, aggregation length, auto assignment, and other relevant parameters.

MetalLB uses BGP for announcing routes only. Only the transmitInterval and minimumTtl parameters are relevant in this mode. Other parameters in the BFD profile should remain close to the default settings. Shorter values might lead to errors and impact performance.

No reference design updates in this release

Supported network interface controllers are listed in "Supported devices".

The SR-IOV Network Operator automatically enables IOMMU on the kernel command line.

SR-IOV VFs do not receive link state updates from PF. If link down detection is needed, it must be done at the protocol level.

MultiNetworkPolicy CRs can be applied to netdevice networks only. This is because the implementation uses the iptables tool, which cannot manage vfio interfaces.

SR-IOV interfaces in vfio mode are typically used to enable additional secondary networks for applications that require high throughput or low latency.

If you exclude the SriovOperatorConfig CR from your deployment, the CR will not be created automatically.

NICs that do not support firmware updates under secure boot or kernel lock-down must be pre-configured with enough VFs enabled to support the number of VFs needed by the application workload.

No reference design updates in this release

The initial networking configuration is applied using NMStateConfig content in the installation CRs. The NMState Operator is used only when needed for network updates.

When SR-IOV virtual functions are used for host networking, the NMState Operator using NodeNetworkConfigurationPolicy is used to configure those VF interfaces, for example, VLANs and the MTU.

Cluster Logging Operator 6.0 is new in this release. Update your existing implementation to adapt to the new version of the API.

The impact of cluster CPU use is based on the number or size of logs generated and the amount of log filtering configured.

The reference configuration does not include shipping of application logs. Inclusion of application logs in the configuration requires evaluation of the application logging rate and sufficient additional CPU resources allocated to the reserved set.

No reference design updates in this release

Power configuration relies on appropriate BIOS configuration, for example, enabling C-states and P-states. Configuration varies between hardware vendors.

Latency: To ensure that latency-sensitive workloads meet their requirements, you will need either a high-power configuration or a per-pod power management configuration. Per-pod power management is only available for Guaranteed QoS Pods with dedicated pinned CPUs.

No reference design updates in this release

Monitoring configuration must enable the dedicated service monitor feature for accurate representation of pod metrics

You configure the Prometheus retention period. The value used is a tradeoff between operational requirements for maintaining historical data on the cluster against CPU and storage resources. Longer retention periods increase the need for storage and require additional CPU to manage the indexing of data.

No reference design updates in this release

The scheduler is a cluster-wide component responsible for selecting the right node for a given workload. It is a core part of the platform and does not require any specific configuration in the common deployment scenarios. However, there are few specific use cases described in the following section. NUMA-aware scheduling can be enabled through the NUMA Resources Operator. For more information, see "Scheduling NUMA-aware workloads".

The default scheduler does not understand the NUMA locality of workloads. It only knows about the sum of all free resources on a worker node. This might cause workloads to be rejected when scheduled to a node with the topology manager policy set to single-numa-node or restricted.

All machine config pools must have consistent hardware configuration for example all nodes are expected to have the same NUMA zone count.

Pods might require annotations for correct scheduling and isolation. For more information on annotations, see "CPU partitioning and performance tuning".

You can configure SR-IOV virtual function NUMA affinity to be ignored during scheduling by using the excludeTopology field in SriovNetworkNodePolicy CR.

Container mount namespace encapsulation and kdump are now available in the telco core RDS.

Container mount namespace encapsulation creates a container mount namespace that reduces system mount scanning and is visible to kubelet and CRI-O.

kdump is an optional configuration that is enabled by default that captures debug information when a kernel panic occurs. The reference CRs which enable kdump include an increased memory reservation based on the set of drivers and kernel modules included in the reference configuration.

Use of kdump and container mount namespace encapsulation is made available through additional kernel modules. You should analyze these modules to determine impact on CPU load, system performance, and ability to meet required KPIs.

Install the following kernel modules with MachineConfig CRs. These modules provide extended kernel functionality to cloud-native functions (CNFs).

Secure boot is now recommended for cluster hosts configured with the telco core reference design.

Enabling secure boot is the recommended configuration.

No reference design updates in this release

A unique name is required for all custom CatalogSources. Do not reuse the default catalog names.

A valid time source must be configured as part of cluster installation.

Secure boot host firmware setting is now recommended for telco core clusters. For more information, see "Host firmware and boot loader configuration".

SecurityContextConstraints (SCC): All workload pods should be run with restricted-v2 or restricted SCC.

Seccomp: All pods should be run with the RuntimeDefault (or stronger) seccomp profile.

Rootless DPDK pods: Many user-plane networking (DPDK) CNFs require pods to run with root privileges. With this feature, a conformant DPDK pod can be run without requiring root privileges. Rootless DPDK pods create a tap device in a rootless pod that injects traffic from a DPDK application to the kernel.

Storage: The storage network should be isolated and non-routable to other cluster networks. See the "Storage" section for additional details.

Rootless DPDK pods requires the following additional configuration steps:

For rootless DPDK pod support, the SELinux boolean container_use_devices must be enabled on the host for the TAP device to be created. This introduces a security risk that is acceptable for short to mid-term use. Other solutions will be explored.

No reference design updates in this release

Cluster should scale to at least 120 nodes.