This is a cache of https://docs.okd.io/4.17/networking/multiple_networks/primary_networks/about-user-defined-networks.html. It is a snapshot of the page at 2026-01-18T00:37:53.788+0000.
UserDefinedNetwork CR - Multiple networks | Networking | OKD 4.17
  1. Documentation
×

About-user-defined networks

UserDefinedNetwork is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

User-defined networks (UDNs) extend OVN-Kubernetes to enable custom layer 2 and layer 3 network segments with default isolation, providing enhanced network flexibility, security, and segmentation capabilities for multitenant deployments and custom network architectures.

Overview of user-defined networks

To secure and improve network segmentation and isolation, administrators can create primary or secondary networks at the namespace level by using the UserDefinedNetwork CR.

Before the implementation of user-defined networks (UDN), the OVN-Kubernetes CNI plugin for OKD only supported a layer 3 topology on the primary or main network. Due to Kubernetes design principles: all pods are attached to the main network, all pods communicate with each other by their IP addresses, and inter-pod traffic is restricted according to network policy.

While the Kubernetes design is useful for simple deployments, this layer 3 topology restricts customization of primary network segment configurations, especially for modern multitenant deployments.

UDN improves the flexibility and segmentation capabilities of the default layer 3 topology for a Kubernetes pod network by enabling custom layer 2 and layer 3 network segments, where all these segments are isolated by default. These segments act as either primary or secondary networks for container pods and virtual machines that use the default OVN-Kubernetes CNI plugin. UDNs enable a wide range of network architectures and topologies, enhancing network flexibility, security, and performance.

The following sections further emphasize the benefits and limitations of user-defined networks,UserDefinedNetwork CR, how to create the CR, and additional configuration details that might be relevant to your deployment.

Benefits of a user-defined network

User-defined networks enable tenant isolation by providing each namespace with its own isolated primary network, reducing cross-tenant traffic risks and simplifying network management by eliminating the need for complex network policies.

User-defined networks offer the following benefits:

  1. Enhanced network isolation for security

    • Tenant isolation: Namespaces can have their own isolated primary network, similar to how tenants are isolated in OpenStack. This improves security by reducing the risk of cross-tenant traffic.

  2. Network flexibility

    • Layer 2 and layer 3 support: Cluster administrators can configure primary networks as layer 2 or layer 3 network types.

  3. Simplified network management

    • Reduced network configuration complexity: With user-defined networks, the need for complex network policies are eliminated because isolation can be achieved by grouping workloads in different networks.

  4. Advanced capabilities

    • Consistent and selectable IP addressing: Users can specify and reuse IP subnets across different namespaces and clusters, providing a consistent networking environment.

    • Support for multiple networks: The user-defined networking feature allows administrators to connect multiple namespaces to a single network, or to create distinct networks for different sets of namespaces.

  5. Simplification of application migration from OpenStack

    • Network parity: With user-defined networking, the migration of applications from OpenStack to OKD is simplified by providing similar network isolation and configuration options.

Developers and administrators can create a user-defined network that is namespace scoped using the custom resource. An overview of the process is as follows:

  1. An administrator creates a namespace for a user-defined network with the k8s.ovn.org/primary-user-defined-network label.

  2. The UserDefinedNetwork CR is created by either the cluster administrator or the user.

  3. The user creates pods in the namespace.

Limitations for UserDefinedNetwork custom resource

To deploy a successful user-defined networks (UDN), you must consider their limitations including DNS resolution behavior, restricted access to default network services such as the image registry, network policy constraints between isolated networks, and the requirement to create namespaces and networks before pods.

Consider the following limitations before implementing a UDN.

  • DNS limitations:

    • DNS lookups for pods resolve to the pod’s IP address on the cluster default network. Even if a pod is part of a user-defined network, DNS lookups will not resolve to the pod’s IP address on that user-defined network. However, DNS lookups for services and external entities will function as expected.

    • When a pod is assigned to a primary UDN, it can access the Kubernetes API (KAPI) and DNS services on the cluster’s default network.

  • Initial network assignment: You must create the namespace and network before creating pods. Assigning a namespace with pods to a new network or creating a UDN in an existing namespace will not be accepted by OVN-Kubernetes.

  • Health check limitations: Kubelet health checks are performed by the cluster default network, which does not confirm the network connectivity of the primary interface on the pod. Consequently, scenarios where a pod appears healthy by the default network, but has broken connectivity on the primary interface, are possible with user-defined networks.

  • Network policy limitations: Network policies that enable traffic between namespaces connected to different user-defined primary networks are not effective. These traffic policies do not take effect because there is no connectivity between these isolated networks.

  • Creation and modification limitation: The ClusterUserDefinedNetwork CR and the UserDefinedNetwork CR cannot be modified after being created.

Layer 2 and layer 3 topologies

A layer 2 topology creates a distributed virtual switch across cluster nodes, this network topology provides a smooth live migration of virtual machine (VM) within the same subnet. A layer 3 topology creates unique segments per node with routing between them, this network topology effectively manages large broadcast domains.

In a flat layer 2 topology, virtual machines and pods connect to the virtual switch so that all these components can communicate with each other within the same subnet. This topology is useful for the live migration of VMs across nodes in the cluster. The following diagram shows a flat layer 2 topology with two nodes that use the virtual switch for live migration purposes:

A flat layer 2 topology with a virtual switch so that virtual machines in node-1 to node-2 can communicate with each other
Figure 1. A flat layer 2 topology that uses a virtual switch for component communication

If you decide not to specify a layer 2 subnet, then you must manually configure IP addresses for each pod in your cluster. When you do not specify a layer 2 subnet, port security is limited to preventing Media Access Control (MAC) spoofing only, and does not include IP spoofing. A layer 2 topology creates a single broadcast domain that can be challenging in large network environments, where the topology might cause a broadcast storm that can degrade network performance.

To access more configurable options for your network, you can integrate a layer 2 topology with a user-defined network (UDN). The following diagram shows two nodes that use a UDN with a layer 2 topology that includes pods that exist on each node. Each node includes two interfaces:

  • A node interface, which is a compute node that connects networking components to the node.

  • An Open vSwitch (OVS) bridge such as br-ex, which creates an layer 2 OVN switch so that pods can communicate with each other and share resources.

An external switch connects these two interfaces, while the gateway or router handles routing traffic between the external switch and the layer 2 OVN switch. VMs and pods in a node can use the UDN to communicate with each other. The layer 2 OVN switch handles node traffic over a UDN so that live migrate of a VM from one node to another is possible.

A UDN that uses a layer 2 topology for migrating a VM from node-1 to node-2
Figure 2. A user-defined network (UDN) that uses a layer 2 topology

A layer 3 topology creates a unique layer 2 segment for each node in a cluster. The layer 3 routing mechanism interconnects these segments so that virtual machines and pods that are hosted on different nodes can communicate with each other. A layer 3 topology can effectively manage large broadcast domains by assigning each domain to a specific node, so that broadcast traffic has a reduced scope. To configure a layer 3 topology, you must configure cidr and hostSubnet parameters.

Best practices for UserDefinedNetwork

To deploy a successful instance of the UserDefinedNetwork (UDN) CR, you must follow masquerade IP address requirements, avoid default and openshift-* namespaces, set a proper namespace selector configuration, and ensure physical network parameter matching.

The following details provide a best practice for designing a UDN CR:

  • openshift-* namespaces should not be used to set up a UDN.

  • 2 masquerade IP addresses are required for user defined networks. You must reconfigure your masquerade subnet to be large enough to hold the required number of networks.

    • For OKD 4.17 and later, clusters use 169.254.0.0/17 for IPv4 and fd69::/112 for IPv6 as the default masquerade subnet. These ranges should be avoided by users. For updated clusters, there is no change to the default masquerade subnet.

    • Changing the cluster’s masquerade subnet is unsupported after a user-defined network has been configured for a project. Attempting to modify the masquerade subnet after a UDN has been set up can disrupt the network connectivity and cause configuration issues.

  • Ensure tenants are using the UserDefinedNetwork resource and not the NetworkAttachmentDefinition (NAD) resource. This can create security risks between tenants.

  • When creating network segmentation, you should only use the NAD resource if user-defined network segmentation cannot be completed using the UDN resource.

  • The cluster subnet and services CIDR for a UDN cannot overlap with the default cluster subnet CIDR. OVN-Kubernetes network plugin uses 100.64.0.0/16 as the default network’s join subnet, you must not use that value to configure a UDN joinSubnets field. If the default address values are used anywhere in the network for the cluster, you must override it by setting the joinSubnets field. For more information, see "Additional configuration details for a UserDefinedNetworks CR".

  • The cluster subnet and services CIDR for a UDN cannot overlap with the default cluster subnet CIDR. OVN-Kubernetes network plugin uses 100.64.0.0/16 as the default join subnet for the network, you must not use that value to configure a UDN joinSubnets field. If the default address values are used anywhere in the network for the cluster you must override the default values by setting the joinSubnets field. For more information, see "Additional configuration details for a UserDefinedNetworks CR".

Creating a UserDefinedNetwork custom resource

Create a UserDefinedNetwork CR by using the CLI to enable namespace-scoped network segmentation and isolation, allowing you to define custom Layer 2 or Layer 3 network topologies for pods within specific namespaces.

The following procedure creates a UserDefinedNetwork CR that is namespace scoped. Based upon your use case, create your request by using either the my-layer-two-udn.yaml example for a Layer2 topology type or the my-layer-three-udn.yaml example for a Layer3 topology type.

Prerequisites

  • You have logged in with cluster-admin privileges, or you have view and edit role-based access control (RBAC).

Procedure

  1. Optional: For a UserDefinedNetwork CR that uses a primary network, create a namespace with the k8s.ovn.org/primary-user-defined-network label by entering the following command:

    $ cat << EOF | oc apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: <udn_namespace_name>
  labels:
    k8s.ovn.org/primary-user-defined-network: ""
EOF

  2. Create a request for either a Layer2 or Layer3 topology type user-defined network:

    1. Create a YAML file, such as my-layer-two-udn.yaml, to define your request for a Layer2 topology as in the following example:

      apiVersion: k8s.ovn.org/v1
kind: UserDefinedNetwork
metadata:
  name: udn-1
  namespace: <some_custom_namespace>
spec:
  topology: Layer2
  layer2: (3)
    role: Primary
    subnets:
      - "10.0.0.0/24"
      - "2001:db8::/60"

      where:

      name

      Name of your UserDefinedNetwork resource. This should not be default or duplicate any global namespaces created by the Cluster Network Operator (CNO).

      topology

      Specifies the network configuration; accepted values are Layer2 and Layer3. Specifying a Layer2 topology type creates one logical switch that is shared by all nodes.

      role

      Specifies a Primary or Secondary role.

      subnets

      For Layer2 topology types the following specifies config details for the subnet field:

      • The subnets field is optional.

      • The subnets field is of type string and accepts standard CIDR formats for both IPv4 and IPv6.

      • The subnets field accepts one or two items. For two items, they must be of a different family. For example, subnets values of 10.100.0.0/16 and 2001:db8::/64.

      • Layer2 subnets can be omitted. If omitted, users must configure IP addresses for the pods. As a consequence, port security only prevents MAC spoofing.

      • The Layer2 subnets field is mandatory when the ipamLifecycle field is specified.

    2. Create a YAML file, such as my-layer-three-udn.yaml, to define your request for a Layer3 topology as in the following example:

      apiVersion: k8s.ovn.org/v1
kind: UserDefinedNetwork
metadata:
  name: udn-2-primary
  namespace: <some_custom_namespace>
spec:
  topology: Layer3
  layer3:
    role: Primary
    subnets:
      - cidr: 10.150.0.0/16
        hostSubnet: 24
      - cidr: 2001:db8::/60
        hostSubnet: 64
# ...

      where:

      name

      Name of your UserDefinedNetwork resource. This should not be default or duplicate any global namespaces created by the Cluster Network Operator (CNO).

      topology

      Specifies the network configuration; accepted values are Layer2 and Layer3. Specifying a Layer2 topology type creates one logical switch that is shared by all nodes.

      role

      Specifies a Primary or Secondary role.

      subnets

      For Layer3 topology types the following specifies config details for the subnet field:

      • The subnets field is mandatory.

      • The type for the subnets field is cidr and hostSubnet:

        • cidr is equivalent to the clusterNetwork configuration settings of a cluster. The IP addresses in the CIDR are distributed to pods in the user defined network. This parameter accepts a string value.

        • hostSubnet defines the per-node subnet prefix.

        • For IPv6, only a /64 length is supported for hostSubnet.

  3. Apply your request by running the following command:

    $ oc apply -f <my_layer_two_udn.yaml>

    Where <my_layer_two_udn.yaml> is the name of your Layer2 or Layer3 configuration file.

  4. Verify that your request is successful by running the following command:

    $ oc get userdefinednetworks udn-1 -n <some_custom_namespace> -o yaml

    Where some_custom_namespace is the namespace you created for your user-defined network.

    Example output
    apiVersion: k8s.ovn.org/v1
kind: UserDefinedNetwork
metadata:
  creationTimestamp: "2024-08-28T17:18:47Z"
  finalizers:
  - k8s.ovn.org/user-defined-network-protection
  generation: 1
  name: udn-1
  namespace: some-custom-namespace
  resourceVersion: "53313"
  uid: f483626d-6846-48a1-b88e-6bbeb8bcde8c
spec:
  layer2:
    role: Primary
    subnets:
    - 10.0.0.0/24
    - 2001:db8::/60
  topology: Layer2
status:
  conditions:
  - lastTransitionTime: "2024-08-28T17:18:47Z"
    message: NetworkAttachmentDefinition has been created
    reason: NetworkAttachmentDefinitionReady
    status: "True"
    type: NetworkReady
Additional resources

Additional configuration details for a UserDefinedNetworks CR

Configure optional advanced settings for ClusterUserDefinedNetwork and UserDefinedNetwork CRs when default values conflict with your network topology or when you need persistent IP addresses, custom gateways, or specific subnet configurations.

It is not recommended to set these fields without explicit need and understanding of OVN-Kubernetes network topology.

  1. Optional configurations for user-defined networks

CUDN field

UDN field

Type

Description

spec.network.<topology>.joinSubnets

spec.<topology>.joinSubnets

object

When omitted, the platform sets default values for the joinSubnets field of 100.65.0.0/16 for IPv4 and fd99::/64 for IPv6. If the default address values are used anywhere in the cluster’s network you must override it by setting the joinSubnets field. If you choose to set this field, ensure it does not conflict with other subnets in the cluster such as the cluster subnet, the default network cluster subnet, and the masquerade subnet.

The joinSubnets field configures the routing between different segments within a user-defined network. Dual-stack clusters can set 2 subnets, one for each IP family; otherwise, only 1 subnet is allowed. This field is only allowed for the Primary network.

spec.network.<topology>.ipam.lifecycle

spec.<topology>.ipam.lifecycle

object

The spec.ipam.lifecycle field configures the IP address management system (IPAM). You might use this field for virtual workloads to ensure persistent IP addresses. The only allowed value is Persistent, which ensures that your virtual workloads have persistent IP addresses across reboots and migration. These are assigned by the container network interface (CNI) and used by OVN-Kubernetes to program pod IP addresses. You must not change this for pod annotations.

Setting a value of Persistent is only supported when ipam.mode parameter is set to Enabled.

spec.network.<topology>.ipam.mode

spec.network.<topology>.ipam.mode

object

The mode parameter controls how much of the IP configuration is managed by OVN-Kubernetes. The following options are available: * Enabled: When enabled, OVN-Kubernetes applies the IP configuration to the SDN infrastructure and assigns IP addresses from the selected subnet to the individual pods. This is the default setting. When set to Enabled, the subnets field must be defined. Enabled is the default configuration. * Disabled: When disabled, OVN-Kubernetes only assigns MAC addresses and provides layer 2 communication, which allows users to configure IP addresses. Disabled is only available for layer 2 (secondary) networks. By disabling IPAM, features that rely on selecting pods by IP, for example, network policy, services, and so on, no longer function. Additionally, IP port security is also disabled for interfaces attached to this network. The subnets field must be empty when spec.ipam.mode is set to Disabled.

spec.network.<topology>.mtu

spec.<topology>.mtu

integer

The maximum transmission units (MTU). The default value is 1400. The boundary for IPv4 is 576, and for IPv6 it is 1280.

where:

<topology>

Is one of layer2 or layer3.