Configuring the integration of MetalLB and FRR-K8s - Load balancing with MetalLB | Networking

Activating the integration of MetalLB and FRR-K8s
FRR configurations
Configuring the FRRConfiguration CRD
How FRR-K8s merges multiple configurations

To access advanced routing services not natively provided by MetalLB, configure the FRRConfiguration custom resource (CR). Defining the CR exposes specific FRRouting (FRR) capabilities and extends the routing functionality of your cluster beyond standard MetalLB advertisements.

The FRRConfiguration custom resource is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

FRRouting (FRR) is a free, open-source internet routing protocol suite for Linux and UNIX platforms. FRR-K8s is a Kubernetes-based DaemonSet that exposes a subset of the FRR API in a Kubernetes-compliant manner. MetalLB generates the FRR-K8s configuration corresponding to the MetalLB configuration applied.

When configuring Virtual Route Forwarding (VRF), you must change the VRFs to a table ID lower than 1000 as higher than 1000 is reserved for OKD.

Activating the integration of MetalLB and FRR-K8s

The following procedure shows you how to activate FRR-K8s as the backend for MetalLB.

Prerequisites

You have a cluster installed on bare-metal hardware.
You have installed the OpenShift CLI (oc).
You have logged in as a user with cluster-admin privileges.

Procedure

Set the bgpBackend field of the MetalLB CR to frr-k8s as in the following example:

apiVersion: metallb.io/v1beta1
kind: MetalLB
metadata:
  name: metallb
  namespace: metallb-system
spec:
  bgpBackend: frr-k8s

FRR configurations

You can create multiple FRRConfiguration CRs to use FRR services in MetalLB.

MetalLB generates an FRRConfiguration object which FRR-K8s merges with all other configurations that all users have created. For example, you can configure FRR-K8s to receive all of the prefixes advertised by a given neighbor. The following example configures FRR-K8s to receive all of the prefixes advertised by a BGPPeer with host 172.18.0.5:

Example FRRConfiguration CR

apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
 name: test
 namespace: metallb-system
spec:
 bgp:
   routers:
   - asn: 64512
     neighbors:
     - address: 172.18.0.5
       asn: 64512
       toReceive:
        allowed:
            mode: all
# ...

You can also configure FRR-K8s to always block a set of prefixes, regardless of the configuration applied. This can be useful to avoid routes going to pods or ClusterIPs CIDRs that might result in cluster malfunctions. The following example blocks the set of prefixes 192.168.1.0/24:

Example MetalLB CR

apiVersion: metallb.io/v1beta1
kind: MetalLB
metadata:
  name: metallb
  namespace: metallb-system
spec:
  bgpBackend: frr-k8s
  frrk8sConfig:
    alwaysBlock:
    - 192.168.1.0/24
# ...

You can set FRR-K8s to block the clusterNetwork CIDR and serviceNetwork CIDR. You can view the values for these CIDR address specifications by running the following command:

$ oc describe network.config/cluster

Configuring the FRRConfiguration CRD

To customize routing behavior beyond standard MetalLB capabilities, configure the FRRConfiguration custom resource (CR).

The following reference examples demonstrate how to define specific FRRouting (FRR) parameters to enable advanced services, such as receiving routes:

The routers parameter: You can use the routers parameter to configure multiple routers, one for each Virtual Routing and Forwarding (VRF) resource. For each router, you must define the Autonomous System Number (ASN).

You can also define a list of Border Gateway Protocol (BGP) neighbors to connect to, as in the following example:

Example FRRConfiguration CR

apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
  name: test
  namespace: frr-k8s-system
spec:
  bgp:
    routers:
    - asn: 64512
      neighbors:
      - address: 172.30.0.3
        asn: 4200000000
        ebgpMultiHop: true
        port: 180
      - address: 172.18.0.6
        asn: 4200000000
        port: 179
# ...

The toAdvertise parameter: By default, FRR-K8s does not advertise the prefixes configured as part of a router configuration. To advertise the prefixes, you use the toAdvertise parameter.

You can advertise a subset of the prefixes, as in the following example:

Example FRRConfiguration CR

apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
  name: test
  namespace: frr-k8s-system
spec:
  bgp:
    routers:
    - asn: 64512
      neighbors:
      - address: 172.30.0.3
        asn: 4200000000
        ebgpMultiHop: true
        port: 180
        toAdvertise:
          allowed:
            prefixes:
            - 192.168.2.0/24
      prefixes:
        - 192.168.2.0/24
        - 192.169.2.0/24
# ...

allowed.prefixes: Advertises a subset of prefixes.

The following example shows you how to advertise all of the prefixes:

Example FRRConfiguration CR

apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
  name: test
  namespace: frr-k8s-system
spec:
  bgp:
    routers:
    - asn: 64512
      neighbors:
      - address: 172.30.0.3
        asn: 4200000000
        ebgpMultiHop: true
        port: 180
        toAdvertise:
          allowed:
            mode: all
      prefixes:
        - 192.168.2.0/24
        - 192.169.2.0/24
# ...

allowed.mode: Advertises all prefixes.

The toReceive parameter

By default, FRR-K8s does not process any prefixes advertised by a neighbor. You can use the toReceive parameter to process such addresses.

You can configure for a subset of the prefixes, as in this example:

Example FRRConfiguration CR

apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
  name: test
  namespace: frr-k8s-system
spec:
  bgp:
    routers:
    - asn: 64512
      neighbors:
      - address: 172.18.0.5
          asn: 64512
          port: 179
          toReceive:
            allowed:
              prefixes:
              - prefix: 192.168.1.0/24
              - prefix: 192.169.2.0/24
                ge: 25
                le: 28
# ...

prefixes: The prefix is applied if the prefix length is less than or equal to the le prefix length and greater than or equal to the ge prefix length.

The following example configures FRR to handle all the prefixes announced:

Example FRRConfiguration CR

apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
  name: test
  namespace: frr-k8s-system
spec:
  bgp:
    routers:
    - asn: 64512
      neighbors:
      - address: 172.18.0.5
          asn: 64512
          port: 179
          toReceive:
            allowed:
              mode: all
# ...

The bgp parameter: You can use the bgp parameter to define various BFD profiles and associate them with a neighbor. In the following example, BFD backs up the BGP session and FRR can detect link failures:

Example FRRConfiguration CR

apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
  name: test
  namespace: frr-k8s-system
spec:
  bgp:
    routers:
    - asn: 64512
      neighbors:
      - address: 172.30.0.3
        asn: 64512
        port: 180
        bfdProfile: defaultprofile
    bfdProfiles:
      - name: defaultprofile
# ...

The nodeSelector parameter: By default, FRR-K8s applies the configuration to all nodes where the daemon is running. You can use the nodeSelector parameter to specify the nodes to which you want to apply the configuration. For example:

Example FRRConfiguration CR

apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
  name: test
  namespace: frr-k8s-system
spec:
  bgp:
    routers:
    - asn: 64512
  nodeSelector:
    labelSelector:
    foo: "bar"
# ...

The fields for the FRRConfiguration custom resource are described in the following table:

Table 1. MetalLB FRRConfiguration custom resource
Parameter	Type	Description
`spec.bgp.routers`	`array`	Specifies the routers that FRR is to configure (one per VRF).
`spec.bgp.routers.asn`	`integer`	The Autonomous System Number (ASN) to use for the local end of the session.
`spec.bgp.routers.id`	`string`	Specifies the ID of the `bgp` router.
`spec.bgp.routers.vrf`	`string`	Specifies the host VRF used to establish sessions from this router.
`spec.bgp.routers.neighbors`	`array`	Specifies the neighbors to establish BGP sessions with.
`spec.bgp.routers.neighbors.asn`	`integer`	Specifies the ASN to use for the remote end of the session.
`spec.bgp.routers.neighbors.address`	`string`	Specifies the IP address to establish the session with.
`spec.bgp.routers.neighbors.port`	`integer`	Specifies the port to dial when establishing the session. Defaults to `179`.
`spec.bgp.routers.neighbors.password`	`string`	Specifies the password to use for establishing the BGP session. `Password` and `PasswordSecret` are mutually exclusive.
`spec.bgp.routers.neighbors.passwordSecret`	`string`	Specifies the name of the authentication secret for the neighbor. The secret must be of type "kubernetes.io/basic-auth", and in the same namespace as the FRR-K8s daemon. The key "password" stores the password in the secret. `Password` and `PasswordSecret` are mutually exclusive.
`spec.bgp.routers.neighbors.holdTime`	`duration`	Specifies the requested BGP hold time, per RFC4271. Defaults to 180s.
`spec.bgp.routers.neighbors.keepaliveTime`	`duration`	Specifies the requested BGP keepalive time, per RFC4271. Defaults to `60s`.
`spec.bgp.routers.neighbors.connectTime`	`duration`	Specifies how long BGP waits between connection attempts to a neighbor.
`spec.bgp.routers.neighbors.ebgpMultiHop`	`boolean`	Indicates if the BGPPeer is a multi-hop away.
`spec.bgp.routers.neighbors.bfdProfile`	`string`	Specifies the name of the BFD Profile to use for the BFD session associated with the BGP session. If not set, the BFD session is not set up.
`spec.bgp.routers.neighbors.toAdvertise.allowed`	`array`	Represents the list of prefixes to advertise to a neighbor, and the associated properties.
`spec.bgp.routers.neighbors.toAdvertise.allowed.prefixes`	`string array`	Specifies the list of prefixes to advertise to a neighbor. This list must match the prefixes that you define in the router.
`spec.bgp.routers.neighbors.toAdvertise.allowed.mode`	`string`	Specifies the mode to use when handling the prefixes. You can set to `filtered` to allow only the prefixes in the prefixes list. You can set to `all` to allow all the prefixes configured on the router.
`spec.bgp.routers.neighbors.toAdvertise.withLocalPref`	`array`	Specifies the prefixes associated with an advertised local preference. You must specify the prefixes associated with a local preference in the prefixes allowed to be advertised.
`spec.bgp.routers.neighbors.toAdvertise.withLocalPref.prefixes`	`string array`	Specifies the prefixes associated with the local preference.
`spec.bgp.routers.neighbors.toAdvertise.withLocalPref.localPref`	`integer`	Specifies the local preference associated with the prefixes.
`spec.bgp.routers.neighbors.toAdvertise.withCommunity`	`array`	Specifies the prefixes associated with an advertised BGP community. You must include the prefixes associated with a local preference in the list of prefixes that you want to advertise.
`spec.bgp.routers.neighbors.toAdvertise.withCommunity.prefixes`	`string array`	Specifies the prefixes associated with the community.
`spec.bgp.routers.neighbors.toAdvertise.withCommunity.community`	`string`	Specifies the community associated with the prefixes.
`spec.bgp.routers.neighbors.toReceive`	`array`	Specifies the prefixes to receive from a neighbor.
`spec.bgp.routers.neighbors.toReceive.allowed`	`array`	Specifies the information that you want to receive from a neighbor.
`spec.bgp.routers.neighbors.toReceive.allowed.prefixes`	`array`	Specifies the prefixes allowed from a neighbor.
`spec.bgp.routers.neighbors.toReceive.allowed.mode`	`string`	Specifies the mode to use when handling the prefixes. When set to `filtered`, only the prefixes in the `prefixes` list are allowed. When set to `all`, all the prefixes configured on the router are allowed.
`spec.bgp.routers.neighbors.disableMP`	`boolean`	Disables MP BGP to prevent it from separating IPv4 and IPv6 route exchanges into distinct BGP sessions.
`spec.bgp.routers.prefixes`	`string array`	Specifies all prefixes to advertise from this router instance.
`spec.bgp.bfdProfiles`	`array`	Specifies the list of BFD profiles to use when configuring the neighbors.
`spec.bgp.bfdProfiles.name`	`string`	The name of the BFD Profile to be referenced in other parts of the configuration.
`spec.bgp.bfdProfiles.receiveInterval`	`integer`	Specifies the minimum interval at which this system can receive control packets, in milliseconds. Defaults to `300ms`.
`spec.bgp.bfdProfiles.transmitInterval`	`integer`	Specifies the minimum transmission interval, excluding jitter, that this system wants to use to send BFD control packets, in milliseconds. Defaults to `300ms`.
`spec.bgp.bfdProfiles.detectMultiplier`	`integer`	Configures the detection multiplier to determine packet loss. To determine the connection loss-detection timer, multiply the remote transmission interval by this value.
`spec.bgp.bfdProfiles.echoInterval`	`integer`	Configures the minimal echo receive transmission-interval that this system can handle, in milliseconds. Defaults to `50ms`.
`spec.bgp.bfdProfiles.echoMode`	`boolean`	Enables or disables the echo transmission mode. This mode is disabled by default, and not supported on multihop setups.
`spec.bgp.bfdProfiles.passiveMode`	`boolean`	Mark session as passive. A passive session does not attempt to start the connection and waits for control packets from peers before it begins replying.
`spec.bgp.bfdProfiles.MinimumTtl`	`integer`	For multihop sessions only. Configures the minimum expected TTL for an incoming BFD control packet.
`spec.nodeSelector`	`string`	Limits the nodes that attempt to apply this configuration. If specified, only those nodes whose labels match the specified selectors attempt to apply the configuration. If it is not specified, all nodes attempt to apply this configuration.
`status`	`string`	Defines the observed state of FRRConfiguration.

How FRR-K8s merges multiple configurations

FRR-K8s uses an additive merge strategy when multiple users configure the same node. By using FRR-K8s, you can extend existing configurations, such as adding neighbors or prefixes, but prevent the removal of components defined by other sources.

Configuration conflicts

Certain configurations can cause conflicts, leading to errors, for example:

different ASN for the same router (in the same VRF)
different ASN for the same neighbor (with the same IP / port)
multiple BFD profiles with the same name but different values

When the daemon finds an invalid configuration for a node, it reports the configuration as invalid and reverts to the previous valid FRR configuration.

Merging

When merging, iyou can complete the following actions:

Extend the set of IP addresses that you want to advertise to a neighbor.
Add an extra neighbor with its set of IP addresses.
Extend the set of IP addresses to which you want to associate a community.
Allow incoming routes for a neighbor.

Each configuration must be self contained. This means, for example, that you cannot allow prefixes that are not defined in the router section by leveraging prefixes coming from another configuration.

If the configurations to be applied are compatible, merging works as follows:

FRR-K8s combines all the routers.
FRR-K8s merges all prefixes and neighbors for each router.
FRR-K8s merges all filters for each neighbor.

A less restrictive filter has precedence over a stricter one. For example, a filter accepting some prefixes has precedence over a filter not accepting any, and a filter accepting all prefixes has precedence over one that accepts some.