Routes - Core Concepts | Architecture | OpenShift Enterprise 3.0

Overview
Routers
- Template Routers
Available Router Plug-ins
- HAProxy Template Router
- F5 Router
Route Host Names
Route Types
Path Based Routes
Secured Routes

Overview

An OpenShift route is a way to expose a service by giving it an externally-reachable hostname like www.example.com.

A defined route and the endpoints identified by its service can be consumed by a router to provide named connectivity that allows external clients to reach your applications. Each route consists of a route name (limited to 63 characters), service selector, and (optionally) security configuration.

Routers

An OpenShift administrator can deploy routers in an OpenShift cluster, which enable routes created by developers to be used by external clients. The routing layer in OpenShift is pluggable, and two available router plug-ins are provided and supported by default.

OpenShift routers provide external host name mapping and load balancing to services over protocols that pass distinguishing information directly to the router; the host name must be present in the protocol in order for the router to determine where to send it.

Router plug-ins assume they can bind to host ports 80 and 443. This is to allow external traffic to route to the host and subsequently through the router. Routers also assume that networking is configured such that it can access all pods in the cluster.

Routers support the following protocols:

HTTP
HTTPS (with SNI)
WebSockets
TLS with SNI

WebSocket traffic uses the same route conventions and supports the same TLS termination types as other traffic.

A router uses the service selector to find the service and the endpoints backing the service. service-provided load balancing is bypassed and replaced with the router’s own load balancing. Routers watch the cluster API and automatically update their own configuration according to any relevant changes in the API objects. Routers may be containerized or virtual. Custom routers can be deployed to communicate modifications of API objects to an external routing solution.

In order to reach a router in the first place, requests for host names must resolve via DNS to a router or set of routers. The suggested method is to define a cloud domain with a wildcard DNS entry pointing to a virtual IP backed by multiple router instances on designated nodes. Router VIP configuration is described in the Administration Guide. DNS for addresses outside the cloud domain would need to be configured individually. Other approaches may be feasible.

Template Routers

A template router is a type of router that provides certain infrastructure information to the underlying router implementation, such as:

A wrapper that watches endpoints and routes.
Endpoint and route data, which is saved into a consumable form.
Passing the internal state to a configurable template and executing the template.
Calling a reload script.

Available Router Plug-ins

The following router plug-ins are provided and supported in OpenShift. Instructions on deploying these routers are available in Deploying a Router.

HAProxy Template Router

The HAProxy template router implementation is the reference implementation for a template router plug-in. It uses the openshift3/ose-haproxy-router repository to run an HAProxy instance alongside the template router plug-in.

The following diagram illustrates how data flows from the master through the plug-in and finally into an HAProxy configuration:

Figure 1. HAProxy Router Data Flow

Sticky Sessions

Implementing sticky sessions is up to the underlying router configuration. The default HAProxy template implements sticky sessions using the balance source directive which balances based on the source IP. In addition, the template router plug-in provides the service name and namespace to the underlying implementation. This can be used for more advanced configuration such as implementing stick-tables that synchronize between a set of peers.

Specific configuration for this router implementation is stored in the haproxy-config.template file located in the /var/lib/haproxy/conf directory of the router container.

The balance source directive does not distinguish between external client IP addresses; because of the NAT configuration, the originating IP address (HAProxy remote) is the same. Unless the HAProxy router is running with hostNetwork: true, all external clients will be routed to a single pod.

F5 Router

The F5 router plug-in is available starting in OpenShift Enterprise 3.0.2.

The F5 router plug-in integrates with an existing F5 BIG-IP® system in your environment. F5 BIG-IP® version 11.4 or newer is required in order to have the F5 iControl REST API. The F5 router supports unsecured, edge terminated, re-encryption terminated, and passthrough terminated routes matching on HTTP vhost and request path.

The F5 router has additional features over the F5 BIG-IP® support in OpenShift Enterprise 2. When comparing with the OpenShift routing-daemon used in earlier versions, the F5 router additionally supports:

path-based routing (using policy rules),
passthrough of encrypted connections (implemented using an iRule that parses the SNI protocol and uses a data group that is maintained by the F5 router for the servername lookup).

Passthrough routes are a special case: path-based routing is technically impossible with passthrough routes because F5 BIG-IP® itself does not see the HTTP request, so it cannot examine the path. The same restriction applies to the template router; it is a technical limitation of passthrough encryption, not a technical limitation of OpenShift.

Routing Traffic to Pods Through the SDN

Because F5 BIG-IP® is external to the OpenShift SDN, a cluster administrator must create a peer-to-peer tunnel between F5 BIG-IP® and a host that is on the SDN, typically an OpenShift node host. This ramp node can be configured as unschedulable for pods so that it will not be doing anything except act as a gateway for the F5 BIG-IP® host. It is also possible to configure multiple such hosts and use the OpenShift ipfailover feature for redundancy; the F5 BIG-IP® host would then need to be configured to use the ipfailover VIP for its tunnel’s remote endpoint.

F5 Integration Details

The operation of the F5 router is similar to that of the OpenShift routing-daemon used in earlier versions. Both use REST API calls to:

create and delete pools,
add endpoints to and delete them from those pools, and
configure policy rules to route to pools based on vhost.

Both also use scp and ssh commands to upload custom TLS/SSL certificates to F5 BIG-IP®.

The F5 router configures pools and policy rules on virtual servers as follows:

When a user creates or deletes a route on OpenShift, the router creates a pool to F5 BIG-IP® for the route (if no pool already exists) and adds a rule to, or deletes a rule from, the policy of the appropriate vserver: the HTTP vserver for non-TLS routes, or the HTTPS vserver for edge or re-encrypt routes. In the case of edge and re-encrypt routes, the router also uploads and configures the TLS certificate and key. The router supports host- and path-based routes.

Passthrough routes are a special case: to support those, it is necessary to write an iRule that parses the SNI ClientHello handshake record and looks up the servername in an F5 data-group. The router creates this iRule, associates the iRule with the vserver, and updates the F5 data-group as passthrough routes are created and deleted. Other than this implementation detail, passthrough routes work the same way as other routes.

When a user creates a service on OpenShift, the router adds a pool to F5 BIG-IP® (if no pool already exists). As endpoints on that service are created and deleted, the router adds and removes corresponding pool members.
When a user deletes the route and all endpoints associated with a particular pool, the router deletes that pool.

Route Host Names

In order for services to be exposed externally, an OpenShift route allows you to associate a service with an externally-reachable host name. This edge host name is then used to route traffic to the service.

When two routes claim the same host, the oldest route wins. If additional routes with different path fields are defined in the same namespace, those paths will be added. If multiple routes with the same path are used, the oldest takes priority.

Example 1. A Route with a Specified Host:

apiVersion: v1
kind: Route
metadata:
  name: host-route
spec:
  host: www.example.com  (1)
  to:
    kind: service
    name: service-name

1	Specifies the externally-reachable host name used to expose a service.

Example 2. A Route Without a Host:

apiVersion: v1
kind: Route
metadata:
  name: no-route-hostname
spec:
  to:
    kind: service
    name: service-name

If a host name is not provided as part of the route definition, then OpenShift automatically generates one for you. The generated host name is of the form:

<route-name>[-<namespace>].<suffix>

The following example shows the OpenShift-generated host name for the above configuration of a route without a host added to a namespace mynamespace:

Example 3. Generated Host Name

no-route-hostname-mynamespace.router.default.svc.cluster.local (1)

1	The generated host name suffix is the default routing subdomain router.default.svc.cluster.local.

A cluster administrator can also customize the suffix used as the default routing subdomain for their environment.

Route Types

Routes can be either secured or unsecured. Secure routes provide the ability to use several types of TLS termination to serve certificates to the client. Routers support edge, passthrough, and re-encryption termination.

Example 4. Unsecured Route Object YAML Definition

apiVersion: v1
kind: Route
metadata:
  name: route-unsecured
spec:
  host: www.example.com
  to:
    kind: service
    name: service-name

Unsecured routes are simplest to configure, as they require no key or certificates, but secured routes offer security for connections to remain private.

A secured route is one that specifies the TLS termination of the route. The available types of termination are described below.

Path Based Routes

Path based routes specify a path component that can be compared against a URL (which requires that the traffic for the route be HTTP based) such that multiple routes can be served using the same hostname, each with a different path. Routers should match routes based on the most specific path to the least; however, this depends on the router implementation. The following table shows example routes and their accessibility:

Table 1. Route Availability
Route	When Compared to	Accessible
www.example.com/test	www.example.com/test	Yes
www.example.com/test	www.example.com	No
www.example.com/test and www.example.com	www.example.com/test	Yes
www.example.com/test and www.example.com	www.example.com	Yes
www.example.com	www.example.com/test	Yes (Matched by the host, not the route)
www.example.com	www.example.com	Yes

Example 5. An Unsecured Route with a Path:

apiVersion: v1
kind: Route
metadata:
  name: route-unsecured
spec:
  host: www.example.com
  path: "/test"   (1)
  to:
    kind: service
    name: service-name

1	The path is the only added attribute for a path-based route.

Path-based routing is not available when using passthrough TLS, as the router does not terminate TLS in that case and cannot read the contents of the request.

Secured Routes

Secured routes specify the TLS termination of the route and, optionally, provide a key and certificate(s).

TLS termination in OpenShift relies on SNI for serving custom certificates. Any non-SNI traffic received on port 443 is handled with TLS termination and a default certificate (which may not match the requested hostname, resulting in validation errors).

Secured routes can use any of the following three types of secure TLS termination.

Edge Termination

With edge termination, TLS termination occurs at the router, prior to proxying traffic to its destination. TLS certificates are served by the front end of the router, so they must be configured into the route, otherwise the router’s default certificate will be used for TLS termination.

Example 6. A Secured Route Using Edge Termination

apiVersion: v1
kind: Route
metadata:
  name: route-edge-secured (1)
spec:
  host: www.example.com
  to:
    kind: service
    name: service-name (1)
  tls:
    termination: edge            (2)
    key: |-                      (3)
      -----BEGIN PRIVATE KEY-----
      [...]
      -----END PRIVATE KEY-----
    certificate: |-              (4)
      -----BEGIN CERTIFICATE-----
      [...]
      -----END CERTIFICATE-----
    caCertificate: |-            (5)
      -----BEGIN CERTIFICATE-----
      [...]
      -----END CERTIFICATE-----

1	The name of the object, which is limited to 63 characters.
2	The `termination` field is `edge` for edge termination.
3	The `key` field is the contents of the PEM format key file.
4	The `certificate` field is the contents of the PEM format certificate file.
5	An optional CA certificate may be required to establish a certificate chain for validation.

Because TLS is terminated at the router, connections from the router to the endpoints over the internal network are not encrypted.

Passthrough Termination

With passthrough termination, encrypted traffic is sent straight to the destination without the router providing TLS termination. Therefore no key or certificate is required.

Example 7. A Secured Route Using Passthrough Termination

apiVersion: v1
kind: Route
metadata:
  name: route-passthrough-secured (1)
spec:
  host: www.example.com
  to:
    kind: service
    name: service-name (1)
  tls:
    termination: passthrough     (2)

1	The name of the object, which is limited to 63 characters. <2>The `termination` field is set to `passthrough`. No other encryption fields are needed.

The destination pod is responsible for serving certificates for the traffic at the endpoint. This is currently the only method that can support requiring client certificates (also known as two-way authentication).

Re-encryption Termination

Re-encryption is a variation on edge termination where the router terminates TLS with a certificate, then re-encrypts its connection to the endpoint which may have a different certificate. Therefore the full path of the connection is encrypted, even over the internal network. The router uses health checks to determine the authenticity of the host.

Example 8. A Secured Route Using Re-Encrypt Termination

apiVersion: v1
kind: Route
metadata:
  name: route-pt-secured (1)
spec:
  host: www.example.com
  to:
    kind: service
    name: service-name (1)
  tls:
    termination: reencrypt        (2)
    key: [as in edge termination]
    certificate: [as in edge termination]
    caCertificate: [as in edge termination]
    destinationCACertificate: |-  (3)
      -----BEGIN CERTIFICATE-----
      [...]
      -----END CERTIFICATE-----

1	The name of the object, which is limited to 63 characters.
2	The `termination` field is set to `reencrypt`. Other fields are as in edge termination.
3	The `destinationCACertificate` field specifies a CA certificate to validate the endpoint certificate, securing the connection from the router to the destination. This field is required, but only for re-encryption.