An OpenShift Container Platform route exposes a service at a host name, like www.example.com, so that external clients can reach it by name.
DNS resolution for a host name is handled separately from routing. Your administrator may have configured a DNS wildcard entry that will resolve to the OpenShift Container Platform node that is running the OpenShift Container Platform router. If you are using a different host name you may need to modify its DNS records independently to resolve to the node that is running the router.
Each route consists of a name (limited to 63 characters), a service selector, and an optional security configuration.
An OpenShift Container Platform administrator can deploy routers to nodes in an OpenShift Container Platform cluster, which enable routes created by developers to be used by external clients. The routing layer in OpenShift Container Platform is pluggable, and two available router plug-ins are provided and supported by default.
See the Installation and Configuration guide for information on deploying a router. |
A router uses the service selector to find the service and the endpoints backing the service. When both router and service provide load balancing, OpenShift Container Platform uses the router load balancing. A router detects relevant changes in the IP addresses of its services and adapts its configuration accordingly. This is useful for custom routers to communicate modifications of API objects to an external routing solution.
The path of a request starts with the DNS resolution of a host name to one or more routers. The suggested method is to define a cloud domain with a wildcard DNS entry pointing to one or more virtual IP (VIP) addresses backed by multiple router instances. Routes using names and addresses outside the cloud domain require configuration of individual DNS entries.
When there are fewer VIP addresses than routers, the routers corresponding to the number of addresses are active and the rest are passive. A passive router is also known as a hot-standby router. For example, with two VIP addresses and three routers, you have an "active-active-passive" configuration. See High Availability for more information on router VIP configuration.
Routes can be sharded among the set of routers. Administrators can set up sharding on a cluster-wide basis and users can set up sharding for the namespace in their project. Sharding allows the operator to define multiple router groups. Each router in the group serves only a subset of traffic.
OpenShift Container Platform routers provide external host name mapping and load balancing of service end points over protocols that pass distinguishing information directly to the router; the host name must be present in the protocol in order for the router to determine where to send it.
Router plug-ins assume they can bind to host ports 80 (HTTP)
and 443 (HTTPS), by default.
This means that routers must be placed on nodes
where those ports are not otherwise in use.
Alternatively, a router can be configured to listen
on other ports by setting the ROUTER_service_HTTP_PORT
and ROUTER_service_HTTPS_PORT
environment variables.
Because a router binds to ports on the host node, only one router listening on those ports can be on each node if the router uses host networking (the default). Cluster networking is configured such that all routers can access all pods in the cluster.
Routers support the following protocols:
HTTP
HTTPS (with SNI)
WebSockets
TLS with SNI
WebSocket traffic uses the same route conventions and supports the same TLS termination types as other traffic. |
A template router is a type of router that provides certain infrastructure information to the underlying router implementation, such as:
A wrapper that watches endpoints and routes.
Endpoint and route data, which is saved into a consumable form.
Passing the internal state to a configurable template and executing the template.
Calling a reload script.
The following router plug-ins are provided and supported in OpenShift Container Platform. Instructions on deploying these routers are available in Deploying a Router.
The HAProxy template router implementation is the reference implementation for a template router plug-in. It uses the openshift3/ose-haproxy-router repository to run an HAProxy instance alongside the template router plug-in.
The following diagram illustrates how data flows from the master through the plug-in and finally into an HAProxy configuration:
Sticky Sessions
Implementing sticky sessions is up to the underlying router configuration. The
default HAProxy template implements sticky sessions using the balance source
directive which balances based on the source IP. In addition, the template
router plug-in provides the service name and namespace to the underlying
implementation. This can be used for more advanced configuration such as
implementing stick-tables that synchronize between a set of peers.
Specific configuration for this router implementation is stored in the haproxy-config.template file located in the /var/lib/haproxy/conf directory of the router container.
The |
For all the items outlined in this section, you can set environment
variables on the deployment config for the router to alter its configuration, or use the oc set env
command:
$ oc set env <object_type>/<object_name> KEY1=VALUE1 KEY2=VALUE2
For example:
$ oc set env dc/router ROUTER_SYSLOG_ADDRESS=127.0.0.1 ROUTER_LOG_LEVEL=debug
Variable | Default | Description |
---|---|---|
|
The contents of a default certificate to use for routes that don’t expose a TLS server cert; in PEM format. |
|
|
A path to a directory that contains a file named tls.crt. If tls.crt is not a PEM file which also contains a private key, it is first combined with a file named tls.key in the same directory. The PEM-format contents are then used as the default certificate. Only used if |
|
|
A path to default certificate to use for routes that don’t expose a TLS server cert; in PEM format. Only used if |
|
|
|
If |
|
A label selector to apply to namespaces to watch, empty means all. |
|
|
A label selector to apply to projects to watch, emtpy means all. |
|
|
The path to the reload script to use to reload the router. |
|
|
5000ms |
Length of time between subsequent "liveness" checks on backends. (TimeUnits) |
|
|
If set to |
|
30s |
Length of time within which a client has to acknowledge or send data. (TimeUnits) |
|
5s |
The maximum connect time. (TimeUnits) |
|
30s |
Length of time within which a server has to acknowledge or send data. (TimeUnits) |
|
1h |
Length of time till which TCP or WebSocket connections will remain open. If you have websockets/tcp connections (and any time HAProxy is reloaded), the old HAProxy processes will "linger" around for that period. (TimeUnits) |
|
warning |
The log level to send to the syslog server. |
|
If set |
|
|
443 |
Port to listen for HTTPS requests. |
|
80 |
Port to listen for HTTP requests. |
|
public |
The name that the router identifies itself in the in route status. |
|
The namespace the router identifies itself in the in route status. Required if |
|
|
10443 |
Internal port for some front-end to back-end communication (see note below). |
|
10444 |
Internal port for some front-end to back-end communication (see note below). |
|
10s |
Length of time the transmission of an HTTP request can take. (TimeUnits) |
|
The template that should be used to generate the host name for a route without spec.host (e.g. ${name}-${namespace}.myapps.mycompany.com). |
|
|
Address to send log messages. Disabled if empty. |
|
|
source |
Load-balancing strategy for multiple endpoints for pass-through routes. Available options are |
|
A label selector to apply to the routes to watch, empty means all. |
|
|
The password needed to access router stats (if the router implementation supports it). |
|
|
Port to expose statistics on (if the router implementation supports it). If not set, stats are not exposed. |
|
|
The user name needed to access router stats (if the router implementation supports it). |
|
|
|
The path to the HAProxy template file (in the container image). |
|
12s |
The minimum frequency the router is allowed to reload to accept new changes. (TimeUnits) |
|
When set to |
TimeUnits are represented by a number followed by the unit: us
(microseconds), ms
(milliseconds, default), s
(seconds), m
(minutes), h
*(hours), d
(days). The regular expression is:
[1-9][0-9](us\|ms\|s\|m\|h\|d)
If you want to run multiple routers on the same machine, you must change the
ports that the router is listening on, |
The F5 router plug-in is available starting in OpenShift Enterprise 3.0.2. |
The F5 router plug-in integrates with an existing F5 BIG-IP® system in your environment. F5 BIG-IP® version 11.4 or newer is required in order to have the F5 iControl REST API. The F5 router supports unsecured, edge terminated, re-encryption terminated, and passthrough terminated routes matching on HTTP vhost and request path.
The F5 router has feature parity with the HAProxy template router, and has additional features over the F5 BIG-IP® support in OpenShift Enterprise 2. Compared with the routing-daemon used in earlier versions, the F5 router additionally supports:
path-based routing (using policy rules),
re-encryption (implemented using client and server SSL profiles), and
passthrough of encrypted connections (implemented using an iRule that parses the SNI protocol and uses a data group that is maintained by the F5 router for the servername lookup).
Passthrough routes are a special case: path-based routing is technically impossible with passthrough routes because F5 BIG-IP® itself does not see the HTTP request, so it cannot examine the path. The same restriction applies to the template router; it is a technical limitation of passthrough encryption, not a technical limitation of OpenShift Container Platform. |
Routing Traffic to Pods Through the SDN
Because F5 BIG-IP® is external to the OpenShift Container Platform SDN, a cluster administrator must create a peer-to-peer tunnel between F5 BIG-IP® and a host that is on the SDN, typically an OpenShift Container Platform node host. This ramp node can be configured as unschedulable for pods so that it will not be doing anything except act as a gateway for the F5 BIG-IP® host. It is also possible to configure multiple such hosts and use the OpenShift Container Platform ipfailover feature for redundancy; the F5 BIG-IP® host would then need to be configured to use the ipfailover VIP for its tunnel’s remote endpoint.
F5 Integration Details
The operation of the F5 router is similar to that of the OpenShift Container Platform routing-daemon used in earlier versions. Both use REST API calls to:
create and delete pools,
add endpoints to and delete them from those pools, and
configure policy rules to route to pools based on vhost.
Both also use scp
and ssh
commands to upload custom TLS/SSL certificates to
F5 BIG-IP®.
The F5 router configures pools and policy rules on virtual servers as follows:
When a user creates or deletes a route on OpenShift Container Platform, the router creates a pool to F5 BIG-IP® for the route (if no pool already exists) and adds a rule to, or deletes a rule from, the policy of the appropriate vserver: the HTTP vserver for non-TLS routes, or the HTTPS vserver for edge or re-encrypt routes. In the case of edge and re-encrypt routes, the router also uploads and configures the TLS certificate and key. The router supports host- and path-based routes.
Passthrough routes are a special case: to support those, it is necessary to write an iRule that parses the SNI ClientHello handshake record and looks up the servername in an F5 data-group. The router creates this iRule, associates the iRule with the vserver, and updates the F5 data-group as passthrough routes are created and deleted. Other than this implementation detail, passthrough routes work the same way as other routes. |
When a user creates a service on OpenShift Container Platform, the router adds a pool to F5 BIG-IP® (if no pool already exists). As endpoints on that service are created and deleted, the router adds and removes corresponding pool members.
When a user deletes the route and all endpoints associated with a particular pool, the router deletes that pool.
In order for services to be exposed externally, an OpenShift Container Platform route allows you to associate a service with an externally-reachable host name. This edge host name is then used to route traffic to the service.
When multiple routes from different namespaces claim the same host, the oldest route wins and claims it for the namespace. If additional routes with different path fields are defined in the same namespace, those paths are added. If multiple routes with the same path are used, the oldest takes priority.
A consequence of this behavior is that if you have two routes for a hostname: an older one and a newer one. If someone else has a route for the same hostname that they created between when you created the other two routes, then if you delete your older route, your claim to the hostname will no longer be in effect. The other namespace now claims the hostname and your claim is lost.
apiVersion: v1
kind: Route
metadata:
name: host-route
spec:
host: www.example.com (1)
to:
kind: service
name: service-name
1 | Specifies the externally-reachable host name used to expose a service. |
apiVersion: v1
kind: Route
metadata:
name: no-route-hostname
spec:
to:
kind: service
name: service-name
If a host name is not provided as part of the route definition, then OpenShift Container Platform automatically generates one for you. The generated host name is of the form:
<route-name>[-<namespace>].<suffix>
The following example shows the OpenShift Container Platform-generated host name for the above configuration of a route without a host added to a namespace mynamespace:
no-route-hostname-mynamespace.router.default.svc.cluster.local (1)
1 | The generated host name suffix is the default routing subdomain router.default.svc.cluster.local. |
A cluster administrator can also customize the suffix used as the default routing subdomain for their environment.
Routes can be either secured or unsecured. Secure routes provide the ability to use several types of TLS termination to serve certificates to the client. Routers support edge, passthrough, and re-encryption termination.
apiVersion: v1
kind: Route
metadata:
name: route-unsecured
spec:
host: www.example.com
to:
kind: service
name: service-name
Unsecured routes are simplest to configure, as they require no key or certificates, but secured routes offer security for connections to remain private.
A secured route is one that specifies the TLS termination of the route. The available types of termination are described below.
Path based routes specify a path component that can be compared against a URL (which requires that the traffic for the route be HTTP based) such that multiple routes can be served using the same hostname, each with a different path. Routers should match routes based on the most specific path to the least; however, this depends on the router implementation. The following table shows example routes and their accessibility:
Route | When Compared to | Accessible |
---|---|---|
www.example.com/test |
www.example.com/test |
Yes |
www.example.com |
No |
|
www.example.com/test and www.example.com |
www.example.com/test |
Yes |
www.example.com |
Yes |
|
www.example.com |
www.example.com/test |
Yes (Matched by the host, not the route) |
www.example.com |
Yes |
apiVersion: v1
kind: Route
metadata:
name: route-unsecured
spec:
host: www.example.com
path: "/test" (1)
to:
kind: service
name: service-name
1 | The path is the only added attribute for a path-based route. |
Path-based routing is not available when using passthrough TLS, as the router does not terminate TLS in that case and cannot read the contents of the request. |
Secured routes specify the TLS termination of the route and, optionally, provide a key and certificate(s).
TLS termination in OpenShift Container Platform relies on SNI for serving custom certificates. Any non-SNI traffic received on port 443 is handled with TLS termination and a default certificate (which may not match the requested hostname, resulting in validation errors). |
Secured routes can use any of the following three types of secure TLS termination.
Edge Termination
With edge termination, TLS termination occurs at the router, prior to proxying traffic to its destination. TLS certificates are served by the front end of the router, so they must be configured into the route, otherwise the router’s default certificate will be used for TLS termination.
apiVersion: v1
kind: Route
metadata:
name: route-edge-secured (1)
spec:
host: www.example.com
to:
kind: service
name: service-name (1)
tls:
termination: edge (2)
key: |- (3)
-----BEGIN PRIVATE KEY-----
[...]
-----END PRIVATE KEY-----
certificate: |- (4)
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
caCertificate: |- (5)
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
1 | The name of the object, which is limited to 63 characters. |
2 | The termination field is edge for edge termination. |
3 | The key field is the contents of the PEM format key file. |
4 | The certificate field is the contents of the PEM format certificate file. |
5 | An optional CA certificate may be required to establish a certificate chain for validation. |
Because TLS is terminated at the router, connections from the router to the endpoints over the internal network are not encrypted.
Edge-terminated routes can specify an insecureEdgeTerminationPolicy
that
enables traffic on insecure schemes (HTTP
) to be disabled, allowed or
redirected.
The allowed values for insecureEdgeTerminationPolicy
are:
None
or empty (for disabled), Allow
or Redirect
.
The default insecureEdgeTerminationPolicy
is to disable traffic on the
insecure scheme. A common use case is to allow content to be served via a
secure scheme but serve the assets (example images, stylesheets and
javascript) via the insecure scheme.
apiVersion: v1
kind: Route
metadata:
name: route-edge-secured-allow-insecure (1)
spec:
host: www.example.com
to:
kind: service
name: service-name (1)
tls:
termination: edge (2)
insecureEdgeTerminationPolicy: Allow (3)
[ ... ]
1 | The name of the object, which is limited to 63 characters. |
2 | The termination field is edge for edge termination. |
3 | The insecure policy to allow requests sent on an insecure scheme HTTP . |
apiVersion: v1
kind: Route
metadata:
name: route-edge-secured-redirect-insecure (1)
spec:
host: www.example.com
to:
kind: service
name: service-name (1)
tls:
termination: edge (2)
insecureEdgeTerminationPolicy: Redirect (3)
[ ... ]
1 | The name of the object, which is limited to 63 characters. |
2 | The termination field is edge for edge termination. |
3 | The insecure policy to redirect requests sent on an insecure scheme HTTP to a secure scheme HTTPS . |
Passthrough Termination
With passthrough termination, encrypted traffic is sent straight to the destination without the router providing TLS termination. Therefore no key or certificate is required.
apiVersion: v1
kind: Route
metadata:
name: route-passthrough-secured (1)
spec:
host: www.example.com
to:
kind: service
name: service-name (1)
tls:
termination: passthrough (2)
1 | The name of the object, which is limited to 63 characters. |
2 | The termination field is set to passthrough . No other encryption fields are needed. |
The destination pod is responsible for serving certificates for the traffic at the endpoint. This is currently the only method that can support requiring client certificates (also known as two-way authentication).
Re-encryption Termination
Re-encryption is a variation on edge termination where the router terminates TLS with a certificate, then re-encrypts its connection to the endpoint which may have a different certificate. Therefore the full path of the connection is encrypted, even over the internal network. The router uses health checks to determine the authenticity of the host.
apiVersion: v1
kind: Route
metadata:
name: route-pt-secured (1)
spec:
host: www.example.com
to:
kind: service
name: service-name (1)
tls:
termination: reencrypt (2)
key: [as in edge termination]
certificate: [as in edge termination]
caCertificate: [as in edge termination]
destinationCACertificate: |- (3)
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
1 | The name of the object, which is limited to 63 characters. |
2 | The termination field is set to reencrypt . Other fields are as in edge
termination. |
3 | The destinationCACertificate field specifies a CA certificate to
validate the endpoint certificate, securing the connection from the router to
the destination. This field is required, but only for re-encryption. |
In OpenShift Container Platform, each route can have any number of
labels
in its metadata
field.
A router uses selectors (also known as a selection expression)
to select a subset of routes from the entire pool of routes to serve.
A selection expression can also involve
labels on the route’s namespace.
The selected routes form a router shard.
You can
create
and
modify
router shards independently from the routes, themselves.
This design supports traditional sharding as well as overlapped sharding.
In traditional sharding, the selection results in no overlapping sets
and a route belongs to exactly one shard.
In overlapped sharding, the selection results in overlapping sets
and a route can belong to many different shards.
For example, a single route may belong to a SLA=high
shard
(but not SLA=medium
or SLA=low
shards),
as well as a geo=west
shard
(but not a geo=east
shard).
Another example of overlapped sharding is a set of routers that select based on namespace of the route:
Router | Selection | Namespaces |
---|---|---|
router-1 |
|
|
router-2 |
|
|
router-3 |
|
|
Both router-2
and router-3
serve routes that are in the
namespaces Q*
, R*
, S*
, T*
.
To change this example from overlapped to traditional sharding,
we could change the selection of router-2
to K*
— P*
,
which would eliminate the overlap.
When routers are sharded, a given route is bound to zero or more routers in the group. The route binding ensures uniqueness of the route across the shard. Uniqueness allows secure and non-secure versions of the same route to exist within a single shard. This implies that routes now have a visible life cycle that moves from created to bound to active.
In the sharded environment the first route to hit the shard reserves the right to exist there indefinitely, even across restarts.
During a green/blue deployment a route may be be selected in multiple routers. An OpenShift Container Platform application administrator may wish to bleed traffic from one version of the application to another and then turn off the old version.
Sharding can be done by the administrator at a cluster level and by the user
at a project/namespace level.
When namespace labels are used, the service account for the router
must have cluster-reader
permission to permit the
router to access the labels in the namespace.
For two or more routes that claim the same host name, the resolution order is based on the age of the route and the oldest route would win the claim to that host. In the case of sharded routers, routes are selected based on their labels matching the router’s selection criteria. There is no consistent way to determine when labels are added to a route. So if an older route claiming an existing host name is "re-labelled" to match the router’s selection criteria, it will replace the existing route based on the above mentioned resolution order (oldest route wins). |
Using environment variables as defined in Configuration Parameters, a router can set the default options for all the routes it exposes. An individual route can override some of these defaults by providing specific configurations in its annotations.
Route Annotations
For all the items outlined in this section, you can set annotations on the route definition for the route to alter its configuration
Variable | Description | Environment Variable Used as Default |
---|---|---|
|
Sets the load-balancing algorithm. Available options are |
|
|
Setting |
|
|
Limits the number of concurrent TCP connections shared by an IP address. |
|
|
Limits the rate at which an IP address can make HTTP requests. |
|
|
Limits the rate at which an IP address can make TCP connections. |
|
|
Sets a server-side timeout for the route. (TimeUnits) |
|
|
Sets the interval for the back-end health checks. (TimeUnits) |
|
apiVersion: v1
kind: Route
metadata:
annotations:
haproxy.router.openshift.io/timeout: 5500ms (1)
[...]
1 | Specifies the new timeout with HAProxy supported units (us, ms, s, m, h, d). If unit not provided, ms is the default. |
Setting a server-side timeout value for passthrough routes too low can cause WebSocket connections to timeout frequently on that route. |
The route status
field is only set by routers. If changes are made to a route
so that a router no longer serves a specific route, the status becomes stale.
The routers do not clear the route status
field. To remove the stale entries
in the route status, use the
clear-route-status
script.