Deploying cluster logging | Logging | OpenShift Container Platform 4.1

Installing the Cluster Logging and Elasticsearch Operators
Additional resources

The process for deploying cluster Logging to OpenShift Container Platform involves:

Review the installation options in About deploying cluster logging.
Review the cluster logging storage considerations.
Install the Cluster Logging subscription using the web console.

Installing the Cluster Logging and Elasticsearch Operators

You can use the OpenShift Container Platform console to install cluster logging, by deploying, the Cluster Logging and Elasticsearch Operators. The Cluster Logging Operator creates and manages the components of the logging stack. The Elasticsearch Operator creates and manages the Elasticsearch cluster used by cluster logging.

The OpenShift Container Platform cluster logging solution requires that you install both the Cluster Logging Operator and Elasticsearch Operator. There is no use case in OpenShift Container Platform for installing the operators individually. You must install the Elasticsearch Operator using the CLI following the directions below. You can install the Cluster Logging Operator using the web console or CLI.

Prerequisites

Ensure that you have the necessary persistent storage for Elasticsearch. Note that each Elasticsearch node requires its own storage volume.

Elasticsearch is a memory-intensive application. Each Elasticsearch node needs 16G of memory for both memory requests and limits. The initial set of OpenShift Container Platform nodes might not be large enough to support the Elasticsearch cluster. You must add additional nodes to the OpenShift Container Platform cluster to run with the recommended or higher memory. Each Elasticsearch node can operate with a lower memory setting though this is not recommended for production deployments.

You must install the Elasticsearch Operator using the CLI following the directions below. You can install the Cluster Logging Operator using the web console or CLI.

Procedure

Create Namespaces for the Elasticsearch Operator and Cluster Logging Operator.

You can also create the Namespaces in the web console using the Administration → Namespaces page. You must apply the cluster-logging and cluster-monitoring labels listed in the sample YAML to the namespaces you create.

Create a Namespace for the Elasticsearch Operator (for example, eo-namespace.yaml):

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-operators-redhat (1)
  annotations:
    openshift.io/node-selector: ""
  labels:
    openshift.io/cluster-monitoring: "true" (2)

1 You must specify the openshift-operators-redhat namespace. To prevent possible conflicts with metrics, you should configure the Prometheus Cluster Monitoring stack to scrape metrics from the openshift-operators-redhat namespace and not the openshift-operators namespace. The openshift-operators namespace can contain community operators, which are untrusted and could publish a metric with the same name as an OpenShift Container Platform metric, which would cause conflicts.

2 You must specify this label as shown to ensure that cluster monitoring scrapes the openshift-operators-redhat namespace.

Run the following command to create the namespace:

$ oc create -f <file-name>.yaml

For example:

$ oc create -f eo-namespace.yaml

Create a Namespace for the Cluster Logging Operator (for example, clo-namespace.yaml):

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-logging (1)
  annotations:
    openshift.io/node-selector: "" (1)
  labels:  (1)
    openshift.io/cluster-logging: "true"
    openshift.io/cluster-monitoring: "true"

1	Specify these labels as shown.

Run the following command to create the namespace:

$ oc create -f <file-name>.yaml

For example:

$ oc create -f clo-namespace.yaml

Install the Elasticsearch Operator by creating the following objects:

Create an Operator Group object YAML file (for example, eo-og.yaml) for the Elasticsearch operator:

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: openshift-operators-redhat
  namespace: openshift-operators-redhat (1)
spec: {}

1	You must specify the `openshift-operators-redhat` namespace.

Create an Operator Group object:
```
$ oc create -f eo-og.yaml
```
Create a CatalogSourceConfig object YAML file (for example, eo-csc.yaml) to enable the Elasticsearch Operator on the cluster.
Example CatalogSourceConfig
```
apiVersion: "operators.coreos.com/v1"
kind: "CatalogSourceConfig"
metadata:
  name: "elasticsearch"
  namespace: "openshift-marketplace"
spec:
  targetNamespace: "openshift-operators-redhat" (1)
  packages: "elasticsearch-operator"
  source: "redhat-operators"
```
1 You must specify the openshift-operators-redhat namespace.

The Operator generates a CatalogSource from your CatalogSourceConfig in the namespace specified in targetNamespace.
Create a CatalogSourceConfig object:
```
$ oc create -f eo-csc.yaml
```

Use the following command to get the channel value required for the next step.

$ oc get packagemanifest elasticsearch-operator -n openshift-marketplace -o jsonpath='{.status.channels[].name}'

preview

Create a Subscription object YAML file (for example, eo-sub.yaml) to subscribe a Namespace to an Operator.

Example Subscription

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  generateName: "elasticsearch-"
  namespace: "openshift-operators-redhat" (1)
spec:
  channel: "preview" (2)
  installPlanApproval: "Automatic"
  source: "elasticsearch"
  sourceNamespace: "openshift-operators-redhat" (1)
  name: "elasticsearch-operator"

1	You must specify the `openshift-operators-redhat` namespace for `namespace` and `sourceNameSpace`.
2	Specify the `.status.channels[].name` value from the previous step.

Create the Subscription object:
```
$ oc create -f eo-sub.yaml
```

Change to the openshift-operators-redhat project:

$ oc project openshift-operators-redhat

Now using project "openshift-operators-redhat"

Create a Role-based Access Control (RBAC) object file (for example, eo-rbac.yaml) to grant Prometheus permission to access the openshift-operators-redhat namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: prometheus-k8s
  namespace: openshift-operators-redhat
rules:
- apiGroups:
  - ""
  resources:
  - services
  - endpoints
  - pods
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: prometheus-k8s
  namespace: openshift-operators-redhat
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: prometheus-k8s
subjects:
- kind: ServiceAccount
  name: prometheus-k8s
  namespace: openshift-operators-redhat

Create the RBAC object:
```
$ oc create -f eo-rbac.yaml
```
The Elasticsearch operator is installed to each project in the cluster.

Install the Cluster Logging Operator using the OpenShift Container Platform web console for best results:
1. In the OpenShift Container Platform web console, click Catalog → OperatorHub.
2. Choose Cluster Logging from the list of available Operators, and click Install.
3. On the Create Operator Subscription page, under A specific namespace on the cluster select openshift-logging. Then, click Subscribe.
Verify the operator installations:
1. Switch to the Catalog → Installed Operators page.
2. Ensure that Cluster Logging is listed in the openshift-logging project with a Status of InstallSucceeded.
3. Ensure that Elasticsearch Operator is listed in the openshift-operator-redhat project with a Status of InstallSucceeded. The Elasticsearch Operator is copied to all other projects.
  
  During installation an operator might display a Failed status. If the operator then installs with an InstallSucceeded message, you can safely ignore the Failed message.
  
  If either operator does not appear as installed, to troubleshoot further:
  - Switch to the Catalog → Operator Management page and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.
  - Switch to the Workloads → Pods page and check the logs in any Pods in the openshift-logging and openshift-operators-redhat projects that are reporting issues.

Create a cluster logging instance:

Switch to the Administration → Custom Resource Definitions page.
On the Custom Resource Definitions page, click ClusterLogging.
On the Custom Resource Definition Overview page, select View Instances from the Actions menu.
On the Cluster Loggings page, click Create Cluster Logging.

You might have to refresh the page to load the data.

In the YAML, replace the code with the following:

This default cluster logging configuration should support a wide array of environments. Review the topics on tuning and configuring the cluster logging components for information on modifications you can make to your cluster logging cluster.

apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance" (1)
  namespace: "openshift-logging"
spec:
  managementState: "Managed"  (2)
  logStore:
    type: "elasticsearch"  (3)
    elasticsearch:
      nodeCount: 3 (4)
      storage:
        storageClassName: gp2 (5)
        size: 200G
      redundancyPolicy: "SingleRedundancy"
  visualization:
    type: "kibana"  (6)
    kibana:
      replicas: 1
  curation:
    type: "curator"  (7)
    curator:
      schedule: "30 3 * * *"
  collection:
    logs:
      type: "fluentd"  (8)
      fluentd: {}

1	The name of the CR. This must be `instance`.
2	The cluster logging management state. In most cases, if you change the default cluster logging defaults, you must set this to `Unmanaged`. However, an unmanaged deployment does not receive updates until the cluster logging is placed back into a managed state. For more information, see Changing cluster logging management state.
3	Settings for configuring Elasticsearch. Using the CR, you can configure shard replication policy and persistent storage. For more information, see Configuring Elasticsearch.
4	Specify the number of Elasticsearch nodes. See the note that follows this list.
5	Specify that each Elasticsearch node in the cluster is bound to a Persistent Volume Claim.
6	Settings for configuring Kibana. Using the CR, you can scale Kibana for redundancy and configure the CPU and memory for your Kibana nodes. For more information, see Configuring Kibana.
7	Settings for configuring Curator. Using the CR, you can set the Curator schedule. For more information, see Configuring Curator.
8	Settings for configuring Fluentd. Using the CR, you can configure Fluentd CPU and memory limits. For more information, see Configuring Fluentd.

The maximum number of Elasticsearch master nodes is three. If you specify a nodeCount greater than 3, OpenShift Container Platform creates three Elasticsearch nodes that are master-eligible nodes, with the master, client, and data roles. The additional Elasticsearch nodes are created as Data-only nodes, using client and data roles. master nodes perform cluster-wide actions such as creating or deleting an index, shard allocation, and tracking nodes. Data nodes hold the shards and perform data-related operations such as CRUD, search, and aggregations. Data-related operations are I/O-, memory-, and CPU-intensive. It is important to monitor these resources and to add more Data nodes if the current nodes are overloaded.

For example, if nodeCount=4, the following nodes are created:

$ oc get deployment

cluster-logging-operator       1/1     1            1           18h
elasticsearch-cd-x6kdekli-1    0/1     1            0           6m54s
elasticsearch-cdm-x6kdekli-1   1/1     1            1           18h
elasticsearch-cdm-x6kdekli-2   0/1     1            0           6m49s
elasticsearch-cdm-x6kdekli-3   0/1     1            0           6m44s

Click Create. This creates the Cluster Logging Custom Resource and Elasticsearch Custom Resource, which you can edit to make changes to your cluster logging cluster.

Verify the install:
1. Switch to the Workloads → Pods page.
2. Select the openshift-logging project.
  
  You should see several pods for cluster logging, Elasticsearch, Fluentd, and Kibana similar to the following list:
  - cluster-logging-operator-cb795f8dc-xkckc
  - elasticsearch-cdm-b3nqzchd-1-5c6797-67kfz
  - elasticsearch-cdm-b3nqzchd-2-6657f4-wtprv
  - elasticsearch-cdm-b3nqzchd-3-588c65-clg7g
  - fluentd-2c7dg
  - fluentd-9z7kk
  - fluentd-br7r2
  - fluentd-fn2sb
  - fluentd-pb2f8
  - fluentd-zqgqx
  - kibana-7fb4fd4cc9-bvt4p

Additional resources

For more information on installing operators,see Installing Operators from the OperatorHub.