Viewing Elasticsearch status

Viewing Elasticsearch status
- Example condition messages
Viewing Elasticsearch component status

You can view the status of the Elasticsearch Operator and for a number of Elasticsearch components.

You can view the status of your Elasticsearch cluster.

Prerequisites

Cluster logging and Elasticsearch must be installed.

Procedure

Change to the openshift-logging project.
```
$ oc project openshift-logging
```

To view the Elasticsearch cluster status:

Get the name of the Elasticsearch instance:

$ oc get Elasticsearch

NAME            AGE
elasticsearch   5h9m

Get the Elasticsearch status:

$ oc get Elasticsearch <Elasticsearch-instance> -o yaml

For example:

$ oc get Elasticsearch elasticsearch -n openshift-logging -o yaml

The output includes information similar to the following:

status: (1)
  cluster: (2)
    activePrimaryShards: 30
    activeShards: 60
    initializingShards: 0
    numDataNodes: 3
    numNodes: 3
    pendingTasks: 0
    relocatingShards: 0
    status: green
    unassignedShards: 0
  clusterHealth: ""
  conditions: [] (3)
  nodes: (4)
  - deploymentName: elasticsearch-cdm-zjf34ved-1
    upgradeStatus: {}
  - deploymentName: elasticsearch-cdm-zjf34ved-2
    upgradeStatus: {}
  - deploymentName: elasticsearch-cdm-zjf34ved-3
    upgradeStatus: {}
  pods: (5)
    client:
      failed: []
      notReady: []
      ready:
      - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
      - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
      - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
    data:
      failed: []
      notReady: []
      ready:
      - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
      - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
      - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
    master:
      failed: []
      notReady: []
      ready:
      - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
      - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
      - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
  shardAllocationEnabled: all

1	In the output, the cluster status fields appear in the `status` stanza.
2	The status of the Elasticsearch cluster: The number of active primary shards. The number of active shards. The number of shards that are initializing. The number of Elasticsearch data nodes. The total number of Elasticsearch nodes. The number of pending tasks. The Elasticsearch status: `green`, `red`, `yellow`. The number of unassigned shards.
3	Any status conditions, if present. The Elasticsearch cluster status indicates the reasons from the scheduler if a pod could not be placed. Any events related to the following conditions are shown: Container Waiting for both the Elasticsearch and proxy containers. Container Terminated for both the Elasticsearch and proxy containers. Pod unschedulable. Also, a condition is shown for a number of issues, see Example condition messages.
4	The Elasticsearch nodes in the cluster, with `upgradeStatus`.
5	The Elasticsearch client, data, and master pods in the cluster, listed under 'failed`, `notReady` or `ready` state.

Example condition messages

The following are examples of some condition messages from the Status section of the Elasticsearch instance.

This status message indicates a node has exceeded the configured low watermark and no shard will be allocated to this node.

status:
  nodes:
  - conditions:
    - lastTransitionTime: 2019-03-15T15:57:22Z
      message: Disk storage usage for node is 27.5gb (36.74%). Shards will be not
        be allocated on this node.
      reason: Disk Watermark Low
      status: "True"
      type: NodeStorage
    deploymentName: example-elasticsearch-cdm-0-1
    upgradeStatus: {}

This status message indicates a node has exceeded the configured high watermark and shards will be relocated to other nodes.

status:
  nodes:
  - conditions:
    - lastTransitionTime: 2019-03-15T16:04:45Z
      message: Disk storage usage for node is 27.5gb (36.74%). Shards will be relocated
        from this node.
      reason: Disk Watermark High
      status: "True"
      type: NodeStorage
    deploymentName: example-elasticsearch-cdm-0-1
    upgradeStatus: {}

This status message indicates the Elasticsearch node selector in the CR does not match any nodes in the cluster:

status:
    nodes:
    - conditions:
      - lastTransitionTime: 2019-04-10T02:26:24Z
        message: '0/8 nodes are available: 8 node(s) didn''t match node selector.'
        reason: Unschedulable
        status: "True"
        type: Unschedulable

This status message indicates that the Elasticsearch CR uses a non-existent PVC.

status:
   nodes:
   - conditions:
     - last Transition Time:  2019-04-10T05:55:51Z
       message:               pod has unbound immediate PersistentVolumeClaims (repeated 5 times)
       reason:                Unschedulable
       status:                True
       type:                  Unschedulable

This status message indicates that your Elasticsearch cluster does not have enough nodes to support your Elasticsearch redundancy policy.

status:
  clusterHealth: ""
  conditions:
  - lastTransitionTime: 2019-04-17T20:01:31Z
    message: Wrong RedundancyPolicy selected. Choose different RedundancyPolicy or
      add more nodes with data roles
    reason: Invalid Settings
    status: "True"
    type: InvalidRedundancy

This status message indicates your cluster has too many master nodes:

status:
  clusterHealth: green
  conditions:
    - lastTransitionTime: '2019-04-17T20:12:34Z'
      message: >-
        Invalid master nodes count. Please ensure there are no more than 3 total
        nodes with master roles
      reason: Invalid Settings
      status: 'True'
      type: InvalidMasters

Viewing Elasticsearch component status

You can view the status for a number of Elasticsearch components.

Elasticsearch indices

You can view the status of the Elasticsearch indices.

Get the name of an Elasticsearch pod:

$ oc get pods --selector component=elasticsearch -o name

pod/elasticsearch-cdm-1godmszn-1-6f8495-vp4lw
pod/elasticsearch-cdm-1godmszn-2-5769cf-9ms2n
pod/elasticsearch-cdm-1godmszn-3-f66f7d-zqkz7

Get the status of the indices:

$ oc exec elasticsearch-cdm-1godmszn-1-6f8495-vp4lw -- indices

Defaulting container name to elasticsearch.
Use 'oc describe pod/elasticsearch-cdm-1godmszn-1-6f8495-vp4lw -n openshift-logging' to see all of the containers in this pod.
Wed Apr 10 05:42:12 UTC 2019
health status index                                            uuid                   pri rep docs.count docs.deleted store.size pri.store.size
red    open   .kibana.647a750f1787408bf50088234ec0edd5a6a9b2ac N7iCbRjSSc2bGhn8Cpc7Jg   2   1
green  open   .operations.2019.04.10                           GTewEJEzQjaus9QjvBBnGg   3   1    2176114            0       3929           1956
green  open   .operations.2019.04.11                           ausZHoKxTNOoBvv9RlXfrw   3   1    1494624            0       2947           1475
green  open   .kibana                                          9Fltn1D0QHSnFMXpphZ--Q   1   1          1            0          0              0
green  open   .searchguard                                     chOwDnQlSsqhfSPcot1Yiw   1   1          5            1          0              0

Elasticsearch pods

You can view the status of the Elasticsearch pods.

Get the name of a pod:

$ oc get pods --selector component=elasticsearch -o name

pod/elasticsearch-cdm-1godmszn-1-6f8495-vp4lw
pod/elasticsearch-cdm-1godmszn-2-5769cf-9ms2n
pod/elasticsearch-cdm-1godmszn-3-f66f7d-zqkz7

Get the status of a pod:

oc describe pod elasticsearch-cdm-1godmszn-1-6f8495-vp4lw

The output includes the following status information:

....
Status:             Running

....

Containers:
  elasticsearch:
    Container ID:   cri-o://b7d44e0a9ea486e27f47763f5bb4c39dfd2
    State:          Running
      Started:      Mon, 08 Apr 2019 10:17:56 -0400
    Ready:          True
    Restart Count:  0
    Readiness:  exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3

....

  proxy:
    Container ID:  cri-o://3f77032abaddbb1652c116278652908dc01860320b8a4e741d06894b2f8f9aa1
    State:          Running
      Started:      Mon, 08 Apr 2019 10:18:38 -0400
    Ready:          True
    Restart Count:  0

....

Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True

....

Events:          <none>

Elasticsearch deployment configuration

You can view the status of the Elasticsearch deployment configuration.

Get the name of a deployment configuration:

$ oc get deployment --selector component=elasticsearch -o name

deployment.extensions/elasticsearch-cdm-1gon-1
deployment.extensions/elasticsearch-cdm-1gon-2
deployment.extensions/elasticsearch-cdm-1gon-3

Get the deployment configuration status:

$ oc describe deployment elasticsearch-cdm-1gon-1

The output includes the following status information:

....
  Containers:
   elasticsearch:
    Image:      registry.redhat.io/openshift4/ose-logging-elasticsearch5:v4.2
    Readiness:  exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3

....

Conditions:
  Type           Status   Reason
  ----           ------   ------
  Progressing    Unknown  deploymentPaused
  Available      True     MinimumReplicasAvailable

....

Events:          <none>

Elasticsearch ReplicaSet

You can view the status of the Elasticsearch ReplicaSet.

Get the name of a replica set:

$ oc get replicaSet --selector component=elasticsearch -o name

replicaset.extensions/elasticsearch-cdm-1gon-1-6f8495
replicaset.extensions/elasticsearch-cdm-1gon-2-5769cf
replicaset.extensions/elasticsearch-cdm-1gon-3-f66f7d

Get the status of the replica set:

$ oc describe replicaSet elasticsearch-cdm-1gon-1-6f8495

The output includes the following status information:

....
  Containers:
   elasticsearch:
    Image:      registry.redhat.io/openshift4/ose-logging-elasticsearch5:v4.2
    Readiness:  exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3

....

Events:          <none>