Configuring Curator - Configuring your cluster logging deployment | Logging

Configuring the Curator schedule
Configuring Curator index deletion
Troubleshooting Curator
Configuring Curator in scripted deployments
Using the Curator Action file

The Elasticsearch Curator tool performs scheduled maintenance operations on a global and/or on a per-project basis. Curator performs actions daily based on its configuration.

The Cluster Logging Operator installs Curator and its configuration. You can configure the Curator cron schedule using the Cluster Logging Custom Resource and further configuration options can be found in the Curator configmap, curator in the openshift-logging project, which incorporates the Curator configuration file, curator5.yaml and an OpenShift Container Platform custom configuration file, config.yaml.

OpenShift Container Platform uses the config.yaml internally to generate the Curator action file.

Optionally, you can use the action file, directly. Editing this file allows you to use any action that Curator has available to it to be run periodically. However, this is only recommended for advanced users as modifying the file can be destructive to the cluster and can cause removal of required indices/settings from Elasticsearch. Most users only must modify the Curator configuration map and never edit the action file.

Configuring the Curator schedule

You can specify the schedule for Curator using the cluster logging Custom Resource created by the cluster logging installation.

Prerequisites

Cluster logging and Elasticsearch must be installed.

Procedure

To configure the Curator schedule:

Edit the Cluster Logging Custom Resource in the openshift-logging project:

$ oc edit clusterlogging instance

apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance"

...

  curation:
    curator:
      schedule: 30 3 * * * (1)
    type: curator

1	Specify the schedule for Curator in cron format.

The time zone is set based on the host node where the Curator pod runs.

Configuring Curator index deletion

You can configure Curator to delete Elasticsearch data based on retention settings. You can configure per-project and global settings. Global settings apply to any project not specified. Per-project settings override global settings.

Prerequisite

Cluster logging must be installed.

Procedure

To delete indices:

Edit the OpenShift Container Platform custom Curator configuration file:
```
$ oc edit configmap/curator
```

Set the following parameters as needed:

config.yaml: |
  project_name:
    action
      unit:value

The available parameters are:

Table 1. Project options
Variable Name	Description
`project_name`	The actual name of a project, such as myapp-devel. For OpenShift Container Platform operations logs, use the name `.operations` as the project name.
`action`	The action to take, currently only `delete` is allowed.
`unit`	The period to use for deletion, `days`, `weeks`, or `months`.
`value`	The number of units.

Table 2. Filter options
Variable Name	Description
`.defaults`	Use `.defaults` as the `project_name` to set the defaults for projects that are not specified.
`.regex`	The list of regular expressions that match project names.
`pattern`	The valid and properly escaped regular expression pattern enclosed by single quotation marks.

For example, to configure Curator to:

Delete indices in the myapp-dev project older than 1 day
Delete indices in the myapp-qe project older than 1 week
Delete operations logs older than 8 weeks
Delete all other projects indices after they are 31 days old
Delete indices older than 1 day that are matched by the ^project\..+\-dev.*$ regex
Delete indices older than 2 days that are matched by the ^project\..+\-test.*$ regex

Use:

  config.yaml: |
    .defaults:
      delete:
        days: 31

    .operations:
      delete:
        weeks: 8

    myapp-dev:
      delete:
        days: 1

    myapp-qe:
      delete:
        weeks: 1

    .regex:
      - pattern: '^project\..+\-dev\..*$'
        delete:
          days: 1
      - pattern: '^project\..+\-test\..*$'
        delete:
          days: 2

When you use months as the $UNIT for an operation, Curator starts counting at the first day of the current month, not the current day of the current month. For example, if today is April 15, and you want to delete indices that are 2 months older than today (delete: months: 2), Curator does not delete indices that are dated older than February 15; it deletes indices older than February 1. That is, it goes back to the first day of the current month, then goes back two whole months from that date. If you want to be exact with Curator, it is best to use days (for example, delete: days: 30).

Troubleshooting Curator

You can use information in this section for debugging Curator. For example, if curator is in failed state, but the log messages do not provide a reason, you could increase the log level and trigger a new job, instead of waiting for another scheduled run of the cron job.

Prerequisites

Cluster logging and Elasticsearch must be installed.

Procedure

Enable the Curator debug log and trigger next Curator iteration manually

Enable debug log of Curator:
```
$ oc set env cronjob/curator CURATOR_LOG_LEVEL=DEBUG CURATOR_SCRIPT_LOG_LEVEL=DEBUG
```
Specify the log level:
- CRITICAL. Curator displays only critical messages.
- ERROR. Curator displays only error and critical messages.
- WARNING. Curator displays only error, warning, and critical messages.
- INFO. Curator displays only informational, error, warning, and critical messages.
- DEBUG. Curator displays only debug messages, in addition to all of the above.
  
  The default value is INFO.

Cluster logging uses the OpenShift Container Platform custom environment variable CURATOR_SCRIPT_LOG_LEVEL in OpenShift Container Platform wrapper scripts (run.sh and convert.py). The environment variable takes the same values as CURATOR_LOG_LEVEL for script debugging, as needed.

Trigger next curator iteration:

$ oc create job --from=cronjob/curator <job_name>

Use the following commands to control the CronJob:

Suspend a CronJob:

$ oc patch cronjob curator -p '{"spec":{"suspend":true}}'

Resume a CronJob:

$ oc patch cronjob curator -p '{"spec":{"suspend":false}}'

Change a CronJob schedule:
```
$ oc patch cronjob curator -p '{"spec":{"schedule":"0 0 * * *"}}' (1)
```
1 The schedule option accepts schedules in cron format.

Configuring Curator in scripted deployments

Use the information in this section if you must configure Curator in scripted deployments.

Prerequisites

Cluster logging and Elasticsearch must be installed.
Set cluster logging to the unmanaged state.

Procedure

Use the following snippets to configure Curator in your scripts:

For scripted deployments
1. Create and modify the configuration:
  1. Copy the Curator configuration file and the OpenShift Container Platform custom configuration file from the Curator configuration map and create separate files for each:
    
    $ oc extract configmap/curator --keys=curator5.yaml,config.yaml --to=/my/config
  2. Edit the /my/config/curator5.yaml and /my/config/config.yaml files.
2. Delete the existing Curator config map and add the edited YAML files to a new Curator config map.
  $ oc delete configmap curator ; sleep 1 $ oc create configmap curator \ --from-file=curator5.yaml=/my/config/curator5.yaml \ --from-file=config.yaml=/my/config/config.yaml \ ; sleep 1
  The next iteration will use this configuration.
If you are using the action file:
1. Create and modify the configuration:
  1. Copy the Curator configuration file and the action file from the Curator configuration map and create separate files for each:
    
    $ oc extract configmap/curator --keys=curator5.yaml,actions.yaml --to=/my/config
  2. Edit the /my/config/curator5.yaml and /my/config/actions.yaml files.
2. Delete the existing Curator config map and add the edited YAML files to a new Curator config map.
  $ oc delete configmap curator ; sleep 1 $ oc create configmap curator \ --from-file=curator5.yaml=/my/config/curator5.yaml \ --from-file=actions.yaml=/my/config/actions.yaml \ ; sleep 1
  The next iteration will use this configuration.

Using the Curator Action file

The Curator configmap in the openshift-logging project includes a Curator action file where you configure any Curator action to be run periodically.

However, when you use the action file, OpenShift Container Platform ignores the config.yaml section of the curator configmap, which is configured to ensure important internal indices do not get deleted by mistake. In order to use the action file, you should add an exclude rule to your configuration to retain these indices. You also must manually add all the other patterns following the steps in this topic.

The actions and config.yaml are mutually-exclusive configuration files. Once the actions file exist, OpenShift Container Platform ignores the config.yaml file. Using the action file is recommended only for advanced users as using this file can be destructive to the cluster and can cause removal of required indices/settings from Elasticsearch.

Prerequisite

Cluster logging and Elasticsearch must be installed.
Set cluster logging to the unmanaged state.

Procedure

To configure Curator to delete indices:

Edit the Curator configmap:
```
oc edit cm/curator -n openshift-logging
```

Make the following changes to the action file:

actions:
1:
      action: delete_indices (1)
      description: >-
        Delete .operations indices older than 30 days.
        Ignore the error if the filter does not
        result in an actionable list of indices (ignore_empty_list).
        See https://www.elastic.co/guide/en/elasticsearch/client/curator/5.2/ex_delete_indices.html
      options:
        # Swallow curator.exception.NoIndices exception
        ignore_empty_list: True
        # In seconds, default is 300
        timeout_override: ${CURATOR_TIMEOUT}
        # Don't swallow any other exceptions
        continue_if_exception: False
        # Optionally disable action, useful for debugging
        disable_action: False
      # All filters are bound by logical AND
      filters:            (2)
      - filtertype: pattern
        kind: regex
        value: '^\.operations\..*$'
        exclude: False    (3)
      - filtertype: age
        # Parse timestamp from index name
        source: name
        direction: older
        timestring: '%Y.%m.%d'
        unit: days
        unit_count: 30
        exclude: False

1	Specify `delete_indices` to delete the specified index.
2	Use the `filers` parameters to specify the index to be deleted. See the Elastic Search curator documentation for information on these parameters.
3	Specify `false` to allow the index to be deleted.

Curation of Elasticsearch Data

Configuring the Curator schedule

Configuring Curator index deletion

Troubleshooting Curator

Configuring Curator in scripted deployments

Using the Curator Action file