This is a cache of https://docs.okd.io/4.14/observability/monitoring/configuring-core-platform-monitoring/configuring-alerts-and-notifications.html. It is a snapshot of the page at 2025-07-07T05:25:48.497+0000.
Configuring alerts and notifications - Monitoring | Observability | OKD 4.14
  1. Documentation
×

Configuring alerts and notifications for core platform monitoring

Configuring external Alertmanager instances

The OKD monitoring stack includes a local Alertmanager instance that routes alerts from Prometheus.

You can add external Alertmanager instances to route alerts for core OKD projects.

If you add the same external Alertmanager configuration for multiple clusters and disable the local instance for each cluster, you can then manage alert routing for multiple clusters by using a single external Alertmanager instance.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role.

  • You have created the cluster-monitoring-config ConfigMap object.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

    $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

  2. Add an additionalAlertmanagerConfigs section with configuration details under data/config.yaml/prometheusK8s:

    apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      additionalAlertmanagerConfigs:
      - <alertmanager_specification> (1)
    1 Substitute <alertmanager_specification> with authentication and other configuration details for additional Alertmanager instances. Currently supported authentication methods are bearer token (bearerToken) and client TLS (tlsConfig).

    The following sample config map configures an additional Alertmanager for Prometheus by using a bearer token with client TLS authentication:

    apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      additionalAlertmanagerConfigs:
      - scheme: https
        pathPrefix: /
        timeout: "30s"
        apiVersion: v1
        bearerToken:
          name: alertmanager-bearer-token
          key: token
        tlsConfig:
          key:
            name: alertmanager-tls
            key: tls.key
          cert:
            name: alertmanager-tls
            key: tls.crt
          ca:
            name: alertmanager-tls
            key: tls.ca
        staticConfigs:
        - external-alertmanager1-remote.com
        - external-alertmanager1-remote2.com

  3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Disabling the local Alertmanager

A local Alertmanager that routes alerts from Prometheus instances is enabled by default in the openshift-monitoring project of the OKD monitoring stack.

If you do not need the local Alertmanager, you can disable it by configuring the cluster-monitoring-config config map in the openshift-monitoring project.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role.

  • You have created the cluster-monitoring-config config map.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

    $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

  2. Add enabled: false for the alertmanagerMain component under data/config.yaml:

    apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    alertmanagerMain:
      enabled: false

  3. Save the file to apply the changes. The Alertmanager instance is disabled automatically when you apply the change.

Additional resources

Configuring secrets for Alertmanager

The OKD monitoring stack includes Alertmanager, which routes alerts from Prometheus to endpoint receivers. If you need to authenticate with a receiver so that Alertmanager can send alerts to it, you can configure Alertmanager to use a secret that contains authentication credentials for the receiver.

For example, you can configure Alertmanager to use a secret to authenticate with an endpoint receiver that requires a certificate issued by a private Certificate Authority (CA). You can also configure Alertmanager to use a secret to authenticate with a receiver that requires a password file for Basic HTTP authentication. In either case, authentication details are contained in the Secret object rather than in the ConfigMap object.

Adding a secret to the Alertmanager configuration

You can add secrets to the Alertmanager configuration by editing the cluster-monitoring-config config map in the openshift-monitoring project.

After you add a secret to the config map, the secret is mounted as a volume at /etc/alertmanager/secrets/<secret_name> within the alertmanager container for the Alertmanager pods.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role.

  • You have created the cluster-monitoring-config config map.

  • You have created the secret to be configured in Alertmanager in the openshift-monitoring project.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

    $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

  2. Add a secrets: section under data/config.yaml/alertmanagerMain with the following configuration:

    apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    alertmanagerMain:
      secrets: (1)
      - <secret_name_1> (2)
      - <secret_name_2>
    1 This section contains the secrets to be mounted into Alertmanager. The secrets must be located within the same namespace as the Alertmanager object.
    2 The name of the Secret object that contains authentication credentials for the receiver. If you add multiple secrets, place each one on a new line.

    The following sample config map settings configure Alertmanager to use two Secret objects named test-secret-basic-auth and test-secret-api-token:

    apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    alertmanagerMain:
      secrets:
      - test-secret-basic-auth
      - test-secret-api-token

  3. Save the file to apply the changes. The new configuration is applied automatically.

Attaching additional labels to your time series and alerts

You can attach custom labels to all time series and alerts leaving Prometheus by using the external labels feature of Prometheus.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role.

  • You have created the cluster-monitoring-config ConfigMap object.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

    $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

  2. Define labels you want to add for every metric under data/config.yaml:

    apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      externalLabels:
        <key>: <value> (1)
    1 Substitute <key>: <value> with key-value pairs where <key> is a unique name for the new label and <value> is its value.

    • Do not use prometheus or prometheus_replica as key names, because they are reserved and will be overwritten.

    • Do not use cluster or managed_cluster as key names. Using them can cause issues where you are unable to see data in the developer dashboards.

    For example, to add metadata about the region and environment to all time series and alerts, use the following example:

    apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      externalLabels:
        region: eu
        environment: prod

  3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Additional resources

Configuring alert notifications

In OKD 4.14, you can view firing alerts in the Alerting UI. You can configure Alertmanager to send notifications about default platform alerts by configuring alert receivers.

Alertmanager does not send notifications by default. It is strongly recommended to configure Alertmanager to receive notifications by configuring alert receivers through the web console or through the alertmanager-main secret.
Additional resources

Configuring alert routing for default platform alerts

You can configure Alertmanager to send notifications to receive important alerts coming from your cluster. Customize where and how Alertmanager sends notifications about default platform alerts by editing the default configuration in the alertmanager-main secret in the openshift-monitoring namespace.

All features of a supported version of upstream Alertmanager are also supported in an OKD Alertmanager configuration. To check all the configuration options of a supported version of upstream Alertmanager, see Alertmanager configuration (Prometheus documentation).
Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Extract the currently active Alertmanager configuration from the alertmanager-main secret and save it as a local alertmanager.yaml file:

    $ oc -n openshift-monitoring get secret alertmanager-main --template='{{ index .data "alertmanager.yaml" }}' | base64 --decode > alertmanager.yaml

  2. Open the alertmanager.yaml file.

  3. Edit the Alertmanager configuration:

    1. Optional: Change the default Alertmanager configuration:

      Example of the default Alertmanager secret YAML
      global:
  resolve_timeout: 5m
route:
  group_wait: 30s (1)
  group_interval: 5m (2)
  repeat_interval: 12h (3)
  receiver: default
  routes:
  - matchers:
    - "alertname=Watchdog"
    repeat_interval: 2m
    receiver: watchdog
receivers:
- name: default
- name: watchdog
      1 Specify how long Alertmanager waits while collecting initial alerts for a group of alerts before sending a notification.
      2 Specify how much time must elapse before Alertmanager sends a notification about new alerts added to a group of alerts for which an initial notification was already sent.
      3 Specify the minimum amount of time that must pass before an alert notification is repeated. If you want a notification to repeat at each group interval, set the repeat_interval value to less than the group_interval value. The repeated notification can still be delayed, for example, when certain Alertmanager pods are restarted or rescheduled.

    2. Add your alert receiver configuration:

      # ...
receivers:
- name: default
- name: watchdog
- name: <receiver> (1)
  <receiver_configuration> (2)
# ...
      1 The name of the receiver.
      2 The receiver configuration. The supported receivers are PagerDuty, webhook, email, Slack, and Microsoft Teams.
      Example of configuring PagerDuty as an alert receiver
      # ...
receivers:
- name: default
- name: watchdog
- name: team-frontend-page
  pagerduty_configs:
  - routing_key: xxxxxxxxxx (1)
# ...
      1 Defines the PagerDuty integration key.
      Example of configuring email as an alert receiver
      # ...
receivers:
- name: default
- name: watchdog
- name: team-frontend-page
  email_configs:
    - to: myemail@example.com (1)
      from: alertmanager@example.com (2)
      smarthost: 'smtp.example.com:587' (3)
      auth_username: alertmanager@example.com  (4)
      auth_password: password
      hello: alertmanager (5)
# ...
      1 Specify an email address to send notifications to.
      2 Specify an email address to send notifications from.
      3 Specify the SMTP server address used for sending emails, including the port number.
      4 Specify the authentication credentials that Alertmanager uses to connect to the SMTP server. This example uses username and password.
      5 Specify the hostname to identify to the SMTP server. If you do not include this parameter, the hostname defaults to localhost.

      Alertmanager requires an external SMTP server to send email alerts. To configure email alert receivers, ensure you have the necessary connection details for an external SMTP server.

    3. Add the routing configuration:

      # ...
route:
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  receiver: default
  routes:
  - matchers:
    - "alertname=Watchdog"
    repeat_interval: 2m
    receiver: watchdog
  - matchers: (1)
    - "<your_matching_rules>" (2)
    receiver: <receiver> (3)
# ...
      1 Use the matchers key name to specify the matching rules that an alert has to fulfill to match the node. If you define inhibition rules, use target_matchers key name for target matchers and source_matchers key name for source matchers.
      2 Specify labels to match your alerts.
      3 Specify the name of the receiver to use for the alerts.

      Do not use the match, match_re, target_match, target_match_re, source_match, and source_match_re key names, which are deprecated and planned for removal in a future release.
      Example of alert routing
      # ...
route:
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  receiver: default
  routes:
  - matchers:
    - "alertname=Watchdog"
    repeat_interval: 2m
    receiver: watchdog
  - matchers: (1)
    - "service=example-app"
    routes: (2)
    - matchers:
      - "severity=critical"
      receiver: team-frontend-page
# ...
      1 This example matches alerts from the example-app service.
      2 You can create routes within other routes for more complex alert routing.

      The previous example routes alerts of critical severity that are fired by the example-app service to the team-frontend-page receiver. Typically, these types of alerts are paged to an individual or a critical response team.

  4. Apply the new configuration in the file:

    $ oc -n openshift-monitoring create secret generic alertmanager-main --from-file=alertmanager.yaml --dry-run=client -o=yaml |  oc -n openshift-monitoring replace secret --filename=-

  5. Verify your routing configuration by visualizing the routing tree:

    $ oc exec alertmanager-main-0 -n openshift-monitoring -- amtool config routes show --alertmanager.url http://localhost:9093
    Example output
    Routing tree:
.
└── default-route  receiver: default
    ├── {alertname="Watchdog"}  receiver: Watchdog
    └── {service="example-app"}  receiver: default
        └── {severity="critical"}  receiver: team-frontend-page
Additional resources

Configuring alert routing with the OKD web console

You can configure alert routing through the OKD web console to ensure that you learn about important issues with your cluster.

The OKD web console provides fewer settings to configure alert routing than the alertmanager-main secret. To configure alert routing with the access to more configuration settings, see "Configuring alert routing for default platform alerts".
Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role.

Procedure

  1. In the Administrator perspective, go to Administration → Cluster Settings → Configuration → Alertmanager.

    Alternatively, you can go to the same page through the notification drawer. Select the bell icon at the top right of the OKD web console and choose Configure in the AlertmanagerReceiverNotConfigured alert.

  2. Click Create Receiver in the Receivers section of the page.

  3. In the Create Receiver form, add a Receiver name and choose a Receiver type from the list.

  4. Edit the receiver configuration:

    • For PagerDuty receivers:

      1. Choose an integration type and add a PagerDuty integration key.

      2. Add the URL of your PagerDuty installation.

      3. Click Show advanced configuration if you want to edit the client and incident details or the severity specification.

    • For webhook receivers:

      1. Add the endpoint to send HTTP POST requests to.

      2. Click Show advanced configuration if you want to edit the default option to send resolved alerts to the receiver.

    • For email receivers:

      1. Add the email address to send notifications to.

      2. Add SMTP configuration details, including the address to send notifications from, the smarthost and port number used for sending emails, the hostname of the SMTP server, and authentication details.

        Alertmanager requires an external SMTP server to send email alerts. To configure email alert receivers, ensure you have the necessary connection details for an external SMTP server.

      3. Select whether TLS is required.

      4. Click Show advanced configuration if you want to edit the default option not to send resolved alerts to the receiver or edit the body of email notifications configuration.

    • For Slack receivers:

      1. Add the URL of the Slack webhook.

      2. Add the Slack channel or user name to send notifications to.

      3. Select Show advanced configuration if you want to edit the default option not to send resolved alerts to the receiver or edit the icon and username configuration. You can also choose whether to find and link channel names and usernames.

  5. By default, firing alerts with labels that match all of the selectors are sent to the receiver. If you want label values for firing alerts to be matched exactly before they are sent to the receiver, perform the following steps:

    1. Add routing label names and values in the Routing labels section of the form.

    2. Click Add label to add further routing labels.

  6. Click Create to create the receiver.

Additional resources

Configuring different alert receivers for default platform alerts and user-defined alerts

You can configure different alert receivers for default platform alerts and user-defined alerts to ensure the following results:

  • All default platform alerts are sent to a receiver owned by the team in charge of these alerts.

  • All user-defined alerts are sent to another receiver so that the team can focus only on platform alerts.

You can achieve this by using the openshift_io_alert_source="platform" label that is added by the Cluster Monitoring Operator to all platform alerts:

  • Use the openshift_io_alert_source="platform" matcher to match default platform alerts.

  • Use the openshift_io_alert_source!="platform" or 'openshift_io_alert_source=""' matcher to match user-defined alerts.

This configuration does not apply if you have enabled a separate instance of Alertmanager dedicated to user-defined alerts.