Metrics, logs, and traces

Log in to the OpenShift Container Platform web console as a user with cluster-admin rights. If you use Red Hat OpenShift Dedicated, you must have an account with the dedicated-admin role.

Navigate to Networking → Routes.

On the Routes page, select the service Mesh control plane project, for example istio-system, from the Namespace menu.

If necessary, use the filter to find the component console whose route you want to access. Click the route Location to launch the console.

Click Log In With OpenShift.

Log in to the OpenShift Container Platform CLI as a user with the cluster-admin role. If you use Red Hat OpenShift Dedicated, you must have an account with the dedicated-admin role.

$ oc login --username=<NAMEOFUSER> https://<HOSTNAME>:6443

Switch to the service Mesh control plane project. In this example, istio-system is the service Mesh control plane project. Run the following command:

$ oc project istio-system

To get the routes for the various Red Hat OpenShift service Mesh consoles, run the folowing command:

$ oc get routes

NAME                    HOST/PORT                         serviceS              PORT    TERMINATION
bookinfo-gateway        bookinfo-gateway-yourcompany.com  istio-ingressgateway          http2
grafana                 grafana-yourcompany.com           grafana               <all>   reencrypt/Redirect
istio-ingressgateway    istio-ingress-yourcompany.com     istio-ingressgateway  8080
jaeger                  jaeger-yourcompany.com            jaeger-query          <all>   reencrypt
kiali                   kiali-yourcompany.com             kiali                 20001   reencrypt/Redirect
prometheus              prometheus-yourcompany.com        prometheus            <all>   reencrypt/Redirect

Copy the URL for the console you want to access from the HOST/PORT column into a browser to open the console.

Click Log In With OpenShift.

Log in to the OpenShift Container Platform web console with an administrator role.

Click Home → Projects.

On the Projects page, if necessary, use the filter to find the name of your project.

Click the name of your project, for example, bookinfo.

On the Project details page, in the Launcher section, click the Kiali link.

Log in to the Kiali console with the same user name and password that you use to access the OpenShift Container Platform console.

Log in to the OpenShift Container Platform web console with a developer role.

Click Project.

On the Project Details page, if necessary, use the filter to find the name of your project.

Click the name of your project, for example, bookinfo.

On the Project page, in the Launcher section, click the Kiali link.

Click Log In With OpenShift.

Log in to the OpenShift Container Platform web console as a user with cluster-admin rights. If you use Red Hat OpenShift Dedicated, you must have an account with the dedicated-admin role.

Navigate to Networking → Routes.

On the Routes page, select the service Mesh control plane project, for example istio-system, from the Namespace menu.

If necessary, use the filter to find the jaeger route. Click the route Location to launch the console.

Click Log In With OpenShift.

Launch the Kiali console.

Click Distributed Tracing in the left navigation pane.

Click Log In With OpenShift.

Log in to the OpenShift Container Platform CLI as a user with the cluster-admin role. If you use Red Hat OpenShift Dedicated, you must have an account with the dedicated-admin role.

$ oc login --username=<NAMEOFUSER> https://<HOSTNAME>:6443

To query for details of the route using the command line, enter the following command. In this example, istio-system is the service Mesh control plane namespace.

$ export JAEGER_URL=$(oc get route -n istio-system jaeger -o jsonpath='{.spec.host}')

Launch a browser and navigate to https://<JAEGER_URL>, where <JAEGER_URL> is the route that you discovered in the previous step.

Log in using the same user name and password that you use to access the OpenShift Container Platform console.

If you have added services to the service mesh and have generated traces, you can use the filters and Find Traces button to search your trace data.

Log in to the OpenShift Container Platform web console.

Click the Project menu and select the project where you installed the service Mesh control plane, for example istio-system.

Click Routes.

Click the link in the Location column for the Grafana row.

Log in to the Grafana console with your OpenShift Container Platform credentials.

Log in to the OpenShift Container Platform web console.

Click the Project menu and select the project where you installed the service Mesh control plane, for example istio-system.

Click Routes.

Click the link in the Location column for the Prometheus row.

Log in to the Prometheus console with your OpenShift Container Platform credentials.

Create a token to Thanos for Kiali by running the following commands:

Configure Kiali for user-workload monitoring:

apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali-user-workload-monitoring
  namespace: istio-system
spec:
  external_services:
    istio:
      url_service_version: 'http://istiod-basic.istio-system:15014/version'
      config_map_name: istio-basic (1)
    prometheus:
      auth:
        token: secret:thanos-querier-web-token:token
        type: bearer
        use_kiali_token: false
      query_scope:
        mesh_id: "basic-istio-system"
      thanos_proxy:
        enabled: true
      url: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091
  version: v1.65

Configure the SMCP for external Prometheus:

apiVersion: maistra.io/v2
kind: serviceMeshControlPlane
metadata:
  name: basic
  namespace: istio-system
spec:
  addons:
    prometheus:
      enabled: false (1)
    grafana:
      enabled: false (2)
    kiali:
      name: kiali-user-workload-monitoring
  meshConfig:
    extensionProviders:
    - name: prometheus
      prometheus: {}

Apply a custom network policy to allow ingress traffic from the monitoring namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: user-workload-access
  namespace: bookinfo (1)
spec:
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          network.openshift.io/policy-group: monitoring
  podSelector: {}
  policyTypes:
  - Ingress

Apply a Telemetry object to enable traffic metrics in Istio proxies:

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: enable-prometheus-metrics
  namespace: istio-system (1)
spec:
  selector: (2)
    matchLabels:
      app: bookinfo
  metrics:
  - providers:
    - name: prometheus

Apply a serviceMonitor object to monitor the Istio control plane:

apiVersion: monitoring.coreos.com/v1
kind: serviceMonitor
metadata:
  name: istiod-monitor
  namespace: istio-system (1)
spec:
  targetLabels:
  - app
  selector:
    matchLabels:
      istio: pilot
  endpoints:
  - port: http-monitoring
    interval: 30s
    relabelings:
    - action: replace
      replacement: "basic-istio-system" (2)
      targetLabel: mesh_id

Apply a PodMonitor object to collect metrics from Istio proxies:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: istio-proxies-monitor
  namespace: istio-system (1)
spec:
  selector:
    matchExpressions:
    - key: istio-prometheus-ignore
      operator: DoesNotExist
  podMetricsEndpoints:
  - path: /stats/prometheus
    interval: 30s
    relabelings:
    - action: keep
      sourceLabels: [__meta_kubernetes_pod_container_name]
      regex: "istio-proxy"
    - action: keep
      sourceLabels: [__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape]
    - action: replace
      regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
      replacement: '[$2]:$1'
      sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_port,
      __meta_kubernetes_pod_ip]
      targetLabel: __address__
    - action: replace
      regex: (\d+);((([0-9]+?)(\.|$)){4})
      replacement: $2:$1
      sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_port,
      __meta_kubernetes_pod_ip]
      targetLabel: __address__
    - action: labeldrop
      regex: "__meta_kubernetes_pod_label_(.+)"
    - sourceLabels: [__meta_kubernetes_namespace]
      action: replace
      targetLabel: namespace
    - sourceLabels: [__meta_kubernetes_pod_name]
      action: replace
      targetLabel: pod_name
    - action: replace
      replacement: "basic-istio-system" (2)
      targetLabel: mesh_id

Open the OpenShift Container Platform web console, and check that metrics are visible.