$ oc login --username=<your_username>
Installing the distributed tracing platform (Tempo) requires the Tempo Operator and choosing which type of deployment is best for your use case:
For microservices mode, deploy a TempoStack instance in a dedicated OpenShift project.
For monolithic mode, deploy a TempoMonolithic instance in a dedicated OpenShift project.
Using object storage requires setting up a supported object store and creating a secret for the object store credentials before deploying a TempoStack or TempoMonolithic instance. |
You can install the Tempo Operator by using the web console or the command line.
You can install the Tempo Operator from the Administrator view of the web console.
You are logged in to the OpenShift Container Platform web console as a cluster administrator with the cluster-admin
role.
For Red Hat OpenShift Dedicated, you must be logged in using an account with the dedicated-admin
role.
You have completed setting up the required object storage by a supported provider: {odf-full}, MinIO, Amazon S3, Azure Blob Storage, Google Cloud Storage. For more information, see "Object storage setup".
Object storage is required and not included with the distributed tracing platform (Tempo). You must choose and set up object storage by a supported provider before installing the distributed tracing platform (Tempo). |
Go to Operators → OperatorHub and search for Tempo Operator
.
Select the Tempo Operator that is provided by Red Hat.
The following selections are the default presets for this Operator:
|
Select the Enable Operator recommended cluster monitoring on this Namespace checkbox.
Select Install → Install → View Operator.
In the Details tab of the page of the installed Operator, under ClusterServiceVersion details, verify that the installation Status is Succeeded.
You can install the Tempo Operator from the command line.
An active OpenShift CLI (oc
) session by a cluster administrator with the cluster-admin
role.
|
You have completed setting up the required object storage by a supported provider: {odf-full}, MinIO, Amazon S3, Azure Blob Storage, Google Cloud Storage. For more information, see "Object storage setup".
Object storage is required and not included with the distributed tracing platform (Tempo). You must choose and set up object storage by a supported provider before installing the distributed tracing platform (Tempo). |
Create a project for the Tempo Operator by running the following command:
$ oc apply -f - << EOF
apiVersion: project.openshift.io/v1
kind: Project
metadata:
labels:
kubernetes.io/metadata.name: openshift-tempo-operator
openshift.io/cluster-monitoring: "true"
name: openshift-tempo-operator
EOF
Create an Operator group by running the following command:
$ oc apply -f - << EOF
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: openshift-tempo-operator
namespace: openshift-tempo-operator
spec:
upgradeStrategy: Default
EOF
Create a subscription by running the following command:
$ oc apply -f - << EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: tempo-product
namespace: openshift-tempo-operator
spec:
channel: stable
installPlanApproval: Automatic
name: tempo-product
source: redhat-operators
sourceNamespace: openshift-marketplace
EOF
Check the Operator status by running the following command:
$ oc get csv -n openshift-tempo-operator
You can install a TempoStack instance by using the web console or the command line.
You can install a TempoStack instance from the Administrator view of the web console.
You are logged in to the OpenShift Container Platform web console as a cluster administrator with the cluster-admin
role.
For Red Hat OpenShift Dedicated, you must be logged in using an account with the dedicated-admin
role.
You have completed setting up the required object storage by a supported provider: {odf-full}, MinIO, Amazon S3, Azure Blob Storage, Google Cloud Storage. For more information, see "Object storage setup".
Object storage is required and not included with the distributed tracing platform (Tempo). You must choose and set up object storage by a supported provider before installing the distributed tracing platform (Tempo). |
Go to Home → Projects → Create Project to create a project of your choice for the TempoStack instance that you will create in a subsequent step.
Go to Workloads → Secrets → Create → From YAML to create a secret for your object storage bucket in the project that you created for the TempoStack instance. For more information, see "Object storage setup".
apiVersion: v1
kind: Secret
metadata:
name: minio-test
stringData:
endpoint: http://minio.minio.svc:9000
bucket: tempo
access_key_id: tempo
access_key_secret: <secret>
type: Opaque
Create a TempoStack instance.
You can create multiple TempoStack instances in separate projects on the same cluster. |
Go to Operators → Installed Operators.
Select TempoStack → Create TempoStack → YAML view.
In the YAML view, customize the TempoStack
custom resource (CR):
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
name: sample
namespace: <project_of_tempostack_instance>
spec:
storageSize: <value>Gi (1)
storage:
secret: (2)
name: <secret_name> (3)
type: <secret_provider> (4)
tls: (5)
enabled: true
caName: <ca_certificate_configmap_name> (6)
template:
queryFrontend:
jaegerQuery:
enabled: true
ingress:
route:
termination: edge
type: route
resources: (7)
total:
limits:
memory: <value>Gi
cpu: <value>m
1 | Size of the persistent volume claim for the Tempo WAL. The default is 10Gi . |
2 | Secret you created in step 2 for the object storage that had been set up as one of the prerequisites. |
3 | Value of the name in the metadata of the secret. |
4 | Accepted values are azure for Azure Blob Storage; gcs for Google Cloud Storage; and s3 for Amazon S3, MinIO, or {odf-full}. |
5 | Optional. |
6 | Optional: Name of a ConfigMap object that contains a CA certificate. |
7 | Optional. |
TempoStack
CR for AWS S3 and MinIO storageapiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
name: simplest
namespace: <project_of_tempostack_instance>
spec:
storageSize: 1Gi
storage: (1)
secret:
name: minio-test
type: s3
resources:
total:
limits:
memory: 2Gi
cpu: 2000m
template:
queryFrontend:
jaegerQuery: (2)
enabled: true
ingress:
route:
termination: edge
type: route
1 | In this example, the object storage was set up as one of the prerequisites, and the object storage secret was created in step 2. |
2 | The stack deployed in this example is configured to receive Jaeger Thrift over HTTP and OpenTelemetry Protocol (OTLP), which permits visualizing the data with the Jaeger UI. |
Select Create.
Use the Project: dropdown list to select the project of the TempoStack instance.
Go to Operators → Installed Operators to verify that the Status of the TempoStack instance is Condition: Ready.
Go to Workloads → Pods to verify that all the component pods of the TempoStack instance are running.
Access the Tempo console:
Go to Networking → routes and Ctrl+F to search for tempo
.
In the Location column, open the URL to access the Tempo console.
The Tempo console initially shows no trace data following the Tempo console installation. |
You can install a TempoStack instance from the command line.
An active OpenShift CLI (oc
) session by a cluster administrator with the cluster-admin
role.
|
You have completed setting up the required object storage by a supported provider: {odf-full}, MinIO, Amazon S3, Azure Blob Storage, Google Cloud Storage. For more information, see "Object storage setup".
Object storage is required and not included with the distributed tracing platform (Tempo). You must choose and set up object storage by a supported provider before installing the distributed tracing platform (Tempo). |
Run the following command to create a project of your choice for the TempoStack instance that you will create in a subsequent step:
$ oc apply -f - << EOF
apiVersion: project.openshift.io/v1
kind: Project
metadata:
name: <project_of_tempostack_instance>
EOF
In the project that you created for the TempoStack instance, create a secret for your object storage bucket by running the following command:
$ oc apply -f - << EOF
<object_storage_secret>
EOF
For more information, see "Object storage setup".
apiVersion: v1
kind: Secret
metadata:
name: minio-test
stringData:
endpoint: http://minio.minio.svc:9000
bucket: tempo
access_key_id: tempo
access_key_secret: <secret>
type: Opaque
Create a TempoStack instance in the project that you created for it:
You can create multiple TempoStack instances in separate projects on the same cluster. |
Customize the TempoStack
custom resource (CR):
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
name: sample
namespace: <project_of_tempostack_instance>
spec:
storageSize: <value>Gi (1)
storage:
secret: (2)
name: <secret_name> (3)
type: <secret_provider> (4)
tls: (5)
enabled: true
caName: <ca_certificate_configmap_name> (6)
template:
queryFrontend:
jaegerQuery:
enabled: true
ingress:
route:
termination: edge
type: route
resources: (7)
total:
limits:
memory: <value>Gi
cpu: <value>m
1 | Size of the persistent volume claim for the Tempo WAL. The default is 10Gi . |
2 | Secret you created in step 2 for the object storage that had been set up as one of the prerequisites. |
3 | Value of the name in the metadata of the secret. |
4 | Accepted values are azure for Azure Blob Storage; gcs for Google Cloud Storage; and s3 for Amazon S3, MinIO, or {odf-full}. |
5 | Optional. |
6 | Optional: Name of a ConfigMap object that contains a CA certificate. |
7 | Optional. |
TempoStack
CR for AWS S3 and MinIO storageapiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
name: simplest
namespace: <project_of_tempostack_instance>
spec:
storageSize: 1Gi
storage: (1)
secret:
name: minio-test
type: s3
resources:
total:
limits:
memory: 2Gi
cpu: 2000m
template:
queryFrontend:
jaegerQuery: (2)
enabled: true
ingress:
route:
termination: edge
type: route
1 | In this example, the object storage was set up as one of the prerequisites, and the object storage secret was created in step 2. |
2 | The stack deployed in this example is configured to receive Jaeger Thrift over HTTP and OpenTelemetry Protocol (OTLP), which permits visualizing the data with the Jaeger UI. |
Apply the customized CR by running the following command:
$ oc apply -f - << EOF
<tempostack_cr>
EOF
Verify that the status
of all TempoStack components
is Running
and the conditions
are type: Ready
by running the following command:
$ oc get tempostacks.tempo.grafana.com simplest -o yaml
Verify that all the TempoStack component pods are running by running the following command:
$ oc get pods
Access the Tempo console:
Query the route details by running the following command:
$ oc get route
Open https://<route_from_previous_step>
in a web browser.
The Tempo console initially shows no trace data following the Tempo console installation. |
The TempoMonolithic instance is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope. |
You can install a TempoMonolithic instance by using the web console or the command line.
The TempoMonolithic
custom resource (CR) creates a Tempo deployment in monolithic mode.
All components of the Tempo deployment, such as the compactor, distributor, ingester, querier, and query frontend, are contained in a single container.
A TempoMonolithic instance supports storing traces in in-memory storage, a persistent volume, or object storage.
Tempo deployment in monolithic mode is preferred for a small deployment, demonstration, testing, and as a migration path of the Red Hat OpenShift distributed tracing platform (Jaeger) all-in-one deployment.
The monolithic deployment of Tempo does not scale horizontally.
If you require horizontal scaling, use the |
The TempoMonolithic instance is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope. |
You can install a TempoMonolithic instance from the Administrator view of the web console.
You are logged in to the OpenShift Container Platform web console as a cluster administrator with the cluster-admin
role.
For Red Hat OpenShift Dedicated, you must be logged in using an account with the dedicated-admin
role.
Go to Home → Projects → Create Project to create a project of your choice for the TempoMonolithic instance that you will create in a subsequent step.
Decide which type of supported storage to use for storing traces: in-memory storage, a persistent volume, or object storage.
Object storage is not included with the distributed tracing platform (Tempo) and requires setting up an object store by a supported provider: {odf-full}, MinIO, Amazon S3, Azure Blob Storage, or Google Cloud Storage. Additionally, opting for object storage requires creating a secret for your object storage bucket in the project that you created for the TempoMonolithic instance. You can do this in Workloads → Secrets → Create → From YAML. For more information, see "Object storage setup". Example secret for Amazon S3 and MinIO storage
|
Create a TempoMonolithic instance:
You can create multiple TempoMonolithic instances in separate projects on the same cluster. |
Go to Operators → Installed Operators.
Select TempoMonolithic → Create TempoMonolithic → YAML view.
In the YAML view, customize the TempoMonolithic
custom resource (CR).
The following TempoMonolithic
CR creates a TempoMonolithic deployment with trace ingestion over OTLP/gRPC and OTLP/HTTP, storing traces in a supported type of storage and exposing Jaeger UI via a route:
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoMonolithic
metadata:
name: <metadata_name>
namespace: <project_of_tempomonolithic_instance>
spec:
storage:
traces:
backend: <supported_storage_type> (1)
size: <value>Gi (2)
s3: (3)
secret: <secret_name> (4)
tls: (5)
enabled: true
caName: <ca_certificate_configmap_name> (6)
jaegerui:
enabled: true (7)
route:
enabled: true (8)
resources: (9)
total:
limits:
memory: <value>Gi
cpu: <value>m
1 | Type of storage for storing traces: in-memory storage, a persistent volume, or object storage. The value for a persistent volume is pv . The accepted values for object storage are s3 , gcs , or azure , depending on the used object store type. The default value is memory for the tmpfs in-memory storage, which is only appropriate for development, testing, demonstrations, and proof-of-concept environments because the data does not persist when the pod is shut down. |
2 | Memory size: For in-memory storage, this means the size of the tmpfs volume, where the default is 2Gi . For a persistent volume, this means the size of the persistent volume claim, where the default is 10Gi . For object storage, this means the size of the persistent volume claim for the Tempo WAL, where the default is 10Gi . |
3 | Optional: For object storage, the type of object storage. The accepted values are s3 , gcs , and azure , depending on the used object store type. |
4 | Optional: For object storage, the value of the name in the metadata of the storage secret. The storage secret must be in the same namespace as the TempoMonolithic instance and contain the fields specified in "Table 1. Required secret parameters" in the section "Object storage setup". |
5 | Optional. |
6 | Optional: Name of a ConfigMap object that contains a CA certificate. |
7 | Enables the Jaeger UI. |
8 | Enables creation of a route for the Jaeger UI. |
9 | Optional. |
Select Create.
Use the Project: dropdown list to select the project of the TempoMonolithic instance.
Go to Operators → Installed Operators to verify that the Status of the TempoMonolithic instance is Condition: Ready.
Go to Workloads → Pods to verify that the pod of the TempoMonolithic instance is running.
Access the Jaeger UI:
Go to Networking → routes and Ctrl+F to search for jaegerui
.
The Jaeger UI uses the |
In the Location column, open the URL to access the Jaeger UI.
When the pod of the TempoMonolithic instance is ready, you can send traces to the tempo-<metadata_name_of_TempoMonolithic_CR>:4317
(OTLP/gRPC) and tempo-<metadata_name_of_TempoMonolithic_CR>:4318
(OTLP/HTTP) endpoints inside the cluster.
The Tempo API is available at the tempo-<metadata_name_of_TempoMonolithic_CR>:3200
endpoint inside the cluster.
The TempoMonolithic instance is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope. |
You can install a TempoMonolithic instance from the command line.
An active OpenShift CLI (oc
) session by a cluster administrator with the cluster-admin
role.
|
Run the following command to create a project of your choice for the TempoMonolithic instance that you will create in a subsequent step:
$ oc apply -f - << EOF
apiVersion: project.openshift.io/v1
kind: Project
metadata:
name: <project_of_tempomonolithic_instance>
EOF
Decide which type of supported storage to use for storing traces: in-memory storage, a persistent volume, or object storage.
Object storage is not included with the distributed tracing platform (Tempo) and requires setting up an object store by a supported provider: {odf-full}, MinIO, Amazon S3, Azure Blob Storage, or Google Cloud Storage. Additionally, opting for object storage requires creating a secret for your object storage bucket in the project that you created for the TempoMonolithic instance. You can do this by running the following command:
For more information, see "Object storage setup". Example secret for Amazon S3 and MinIO storage
|
Create a TempoMonolithic instance in the project that you created for it.
You can create multiple TempoMonolithic instances in separate projects on the same cluster. |
Customize the TempoMonolithic
custom resource (CR).
The following TempoMonolithic
CR creates a TempoMonolithic deployment with trace ingestion over OTLP/gRPC and OTLP/HTTP, storing traces in a supported type of storage and exposing Jaeger UI via a route:
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoMonolithic
metadata:
name: <metadata_name>
namespace: <project_of_tempomonolithic_instance>
spec:
storage:
traces:
backend: <supported_storage_type> (1)
size: <value>Gi (2)
s3: (3)
secret: <secret_name> (4)
tls: (5)
enabled: true
caName: <ca_certificate_configmap_name> (6)
jaegerui:
enabled: true (7)
route:
enabled: true (8)
resources: (9)
total:
limits:
memory: <value>Gi
cpu: <value>m
1 | Type of storage for storing traces: in-memory storage, a persistent volume, or object storage. The value for a persistent volume is pv . The accepted values for object storage are s3 , gcs , or azure , depending on the used object store type. The default value is memory for the tmpfs in-memory storage, which is only appropriate for development, testing, demonstrations, and proof-of-concept environments because the data does not persist when the pod is shut down. |
2 | Memory size: For in-memory storage, this means the size of the tmpfs volume, where the default is 2Gi . For a persistent volume, this means the size of the persistent volume claim, where the default is 10Gi . For object storage, this means the size of the persistent volume claim for the Tempo WAL, where the default is 10Gi . |
3 | Optional: For object storage, the type of object storage. The accepted values are s3 , gcs , and azure , depending on the used object store type. |
4 | Optional: For object storage, the value of the name in the metadata of the storage secret. The storage secret must be in the same namespace as the TempoMonolithic instance and contain the fields specified in "Table 1. Required secret parameters" in the section "Object storage setup". |
5 | Optional. |
6 | Optional: Name of a ConfigMap object that contains a CA certificate. |
7 | Enables the Jaeger UI. |
8 | Enables creation of a route for the Jaeger UI. |
9 | Optional. |
Apply the customized CR by running the following command:
$ oc apply -f - << EOF
<tempomonolithic_cr>
EOF
Verify that the status
of all TempoMonolithic components
is Running
and the conditions
are type: Ready
by running the following command:
$ oc get tempomonolithic.tempo.grafana.com <metadata_name_of_tempomonolithic_cr> -o yaml
Run the following command to verify that the pod of the TempoMonolithic instance is running:
$ oc get pods
Access the Jaeger UI:
Query the route details for the tempo-<metadata_name_of_tempomonolithic_cr>-jaegerui
route by running the following command:
$ oc get route
Open https://<route_from_previous_step>
in a web browser.
When the pod of the TempoMonolithic instance is ready, you can send traces to the tempo-<metadata_name_of_tempomonolithic_cr>:4317
(OTLP/gRPC) and tempo-<metadata_name_of_tempomonolithic_cr>:4318
(OTLP/HTTP) endpoints inside the cluster.
The Tempo API is available at the tempo-<metadata_name_of_tempomonolithic_cr>:3200
endpoint inside the cluster.
You can use the following configuration parameters when setting up a supported object storage.
Storage provider | Secret parameters |
---|---|
|
|
MinIO |
See MinIO Operator.
|
Amazon S3 |
|
Amazon S3 with Security Token Service (STS) |
|
Microsoft Azure Blob Storage |
|
Google Cloud Storage on Google Cloud Platform (GCP) |
|
You can set up the Amazon S3 storage with the Security Token Service (STS) by using the AWS Command Line Interface (AWS CLI).
The Amazon S3 storage with the Security Token Service is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope. |
You have installed the latest version of the AWS CLI.
Create an AWS S3 bucket.
Create the following trust.json
file for the AWS IAM policy that will set up a trust relationship for the AWS IAM role, created in the next step, with the service account of the TempoStack instance:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::${<aws_account_id>}:oidc-provider/${<oidc_provider>}" (1)
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"${OIDC_PROVIDER}:sub": [
"system:serviceaccount:${<openshift_project_for_tempostack>}:tempo-${<tempostack_cr_name>}" (2)
"system:serviceaccount:${<openshift_project_for_tempostack>}:tempo-${<tempostack_cr_name>}-query-frontend"
]
}
}
}
]
}
1 | OIDC provider that you have configured on the OpenShift Container Platform. You can get the configured OIDC provider value also by running the following command: $ oc get authentication cluster -o json | jq -r '.spec.serviceAccountIssuer' | sed 'shttp[s]*://~g' . |
2 | Namespace in which you intend to create the TempoStack instance. |
Create an AWS IAM role by attaching the trust.json
policy file that you created:
$ aws iam create-role \
--role-name "tempo-s3-access" \
--assume-role-policy-document "file:///tmp/trust.json" \
--query Role.Arn \
--output text
Attach an AWS IAM policy to the created role:
$ aws iam attach-role-policy \
--role-name "tempo-s3-access" \
--policy-arn "arn:aws:iam::aws:policy/AmazonS3FullAccess"
In the OpenShift Container Platform, create an object storage secret with keys as follows:
apiVersion: v1
kind: Secret
metadata:
name: minio-test
stringData:
bucket: <s3_bucket_name>
region: <s3_region>
role_arn: <s3_role_arn>
type: Opaque