Using Tekton Chains for OpenShift Pipelines supply chain security - Pipelines | CI/CD

Key features
Installing Tekton Chains using the Red Hat OpenShift Pipelines Operator
Configuring Tekton Chains
- Supported keys for Tekton Chains configuration
Signing secrets in Tekton Chains
Authenticating to an OCI registry
- Creating and verifying task run signatures without any additional authentication
Using Tekton Chains to sign and verify image and provenance
Additional resources

Tekton Chains is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Tekton Chains is a Kubernetes Custom Resource Definition (CRD) controller. You can use it to manage the supply chain security of the tasks and pipelines created using Red Hat OpenShift Pipelines.

By default, Tekton Chains observes all task run executions in your OpenShift Container Platform cluster. When the task runs complete, Tekton Chains takes a snapshot of the task runs. It then converts the snapshot to one or more standard payload formats, and finally signs and stores all artifacts.

To capture information about task runs, Tekton Chains uses the Result and PipelineResource objects. When the objects are unavailable, Tekton Chains the URLs and qualified digests of the OCI images.

The PipelineResource object is deprecated and will be removed in a future release; for manual use, the Results object is recommended.

Key features

You can sign task runs, task run results, and OCI registry images with cryptographic key types and services such as cosign.
You can use attestation formats such as in-toto.
You can securely store signatures and signed artifacts using OCI repository as a storage backend.

Installing Tekton Chains using the Red Hat OpenShift Pipelines Operator

Cluster administrators can use the TektonChain custom resource (CR) to install and manage Tekton Chains.

Tekton Chains is an optional component of Red Hat OpenShift Pipelines. Currently, you cannot install it using the TektonConfig CR.

Prerequisites

Ensure that the Red Hat OpenShift Pipelines Operator is installed in the openshift-pipelines namespace on your cluster.

Procedure

Create the TektonChain CR for your OpenShift Container Platform cluster.

apiVersion: operator.tekton.dev/v1alpha1
kind: TektonChain
metadata:
  name: chain
spec:
  targetNamespace: openshift-pipelines

Apply the TektonChain CR.
```
$ oc apply -f TektonChain.yaml (1)
```
1 Substitute with the file name of the TektonChain CR.

Check the status of the installation.

$ oc get tektonchains.operator.tekton.dev

Configuring Tekton Chains

Tekton Chains uses a ConfigMap object named chains-config in the openshift-pipelines namespace for configuration.

To configure Tekton Chains, use the following example:

Example: Configuring Tekton Chains

$ oc patch configmap chains-config -n openshift-pipelines -p='{"data":{"artifacts.oci.storage": "", "artifacts.taskrun.format":"tekton", "artifacts.taskrun.storage": "tekton"}}' (1)

1	Use a combination of supported key-value pairs in the JSON payload.

Supported keys for Tekton Chains configuration

Cluster administrators can use various supported keys and values to configure specifications about task runs, OCI images, and storage.

Supported keys for task run

Table 1. Chains configuration: Supported keys for task run
Supported keys	Description	Supported values	Default values
`artifacts.taskrun.format`	The format to store task run payloads.	`tekton`, `in-toto`	`tekton`
`artifacts.taskrun.storage`	The storage backend for task run signatures. You can specify multiple backends as a comma-separated list, such as `“tekton,oci”`. To disable this artifact, provide an empty string `“”`.	`tekton`, `oci`	`tekton`
`artifacts.taskrun.signer`	The signature backend to sign task run payloads.	`x509`	`x509`

Supported keys for OCI

Table 2. Chains configuration: Supported keys for OCI
Supported keys	Description	Supported values	Default values
`artifacts.oci.format`	The format to store OCI payloads.	`simplesigning`	`simplesigning`
`artifacts.oci.storage`	The storage backend to for OCI signatures. You can specify multiple backends as a comma-separated list, such as `“oci,tekton”`. To disable the OCI artifact, provide an empty string `“”`.	`tekton`, `oci`	`oci`
`artifacts.oci.signer`	The signature backend to sign OCI payloads.	`x509`, `cosign`	`x509`

Supported keys for storage

Table 3. Chains configuration: Supported keys for storage
Supported keys	Description	Supported values	Default values
`artifacts.oci.repository`	The OCI repository to store OCI signatures.	Currently, Chains support only the internal OpenShift OCI registry; other popular options such as Quay is not supported.

Signing secrets in Tekton Chains

Cluster administrators can generate a key pair and use Tekton Chains to sign artifacts using a Kubernetes secret. For Tekton Chains to work, a private key and a password for encrypted keys must exist as part of the signing-secrets Kubernetes secret, in the openshift-pipelines namespace.

Currently, Tekton Chains supports the x509 and cosign signature schemes.

Use only one of the supported signature schemes.

Signing using x509

To use the x509 signing scheme with Tekton Chains, store the x509.pem private key of the ed25519 or ecdsa type in the signing-secrets Kubernetes secret. Ensure that the key is stored as an unencrypted PKCS8 PEM file (BEGIN PRIVATE KEY).

Signing using cosign

To use the cosign signing scheme with Tekton Chains:

Install cosign.
Generate the cosign.key and cosign.pub key pairs.
```
$ cosign generate-key-pair k8s://openshift-pipelines/signing-secrets
```
Cosign prompts you for a password, and creates a Kubernetes secret.
Store the encrypted cosign.key private key and the cosign.password decryption password in the signing-secrets Kubernetes secret. Ensure that the private key is stored as an encrypted PEM file of the ENCRYPTED COSIGN PRIVATE KEY type.

Troubleshooting signing

If the signing secrets are already populated, you might get the following error:

Error from server (AlreadyExists): secrets "signing-secrets" already exists

To resolve the error:

Delete the secrets:

$ oc delete secret signing-secrets -n openshift-pipelines

Recreate the key pairs and store them in the secrets using your preferred signing scheme.

Authenticating to an OCI registry

Before pushing signatures to an OCI registry, cluster administrators must configure Tekton Chains to authenticate with the registry. The Tekton Chains controller uses the same service account under which the task runs execute. To set up a service account with the necessary credentials for pushing signatures to an OCI registry, perform the following steps:

Procedure

Set the namespace and name of the Kubernetes service account.
```
$ export NAMESPACE=<namespace> (1)
$ export SERVICE_ACCOUNT_NAME=<service_account> (2)
```
1 The namespace associated with the service account.

2 The name of the service account.

Create a Kubernetes secret.

$ oc create secret registry-credentials \
  --from-file=.dockerconfigjson \ (1)
  --type=kubernetes.io/dockerconfigjson \
  -n $NAMESPACE

1	Substitute with the path to your Docker config file. Default path is `~/.docker/config.json`.

Give the service account access to the secret.
```
$ oc patch serviceaccount $SERVICE_ACCOUNT_NAME \
  -p "{\"imagePullsecrets\": [{\"name\": \"registry-credentials\"}]}" -n $NAMESPACE
```
If you patch the default pipeline service account that Red Hat OpenShift Pipelines assigns to all task runs, the Red Hat OpenShift Pipelines Operator will override the service account. As a best practice, you can perform the following steps:
1. Create a separate service account to assign to user’s task runs.
  $ oc create serviceaccount <service_account_name>
2. Associate the service account to the task runs by setting the value of the serviceaccountname field in the task run template.
  apiVersion: tekton.dev/v1beta1 kind: TaskRun metadata: name: build-push-task-run-2 spec: serviceAccountName: build-bot (1) taskRef: name: build-push ...
  1 Substitute with the name of the newly created service account.

Creating and verifying task run signatures without any additional authentication

To verify signatures of task runs using Tekton Chains with any additional authentication, perform the following tasks:

Create an encrypted x509 key pair and save it as a Kubernetes secret.
Configure the Tekton Chains backend storage.
Create a task run, sign it, and store the signature and the payload as annotations on the task run itself.
Retrieve the signature and payload from the signed task run.
Verify the signature of the task run.

Prerequisites

Ensure that the following are installed on the cluster:

Red Hat OpenShift Pipelines Operator
Tekton Chains
Cosign

Procedure

Create an encrypted x509 key pair and save it as a Kubernetes secret:
```
$ cosign generate-key-pair k8s://openshift-pipelines/signing-secrets
```
Provide a password when prompted. Cosign stores the resulting private key as part of the signing-secrets Kubernetes secret in the openshift-pipelines namespace.

In the Tekton Chains configuration, disable the OCI storage, and set the task run storage and format to tekton.

$ oc patch configmap chains-config -n openshift-pipelines -p='{"data":{"artifacts.oci.storage": "", "artifacts.taskrun.format":"tekton", "artifacts.taskrun.storage": "tekton"}}'

Restart the Tekton Chains controller to ensure that the modified configuration is applied.
```
$ oc delete po -n openshift-pipelines -l app=tekton-chains-controller
```

Create a task run.

$ oc create -f https://raw.githubusercontent.com/tektoncd/chains/main/examples/taskruns/task-output-image.yaml (1)
taskrun.tekton.dev/build-push-run-output-image-qbjvh created

1	Substitute with the URI or file path pointing to your task run.

Check the status of the steps, and wait till the process finishes.

$ tkn tr describe --last
[...truncated output...]
NAME                            STATUS
∙ create-dir-builtimage-9467f   Completed
∙ git-source-sourcerepo-p2sk8   Completed
∙ build-and-push                Completed
∙ echo                          Completed
∙ image-digest-exporter-xlkn7   Completed

Retrieve the signature and payload from the object stored as base64 encoded annotations:

$ export TASKRUN_UID=$(tkn tr describe --last -o  jsonpath='{.metadata.uid}')
$ tkn tr describe --last -o jsonpath="{.metadata.annotations.chains\.tekton\.dev/signature-taskrun-$TASKRUN_UID}" > signature
$ tkn tr describe --last -o jsonpath="{.metadata.annotations.chains\.tekton\.dev/payload-taskrun-$TASKRUN_UID}" | base64 -d > payload

Verify the signature.

$ cosign verify-blob --key k8s://openshift-pipelines/signing-secrets --signature ./signature ./payload
Verified OK

Using Tekton Chains to sign and verify image and provenance

Cluster administrators can use Tekton Chains to sign and verify images and provenances, by performing the following tasks:

Create an encrypted x509 key pair and save it as a Kubernetes secret.
Set up authentication for the OCI registry to store images, image signatures, and signed image attestations.
Configure Tekton Chains to generate and sign provenance.
Create an image with Kaniko in a task run.
Verify the signed image and the signed provenance.

Prerequisites

Ensure that the following are installed on the cluster:

Red Hat OpenShift Pipelines Operator
Tekton Chains
Cosign
Rekor
jq

Procedure

Create an encrypted x509 key pair and save it as a Kubernetes secret:
```
$ cosign generate-key-pair k8s://openshift-pipelines/signing-secrets
```
Provide a password when prompted. Cosign stores the resulting private key as part of the signing-secrets Kubernetes secret in the openshift-pipelines namespace, and writes the public key to the cosign.pub local file.
Configure authentication for the image registry.
1. To configure the Tekton Chains controller for pushing signature to an OCI registry, use the credentials associated with the service account of the task run. For detailed information, see the "Authenticating to an OCI registry" section.
2. To configure authentication for a Kaniko task that builds and pushes image to the registry, create a Kubernetes secret of the docker config.json file containing the required credentials.
  $ oc create secret generic <docker_config_secret_name> \ (1) --from-file <path_to_config.json> (2)
  1 Substitute with the name of the docker config secret.
  
  2 Substitute with the path to docker config.json file.

Configure Tekton Chains by setting the artifacts.taskrun.format, artifacts.taskrun.storage, and transparency.enabled parameters in the chains-config object:

$ oc patch configmap chains-config -n openshift-pipelines -p='{"data":{"artifacts.taskrun.format": "in-toto"}}'

$ oc patch configmap chains-config -n openshift-pipelines -p='{"data":{"artifacts.taskrun.storage": "oci"}}'

$ oc patch configmap chains-config -n openshift-pipelines -p='{"data":{"transparency.enabled": "true"}}'

Start the Kaniko task.
1. Apply the Kaniko task to the cluster.
  $ oc apply -f examples/kaniko/kaniko.yaml (1)
  1 Substitute with the URI or file path to your Kaniko task.
2. Set the appropriate environment variables.
  $ export REGISTRY=<url_of_registry> (1) $ export DOCKERCONFIG_SECRET_NAME=<name_of_the_secret_in_docker_config_json> (2)
  1 Substitute with the URL of the registry where you want to push the image.
  
  2 Substitute with the name of the secret in the docker config.json file.
3. Start the Kaniko task.
  $ tkn task start --param IMAGE=$REGISTRY/kaniko-chains --use-param-defaults --workspace name=source,emptyDir="" --workspace name=dockerconfig,secret=$DOCKERCONFIG_SECRET_NAME kaniko-chains
  Observe the logs of this task until all steps are complete. On successful authentication, the final image will be pushed to $REGISTRY/kaniko-chains.
Wait for a minute to allow Tekton Chains to generate the provenance and sign it, and then check the availability of the chains.tekton.dev/signed=true annotation on the task run.
```
$ oc get tr <task_run_name> \ (1)
-o json | jq -r .metadata.annotations

{
  "chains.tekton.dev/signed": "true",
  ...
}
```
1 Substitute with the name of the task run.

Verify the image and the attestation.

$ cosign verify --key cosign.pub $REGISTRY/kaniko-chains

$ cosign verify-attestation --key cosign.pub $REGISTRY/kaniko-chains

Find the provenance for the image in Rekor.
1. Get the digest of the $REGISTRY/kaniko-chains image. You can search for it ing the task run, or pull the image to extract the digest.
2. Search Rekor to find all entries that match the sha256 digest of the image.
  $ rekor-cli search --sha <image_digest> (1) <uuid_1> (2) <uuid_2> (3) ...
  1 Substitute with the sha256 digest of the image.
  
  2 The first matching universally unique identifier (UUID).
  
  3 The second matching UUID.
  
  The search result displays UUIDs of the matching entries. One of those UUIDs holds the attestation.
3. Check the attestation.
  $ rekor-cli get --uuid <uuid> --format json | jq -r .Attestation | base64 --decode | jq

Additional resources

Installing OpenShift Pipelines

1	The namespace associated with the service account.
2	The name of the service account.

1	Substitute with the name of the docker config secret.
2	Substitute with the path to docker `config.json` file.

1	Substitute with the URL of the registry where you want to push the image.
2	Substitute with the name of the secret in the docker `config.json` file.

1	Substitute with the `sha256` digest of the image.
2	The first matching universally unique identifier (UUID).
3	The second matching UUID.