This is a cache of https://docs.openshift.com/container-platform/4.6/operators/operator_sdk/osdk-scorecard.html. It is a snapshot of the page at 2024-11-25T23:27:05.772+0000.
Validating Operators using the scorecard - Developing Operators | Operators | OpenShift Container Platform 4.6
    OpenShift docs are moving and will soon only be available at docs.redhat.com, the home of all Red Hat product documentation. Explore the new docs experience today.
  1. Documentation
    2. history
×

Validating Operators using the scorecard

Operator authors should validate that their Operator is packaged correctly and free of syntax errors. As an Operator author, you can use the Operator SDK scorecard tool to validate your Operator packaging and run tests.

OpenShift Container Platform 4.6 supports Operator SDK v0.19.4.

About the scorecard tool

To validate an Operator, the scorecard tool provided by the Operator SDK begins by creating all resources required by any related custom resources (CRs) and the Operator. The scorecard then creates a proxy container in the deployment of the Operator which is used to record calls to the API server and run some of the tests. The tests performed also examine some of the parameters in the CRs.

Scorecard configuration

The scorecard tool uses a configuration file that allows you to configure internal plug-ins, as well as several global configuration options.

Configuration file

The default location for the scorecard tool configuration is the <project_dir>/.osdk-scorecard.*. The following is an example of a YAML-formatted configuration file:

Scorecard configuration file
scorecard:
  output: json
  plugins:
    - basic: (1)
        cr-manifest:
          - "deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml"
          - "deploy/crds/cache.example.com_v1alpha1_memcachedrs_cr.yaml"
    - olm: (2)
        cr-manifest:
          - "deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml"
          - "deploy/crds/cache.example.com_v1alpha1_memcachedrs_cr.yaml"
        csv-path: "deploy/olm-catalog/memcached-operator/0.0.3/memcached-operator.v0.0.3.clusterserviceversion.yaml"
1 basic tests configured to test two custom resources (CRs).
2 olm tests configured to test two CRs.

Configuration methods for global options take the following priority, highest to lowest:

Command arguments (if available) → configuration file → default

The configuration file must be in YAML format. As the configuration file might be extended to allow configuration of all operator-sdk subcommands in the future, the scorecard configuration must be under a scorecard subsection.

Configuration file support is provided by the viper package. For more info on how viper configuration works, see the README.

Command arguments

While most of the scorecard tool configuration is done using a configuration file, you can also use the following arguments:

Table 1. Scorecard tool arguments
Flag Type Description

--bundle, -b

string

The path to a bundle directory used for the bundle validation test.

--config

string

The path to the scorecard configuration file. The default is <project_dir>/.osdk-scorecard. The file type and extension must be .yaml. If a configuration file is not provided or found at the default location, the scorecard exits with an error.

--output, -o

string

Output format. Valid options are text and json. The default format is text, which is designed to be a human readable format. The json format uses the JSON schema output format used for plug-ins defined later.

--kubeconfig, -o

string

The path to the kubeconfig file. It sets the kubeconfig for internal plug-ins.

--version

string

The version of scorecard to run. The default and only valid option is v1alpha2.

--selector, -l

string

The label selector to filter tests on.

--list, -L

bool

If true, only print the test names that would be run based on selector filtering.

Configuration file options

The scorecard configuration file provides the following options:

Table 2. Scorecard configuration file options
Option Type Description

bundle

string

Equivalent of the --bundle flag. Operator Lifecycle Manager (OLM) bundle directory path, when specified, runs bundle validation.

output

string

Equivalent of the --output flag. If this option is defined by both the configuration file and the flag, the flag value takes priority.

kubeconfig

string

Equivalent of the --kubeconfig flag. If this option is defined by both the configuration file and the flag, the flag value takes priority.

plugins

array

An array of plug-in names.

Basic and OLM plug-ins

The scorecard supports the internal basic and olm plug-ins, which are configured by a plugins section in the configuration file.

Table 3. Plug-in options
Option Type Description

cr-manifest

[]string

The path(s) for CRs being tested. Required if olm-deployed is unset or false.

csv-path

string

The path to the cluster service version (CSV) for the Operator. Required for OLM tests or if olm-deployed is set to true.

olm-deployed

bool

Indicates that the CSV and relevant CRDs have been deployed onto the cluster by OLM.

kubeconfig

string

The path to the kubeconfig file. If both the global kubeconfig and this field are set, this field is used for the plug-in.

namespace

string

The namespace to run the plug-ins in. If unset, the default specified by the kubeconfig file is used.

init-timeout

int

Time in seconds until a timeout during initialization of the Operator.

crds-dir

string

The path to the directory containing CRDs that must be deployed to the cluster.

namespaced-manifest

string

The manifest file with all resources that run within a namespace. By default, the scorecard combines the service_account.yaml, role.yaml, role_binding.yaml, and operator.yaml files from the deploy directory into a temporary manifest to use as the namespaced manifest.

global-manifest

string

The manifest containing required resources that run globally (not namespaced). By default, the scorecard combines all CRDs in the crds-dir directory into a temporary manifest to use as the global manifest.

Currently, using the scorecard with a CSV does not permit multiple CR manifests to be set through the CLI, configuration file, or CSV annotations. You must tear down your Operator in the cluster, re-deploy, and re-run the scorecard for each CR that is tested.
Additional resources

  • You can either set cr-manifest or your CSV metadata.annotations['alm-examples'] to provide CRs to the scorecard, but not both. See CRD templates for details.

Tests performed

By default, the scorecard tool has a set of internal tests it can run available across two internal plug-ins. If multiple CRs are specified for a plug-in, the test environment is fully cleaned up after each CR so that each CR gets a clean testing environment.

Each test has a short name that uniquely identifies the test. This is useful when selecting a specific test or tests to run. For example:

$ operator-sdk scorecard -o text --selector=test=checkspectest
$ operator-sdk scorecard -o text --selector='test in (checkspectest,checkstatustest)'

Basic plug-in

The following basic Operator tests are available from the basic plug-in:

Table 4. basic plug-in tests
Test Description Short name

Spec Block Exists

This test checks the custom resources (CRs) created in the cluster to make sure that all CRs have a spec block. This test has a maximum score of 1.

checkspectest

Status Block Exists

This test checks the CRs created in the cluster to make sure that all CRs have a status block. This test has a maximum score of 1.

checkstatustest

Writing Into CRs Has An Effect

This test reads the scorecard proxy logs to verify that the Operator is making PUT or POST, or both, requests to the API server, indicating that it is modifying resources. This test has a maximum score of 1.

writingintocrshaseffecttest

OLM plug-in

The following Operator Lifecycle Manager (OLM) integration tests are available from the olm plug-in:

Table 5. olm plug-in tests
Test Description Short name

OLM Bundle Validation

This test validates the OLM bundle manifests found in the bundle directory as specified by the bundle flag. If the bundle contents contain errors, then the test result output includes the validator log as well as error messages from the validation library.

bundlevalidationtest

Provided APIs Have Validation

This test verifies that the CRDs for the provided CRs contain a validation section and that there is validation for each spec and status field detected in the CR. This test has a maximum score equal to the number of CRs provided by the cr-manifest option.

crdshavevalidationtest

Owned CRDs Have Resources Listed

This test makes sure that the CRDs for each CR provided by the cr-manifest option have a resources subsection in the owned CRDs section of the CSV. If the test detects used resources that are not listed in the resources section, it lists them in the suggestions at the end of the test. This test has a maximum score equal to the number of CRs provided by the cr-manifest option.

crdshaveresourcestest

Spec Fields With Descriptors

This test verifies that every field in the spec sections of custom resources have a corresponding descriptor listed in the CSV. This test has a maximum score equal to the total number of fields in the spec sections of each custom resource passed in by the cr-manifest option.

specdescriptorstest

Status Fields With Descriptors

This test verifies that every field in the status sections of custom resources have a corresponding descriptor listed in the CSV. This test has a maximum score equal to the total number of fields in the status sections of each custom resource passed in by the cr-manifest option.

statusdescriptorstest
Additional resources

Running the scorecard

Prerequisites

The following prerequisites for the Operator project are checked by the scorecard tool:

  • Access to a cluster running Kubernetes 1.11.3 or later.

  • If you want to use the scorecard to check the integration of your Operator project with Operator Lifecycle Manager (OLM), then a cluster service version (CSV) file is also required. This is a requirement when the olm-deployed option is used.

  • For Operators that were not generated using the Operator SDK (non-SDK Operators):

    • Resource manifests for installing and configuring the Operator and custom resources (CRs).

    • Configuration getter that supports reading from the KUBECONFIG environment variable, such as the clientcmd or controller-runtime configuration getters. This is required for the scorecard proxy to work correctly.

Procedure

  1. Define a .osdk-scorecard.yaml configuration file in your Operator project.

  2. Create the namespace defined in the RBAC files (role_binding).

  3. Run the scorecard from the root directory of your Operator project:

    $ operator-sdk scorecard

    The scorecard return code is 1 if any of the executed texts did not pass and 0 if all selected tests passed.

Running the scorecard with an OLM-managed Operator

The scorecard can be run using a cluster service version (CSV), providing a way to test cluster-ready and non-Operator SDK Operators.

Procedure

  1. The scorecard requires a proxy container in the deployment pod of the Operator to read Operator logs. A few modifications to your CSV and creation of one extra object are required to run the proxy before deploying your Operator with Operator Lifecycle Manager (OLM).

    This step can be performed manually or automated using bash functions. Choose one of the following methods.

    • Manual method:

      1. Create a proxy server secret containing a local kubeconfig file`.

        1. Generate a user name using the namespaced owner reference of the scorecard proxy.

          $ echo '{"apiVersion":"","kind":"","name":"scorecard","uid":"","Namespace":"'<namespace>'"}' | base64 -w 0 (1)
          1 Replace <namespace> with the namespace your Operator will deploy in.

        2. Write a Config manifest scorecard-config.yaml using the following template, replacing <username> with the base64 user name generated in the previous step:

          apiVersion: v1
kind: Config
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: http://<username>@localhost:8889
  name: proxy-server
contexts:
- context:
    cluster: proxy-server
    user: admin/proxy-server
  name: <namespace>/proxy-server
current-context: <namespace>/proxy-server
preferences: {}
users:
- name: admin/proxy-server
  user:
    username: <username>
    password: unused

        3. Encode the Config as base64:

          $ cat scorecard-config.yaml | base64 -w 0

        4. Create a secret manifest scorecard-secret.yaml:

          apiVersion: v1
kind: secret
metadata:
  name: scorecard-kubeconfig
  namespace: <namespace> (1)
data:
  kubeconfig: <kubeconfig_base64> (2)
          1 Replace <namespace> with the namespace your Operator will deploy in.
          2 Replace <kubeconfig_base64> with the Config encoded as base64.

        5. Apply the secret:

          $ oc apply -f scorecard-secret.yaml

        6. Insert a volume referring to the secret into the deployment for the Operator:

          spec:
  install:
    spec:
      deployments:
      - name: memcached-operator
        spec:
          ...
          template:
            ...
            spec:
              containers:
              ...
              volumes:
              - name: scorecard-kubeconfig (1)
                secret:
                  secretName: scorecard-kubeconfig
                  items:
                  - key: kubeconfig
                    path: config
          1 Scorecard kubeconfig volume.

      2. Insert a volume mount and KUBECONFIG environment variable into each container in the deployment of your Operator:

        spec:
  install:
    spec:
      deployments:
      - name: memcached-operator
        spec:
          ...
          template:
            ...
            spec:
              containers:
              - name: container1
                ...
                volumeMounts:
                - name: scorecard-kubeconfig (1)
                  mountPath: /scorecard-secret
                env:
                - name: KUBECONFIG (2)
                  value: /scorecard-secret/config
              - name: container2 (3)
                ...
        1 Scorecard kubeconfig volume mount.
        2 Scorecard kubeconfig environment variable.
        3 Repeat the same for this and all other containers.

      3. Insert the scorecard proxy container into the deployment of your Operator:

        spec:
  install:
    spec:
      deployments:
      - name: memcached-operator
        spec:
          ...
          template:
            ...
            spec:
              containers:
              ...
              - name: scorecard-proxy (1)
                command:
                - scorecard-proxy
                env:
                - name: WATCH_NAMESPACE
                  valueFrom:
                    fieldRef:
                      apiVersion: v1
                      fieldPath: metadata.namespace
                image: quay.io/operator-framework/scorecard-proxy:master
                imagePullPolicy: Always
                ports:
                - name: proxy
                  containerPort: 8889
        1 Scorecard proxy container.

    • Automated method:

      The community-operators repository has several bash functions that can perform the previous steps in the procedure for you.

      1. Run the following curl command:

        $ curl -Lo csv-manifest-modifiers.sh \
    https://raw.githubusercontent.com/operator-framework/community-operators/master/scripts/lib/file

      2. Source the csv-manifest-modifiers.sh file:

        $ . ./csv-manifest-modifiers.sh

      3. Create the kubeconfig secret file:

        $ create_kubeconfig_secret_file scorecard-secret.yaml "<namespace>" (1)
        1 Replace <namespace> with the namespace your Operator will deploy in.

      4. Apply the secret:

        $ oc apply -f scorecard-secret.yaml

      5. Insert the kubeconfig volume:

        $ insert_kubeconfig_volume "<csv_file>" (1)
        1 Replace <csv_file> with the path to your CSV manifest.

      6. Insert the kubeconfig secret mount:

        $ insert_kubeconfig_secret_mount "<csv_file>"

      7. Insert the proxy container:

        $ insert_proxy_container "<csv_file>" "quay.io/operator-framework/scorecard-proxy:master"

  2. After inserting the proxy container, follow the steps in the Getting started with the Operator SDK guide to bundle your CSV and custom resource definitions (CRDs) and deploy your Operator on OLM.

  3. After your Operator has been deployed on OLM, define a .osdk-scorecard.yaml configuration file in your Operator project and ensure both the csv-path: <csv_manifest_path> and olm-deployed options are set.

  4. Run the scorecard with both the csv-path: <csv_manifest_path> and olm-deployed options set in your scorecard configuration file:

    $ operator-sdk scorecard