Hybrid Helm Operator - Developing Operators | Operators

Prerequisites
Creating a project
Creating a Helm API
- Operator logic for the Helm API
- Custom Helm reconciler configurations using provided library APIs
Creating a Go API
Running locally outside the cluster
Running as a deployment on the cluster
Creating custom resources
Project layout

The standard Helm-based Operator support in the Operator SDK has limited functionality compared to the Go-based and Ansible-based Operator support that has reached the Auto Pilot capability (level V) in the Operator maturity model.

The Hybrid Helm Operator enhances the existing Helm-based support’s abilities through Go APIs. With this hybrid approach of Helm and Go, the Operator SDK enables Operator authors to use the following process:

Generate a default structure for, or scaffold, a Go API in the same project as Helm.
Configure the Helm reconciler in the main.go file of the project, through the libraries provided by the Hybrid Helm Operator.

The Hybrid Helm Operator is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

This tutorial walks through the following process using the Hybrid Helm Operator:

Create a Memcached deployment through a Helm chart if it does not exist
Ensure that the deployment size is the same as specified by Memcached custom resource (CR) spec
Create a MemcachedBackup deployment by using the Go API

Prerequisites

Operator SDK CLI installed
OpenShift CLI (oc) 4.15+ installed
Logged into an OKD 4.15 cluster with oc with an account that has cluster-admin permissions
To allow the cluster to pull the image, the repository where you push your image must be set as public, or you must configure an image pull secret

Additional resources

Creating a project

Use the Operator SDK CLI to create a project called memcached-operator.

Procedure

Create a directory for the project:

$ mkdir -p $HOME/github.com/example/memcached-operator

Change to the directory:

$ cd $HOME/github.com/example/memcached-operator

Run the operator-sdk init command to initialize the project. This example uses a domain of my.domain so that all API groups are <group>.my.domain:
```
$ operator-sdk init \
    --plugins=hybrid.helm.sdk.operatorframework.io \
    --project-version="3" \
    --domain my.domain \
    --repo=github.com/example/memcached-operator
```
The init command generates the RBAC rules in the config/rbac/role.yaml file based on the resources that would be deployed by the chart’s default manifests. Verify that the rules generated in the config/rbac/role.yaml file meet your Operator’s permission requirements.

Additional resources

This procedure creates a project structure that is compatible with both Helm and Go APIs. To learn more about the project directory structure, see Project layout.

Creating a Helm API

Use the Operator SDK CLI to create a Helm API.

Procedure

Run the following command to create a Helm API with group cache, version v1, and kind Memcached:

$ operator-sdk create api \
    --plugins helm.sdk.operatorframework.io/v1 \
    --group cache \
    --version v1 \
    --kind Memcached

This procedure also configures your Operator project to watch the Memcached resource with API version v1 and scaffolds a boilerplate Helm chart. Instead of creating the project from the boilerplate Helm chart scaffolded by the Operator SDK, you can alternatively use an existing chart from your local file system or remote chart repository.

For more details and examples for creating Helm API based on existing or new charts, run the following command:

$ operator-sdk create api --plugins helm.sdk.operatorframework.io/v1 --help

Additional resources

Existing Helm charts

Operator logic for the Helm API

By default, your scaffolded Operator project watches Memcached resource events as shown in the watches.yaml file and executes Helm releases using the specified chart.

Example watches.yaml file

# Use the 'create api' subcommand to add watches to this file.
- group: cache.my.domain
  version: v1
  kind: Memcached
  chart: helm-charts/memcached
#+kubebuilder:scaffold:watch

Additional resources

For detailed documentation on customizing the Helm Operator logic through the chart, see Understanding the Operator logic.

Custom Helm reconciler configurations using provided library APIs

A disadvantage of existing Helm-based Operators is the inability to configure the Helm reconciler, because it is abstracted from users. For a Helm-based Operator to reach the Seamless Upgrades capability (level II and later) that reuses an already existing Helm chart, a hybrid between the Go and Helm Operator types adds value.

The APIs provided in the helm-operator-plugins library allow Operator authors to make the following configurations:

Customize value mapping based on cluster state
Execute code in specific events by configuring the reconciler’s event recorder
Customize the reconciler’s logger
Setup Install, Upgrade, and Uninstall annotations to enable Helm’s actions to be configured based on the annotations found in custom resources watched by the reconciler
Configure the reconciler to run with Pre and Post hooks

The above configurations to the reconciler can be done in the main.go file:

Details

Example main.go file

// Operator's main.go
// With the help of helpers provided in the library, the reconciler can be
// configured here before starting the controller with this reconciler.
reconciler := reconciler.New(
 reconciler.WithChart(*chart),
 reconciler.WithGroupVersionKind(gvk),
)

if err := reconciler.SetupWithManager(mgr); err != nil {
 panic(fmt.Sprintf("unable to create reconciler: %s", err))
}

Creating a Go API

Use the Operator SDK CLI to create a Go API.

Procedure

Run the following command to create a Go API with group cache, version v1, and kind MemcachedBackup:

$ operator-sdk create api \
    --group=cache \
    --version v1 \
    --kind MemcachedBackup \
    --resource \
    --controller \
    --plugins=go/v3

When prompted, enter y for creating both resource and controller:
```
$ Create Resource [y/n]
y
Create Controller [y/n]
y
```

This procedure generates the MemcachedBackup resource API at api/v1/memcachedbackup_types.go and the controller at controllers/memcachedbackup_controller.go.

Defining the API

Define the API for the MemcachedBackup custom resource (CR).

Represent this Go API by defining the MemcachedBackup type, which will have a MemcachedBackupSpec.Size field to set the quantity of Memcached backup instances (CRs) to be deployed, and a MemcachedBackupStatus.Nodes field to store a CR’s pod names.

The Node field is used to illustrate an example of a Status field.

Procedure

Define the API for the MemcachedBackup CR by modifying the Go type definitions in the api/v1/memcachedbackup_types.go file to have the following spec and status:

Example api/v1/memcachedbackup_types.go file

// MemcachedBackupSpec defines the desired state of MemcachedBackup
type MemcachedBackupSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file

	//+kubebuilder:validation:Minimum=0
	// Size is the size of the memcached deployment
	Size int32 `json:"size"`
}

// MemcachedBackupStatus defines the observed state of MemcachedBackup
type MemcachedBackupStatus struct {
	// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
	// Important: Run "make" to regenerate code after modifying this file
	// Nodes are the names of the memcached pods
	Nodes []string `json:"nodes"`
}

Update the generated code for the resource type:
```
$ make generate
```
After you modify a *_types.go file, you must run the make generate command to update the generated code for that resource type.
After the API is defined with spec and status fields and CRD validation markers, generate and update the CRD manifests:
```
$ make manifests
```

This Makefile target invokes the controller-gen utility to generate the CRD manifests in the config/crd/bases/cache.my.domain_memcachedbackups.yaml file.

Controller implementation

The controller in this tutorial performs the following actions:

Create a Memcached deployment if it does not exist.
Ensure that the deployment size is the same as specified by the Memcached CR spec.
Update the Memcached CR status with the names of the memcached pods.

For a detailed explanation on how to configure the controller to perform the above mentioned actions, see Implementing the controller in the Operator SDK tutorial for standard Go-based Operators.

Differences in main.go

For standard Go-based Operators and the Hybrid Helm Operator, the main.go file handles the scaffolding the initialization and running of the Manager program for the Go API. For the Hybrid Helm Operator, however, the main.go file also exposes the logic for loading the watches.yaml file and configuring the Helm reconciler.

Example main.go file

...
	for _, w := range ws {
		// Register controller with the factory
		reconcilePeriod := defaultReconcilePeriod
		if w.ReconcilePeriod != nil {
			reconcilePeriod = w.ReconcilePeriod.Duration
		}

		maxConcurrentReconciles := defaultMaxConcurrentReconciles
		if w.MaxConcurrentReconciles != nil {
			maxConcurrentReconciles = *w.MaxConcurrentReconciles
		}

		r, err := reconciler.New(
			reconciler.WithChart(*w.Chart),
			reconciler.WithGroupVersionKind(w.GroupVersionKind),
			reconciler.WithOverrideValues(w.OverrideValues),
			reconciler.SkipDependentWatches(w.WatchDependentResources != nil && !*w.WatchDependentResources),
			reconciler.WithMaxConcurrentReconciles(maxConcurrentReconciles),
			reconciler.WithReconcilePeriod(reconcilePeriod),
			reconciler.WithInstallAnnotations(annotation.DefaultInstallAnnotations...),
			reconciler.WithUpgradeAnnotations(annotation.DefaultUpgradeAnnotations...),
			reconciler.WithUninstallAnnotations(annotation.DefaultUninstallAnnotations...),
		)
...

The manager is initialized with both Helm and Go reconcilers:

Example Helm and Go reconcilers

...
// Setup manager with Go API
   if err = (&controllers.MemcachedBackupReconciler{
		Client: mgr.GetClient(),
		Scheme: mgr.GetScheme(),
	}).SetupWithManager(mgr); err != nil {
		setupLog.Error(err, "unable to create controller", "controller", "MemcachedBackup")
		os.Exit(1)
	}

   ...
// Setup manager with Helm API
	for _, w := range ws {

      ...
		if err := r.SetupWithManager(mgr); err != nil {
			setupLog.Error(err, "unable to create controller", "controller", "Helm")
			os.Exit(1)
		}
		setupLog.Info("configured watch", "gvk", w.GroupVersionKind, "chartPath", w.ChartPath, "maxConcurrentReconciles", maxConcurrentReconciles, "reconcilePeriod", reconcilePeriod)
	}

// Start the manager
   if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
		setupLog.Error(err, "problem running manager")
		os.Exit(1)
	}

Permissions and RBAC manifests

The controller requires certain role-based access control (RBAC) permissions to interact with the resources it manages. For the Go API, these are specified with RBAC markers, as shown in the Operator SDK tutorial for standard Go-based Operators.

For the Helm API, the permissions are scaffolded by default in roles.yaml. Currently, however, due to a known issue when the Go API is scaffolded, the permissions for the Helm API are overwritten. As a result of this issue, ensure that the permissions defined in roles.yaml match your requirements.

This known issue is being tracked in https://github.com/operator-framework/helm-operator-plugins/issues/142.

The following is an example role.yaml for a Memcached Operator:

Example Helm and Go reconcilers

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: manager-role
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
- apiGroups:
  - apps
  resources:
  - deployments
  - daemonsets
  - replicasets
  - statefulsets
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - cache.my.domain
  resources:
  - memcachedbackups
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - cache.my.domain
  resources:
  - memcachedbackups/finalizers
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - pods
  - services
  - services/finalizers
  - endpoints
  - persistentvolumeclaims
  - events
  - configmaps
  - secrets
  - serviceaccounts
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - cache.my.domain
  resources:
  - memcachedbackups/status
  verbs:
  - get
  - patch
  - update
- apiGroups:
  - policy
  resources:
  - events
  - poddisruptionbudgets
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - cache.my.domain
  resources:
  - memcacheds
  - memcacheds/status
  - memcacheds/finalizers
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch

Additional resources

RBAC markers for Go-based Operators

Running locally outside the cluster

You can run your Operator project as a Go program outside of the cluster. This is useful for development purposes to speed up deployment and testing.

Procedure

Run the following command to install the custom resource definitions (CRDs) in the cluster configured in your ~/.kube/config file and run the Operator locally:
```
$ make install run
```

Running as a deployment on the cluster

You can run your Operator project as a deployment on your cluster.

Procedure

Run the following make commands to build and push the Operator image. Modify the IMG argument in the following steps to reference a repository that you have access to. You can obtain an account for storing containers at repository sites such as Quay.io.

Build the image:

$ make docker-build IMG=<registry>/<user>/<image_name>:<tag>

The Dockerfile generated by the SDK for the Operator explicitly references GOARCH=amd64 for go build. This can be amended to GOARCH=$TARGETARCH for non-AMD64 architectures. Docker will automatically set the environment variable to the value specified by –platform. With Buildah, the –build-arg will need to be used for the purpose. For more information, see Multiple Architectures.

Push the image to a repository:

$ make docker-push IMG=<registry>/<user>/<image_name>:<tag>

The name and tag of the image, for example IMG=<registry>/<user>/<image_name>:<tag>, in both the commands can also be set in your Makefile. Modify the IMG ?= controller:latest value to set your default image name.

Run the following command to deploy the Operator:
```
$ make deploy IMG=<registry>/<user>/<image_name>:<tag>
```
By default, this command creates a namespace with the name of your Operator project in the form <project_name>-system and is used for the deployment. This command also installs the RBAC manifests from config/rbac.

Run the following command to verify that the Operator is running:

$ oc get deployment -n <project_name>-system

Example output

NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
<project_name>-controller-manager       1/1     1            1           8m

Creating custom resources

After your Operator is installed, you can test it by creating custom resources (CRs) that are now provided on the cluster by the Operator.

Procedure

Change to the namespace where your Operator is installed:
```
$ oc project <project_name>-system
```

Update the sample Memcached CR manifest at the config/samples/cache_v1_memcached.yaml file by updating the replicaCount field to 3:

Example config/samples/cache_v1_memcached.yaml file

apiVersion: cache.my.domain/v1
kind: Memcached
metadata:
  name: memcached-sample
spec:
  # Default values copied from <project_dir>/helm-charts/memcached/values.yaml
  affinity: {}
  autoscaling:
    enabled: false
    maxReplicas: 100
    minReplicas: 1
    targetCPUUtilizationPercentage: 80
  fullnameOverride: ""
  image:
    pullPolicy: IfNotPresent
    repository: nginx
    tag: ""
  imagePullSecrets: []
  ingress:
    annotations: {}
    className: ""
    enabled: false
    hosts:
    - host: chart-example.local
      paths:
      - path: /
        pathType: ImplementationSpecific
    tls: []
  nameOverride: ""
  nodeSelector: {}
  podAnnotations: {}
  podSecurityContext: {}
  replicaCount: 3
  resources: {}
  securityContext: {}
  service:
    port: 80
    type: ClusterIP
  serviceAccount:
    annotations: {}
    create: true
    name: ""
  tolerations: []

Create the Memcached CR:

$ oc apply -f config/samples/cache_v1_memcached.yaml

Ensure that the Memcached Operator creates the deployment for the sample CR with the correct size:

$ oc get pods

Example output

NAME                                  READY     STATUS    RESTARTS   AGE
memcached-sample-6fd7c98d8-7dqdr      1/1       Running   0          18m
memcached-sample-6fd7c98d8-g5k7v      1/1       Running   0          18m
memcached-sample-6fd7c98d8-m7vn7      1/1       Running   0          18m

Update the sample MemcachedBackup CR manifest at the config/samples/cache_v1_memcachedbackup.yaml file by updating the size to 2:
Example config/samples/cache_v1_memcachedbackup.yaml file
apiVersion: cache.my.domain/v1 kind: MemcachedBackup metadata: name: memcachedbackup-sample spec: size: 2

Create the MemcachedBackup CR:

$ oc apply -f config/samples/cache_v1_memcachedbackup.yaml

Ensure that the count of memcachedbackup pods is the same as specified in the CR:

$ oc get pods

Example output

NAME                                        READY     STATUS    RESTARTS   AGE
memcachedbackup-sample-8649699989-4bbzg     1/1       Running   0          22m
memcachedbackup-sample-8649699989-mq6mx     1/1       Running   0          22m

You can update the spec in each of the above CRs, and then apply them again. The controller reconciles again and ensures that the size of the pods is as specified in the spec of the respective CRs.
Clean up the resources that have been created as part of this tutorial:
1. Delete the Memcached resource:
  $ oc delete -f config/samples/cache_v1_memcached.yaml
2. Delete the MemcachedBackup resource:
  $ oc delete -f config/samples/cache_v1_memcachedbackup.yaml
3. If you used the make deploy command to test the Operator, run the following command:
  $ make undeploy

Project layout

The Hybrid Helm Operator scaffolding is customized to be compatible with both Helm and Go APIs.

File/folders Purpose

File/folders	Purpose
`Dockerfile`	Instructions used by a container engine to build your Operator image with the `make docker-build` command.
`Makefile`	Build file with helper targets to help you work with your project.
`PROJECT`	YAML file containing metadata information for the Operator. Represents the project’s configuration and is used to track useful information for the CLI and plugins.
`bin/`	Contains useful binaries such as the `manager` which is used to run your project locally and the `kustomize` utility used for the project configuration.
`config/`	Contains configuration files, including all Kustomize manifests, to launch your Operator project on a cluster. Plugins might use it to provide functionality. For example, for the Operator SDK to help create your Operator bundle, the CLI looks up the CRDs and CRs which are scaffolded in this directory. `config/crd/` Contains custom resource definitions (CRDs). `config/default/` Contains a Kustomize base for launching the controller in a standard configuration. `config/manager/` Contains the manifests to launch your Operator project as pods on the cluster. `config/manifests/` Contains the base to generate your OLM manifests in the `bundle/` directory. `config/prometheus/` Contains the manifests required to enable project to serve metrics to Prometheus such as the `ServiceMonitor` resource. `config/scorecard/` Contains the manifests required to allow you test your project with the scorecard tool. `config/rbac/` Contains the RBAC permissions required to run your project. `config/samples/` Contains samples for custom resources.
`api/`	Contains the Go API definition.
`controllers/`	Contains the controllers for the Go API.
`hack/`	Contains utility files, such as the file used to scaffold the license header for your project files.
`main.go`	Main program of the Operator. Instantiates a new manager that registers all custom resource definitions (CRDs) in the `apis/` directory and starts all controllers in the `controllers/` directory.
`helm-charts/`	Contains the Helm charts which can be specified using the `create api` command with the Helm plugin.
`watches.yaml`	Contains group/version/kind (GVK) and Helm chart location. Used to configure the Helm watches.

Dockerfile

Instructions used by a container engine to build your Operator image with the make docker-build command.

Makefile

Build file with helper targets to help you work with your project.

PROJECT

YAML file containing metadata information for the Operator. Represents the project’s configuration and is used to track useful information for the CLI and plugins.

bin/

Contains useful binaries such as the manager which is used to run your project locally and the kustomize utility used for the project configuration.

config/

Contains configuration files, including all Kustomize manifests, to launch your Operator project on a cluster. Plugins might use it to provide functionality. For example, for the Operator SDK to help create your Operator bundle, the CLI looks up the CRDs and CRs which are scaffolded in this directory.

config/crd/: Contains custom resource definitions (CRDs).
config/default/: Contains a Kustomize base for launching the controller in a standard configuration.
config/manager/: Contains the manifests to launch your Operator project as pods on the cluster.
config/manifests/: Contains the base to generate your OLM manifests in the bundle/ directory.
config/prometheus/: Contains the manifests required to enable project to serve metrics to Prometheus such as the ServiceMonitor resource.
config/scorecard/: Contains the manifests required to allow you test your project with the scorecard tool.
config/rbac/: Contains the RBAC permissions required to run your project.
config/samples/: Contains samples for custom resources.

api/

Contains the Go API definition.

controllers/

Contains the controllers for the Go API.

hack/

Contains utility files, such as the file used to scaffold the license header for your project files.

main.go

Main program of the Operator. Instantiates a new manager that registers all custom resource definitions (CRDs) in the apis/ directory and starts all controllers in the controllers/ directory.

helm-charts/

Contains the Helm charts which can be specified using the create api command with the Helm plugin.

watches.yaml

Contains group/version/kind (GVK) and Helm chart location. Used to configure the Helm watches.

Operator SDK tutorial for Hybrid Helm Operators

Prerequisites

Creating a project

Creating a Helm API

Operator logic for the Helm API

Custom Helm reconciler configurations using provided library APIs

Creating a Go API

Defining the API

Controller implementation

Differences in main.go

Permissions and RBAC manifests

Running locally outside the cluster

Running as a deployment on the cluster

Creating custom resources

Project layout