$ mkdir -p $HOME/github.com/example/memcached-operator
The standard Helm-based Operator support in the Operator SDK has limited functionality compared to the Go-based and Ansible-based Operator support that has reached the Auto Pilot capability (level V) in the Operator maturity model.
The Hybrid Helm Operator enhances the existing Helm-based support’s abilities through Go APIs. With this hybrid approach of Helm and Go, the Operator SDK enables Operator authors to use the following process:
Generate a default structure for, or scaffold, a Go API in the same project as Helm.
Configure the Helm reconciler in the main.go
file of the project, through the libraries provided by the Hybrid Helm Operator.
The Hybrid Helm Operator is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope. |
This tutorial walks through the following process using the Hybrid Helm Operator:
Create a Memcached
deployment through a Helm chart if it does not exist
Ensure that the deployment size is the same as specified by Memcached
custom resource (CR) spec
Create a MemcachedBackup
deployment by using the Go API
Operator SDK cli installed
OpenShift cli (oc
) v4.11+ installed
Logged into an OpenShift Container Platform 4.11 cluster with oc
with an account that has cluster-admin
permissions
To allow the cluster to pull the image, the repository where you push your image must be set as public, or you must configure an image pull secret
Use the Operator SDK cli to create a project called memcached-operator
.
Create a directory for the project:
$ mkdir -p $HOME/github.com/example/memcached-operator
Change to the directory:
$ cd $HOME/github.com/example/memcached-operator
Run the operator-sdk init
command to initialize the project. Use a domain of example.com
so that all API groups are <group>.example.com
:
$ operator-sdk init \
--plugins=hybrid.helm.sdk.operatorframework.io \
--project-version="3" \
--domain example.com \
--repo=github.com/example/memcached-operator
The init
command generates the RBAC rules in the config/rbac/role.yaml
file based on the resources that would be deployed by the chart’s default manifests. Verify that the rules generated in the config/rbac/role.yaml
file meet your Operator’s permission requirements.
This procedure creates a project structure that is compatible with both Helm and Go APIs. To learn more about the project directory structure, see Project layout.
Use the Operator SDK cli to create a Helm API.
Run the following command to create a Helm API with group cache
, version v1
, and kind Memcached
:
$ operator-sdk create api \
--plugins helm.sdk.operatorframework.io/v1 \
--group cache \
--version v1 \
--kind Memcached
This procedure also configures your Operator project to watch the For more details and examples for creating Helm API based on existing or new charts, run the following command:
|
By default, your scaffolded Operator project watches Memcached
resource events as shown in the watches.yaml
file and executes Helm releases using the specified chart.
watches.yaml
file# Use the 'create api' subcommand to add watches to this file.
- group: cache.my.domain
version: v1
kind: Memcached
chart: helm-charts/memcached
#+kubebuilder:scaffold:watch
For detailed documentation on customizing the Helm Operator logic through the chart, see Understanding the Operator logic.
A disadvantage of existing Helm-based Operators is the inability to configure the Helm reconciler, because it is abstracted from users. For a Helm-based Operator to reach the Seamless Upgrades capability (level II and later) that reuses an already existing Helm chart, a hybrid between the Go and Helm Operator types adds value.
The APIs provided in the helm-operator-plugins
library allow Operator authors to make the following configurations:
Customize value mapping based on cluster state
Execute code in specific events by configuring the reconciler’s event recorder
Customize the reconciler’s logger
Setup Install
, Upgrade
, and Uninstall
annotations to enable Helm’s actions to be configured based on the annotations found in custom resources watched by the reconciler
Configure the reconciler to run with Pre
and Post
hooks
The above configurations to the reconciler can be done in the main.go
file:
main.go
file// Operator's main.go
// With the help of helpers provided in the library, the reconciler can be
// configured here before starting the controller with this reconciler.
reconciler := reconciler.New(
reconciler.WithChart(*chart),
reconciler.WithGroupVersionKind(gvk),
)
if err := reconciler.SetupWithManager(mgr); err != nil {
panic(fmt.Sprintf("unable to create reconciler: %s", err))
}
Use the Operator SDK cli to create a Go API.
Run the following command to create a Go API with group cache
, version v1
, and kind MemcachedBackup
:
$ operator-sdk create api \
--group=cache \
--version v1 \
--kind MemcachedBackup \
--resource \
--controller \
--plugins=go/v3
When prompted, enter y
for creating both resource and controller:
$ Create Resource [y/n]
y
Create Controller [y/n]
y
This procedure generates the MemcachedBackup
resource API at api/v1/memcachedbackup_types.go
and the controller at controllers/memcachedbackup_controller.go
.
Define the API for the MemcachedBackup
custom resource (CR).
Represent this Go API by defining the MemcachedBackup
type, which will have a MemcachedBackupSpec.Size
field to set the quantity of Memcached backup instances (CRs) to be deployed, and a MemcachedBackupStatus.Nodes
field to store a CR’s pod names.
The |
Define the API for the MemcachedBackup
CR by modifying the Go type definitions in the api/v1/memcachedbackup_types.go
file to have the following spec
and status
:
api/v1/memcachedbackup_types.go
file// MemcachedBackupSpec defines the desired state of MemcachedBackup
type MemcachedBackupSpec struct {
// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
// Important: Run "make" to regenerate code after modifying this file
//+kubebuilder:validation:Minimum=0
// Size is the size of the memcached deployment
Size int32 `json:"size"`
}
// MemcachedBackupStatus defines the observed state of MemcachedBackup
type MemcachedBackupStatus struct {
// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
// Important: Run "make" to regenerate code after modifying this file
// Nodes are the names of the memcached pods
Nodes []string `json:"nodes"`
}
Update the generated code for the resource type:
$ make generate
After you modify a |
After the API is defined with spec
and status
fields and CRD validation markers, generate and update the CRD manifests:
$ make manifests
This Makefile target invokes the controller-gen
utility to generate the CRD manifests in the config/crd/bases/cache.my.domain_memcachedbackups.yaml
file.
The controller in this tutorial performs the following actions:
Create a Memcached
deployment if it does not exist.
Ensure that the deployment size is the same as specified by the Memcached
CR spec.
Update the Memcached
CR status with the names of the memcached
pods.
For a detailed explanation on how to configure the controller to perform the above mentioned actions, see Implementing the controller in the Operator SDK tutorial for standard Go-based Operators.
For standard Go-based Operators and the Hybrid Helm Operator, the main.go
file handles the scaffolding the initialization and running of the Manager
program for the Go API. For the Hybrid Helm Operator, however, the main.go
file also exposes the logic for loading the watches.yaml
file and configuring the Helm reconciler.
main.go
file...
for _, w := range ws {
// Register controller with the factory
reconcilePeriod := defaultReconcilePeriod
if w.ReconcilePeriod != nil {
reconcilePeriod = w.ReconcilePeriod.Duration
}
maxConcurrentReconciles := defaultMaxConcurrentReconciles
if w.MaxConcurrentReconciles != nil {
maxConcurrentReconciles = *w.MaxConcurrentReconciles
}
r, err := reconciler.New(
reconciler.WithChart(*w.Chart),
reconciler.WithGroupVersionKind(w.GroupVersionKind),
reconciler.WithOverrideValues(w.OverrideValues),
reconciler.SkipDependentWatches(w.WatchDependentResources != nil && !*w.WatchDependentResources),
reconciler.WithMaxConcurrentReconciles(maxConcurrentReconciles),
reconciler.WithReconcilePeriod(reconcilePeriod),
reconciler.WithInstallAnnotations(annotation.DefaultInstallAnnotations...),
reconciler.WithUpgradeAnnotations(annotation.DefaultUpgradeAnnotations...),
reconciler.WithUninstallAnnotations(annotation.DefaultUninstallAnnotations...),
)
...
The manager is initialized with both Helm
and Go
reconcilers:
Helm
and Go
reconcilers...
// Setup manager with Go API
if err = (&controllers.MemcachedBackupReconciler{
client: mgr.Getclient(),
Scheme: mgr.GetScheme(),
}).SetupWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "MemcachedBackup")
os.Exit(1)
}
...
// Setup manager with Helm API
for _, w := range ws {
...
if err := r.SetupWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "Helm")
os.Exit(1)
}
setupLog.Info("configured watch", "gvk", w.GroupVersionKind, "chartPath", w.ChartPath, "maxConcurrentReconciles", maxConcurrentReconciles, "reconcilePeriod", reconcilePeriod)
}
// Start the manager
if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
setupLog.Error(err, "problem running manager")
os.Exit(1)
}
The controller requires certain role-based access control (RBAC) permissions to interact with the resources it manages. For the Go API, these are specified with RBAC markers, as shown in the Operator SDK tutorial for standard Go-based Operators.
For the Helm API, the permissions are scaffolded by default in roles.yaml
. Currently, however, due to a known issue when the Go API is scaffolded, the permissions for the Helm API are overwritten. As a result of this issue, ensure that the permissions defined in roles.yaml
match your requirements.
This known issue is being tracked in https://github.com/operator-framework/helm-operator-plugins/issues/142. |
The following is an example role.yaml
for a Memcached Operator:
Helm
and Go
reconcilers---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: manager-role
rules:
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- apiGroups:
- apps
resources:
- deployments
- daemonsets
- replicasets
- statefulsets
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- cache.my.domain
resources:
- memcachedbackups
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- cache.my.domain
resources:
- memcachedbackups/finalizers
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- ""
resources:
- pods
- services
- services/finalizers
- endpoints
- persistentvolumeclaims
- events
- configmaps
- secrets
- serviceaccounts
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- cache.my.domain
resources:
- memcachedbackups/status
verbs:
- get
- patch
- update
- apiGroups:
- policy
resources:
- events
- poddisruptionbudgets
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- cache.my.domain
resources:
- memcacheds
- memcacheds/status
- memcacheds/finalizers
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
You can run your Operator project as a Go program outside of the cluster. This is useful for development purposes to speed up deployment and testing.
Run the following command to install the custom resource definitions (CRDs) in the cluster configured in your ~/.kube/config
file and run the Operator locally:
$ make install run
You can run your Operator project as a deployment on your cluster.
Run the following make
commands to build and push the Operator image. Modify the IMG
argument in the following steps to reference a repository that you have access to. You can obtain an account for storing containers at repository sites such as Quay.io.
Build the image:
$ make docker-build IMG=<registry>/<user>/<image_name>:<tag>
The Dockerfile generated by the SDK for the Operator explicitly references |
Push the image to a repository:
$ make docker-push IMG=<registry>/<user>/<image_name>:<tag>
The name and tag of the image, for example |
Run the following command to deploy the Operator:
$ make deploy IMG=<registry>/<user>/<image_name>:<tag>
By default, this command creates a namespace with the name of your Operator project in the form <project_name>-system
and is used for the deployment. This command also installs the RBAC manifests from config/rbac
.
Run the following command to verify that the Operator is running:
$ oc get deployment -n <project_name>-system
NAME READY UP-TO-DATE AVAILABLE AGE
<project_name>-controller-manager 1/1 1 1 8m
After your Operator is installed, you can test it by creating custom resources (CRs) that are now provided on the cluster by the Operator.
Change to the namespace where your Operator is installed:
$ oc project <project_name>-system
Update the sample Memcached
CR manifest at the config/samples/cache_v1_memcached.yaml
file by updating the replicaCount
field to 3
:
config/samples/cache_v1_memcached.yaml
fileapiVersion: cache.my.domain/v1
kind: Memcached
metadata:
name: memcached-sample
spec:
# Default values copied from <project_dir>/helm-charts/memcached/values.yaml
affinity: {}
autoscaling:
enabled: false
maxReplicas: 100
minReplicas: 1
targetCPUUtilizationPercentage: 80
fullnameOverride: ""
image:
pullPolicy: IfNotPresent
repository: nginx
tag: ""
imagePullSecrets: []
ingress:
annotations: {}
className: ""
enabled: false
hosts:
- host: chart-example.local
paths:
- path: /
pathType: ImplementationSpecific
tls: []
nameOverride: ""
nodeSelector: {}
podAnnotations: {}
podSecurityContext: {}
replicaCount: 3
resources: {}
securityContext: {}
service:
port: 80
type: ClusterIP
serviceAccount:
annotations: {}
create: true
name: ""
tolerations: []
Create the Memcached
CR:
$ oc apply -f config/samples/cache_v1_memcached.yaml
Ensure that the Memcached Operator creates the deployment for the sample CR with the correct size:
$ oc get pods
NAME READY STATUS RESTARTS AGE
memcached-sample-6fd7c98d8-7dqdr 1/1 Running 0 18m
memcached-sample-6fd7c98d8-g5k7v 1/1 Running 0 18m
memcached-sample-6fd7c98d8-m7vn7 1/1 Running 0 18m
Update the sample MemcachedBackup
CR manifest at the config/samples/cache_v1_memcachedbackup.yaml
file by updating the size
to 2
:
config/samples/cache_v1_memcachedbackup.yaml
fileapiVersion: cache.my.domain/v1
kind: MemcachedBackup
metadata:
name: memcachedbackup-sample
spec:
size: 2
Create the MemcachedBackup
CR:
$ oc apply -f config/samples/cache_v1_memcachedbackup.yaml
Ensure that the count of memcachedbackup
pods is the same as specified in the CR:
$ oc get pods
NAME READY STATUS RESTARTS AGE
memcachedbackup-sample-8649699989-4bbzg 1/1 Running 0 22m
memcachedbackup-sample-8649699989-mq6mx 1/1 Running 0 22m
You can update the spec
in each of the above CRs, and then apply them again. The controller reconciles again and ensures that the size of the pods is as specified in the spec
of the respective CRs.
Clean up the resources that have been created as part of this tutorial:
Delete the Memcached
resource:
$ oc delete -f config/samples/cache_v1_memcached.yaml
Delete the MemcachedBackup
resource:
$ oc delete -f config/samples/cache_v1_memcachedbackup.yaml
If you used the make deploy
command to test the Operator, run the following command:
$ make undeploy
The Hybrid Helm Operator scaffolding is customized to be compatible with both Helm and Go APIs.
File/folders | Purpose |
---|---|
|
Instructions used by a container engine to build your Operator image with the |
|
Build file with helper targets to help you work with your project. |
|
YAML file containing metadata information for the Operator. Represents the project’s configuration and is used to track useful information for the cli and plugins. |
|
Contains useful binaries such as the |
|
Contains configuration files, including all Kustomize manifests, to launch your Operator project on a cluster. Plugins might use it to provide functionality. For example, for the Operator SDK to help create your Operator bundle, the cli looks up the CRDs and CRs which are scaffolded in this directory.
|
|
Contains the Go API definition. |
|
Contains the controllers for the Go API. |
|
Contains utility files, such as the file used to scaffold the license header for your project files. |
|
Main program of the Operator. Instantiates a new manager that registers all custom resource definitions (CRDs) in the |
|
Contains the Helm charts which can be specified using the |
|
Contains group/version/kind (GVK) and Helm chart location. Used to configure the Helm watches. |