$ mkdir -p $HOME/projects/nginx-operator
Operator developers can take advantage of Helm support in the Operator SDK to build an example Helm-based Operator for Nginx and manage its lifecycle. This tutorial walks through the following process:
Create a Nginx deployment
Ensure that the deployment size is the same as specified by the Nginx
custom resource (CR) spec
Update the Nginx
CR status using the status writer with the names of the nginx
pods
This process is accomplished using two centerpieces of the Operator Framework:
The operator-sdk
cli tool and controller-runtime
library API
Installation, upgrade, and role-based access control (RBAC) of Operators on a cluster
This tutorial goes into greater detail than Getting started with Operator SDK for Helm-based Operators. |
Logged into an OpenShift Container Platform 4.10 cluster with oc
with an account that has cluster-admin
permissions
To allow the cluster to pull the image, the repository where you push your image must be set as public, or you must configure an image pull secret
Use the Operator SDK cli to create a project called nginx-operator
.
Create a directory for the project:
$ mkdir -p $HOME/projects/nginx-operator
Change to the directory:
$ cd $HOME/projects/nginx-operator
Run the operator-sdk init
command
with the helm
plugin
to initialize the project:
$ operator-sdk init \
--plugins=helm \
--domain=example.com \
--group=demo \
--version=v1 \
--kind=Nginx
By default, the |
The init
command creates the nginx-operator
project specifically for watching a resource with API version example.com/v1
and kind Nginx
.
For Helm-based projects, the init
command generates the RBAC rules in the config/rbac/role.yaml
file based on the resources that would be deployed by the default manifest for the chart. Verify that the rules generated in this file meet the permission requirements of the Operator.
Instead of creating your project with a boilerplate Helm chart, you can alternatively use an existing chart, either from your local file system or a remote chart repository, by using the following flags:
--helm-chart
--helm-chart-repo
--helm-chart-version
If the --helm-chart
flag is specified, the --group
, --version
, and --kind
flags become optional. If left unset, the following default values are used:
Flag | Value |
---|---|
|
|
|
|
|
|
|
Deduced from the specified chart |
If the --helm-chart
flag specifies a local chart archive, for example example-chart-1.2.0.tgz
, or directory, the chart is validated and unpacked or copied into the project. Otherwise, the Operator SDK attempts to fetch the chart from a remote repository.
If a custom repository URL is not specified by the --helm-chart-repo
flag, the following chart reference formats are supported:
Format | Description |
---|---|
|
Fetch the Helm chart named |
|
Fetch the Helm chart archive at the specified URL. |
If a custom repository URL is specified by --helm-chart-repo
, the following chart reference format is supported:
Format | Description |
---|---|
|
Fetch the Helm chart named |
If the --helm-chart-version
flag is unset, the Operator SDK fetches the latest available version of the Helm chart. Otherwise, it fetches the specified version. The optional --helm-chart-version
flag is not used when the chart specified with the --helm-chart
flag refers to a specific version, for example when it is a local path or a URL.
For more details and examples, run:
$ operator-sdk init --plugins helm --help
Among the files generated by the operator-sdk init
command is a Kubebuilder PROJECT
file. Subsequent operator-sdk
commands, as well as help
output, that are run from the project root read this file and are aware that the project type is Helm. For example:
domain: example.com
layout:
- helm.sdk.operatorframework.io/v1
plugins:
manifests.sdk.operatorframework.io/v2: {}
scorecard.sdk.operatorframework.io/v2: {}
sdk.x-openshift.io/v1: {}
projectName: nginx-operator
resources:
- api:
crdVersion: v1
namespaced: true
domain: example.com
group: demo
kind: Nginx
version: v1
version: "3"
For this example, the nginx-operator
project executes the following reconciliation logic for each Nginx
custom resource (CR):
Create an Nginx deployment if it does not exist.
Create an Nginx service if it does not exist.
Create an Nginx ingress if it is enabled and does not exist.
Ensure that the deployment, service, and optional ingress match the desired configuration as specified by the Nginx
CR, for example the replica count, image, and service type.
By default, the nginx-operator
project watches Nginx
resource events as shown in the watches.yaml
file and executes Helm releases using the specified chart:
# Use the 'create api' subcommand to add watches to this file.
- group: demo
version: v1
kind: Nginx
chart: helm-charts/nginx
# +kubebuilder:scaffold:watch
When a Helm Operator project is created, the Operator SDK creates a sample Helm chart that contains a set of templates for a simple Nginx release.
For this example, templates are available for deployment, service, and ingress resources, along with a NOTES.txt
template, which Helm chart developers use to convey helpful information about a release.
If you are not already familiar with Helm charts, review the Helm developer documentation.
Helm uses a concept called values to provide customizations to the defaults of a Helm chart, which are defined in the values.yaml
file.
You can override these defaults by setting the desired values in the custom resource (CR) spec. You can use the number of replicas as an example.
The helm-charts/nginx/values.yaml
file has a value called replicaCount
set to 1
by default. To have two Nginx instances in your deployment, your CR spec must contain replicaCount: 2
.
Edit the config/samples/demo_v1_nginx.yaml
file to set replicaCount: 2
:
apiVersion: demo.example.com/v1
kind: Nginx
metadata:
name: nginx-sample
...
spec:
...
replicaCount: 2
Similarly, the default service port is set to 80
. To use 8080
, edit the config/samples/demo_v1_nginx.yaml
file to set spec.port: 8080
,which adds the service port override:
apiVersion: demo.example.com/v1
kind: Nginx
metadata:
name: nginx-sample
spec:
replicaCount: 2
service:
port: 8080
The Helm Operator applies the entire spec as if it was the contents of a values file, just like the helm install -f ./overrides.yaml
command.
Operator authors can develop Operators that support network proxies. Cluster administrators configure proxy support for the environment variables that are handled by Operator Lifecycle Manager (OLM). To support proxied clusters, your Operator must inspect the environment for the following standard proxy variables and pass the values to Operands:
HTTP_PROXY
HTTPS_PROXY
NO_PROXY
This tutorial uses |
A cluster with cluster-wide egress proxy enabled.
Edit the watches.yaml
file to include overrides based on an environment variable by adding the overrideValues
field:
...
- group: demo.example.com
version: v1alpha1
kind: Nginx
chart: helm-charts/nginx
overrideValues:
proxy.http: $HTTP_PROXY
...
Add the proxy.http
value in the helm-charts/nginx/values.yaml
file:
...
proxy:
http: ""
https: ""
no_proxy: ""
To make sure the chart template supports using the variables, edit the chart template in the helm-charts/nginx/templates/deployment.yaml
file to contain the following:
containers:
- name: {{ .Chart.Name }}
securityContext:
- toYaml {{ .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
env:
- name: http_proxy
value: "{{ .Values.proxy.http }}"
Set the environment variable on the Operator deployment by adding the following to the config/manager/manager.yaml
file:
containers:
- args:
- --leader-elect
- --leader-election-id=ansible-proxy-demo
image: controller:latest
name: manager
env:
- name: "HTTP_PROXY"
value: "http_proxy_test"
There are three ways you can use the Operator SDK cli to build and run your Operator:
Run locally outside the cluster as a Go program.
Run as a deployment on the cluster.
Bundle your Operator and use Operator Lifecycle Manager (OLM) to deploy on the cluster.
You can run your Operator project as a Go program outside of the cluster. This is useful for development purposes to speed up deployment and testing.
Run the following command to install the custom resource definitions (CRDs) in the cluster configured in your ~/.kube/config
file and run the Operator locally:
$ make install run
...
{"level":"info","ts":1612652419.9289865,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1612652419.9296563,"logger":"helm.controller","msg":"Watching resource","apiVersion":"demo.example.com/v1","kind":"Nginx","namespace":"","reconcilePeriod":"1m0s"}
{"level":"info","ts":1612652419.929983,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1612652419.930015,"logger":"controller-runtime.manager.controller.nginx-controller","msg":"Starting EventSource","source":"kind source: demo.example.com/v1, Kind=Nginx"}
{"level":"info","ts":1612652420.2307851,"logger":"controller-runtime.manager.controller.nginx-controller","msg":"Starting Controller"}
{"level":"info","ts":1612652420.2309358,"logger":"controller-runtime.manager.controller.nginx-controller","msg":"Starting workers","worker count":8}
You can run your Operator project as a deployment on your cluster.
Run the following make
commands to build and push the Operator image. Modify the IMG
argument in the following steps to reference a repository that you have access to. You can obtain an account for storing containers at repository sites such as Quay.io.
Build the image:
$ make docker-build IMG=<registry>/<user>/<image_name>:<tag>
The Dockerfile generated by the SDK for the Operator explicitly references |
Push the image to a repository:
$ make docker-push IMG=<registry>/<user>/<image_name>:<tag>
The name and tag of the image, for example |
Run the following command to deploy the Operator:
$ make deploy IMG=<registry>/<user>/<image_name>:<tag>
By default, this command creates a namespace with the name of your Operator project in the form <project_name>-system
and is used for the deployment. This command also installs the RBAC manifests from config/rbac
.
Verify that the Operator is running:
$ oc get deployment -n <project_name>-system
NAME READY UP-TO-DATE AVAILABLE AGE
<project_name>-controller-manager 1/1 1 1 8m
The Operator bundle format is the default packaging method for Operator SDK and Operator Lifecycle Manager (OLM). You can get your Operator ready for use on OLM by using the Operator SDK to build and push your Operator project as a bundle image.
Operator SDK cli installed on a development workstation
OpenShift cli (oc
) v4.10+ installed
Operator project initialized by using the Operator SDK
Run the following make
commands in your Operator project directory to build and push your Operator image. Modify the IMG
argument in the following steps to reference a repository that you have access to. You can obtain an account for storing containers at repository sites such as Quay.io.
Build the image:
$ make docker-build IMG=<registry>/<user>/<operator_image_name>:<tag>
The Dockerfile generated by the SDK for the Operator explicitly references |
Push the image to a repository:
$ make docker-push IMG=<registry>/<user>/<operator_image_name>:<tag>
Create your Operator bundle manifest by running the make bundle
command, which invokes several commands, including the Operator SDK generate bundle
and bundle validate
subcommands:
$ make bundle IMG=<registry>/<user>/<operator_image_name>:<tag>
Bundle manifests for an Operator describe how to display, create, and manage an application. The make bundle
command creates the following files and directories in your Operator project:
A bundle manifests directory named bundle/manifests
that contains a ClusterServiceVersion
object
A bundle metadata directory named bundle/metadata
All custom resource definitions (CRDs) in a config/crd
directory
A Dockerfile bundle.Dockerfile
These files are then automatically validated by using operator-sdk bundle validate
to ensure the on-disk bundle representation is correct.
Build and push your bundle image by running the following commands. OLM consumes Operator bundles using an index image, which reference one or more bundle images.
Build the bundle image. Set BUNDLE_IMG
with the details for the registry, user namespace, and image tag where you intend to push the image:
$ make bundle-build BUNDLE_IMG=<registry>/<user>/<bundle_image_name>:<tag>
Push the bundle image:
$ docker push <registry>/<user>/<bundle_image_name>:<tag>
Operator Lifecycle Manager (OLM) helps you to install, update, and manage the lifecycle of Operators and their associated services on a Kubernetes cluster. OLM is installed by default on OpenShift Container Platform and runs as a Kubernetes extension so that you can use the web console and the OpenShift cli (oc
) for all Operator lifecycle management functions without any additional tools.
The Operator bundle format is the default packaging method for Operator SDK and OLM. You can use the Operator SDK to quickly run a bundle image on OLM to ensure that it runs properly.
Operator SDK cli installed on a development workstation
Operator bundle image built and pushed to a registry
OLM installed on a Kubernetes-based cluster (v1.16.0 or later if you use apiextensions.k8s.io/v1
CRDs, for example OpenShift Container Platform 4.10)
Logged in to the cluster with oc
using an account with cluster-admin
permissions
Enter the following command to run the Operator on the cluster:
$ operator-sdk run bundle \
[-n <namespace>] \(1)
<registry>/<user>/<bundle_image_name>:<tag>
1 | By default, the command installs the Operator in the currently active project in your ~/.kube/config file. You can add the -n flag to set a different namespace scope for the installation. |
This command performs the following actions:
Create an index image referencing your bundle image. The index image is opaque and ephemeral, but accurately reflects how a bundle would be added to a catalog in production.
Create a catalog source that points to your new index image, which enables OperatorHub to discover your Operator.
Deploy your Operator to your cluster by creating an OperatorGroup
, Subscription
, InstallPlan
, and all other required objects, including RBAC.
After your Operator is installed, you can test it by creating a custom resource (CR) that is now provided on the cluster by the Operator.
Example Nginx Operator, which provides the Nginx
CR, installed on a cluster
Change to the namespace where your Operator is installed. For example, if you deployed the Operator using the make deploy
command:
$ oc project nginx-operator-system
Edit the sample Nginx
CR manifest at config/samples/demo_v1_nginx.yaml
to contain the following specification:
apiVersion: demo.example.com/v1
kind: Nginx
metadata:
name: nginx-sample
...
spec:
...
replicaCount: 3
The Nginx service account requires privileged access to run in OpenShift Container Platform. Add the following security context constraint (SCC) to the service account for the nginx-sample
pod:
$ oc adm policy add-scc-to-user \
anyuid system:serviceaccount:nginx-operator-system:nginx-sample
Create the CR:
$ oc apply -f config/samples/demo_v1_nginx.yaml
Ensure that the Nginx
Operator creates the deployment for the sample CR with the correct size:
$ oc get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-operator-controller-manager 1/1 1 1 8m
nginx-sample 3/3 3 3 1m
Check the pods and CR status to confirm the status is updated with the Nginx pod names.
Check the pods:
$ oc get pods
NAME READY STATUS RESTARTS AGE
nginx-sample-6fd7c98d8-7dqdr 1/1 Running 0 1m
nginx-sample-6fd7c98d8-g5k7v 1/1 Running 0 1m
nginx-sample-6fd7c98d8-m7vn7 1/1 Running 0 1m
Check the CR status:
$ oc get nginx/nginx-sample -o yaml
apiVersion: demo.example.com/v1
kind: Nginx
metadata:
...
name: nginx-sample
...
spec:
replicaCount: 3
status:
nodes:
- nginx-sample-6fd7c98d8-7dqdr
- nginx-sample-6fd7c98d8-g5k7v
- nginx-sample-6fd7c98d8-m7vn7
Update the deployment size.
Update config/samples/demo_v1_nginx.yaml
file to change the spec.size
field in the Nginx
CR from 3
to 5
:
$ oc patch nginx nginx-sample \
-p '{"spec":{"replicaCount": 5}}' \
--type=merge
Confirm that the Operator changes the deployment size:
$ oc get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-operator-controller-manager 1/1 1 1 10m
nginx-sample 5/5 5 5 3m
Delete the CR by running the following command:
$ oc delete -f config/samples/demo_v1_nginx.yaml
Clean up the resources that have been created as part of this tutorial.
If you used the make deploy
command to test the Operator, run the following command:
$ make undeploy
If you used the operator-sdk run bundle
command to test the Operator, run the following command:
$ operator-sdk cleanup <project_name>
See Project layout for Helm-based Operators to learn about the directory structures created by the Operator SDK.
If a cluster-wide egress proxy is configured, cluster administrators can override the proxy settings or inject a custom CA certificate for specific Operators running on Operator Lifecycle Manager (OLM).