Running a custom scheduler - Controlling pod placement onto nodes (scheduling) | Nodes

Deploying a custom scheduler
Deploying pods using a custom scheduler
Additional resources

You can run multiple custom schedulers alongside the default scheduler and configure which scheduler to use for each pod.

It is supported to use a custom scheduler with OKD, but Red Hat does not directly support the functionality of the custom scheduler.

For information on how to configure the default scheduler, see Controlling pod placement using the scheduler.

To schedule a given pod using a specific scheduler, specify the name of the scheduler in that Pod specification.

Deploying a custom scheduler

To include a custom scheduler in your cluster, include the image for a custom scheduler in a deployment.

Prerequisites

You have access to the cluster as a user with the cluster-admin role.

You have a scheduler binary.

Information on how to create a scheduler binary is outside the scope of this document. For an example, see Configure Multiple Schedulers in the Kubernetes documentation. Note that the actual functionality of your custom scheduler is not supported by Red Hat.

You have created an image containing the scheduler binary and pushed it to a registry.

Procedure

Create a file that contains a config map that holds the scheduler configuration file:

Example scheduler-config-map.yaml

apiVersion: v1
kind: configmap
metadata:
  name: scheduler-config
  namespace: kube-system (1)
data:
  scheduler-config.yaml: | (2)
    apiVersion: kubescheduler.config.k8s.io/v1beta2
    kind: KubeSchedulerConfiguration
    profiles:
      - schedulerName: custom-scheduler (3)
    leaderElection:
      leaderElect: false

1	This procedure uses the `kube-system` namespace, but you can use the namespace of your choosing.
2	When you define your `Deployment` resource later in this procedure, you pass this file in to the scheduler command by using the `--config` argument.
3	Define a scheduler profile for your custom scheduler. This scheduler name is used when defining the `schedulerName` in the `Pod` configuration.

Create the config map:

$ oc create -f scheduler-config-map.yaml

Create a file that contains the deployment resources for the custom scheduler:

Example custom-scheduler.yaml file

apiVersion: v1
kind: ServiceAccount
metadata:
  name: custom-scheduler
  namespace: kube-system (1)
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: custom-scheduler-as-kube-scheduler
subjects:
- kind: ServiceAccount
  name: custom-scheduler
  namespace: kube-system (1)
roleRef:
  kind: ClusterRole
  name: system:kube-scheduler
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: custom-scheduler-as-volume-scheduler
subjects:
- kind: ServiceAccount
  name: custom-scheduler
  namespace: kube-system (1)
roleRef:
  kind: ClusterRole
  name: system:volume-scheduler
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    component: scheduler
    tier: control-plane
  name: custom-scheduler
  namespace: kube-system (1)
spec:
  selector:
    matchLabels:
      component: scheduler
      tier: control-plane
  replicas: 1
  template:
    metadata:
      labels:
        component: scheduler
        tier: control-plane
        version: second
    spec:
      serviceAccountName: custom-scheduler
      containers:
      - command:
        - /usr/local/bin/kube-scheduler
        - --config=/etc/config/scheduler-config.yaml (2)
        image: "<namespace>/<image_name>:<tag>" (3)
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10259
            scheme: HTTPS
          initialDelaySeconds: 15
        name: kube-second-scheduler
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10259
            scheme: HTTPS
        resources:
          requests:
            cpu: '0.1'
        securityContext:
          privileged: false
        volumeMounts:
        - name: config-volume
          mountPath: /etc/config
      hostNetwork: false
      hostPID: false
      volumes:
        - name: config-volume
          configmap:
            name: scheduler-config

1	This procedure uses the `kube-system` namespace, but you can use the namespace of your choosing.
2	The command for your custom scheduler might require different arguments.
3	Specify the container image that you created for the custom scheduler.

Create the deployment resources in the cluster:
```
$ oc create -f custom-scheduler.yaml
```

Verification

Verify that the scheduler pod is running:

$ oc get pods -n kube-system

The custom scheduler pod is listed as Running:

NAME                                                       READY   STATUS    RESTARTS   AGE
custom-scheduler-6cd7c4b8bc-854zb                          1/1     Running   0          2m

Deploying pods using a custom scheduler

After the custom scheduler is deployed in your cluster, you can configure pods to use that scheduler instead of the default scheduler.

Each scheduler has a separate view of resources in a cluster. For that reason, each scheduler should operate over its own set of nodes.

If two or more schedulers operate on the same node, they might intervene with each other and schedule more pods on the same node than there are available resources for. Pods might get rejected due to insufficient resources in this case.

Prerequisites

You have access to the cluster as a user with the cluster-admin role.
The custom scheduler has been deployed in the cluster.

Procedure

If your cluster uses role-based access control (RBAC), add the custom scheduler name to the system:kube-scheduler cluster role.

Edit the system:kube-scheduler cluster role:

$ oc edit clusterrole system:kube-scheduler

Add the name of the custom scheduler to the resourceNames lists for the leases and endpoints resources:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: "2021-07-07T10:19:14Z"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-scheduler
  resourceVersion: "125"
  uid: 53896c70-b332-420a-b2a4-f72c822313f2
rules:
...
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - create
- apiGroups:
  - coordination.k8s.io
  resourceNames:
  - kube-scheduler
  - custom-scheduler (1)
  resources:
  - leases
  verbs:
  - get
  - update
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
  - create
- apiGroups:
  - ""
  resourceNames:
  - kube-scheduler
  - custom-scheduler (1)
  resources:
  - endpoints
  verbs:
  - get
  - update
...

1	This example uses `custom-scheduler` as the custom scheduler name.

Create a Pod configuration and specify the name of the custom scheduler in the schedulerName parameter:
Example custom-scheduler-example.yaml file
```
apiVersion: v1
kind: Pod
metadata:
  name: custom-scheduler-example
  labels:
    name: custom-scheduler-example
spec:
  schedulerName: custom-scheduler (1)
  containers:
  - name: pod-with-second-annotation-container
    image: docker.io/ocpqe/hello-pod
```
1 The name of the custom scheduler to use, which is custom-scheduler in this example. When no scheduler name is supplied, the pod is automatically scheduled using the default scheduler.

Create the pod:

$ oc create -f custom-scheduler-example.yaml

Verification

Enter the following command to check that the pod was created:

$ oc get pod custom-scheduler-example

The custom-scheduler-example pod is listed in the output:

NAME                       READY     STATUS    RESTARTS   AGE
custom-scheduler-example   1/1       Running   0          4m

Enter the following command to check that the custom scheduler has scheduled the pod:

$ oc describe pod custom-scheduler-example

The scheduler, custom-scheduler, is listed as shown in the following truncated output:

Events:
  Type    Reason          Age        From                                               Message
  ----    ------          ----       ----                                               -------
  Normal  Scheduled       <unknown>  custom-scheduler                                   Successfully assigned default/custom-scheduler-example to <node_name>

Additional resources

Learning container best practices