$ oc get -n openshift-dns-operator deployment/dns-operator
In OKD, the DNS Operator deploys and manages a CoreDNS instance to provide a name resolution service to pods inside the cluster, enables DNS-based Kubernetes Service discovery, and resolves internal cluster.local names.
You can check the DNS Operator deployment and cluster operator status. The DNS Operator is deployed during installation with a Deployment object.
The DNS Operator implements the dns API from the operator.openshift.io API
group. The Operator deploys CoreDNS using a daemon set, creates a service for
the daemon set, and configures the kubelet to instruct pods to use the CoreDNS
service IP address for name resolution.
Use the oc get command to view the deployment status:
$ oc get -n openshift-dns-operator deployment/dns-operator
NAME READY UP-TO-DATE AVAILABLE AGE
dns-operator 1/1 1 1 23h
Use the oc get command to view the state of the DNS Operator:
$ oc get clusteroperator/dns
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
dns 4.1.15-0.11 True False False 92m
AVAILABLE, PROGRESSING, and DEGRADED provide information about the status of the Operator. AVAILABLE is True when at least 1 pod from the CoreDNS daemon set reports an Available status condition, and the DNS service has a cluster IP address.
View the default DNS resource and cluster DNS settings to verify the DNS configuration or troubleshoot DNS issues.
Every new OKD installation has a dns.operator named default.
Use the oc describe command to view the default dns:
$ oc describe dns.operator/default
Name: default
Namespace:
Labels: <none>
Annotations: <none>
API Version: operator.openshift.io/v1
Kind: DNS
...
Status:
Cluster Domain: cluster.local
Cluster IP: 172.30.0.10
...
where:
Status.Cluster DomainSpecifiecs the base DNS domain used to construct fully qualified pod and service domain names.
Status.Cluster IPSpecifies the address that pods query for name resolution. The IP is defined as the 10th address in the service CIDR range.
To find the service CIDR range, such as 172.30.0.0/16, of your cluster, use the oc get command:
$ oc get networks.config/cluster -o jsonpath='{$.status.serviceNetwork}'
Configure DNS forwarding servers and upstream resolvers for the cluster.
You can use DNS forwarding to override the default forwarding configuration in the /etc/resolv.conf file in the following ways:
Specify name servers (spec.servers) for every zone. If the forwarded zone is the ingress domain managed by OKD, then the upstream name server must be authorized for the domain.
Provide a list of upstream DNS servers (spec.upstreamResolvers).
Change the default forwarding policy.
A DNS forwarding configuration for the default domain can have both the default servers specified in the /etc/resolv.conf file and the upstream DNS servers.
|
During pod creation, Kubernetes uses the |
Modify the DNS Operator object named default:
$ oc edit dns.operator/default
After you issue the previous command, the Operator creates and updates the config map named dns-default with additional server configuration blocks based on spec.servers.
If none of the servers have a zone that matches the query, then name resolution falls back to the upstream DNS servers.
apiVersion: operator.openshift.io/v1
kind: DNS
metadata:
name: default
spec:
cache:
negativeTTL: 0s
positiveTTL: 0s
logLevel: Normal
nodePlacement: {}
operatorLogLevel: Normal
servers:
- name: example-server
zones:
- example.com
forwardPlugin:
policy: Random
upstreams:
- 1.1.1.1
- 2.2.2.2:5353
upstreamResolvers:
policy: Random
protocolStrategy: ""
transportConfig: {}
upstreams:
- type: SystemResolvConf
- type: Network
address: 1.2.3.4
port: 53
status:
clusterDomain: cluster.local
clusterIP: x.y.z.10
conditions:
...
where:
spec.servers.nameMust comply with the rfc6335 service name syntax.
spec.servers.zonesMust conform to the rfc1123 subdomain syntax. The cluster domain cluster.local is invalid for zones.
spec.servers.forwardPlugin.policySpecifies the upstream selection policy. Defaults to Random; allowed values are RoundRobin and Sequential.
spec.servers.forwardPlugin.upstreamsMust provide no more than 15 upstreams entries per forwardPlugin.
spec.upstreamResolvers.upstreamsSpecifies an upstreamResolvers to override the default forwarding policy and forward DNS resolution to the specified DNS resolvers (upstream resolvers) for the default domain. You can use this field when you need custom upstream resolvers; otherwise queries use the servers declared in /etc/resolv.conf.
spec.upstreamResolvers.policySpecifies the upstream selection order. Defaults to Sequential; allowed values are Random, RoundRobin, and Sequential.
spec.upstreamResolvers.protocolStrategySpecify TCP to force the protocol to use for upstream DNS requests, even if the request uses UDP. Valid values are TCP and omitted. When omitted, the platform chooses a default, normally the protocol of the original client request.
spec.upstreamResolvers.transportConfigSpecifies the transport type, server name, and optional custom CA or CA bundle to use when forwarding DNS requests to an upstream resolver.
spec.upstreamResolvers.upstreams.typeSpecifies two types of upstreams: SystemResolvConf or Network. SystemResolvConf configures the upstream to use /etc/resolv.conf and Network defines a Networkresolver. You can specify one or both.
spec.upstreamResolvers.upstreams.addressSpecifies a valid IPv4 or IPv6 address when type is Network.
spec.upstreamResolvers.upstreams.portSpecifies an optional field to provide a port number. Valid values are between 1 and 65535; defaults to 853 when omitted.
You can inspect the status and view the details of the DNS Operator by using the oc describe command.
View the status of the DNS Operator:
$ oc describe clusteroperators/dns
Though the messages and spelling might vary in a specific release, the expected status output looks like:
Status:
Conditions:
Last Transition Time: <date>
Message: DNS "default" is available.
Reason: AsExpected
Status: True
Type: Available
Last Transition Time: <date>
Message: Desired and current number of DNSes are equal
Reason: AsExpected
Status: False
Type: Progressing
Last Transition Time: <date>
Reason: DNSNotDegraded
Status: False
Type: Degraded
Last Transition Time: <date>
Message: DNS default is upgradeable: DNS Operator can be upgraded
Reason: DNSUpgradeable
Status: True
Type: Upgradeable
You can view DNS Operator logs to troubleshoot DNS issues, verify configuration changes, and monitor activity by using the by using the oc logs command.
View the logs of the DNS Operator by running the following command:
$ oc logs -n openshift-dns-operator deployment/dns-operator -c dns-operator
Set CoreDNS log levels to control the detail of DNS error logging.
Log levels for CoreDNS and the CoreDNS Operator are set by using different methods. You can configure the CoreDNS log level to determine the amount of detail in logged error messages. The valid values for CoreDNS log level are Normal, Debug, and Trace. The default logLevel is Normal.
|
The CoreDNS error log level is always enabled. The following log level settings report different error responses:
|
To set logLevel to Debug, enter the following command:
$ oc patch dnses.operator.openshift.io/default -p '{"spec":{"logLevel":"Debug"}}' --type=merge
To set logLevel to Trace, enter the following command:
$ oc patch dnses.operator.openshift.io/default -p '{"spec":{"logLevel":"Trace"}}' --type=merge
To ensure the desired log level was set, check the config map:
$ oc get configmap/dns-default -n openshift-dns -o yaml
For example, after setting the logLevel to Trace, you should see this stanza in each server block:
errors
log . {
class all
}
You can view CoreDNS pod logs to troubleshoot DNS issues by using the oc logs command.
View the logs of a specific CoreDNS pod by entering the following command:
$ oc -n openshift-dns logs -c dns <core_dns_pod_name>
Follow the logs of all CoreDNS pods by entering the following command:
$ oc -n openshift-dns logs -c dns -l dns.operator.openshift.io/daemonset-dns=default -f --max-log-requests=<number> (1)
<number>: Specifies the number of DNS pods to stream logs from. The maximum is 6.
You can configure the Operator log level to quickly track down OpenShift DNS issues.
The valid values for operatorLogLevel are Normal, Debug, and Trace. Trace has the most detailed information. The default operatorlogLevel is Normal. There are seven logging levels for Operator issues: Trace, Debug, Info, Warning, Error, Unrecoverable, and Panic. After the logging level is set, log entries with that severity or anything above it will be logged.
operatorLogLevel: "Normal" sets logrus.SetLogLevel("Info").
operatorLogLevel: "Debug" sets logrus.SetLogLevel("Debug").
operatorLogLevel: "Trace" sets logrus.SetLogLevel("Trace").
To set operatorLogLevel to Debug, enter the following command:
$ oc patch dnses.operator.openshift.io/default -p '{"spec":{"operatorLogLevel":"Debug"}}' --type=merge
To set operatorLogLevel to Trace, enter the following command:
$ oc patch dnses.operator.openshift.io/default -p '{"spec":{"operatorLogLevel":"Trace"}}' --type=merge
To review the resulting change, enter the following command:
$ oc get dnses.operator -A -oyaml
You should see two log level entries. The operatorLogLevel applies to OpenShift DNS Operator issues, and the logLevel applies to the daemonset of CoreDNS pods:
logLevel: Trace
operatorLogLevel: Debug
To review the logs for the daemonset, enter the following command:
$ oc logs -n openshift-dns ds/dns-default
For CoreDNS, you can configure the maximum duration of both successful or unsuccessful caching, also known respectively as positive or negative caching. Tuning the cache duration of DNS query responses can reduce the load for any upstream DNS resolvers.
|
Setting TTL fields to low values could lead to an increased load on the cluster, any upstream resolvers, or both. |
Edit the DNS Operator object named default by running the following command:
$ oc edit dns.operator.openshift.io/default
Modify the time-to-live (TTL) caching values:
apiVersion: operator.openshift.io/v1
kind: DNS
metadata:
name: default
spec:
cache:
positiveTTL: 1h
negativeTTL: 0.5h10m
where:
spec.cache.positiveTTLSpecifies a string value that is converted to its respective number of seconds by CoreDNS. If this field is omitted, the value is assumed to be 0s and the cluster uses the internal default value of 900s as a fallback.
spec.cache.negativeTTLSpecifies a string value that is converted to its respective number of seconds by CoreDNS. If this field is omitted, the value is assumed to be 0s and the cluster uses the internal default value of 30s as a fallback.
To review the change, look at the config map again by running the following command:
$ oc get configmap/dns-default -n openshift-dns -o yaml
Verify that you see entries that look like the following example:
cache 3600 {
denial 9984 2400
}
You can change from the default Managed state to Unmanaged to stop the DNS Operator from managing its resources in order to apply a workaround or test a configuration change.
The DNS Operator manages the CoreDNS component to provide a name resolution service for pods and services in the cluster. The managementState of the DNS Operator is set to Managed by default, which means that the DNS Operator is actively managing its resources. You can change it to Unmanaged, which means the DNS Operator is not managing its resources.
The following are use cases for changing the DNS Operator managementState:
You are a developer and want to test a configuration change to see if it fixes an issue in CoreDNS. You can stop the DNS Operator from overwriting the configuration change by setting the managementState to Unmanaged.
You are a cluster administrator and have reported an issue with CoreDNS, but need to apply a workaround until the issue is fixed. You can set the managementState field of the DNS Operator to Unmanaged to apply the workaround.
|
You cannot upgrade while the |
Change managementState to Unmanaged in the DNS Operator by running the following command:
oc patch dns.operator.openshift.io default --type merge --patch '{"spec":{"managementState":"Unmanaged"}}'
Review managementState of the DNS Operator by using the jsonpath command-line JSON parser:
$ oc get dns.operator.openshift.io default -ojsonpath='{.spec.managementState}'
Control where CoreDNS and node-resolver pods run by using taints, tolerations, and selectors.
The DNS Operator has two daemon sets: one for CoreDNS called dns-default and one for managing the /etc/hosts file called node-resolver.
You can assign and run CoreDNS pods on specified nodes. For example, if the cluster administrator has configured security policies that prohibit communication between pairs of nodes, you can configure CoreDNS pods to run on a restricted set of nodes.
DNS service is available to all pods if the following circumstances are true:
DNS pods are running on some nodes in the cluster.
The nodes on which DNS pods are not running have network connectivity to nodes on which DNS pods are running,
The node-resolver daemon set must run on every node host because it adds an entry for the cluster image registry to support pulling images. The node-resolver pods have only one job: to look up the image-registry.openshift-image-registry.svc service’s cluster IP address and add it to /etc/hosts on the node host so that the container runtime can resolve the service name.
As a cluster administrator, you can use a custom node selector to configure the daemon set for CoreDNS to run or not run on certain nodes.
You installed the oc CLI.
You are logged in to the cluster as a user with cluster-admin privileges.
Your DNS Operator managementState is set to Managed.
To allow the daemon set for CoreDNS to run on certain nodes, configure a taint and toleration:
Set a taint on the nodes that you want to control DNS pod placement by entering the following command:
$ oc adm taint nodes <node_name> dns-only=abc:NoExecute
Replace <node_name> with the actual name of the node.
Modify the DNS Operator object named default to include the corresponding toleration by entering the following command:
$ oc edit dns.operator/default
Specify a taint key and a toleration for the taint. The following toleration matches the taint set on the nodes.
apiVersion: operator.openshift.io/v1
kind: DNS
metadata:
name: default
spec:
nodePlacement:
tolerations:
- effect: NoExecute
key: "dns-only"
operator: Equal
value: abc
tolerationSeconds: 3600
If the key field is set to dns-only, it can be tolerated indefinitely.
The tolerationSeconds field is optional.
Optional: To specify node placement using a node selector, modify the default DNS Operator:
Edit the DNS Operator object named default to include a node selector:
apiVersion: operator.openshift.io/v1
kind: DNS
metadata:
name: default
spec:
nodePlacement:
nodeSelector:
node-role.kubernetes.io/control-plane: ""
The nodeselector field in the example ensures that the CoreDNS pods run only on control plane nodes.
Configure DNS forwarding with TLS to secure queries to upstream resolvers.
When working in a highly regulated environment, you might need the ability to secure DNS traffic when forwarding requests to upstream resolvers so that you can ensure additional DNS traffic and data privacy.
Be aware that CoreDNS caches forwarded connections for 10 seconds. CoreDNS will hold a TCP connection open for those 10 seconds if no request is issued.
|
With large clusters, ensure that your DNS server is aware that it might get many new connections to hold open because you can initiate a connection per node. Set up your DNS hierarchy accordingly to avoid performance issues. |
Modify the DNS Operator object named default:
$ oc edit dns.operator/default
Cluster administrators can configure transport layer security (TLS) for forwarded DNS queries.
apiVersion: operator.openshift.io/v1
kind: DNS
metadata:
name: default
spec:
servers:
- name: example_server
zones:
- example.com
forwardPlugin:
transportConfig:
transport: TLS
tls:
caBundle:
name: mycacert
serverName: dnstls.example.com
policy: Random
upstreams:
- 1.1.1.1
- 2.2.2.2:5353
upstreamResolvers:
transportConfig:
transport: TLS
tls:
caBundle:
name: mycacert
serverName: dnstls.example.com
upstreams:
- type: Network
address: 1.2.3.4
port: 53
where:
spec.servers.nameMust comply with the rfc6335 service name syntax.
spec.servers.zonesMust conform to the rfc1123 subdomain syntax. The cluster domain, cluster.local, is invalid for zones.
spec.servers.forwardPlugin.transportConfig.transportMust be set to TLS when configuring TLS forwarding.
spec.servers.forwardPlugin.transportConfig.tls.serverNameMust be set to the server name indication (SNI) server name used to validate the upstream TLS certificate.
spec.servers.forwardPlugin.policySpecifies the upstream selection policy. Defaults to Random; valid values are RoundRobin and Sequential.
spec.servers.forwardPlugin.upstreamsMust provide upstream resolvers; maximum 15 entries per forwardPlugin.
spec.upstreamResolvers.upstreamsSpecifies an optional field to override the default policy for the default domain. Use the Network type only when TLS is enabled and provide an IP address. If omitted, queries use /etc/resolv.conf.
spec.upstreamResolvers.upstreams.addressMust be a valid IPv4 or IPv6 address.
spec.upstreamResolvers.upstreams.portSpecifies an optional field to provide a port number. Valid values are between 1 and 65535; defaults to 853 when omitted.
|
If |
View the config map:
$ oc get configmap/dns-default -n openshift-dns -o yaml
apiVersion: v1
data:
Corefile: |
example.com:5353 {
forward . 1.1.1.1 2.2.2.2:5353
}
bar.com:5353 example.com:5353 {
forward . 3.3.3.3 4.4.4.4:5454
}
.:5353 {
errors
health
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
upstream
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
forward . /etc/resolv.conf 1.2.3.4:53 {
policy Random
}
cache 30
reload
}
kind: ConfigMap
metadata:
labels:
dns.operator.openshift.io/owning-dns: default
name: dns-default
namespace: openshift-dns
The data.Corefile key contains the Corefile configuration for the DNS server. Changes to the forwardPlugin triggers a rolling update of the CoreDNS daemon set.