Debugging CoreDNS: When Kubernetes Service DNS Fails

When I see 'networking seems broken' my first instinct is to verify DNS. In a Kubernetes cluster DNS is responsible for every service name resolution, and intermittent DNS issues are the single most common hidden cause of vague connectivity problems.

My first command

• kubectl run dns-test --rm -it --image=busybox --restart=Never -- nslookup kubernetes.default.
• If that fails, DNS is totally broken — CoreDNS down or unreachable; check kube-system pods and NetworkPolicy.
• If kubernetes.default works but my service name fails, it's a missing Service, wrong namespace, or label mismatch — kubectl get svc and describe it.
• If internal works but external doesn't, it's an upstream resolver issue — CoreDNS inherits the node's resolv.conf by default; check the VPC DNS or force an explicit upstream in the Corefile.

Check CoreDNS health

• kubectl get pods -n kube-system -l k8s-app=kube-dns — any CrashLoopBackOff? Any pod at 100% CPU under load?
• kubectl logs on CoreDNS pods with --previous if they've restarted — Corefile errors show up here.
• Scale CoreDNS replicas immediately if CPU is pegged — kubectl scale deploy coredns -n kube-system --replicas=5.

The classic Kubernetes DNS performance issues

• ndots: 5 default — every lookup with fewer than 5 dots goes through the search list, so google.com becomes 4 queries. Lower ndots to 2 via pod dnsConfig for external-heavy services; cuts external DNS volume 5x.
• The 5-second timeout — glibc sends A and AAAA in parallel on the same UDP socket and conntrack can race, dropping one and retransmitting after 5 seconds. Fixes are single-request-reopen in pod dnsConfig or enabling NodeLocal DNSCache which uses TCP.
• NodeLocal DNSCache is the standard scalability answer — a DaemonSet runs on every node with a local cache; pods hit the local cache first, CoreDNS sees only cache misses. Dramatically cuts load and eliminates conntrack races. Standard recommendation for clusters over 100 nodes.

NetworkPolicy is the production trap

• A strict default-deny egress policy without an explicit allow rule for UDP/53 to kube-dns breaks every pod in the namespace.
• The rule must allow both UDP and TCP port 53 to pods labeled k8s-app: kube-dns in kube-system.
• Classic production incident: a security-hardening policy rolls out, all services start failing to resolve, the fix is a one-line NetworkPolicy addition but it takes hours to find without the runbook.

Essential alerting I put in place

• CoreDNS P99 request duration > 100ms for 5 minutes — CoreDNS is overloaded.
• CoreDNS cache hit ratio drops suddenly — upstream failures.
• At least 2 CoreDNS pods ready — less than 2 is an incident.
• A synthetic DNS probe pod doing nslookup every 15 seconds — page at >1% failure rate.

Production hardening

• At least 3 CoreDNS replicas with topology spread across AZs and PDB minAvailable 2.
• NodeLocal DNSCache enabled.
• Pod-level ndots and single-request-reopen tuning.
• Explicit NetworkPolicy allow-dns rule in every namespace that uses default-deny.
• Document the DNS runbook — this is the single highest-ROI runbook because DNS-induced incidents are frequent and misdiagnosed.