Debugging Pending Pods: Scheduling Failures in Kubernetes

When a pod is stuck in Pending, my first move is always kubectl describe pod and I read the Events section. The scheduler writes its failure reason there — it will literally say why each node was rejected.

Walking the decision tree

• The most common reason is resource shortage — "0/6 nodes are available: Insufficient cpu". I check kubectl top nodes for headroom and compare requests to allocatable; then check the Cluster Autoscaler or Karpenter logs to see why it isn't adding nodes.
• A key concept: the scheduler uses requests, not actual usage. A node at 30% CPU usage can still reject pods if committed requests already total 95% of allocatable — the scheduler is pessimistic.
• If Events mention untolerated taints, I run kubectl describe node and grep for Taints; the fix is either adding a matching toleration or removing the taint.
• If Events mention unbound PVC, I check kubectl get pvc and describe pvc — usually wrong zone, missing StorageClass, or CSI driver unhealthy.
• If Events mention topology spread or affinity, I check the pod spec's constraints — DoNotSchedule spread with tight maxSkew is a classic cause during rolling updates.
• If Events mention exceeded quota, kubectl describe resourcequota shows used vs hard limits.

Catching things `describe pod` doesn't tell you

• If Events are empty after several minutes, the scheduler itself might be unhealthy or a validating admission webhook is blocking — check kube-scheduler pod and validatingwebhookconfigurations.
• Nodes auto-taint on NotReady or memory-pressure — if every new pod is Pending with taint reasons cluster-wide, the cluster is unhealthy, not the pod.
• Pod density limits per node on AWS VPC CNI — a t3.small only holds 11 pods, so CPU-free doesn't mean pod-free.

Preventive patterns I put in place

• Use VPA in recommend-only mode to right-size requests — oversized requests cause unnecessary Pending.
• PodDisruptionBudget plus minReplicas 2 so rolling updates don't pin a lone pod unable to find a new home.
• PriorityClasses on critical workloads so they preempt lower-priority pods rather than sit Pending.
• Prometheus alert on kube_pod_status_phase{phase="Pending"} > 5m — anything Pending more than 5 minutes is an incident.

Escalation order

• Single pod Pending → look at the pod's constraints.
• All new pods in one namespace → Quota, StorageClass, default LimitRange.
• All new pods cluster-wide → scheduler health, CNI/CSI driver, node readiness.