Control plane and cluster-scale failures

Key points

• A control plane outage does not stop running pods. kubelet keeps containers alive and kube-proxy keeps routing, so existing traffic is served - you simply cannot change anything.
• What stops is everything that needs the API: scheduling, rollouts, scaling, endpoint updates. A pod that dies is not replaced, and that is how a control plane problem becomes a gradual application outage.
• etcd is the real single point of truth. It needs a quorum, so a three-node etcd tolerates one failure and no more, and it is very sensitive to disk latency - slow disks show up as cluster-wide API slowness.
• API priority and fairness exists because a badly behaved controller listing every pod every second can starve the API server. Requests are classified and throttled rather than served first-come.
• Node pressure triggers eviction in QoS order: BestEffort first, then Burstable, and Guaranteed last. A pod with no resource requests is the first casualty of memory pressure elsewhere on the node.
• DNS is the classic cluster-wide dependency. CoreDNS under-provisioned or throttled makes every service look broken at once, and the symptom - intermittent connection failures everywhere - misleads almost everyone.

Example

# What survives a control plane outage:
#
#   KEEPS WORKING                      STOPS
#   running pods                       scheduling new pods
#   kube-proxy routing                 rollouts and scaling
#   existing Service endpoints         endpoint updates on pod death
#   node-local storage                 ConfigMap/Secret updates
#
# So the failure is invisible until something needs to change - and then
# it looks like a slow-motion outage rather than an incident.

# etcd health - check the disk, it is nearly always the disk:
$ etcdctl endpoint status --write-out=table
$ etcdctl endpoint health
#   etcd_disk_wal_fsync_duration_seconds p99 > 10ms -> the cluster will
#   feel slow everywhere, and nobody will suspect storage

# Is the API server throttling someone?
$ kubectl get --raw /metrics | grep apiserver_flowcontrol_rejected
#   a controller in a hot loop can starve everything else

# Eviction order under node pressure - QoS decides:
$ kubectl get pods -o custom-columns=\
NAME:.metadata.name,QOS:.status.qosClass --all-namespaces | grep BestEffort
#   BestEffort pods are killed FIRST. Setting requests is not just about
#   scheduling - it is what stops your pod being the sacrifice.

# DNS: one dependency every pod has.
$ kubectl -n kube-system top pods -l k8s-app=kube-dns
$ kubectl -n kube-system logs -l k8s-app=kube-dns | grep -i "throttl\|timeout"
#   Symptom of CoreDNS trouble: intermittent connection failures across
#   many unrelated services at once. Check DNS before anything else when
#   everything breaks simultaneously.

Running pods survive a control plane outage but nothing can change - and when everything breaks at once, suspect DNS first.