How a Service IP becomes a packet

Key points

• A ClusterIP does not exist on any interface. Nothing answers a ping. It is a virtual address rewritten by rules on each node, which is why so many normal network tools give useless answers.
• kube-proxy programs those rules. In iptables mode it builds one chain per Service with random-probability jumps to each endpoint; in IPVS mode it uses a kernel load balancer that scales far better with thousands of Services.
• Load balancing is per connection, not per request. A long-lived HTTP/2 or gRPC connection pins to one pod, so a client that opens one connection sends all its traffic to a single replica no matter how many exist.
• That is the usual explanation for "we scaled to ten pods and one is doing all the work". The fix is client-side balancing, a headless Service, or a mesh - not more replicas.
• DNS is the other common surprise. A cluster ndots:5 setting means a name with fewer than five dots is tried against every search domain first, so api.example.com can cost four failed lookups before the right one.
• Endpoints follow readiness, not liveness. A pod failing readiness is removed from the Service while still running - which is exactly what you want during warm-up or shutdown.

Example

# The Service has an IP that exists nowhere. This is expected:
$ kubectl get svc orders
#   NAME     TYPE        CLUSTER-IP     PORT(S)
#   orders   ClusterIP   10.96.132.44   8080/TCP
$ ping 10.96.132.44        # no reply, and that is not a fault

# What it actually resolves to - the endpoints behind it:
$ kubectl get endpointslices -l kubernetes.io/service-name=orders
#   ADDRESSES                          READY
#   10.244.1.7,10.244.2.3,10.244.3.9   true

# The rules doing the work, on a node:
$ iptables -t nat -L KUBE-SERVICES -n | grep 10.96.132.44
$ iptables -t nat -L KUBE-SVC-XXXX -n
#   statistic mode random probability 0.33333  -> KUBE-SEP-A
#   statistic mode random probability 0.50000  -> KUBE-SEP-B
#   (always)                                   -> KUBE-SEP-C
#
# Note the probabilities are chained, not equal - 1/3, then 1/2 of the
# remainder, then the rest. That yields an even split overall.

---
# PER-CONNECTION balancing. One gRPC channel = one pod, forever:
#
#   client --[one HTTP/2 connection]--> pod-1   all traffic
#                                       pod-2   idle
#                                       pod-3   idle
#
# Fixes: a headless Service plus client-side balancing, a proxy that
# understands HTTP/2, or periodic connection recycling.

---
# ndots: cut needless DNS lookups for external names.
spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"
# Or sidestep search domains entirely with a trailing dot:
#   https://api.example.com.

A Service balances connections, not requests - so one long-lived HTTP/2 connection pins to one pod however many replicas you add.