DaemonSets: one pod per node

Key points

• A DaemonSet keeps exactly one pod on every matching node. There is no replicas field: the node count is the count.
• Add a node and its pod appears automatically. Drain the node and the pod goes with it.
• The usual tenants are Fluent Bit, node-exporter, the CNI agent and CSI node drivers - things that must see the host.
• Narrow it with nodeSelector, and add tolerations when you also want it on tainted or control-plane nodes.
• Keep requests small. Whatever you ask for here is multiplied by every node in the cluster.
• Updates roll with maxUnavailable: 1 by default, so a large cluster takes a while to finish a version bump.

Example

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-shipper
spec:
  selector:
    matchLabels:
      app: log-shipper
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: log-shipper
    spec:
      tolerations:                 # run on control-plane nodes too
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:3.1
          resources:
            requests:
              cpu: "50m"           # x every node in the cluster
              memory: "96Mi"
          volumeMounts:
            - name: varlog
              mountPath: /var/log
              readOnly: true
      volumes:
        - name: varlog
          hostPath:
            path: /var/log

A Deployment gives you N pods somewhere. A DaemonSet gives you one pod everywhere.