Layers, OverlayFS and copy-up cost

Key points

• An image is a stack of read-only layers unified by OverlayFS. The container adds one thin writable layer on top, and everything below it is shared between containers.
• Deleting a file in a later layer does not remove it. Overlay records a whiteout that hides it, so the bytes are still in the image and still downloaded - which is why removing a secret in a later RUN does not remove it.
• Modifying any file triggers copy-up: the whole file is copied into the writable layer first. Appending one line to a 1GB log means copying 1GB, which is a genuine and surprising latency spike.
• That is why databases and other write-heavy workloads must use a volume. A volume bypasses the union filesystem entirely and writes straight to the host.
• Layer count matters less than layer content. What matters is ordering: put rarely changing things first so the cache survives, and never let a frequently-changing COPY invalidate the dependency layer beneath it.
• Layers are content-addressed and shared across images, so ten images on one base download that base once. A gratuitously different base defeats that sharing for every image you build.

Example

# The secret is still in the image. Both layers ship.
FROM alpine
COPY secrets.env /tmp/secrets.env      # layer 2: the file exists
RUN rm /tmp/secrets.env                # layer 3: a whiteout hides it

$ docker history --no-trunc myimage    # layer 2 is still there
$ docker save myimage | tar -x         # and readable from the tar

# CORRECT - a multi-stage build, so the secret layer is never in the result:
FROM alpine AS build
COPY secrets.env /tmp/secrets.env
RUN ./configure-with-secret.sh

FROM alpine
COPY --from=build /app/output /app     # only the output crosses over

# Or a BuildKit secret mount - never written to any layer at all:
# syntax=docker/dockerfile:1
RUN --mount=type=secret,id=env \
    ./configure.sh < /run/secrets/env
#   docker build --secret id=env,src=secrets.env .

---
# CACHE ORDERING - dependencies before source, always.

# WRONG: any source change re-downloads every dependency.
COPY . /app
RUN ./gradlew build

# RIGHT: dependencies are their own layer and survive source edits.
COPY build.gradle settings.gradle /app/
RUN ./gradlew dependencies --no-daemon      # cached until the build file changes
COPY src /app/src
RUN ./gradlew bootJar --no-daemon

---
# COPY-UP: writing to a container filesystem copies the whole file first.
#
#   append 1 line to a 1GB file in the container FS -> copies 1GB
#   the same append on a volume                     -> writes 1 line
#
# Which is why any write-heavy path needs a volume:
#   docker run -v pgdata:/var/lib/postgresql/data postgres

A deleted file is only hidden, not removed - and any write copies the whole file up, so write-heavy paths belong on a volume.