“The true art of container engineering isn’t about what you put in—it’s about what you leave out. Every unnecessary package, every redundant library isn’t just bloating your image—it’s expanding your attack surface and slowing down your delivery pipeline.” – Kelsey Hightower
Optimizing Containers and Layers: A Pragmatic Guide for Engineering Leaders
The first time I optimized a container, I did what most engineers do: I copied an existing Dockerfile from the internet, built my image, and called it a day. It worked—until it didn’t. The image ballooned in size, builds took forever, and debugging was a nightmare. Eventually, it became clear that while containers are great, poorly designed ones can be a liability.
Over the years, I’ve learned a few hard lessons about optimizing container images, avoiding common pitfalls, and making sure they remain secure and maintainable. Whether you’re leading a team that ships containers daily or you’re just starting to optimize your DevOps workflows, these best practices will save you time, headaches, and security risks.
Understanding Layers and Why They Matter
Every container image is made up of layers. Each command in a Dockerfile that changes the filesystem (e.g., RUN, COPY, ADD) creates a new layer. This layering system brings benefits—like caching and efficient storage—but also comes with challenges.
What to Do
Minimize the Number of Layers
Instead of:
RUN apt-get update
RUN apt-get install -y curl
RUN apt-get install -y gitDo this:
RUN apt-get update && apt-get install -y curl git && rm -rf /var/lib/apt/lists/*This reduces unnecessary layers and avoids storing outdated package lists.
Order Commands to Maximize Caching
Docker caches layers, but if a layer changes, every layer after it gets rebuilt. Place frequently changing layers near the bottom and stable ones at the top. For example, installing dependencies before copying code:
COPY package.json package-lock.json /app/
RUN npm install
COPY . /app/This way, unless package.json changes, the install step is cached.
Use Multi-Stage Builds
Multi-stage builds allow you to use one image for building and another for running. This keeps the final image lean.
# Stage 1: Build
FROM golang:1.20 AS builder
WORKDIR /app
COPY . .
RUN go build -o myapp
# Stage 2: Runtime
FROM alpine:latest
WORKDIR /app
COPY --from=builder /app/myapp .
CMD ["./myapp"]This prevents the final image from carrying unnecessary build tools.
What to Avoid
Using Heavy Base Images
Base images like ubuntu or debian are convenient, but they come with bloat. Instead, consider smaller options like alpine, distroless, or scratch.
Installing Unnecessary Dependencies
Only install what your application needs. If a dependency is only needed at build time (e.g., compilers, package managers), use a multi-stage build so it doesn’t end up in the final image.
Leaving Behind Temporary Files
Many tools download temp files but don’t clean them up. Be sure to remove them within the same layer.
RUN curl -o package.tar.gz https://example.com/package.tar.gz && \
tar -xzf package.tar.gz && \
rm package.tar.gzThis ensures the temp file never gets added to an unnecessary layer.
Security Considerations
Optimizing a container isn’t just about size and speed—it’s about security.
Use Minimal Base Images
Smaller images mean fewer vulnerabilities. distroless is a great option since it has no package manager, reducing attack surface.
Scan for Vulnerabilities
Use tools like:
- Trivy (trivy image myimage:latest)
- Grype (grype myimage:latest)
- Docker Scout
These tools detect outdated libraries, known CVEs, and other security issues.
Avoid Running as Root
By default, containers run as root. Instead, create a non-root user:
RUN addgroup --system appgroup && adduser --system --ingroup appgroup appuser
USER appuserThis prevents privilege escalation attacks.
Keep Dependencies Updated
Regularly rebuild and scan images to check for outdated packages. Automate this with tools like Dependabot, Renovate, or GitHub Actions.
When to Create Your Own Base Image
Most teams can get by with existing images (alpine, node, python), but there are times when a custom base image is the better option:
When You Need Absolute Control Over Dependencies
If you only need a few system packages and don’t want a bloated image, creating your own minimal base image ensures you ship only what’s necessary.
When You Need Consistency Across Teams
A custom base image ensures all teams start from the same known-good environment, reducing “works on my machine” issues.
When Security is a Priority
Public base images can contain vulnerabilities. With a private base image, you can strip out anything unnecessary and harden security.
How to Create a Minimal Base Image
- Start with scratch, the smallest possible image.
FROM scratch
COPY mybinary /
CMD ["/mybinary"]If you need some Linux utilities, use alpine:
FROM alpine:latest
RUN apk add --no-cache bash curlRegularly update and scan it for security vulnerabilities.
Debugging and Optimization Tools
When something goes wrong—or an image is mysteriously large—these tools can help:
Analyze Image Size
- docker images – Check all images and sizes.
- docker history myimage:latest – See which layers are largest.
- dive myimage:latest – A great visual tool to inspect image layers.
Debug a Running Container
- docker exec -it container_name sh – Opens a shell inside the container.
- docker logs container_name – View logs.
- strace – Trace system calls to debug issues inside the container.
Wrapping up…
Optimizing containers is an ongoing process, but the core principles remain the same:
- Keep images small by minimizing layers and dependencies.
- Use multi-stage builds and caching effectively.
- Prioritize security with minimal base images, user permissions, and vulnerability scanning.
- Create your own base image when it makes sense for your team’s needs.
The next time you’re writing a Dockerfile, think about the long-term impact. A well-optimized image won’t just save space—it’ll make your builds faster, deployments smoother, and infrastructure more secure.
