Why My Server Froze: Docker Build Cache Filled the Disk

One morning I noticed that both this site and its content management system (CMS) had become strangely slow. Pages took ages to load, the save button in the CMS didn't respond, and even connecting to the server over SSH took minutes. Nothing had changed in the application code, and memory was free. The culprit was Docker build leftovers, piling up silently with every deploy and never removed. In this article I go through the symptoms, the diagnosis, the fix and the safeguards I put in place so it doesn't happen again. If you deploy to a server with docker compose up --build, the same trap may be waiting for you.

The setup: building on the server on every push

The site and the CMS run on the same server with Docker Compose: two .NET applications, PostgreSQL and Redis. The server is a virtual machine with 4 cores, 6 GB of memory and a 90 GB disk, running Ubuntu 24.04. Deploys run from GitHub Actions over SSH, and on the server they roughly did this:

git fetch origin master
git reset --hard origin/master
docker compose up -d --build
docker image prune -f

So the images were built on the server itself, and old untagged images were removed afterwards. At first sight a tidy setup: unused images get cleaned up, the disk won't fill. It did.

Symptoms: free memory, CPU at the ceiling

When I managed to connect over SSH, the first thing I checked was load and memory:

$ uptime && free -m
 14:44:51 up 3 days, 12:19,  1 user,  load average: 76.70, 60.86, 35.55
               total        used        free      shared  buff/cache   available
Mem:            5925        1175        1094         179        4133        4750

A load average of 76 on a four-core machine. Memory, on the other hand, was almost untouched: 4.7 GB available. So this was not "the application ate the memory". The top output told more:

%Cpu(s): 39.3 us, 58.4 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  2.2 si
    PID USER   %CPU  COMMAND
2422120 root   54.3  kworker+
2393805 1654   23.6  dotnet
   2095 dnsmasq 18.1 redis-s+
2423805 70     11.0  postgres  (zombie)
2423442 root    3.9  runc:[2+
2423815 1654    3.9  curl

Three clues here:

• sy at 58 percent: more than half of the CPU time was spent in the kernel, not in the applications. On a normal web server this is a few percent.
• The top CPU consumer was a kworker: a kernel thread. Not the applications, but the operating system was struggling with something.
• runc and curl processes: Docker's health checks run a curl inside each container every 10 seconds. As the server slowed down, these checks didn't finish in time, new ones piled on top and added even more load. Health check responses took between 8 and 28 seconds.

That also explains the huge load average. On Linux, load average counts not only processes waiting for a CPU but also processes waiting for something inside the kernel. When the kernel is clogged, waiting processes pile up and the number climbs into the dozens.

I also tried a restart: reboot hung during shutdown. Another sign that this was not an ordinary application bug but something at the operating system level.

Diagnosis: df and docker system df

One of the most common reasons for a kernel to struggle like this is a disk that is about to fill up. As a disk nears full, the file system has to work much harder to find free space, and processes that write data, such as the database and loggers, start waiting. Checking takes one command:

df -h /

The disk was almost completely full. The images users had uploaded couldn't be the cause; there were about thirty of them in total. Docker's own report shows what takes the space:

docker system df

It lists the space used by images, containers, volumes and the build cache separately. In my case, the build cache was carrying most of the weight.

The root cause: the legacy builder and layers nobody removed

When Docker builds an image, it produces a layer for every step in the Dockerfile and keeps these intermediate layers to speed up the next build. In .NET projects these layers are not small: the SDK image, the packages downloaded by dotnet restore, the build output… Every deploy left a new stack of layers behind for both applications.

The docker image prune -f in the deploy script only removes dangling (untagged) images; it doesn't touch the build cache. On top of that, the server didn't have buildx, the command-line plugin for Docker's newer builder, BuildKit. Docker said so itself when I ran the cleanup:

$ docker builder prune -af
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            Install the buildx component to build images with BuildKit

So the images were being built with the old "legacy builder", and the intermediate layers of every deploy kept accumulating. With several deploys in a day, the 90 GB disk filled up.

The fix: builder prune, and patience

The cleanup command is simple:

docker builder prune -af

-a removes all unused build cache, -f skips the confirmation. It freed about 55 GB and brought disk usage down to 18 percent. It wasn't fast, though: deleting that much data on an already struggling disk took about four hours. While it ran, the site kept serving pages, slowly.

A note for honesty: around the same time there was a separate hardware failure on the hosting provider's side, and the server was completely unreachable for a while. The provider fixed that, and it had nothing to do with the disk problem. When investigating an incident, not mixing up two separate problems is the first condition for not drawing the wrong conclusion.

Safeguards: automatic cleanup on deploy and a log limit

To keep it from happening again I made two changes. The first is build cache cleanup at the end of the deploy script:

docker compose up -d --build
docker image prune -f
# Remove build cache older than a day: the next deploy still
# benefits from recent layers, but leftovers don't pile up.
docker builder prune -af --filter "until=24h"

--filter "until=24h" keeps the cache from the last 24 hours. Back-to-back deploys still get faster from the cache, but old layers are removed every time.

The second is a limit on container logs. By default Docker keeps each container's output in a single JSON file with no size limit. On a long-running site that file can fill the disk too. In Docker Compose it is easy to give every service a shared logging setting:

x-logging: &default-logging
  driver: json-file
  options:
    max-size: "10m"
    max-file: "3"

services:
  web:
    logging: *default-logging
  cms:
    logging: *default-logging

With this, each service's log takes at most 3 × 10 MB.

In addition, I recommend:

• Install buildx. BuildKit builds faster and has its own garbage collection for its cache. The legacy builder is being retired anyway.
• Set up a disk alert. A simple monitor that warns when disk usage passes 80 percent would have caught this hours earlier.
• If you can, don't build images on the server. Building the image in CI, pushing it to a registry and running only docker compose pull on the server spares both the disk and the CPU.

A checklist for a similar situation

If your server is inexplicably slow, memory is free and the CPU is busy in the kernel, check these in order:

1. uptime and free -m: is the load high while memory is free?
2. top: is sy high, are kernel processes such as kworker at the top?
3. df -h: is the disk about to fill? Check inodes too with df -i; many small files can exhaust inodes even when there is space left.
4. docker system df: is the space taken by images, volumes or the build cache?
5. du -sh /var/lib/docker/containers/*: have container logs grown?

Conclusion

The code was not to blame in this incident; the problem was hidden in the deploy setup. Any setup that builds images on the server also has to manage the build cache, and docker image prune is not enough for that. A docker builder prune at the end of every deploy, capped container logs and a simple disk alert prevent this kind of outage from the start. For more on site speed, see Website Speed and Core Web Vitals; why the first requests after a deploy are slow is covered in ReadyToRun and Cold Starts.