93 GB of Chrome Crash Dumps in My Docker Image: Debugging a Headless Browser Container

My unRAID box sent a 75% alert for docker.img, and the thing eating it wasn't a database or a media cache. It was a headless browser container holding 93 GB of Chrome crash dumps. About 2.28 million of them, piling up quietly since the middle of May.

TL;DR: a client that couldn't connect kept making browserless launch Chrome, Chrome kept dying and writing a crash report, and an hourly docker restart never cleared any of it because restart keeps the writable layer. The fix was a recreate plus a 256 MB tmpfs over Chrome's config dir. A week later the same container froze the whole host through its pids limit, which is a different lesson.

What was actually in the layer

On unRAID, every container's writable layer lives inside one loopback image, docker.img. So a single container that writes where it shouldn't doesn't fill "its" disk. It fills everyone's.

The files were here:

/home/blessuser/.config/google-chrome-for-testing/Crash Reports/pending

That's Chrome's crash handler queue. Nothing in the container uploads or clears that queue, so pending just grows. 93 GB over 2.28M files works out to roughly 40 KB a dump. Small files, a lot of them, for months.

The loop that fed it

The client was Karakeep, my bookmark archiver, which uses browserless to render pages. The cycle looked like this:

1. Karakeep connects, about every 10 seconds.
2. browserless launches Chrome for the session.
3. Chrome dies with signal 6 and spawn chrome EAGAIN.
4. Crashpad writes a dump into pending.
5. Karakeep logs Failed to connect ... will retry in 5 secs and goes back to step 1.

EAGAIN from spawn means the kernel refused to create the process, so Chrome never got a fair start. Each failure was cheap and quiet. Nothing alerted, because nothing was down in the "service is down" sense. Karakeep's crawls just kept failing, and the disk paid for every retry.

Part of the connection failure was Karakeep's own config. It uses Playwright chromium.connect() with a 5 second timeout, and I had pointed it at browserless's root path. The root path speaks raw CDP only, so every Playwright connect timed out. The correct URL is the Playwright route:

BROWSER_WEBSOCKET_URL=ws://host.docker.internal:3741/chromium/playwright?token=<token>&stealth=true&launch={...,"--disable-breakpad","--disable-crash-reporter"}
BROWSER_CONNECT_ONDEMAND=true

BROWSER_CONNECT_ONDEMAND matters because browserless runs with TIMEOUT=60000, which kills a persistent session after 60 seconds. On demand, Karakeep opens a session per job instead of holding one that gets shot.

Why the hourly restart never helped

I already had a restart-browser script running every hour. It restarted this container like clockwork, and the dumps were still there.

That's the trap. docker restart stops and starts the same container. The writable layer is part of the container, not the process, so everything Chrome wrote survives. Only docker rm plus a fresh docker run gives you a clean layer.

Two practical notes from doing that on unRAID:

• docker ps -a --size is how you see which layer is fat. It's slow, minutes on a box like this, because it walks every layer.
• On btrfs, docker rm of a huge layer also runs for minutes and keeps the name reserved. Your docker run fails with a name conflict until the delete finishes. Retry, don't panic.

The guard: make the crash dir RAM with a cap

Fixing the client URL stopped this particular loop. It doesn't stop the next one. Any future client bug that crashes Chrome in a loop would do the same thing. So the crash directory now lives on a capped tmpfs, set in the container's ExtraParams:

--tmpfs /home/blessuser/.config/google-chrome-for-testing:size=256m,uid=999,gid=999,mode=700

The uid and gid matter. Chrome runs as the image's non-root user, and a root-owned tmpfs would trade the disk problem for a permissions failure. With the cap, the worst case is 256 MB of RAM that resets on every recreate, not a full docker.img.

Belt and braces, both clients also pass --disable-breakpad --disable-crash-reporter in their launch args. browserless v2 has no DEFAULT_LAUNCH_ARGS, so the flags have to ride in each client's URL. And the check I run after any change:

docker exec browserless sh -c 'ls "/home/blessuser/.config/google-chrome-for-testing/Crash Reports/pending" | wc -l'

It should stay near 0.

A different failure: the pids limit froze the host

A week later the NUC froze and needed a reboot, which kicked off a parity check. First suspect was the boot flash, which had been flaky. It wasn't: zero SQUASHFS errors.

The real story was in the kernel log. Another client of mine was scraping a shopping site through browserless's raw CDP route over WireGuard, about 5 jobs a minute, 2 at a time, on pages that served recaptcha. Those pages are heavy, and Chrome forks a lot. At 18:11 the kernel logged:

fork rejected by pids controller

for the browserless cgroup. The container runs with --pids-limit=400. That limit did its job in the narrow sense: the browser couldn't fork without bound. But the Docker API hung right after. Every container monitor in Uptime Kuma went down at once, unRAID's web UI had php-fpm SIGKILLed 83 times, and load hit 29, mostly from processes stuck in D state waiting on something that wasn't coming back.

I don't have a stack trace that ties the two together, but the order is hard to argue with. A pids limit isn't a concurrency limit. It caps the process count, but the work keeps arriving, and once the Docker daemon is the thing waiting, the whole host looks dead.

The fix was to limit work at the front door:

One Chrome at a time, a queue of two, and anything past that gets rejected fast instead of forking into the wall. The container also has a cpuset of two threads, 1.5 CPUs, and 8 GB, so even a bad page can't take the whole NUC. The signature I watch now is docker stats pids for browserless climbing toward 400.

A side effect worth writing down: during the freeze, sshd was penalizing my own LAN retries as if they were attacks, because timed-out logins look like failures. I added PerSourcePenaltyExemptList for the LAN and WireGuard ranges so I can still get in when the box is struggling.

Since recreating this container is now a routine move, it's scripted: a user script parses env and ExtraParams from the dockerMan template and aborts if it can't parse the token. A recreate that silently drops the auth token is worse than no recreate.

Why the image tag is pinned

browserless runs pinned to ghcr.io/browserless/chromium:v2.52.2, and I'll only move it on purpose.

The /chromium/playwright route only accepts Playwright client versions the image bundles. Anything else gets:

428 Precondition Required
Playwright version mismatch

My two clients ship different Playwright versions:

v2.52.x bundles core 1.60.0 plus 1.56 through 1.59, so both work. :latest at the time was v2.56.0, which bundles 1.58 through 1.62. That keeps Open WebUI and breaks Karakeep. A floating tag would have turned a routine image update into the exact silent connect-fail loop from the top of this post.

So before any tag change:

curl -s https://raw.githubusercontent.com/browserless/browserless/<tag>/package.json | grep playwright

and after any recreate, a real chromium.connect() from inside the Karakeep container against its BROWSER_WEBSOCKET_URL. One more recreate gotcha: don't copy NODE_PATH or PLAYWRIGHT_BROWSERS_PATH from docker inspect of the old container. They're image-specific, and carrying them to a new tag points Node at paths that no longer exist.