My AI Agents Took Down My Own Server: A Read-Only Saturation Cascade on unRAID

At 23:03 one night my NUC accepted 33 root SSH sessions in a single minute, all from my own PC. No attacker. It was my AI agents, doing exactly what I asked: reading things. About forty minutes later the flash drive alert fired, unRAID's web workers were failing, and load was on its way to 60.51 on an 8-thread CPU.

TL;DR: two agent audits ran in parallel against the live server, one with two workers of its own. Every command was a read: SSH, docker exec, SQLite queries. Together they tipped a host with no swap, two VMs, and a long backup into a feedback loop of dockerd, memory reclaim, and retries. The flash alert that looked like the cause was a victim. The box recovered on its own a little over an hour after it started, with nothing restarted.

The timeline

All times are local.

Time	What happened
23:02	First audit starts repeated SSH and docker exec reads
23:03	Home Assistant audit starts one root task and two parallel workers
23:03	33 root SSH sessions accepted in one minute
23:03-23:16	Dozens of direct SQLite reads against the live Home Assistant recorder
23:15	One query scans and groups 13,928,522 state rows in 34.3 s
23:17	unRAID PHP worker failures start
23:33	PHP worker failures increase
23:37	Last normal flash heartbeat
23:40	Flash monitor goes down after three missed runs
00:03	Load 60.51, 53.23, 41.30; dockerd 76% CPU, kswapd0 50%
~00:09	Load 4.76, no D-state process left
~00:15	Load 2.73, SSH works again

The SSH count per minute tells the shape better than any graph. 6 at 23:02, then 33, 15, 14, 13, and it stays around 10 a minute until 23:12. That's 151 sessions in the first quarter hour, each one a fresh login with its own key exchange, its own shell, and usually its own docker exec.

The flash alert was a red herring

The alert said the boot flash was down. On unRAID that's scary, because /boot holds the config and the OS image lives on it.

It was also believable. The old rotated syslog held 37,273 SQUASHFS errors from a real flash failure five days earlier. So "the flash died again" was the obvious story.

The heartbeat is a cron job that reads 1 MB of /boot/bzmodules with iflag=direct every minute and pushes to Uptime Kuma. O_DIRECT skips the page cache, so it's a real read from the stick. When the host is saturated, that job simply doesn't get to run on time. Three missed runs and the monitor goes down.

What the current syslog didn't have was the trail a dying flash leaves:

Signature	Matching lines
USB reset or disconnect	0
EIO	0
FAT error	0
SQUASHFS error	0

And the heartbeat came back by itself, with delayed successes at 23:53 and 00:02, exactly when the host briefly made progress. Afterwards the flash reported running. That's not what a dead device looks like. That's a job stuck in a queue.

The rule I took from this: before you trust an alert about component X, grep for X's own failure signature. If it's not there, the alert is telling you about the scheduler, not the hardware.

What the loop looked like

No single command did this. The picture at the peak:

• dockerd at 76-79% CPU with 107 threads. Every docker exec is an API call to the daemon, and it was fielding them from three agents at once.
• docker-buildx helpers, 8 at 23:51 and 12 by 00:03 at 59% combined CPU, with their group in D state. As far as I can tell that's the Docker CLI probing its plugins on startup, so every short docker command drags a buildx process along.
• kswapd0 at 50% CPU.
• unRAID PHP workers failing and retrying, and monitoring requests piling on top.

The kswapd number is the tell. This box has no swap. Without swap the kernel's only way to free memory is to drop file pages, and when the busy processes need those pages back right away, reclaim just scans and evicts in a circle. That's CPU spent making no progress, on a host whose recovery capacity was already spoken for: two VMs running, and a long rclone backup in flight.

Then the feedback part. A slow daemon makes each docker exec take longer. Monitoring requests time out and come back, and unRAID's PHP workers retry. More requests, more memory, more reclaim, slower daemon. At 00:04 the hourly browserless restart fired late and added a Chrome launch into the middle of the recovery, which didn't help.

The 34.3 second recorder query deserves its own line. Scanning and grouping 13.9 million rows in the live Home Assistant SQLite file is fine on a quiet box. On this one it held memory and I/O for half a minute, against a file the recorder was using live.

The host fixed itself

Nobody restarted anything. The agent work wound down, the last light docker exec checks ran at 23:30, and the loop eventually ran out of fuel. By about 00:09 load was 4.76 with no process left in D state. By 00:15 it was 2.73, available memory was about 4 GB, and SSH worked again.

That matters, because the obvious move at 23:45 was a reboot, and possibly a flash swap. Both would have destroyed the evidence and "fixed" a device that wasn't broken. A hard reboot on unRAID also means an unclean array stop and a parity check.

How I investigated it

I used an agent for the diagnosis too, with tighter rules. It read the persistent syslog snapshots from a read-only mount, changed nothing on the NUC, and stored hashes of the raw files next to the timeline. The agent session transcripts stayed out of the evidence folder, because they held private context. The timeline keeps only sanitized counts and command categories.

The write-up also says what it can't prove. There's no stack trace from the peak, so I don't know which Docker lock or system call everything was blocked on. That would need a process stack or a daemon profile captured during the saturation. "Resource saturation feedback loop" is the best-supported explanation, not a proven single cause, and I'd rather write that than invent a culprit.

What the agent rules were missing

The uncomfortable bit is that my agent rules were fine on paper. On this server, any command that changes state needs my explicit yes. None of these commands changed state. The rule assumed reads are free, and on a 4-core NUC with no swap, they aren't.

So agents on this box need a load budget, not just a write gate:

• One SSH connection, reused. 151 fresh logins in fifteen minutes is the problem, not the reads themselves. A multiplexed connection, or one script that does the whole batch in one session, costs a fraction of that.
• No parallel workers against the live host. Parallel agents are great on my PC. Three of them pointed at one small server multiply the load, and none of them can see what the others are doing.
• Never scan a live database. Big analytical queries go against a copy, not the file the application is writing to.
• Check the host before starting heavy work. If load is already up, a VM is busy, or a backup is running, the audit waits.
• Diagnose before you reboot. An agent's first move in an outage is read-only evidence capture, and it looks for the component's own failure signature before it blames the component.

The browserless side already had its limits from a freeze the day before this, which I wrote up in 93 GB of Chrome Crash Dumps in My Docker Image. Different trigger, same lesson: on a small host, every client needs a limit, and that includes the ones I run myself.