The USB Boot Drive That Dropped Twice: Detecting and Replacing an unRAID Flash

unRAID runs from a USB stick. Not just the bootloader: /boot lives on it, and parts of the running OS are mounted off it as loop devices for as long as the box is up. So when the stick falls off the bus, the server doesn't crash cleanly. It goes half-dead, and some things keep working just well enough to fool you.

Mine did that twice in nine days, on 2026-08-25 and again on 2026-09-02. The second time, the alert hit my phone about three minutes after the drop, and the stick was replaced an hour later.

TL;DR: read the flash with O_DIRECT every minute and push the result to a heartbeat monitor, then watch that monitor from a machine that isn't the NUC. Keep an off-box copy of the flash every day. When you swap sticks, the new one must be labeled UNRAID, only one stick can be plugged in, and the license move needs internet from a box whose DNS container is down.

What a flash drop looks like

The box is an Intel NUC running unRAID, and the boot stick was an HP v222w in the front top-left USB port.

The first drop, on 2026-08-25 at 04:56, was the loud kind. The stick (sda) disconnected, reads of bzmodules and bzfirmware returned EIO, the log filled with about 37k SQUASHFS error lines, and the host froze.

The second one, at 23:55:51 on 2026-09-02, was quieter and more interesting. The stick disconnected with no event before it in the syslog and came back 4 seconds later as a new device, sdg. That doesn't help. Everything was still pointing at the old one:

• /boot (sda1), /usr (loop0), and /lib (loop1) all returned EIO.
• emhttpd logged ENOFLASH3.
• dockerd died with SIGBUS, and containerd and rsyslogd went with it. The persistent syslog stops at 23:55:59.

Then the part that fools you: the 47 container shims kept running. Caddy, Home Assistant, AdGuard, Uptime Kuma, and Postgres all stayed up. The web UI, Samba, the Docker health checks, and any container that restarted were dead. From the couch, the house looked fine.

SSH was the next trap. Key login failed, because /root/.ssh is a link to /boot/config/ssh/root and /boot was gone. Password login still worked, so that's what I used. I checked USB autosuspend while I was in there, and it wasn't the cause: power/control was already on for the stick.

The flash also gives you a clean way to rule it out. A host freeze on 2026-08-29 looked similar from the outside, but the syslog had zero SQUASHFS error lines. That one turned out to be a headless browser container hitting its process limit and hanging the Docker API. No SQUASHFS errors, not the flash.

Recovery without /usr

Recovery took 00:16 to 00:24. nohup wouldn't even load, because it lives in /usr, and /usr was on the stick. This worked over password SSH:

sync; /sbin/reboot

Shutdown took 90 seconds. The array couldn't stop cleanly with the flash gone, so a parity check started on boot. All three disks came back DISK_OK. Two containers needed a hand: a Telegram bot that timed out before the network was up (unRAID doesn't retry autostart), and Forgejo, which it turned out had never been on autostart at all. That's fixed now, and it has its own monitor.

Detection: read the actual flash

The heartbeat is a user script on a one-minute cron. This is the live version with the token swapped out:

#!/bin/bash
#clearLog=true
KUMA_PUSH_URL="https://kuma.example/api/push/<token>"
 
push() {
    curl -fsS -m 10 -G --data-urlencode "status=$1" --data-urlencode "msg=$2" "$KUMA_PUSH_URL" >/dev/null 2>&1 \
        || logger -t nuc_flash_heartbeat "kuma push failed (status=$1)"
}
 
if dd if=/boot/bzmodules iflag=direct bs=1M skip=$((RANDOM % 80)) count=1 of=/dev/null 2>/dev/null; then
    push up "flash read ok"
else
    logger -t nuc_flash_heartbeat "flash read FAILED"
    push down "flash read failed"
    exit 1
fi

iflag=direct is the whole trick. Without it, Linux hands you the file from the page cache and the read succeeds with the stick unplugged. With it, 1 MB actually comes off the flash every minute, from a random spot in the first 80 MB of bzmodules. A failed push goes to the syslog, and #clearLog=true stops the User Scripts plugin from growing its log in RAM by a few hundred bytes a run. A dead cron and a dead stick look the same from Kuma: the pushes stop.

The Kuma push monitor has a 90-second interval and retries 60 s x 2, so it goes down about 3.5 minutes after the last good push. It started at 60 seconds, the same as the cron period. That hit an open upstream race (uptime-kuma #5357): Kuma's retry timer and the push wrote the same stat row in the same second, and after that Kuma rejected every push for the rest of the hour. I got a false NUC-down alert at 01:03 on 2026-09-04 for a box that was fine. Moving to 90 seconds fixed it.

Detection: watch from outside

Kuma runs on the NUC. When the NUC is the thing that's broken, Kuma can't tell you. So a small VPS I call the hub runs a systemd timer every minute that checks three things:

1. Ping the NUC over WireGuard.
2. GET Kuma's public status page through Caddy. That page exposes exactly one monitor: the flash heartbeat.
3. Check the last heartbeat of that monitor is 1.

Three failures in a row sends a Telegram alert with the reason. It repeats every 30 minutes while the NUC is down, and sends one "back" message on recovery. Each check covers something different: the ping covers power, the kernel, and WireGuard; the status page covers Caddy, Docker, the cache SSD, and Kuma; and the heartbeat covers a USB flash drop or a dead cron.

The watcher also pushes to its own Kuma monitor each run, so if the hub dies, Kuma on the NUC complains. And there's no SSH and no keys in either direction, by design. A compromised hub shouldn't be a way into the NUC.

On 2026-09-02 the timeline was: drop at 23:55:51, heartbeat monitor down at 23:58, hub alert at 23:59.

The swap

The old stick had dropped twice in nine days, and it had a habit of doing it about once a month. It went out on 2026-09-03, between 00:30 and 00:55, for a SanDisk Cruzer Blade.

The procedure:

1. On the PC, build the new stick from the newest off-box flash zip, with the unRAID USB Creator's local zip option.
2. Shut the NUC down, pull the old stick, insert the new one, boot.
3. The array stays stopped. Run Replace Key, then start the array.

Never boot with both sticks in. unRAID mounts the first drive labeled UNRAID, and you don't get to pick which one that is.

The label is where it bit me. The USB Creator's local zip write left the volume labeled BOOT, and unRAID won't boot without UNRAID. The fallback script, make_bootable.bat, checks the label but doesn't set it. So on Windows:

label E: UNRAID
rem then, from an admin prompt:
make_bootable.bat

The zip already had the EFI folder, so UEFI boot needed nothing extra.

Moving the license

An unRAID key is tied to the flash drive's hardware GUID, so a new stick means a new GUID and a stopped array. Replace Key moves the key to the new stick and blacklists the old GUID. That's the other reason the HP will never go back in: its GUID is dead to the license server.

A few things the docs don't say loudly:

• Replace Key runs from the Unraid.net account app (sign in, Replace Key, tick the blacklist box). The local registration page only links there.
• It needs internet. My house DNS is an AdGuard container, and Docker hadn't started. That didn't matter, because the host resolves through /etc/resolv.conf, not the container.
• The automated replace works once every 12 months. The next one before that date has to go through support.

A clean shutdown also cleared the queued parity check, so I ran one by hand later.

The flash backups that made it boring

The swap was a 25-minute job because the zip was already sitting on my PC. There are two copies:

The PC copy is the one that matters in a flash failure, because it doesn't need the NUC to be alive. It runs once per calendar day behind a flock and a .last-run stamp, pushes its own Kuma monitor, and pushes status=down with the reason if a step fails. It also keeps a copy of the NUC's rclone config, so the encrypted S3 and Glacier backups stay readable with the NUC dead.

Restore from either one is the same: USB Creator, or unzip onto a FAT32 stick labeled UNRAID, run make_bootable, then Replace Key.

The real fix is to stop booting from a flash stick at all. unRAID 7.3 can boot from an internal SSD and keep the stick only as the license key. This NUC has no free internal slot for it, so that's a separate job. Until then, the SanDisk sits in the same front port, and the HP model is on the never-again list.