A Telegram Bot That Commits to Git and Survives docker stop

The bot that writes my notes is PID 1 in its container, and its first build had no SIGTERM handler. So docker stop sent SIGTERM, the kernel ignored it, and Docker sent SIGKILL when the grace period ran out. That kill can land between writing a line and marking the Telegram message as seen, and in my notes repo a duplicate line is permanent.

TL;DR: the bot sets a flag on SIGTERM, finishes the update in flight, and exits 0. The deploy waits for the bot's git flock before it stops the container, and the bot repairs a stale index.lock or a half-done rebase at startup. A Forgejo pre-receive hook keeps the bot's account on main and out of the folders it has no business in, and force pushes to main are refused for everyone. The first deploy with the handler still took 36 seconds.

What the bot does

I send it a text or a voice note on Telegram. It answers exactly one user ID and ignores everyone else. A text becomes one line in that day's file:

- HH:MMZ [telegram] #inbox <text>

Time is UTC, but the Dubai date picks the file. A voice note gets saved, queued for transcription, and shows up later as a #raw line with a pointer to the audio.

Every write is a git commit. The daily files are append-only: a #raw line is a transcript and nothing is allowed to edit or delete it. That rule is the reason everything below exists. If the bot writes a line twice, the second copy stays forever.

One writer, one lock

Every automated commit to the repo goes through one module, commit.py. It takes an exclusive flock on state/git.lock, runs git add on the exact files (never a directory), commits, runs pull --rebase and pushes with 3 retries. The push token goes in through an inline credential helper, so it never lands in .git/config or the process list.

The bot runs from its own clone on the NUC. I push to the same main from my PC, which is what the pull --rebase is for. One writable checkout per job, one lock per checkout.

The window where a kill duplicates a line

Handling one Telegram update is three durable steps: append the line, commit, then mark the update as seen and advance the offset. Kill the process after the append and before the offset write, and Telegram redelivers the update on the next start. The bot writes the same line again.

On a deploy, that kill is not an accident. It's the default.

The drain

The handler does one thing:

def request_stop(self, signum=None, frame=None) -> None:  # signal handler: set a flag and nothing else
    self.stopping = True
 
for sig in (signal.SIGTERM, signal.SIGINT):
    signal.signal(sig, self.request_stop)

The poll loop checks the flag at every boundary. After a long poll returns, it starts no update if the flag is set. After each update is fully done (appended, committed, offset written), it stops before the next one. If Telegram handed over several updates in one poll and SIGTERM arrived during the first, the rest are simply not acknowledged, and Telegram redelivers them to the new container. Nothing gets written twice.

The long poll is 25 seconds, so the deploy runs docker stop -t 30. That is room for one poll plus one capture.

Why the handler matters more than usual

On a normal Linux process, SIGTERM with no handler kills it. PID 1 is special: the kernel gives it no default action for SIGTERM. The bot is python3 scripts/brainbot/bot.py as the container command, so it is PID 1, and without a handler the signal just vanishes.

The deploy that introduced the drain measured exactly that. The old image had no handler, so docker stop -t 30 waited the full grace period and SIGKILLed it. Downtime was 36 seconds. Later deploys drain in one poll or less.

You can check whether a running container caught the signal without reading its code:

docker exec BrainBot grep SigCgt /proc/1/status

SigCgt is the caught-signal bitmask. SIGTERM is signal 15 and SIGINT is signal 2, so a mask ending in 4002 means both handlers are in.

The lock wait before the stop

Before the stop, the deploy runs:

docker exec BrainBot flock -w 120 /repo/scripts/brainbot/state/git.lock true

That returns only once nobody holds the git lock, so the stop doesn't arrive in the middle of a commit that has already started. If it times out after 120 seconds, the deploy aborts. A bot that holds the lock that long is stuck, and I want to read its logs before killing it.

The flock isn't airtight on its own: the bot can pick up a new update between flock returning and docker stop. The drain covers that gap, since SIGTERM lets the update in flight finish.

The 10-second stop I had to route around

Unraid recreates a container through its own helper, and that helper stops the container first with a timeout fixed at 10 seconds (DockerClient::stopContainer defaults DOCKER_TIMEOUT to 10). That's too short for a capture plus a push. So the deploy stops the bot itself with the longer grace, and then calls the helper, which finds the container already stopped, skips its own stop, and starts the new one.

Repairing a killed git

A drain only helps when the process gets SIGTERM. A crash, an OOM kill at the 256 MB limit, or a stop that hits the grace period still kills it outright. If that happens during git commit or git pull --rebase, /repo/.git keeps an index.lock or a half-finished rebase, and every later commit fails until someone logs in.

So recover_repo runs at startup:

for name in ("rebase-merge", "rebase-apply"):
    if (git_dir / name).exists():
        _git(repo, "rebase", "--abort")
lock = git_dir / "index.lock"
if lock.exists():
    lock.unlink()

(Trimmed: it also logs one commit: ... line per repair and raises if the lock can't be removed.)

Deleting index.lock is normally the thing you're told never to do. Here it's safe because of where it runs. At startup the container has exactly one process, so any lock is stale. The bot also calls it inside the commit path, under the flock, which already excludes every other writer on that clone.

Fencing the bot in on the server

The bot has a Forgejo account with write access, and its token sits on the NUC. I didn't want that token to be able to rewrite my agent config, the private folders, or the homelab ops folder. There's a server-side pre-receive hook keyed on GITEA_PUSHER_NAME. When the pusher is the bot:

• any ref other than refs/heads/main is rejected, so no new branches or tags
• a zero new SHA is rejected, so no deletions
• the changed paths come from git diff --name-only old new, and any path under a blocked folder is rejected with the path in the message

I ran it against a 10-case matrix on a throwaway repository before installing it on the real one.

Forgejo's branch protection has protected_file_patterns, which would have been less code. It blocks every direct push that touches those paths, mine included, so every config change would have to go through a PR. The hook applies its rules only when the bot is the pusher.

The hook limits writes, not reads, and I accepted that trade-off. It lives with the same trust as every other credential on that box.

Then main got force push protection, for everyone. A rejected force push says branch main is protected from force push. With an append-only diary and a bot that rebases onto whatever is there, rewriting history is the mistake that would hurt most. A mistake gets fixed with a new commit.

Voice notes: small archive, full-quality transcription

Audio committed to git stays in the history for good, so the archive copy is re-encoded to mono Opus at 32 kbps, about 5x smaller than what Telegram sends. The transcriber never sees that copy. The untouched download waits in state/audio-orig/, speech to text runs on it, and it's deleted once the clip is transcribed. If ffmpeg is missing, or the transcode fails or comes out bigger, the original bytes are archived instead.

Transcription is a chain, tried in order:

1. Azure's fast transcription REST endpoint with MAI-Transcribe 1.5, called directly
2. gpt-transcribe through my LiteLLM gateway
3. Whisper through LiteLLM, last

A 20-clip bake-off between the first two, on my own voice notes in mixed Arabic and English, set the order. STT_PRIMARY in the env swaps the top two. Every provider gets the same phrase list from a text file in the repo, and the bot reads that file from the mounted clone, so a new name works without an image rebuild.

One Azure gotcha from that setup: MAI-Transcribe runs in enhanced mode, and enhanced mode is served only from a short list of regions. The same call against my eastus2 resources returned 400 Enhanced mode with model is currently not supported yet. The resource itself has to live in a supported region, eastus in my case. Plain fast transcription works everywhere.

Failed clips go on a JSONL queue with the states queued, claimed, done, failed, dead: at most 5 attempts, waiting 10 minutes times the attempt count, and a claim goes stale after 15 minutes. A dead clip keeps its original in audio-orig/ for a manual retry. And because every clip is archived for good, switching models later is a re-run, not a loss.