Why an Idle Postgres Connection Crashed My Node.js Service
node-postgres emits 'error' when the database drops an idle socket. With no listener, Node.js exits. How I found it and made the pool survive.
Engineering. Updated . 4 min read.
Once a week, give or take, one of my services threw a burst of 502s. No deploy, no traffic spike, nothing in the request logs. The process was just dying. The culprit turned out to be one missing line: an error listener on the Postgres pool.
TL;DR: node-postgres (pg) emits error when the database or the network drops a socket. Node treats an unhandled error event as an uncaught exception and exits. You need a listener on the pool and on every client, plus TCP keepalive.
How it dies
The pool keeps a few connections warm between requests (that's the min setting). At some point the database side, in my case a managed RDS instance, closes one of those idle sockets. pg dutifully emits error on the pool for that client. Nobody's listening, so Node throws, and the whole process goes down with every request it was holding.
Nothing in the application triggered it, which is why it looked random. The trigger was on the other end of the wire.
The path most fixes miss
Search for this and you'll find pool.on('error'). That's half the fix. It only covers idle clients.
A client that's checked out, say in the middle of a transaction, emits error on itself, not on the pool. Neither pg-pool nor Kysely, the query builder I use, attaches a listener there. So a socket drop mid-transaction kills the process exactly the same way, and the "fixed" service still crashes, just less often.
The fix
import { Pool } from "pg";
const pool = new Pool({
...config,
keepAlive: true,
// The `min` clients never reach the idle timeout, so probes are what
// notice a peer that dropped the socket without a reset.
keepAliveInitialDelayMillis: 30_000,
});
// An idle drop reaches the client listener and then the pool listener
// with the same error object; report it once.
const reported = new WeakSet<Error>();
const report = (error: Error) => {
if (reported.has(error)) return;
reported.add(error);
logger.error("PG client error", error, { host: config.host });
errorTracker.capture(error);
};
pool.on("error", report); // idle clients
pool.on("connect", (client) => client.on("error", report)); // checked-out clientsA few things about this that aren't obvious:
- It doesn't swallow the failure. The query on the dead socket still rejects, and the caller handles it like any other database error.
pghas already evicted the broken client, so the next query gets a fresh connection. The listener only stops the process from exiting. - An idle drop hits both listeners with the same error object. The
WeakSetdedupes the report without holding on to memory. - Every entry point needs a logger. A migration CLI reused the same connection code but never set one up, so the listener itself would have thrown. Fixing a crash by adding a new crash is a special kind of embarrassing.
Testing it without waiting a week
You don't need production to drop a socket for you. The tests emit the event directly and check that a pool error doesn't throw and gets reported, that a checked-out client error doesn't throw either, that a drop reaching both listeners is reported once, that the CLI path installs its logger, and that keepalive is on.
The fix that didn't ship
The PR merged. About ten minutes later, a revert of it landed. The release branch was cut after both, so the release went out without the fix, and I had to apply it again on the release branch.
Merged is not deployed. Before I call a production bug fixed now, I check the release branch or the running version.
Questions
- Why does an idle Postgres connection crash a Node.js process?
- node-postgres emits an 'error' event when the server or the network drops a client's socket. In Node.js, an 'error' event with no listener is thrown as an uncaught exception, and the process exits. The pool emits it for idle clients, and the client emits it for checked-out clients.
- Is pool.on('error') enough?
- No. The pool listener covers idle clients only. A client that is checked out, for example inside a transaction, emits 'error' on itself. Attach a listener to every client in the pool's 'connect' event as well.
- Does the failed query still reject after I add the listener?
- Yes. The listener only stops the process from exiting. The query that used the dead socket still rejects, and node-postgres removes the broken client from the pool, so the next query gets a new connection.
- Why enable TCP keepalive on the pool?
- The clients kept open by the pool's min setting never reach the idle timeout. A peer that drops the socket without a reset is only noticed when the next query fails. Keepalive probes find the dead socket earlier.