End-to-End Tests That Cannot Fail: Removing Fake Database Writes
An end-to-end test that writes its own setup to the database can stay green while the product is broken. How I found about 150 such writes and sorted them.
Engineering. Updated . 3 min read.
Our end-to-end suite was green, and I took that as proof the payment flows worked. Then I actually read how the tests set up their data. A lot of them wrote the state they needed straight into the database, and some then asserted the exact values they'd just written. Those tests couldn't fail. They were checking their own homework.
TL;DR: an end-to-end test proves something only when the product created the state being checked. Every direct database write in test setup is one of three things: a fake precondition, legitimate cleanup, or a tautology. Keep the cleanup, replace the fakes with real product calls, and delete the tautologies.
Three kinds of write
| Kind | What it does | What to do |
|---|---|---|
| Fake precondition | Writes a field or row the product should create | Drive the product to create it, then assert it |
| Cleanup | Deletes rows the test created | Keep it |
| Tautology | Writes a value, then asserts that same value | Delete it, or assert a real product outcome |
The fake precondition is the nasty one, because it hides exactly the bug the test exists to catch. If the product stops deriving a field, the helper keeps writing it, and every test downstream stays green.
My first count was off by half
I sized the job with a search for direct inserts and updates. That search couldn't see writes hidden inside shared helpers. One helper file exported 15 functions that write to the database, and they accounted for 66 call sites across 28 files. The real surface was about 150 write sites in about 45 files.
That miss would have bitten twice. A later check compared a source scan against an allowlist of approved writes, so any file my search missed would have been certified clean while it was still writing. The first fix wasn't to a test at all. It was to the inventory.
Of the 65 helper call sites I sorted first, 48 were fake preconditions, 10 were cleanup, and 7 were tautologies.
What they looked like in the wild
Names are generalized. The patterns are real.
- A patch with a stale comment. A helper set a flag on new records "because the upload path does not set it." The upload path had set it for ages. The helper was now just masking any future regression. Fix: delete the patch, assert the product sets the flag.
- Checking a name instead of an identity. A test confirmed a record linked to the right owner by comparing names. On one parse failure, the product created a new owner with the same name, and the test passed either way. Fix: compare IDs.
- A test with no assertion. "Should handle a missing record gracefully," with no
expectanywhere. It could not fail. Fix: delete it. - Faking the entire precondition of a destructive action. Before a deletion test, a helper cancelled open requests and forced the record into its final state, then the test asserted the fields the helper wrote. A broken guard on the deletion would never show up. This one needed a product decision first: does the deletion own that cleanup, or must it already be done? On a destructive path I'd rather block on the question than guess.
Rules I set before touching anything
- Never hide a bug. A change that keeps a test green while the product is broken is a failure, not a fix.
- A visible gap beats a green lie. If a write can't be replaced without losing evidence, it stays, and the gap gets written down by name.
- Never weaken or delete an assertion to make a change fit. Report the conflict.
- Never replace cleanup with an API call that can fail on a fake row. The easy "fix" is to swallow that error, and then you've hidden a failure again.
- Keep anything that can fail for an unrelated reason out of an
it.failsbody.
That last one is a Vitest trap. it.fails pins a known bug: the case passes when its body throws. It passes on any throw. If a setup step inside it times out, you get a green pin for a bug you never actually reproduced. Setup goes in a plain it before it.
Why I had the plans attacked
I wrote a plan for each group of test files and handed each one to a reviewer whose only job was to find what breaks. None came back unchanged. Four needed changes and one came back flat-out unsafe. Across all of them, the reviews found 21 high-risk cases where my "replacement" would have hidden a bug. Most looked perfectly clean until someone traced what the product actually does on the failure branch.
Questions
- Why is a direct database write in an end-to-end test a problem?
- The test then checks data it made itself, not data the product made. If the product stops creating that data, the write still creates it, and the test stays green while the feature is broken.
- Are all database writes in tests bad?
- No. Cleanup after a test, such as deleting the rows it created, is fine and should stay. The problem is a write that fakes a precondition the product should create, or a write whose value the test then asserts.
- How do I find every database write in a test suite?
- A search for insert and update calls is not enough, because shared helper functions hide writes behind a name. List the helpers that write, then count their call sites too. In my suite, the helpers added 66 call sites that a plain search missed.
- What is the risk with Vitest it.fails?
- it.fails passes when the test body throws for any reason. If a setup step inside it times out, the case reads as a passing known-bug pin. Keep every step that can fail for another reason outside the it.fails body.