Running an AI Code Reviewer in Production

A demo of an AI code reviewer takes an afternoon: send the diff to a model, post the answer as a comment. Running one on every pull request of a team, every day, is a different problem. The model call is the easy part. The hard parts are cost, long reviews that drop halfway, comments about code that does not exist, and a team that stops reading the bot. I run a reviewer as a GitHub App in production. This is how each of those parts works.

The path of one review

1. 1. Pull request event
   

   Opened, or new commits pushed

2. 2. Quota gate
   

   Skip visibly when over budget

3. 3. Choose the mode
   

   Incremental since the last review, or full

4. 4. Review in chunks
   

   Each finished chunk is saved, so a drop resumes

5. 5. Verify claims
   

   Drop findings that cite code not in the diff

A quota gate before the model

A large pull request can cost more than a day of small ones. Without a limit, one generated file or one big refactor uses the budget of the week. The gate runs before any model call and estimates the size of the review:

type Gate = { allowed: true } | { allowed: false; reason: string };
 
async function quotaGate(repo: string, estimatedTokens: number): Promise<Gate> {
  const used = await usage.sumSince(repo, startOfDayUtc());
  const budget = dailyBudget(repo);
  if (used + estimatedTokens > budget) {
    return { allowed: false, reason: `daily review budget reached (${used}/${budget} tokens)` };
  }
  return { allowed: true };
}

The important part is what happens on "no". The bot never skips in silence. It posts a neutral status on the pull request with the reason. A silent skip looks exactly like a clean review, and people merge on it.

Incremental or full

Most pushes to an open pull request add one small commit. Reviewing the whole pull request again costs more, and it repeats every earlier comment. So the bot remembers the last commit it reviewed and, by default, reviews only what came after it:

type Mode = { kind: "full" } | { kind: "incremental"; fromSha: string };
 
async function chooseMode(pr: PullRequest, lastReviewedSha?: string): Promise<Mode> {
  if (!lastReviewedSha) return { kind: "full" };
  if (pr.labels.includes("review:full")) return { kind: "full" };
  // A force-push or a rebase rewrites history. The old commit is no longer
  // an ancestor of the head, and an incremental diff would be meaningless.
  if (!(await isAncestor(lastReviewedSha, pr.headSha))) return { kind: "full" };
  return { kind: "incremental", fromSha: lastReviewedSha };
}

The ancestor check is the detail that matters. Without it, a rebase produces an incremental diff between two unrelated commits, and the bot reviews noise.

Resume after a dropped connection

A full review of a large pull request is a long request, and long requests drop: a timeout, a reset connection, a restarted worker. Starting again from zero doubles the cost and can fail the same way.

The bot splits a review into chunks, by file or by group of small files. Each finished chunk is saved under the head commit of the pull request. A retry skips every chunk that is already done:

async function reviewInChunks(pr: PullRequest, chunks: Chunk[]): Promise<Finding[]> {
  const findings: Finding[] = [];
  for (const chunk of chunks) {
    const key = `${pr.id}:${pr.headSha}:${chunk.id}`;
    const saved = await store.get(key);
    if (saved) {
      findings.push(...saved);
      continue;
    }
    const result = await withRetry(() => reviewChunk(chunk)); // retries only this chunk
    await store.set(key, result);
    findings.push(...result);
  }
  return findings;
}

The key includes the head commit on purpose. A new push makes new keys, so a resumed review never mixes findings from two versions of the code.

Blocking false claims

A model will say, with confidence, that line 42 dereferences a null value, when line 42 is a comment, or when the file is not in the pull request. One such comment costs more trust than ten good ones earn. So every finding must point at code, and the bot checks the pointer before it posts:

type Finding = {
  path: string;
  line: number;
  quote: string; // the exact code the finding is about
  severity: "blocker" | "major" | "minor";
  body: string;
};
 
function isGrounded(f: Finding, diff: ParsedDiff): boolean {
  const file = diff.files.get(f.path);
  if (!file) return false; // the file is not in this pull request
  const text = file.addedOrContextLines.get(f.line);
  if (text === undefined) return false; // the line is not in the diff
  return normalize(text).includes(normalize(f.quote)); // the quoted code exists
}

Findings that fail the check are dropped, not posted, and counted. The drop rate is a health metric. When it rises, the prompt or the chunking needs work.

This check catches the claims that are false about the code's text. It cannot catch a wrong opinion about correct code. That is what the severity bar is for.

Fixing alert fatigue

The first version was too loud, and people stopped reading it. A reviewer that nobody reads is worse than none, because it looks like coverage. I retuned it around one rule: a comment must be worth the time of the person who reads it.

• Only blocker and major findings become inline comments. Minor ones go into one summary comment.
• A finding that was already posted on an earlier commit is not posted again. Each finding has a fingerprint from its path and the quoted code.
• Style is out of scope. Linters and formatters own style, and they are faster and never wrong about it.
• The bot comments. It never approves and never blocks a merge. People do that.

Alternatives I rejected

• A full review on every push. It costs the most, and it repeats the same comments on every commit.
• Trust the line numbers from the model. Comments land on the wrong lines, or on code outside the pull request.
• Skip in silence when over budget. It looks like approval.
• Let the bot block merges. A tool that can be wrong must not hold the merge button. A person reads the finding and decides.

What I would do differently

I would build the claim check and the metrics before the first comment reached the team. The drop rate and the rate of findings that led to a code change are the only honest measures of a reviewer. They show fatigue coming before the team feels it.

I would also start with a high severity bar and lower it over time, not the reverse. A team forgives a quiet bot that is always right. It does not forgive a loud one, even after it improves.