Let git find the regression: git bisect run

Last month an integration test started timing out on main. It hadn’t been touched in a year, it passed on the release tag from three weeks earlier, and between the two there were 340 commits from eleven people. Reading the log was hopeless. git bisect run found the culprit in nine steps while I made coffee.

I use bisect maybe four times a year, which is just rarely enough to forget the details every time. This is the note I keep for myself.

The manual version#

Bisect is a binary search over commits. You tell git one commit that’s bad and one that’s good, it checks out a commit halfway between, and you tell it whether that one is good or bad:

git bisect start
git bisect bad HEAD
git bisect good v2.7.0
# git checks out a commit in the middle; test it, then:
git bisect good    # or: git bisect bad
# ...repeat until git prints "<sha> is the first bad commit"
git bisect reset

A few things worth knowing:

• git bisect start HEAD v2.7.0 is a shorthand for the first three lines: bad first, then good.
• With 340 commits you need about log2(340) ≈ 9 steps, and that stays small even for huge ranges; a thousand commits is only ten.

Automating it#

Nine rounds of “build, run test, type good or bad” is tedious and error-prone. If you can write a script that exits 0 when the commit is good and non-zero when it’s bad, git does all of it:

git bisect start HEAD v2.7.0
git bisect run ./check.sh

The exit code contract#

The script#

My script for that timeout was this:

#!/usr/bin/env bash
# check.sh: good if the import test finishes within 60 seconds.
set -u

make build >/dev/null 2>&1 || exit 125   # doesn't compile: skip, don't blame

timeout 60 go test ./internal/importer -run TestLargeImport -count=1 >/dev/null 2>&1
status=$?

if (( status == 124 )); then
  exit 1    # timed out: the regression we're looking for
fi
exit "$status"

Two things in there are deliberate:

• Build failures exit 125. In a long history there’s always a commit where the build was broken for an hour. If the script returns 1 for that, bisect thinks it found the bug and blames the wrong commit. With 125 it moves to a neighbouring commit instead.
• The exit code is mapped carefully. timeout returns 124 when it kills the command. And if anything were killed by a signal, the shell would report 128 + signal number, which bisect treats as “abort”. Mapping explicitly avoids both surprises.

Keep the script outside the tree#

Bisect checks out old commits, so a script that lives in the repo might not exist at those commits, or might exist in an older, different version. I copy it out first:

cp check.sh /tmp/check.sh
git bisect run /tmp/check.sh

Same for test data or fixtures your check depends on.

Useful extras#

Skip manually#

If you know a range of commits can’t be tested (a half-finished migration, say), git bisect skip <sha> or git bisect skip v2.7.1..v2.7.3 excludes them up front.

Merge-heavy histories#

On a repo where every feature lands as a merge commit, git bisect start --first-parent only steps through the merges on main. You find the merged PR that introduced the problem, then bisect inside that branch if needed. Needs git 2.29 or newer.

Words other than good and bad#

When you’re hunting for the commit that fixed something, “good” and “bad” get confusing fast. Custom terms help:

git bisect start --term-old=broken --term-new=fixed
git bisect fixed HEAD
git bisect broken v2.6.0

Log and replay#

git bisect log > bisect.log saves your decisions. If you made a wrong call halfway through, edit the log and git bisect replay bisect.log restarts from the corrected state instead of from scratch.

When bisect lies#

Bisect assumes the bug appears once and stays. A flaky test breaks that assumption: one unlucky run marks a good commit as bad and the search goes off in the wrong direction. For flaky failures I make the check run the test several times and only call the commit good if every run passes:

for i in 1 2 3 4 5; do
  go test ./internal/importer -run TestLargeImport -count=1 >/dev/null 2>&1 || exit 1
done
exit 0

It’s slower, but nine slow steps still beat an afternoon of reading diffs. In my case the culprit was a one-line change that switched a buffered writer to an unbuffered one. The diff looked harmless, and I would never have found it by reading.