30.08 ms and 10.31 ms on the same tree: measuring is harder than fixing

In a round where the code was measurably unchanged, three first-byte pools were run against the same tree and the only variable was machine load. At a load average of 25.55 the pooled median came out at 22.35 ms with raw reads from 5.23 to 67.09; with load at 23 and falling, 30.08 ms with raw reads from 5.92 to 343.20, one sub-run alone giving a median of 89.28; and once the machine settled to a load of 3.6 to 4.8, 10.31 ms with raw reads from 6.93 to 24.58. The rule came out of that: never compare single-run numbers with each other, read a pooled median against a raw range. The same noise turned a gate red once, but what fell was a timeout rather than an assertion, and the suite that failed at load 42.9 passed 2359 of 2359 in 24.80 seconds instead of 293.87 once the machine had settled to 11.3.

Tools lie too

The second difficulty is that tools lie, and three cases were measured. pnpm --filter web test --run, and the same command given --exclude, silently run nothing and exit 0, which reads as a green suite; in the other direction, pnpm vitest run with two filters reports that it found no test files and exits 1, which reads as red. The conclusion is that in mutation rounds you read the passing-test line in the log, never the exit code. The second case is the shell itself: putting a command substitution inside an echo line resets the last exit code, because the substitution runs first, and every exit code of one round was written down wrongly for that reason, including a probe that genuinely exited 1 and was recorded as 0. The third is bash 3.2 on macOS, which has no associative arrays; the mutation runner that used them had a trap that restored nothing while printing that it had rolled back, mutations piled up in the tree and contaminated one attribution, and what caught it was a git status reading rather than a test.