Ronny Kohavi's Sample Ratio Mismatch Check: Test the Split Before the Result

The framework

A split that is off by 0.2% is a red flag

The gap does not have to look big. With a million users, 50.2 against 49.8 sounds like noise. Put the counts into the formula and it says a split that uneven should turn up about once in half a million experiments. Kohavi keeps a spreadsheet that does the sum. In a 2016 talk he read out an alert from that week: 821,588 users in control and 815,582 in treatment. The ratio was 50.2%, and the odds of that by chance were one in 500,000.

This is also one of the few places where a p-value, the chance of seeing a result if nothing real is going on, means what people think it means. The design fixes the split at 50/50. So a tiny p-value really is the chance of seeing that split by luck.

The split matters because the traffic you lose is usually not a random slice. Often it is the people who viewed zero pages, or the heavy clickers who got filtered out as bots. Kohavi, on the first mistake his team made:

A failed split makes results look either super good or super bad. The bad ones kill ideas that might have worked. The good ones get celebrated. So the check comes first. If it fails, the metrics are not evidence of anything.

How to apply it

How do you check an A/B test for sample ratio mismatch?

The first three steps run on every test. The last three make the check stick.

1. 1

   Write down the split before launch.

   The check compares what arrived against what you designed, so the design has to exist first. Kohavi's examples run from 50/50 to 20/20, and the same test works for unequal splits.

2. 2

   Run the numbers on the split.

   Compare the users in each variant against the designed ratio with a goodness-of-fit test. Chi-squared is the standard one. If the probability of that split is tiny, stop.

3. 3

   Hide the scorecard when the check fails.

   Microsoft first showed a warning banner, and people presented the results anyway. The team blanked the scorecard behind an OK button. Then it marked every number with a red line, so a screenshot still carried the warning.

4. 4

   Look in the data pipeline before the randomizer.

   The hash almost always works. What breaks is a step that removes data unevenly. Kohavi names bot filtering, fraud removal and bad-traffic rules that hit one variant harder. A marketing campaign that sends its visitors into only one variant does the same.

5. 5

   Run A/A tests as well.

   Split traffic between two identical versions. At a 0.05 threshold, the system should find no significant difference about 95% of the time. Kohavi calls this one of the most useful tools for finding bugs in the software, the method and the variance estimates.

6. 6

   Count your past invalid results.

   When vendors added the check, they found about 8% of the experiments they had already reported to users were invalid. Teams running it for the first time often find more.

Boundary conditions

When does the SRM check mislead you?

Kohavi tells the banner story in two of his recordings. Telling people not to trust a result did not stop them presenting it:

Passing the check leaves every other bug in place. Right after SRM on Lenny's Podcast, Kohavi moved to Twyman's law. Say your experiments normally move a metric by under 1%, and one suddenly moves it by 10%. Hold the celebration dinner and investigate. His Overall Evaluation Criterion covers the other half: deciding what the test should optimize before it runs.

Sources

Where Kohavi discusses this

Of the seven long Kohavi recordings we pulled for this page on 2026-09-30, five raise the sample ratio check, from a 2016 conference talk to a 2024 lesson.

Where experts disagree

Where operators disagree: check every split, or ship the winners fast?

The deciding variable is whether anything outside the test can confirm the win. McEvoy had that check, because monthly revenue nearly quadrupled and a broken split cannot fake revenue. A team shipping small lifts it can see only in the scorecard has no outside check. At the rates Kohavi reports, about one test in twelve has a broken split, so the SRM check is what stands between that team and a roadmap built on noise.