A framework by Ronny Kohavi
Ronny Kohavi's Sample Ratio Mismatch Check for Product Teams
A sample ratio mismatch is a gap between the traffic split you designed an A/B test to get and the split it actually got. Ronny Kohavi worked on experimentation at Amazon, Microsoft and Airbnb, and he calls it by far the most common way an experiment breaks. Check the split before you read a single metric. A 50/50 test that comes back 50.2/49.8 on a million users is broken, however good the result looks.
- Best for
- Product teams reading A/B test results who need to know the traffic split is sound first
- Price
- Free to read. To apply it to your own situation, Gavel has a free plan; Plus is $19/mo.