Ex-Airbnb, Microsoft and Amazon, author of Trustworthy Online Controlled Experiments
Ronny Kohavi wrote Trustworthy Online Controlled Experiments. He led the search experience team at Airbnb, Microsoft's experimentation platform team before that, and data mining and personalization at Amazon. He wants every code change in an experiment, since even small bug fixes can have surprising impact. Most ideas fail, in his experience: about two-thirds at Microsoft and 92% at Airbnb.
Idea validation
Research
Decide what an A/B test optimizes for before it runs, and pair it with a countervailing metric, or the experiment will faithfully optimize the wrong thing.
you have to define the OEC such that it is causally predictive of the lifetime value of the user
Research
Check the traffic split before you read any metric. A 50.2/49.8 split on a million users means the test is broken.
if you get something off from 50 percent it's a red flag
Research
Pick the smallest lift worth catching, 5% or less, and size the A/B test for it before launch. A win from a low-power test is usually an exaggerated one.
the statistical power of your test is the probability that if there is a difference say B is better by 5% that you will detect it
Ronny Kohavi
says write the success metric down before you build, ask how you would cheat it, and add the countervailing metric that catches the cheat. Any metric you can move you can also game, and an experiment with no OEC will faithfully optimise the wrong thing.
Archie Abrams (Shopify)
runs core product without traditional KPIs, on the view that intuition and taste guide the decisions that matter and a metric target quietly narrows what the team is willing to try.
Ronny Kohavi
says no result counts as evidence until the split passes the sample ratio check. Lost traffic is biased traffic, so a failed split makes results look super good or super bad. His team at Microsoft blanked the scorecard rather than let anyone read one.
John McEvoy
ran about ten paywall and onboarding experiments on his own app over two or three months. Conversion went from 0.5% to 8%, and revenue from around $8K to over $30K a month. His rule: A/B test paywalls and onboarding flows relentlessly (Starter Story, 9:12).
Ronny Kohavi
says size the test before it runs: an MDE of 5% or less at 80% power, which for a store converting at 5% means about 200,000 users. Below that a significant result is likely exaggerated, so build the culture and the platform, and test only for large effects.
tests a prototype with five target customers on the Friday of a design sprint, not with internal stakeholders. In Sprint, five customers reveal the major patterns without statistical overhead, so there is no reason to wait for a large sample.
