Ex-Airbnb, Microsoft and Amazon, author of Trustworthy Online Controlled Experiments
Ronny Kohavi wrote Trustworthy Online Controlled Experiments. He led the search experience team at Airbnb, Microsoft's experimentation platform team before that, and data mining and personalization at Amazon. He wants every code change in an experiment, since even small bug fixes can have surprising impact. Most ideas fail, in his experience: about two-thirds at Microsoft and 92% at Airbnb.
Lenny's Podcast 01:12 01:32 01:36 13:37 14:02 21:08 37:45
Research
Decide what an A/B test optimizes for before it runs, and pair it with a countervailing metric, or the experiment will faithfully optimize the wrong thing.
the OEC or the overall evaluation criteria is something that I think many people that start to dabble in A/B testing miss
Ronny Kohavi
says write the success metric down before you build, ask how you would cheat it, and add the countervailing metric that catches the cheat. Any metric you can move you can also game, and an experiment with no OEC will faithfully optimise the wrong thing.
Archie Abrams (Shopify)
runs core product without traditional KPIs, on the view that intuition and taste guide the decisions that matter and a metric target quietly narrows what the team is willing to try.
