TY - GEN AU - Kohavi, Ron A2 - T1 - Online Controlled Experiments: Introduction, Learnings, and Humbling Statistics JO - PB - C1 - PY - 2012/10 VL - IS - SP - EP - UR - http://www.exp-platform.com/Pages/2012RecSys.aspx DO - KW - 2012 KW - amazon KW - bing KW - evaluation KW - experiment KW - industry KW - keynote KW - online KW - recommender KW - recsys KW - statistics L1 - N1 - N1 - AB - The web provides an unprecedented opportunity to accelerate innovation by evaluating ideas quickly and accurately using controlled experiments (e.g., A/B tests and their generalizations). Whether for front-end user-interface changes, or backend recommendation systems and relevance algorithms, online controlled experiments are now utilized to make data-driven decisions at Amazon, Microsoft, eBay, Facebook, Google, Yahoo, Zynga, and at many other companies. While the theory of a controlled experiment is simple, and dates back to Sir Ronald A. Fisher’s experiments at the Rothamsted Agricultural Experimental Station in England in the 1920s, the deployment and mining of online controlled experiments at scale—thousands of experiments now—has taught us many lessons. We provide an introduction, share real examples, key learnings, cultural challenges, and humbling statistics. ER -