PUMA publications for /user/jaeschke/keynotehttps://puma.uni-kassel.de/user/jaeschke/keynotePUMA RSS feed for /user/jaeschke/keynote2024-03-28T17:36:18+01:00Online Controlled Experiments: Introduction, Learnings, and Humbling Statisticshttps://puma.uni-kassel.de/bibtex/2aa31e13651d5d1eab42e449e55a0e745/jaeschkejaeschke2012-09-20T09:59:05+02:002012 amazon bing evaluation experiment industry keynote online recommender recsys statistics <span class="authorEditorList"><span itemtype="http://schema.org/Person" itemscope="itemscope" itemprop="author"><a title="Ron Kohavi" itemprop="url" href="/author/Ron%20Kohavi"><span itemprop="name">R. Kohavi</span></a></span>. </span>(<em><span>12.09.2012<meta content="12.09.2012" itemprop="datePublished"/></span></em>)Thu Sep 20 09:59:05 CEST 2012sepOnline Controlled Experiments: Introduction, Learnings, and Humbling StatisticsIndustry keynote at ACM Recommender Systems20122012 amazon bing evaluation experiment industry keynote online recommender recsys statistics 12The web provides an unprecedented opportunity to accelerate innovation by evaluating ideas quickly and accurately using controlled experiments (e.g., A/B tests and their generalizations). Whether for front-end user-interface changes, or backend recommendation systems and relevance algorithms, online controlled experiments are now utilized to make data-driven decisions at Amazon, Microsoft, eBay, Facebook, Google, Yahoo, Zynga, and at many other companies. While the theory of a controlled experiment is simple, and dates back to Sir Ronald A. Fisher’s experiments at the Rothamsted Agricultural Experimental Station in England in the 1920s, the deployment and mining of online controlled experiments at scale—thousands of experiments now—has taught us many lessons. We provide an introduction, share real examples, key learnings, cultural challenges, and humbling statistics.