64th ISI World Statistics Congress

64th ISI World Statistics Congress

Download PDF

An Overview of Statistical Challenges in Online Controlled Experiments

Author

S
Prof. Nathaniel Stevens

Co-author

Conference

64th ISI World Statistics Congress

Format: IPS Abstract

Keywords: experimental-design

Session: IPS 78 - Statistical Challenges in Computational Advertising

Monday 17 July 4 p.m. - 5:25 p.m. (Canada/Eastern)

Abstract

The rise of internet-based services and products in the late 1990's brought about an unprecedented opportunity for online businesses to engage in large scale data-driven decision making. Over the past two decades, organizations such as Airbnb, Alibaba, Amazon, Baidu, Booking, Alphabet's Google, LinkedIn, Lyft, Meta's Facebook, Microsoft, Netflix, Twitter, Uber, and Yandex have invested tremendous resources in online controlled experiments (OCEs) to assess the impact of innovation on their customers and businesses. Running OCEs at scale has presented a host of challenges requiring solutions from many domains. In this talk we discuss the practice and culture of online experimentation, and we review practical challenges that require new statistical methodologies to address them. The goal is to raise academic statisticians' awareness of these new research opportunities so as to increase collaboration between academia and the online industry.

    •