Modern Statistical Challenges in A/B Tests and Recent Work in Metric Decomposition
Author
Prof. Nathaniel Stevens
Co-author
-
Luke Hagar
-
Alex Deng
-
Tatiana Xifara
-
Amit Gandhi
-
-
-
-
Conference
65th ISI World Statistics Congress 2025
Format: IPS Abstract - WSC 2025
Keywords: causal treatment effect, experiment
Session: IPS 739 - Statistics in the Knowledge Economy
Tuesday 7 October 10:50 a.m. - 12:30 p.m. (Europe/Amsterdam)
Abstract
The rise of internet-based services and products has brought about an unprecedented opportunity for online businesses to engage in large scale data-driven decision making. Over the past two decades, tech organizations have invested tremendous resources in online controlled experiments (colloquially referred to as A/B tests) to assess the impact of innovation on their customers and businesses. Running these experiments at scale has presented a host of novel statistical challenges and hence research opportunities. In this talk we review some of these challenges and present new work on one problem in particular, arising from an academia-industry collaboration with Airbnb. In particular, we’ll look at a new direction for sensitivity improvement whereby a target metric of interest is decomposed into components with high signal-to-noise disparity. Through both frequentist and Bayesian theory as well as real world applications, we’ll demonstrate the agility metric decomposition yields relative to an un-decomposed analysis.