How to Deal With Ratio Metrics When Accounting for Intra-User Correlation in A/B Testing

We consider the A/B testing problem at the presence of correlation among observations coming from the same user. Furthermore, users may come from various segments where levels of correlation are different. A uniformly minimum-variance unbiased estimator of the population mean, called correlation-adjusted mean, is proposed to account for such correlation structure. It is proved theoretically and numerically better than the usual naive mean estimator and normalized mean estimator (taking average within users and then across users). The correlation-adjusted mean method is still unbiased but has reduced variance so it gains additional power. Several simulation studies are designed to show the estimation accuracy of the correlation structure, effectiveness in reducing variance, and capability of obtaining more power. An application to the eBay data is conducted to conclude this paper.