statisticsbayesianab-testinggoogle-optimize

Is SRM in Google Optimize (Bayesian Model) a thing


So checking for Sample Ratio Mismatch is good for data quality. But in Google Optimize i can't influence the sample size or do something against it.

My problem is, out of 15 A/B Tests I only got 2 Experiment with no SRM. (Used this tool https://www.lukasvermeer.nl/srm/microsite/) In the other hand the bayesian model deals with things like different sample sizes and I dont need to worry about, but the opinions on this topic are different.

Is SRM really a problem in Google Optimize or can I ignore it?


Solution

  • SRM affects Bayesian experiments just as much as it affects Frequentist. SRM happens when you expect a certain traffic split, but end up with a different one. Google Optimize is a black box, so it's impossible to tell if the uneven sample sizes you are experiencing are intentional or not.

    Lots of things can cause a SRM, for example if your variation's javascript code has a bug in some browsers those users may not be tracked properly. Another common cause is if your variation causes page load times to increase, more people will abandon the page and you'll see a smaller sample size than expected.

    That lack of statistical rigor and transparency is one of the reasons I built Growth Book, which is an open source A/B testing platform with a Bayesian stats engine and automatic SRM checks for every experiment.