I've been previously using tools such as Google Website Optimizer to run multi-variation or A/B tests. However right now I am looking for a solution that works for a larger site (400-500 000 unique visitor per month) with a very locked down source code environment. Basically:
This means that I am not allowed/it's hard to add "new code" (even if it's tested and verified) via Google Website Optimizer or any other of the GUI-paste-your-new-variation-here type of solutions.
We can however on server side decide which users gets which variation. Basically we can push the new version on X of the servers making 10-30% of the users view it for their entire session. The question is: Which tools do we use to measure "success" (i.e improved conversion rate). My idea so far has been:
Tag the new version in Google Analytics using a session variable (and then make reports based on segment) (similar to what is described on http://searchengineland.com/how-to-analyze-ab-tests-using-google-analytics-67404 )
Use Optimizely which has API support:
window.optimizely = window.optimizely || [];
window.optimizely.push(['bucketUser', EXPERIMENT_ID, VARIATION_ID])
What solutions have you tried for locked-down environments? Am I missing some obvious solution?
The site is in .NET/Episerver on IIS.
Regards, Niklas
We ended up going with Google Analytics and adding a session variable such as "abtest" with value "variation-4" and publishing it on certain nodes. It worked fairly well, with some limitations, namely that google analytics funnels doesn't have segment support.