I am using the power.prop.test
function in R.
I am doing an A/B Test where I am determining the lift from a minimum amount of impressions per group in order for the A/B Test to be significant.
When I run the function like below, I get the second proportion (p2) to be 0.0001870215 with n being 2,571,429 per group:
original_conversion_rate<-0.00009
power.prop.test(n=2571429,
p1=original_conversion_rate,
power=0.8,
sig.level=0.05)
My answer is this:
Two-sample comparison of proportions power calculation
n = 2571429
p1 = 9e-05
p2 = 0.0001870215
sig.level = 0.05
power = 0.8
alternative = two.sided
NOTE: n is number in *each* group
When I rerun this with my answer for p2 (0.0001870215) to solve for n, a different n comes up (230,952.6):
original_conversion_rate<-0.00009
power.prop.test(
p1=original_conversion_rate,
p2=0.0001870215,
power=0.8,
sig.level=0.05)
My n changes to this:
Two-sample comparison of proportions power calculation
n = 230952.6
p1 = 9e-05
p2 = 0.0001870215
sig.level = 0.05
power = 0.8
alternative = two.sided
Why would n change in this case?
Your frequencies are so small that miniscule changes will have a huge impact - especially with the default tolerance level used in power.prop.test
.
If you take one iteration more you get
power.prop.test(n=230952.5,
p1=original_conversion_rate,
power=0.8,
sig.level=0.05)
Two-sample comparison of proportions power calculation
n = 230952.5
p1 = 9e-05
p2 = 0.0001870215
sig.level = 0.05
power = 0.8
alternative = two.sided
NOTE: n is number in *each* group
which is the same value for p2
as before. Now, if we go back to your first calculation and lower the tolerance we get
power.prop.test(n=2571429,
p1=original_conversion_rate,
power=0.8,
sig.level=0.05, tol=.Machine$double.eps^.8)
Two-sample comparison of proportions power calculation
n = 2571429
p1 = 9e-05
p2 = 0.0001150142
sig.level = 0.05
power = 0.8
alternative = two.sided
NOTE: n is number in *each* group
You can see that p2
changed. If we insert the updated value for p2
and solve for n
we get
power.prop.test(
p1=original_conversion_rate,
p2=0.0001150142,
power=0.8,
sig.level=0.05)
Two-sample comparison of proportions power calculation
n = 2571429
p1 = 9e-05
p2 = 0.0001150142
sig.level = 0.05
power = 0.8
alternative = two.sided
NOTE: n is number in *each* group
and you get the same n
.