rstatisticsab-testingsignificanceabtest

power.prop.test function not interchangeable


I am using the power.prop.test function in R.

I am doing an A/B Test where I am determining the lift from a minimum amount of impressions per group in order for the A/B Test to be significant.

When I run the function like below, I get the second proportion (p2) to be 0.0001870215 with n being 2,571,429 per group:

original_conversion_rate<-0.00009

power.prop.test(n=2571429,
  p1=original_conversion_rate, 
            power=0.8, 
            sig.level=0.05)

My answer is this:

     Two-sample comparison of proportions power calculation 

          n = 2571429
         p1 = 9e-05
         p2 = 0.0001870215
  sig.level = 0.05
      power = 0.8
alternative = two.sided

NOTE: n is number in *each* group

When I rerun this with my answer for p2 (0.0001870215) to solve for n, a different n comes up (230,952.6):

original_conversion_rate<-0.00009

power.prop.test(
  p1=original_conversion_rate, 
  p2=0.0001870215,
            power=0.8, 
            sig.level=0.05)

My n changes to this:

 Two-sample comparison of proportions power calculation 

          n = 230952.6
         p1 = 9e-05
         p2 = 0.0001870215
  sig.level = 0.05
      power = 0.8
alternative = two.sided

Why would n change in this case?


Solution

  • Your frequencies are so small that miniscule changes will have a huge impact - especially with the default tolerance level used in power.prop.test.

    If you take one iteration more you get

    power.prop.test(n=230952.5,
      p1=original_conversion_rate, 
                power=0.8, 
                sig.level=0.05)
    
         Two-sample comparison of proportions power calculation 
    
                  n = 230952.5
                 p1 = 9e-05
                 p2 = 0.0001870215
          sig.level = 0.05
              power = 0.8
        alternative = two.sided
    
    NOTE: n is number in *each* group
    

    which is the same value for p2 as before. Now, if we go back to your first calculation and lower the tolerance we get

    power.prop.test(n=2571429,
      p1=original_conversion_rate, 
                power=0.8, 
                sig.level=0.05, tol=.Machine$double.eps^.8) 
    
         Two-sample comparison of proportions power calculation 
    
                  n = 2571429
                 p1 = 9e-05
                 p2 = 0.0001150142
          sig.level = 0.05
              power = 0.8
        alternative = two.sided
    
    NOTE: n is number in *each* group
    

    You can see that p2 changed. If we insert the updated value for p2 and solve for n we get

    power.prop.test(
      p1=original_conversion_rate, 
      p2=0.0001150142,
                power=0.8, 
                sig.level=0.05)
    
         Two-sample comparison of proportions power calculation 
    
                  n = 2571429
                 p1 = 9e-05
                 p2 = 0.0001150142
          sig.level = 0.05
              power = 0.8
        alternative = two.sided
    
    NOTE: n is number in *each* group
    

    and you get the same n.