rstatisticsalgebraab-testingabtest

Z Test of Proportions - Solve for minimum proportion to reach significance


The following equation provides the test statistic used in the Test of Proportions.

For a proposed a/b test, I am attempting to show the minimum value needed for the treated group (p2) to show a statistical significance at a 95% confidence level. In other words, I am trying to solve this equation for p2. Given that I know my total population size, the percent that will be treated, and Z-value, this would seem to be straightforward. However, I am getting stuck on the algebra.

I have written an R script that will run through a range of values for p2 until a p-value of a given confidence is met, but that is a sloppy way of solving the problem.


Solution

  • I wouldn't bother doing this algebraically (or however that's spelled).

    Notice that if

    Z = diff(p) / se(p)

    then

    0 = diff(p) / se(p) - Z

    The uniroot function can then do the work for you. you provide the values of everything but p2, and uniroot will seek out the value that resolves to 0.

    zdiff <- function(p2, p1, n1, n2, alpha = 0.025) 
    {
      ((p1 - p2) - 0) / sqrt(p1 * (1-p1) / n1 + p2 * (1-p2) / n2) - qnorm(alpha, lower.tail = FALSE)
    }
    
    uniroot(f = zdiff,
            p1 = .5, n1 = 50, n2= 50, 
            interval = c(0, 1))
    
    $root
    [1] 0.311125
    
    $f.root
    [1] -1.546283e-06
    
    $iter
    [1] 5
    
    $init.it
    [1] NA
    
    $estim.prec
    [1] 6.103516e-05
    

    So with equal sample sizes of 50 and p1 = .5, p2 would have to be less than 0.311125 to generate a statistically significant result at the two-sides alpha = 0.05 level.