Calculate a 95% confidence interval for relative difference in conversion rates

Step-by-step Playbooks
, , , ,
1

Business Benefits

Add uncertainty estimates to A/B test results for better informed decisions.

Calculate the Z score (Z) corresponding to the confidence level as the Inverse Probability Density function of the confidence level.

If the confidence level is expressed as a percentage, convert it to a proportion first by dividing it by 100. For example, a 95% confidence level would become 0.95.

Use a tool like Microsoft Excel, R, or the GIGA online z-score calculator to calculate the inverse probability function (also known as a quantile function):

  • Use the NORM.S.INV() function in Excel to calculate Z.
  • Use the qnorm() function in R to calculate Z.
  • Use the GIGA online z-score calculator to calculate Z from Probability.

To make sure the calculation is correct, check it using this reference: for a confidence level of 0.95 the Z score would be 1.644854.

Calculate the relative difference between the conversion rates Pc and Pv, Deltarel by using the equation Deltarel = ( Pv - Pc ) / Pc

For example, if Pc = 0.10 (10%) and Pv = 0.12 (12%), Deltarel> = ( 0.12 - 0.10 ) / 0.10 = 0.02 / 0.10 = 0.2. That represents a lift of 0.2 * 100 = 20% in the variant versus the control.

Calculate the standard deviations of the conversion rate in each group SDc and SDv by using the formula SD = SQRT( P * (1 - P) / N )

SQRT is the square root function available in most programming languages. It can be calculated using the sqrt() function in Microsoft Excel, R, or other similar software.

Continuing with the example in which Pc = 0.10 and Pv = 0.12, and also adding the sample sizes Nc = 10010 and Nv = 10050, we get SDc = SQRT( 0.10 * (1 - 0.10) / 10010 ) = SQRT( 0.10 * 0.90 / 10010 ) = SQRT( 0.09 / 10010 ) = 0.002998 and similarly SDv = SQRT( 0.12 * (1 - 0.12) / 10050 ) = SQRT( 0.12 * 0.88 / 10050 ) = SQRT( 0.1056 / 10050 ) = 0.003241.

Calculate the coefficients of variation in each group CVc and CVv by dividing its standard deviation by its conversion rate using the equation CV = SD / P.

Continuing with the previous example, we have CVc = SDc / Pc = 0.002998 / 0.10 = 0.02998 and CVv = SDv / Pv = 0.003241 / 0.12 = 0.027.

Calculate the lower confidence interval bound by solving CIlower = (Deltarel + 1) * ((1 - Z * SQRT(CVc2 + CVv2 - Z2 * CVc2 * CVv2)) / (1 - Z * CVc2)) - 1.

While it may seem complicated, it is a rather straightforward calculation. Taking the values previously calculated and following the same example, we get:

CIlower = (0.2 + 1) * (( 1 - 1.644854 * SQRT(0.029982 + 0.0272 - 1.6448542 * 0.029982 * 0.0272)) / (1 - 1.644854 * 0.029982 )) - 1
= 1.2 * (( 1 - 1.644854 * SQRT( 0.0016287 - 0.00000177 )) / ( 1 - 0.00148 )) - 1
= 1.2 * (( 1 - 1.644854 * 0.040 ) / 0.99852 ) - 1
= 1.2 * (( 1 - 0.06579416 ) / 0.99852 ) - 1
= 1.2 * ( 0.93420584 / 0.99852 ) - 1
= 1.2 * 0.94 - 1
= 0.12270

To present the result as a percentage lift, multiply by 100: 0.12270 * 100 = 12.27% lift. Therefore, the lower bound of the 95% one-sided confidence interval is 12.27%. Effect sizes lower than that number can be rejected with 95% confidence.

To find the upper bound, replace 1 - Z in the above equation with 1 + Z, and keep the rest the same.

Last edited by @hesh_fekry 2023-11-14T15:25:20Z