# Calculate a 95% confidence interval for relative difference in conversion rates

Add uncertainty estimates to A/B test results for better informed decisions.

## Calculate the Z score (Z) corresponding to the confidence level as the Inverse Probability Density function of the confidence level.

If the confidence level is expressed as a percentage, convert it to a proportion first by dividing it by 100. For example, a 95% confidence level would become 0.95.

Use a tool like Microsoft Excel, R, or the GIGA online z-score calculator to calculate the inverse probability function (also known as a quantile function):

• Use the `NORM.S.INV()` function in Excel to calculate Z.
• Use the `qnorm()` function in R to calculate Z.
• Use the GIGA online z-score calculator to calculate Z from Probability.

To make sure the calculation is correct, check it using this reference: for a confidence level of 0.95 the Z score would be 1.644854.

## Calculate the relative difference between the conversion rates Pc and Pv, Deltarel by using the equation `Deltarel = ( Pv - Pc ) / Pc`

For example, if Pc = 0.10 (10%) and Pv = 0.12 (12%), `Deltarel> = ( 0.12 - 0.10 ) / 0.10 = 0.02 / 0.10 = 0.2`. That represents a lift of `0.2 * 100 = 20%` in the variant versus the control.

## Calculate the standard deviations of the conversion rate in each group SDc and SDv by using the formula `SD = SQRT( P * (1 - P) / N )`

SQRT is the square root function available in most programming languages. It can be calculated using the `sqrt()` function in Microsoft Excel, R, or other similar software.

Continuing with the example in which Pc = 0.10 and Pv = 0.12, and also adding the sample sizes Nc = 10010 and Nv = 10050, we get `SDc = SQRT( 0.10 * (1 - 0.10) / 10010 ) = SQRT( 0.10 * 0.90 / 10010 ) = SQRT( 0.09 / 10010 ) = 0.002998` and similarly `SDv = SQRT( 0.12 * (1 - 0.12) / 10050 ) = SQRT( 0.12 * 0.88 / 10050 ) = SQRT( 0.1056 / 10050 ) = 0.003241`.

## Calculate the coefficients of variation in each group CVc and CVv by dividing its standard deviation by its conversion rate using the equation `CV = SD / P`.

Continuing with the previous example, we have `CVc = SDc / Pc = 0.002998 / 0.10 = 0.02998` and `CVv = SDv / Pv = 0.003241 / 0.12 = 0.027`.

## Calculate the lower confidence interval bound by solving `CIlower = (Deltarel + 1) * ((1 - Z * SQRT(CVc2 + CVv2 - Z2 * CVc2 * CVv2)) / (1 - Z * CVc2)) - 1`.

While it may seem complicated, it is a rather straightforward calculation. Taking the values previously calculated and following the same example, we get:

```CIlower = (0.2 + 1) * (( 1 - 1.644854 * SQRT(0.029982 + 0.0272 - 1.6448542 * 0.029982 * 0.0272)) / (1 - 1.644854 * 0.029982 )) - 1 = 1.2 * (( 1 - 1.644854 * SQRT( 0.0016287 - 0.00000177 )) / ( 1 - 0.00148 )) - 1 = 1.2 * (( 1 - 1.644854 * 0.040 ) / 0.99852 ) - 1 = 1.2 * (( 1 - 0.06579416 ) / 0.99852 ) - 1 = 1.2 * ( 0.93420584 / 0.99852 ) - 1 = 1.2 * 0.94 - 1 = 0.12270```

To present the result as a percentage lift, multiply by 100: `0.12270 * 100 = 12.27%` lift. Therefore, the lower bound of the 95% one-sided confidence interval is 12.27%. Effect sizes lower than that number can be rejected with 95% confidence.

To find the upper bound, replace `1 - Z` in the above equation with `1 + Z`, and keep the rest the same.