### Business Benefits

## Calculate the Z score (Z) corresponding to the confidence level as the Inverse Probability Density function of the confidence level.

If the confidence level is expressed as a percentage, convert it to a proportion first by dividing it by 100. For example, a 95% confidence level would become 0.95.

Use a tool like Microsoft Excel, R, or the GIGA online z-score calculator to calculate the inverse probability function (also known as a quantile function):

- Use the
`NORM.S.INV()`

function in Excel to calculate Z. - Use the
`qnorm()`

function in R to calculate Z. - Use the GIGA online z-score calculator to calculate Z from Probability.

To make sure the calculation is correct, check it using this reference: for a confidence level of 0.95 the Z score would be 1.644854.

##
Calculate the relative difference between the conversion rates P_{c} and P_{v}, Delta_{rel} by using the equation `Delta`_{rel} = ( P_{v} - P_{c} ) / P_{c}

_{rel}= ( P

_{v}- P

_{c}) / P

_{c}

For example, if Pc = 0.10 (10%) and Pv = 0.12 (12%), `Delta`

. That represents a lift of _{rel}> = ( 0.12 - 0.10 ) / 0.10 = 0.02 / 0.10 = 0.2`0.2 * 100 = 20%`

in the variant versus the control.

##
Calculate the standard deviations of the conversion rate in each group SDc and SDv by using the formula `SD = SQRT( P * (1 - P) / N )`

SQRT is the square root function available in most programming languages. It can be calculated using the `sqrt()`

function in Microsoft Excel, R, or other similar software.

Continuing with the example in which P_{c} = 0.10 and P_{v} = 0.12, and also adding the sample sizes N_{c} = 10010 and N_{v} = 10050, we get `SD`

and similarly _{c} = SQRT( 0.10 * (1 - 0.10) / 10010 ) = SQRT( 0.10 * 0.90 / 10010 ) = SQRT( 0.09 / 10010 ) = 0.002998`SD`

._{v} = SQRT( 0.12 * (1 - 0.12) / 10050 ) = SQRT( 0.12 * 0.88 / 10050 ) = SQRT( 0.1056 / 10050 ) = 0.003241

##
Calculate the coefficients of variation in each group CVc and CVv by dividing its standard deviation by its conversion rate using the equation `CV = SD / P`

.

Continuing with the previous example, we have `CV`

and _{c} = SD_{c} / P_{c} = 0.002998 / 0.10 = 0.02998`CV`

._{v} = SD_{v} / P_{v} = 0.003241 / 0.12 = 0.027

##
Calculate the lower confidence interval bound by solving `CI`_{lower} = (Delta_{rel} + 1) * ((1 - Z * SQRT(CV_{c}^{2} + CV_{v}^{2} - Z^{2} * CV_{c}^{2} * CV_{v}^{2})) / (1 - Z * CV_{c}^{2})) - 1

.

_{lower}= (Delta

_{rel}+ 1) * ((1 - Z * SQRT(CV

_{c}

^{2}+ CV

_{v}

^{2}- Z

^{2}* CV

_{c}

^{2}* CV

_{v}

^{2})) / (1 - Z * CV

_{c}

^{2})) - 1

While it may seem complicated, it is a rather straightforward calculation. Taking the values previously calculated and following the same example, we get:

`CI`

_{lower} = (0.2 + 1) * (( 1 - 1.644854 * SQRT(0.02998^{2} + 0.027^{2} - 1.644854^{2} * 0.02998^{2} * 0.027^{2})) / (1 - 1.644854 * 0.02998^{2} )) - 1

= 1.2 * (( 1 - 1.644854 * SQRT( 0.0016287 - 0.00000177 )) / ( 1 - 0.00148 )) - 1

= 1.2 * (( 1 - 1.644854 * 0.040 ) / 0.99852 ) - 1

= 1.2 * (( 1 - 0.06579416 ) / 0.99852 ) - 1

= 1.2 * ( 0.93420584 / 0.99852 ) - 1

= 1.2 * 0.94 - 1

= 0.12270

To present the result as a percentage lift, multiply by 100: `0.12270 * 100 = 12.27%`

lift. Therefore, the lower bound of the 95% one-sided confidence interval is 12.27%. Effect sizes lower than that number can be rejected with 95% confidence.

To find the upper bound, replace `1 - Z`

in the above equation with `1 + Z`

, and keep the rest the same.