Business Benefits
Enrich conversion rate difference A/B test reports and allow for more informed decisions.
Calculate the conversion rates (Pc, Pv) in each of the two test groups by dividing the number of conversions by the total number of observations.
If we use:

E_{c}
: number of conversions in the control group. 
E_{v}
: number of conversions in the variant group. 
N_{c}
: total number of observations in the control group. 
N_{v}
: total number of observations in the variant group. 
P_{c}
: conversion rate for the control group. 
P_{v}
: conversion rate for the variant group.
then calculate:
P_{c}= E_{c} / N_{c}
P_{v} = E_{v} / N_{v}
For example, if the metric of interest is a userbased conversion rate, then the total number of observations in the control group is the number of users enrolled in the control, and the number of conversions is the number of conversions by users in the control group. If there were 10,000 users in the control group and 1,200 checkouts started, then:
P_{c} = 1,200 / 10,000 = 0.12
Calculate the difference (Delta) between the two conversion rates by subtracting the conversion rate in the control group from the conversion rate in the variant group.
Following the same notation, the difference in conversion rates would be:
Delta = P_{v}  P_{c}
Following the example above, P_{v} = 0.12
and P_{c} = 0.10
.
Delta = 0.12  0.10 = 0.02
Compute the pooled proportion using the equation P_{pooled} = (E_{v} + E_{c}) / (N_{v} + N_{c})
For example, if E_{v} = 1200
, E_{c} = 1000
, N_{v} = 10000
and N_{c} = 10000
, then:
P_{pooled} = (1200 + 1000) / (10000 + 10000) = 2200 / 20000 = 0.11
Calculate the standard error (SE) of the difference in conversion rates using the equation SE = SQRT( P_{pooled} * ( 1  P_{pooled} ) * ( 1 / N_{v} + 1 / N_{c} ))
SQRT is the square root function available in most software languages. It can be computed using the sqrt()
function in Excel, R, and similar software.
For example, if P_{pooled} = 0.11
, N_{v} = 10000
and N_{c} = 10000
, we have:
SE = SQRT( 0.11 * (1  0.11) * (1 / 10000 + 1 / 10000) )
SE = SQRT( 0.0979 * 0.0002 )
SE = SQRT( 0.00001958 )
SE = 0.004425
Calculate the Z score (Z) by using the equation Z = Delta / SE
This is a simple division to express the observed difference Delta in terms of number of standard deviations.
Continuing with the example above:
Z = 0.02 / 0.0044 = 4.52
Use a tool like Excel, R, or the GIGA online zscore calculator to calculate the cumulative distribution function of the standard normal distribution for the value of Z and subtract it from 1 to get the pvalue.
 Use the
NORM.S.DIST()
function in Excel with a second parameter set toTRUE
.  Use the
pnorm()
function in R.  Use the GIGA online zscore calculator for a normal distribution with a mean of 0 and a standard deviation of 1.
To make sure your calculation is correct you can check it using this reference: for a Z score of 1.644854, the CDF is ~0.95 and the pvalue would be 1  0.95 = 0.05.
If the pvalue is less than your significance threshold, the result is statistically significant and you can reject the hypothesis of no difference between the test groups.
Otherwise, it is not statistically significant and the null stands intact.