Evaluate A/B test results to make confident decisions while controlling business risk.
Calculate the arithmetic means (Mc, Mv) of the two groups by dividing the sum of values by the number of observations in each.
If the metric of interest is average revenue per user, the total number of observations in the control group is simply the number of users enrolled in the control, whereas the sum of values would be the summation of the revenues from each user in the group. Be sure to include zero-valued rows, for example, users who did not purchase anything during the A/B test.
For example, the arithmetic mean for the following raw data in the variant group would be $73.94 / 15 = $4.93:
$14.99 $14.99 $9.99 $9.99 $9.99 $8.99 $5.00 $0.00 $0.00 $0.00 $0.00 $0.00 $0.00 $0.00 $0.00
Estimate the standard deviation of each group (SDc and SDv).
Calculate the pooled standard deviation (SDp) using the equation
SDp = SQRT( ( ( Nv - 1 ) * SDv2 + ( Nc - 1 ) * SDc2 ) / ( Nc + Nv - 2 ) )
For example, if the sample sizes Nc and Nv are 1000 and 1010, and the standard deviations were estimated to be 5.5 and 6 respectively, then:
SDp = SQRT( ( (1000 - 1) * 5.52+ (1010 - 1) * 62) / (1000 + 1010 - 2))
SDp = SQRT( (999 * 30.25 + 1009 * 36) / 2008 )
SDp = SQRT( (30219.75 + 36324) / 2008 )
SDp = SQRT(33.14)
SDp = 5.756
For example, with
SDp = 5.756, Nv = 1000 and Nc = 1010
SE = 5.756 * SQRT(1 / 1000 + 1 / 1010) SE = 5.756 * SQRT( 0.002 ) SE = 5.756 * 0.0447 SE = 0.257
Calculate the difference (Delta) of the two means by subtracting the mean of the control group from the mean of the variant group.
Following the same notation, the difference in means would be
Delta = Mv - Mc.
For example, if
Mv = 4.93 and
Mc = 4.00, then Delta would be
4.93 - 4.00 = 0.93.
Continuing with the example where
Delta = 0.93 and SE = 5.756, you would get
Z = 0.93 / 0.257 = 3.62. This tells you that the observed outcome is 3.62 standard deviations away from the expected value under the null hypothesis of zero or negative difference (H0: Delta ≤ 0).
Calculate the cumulative distribution function of the standard normal distribution for the value of Z and subtract it from one to get the p-value.
- Use the
NORM.S.DIST()function in Excel with a second parameter set to TRUE.
- Use the
pnorm()function in R.
- Use the GIGA online z-score calculator for a normal distribution with a mean of 0 and a standard deviation of 1.
Following the above example, the CDF for a Z score of 3.62 would be 0.99985, therefore the p-value would be
1 - 0.99985 = 0.00015.
If the p-value is less than your significance threshold, the result is statistically significant and you can reject the hypothesis of no difference between the test groups. Otherwise it is not statistically significant and the null stands intact.
For example, if the p-value threshold for a given test is p = 0.01 and you observe p = 0.00015, then you can conclude that the outcome is statistically significant since 0.00015 < 0.01.