Calculate statistical significance for absolute difference in means

Business Benefits

Evaluate A/B test results to make confident decisions while controlling business risk.


Calculate the arithmetic means (Mc, Mv) of the two groups by dividing the sum of values by the number of observations in each.

If the metric of interest is average revenue per user, the total number of observations in the control group is simply the number of users enrolled in the control, whereas the sum of values would be the summation of the revenues from each user in the group. Be sure to include zero-valued rows, for example, users who did not purchase anything during the A/B test.

For example, the arithmetic mean for the following raw data in the variant group would be $73.94 / 15 = $4.93:

$14.99
$14.99
$9.99
$9.99
$9.99
$8.99
$5.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

Estimate the standard deviation of each group (SDc and SDv).

Calculate the pooled standard deviation (SDp) using the equation SDp = SQRT( ( ( Nv - 1 ) * SDv2 + ( Nc - 1 ) * SDc2 ) / ( Nc + Nv - 2 ) )

.

SQRT is the square root function available in most programming languages. It can be calculated using the sqrt() function in Microsoft Excel or R.

For example, if the sample sizes Nc and Nv are 1000 and 1010, and the standard deviations were estimated to be 5.5 and 6 respectively, then:

SDp = SQRT( ( (1000 - 1) * 5.52+ (1010 - 1) * 62) / (1000 + 1010 - 2))
SDp = SQRT( (999 * 30.25 + 1009 * 36) / 2008 )
SDp = SQRT( (30219.75 + 36324) / 2008 )
SDp = SQRT(33.14)
SDp = 5.756

Calculate the standard error of the mean using the equation SE = SDp * SQRT( 1 / Nv + 1 / Nc )

.

SQRT can be calculated using the sqrt() function in Microsoft Excel, R, or other similar software.

For example, with SDp = 5.756, Nv = 1000 and Nc = 1010

we have:

SE =  5.756 * SQRT(1 / 1000 + 1 / 1010)
SE = 5.756 * SQRT( 0.002 )
SE = 5.756 * 0.0447
SE = 0.257

Calculate the difference (Delta) of the two means by subtracting the mean of the control group from the mean of the variant group.

Following the same notation, the difference in means would be Delta = Mv - Mc.

For example, if Mv = 4.93 and Mc = 4.00, then Delta would be 4.93 - 4.00 = 0.93.

Calculate the Z score for difference in means by the formula Z = Delta / SE.

Continuing with the example where Delta = 0.93 and SE = 5.756, you would get Z = 0.93 / 0.257 = 3.62. This tells you that the observed outcome is 3.62 standard deviations away from the expected value under the null hypothesis of zero or negative difference (H0: Delta ≤ 0).

Calculate the cumulative distribution function of the standard normal distribution for the value of Z and subtract it from one to get the p-value.

Use a tool like Microsoft Excel, R, the GIGA online z-score calculator to calculate the cumulative distribution function:

  • Use the NORM.S.DIST() function in Excel with a second parameter set to TRUE.
  • Use the pnorm() function in R.
  • Use the GIGA online z-score calculator for a normal distribution with a mean of 0 and a standard deviation of 1.

Following the above example, the CDF for a Z score of 3.62 would be 0.99985, therefore the p-value would be 1 - 0.99985 = 0.00015.

If the p-value is less than your significance threshold, the result is statistically significant and you can reject the hypothesis of no difference between the test groups. Otherwise it is not statistically significant and the null stands intact.

For example, if the p-value threshold for a given test is p = 0.01 and you observe p = 0.00015, then you can conclude that the outcome is statistically significant since 0.00015 < 0.01.