Calculate statistical significance for absolute difference in conversion rates

Business Benefits

Enrich conversion rate difference A/B test reports and allow for more informed decisions.


Calculate the conversion rates (Pc, Pv) in each of the two test groups by dividing the number of conversions by the total number of observations.

If we use:

  • Ec: number of conversions in the control group.
  • Ev: number of conversions in the variant group.
  • Nc: total number of observations in the control group.
  • Nv: total number of observations in the variant group.
  • Pc: conversion rate for the control group.
  • Pv: conversion rate for the variant group.

then calculate:

Pc= Ec / Nc
Pv = Ev / Nv

For example, if the metric of interest is a user-based conversion rate, then the total number of observations in the control group is the number of users enrolled in the control, and the number of conversions is the number of conversions by users in the control group. If there were 10,000 users in the control group and 1,200 checkouts started, then:

Pc = 1,200 / 10,000 = 0.12

Calculate the difference (Delta) between the two conversion rates by subtracting the conversion rate in the control group from the conversion rate in the variant group.

Following the same notation, the difference in conversion rates would be:

Delta = Pv - Pc

Following the example above, Pv = 0.12

and Pc = 0.10

.

Delta = 0.12 - 0.10 = 0.02

Compute the pooled proportion using the equation Ppooled = (Ev + Ec) / (Nv + Nc)

For example, if Ev = 1200, Ec = 1000, Nv = 10000 and Nc = 10000, then:

Ppooled = (1200 + 1000) / (10000 + 10000) = 2200 / 20000 = 0.11

Calculate the standard error (SE) of the difference in conversion rates using the equation SE = SQRT( Ppooled * ( 1 - Ppooled ) * ( 1 / Nv + 1 / Nc ))

SQRT is the square root function available in most software languages. It can be computed using the sqrt() function in Excel, R, and similar software.

For example, if Ppooled = 0.11, Nv = 10000 and Nc = 10000, we have:

SE = SQRT( 0.11 * (1 - 0.11) * (1 / 10000 + 1 / 10000) )
SE = SQRT( 0.0979 * 0.0002 )
SE = SQRT( 0.00001958 )
SE = 0.004425

Calculate the Z score (Z) by using the equation Z = Delta / SE

This is a simple division to express the observed difference Delta in terms of number of standard deviations.

Continuing with the example above:

Z = 0.02 / 0.0044 = 4.52

Use a tool like Excel, R, or the GIGA online z-score calculator to calculate the cumulative distribution function of the standard normal distribution for the value of Z and subtract it from 1 to get the p-value.

  • Use the NORM.S.DIST() function in Excel with a second parameter set to TRUE.
  • Use the pnorm() function in R.
  • Use the GIGA online z-score calculator for a normal distribution with a mean of 0 and a standard deviation of 1.

To make sure your calculation is correct you can check it using this reference: for a Z score of 1.644854, the CDF is ~0.95 and the p-value would be 1 - 0.95 = 0.05.

If the p-value is less than your significance threshold, the result is statistically significant and you can reject the hypothesis of no difference between the test groups.

Otherwise, it is not statistically significant and the null stands intact.