Create better A/B testing programs by running a pre-registration process.
Create a document with the following sections: Hypothesis, Dependent variables, Sample size, Analysis plan, Testing execution plan, and Dependencies.
- Divide the Hypothesis list into two subsections: Testing Hypothesis(HA) and Null Hypothesis(HO).
Use a testable and falsifiable statement starting with “I believe”. For example:
- I believe my company will gain a 50% increase in revenue for every 50% increase in price.
- I believe changing our brand colors to violet and gray will increase our customer retention rate by 10%.
- I believe increasing our ad spend will improve our ads reach.
These are falsifiable and testable, while “I am an occult mystic because I can see ghosts with my third eye” is not.
Your null hypothesis states the opposite of your testing hypothesis. For example:
- Our revenue will not increase by 50% if we increase our price by 50%.
- Changing our brand colors to violet and gray will not improve our customer retention rate.
- Increasing our ad spend will not improve our ads reach.
The standard significance levels is 95%, while the standard power level is 80%. You can calculate your sample size easily with the CXL A/B test calculator.
Add how your test will be run, where the test will be run, and when your tests will be run to this section.