Develop an A/B test hypothesis

Contributors

@sonja4dp-com-au


Business Benefits

Increase the effectiveness of your website by solving problems that prevent users from using the website to the fullest potential.


Compose your hypothesis to include the data you found, the change and impact you expect, and how you’ll measure the results.

Use the same structure across all your hypotheses:

  • Because we saw [data or feedback]
  • We expect that [change] will cause [impact]
  • We’ll measure this using [metric].

Encourage your team to use data as the starting point of a hypothesis and not assumptions generated from their ideas.

Avoid turning brainstorming ideas into hypotheses as you don’t have real data to back them up.

Conduct conversion research through heuristics analysis to collect data on your users, their purchasing patterns, and what they want to purchase, and identify existing issues.

Use qualitative and quantitative research to gather data on website users and increase your data sources.

Scan your data, looking for issues that will impact your business if you change one element or part of your website.

For each identified problem in your data, form multiple hypotheses for a solution. For example, if you identify that it’s not clear what exactly is being sold on a product page. People don’t buy what they don’t understand.

Write a statement describing how if you make one change and improve one part of your business, for example your website or copy, it could impact a part of your business.

Write the statement to include a concrete action and your expectation. For example, by improving the clarity of the product copy and overall presentation, people can better understand our offering.

Write your final hypothesis to include your desired outcome and use a single metric to track whether there was a change.

Track performance with a single key metric such as average order value and use other metrics only as information. For example, rewrite product copy so that it’s easy to understand what the product is, for whom it’s designed, and what the benefits are, to increase X product sales.

Determine up front the amount of time you’ll allocate to your test and include in your hypothesis the expected change in your metrics over a period of time or number of business cycles.