Business Benefits
Create tests that strike a good balance between data quality and efficiency.
Build a simulation or growth model using random or historical experiment data to determine the optimal number of test variants.
You’re trying to predict the likelihood of various outcomes given a certain number of variants per test. When building a model like this, it’s usually surprising for people to see how high the optimal number of variants is.
Create new concepts, executions of concepts, and high contrast variations to avoid getting too narrow a variation focus.
Typically the more substantial the difference between variations, the bigger and clearer the results difference will be, statistically speaking
Use an iterative approach when you want to explore user behavior on a granular level.
For example, you can test adding a value proposition against your current (or lack of a) value proposition.
Use quick wins with one variation at a time to move onto increasing test velocity, program efficiency, and program support.
By demonstrating a few quick wins, you can get your organization enthusiastic about a bigger testing program.
Use a minimal number of variations if you are concerned users will be exposed to multiple variations in a test.
If your users often switch devices, they will become exposed to the experiment multiple times and create sample pollution. The more variations, the bigger the pollution. With one variation, there is a 50% chance a user will end up in the same variation if they return to the experiment, while with three, there is only a 25% chance.
Ask yourself, How much traffic do we have? and How long will it take to pull off a valid test?
If you have low traffic, it is better to run 10 A/B experiments over several locations of your site, rather than 1 big experiment on 1 location. Make sure that the estimated test duration is in line with your business goals.
Use a single variation against a control to understand buyer or user motivations.
If your goal is to explore what is working and what is not, user behavior and motivation factors will provide this knowledge.
Ask yourself, What is the maximum level of risk we are willing to take? and use an appropriate rate of alpha error.
For example, it might be smarter for your business to start with a high confidence level and fewer test variants.
Last edited by @hesh_fekry 2023-11-14T11:50:18Z