Choose a strategy for running multiple tests

Business Benefits

Determine how your goals align with different test methods.

Write a list of goals you want to accomplish, or elements you want to test, and determine if these are paradigm-changing or not.

For example, if you are testing something that will impact your business model or the future of the company, it is best to test with multiple, separate tests (MST).

Decide if there is a lot, or a little, overlap between the elements you are testing.

Running MST is the best strategy if there is little overlap. Avoid this method if there is huge overlap between tests.

Assess if there is a risk of interaction present between the factors.

If there is, reduce the number of simultaneous tests, and/or let the tests run longer for improved accuracy. Avoid MST if you suspect extreme interactions.

Combine multiple tests into one MVT test for the following reasons:

  • If you are running tests to measure the same goal, for example, a purchase.
  • Elements in the same flow, for example, running tests on each of your multi-step checkout steps.
  • Suspect strong interaction between tests.

Use mutually exclusive tests (MET) if you want to focus on eliminating bias or noise.

With this method you are ensuring that people won’t be part of more than one test.

Determine what your sample size will be as an MET requires an adequate sample size.

Establish the timeframe for your tests.

An MET strategy is complex, and may slow down testing. If you plan to run tests for the same duration, combining multiple tests into one MVT is the best strategy.

Test a large percentage of your traffic with one high speed, high volume test.

Then use this data to invest more time and re-test important elements with multiple smaller, slower, more complex tests.