Prevent Validity Threats

Business Benefits

Prevent your business getting wrong data that can affect your long term goals.


Take note of significant holidays happening during your testing period and monitor media events around popular search terms.

Talk within your company to inform them about your current test and keep them in the loop to avoid false data.

For example, you run tests when there is a lot of outcoming traffic from paid advertising, which is not representative of your overall traffic.

Ask your A/B testing developers to Q/A test in every browser and device to ensure your testing instrument works without problems.

Document your Q/A process, assign it to dedicated individuals, or outsource it to specialists to prevent your tests from failing due to lack of software reliability.

Identify if the test is functional on every single browser, device, or operating system.

Scan your results to identify if you are tracking every event or goal in your analytics without any errors.

Connect your tests to a secondary analysis platform such as Google Analytics and turn on the GA for each test.

For example, if your test data shows an increase with 40% of revenue, talk to your finance department to see if the numbers match your accounting and test data.

Run a dual control experiment where you run two control pages and a variation page to identify if you have a code issue.

  • Create two identical control pages and a variation page and send the traffic to your test as followed 25%/25%/50%.
  • Observe if users interact the same with the two control pages, make edits and test again.

Talk to your marketing team to know if they are running an aggressive paid marketing campaign during your tests to prevent unrepresentative traffic to your tests.

  • Exclude paid campaign traffic to interact with your tests variations
  • Identify in your test results the number of results per traffic source to know if your test is valid

Examine your test on different browsers, devices, and operating systems to identify if you have an issue with your code.

Look for large discrepancies such as loading speed and layouts when examining your code