Conflicting A/B testing results more add to carts vs less transactions

So I have this A/B test for a store that test weather a variant renders more add to carts than the control. I set up de Add to cart as primary objective and the purchase as a secondary. The results are pretty conclusive and point towards an uplift in add to carts and a downtrend on purchases.

Is more carts with slightly less conversion better than less carts with more conversion? Will the net result be bigger volume with slight conversion downtrend? How will you interpret the results?

Here are some graphs to illustrate:
Primary Objective gets an uplift:

Secondary objective gets a downtrend:

2 Likes

Interesting question we have faced something similarl recently and decided to manually breakdown revenue to get some further insights.

Ended up reruning the expirement with revenue being the primary goal.

This was after some advice from @merritt and @paul_kirspuu whonmay have some added advice here.

@dcastro would setting revenue as a primary goal help in this situation?

1 Like

Not sure since what is possibly explaining the behavior is that the buying intention is greater for the original vs the variant because of te actual hypothesis. Giving more context:

The client substituted product page CTA (Button copy) form “Add to cart” to “Buy” although the action the button triggers is an add to cart. The intention behind doing this was to incentivize the buying behavior. Naturally, this provoked a +90% dop on add to carts, so our actual hypothesis is as follows:

“Changing the product page CTA back from “Buy” to “Add to cart” should generate an uplift in cart to detail rate since probably the users are not using the CTA for fear of triggering a one click buy.”

So far, the hypothesis has proven true because the add to cart rate has risen, but the transaction as secondary has dropped. This could be because the “original” tends to direct users with higher buy intent than the “variant”, since the “original” has a “buy” CTA.

There is a possibility that revenue would render a similar result since the AOV is almost the same between “original” and “variant”

1 Like

I would recommend analyze the results using users metrics instead of using sessions metrics, because you are interested if a user converted or not with the control or variant, not in how many sessions they clicked the button. Same applies to transactions: you are interested in how many users exposed to the experiment made a transaction. The results can be very different using this metric! Add to cart sessions could be inflated by one user that made a lot of clicks, the same happens with transactions. I’m looking at you, false positives :laughing:

So, I recommend analyze with users metrics first and then make a conclusion.

You can find more info here:

3 Likes

Well just did the analysis and got to very interesting results:

Users that viewed the experiment:
Variant = 30,920
Control = 31,308

Users that viewed the experiment and have min 1 transaction:
Variant = 1,035
Control = 1,113

User to transactions Conversio Rate
Control = 1113/31308 = 3.55
Variant = 1035/30930 = 3.35
Diff = 0.20% (In favor to the control)

This suggest that having a “buy” button leads to an uplift of 0.2% in total conversion rate based on users which is a very surprising result. Now, the next thing to test is what happens if we add a double CTA like Amazon: “Buy now” or “Add to cart”, this could lead to an uplift in carts without hindering that uplift given by the buy now factor.

Thaks for the help, jus wanted to update you in the result

2 Likes

Great stuff @felipe.garcia.fnz brilliant input here. Some really good resources and advice.

@dcastro do you feel your questions/concerned where addressed with the advice?

Very curious to see the results of your next test too.

1 Like

It is not uncommon for A/B testing results to conflict, mainly when testing different aspects of a website or application. In this case, the results may be conflicting because the two test variations impact different user experience aspects.

For example, the variation that resulted in more add-to carts may have made it easier for users to find and add products to their cart, but it may not have necessarily translated to more transactions. On the other hand, the variation with fewer add-to carts may have resulted in fewer initial additions to the cart. Still, it may have resulted in more transactions due to other factors, such as a more effective checkout flow or compelling calls to action.

To better understand the conflicting results, conducting further analysis and gathering additional data may be helpful. This could include analyzing the user behavior data from both test variations, surveying users to gather their feedback, or conducting additional A/B tests to isolate specific elements of the user experience.

It is also important to remember that A/B testing is just one tool for gathering data and understanding user behavior. It is always beneficial to consider other data sources and approach the analysis with a holistic view of the user experience.

2 Likes