It can make sense to switch your choice in an ABC testing scenario if you have determined that option C is definitely not going to work, and the difference in performance between options A and B is not significant. In this case, switching from option A to option B may increase your chances of choosing the correct option, based on the principles of the Monty Hall paradox.
However, it’s important to consider other factors before making a decision to switch. The cost of switching, such as the resources required to implement the change, should be taken into account. Additionally, you should also consider the potential impact on your business and whether the increase in probability of choosing the correct option justifies the cost of switching.
It’s also important to keep in mind that this is not a standard method in A/B testing. Proper experimental design, randomization, and control groups should be applied to make sure that the results are reliable and not biased. And also, you should use the appropriate statistical methods to analyze the data and make inferences about the results.
Overall, the Monty Hall Paradox can be used as a guide to make a decision, but it should not be the only factor to consider.
What you may also be referring to is called "Probability Matching" - in the Monty Hall paradox, Monty always nudged the user away from the door the user chose even if it was the correct door. In this scenario, you’re assuming that by switching focus from Option A to Option B you’ll have a 66% chance of always picking a winner. The problem with this strategy tends to not work well or have enough evidence that it works due to personal bias or at some point human pattern recognition tries to intervene.
A/B testing is about learnings, not always trying to pick the best option, if that’s what you’re trying to do, then it’s just better to stick to the Bayesian methodology, where probability matching and even the Monty Hall paradox fall under.