Maxymiser (now part of Oracle) laid out some pros and cons of using either test: You can then just read the test as if it were two-tailed.” Pros and cons of each method So if you want your two-tail test to be at the 95% confidence level, then you would actually input a confidence level of 97.5%, or if at a 99% level, you need to input 99.5%. If your software uses a one-tail test, just divide the p-value associated with the confidence level you are looking to run the test by two. All that’s different is the significance threshold level. It is super simple to convert one type to the other (but you need to do this beforeE you run the test) since all of the math is exactly the same in both tests. If your testing software only does one type or the other, don’t sweat it. Of all of the issues facing you when embarking on testing, this isn’t really the one you should worry about. “For some reason, folks are making a big deal about one-tail vs. Okay, so now that we went over what the tests actually are, we can ask the important question: Does it even matter which you use? Turns out, that’s a complicated question. ![]() The one-tailed test takes as a null hypothesis the belief that the variation is not better than the control, but could be worse. The two-tailed test takes as a null hypothesis the belief that both variations have equal conversion rates. ![]() Now suppose you are A/B testing a control and a variation, and you want to measure the difference in conversion rate between both variants. The lower the p-value, the less plausible it is that the null hypothesis is true. The p-value represents the probability of seeing a result at least that “extreme” in the event the null hypothesis were true. Now suppose you’ve run a test and received a p-value. ![]() The null hypothesis is what you believe to be true absent evidence to the contrary. In frequentist tests, you have a null hypothesis. Why would you choose one over another? The two-tailed test can show evidence that the control and variation are different, but the one-tailed test can show evidence if variation is better than the control.Ĭhris Stucchio does a great job explaining the difference between the two tests in context:
0 Comments
Leave a Reply. |