Most testing solutions rely on statistical methods that take a frequentist approach to significance testing, such as the Student’s t-test and the Z test. While these methods have proved themselves valuable, and are viable in sectors such as pharmaceuticals and agriculture, they are not well-suited to the constantly changing online world.
Using these methods requires a strict methodology known as fixed horizon statistics, in which a sample size must first be defined, and in which no conclusion may be made before this size is reached. These constraints are no longer relevant to a world in which data and information constantly flows, and are accessible in real time.
Here at AB Tasty, we’re convinced that the user should not have to bend to constraints, but that solutions should be adapted to give you greater flexibility. This is why we’ve developed Clever Stats, our new results analysis algorithm.
In this real-time age, and when applied to A/B testing on web and mobile sites, you want to be able to see how results are tracking before your page has been shown to your entire specified sample size. In other words, if you’ve chosen to show a certain version of a page to a sample size of 100 people, Clever Stats allows you to start checking results after your page is shown to just a handful of them. Why wait, when usable data have already been collected? You should be able to quickly interpret the data, and act accordingly as soon as results seem promising.
In our relentless efforts to make A/B testing easy to implement for everyone, we have developed the first testing solution to rely on Bayesian statistics. Thanks to almost 12 months of development, this new calculation method gives you greater flexibility on how soon you can start to view your test results, meaning you can make decisions faster, with less risk of incorrectly identifying a high-performance version of a page.
“With Clever Stats, we are taking things even further in our promise to provide simplicity. Since freeing marketing teams from highly technical methods of modifying pages and client browsing paths, we’re now also freeing them from statistical constraints and allowing them to more easily exploit the data collected in tests, and to act with complete confidence.” - Rémi Aubert, co-founder of AB Tasty
British mathematician Thomas Bayes (1702-1761) is the man behind the Bayes Theorem, an important law of probability which we use as a basis for our algorithm. The Bayesian approach to statistics is therefore not new, and has always countered the frequentist approach usually used in significance testing.
We had to wait until 2015 for an A/B testing solution to finally adopt this approach, which makes up for many weaknesses in frequentist testing. Without revealing the finer details of our algorithm, we can tell you that our data scientists and statisticians have worked tirelessly to adapt this formula to your A/B testing and satisfy your need to be agile. You can thank them later.
