A/B testing is a simple way to test changes to a webpage against the current design and determine which ones produce positive results. Often, seemingly small tweaks to a product yield hugely different outcomes. Ask the Obama campaign, who received more than six times more donations by varying email subjects — and the office pool guessed wrong! When intuition falters, A/B testing allows you to try different variants and measure users’ responses. It takes the guesswork out of website optimization and enables data-backed decisions that shift business conversations from “we think” to “we know”.
During Data-driven Design through A/B Testing panel session, the panelists discussed the design of both basic and sophisticated A/B testing frameworks and their experiences on running the tests.
There are popular A/B testing tools, such as Google Analytics Content Experiments, which is a free tool and can test up to five full versions of a single page, each delivered to visitors from a separate URL. You can compare how different web pages perform using a random sample of your visitors.
Andrea Burbank from Pinterest talked about their customized A/B experimentation framework, which includes the ability assigning users to different experiences. The post-hoc analysis is performed segmentally. Jen Dolson from Facebook talked about using Google Analytics and her experiences on measuring the impact of UI, data, and search ranking changes on user engagement. Leslie Nguyen from Box talked about her experience implementing A/B tests on mobile apps, an area new to the A/B testing space due to the lengthy approval process and difficulty of real-time updates. Mona Gandh from AirBnB talked about designing and implementing frameworks that enable rapid testing and iteration of marketing messages, landing pages & conversion funnels. She also talked about building tracking and reporting infrastructure to measure the effectiveness of these A/B tests.
Things need to bear in mind when doing A/B testing
- The smaller the change is, the more data is needed to be sure that the conclusion is statistically significant
- Sample size should be set beforehand, and commit to not stop the test until the sample size has been exceeded
- A better confidence metric to use is the G-test
A/B testing requires some knowledge to perform properly.