Beginner’s Guide to AB Split Testing

Optimization is growing quickly as an industry as more companies increase their online presence. Like any other industry that’s exploding, there’s a lot of confusion in the market around the core science. What exactly is optimization? How do you run split tests?

Before you get into any type of advanced testing, it’s first important to understand the basics of split testing. This includes not only the science behind it but the limitations on what type of impact A/B split testing can have on your site.

Here’s a beginner’s guide to A/B split tests so you can get the basics:

That’s it! The core concept is pretty simple.

There are two basic ways to run AB Tests:

1. Run Yourself Manually: Simply divide the traffic evenly between the two sites, calculate the change in conversion rate, and then do a quick statistical test to determine if the difference is statistically significant.
2. Use Software: Visual Website Optimizer or Optimizely will do just fine. These programs will let you make visual adjustments to your website and then automatically calculate statistical significance for you.

Because this removes a lot of the manual labor and speeds up the process, using software is my preferred approach.

If you’re completely new to split testing, I recommend that you divide up your tests between quick wins/low hanging fruit and then leave more advanced testing to the experts.

Here are a few examples of what you can test in each category:

Quick Wins

1. Calls to Action: Simply changing the text on your buttons on landing pages etc. can have a dramatic impact on conversion rate. This is easy to test using self-serve software.
2. Headlines: Your “elevator pitch” for your product on your homepage is one of the most important features that you have for maximizing conversion rate. Again, swapping these out can give you a quick win.
3. Color: What color do you make your background image? Call to action buttons? Do you have enough contrast?
4. Font Size & Style: Simply changing the type can sometimes be enough to improve conversion (as strange as this seems).
5. Background Images: Do you feature a guy or girl on the homepage? Bright or dark colors?

These are simple but can easily drive a 5-15% improvement in conversion rate.

Dramatic & Harder Changes

1. Mobile and Tablet: How “responsive” is your site? Do you have different versions of your site that are optimized for conversion for both of these designs? Especially in e-commerce, mobile and tablet designs need to be perfected because of how high the traffic is.
2. Speed: Website speed affects conversion rate. Simply getting your page to load 0.5s faster can improve conversion by 5-10%.
3. Price: In competitive categories, such as e-commerce, increasing or decreasing the price can affect conversion rate and revenue dramatically. This is difficult to execute correctly, but if you do it right, optimizing your price can do a lot to boost your net margin.
4. Drastic Changes: A complete redesign of your site – a “risky” test – can carry a dramatic reward in conversion. These include redesigning your color scheme, layout, or even a complete rebrand. This takes time and considerable effort, so I don’t recommend doing this until you’ve gone after all the low hanging fruit.

I wouldn’t recommend doing any of the more advanced tests on your own – these are best to hire an expert for (more on that below).

If you were to run a test manually, you would:

1. Divide the Traffic Evenly Between Two Versions
2. Measure the Conversion Rate of Each
3. Do a Chi-Square Test and Reach 95% Confidence Level

When you are testing, you need to ensure that you are splitting the traffic between the two sites at exactly the same time. You *cannot* run one version of the site one week, another version of the site another week, and then measure conversion rate. Why?

Seeing a conversion rate increase in an AB test does not mean your conversion rate will improve by that amount once the changes go live.

For example – let’s say that you performed a test showing a 10% improvement in conversion rate. Your original conversion rate was 6%, and the new version of the site converted at 6.6%.

Then, you make the change live, and you see that your conversion rate is still 6%. Did the test not work?

No!

This is a common mistake. You have to understand that there are many factors that affect your conversion rate, including:

1. Time of Day / Week
2. Quality of Source Traffic – for example, if you suddenly get a PR splash, your traffic will increase, but with higher volume, a lower percentage of people will be qualified.

You need to be extremely rational when performing split tests – be sure to compare conversion rates at the same time and understand that your conversion rates will fluctuate.

There are two main resources for you:

1. Do It Yourself: My favorite self-help educational materials are Visual Website Optimizers library of case studies as well as Which Test Won, a library of A/B tests showing an improvement in performance.
2. Hire an Expert: There is a growing number of conversion rate optimization experts given the increased demand for these services. The best resource for you to find talent is by hanging around Moz – most of the authors writing articles on conversion rate optimization will be professionals who can help with advanced testing.