How to Do Mobile A/B Split Testing to Improve Conversion Rate

Despite mobile phones dominating modern-day culture, most people are unaware of how to properly optimize their mobile sites for conversion. This is significant because mobile is quickly becoming the dominant force on the internet – at Facebook, for example, about half of users are mobile-only.

Most major PPC and SEO advertisers will tell you that a growing share, if not the majority, of traffic they are seeing is coming from mobile. So, how do you perform proper mobile A/B split tests?

Do you see the differences here?

What WWF has implemented on this site is pretty common – there’s a completely separate mobile experience from the desktop experience. So, you can’t simply optimize the desktop and “hope” that the mobile version is going to respond just as well to the same changes – they are different.

The best way to complete mobile A/B testing is to treat mobile and desktop sites as two different websites. This means that each site is going to be treated like a project with its own baseline conversion rate and “goal” uplift.

It’s actually quite difficult to perform tests on websites with two different versions like this with current testing software. In my opinion, none of the testing platforms (VWO, Optimizely) are really well equipped to handle these different experiences or preview what your tests look like across devices.

So, how do you set up tests technically?

My biggest recommendation is that you put your mobile site on a separate domain and redirect all mobile traffic to this site. Then, set up two separate projects – one for the original “desktop” domain, and the other for the separate “mobile” domain.

Otherwise, if you make changes to the desktop site, you’re only going to be seeing those changes to a portion of your traffic (as high as only 50% of traffic in some cases).

Now that you have a separate mobile-only project, how do you prioritize tests to run?

The key to optimizing a mobile website or mobile app is to reduce as much friction as possible. It’s even harder to input text on mobile than it is on desktop, so you have to take out as many steps and field as you possibly can to boost conversion rate.

Looking at the WWF site above, there are a few themes that stand out to reduce friction:

1. Reducing CTAs: What is the #1 thing that people are likely to do when they come to your mobile site? In WWF’s case, there are already 3 separate calls to action: (1) donate, (2) about, and (3) get more information on the articles they put on your homepage. Which one of these is the most important? In this case, my guess is that simply getting someone on the email list (instead of asking for a donation) is going to be the best “path” to ultimately getting dollars.
2. Focusing on Above the Fold: Most mobile sites have the same problem that desktop sites have – they tend to have information reach homepages with text, images, etc. While having in-depth content can sometimes lift conversion rate (especially for B2B sales companies with longer sales cycles), reducing the amount of content on mobile sites will usually boost conversion. Otherwise, the user can get “caught up” in too much scrolling which distracts him/her from taking the goal CTA.

You could run these tests on the desktop too, but they are especially important on mobile-only.

Here are some additional tests to run across mobile and desktop:

1. Complete Mobile Redesign: What if you focused on a unique selling point/value proposition on the homepage with an email CTA rather than all the variables here? This would be a “radical” design but reducing all the distractions on the page would likely drive the highest uplift in conversion.
2. Personalized Experience: Are you able to give a different mobile experience to people who already have a relationship with WWF – such as existing donors? This is “pushing the limit” on mobile testing right now but would also contribute to getting a higher lift.
3. Speed Tests: Website speed is known to affect conversion rate and mobile load times are no different. How fast does this site load? Can you reduce load times on mobile devices to improve the # of people subscribing?

Remember that when you say “Mobile A/B split testing”, you need to be clear what we’re talking about – just testing for websites that are “responsive” on mobile devices? Or actually testing web apps?

Most software solutions will offer both, but here’s a quick rundown with more detail:

• VWO: Mobile and Mobile App Testing
• Optimizely: Mobile and Mobile App Testing
• Arise: Mobile Apps
• Leanplum: Mobile Apps
• Apptimize: Mobile Apps
• Swrve: Mobile Apps

As you can see there’s quite a number of options to choose from. But, again, remember to keep a few things in mind:

• Desktop Testing does *not* mean Mobile Testing: Most AB testing software is built for desktop testing, *not* mobile. Even if your website is responsive, you need to understand that changes you make on the desktop will not automatically translate 1:1 with mobile.
• Mobile Apps Require Specialization: You can’t apply the same type of split testing frameworks to mobile apps – they are similar, but because everything is an “in-app” experience, you can’t always measure success the same way.