Where Most People Go Wrong When Split Testing

There’s a lot of information out there about split testing; after all, if there’s one thing marketers like to do, it’s create marketing content.

But, as you may know if you’ve dabbled with it in the past, a lot of the “best practices” out there probably won’t work for you.

Why?

Because most of that content about split testing was written by companies that have a million unique visitors a month or more… which is actually a pretty small percentage of the sites out there on the internet.

Even businesses that are making a few million dollars year often don’t see that many visitors, and that means most of those best practices on split testing simply aren’t relevant for most sites.

So today I’m going to distill down my experience from the last 9 years, during which I’ve been involved in countless split tests for countless companies, and share best practices that work for “the rest of us” — that is, those of us with sites that aren’t Amazon or Google.

What to Measure: Looking at the Results of a Split Test

Ultimately, when it comes down to deciding whether or not a split test was a success, it comes down to looking at the sales numbers.

And that’s important to recognize.

All too often when looking at split testing marketers fall into the habit of measuring clicks or views instead of dollars. But sometimes that’s misleading.

For example, imagine this split testing experiment for a company selling helicopter tours of the Grand Canyon. They have 1,000 visitors come to a sales page. Half of those visitors see version A of the page; half see version B.

• Of the 500 people who see version A, 300 of them click on a button at the top of the page that says “Learn more.”

• Of the 500 people who see version B, only 100 click on a button at the top of the page that says, “Watch a video of the helicopter tour.”

At first glance, it looks like version A is more successful, right? But when we look at the sales numbers, we see a different story.

Of the 300 who clicked on version A, only 10 people wound up actually booking a helicopter tour, while 75 of those who clicked on version B made it all the way through the sales process.

If we were just measuring clicks, version A would have seemed much more successful than version B — but when we look at dollars, it’s clear that version B is the winner.

How can you tell this yourself? By using a split test calculator. Once you’ve finished running your test, you want to use a split test calculator to confirm which test was more impactful.

Want to confirm the split test results yourself? Visit our split test calculator, enter the numbers of Visitors and Conversions for the Control and Variation above, and see the statistically significance of the results.