Where Most People Go Wrong When Split Testing

There’s a lot of information out there about split testing; after all, if there’s one thing marketers like to do, it’s create marketing content.

But, as you may know if you’ve dabbled with it in the past, a lot of the “best practices” out there probably won’t work for you.


Because most of that content was written by companies that have a million uniques a month or more… Which is actually a pretty small percentage of the sites out there on the internet.

Even businesses that are making a few million dollars year often don’t see that many visitors, and that means most of those best practices simply aren’t relevant for most sites.

So today I’m going to distill down my experience from the last 9 years, during which I’ve been involved in countless split tests for countless companies, and share best practices that work for “the rest of us” — that is, those of us with sites that aren’t Amazon or Google.

What to Measure: Looking at the Results of a Split Test

Ultimately, when it came down to deciding whether or not the new helicopter tour company’s website was a  success, it came down to looking at sales numbers.

And that’s important to recognize.

All too often when looking at split testing marketers fall into the habit of measuring clicks or views instead of dollars. But sometimes that’s misleading.

For example, imagine that we have 1,000 visitors come to a sales page. Half of those visitors see version A of the page; half see version B.

Of the 500 people who see version A, 300 of them click on a button at the top of the page that says “Learn more.”

Of the 500 people who see version B, only 100 click on a button at the top of the page that says, “Get your copy now.”

But of the 300 who clicked on version A, only 10 people wound up actually buying the widget, while all 100 of those who clicked on version B made it all the way through the sales process.

If we were just measuring clicks, version A would have seemed much more successful than version B — but when we look at dollars, it’s clear that version B is the winner.