We're big advocates of testing. But to us, testing is only one tool in a marketer's toolbox. The bigger mission: find great customers, engage and impress them, start a good relationship, and do it efficiently with a strong return on marketing investment (ROMI).

That's why when it comes to landing pages -- particularly multi-page landing experiences -- we favor A/B split testing over multivariate testing (MVT). It helps keep the big picture in focus.

I'll explain.

I know, MVT is "in" these days. It promises a kind of magical box: drop dozens or hundreds of elements into the box, pour in your respondents, the box shakes around trying thousands and thousands of combinations, and voila!, out pops THE ANSWER. Only a few of us math geeks understand exactly how it reaches that answer, but for everyone else, that's okay, the only thing that really matters is the answer. Right?

Well, sacred cows make great hamburger.

There are a number of caveats to MVT that tend to get lost in the fine print.

For instance, fractional factorial MVT -- popularized by some vendors because they promise to give you the answer without having to test all of those thousands of combinations directly -- are usually based on the assumption that there are no "interaction effects" between the elements, say between a headline on the page and an image on the page. But do you as a marketer believe that a headline and its accompanying image have no interplay?

I've described other concerns with MVT -- optimizing to the average user instead of finding the best answer for each segment independently, difficulty in running apples-to-oranges experiments, etc. -- in a paper I wrote a couple of months ago.

But I'd like to bring attention to a very basic problem with the MVT approach that is often overlooked: the Russian roulette caveat.

Let's say you're testing a landing page with a structure of 5 testable elements: headline, subhead, image, body copy, and call-to-action button. The MVT approach encourages you to load up many variations of each, say 10 headlines, 5 subheads, 5 images, 2 body copy blocks, and 3 different call to action buttons.

That's 10 x 5 x 5 x 2 x 3 = 1,500 possible combinations.

A supposed benefit of MVT is that you as a marketer don't have to visualize each of these combinations. The good news is that somewhere in there is hopefully one combination that is better than all the others, your star, and that's what you're searching for. The bad news, however, is that there are probably a fair number of dogs in there too. Maybe even some really bad ones.

What is a bad combination? (Seth Godin's imagery of a meatball sundae comes to mind.) It's when two or more elements clash so as to confuse or mislead the respondent, give them the wrong impression, or simply come across as a disjointed presentation that reflects poorly on your brand.

For example, say you're marketing your Caribbean resort, and you have two headlines: "Enjoy our delicious food!" and "Relax in our luxurious health spa!". You also have two images: a photo of a big buffet with rich, sinful desserts and another photo of someone being wrapped with cucumber slices over their eyes. Four possible combinations, two of which are fine. However, the other two are not: one is a head scratcher, and the other is downright frightening.

Talk about headline/image interaction effects. Yow.

While this example is obviously contrived, it's not hard to imagine how even well-intentioned MVT trials can lead to unexpected -- and undesired -- combinations. If you were presented with one of these combinations outright, you would immediately veto it. Unfortunately, since it's 1 of 1,500 combinations, you don't get the chance to picture it on its own. It's lost in the math of possibilities.

Now, of course, these bad combinations aren't going to win out in your trials. After you run your MVT experiment long enough to achieve statistical significance, these bad combinations will go away.

But the Russian roulette caveat is that by then a number of legitimate respondents have already been exposed to them.

Any one respondent doesn't know that they're participating in a test of 1,500 different combinations. They don't say, "Heh, heh, sorry guys, this one's wacky, give me an alternative instead." All they know is what they see. To quote The Truman Show, "We accept the reality of the world with which we're presented." If their landing page is disjointed, weird, wacky, misleading, confusing, etc., they simply have a negative experience of your brand.

What percentage of MVT-generated combinations deliver these bad experiences? It's hard to predict. After the fact, you can look at the list of worst performers and see that 10% were awful. But for the sake of argument, let's say it's low, maybe 5%. In which case, 75 of those 1,500 combinations are unfit to represent your brand.

Is your landing page optimization mission worth the risk that 5% of your would-be customers have a bad experience? Particularly as their first impression of you?

The risk here is increased by the fact that many companies are subject to the Pareto principle: 80% of their business comes from 20% of their customers. So on average, if 5% of your landing pages give bad brand and 20% of your customers become superstar customers, then 20% x 5% = 1% of your combinations to would-be customers are resulting in a superstar prospect being mishandled on their first touch.

Statistics are easy to accept impersonally. But when they translate into real people, with high stakes, it's hard to be as cavalier. Every 100th deep-pocketed customer who walks through your door you throw a water balloon at them.

The thing is: you don't have to do it that way.

With A/B testing, you as the marketer get to explicitly visualize each combination before it's in production. Yes, that means a smaller number of combinations are tested, but in exchange you can be guaranteed 100% that no respondent will ever receive a BAD experience.

Every respondent will receive a good experience, and you're simply testing to find the BEST experience.

Narrowing the number of combinations you're trying isn't necessarily a trade off in your results either. The history of direct marketing is rich with examples -- online and offline -- of A/B tests that have produced as significant results as what is advertised by MVT vendors. More combinations does not mean better combinations.

With a smaller number of tests, where you're actually able to visualize each test, you don't waste time with junk combinations or redundant variations. You focus on real hypotheses and clear ideas. This is the sort of thinking that leads to breakthrough experiences that change the game and double or triple your conversion rate.

A/B tests are easy to set up. They're easy to understand. You can visualize exactly what is being tested. And there is no Russian roulette.

-- Scott Brinker