Kinds Of Split Testing Software

Main Menu

Written by Dominic DeLong

Wednesday, 02 June 2010 08:54

by DominicDeLong

I read once about a man who tripled his conversion rate by changing a single letter in his headline. While that kind of improvement is not common (not by a long stretch), you can quite probably double your conversion rate after testing 10 or 20 items on your page. With that said, there are many things to consider when getting started with split testing.

Let me give you an overview, so you can understand the choices you have before you. I'll walk you through the main distinctions that rule the field.

First, there's a difference between multivariate and single variable testing.

Simple split testing is easy. You're just "trying stuff" one thing at a time. You want to try a different headline? Split test it. When that test finishes, you want to "try" something else? Split test that. You can test one alternative version or many. As long as it's a single element of your page you're testing, that's simple split testing. All you need is a tool that rotates the different versions of the headline (or whatever) on your page, and keeps track of which one produces the most sales.

There are a few simple split testing tools available. Some are free and some cost money. Google offers a free tool, though you do have to have an Adwords, and an Analytics account to use it.

With multivariate testing, you can test many factors at the same time on your page. So, instead of just testing your headline, you can test your headline, your offer, the P.S. statement, and your lead image all at the same time. And with multivariate testing, if it's set up right, you can use the same amount of traffic testing many variables that you would use to test a single variable.

So there's the main distinction between single variable testing and multivariate testing. But once you get into the realm of multivariate testing the options seem to multiply again. One difference among multivariate tools is between tools that are hosted on someone else's server, and ones that you install on the same server as your webpages. Hosted solutions can be slower, and more expensive, but you don't have to install any software.

Google Website Optimizer is a hosted plan. But it requires you to have an Adwords account and an analytics account. There is also a reporting delay. Other hosted plans include SiteSpect, and Offermatica. The Split Test Accelerator an MuVar are installed solutions.

Taguchi testing is only one kind of multivariate testing. There are also "full factorial" tools, and random independent rotators. When you run a Taguchi test, you run only a small fraction of the possible combinations of your factor levels. You might, for instance, run only 18 pages to test 4374 combinations. This is made possible by orthongonal arrays. This allows you to find the best combination with out running through all of them. You do have to be careful about your factors interacting negatively with each other, but if you take care as you design your test, you can avoid the bulk of the problem with these. The Split Test Accelerator, Vertster, and Kaizentrack are Taguchi testing tools.

Full factorial tools allow you to cycle through all the combinations for your factors. So if you have 3 headlines, 3 offers, 3 P.S. statements, and 3 images you're testing, you will cycle through 81 pages. One advantage of full factorial testing is that you don't have to worry as much about interaction effects. Good and bad interactions should show up in the data. However, if you want to receive this advantage, you have to run many times the traffic through your test as you would with a Taguchi test. You also can't run as many combinations. Google's website optimizer is a full factorial tool.

The Random independent rotators are a third option. These treat factors independently. So you can just plug in a new factor with as many options as you want at any time, and ignore the fact that other factors are being tested at the same time. This is the most flexible form of testing, but it is also plagued with the most interaction effects.

Now that's just the overview. There are many other features to compare and questions to ask. Does a tester allow you to exclude bots and spiders? Does it allow real time access to data? What kind of reports does it show? What kind of information does it track for each visitor, and is the visitor-level data available for viewing. Etc.

About the Author:

Dominic DeLong has used split testing tools for years to improve landing pages.