There's no replacement for a good research plan

Yes, you should test your Web properties, especially the transactional ones (e-commerce, subscriptions, digital libraries); we routinely recommend usability testing combined with other techniques such as multivariate testing or A/B testing. Anchor Wave's Anthony Rivera (@ant1832) asked us about The Ultimate Comparison of Multivariate Testing, originally tweeted by always-on-top-of-it Smashing Mag - which solution do we recommend? Good question, since startup costs range from free (Google Website Optimizer) to $33,000+ (Accenture). Where to begin?

Here's the short answer: We recommend selecting the best fit for your company. (You need to decide whether you need both multivariate and A/B testing; check out Avinash Kaushik's simple explanation of the difference. Or check out the handy glossary from Web Analytics Blog.) Essentially, multivariate testing swaps out components of a Web page, while A/B testing swaps out two (or more) versions of a Web page. Both are valuable and merit consideration.

Basic criteria to consider when choosing a solution:
- Scale: How extensive is your digital product?
- Flexibility: Can you modify your test targets?
- Self-service: Do you need or want hand-holding?
- Return on investment: If your digital product is a major source of revenue, shouldn't you invest in testing it with something more than a free solution?

There are two basic fallacies to avoid in considering multivariate testing or A/B testing:

The first fallacy is the idea that testing one page is enough. User experience consists of multiple steps (or screen views) through a Web site, Web application, or kiosk. Whether you select multivariate or A/B testing, the tool you choose must account for task sequences - the steps that users take through your digital product.

This is not to say that one page can't make a huge difference. Our usability testing shows that common, simple missteps - such as an interface change at a critical transaction point - will discourage users from completing a purchase. But the larger point is that measurement must be designed to span the entire process - especially because conventional metrics may fall off the map when the user proceeds to a different URL.

The second fallacy is the idea that ANY product replaces a considered research plan. In our practice, we rarely encounter well-planned research, where the organization routinely measures success against defined goals. (A notable exception: Playboy, where measurement is a regular activity conducted by research professionals.) As with usability testing results, well-planned research is most valuable when shared within the organization. The purpose of conducting research, after all, is to inform your business activities.

If you have not established a research plan - even at a back-of-the-napkin level, with definition of user profiles at a minimum - it's probably too soon to engage in multivariate or A/B testing. Because you really don't know what you need yet.

Are you an enterprise, a medium-sized business, or a small business? Your research initiatives should map to the size of the business. The enterprise MUST invest in professional research (and if it does not, it will eventually fail or under-perform, which amounts to the same thing). The small business can take advantage of free or almost-free tools. The medium-sized business can find solutions somewhere in the middle. Regardless of budget, a basic research plan must be in place before solutions are selected. It's a basic part of your Web strategy.