There's no replacement for a good research plan

Yes, you should test your Web properties, especially the transactional ones (e-commerce, subscriptions, digital libraries); we routinely recommend usability testing combined with other techniques such as multivariate testing or A/B testing. Anchor Wave's Anthony Rivera (@ant1832) asked us about The Ultimate Comparison of Multivariate Testing, originally tweeted by always-on-top-of-it Smashing Mag - which solution do we recommend? Good question, since startup costs range from free (Google Website Optimizer) to $33,000+ (Accenture). Where to begin?

Here's the short answer: We recommend selecting the best fit for your company. (You need to decide whether you need both multivariate and A/B testing; check out Avinash Kaushik's simple explanation of the difference. Or check out the handy glossary from Web Analytics Blog.) Essentially, multivariate testing swaps out components of a Web page, while A/B testing swaps out two (or more) versions of a Web page. Both are valuable and merit consideration.

Basic criteria to consider when choosing a solution:
- Scale: How extensive is your digital product?
- Flexibility: Can you modify your test targets?
- Self-service: Do you need or want hand-holding?
- Return on investment: If your digital product is a major source of revenue, shouldn't you invest in testing it with something more than a free solution?

There are two basic fallacies to avoid in considering multivariate testing or A/B testing:

The first fallacy is the idea that testing one page is enough. User experience consists of multiple steps (or screen views) through a Web site, Web application, or kiosk. Whether you select multivariate or A/B testing, the tool you choose must account for task sequences - the steps that users take through your digital product.

This is not to say that one page can't make a huge difference. Our usability testing shows that common, simple missteps - such as an interface change at a critical transaction point - will discourage users from completing a purchase. But the larger point is that measurement must be designed to span the entire process - especially because conventional metrics may fall off the map when the user proceeds to a different URL.

The second fallacy is the idea that ANY product replaces a considered research plan. In our practice, we rarely encounter well-planned research, where the organization routinely measures success against defined goals. (A notable exception: Playboy, where measurement is a regular activity conducted by research professionals.) As with usability testing results, well-planned research is most valuable when shared within the organization. The purpose of conducting research, after all, is to inform your business activities.

If you have not established a research plan - even at a back-of-the-napkin level, with definition of user profiles at a minimum - it's probably too soon to engage in multivariate or A/B testing. Because you really don't know what you need yet.

Are you an enterprise, a medium-sized business, or a small business? Your research initiatives should map to the size of the business. The enterprise MUST invest in professional research (and if it does not, it will eventually fail or under-perform, which amounts to the same thing). The small business can take advantage of free or almost-free tools. The medium-sized business can find solutions somewhere in the middle. Regardless of budget, a basic research plan must be in place before solutions are selected. It's a basic part of your Web strategy.

Agile development and UX: The heart of the matter

Re my recent post on developers and usability:

If the user experience practice in your company was weak before Agile, Agile development isn't going to help things.

If your user experience practice was strong before Agile, it'll remain strong after Agile, and evolve to adapt.

So says Jeff Patton, and I couldn't agree more. No matter how you approach user experience, you must set aside time and resources for it. It has to be in the company DNA - even if that value must be ported from an offline discipline.

Agile development provides fertile ground for user experience design. It may not provide as much support for usability. Regardless, user experience design - most notably including user-facing information architecture and usability testing - must be accounted for in budget and schedule.

The power of data visualization

Great examples of data visualization ( telling the story in images, folks) at io9.com... using science fiction and zombies as accessible pop-culture data points, these information graphics provide excellent examples for reuse in business. Science with a sense of humor. For a more serious take, see C-SPAN's visualization of participation in the presidential debates.

Why are we writing about this? Because in the age of infinite channels, visuals offer an immediacy that words cannot. Saying "the United States is polarized" does not have as visceral an effect as viewing a daily visualization of polling data.

Visualization takes on new power in the torrent of information available to much of the world's population today. Our recent usability testing has shown a predilection towards visualization among younger users; this may indicate a different yet sophisticated way of thinking. Government and military are developing greater visualization capabilities into computer support for emergencies and deployment. Complex sets of data challenge the best and brightest, who see the value of creating visual systems to contextualize the meaning of data.

Consider doing the same with business data, especially online.