There's no replacement for a good research plan

Yes, you should test your Web properties, especially the transactional ones (e-commerce, subscriptions, digital libraries); we routinely recommend usability testing combined with other techniques such as multivariate testing or A/B testing. Anchor Wave's Anthony Rivera (@ant1832) asked us about The Ultimate Comparison of Multivariate Testing, originally tweeted by always-on-top-of-it Smashing Mag - which solution do we recommend? Good question, since startup costs range from free (Google Website Optimizer) to $33,000+ (Accenture). Where to begin?

Here's the short answer: We recommend selecting the best fit for your company. (You need to decide whether you need both multivariate and A/B testing; check out Avinash Kaushik's simple explanation of the difference. Or check out the handy glossary from Web Analytics Blog.) Essentially, multivariate testing swaps out components of a Web page, while A/B testing swaps out two (or more) versions of a Web page. Both are valuable and merit consideration.

Basic criteria to consider when choosing a solution:
- Scale: How extensive is your digital product?
- Flexibility: Can you modify your test targets?
- Self-service: Do you need or want hand-holding?
- Return on investment: If your digital product is a major source of revenue, shouldn't you invest in testing it with something more than a free solution?

There are two basic fallacies to avoid in considering multivariate testing or A/B testing:

The first fallacy is the idea that testing one page is enough. User experience consists of multiple steps (or screen views) through a Web site, Web application, or kiosk. Whether you select multivariate or A/B testing, the tool you choose must account for task sequences - the steps that users take through your digital product.

This is not to say that one page can't make a huge difference. Our usability testing shows that common, simple missteps - such as an interface change at a critical transaction point - will discourage users from completing a purchase. But the larger point is that measurement must be designed to span the entire process - especially because conventional metrics may fall off the map when the user proceeds to a different URL.

The second fallacy is the idea that ANY product replaces a considered research plan. In our practice, we rarely encounter well-planned research, where the organization routinely measures success against defined goals. (A notable exception: Playboy, where measurement is a regular activity conducted by research professionals.) As with usability testing results, well-planned research is most valuable when shared within the organization. The purpose of conducting research, after all, is to inform your business activities.

If you have not established a research plan - even at a back-of-the-napkin level, with definition of user profiles at a minimum - it's probably too soon to engage in multivariate or A/B testing. Because you really don't know what you need yet.

Are you an enterprise, a medium-sized business, or a small business? Your research initiatives should map to the size of the business. The enterprise MUST invest in professional research (and if it does not, it will eventually fail or under-perform, which amounts to the same thing). The small business can take advantage of free or almost-free tools. The medium-sized business can find solutions somewhere in the middle. Regardless of budget, a basic research plan must be in place before solutions are selected. It's a basic part of your Web strategy.

Social media and culture fit: You can't fake it

While listening to my colleague Mike Schmidt's podcast on "social marketing," (a good primer for newbies, featuring Jerry Harkins of MacTutorTucson.com) I considered my setup experience with Google Buzz this morning, and my Mayorship of the Hotel Congress on Foursquare, and the status updates I posted to Facebook and LinkedIn on the way. Another day in the roiling surf of social media. Which service gets your allegiance and time?

As with all digital media, it depends on your users and your business. Some truisms are helpful; we know that bands prosper on MySpace, and charitable causes flourish on Facebook. But in our view, the key to social media is culture. Are you willing to be what Schmidt calls a "good social media citizen"? Are you willing to help people solve problems even when the dollar is not immediately attached? Or are you only willing to invest time in exchange for dollars?

The greatest challenge for conventionally structured organizations is to become those "good social media citizens" when lack of transparency and buyer-beware have been the standard marching orders - until the emperor-has-no-clothes power of the Web came along.

The selection of social media most valuable to your business is a tactic. The strategy: Support a corporate culture whose values are revealed by social media. Successful strategic values include collaboration, humility, helpfulness, expertise, hard work. Going the extra mile. You can't fake these values in social media. At least, not for long.

You should never ignore a technology because you don't know what it is. In our practice, we frequently encounter Web teams that don't use the very technologies about which they are expected to make decisions. "You have to try it to understand it," says Schmidt, the principal of Anchor Wave, and there we agree wholeheartedly. Kudos to managers who understand they may have to rock the boat by introducing - dare we say it - meritocracy. Who are willing to experiment. Who understand that workers must demonstrate interest AND capability. In some organizations, where seniority and longevity rule, this is a tough battle. But it must be fought, and it must be won.

Those who hold back until someone else figures it out (a philosophy proudly stated to me by the principal of a New York "digital design agency" a couple of years ago) are simply walking into the theatre in the middle of the metaphorical movie. You'll never catch up with the story, much less the subtleties. How can you make good decisions about digital media that way? The answer is, you can't.

But where do we find the time to filter out the valuable social media endeavors from the less productive ones? Find that time - because as co-podcaster Jerry Harkins points out, people filter out conventional marketing messages without even realizing the are doing so. People are likelier to trust peers over authority - a finding Interface Guru first realized during a usability test for Clickability in San Francisco in 2001 - and your conventional marketing message may be pointless in 2010. Harkins directs listeners to The Purple Goldfish Project - a great resource on how businesses (and people) can - literally - blow people away with a gesture of respect and service. When you see the examples, you realize the winning companies' strategies are less about what they spend, and more about the core values they hold dear. What can you learn about your own brand that's worth sharing with the world via social media?

You can't have a robber-baron culture and a collaborative social media strategy. One of the two will lose.