There's no replacement for a good research plan

Yes, you should test your Web properties, especially the transactional ones (e-commerce, subscriptions, digital libraries); we routinely recommend usability testing combined with other techniques such as multivariate testing or A/B testing. Anchor Wave's Anthony Rivera (@ant1832) asked us about The Ultimate Comparison of Multivariate Testing, originally tweeted by always-on-top-of-it Smashing Mag - which solution do we recommend? Good question, since startup costs range from free (Google Website Optimizer) to $33,000+ (Accenture). Where to begin?

Here's the short answer: We recommend selecting the best fit for your company. (You need to decide whether you need both multivariate and A/B testing; check out Avinash Kaushik's simple explanation of the difference. Or check out the handy glossary from Web Analytics Blog.) Essentially, multivariate testing swaps out components of a Web page, while A/B testing swaps out two (or more) versions of a Web page. Both are valuable and merit consideration.

Basic criteria to consider when choosing a solution:
- Scale: How extensive is your digital product?
- Flexibility: Can you modify your test targets?
- Self-service: Do you need or want hand-holding?
- Return on investment: If your digital product is a major source of revenue, shouldn't you invest in testing it with something more than a free solution?

There are two basic fallacies to avoid in considering multivariate testing or A/B testing:

The first fallacy is the idea that testing one page is enough. User experience consists of multiple steps (or screen views) through a Web site, Web application, or kiosk. Whether you select multivariate or A/B testing, the tool you choose must account for task sequences - the steps that users take through your digital product.

This is not to say that one page can't make a huge difference. Our usability testing shows that common, simple missteps - such as an interface change at a critical transaction point - will discourage users from completing a purchase. But the larger point is that measurement must be designed to span the entire process - especially because conventional metrics may fall off the map when the user proceeds to a different URL.

The second fallacy is the idea that ANY product replaces a considered research plan. In our practice, we rarely encounter well-planned research, where the organization routinely measures success against defined goals. (A notable exception: Playboy, where measurement is a regular activity conducted by research professionals.) As with usability testing results, well-planned research is most valuable when shared within the organization. The purpose of conducting research, after all, is to inform your business activities.

If you have not established a research plan - even at a back-of-the-napkin level, with definition of user profiles at a minimum - it's probably too soon to engage in multivariate or A/B testing. Because you really don't know what you need yet.

Are you an enterprise, a medium-sized business, or a small business? Your research initiatives should map to the size of the business. The enterprise MUST invest in professional research (and if it does not, it will eventually fail or under-perform, which amounts to the same thing). The small business can take advantage of free or almost-free tools. The medium-sized business can find solutions somewhere in the middle. Regardless of budget, a basic research plan must be in place before solutions are selected. It's a basic part of your Web strategy.

So, how were your e-sales this year? E-com usability

So, how were your e-sales this year? E-com usability may be easier to discuss in recent hindsight, after the dust has settled and the wishful thinking has stopped. How well did you do? What sort of ROI (return on investment) are you seeing?

E-commerce success can be measured in many ways; gross sales numbers are just a start. A more critical number in our view: Abandonment rates. How many people abandoned a shopping cart? Do you know exactly when and where it was abandoned? Most importantly - Do you know why someone abandoned a transaction?

The UK firm userfocus recently discussed "Nine tests for a usable checkout" which certainly merit a look. It's great to see another firm reiterate what we've been observing - and telling our clients - for years. "Ask the minimum number of questions. Use consistent and standard form controls. Place error messages next to the entries that need correcting." Truly commonsensical advice.

While the Nine Tests checklist is a great contribution, we still need to understand our users on a more specific level. The Amazon one-click book shopper is not the Playboy online club member, who is not the engineer joining IEEE online. Mental models, and the reasons for our purchases, play a role. The Amazon book shopper has learned to trust one-click ordering and needs little else. The Playboy user may like the added thrill of the chase that results from "teaser" screens. The engineer at IEEE will grumble if date entry fields don't enforce formats (he's an engineer!).

User experience is about seeing the world through the user's eyes, even if briefly. The personal interaction that comes from user involvement - be it as advisor to the design team, or simply as usability test subject - is irreplaceable. Feel-good aside, sincere understanding of - and concern for - your user is the key to successful business online.

You can and should know how many transactions were abandoned; that's tactical. Understanding that user and designing to his needs is strategic.

Coda: I was about to join an association I really admire last week. They lost me at unenforced date format for my date of birth. Why did they lose me? Because I know they're smarter than that.

Shiny New Toy Syndrome: Crippling your Web presence?

Is your Web presence hobbled by Shiny New Toy Syndrome?

Used to be that the CEO was most prone to Shiny New Toy Syndrome. (Actual quote from a major media boardroom: "ESPN's doing it... it must be working for them... why can't do that too?"). We tech types breathed a sigh of relief when Chief Technology Officers became commonplace, since the CTO is supposed to disabuse the CEO of the dancing sugarplums that usually surround Shiny New Toy.

Turns out CTOs are just as - if not more - susceptible to Shiny New Toy Syndrome (SNTS). Especially when pushed in a flashy new demo from an 800-pound-gorilla vendor. Why does this happen?

Vendor schmoozing aside, this happens because the company has not established requirements. Sure, the toy is shiny, but: Does it meet your requirements? Does it duplicate the functionality of an existing, better-developed tool? Will it create ROI, or will it add a maintenance burden that you didn't plan to meet? Or is this yet another instance of the technologist's worst nightmare: Wishful thinking?

Major American businesses, associations, and scientific projects are losing time, money, and opportunity to SNTS. We see it on far too many projects. The cure: The creation of objective requirements - based on a coherent Web strategy - that set a standard for selection and implementation of technologies.

Play is clearly more fun than work. But CTOs are paid big salaries to make responsible decisions that affect the livelihoods of hundreds of people - not to mention the company's brand.

Toys are for kids.

Great ad - it's about relevant placement

Previously we talked about how important placement is in online advertising and how poor placement can make any ad bad. Now let's talk about which ad types are the most effective and bring the best return on investment.

NON-INTRUSIVE

"This could be good"
Ads that don't intrude on requested content don't annoy users. Text overlays on video windows take advantage of the un-used space without detracting from the desired video.
Above: A text overlay ad displays across the bottom of a video window without obscuring the content.

CONTEXTUAL

"I need one of those"
Users with "banner blindness" still respond to contextual advertising and good SEO. Users blind to popups and interstitials will still click on an ad when it's relevant to what they're already doing.
Above: Chart depicts ROI success per ad type.
(source: Marketing Sherpa and Ad:Tech survey)

RELEVANT

"It applies to me"
Online articles that include related brand information work well on younger users, who view them as more trustworthy than random banners. Advertisements relevant to what users are already looking at are more likely to get traffic.
Above: Graph depicts which ad formats users are most likely to respond to.
(source: ARAnet AdFusion survey)