Current Issue
Evolution Trumps Usability Guidelines - 15 September 2006
by Jared M. Spool
Read this article in Chinese (translated by Zhen Zhang, proofread by Yimeng Ding)
“Use a Search Box instead of a link to a Search page.”
This is one guideline from the plethora of recently created usability guidelines to help designers produce more usable web sites. It makes sense. After all, there are more than 42 million web sites on the Internet. It should be simple to study these sites and put together a list of “do’s” and “don’ts” that, when followed, will produce easy-to-use sites.
Designing a web site, either usable or unusable, is hard work. There are many details that designers need to take into account, such as browser differences, content management, information architecture, and graphic design. Providing proven guidelines to developers can reduce their already overburdened workload, making one aspect of design that much simpler.
However, we are assuming the guidelines actually result in more usable sites. This is where things start to get murky.
For example, the guideline above suggests that access to your site’s Search functionality should be through a type-in box, not a link. We’ve seen various forms of this guideline in multiple places. The stated implication always is users are less likely to find Search if it is only a link to a separate search page and therefore users are less likely to succeed at accomplishing their goals.
What’s most interesting is that the guideline’s publishers never present any evidence that following it will actually improve the site. The best we’ve seen is one publisher who stated that on their site, when they changed the link to a type-in box, the use of Search increased 91%.
While 91% seems like a lot, if only 1.5% of the site’s total visitors originally used Search, a 91% increase would only bring it up to 2.9%—still unused by 97.1% of the folks visiting the site. In addition, that publisher states that they don’t know if people actually found the information they were seeking because of the change. Given this, is the guideline worth following?
Testing Guidelines
In the last 18 months, we’ve been testing guidelines against our growing collection of usability data. We can see which guidelines actually predict better results.
Testing a guideline is usually straightforward. We can translate a well-written guideline into a hypothesis and test it against our data. For example, we can look at e-commerce sites that have Search boxes on every page versus those sites that don’t and see if the presence of the Search box predicts that shoppers will buy more.
However, not all guidelines are written well. For example, one publisher of more than 200 e-commerce guidelines, had many guidelines we couldn’t test because they were not measurable, such as “Make sure your checkout form fields are written clearly.” When a guideline has a subjective standard (how does a designer know when something is ‘written clearly’?), we can’t form a measurable hypothesis to test.
Case study: Expected Placement of Design Elements
By testing guidelines, we can identify those that will provide the best value. As we’ve been studying guidelines published by various folks, we’ve noticed something very interesting: web usability guidelines are very sensitive to the nature of the tasks and the subtle differences in the content of the site.
We see that a guideline’s effectiveness will change when the wording of the usability test’s tasks is changed slightly. For example, if we tell shoppers to buy a sweater, whether they are interested in buying one or not, we see different results from when we ask them to buy something they really need.
We’ve also noticed that the content dictates whether a guideline will pass the test. Many guidelines for Search, for example, depend on whether the content is uniquely identified (like books or proposed legislation) or not (like apparel or disease symptoms). (See Strategies for Categorizing Categories for more information about uniquely identified content.)
If the tasks we use or the nature of the content can affect whether a guideline passes the test, then that means that we won’t have many generalized guidelines which are always applicable. We need to conduct more research to understand what variables make each guideline effective, so that designers understand the context that makes the guidelines valuable.
Following Untested Guidelines Possibly Harmful
Sites still need designing while we’re waiting for the research. One option is to follow the guidelines as they stand. After all, smart people are producing these guidelines—they’ve got to be worth something.
Unfortunately, three outcomes result from testing guidelines. The guideline, when followed, either predicts success, doesn’t have any effect on success, or, (the worst case scenario) makes the site harder to use.
In a very recent study of some guidelines, we found that following the guidelines actually predicted failure in the clickstreams. For example, one published Search guideline is “Provide a clearly marked link to Advanced Search”. From this guideline, we hypothesized that users who found Advanced Search would succeed more often than those who didn’t.
However, in our data, we found that users who utilized advanced search rarely found their desired content. Use of advanced search functionality actually predicts failure of the task.
It turns out that following this guideline (making advanced search more visible) will potentially decrease the usability of the site. Those designs that didn’t provide advanced search actually did better in our testing.
This means that following untested guidelines is like drinking water from an unidentified source. It might quench your thirst, but it could also make you very ill.
A Practical Alternative to Guidelines: Evolution
If you shouldn’t follow untested guidelines, what should you do? Fortunately, evolution can help.
For most types of sites, there are existing models already out there. Someone who wants to produce a new site with pharmaceutical information, for example, can find dozens of existing pharmaceutical sites.
A small investment in studying how users interact with existing sites can reveal a lot about what works for your users on their tasks. You could easily develop an understanding of the ‘best practices’ and, from that, produce your own guidelines.
Because you will generate your guidelines by directly observing your users, these guidelines are far more likely to be of value than generalized guidelines produced from sites that have little or nothing to do with your work. Evolution has produced these sites and you can identify which have won the ‘survival of the fittest’ competition.
Even if you don’t have many comparable sites to look at, you can still use evolution to your advantage. Create your own mutations by trying out ideas and seeing how they work.
This is exactly how sites like eBay and Amazon have gotten to where they are today. They’ll isolate one of their many servers and change the design of a few pages, just for users of that server. They’ll compare the results with users of the other servers running the ‘existing site’. Because of their traffic volume, they can learn a lot in just a few hours.
You may need to leave things a little longer on your site, but the effect is still the same. You’ll quickly learn what works and what doesn’t, evolving your site into something that meets both the needs of your organization and your users.
Jared is Founding Principal of UIE . He’s been working in the field of usability and design since 1978, before the term “usability” was ever associated with computers. Jared has guided the research agenda and built UIE into the largest research organization of its kind in the world.
Jared is a top-rated speaker at more than 20 conferences every year. He is also the conference chair and keynote speaker at the annual User Interface Conference ., and is on the faculty of the Tufts University Gordon Institute.
This article originally published: Sep 08, 2002
Comments made
Possible Related Articles:
Dave Roberts
Being new to this site I just found your article. I thought it was very good. However, I have to take exception to one sentence: “You could easily develop an understanding of the ‘best practices’ and, from that, produce your own guidelines.”.
I know that your organisation would have no problem with this. However, in my experience this is often a problem for those with less skill. I have seen several teams develop the wrong guidelines from this sort of effort.
In my experience this is why most guidelines collections exist. Only some people are skilled enough to derive guidelines. So for any set, it is important to know if they were created by a team with the right skill. Sadly, as you note, there are many that are developed without the right skill.
Jared M. Spool
Dave,
I hear what you’re saying about having the right skills on the team. This is absolutely critical.
However, in addition to the right skills, the team also needs the right experience. They need to interact with users on a regular basis to learn what the users are trying to do and how they go about doing it.
No amount of third party skills will produce this experience for your users working with your designs.
In my experience, if you have to sacrifice either skills or experience, I’d sacrifice skills. Skills are learned through practice. If a team has experience, they’ll pick up the skills.
A third party may have the skills, but they can’t pick up the experience you need for your design. So, no amount of third-party guideline assistance will work.
Jared
UE team member
Great article. Agree with the above comments. Guidelines should certainly be evaluated against user goal attainment. Also, they should be written with actual goal attainment in mind, not an interpretation of what will create goal attainment. In my company, web design guidelines were written with ‘consistency’ in mind – in fact, design consistency across multiple contexts and goals does not always make sense or facilitate goal attainment! (And actual data helps challenge the ‘consistency for sake of consistency’ mentality.)
Dave Roberts
Jared,
I agree. I was only taking exception to one sentence. At the chance of offending any academics or teachers, I would always prefer to see skills that came from experience rather than those that might come from other places such as training course.
I was not suggesting that third party guidelines are any substitute. What I was cautioning against was running before you can walk: creating guidelines before you knew how to.
A set of guidelines not only relies on the experience of creating good interfaces but on the experience of disseminating guidelines. If you create guidelines that are hard to interpret you can get results that you did not wish for. You need a much wider experience and a broader skill set to ensure you will have guidelines that work.
I’d go with your practical alternative of evolution any day.
Dave
Edwin
Nice article, but what is working for eBay and Amazon can not always be implemented in smaller projects or in projects with a rather small user base.
Many teams are already struggling to finish just one version to go live. In many cases there’s no room for such fancy things.
Also, I am wondering where is the need for the usability expert?
He or she is the one who should have the knowledge to forecast which version will work and which one will not.
No?
In my opinion it is.
A capable usability expert is still cheaper than doing the analysis, the architectural design, the actual development and testing TWICE…
Sure for the big ones, like Amazon it is worth the price. But again, in my opinion almost all websites are a little smaller.
Jared M. Spool
A capable usability expert is still cheaper than doing the analysis, the architectural design, the actual development and testing TWICE…
Assuming the capable usability expert gets it right. If they don’t, then it’s more expensive. (And, unfortunately, they often don’t get it right. I know. I am one.)