Notable passages from
Don't Make Me Think by Steve Krug

From the usability portion of the book, which is tremendously valuable in the original edition (reviewed here), and has been expanded in the new edition.

New Riders, Indianapolis, 2000
(a revised second edition with additional content was published in 2005)

In a usability test, one user at a time is shown something ... and asked to either (a) figure out what it is, or (b) try to use it to do a typical task.

[ ‹ ]

Here are the main things I know about testing:

• If you want a great site, you've got to test. After you've worked on a site for even a few weeks, you can't see it fresh anymore. You know too much. The only way to find out if it really works is to test it.
• Testing one user is 100 percent better than testing none. Testing always works. Even the worst test with the wrong user will show you things you can do that will improve your site.
• Testing one user early in the project is better than testing 50 near the end. A simple test early—while you still have time to use what you learn from it—is almost always more valuable than a sophisticated test later.
• The importance of recruiting representative users is overrated. It's good to do your testing with people who are like the people who will use your site, but it's more important to test early and often.
• The point of testing is not to prove or disprove something. It's to inform your judgment. People like to think, for instance, that they can use testing to prove whether navigation system "a" is better than navigation system"b", but you can't. No one has the resources to set up the kind of controlled experiment you'd need. What testing can do is provide you with invaluable input which, taken together with your experience, professional judgment, and common sense, will make it easier for you to choose wisely—and with greater confidence—between "a" and "b."
• Testing is an iterative process. Testing isn't something you do once. You make something, test it, fix it, and test it again.
• Nothing beats a live audience reaction.

[ ‹ ]

Lost our lease, going-out-of-business-sale usability testing

Usability testing has been around for a long time, and the basic idea is pretty simple: If you want to know whether your software or your Web site or your VCR remote control is easy enough to use, watch some people while they try to use it and note where they run into trouble. Then fix it, and test again.

• Number of users per test - Three or four. [The first three users are very likely to encounter all of the most significant problems.]
• Recruiting effort - Grab some people. Almost anybody who uses the Web will do. [We're all beginners under the skin. It's usually not a good idea to design a site so that only your target audience can use it. Experts are rarely insulted by something that is clear enough for beginners.]
• Where to test - Any office or conference room.
• Who does testing - Any reasonably patient human being.
• Advance planning - Tests can be done almost any time, with little advance scheduling.
• Preparation - Decide what you're going to show.
• What/when do you test? - Run small tests continually throughout the development process. ["Get it" testing is just what it sounds like: show them the site, and see if they get it—do they understand the purpose of the site, the value proposition, how it's organized, how it works, and so on. Key task testing means asking the user to do something, then watching how well they do. As a rule, you'll always get more revealing results if you can find a way to observe users doing tasks that they have a hand in choosing.]
• Cost - About $300 (a $50 to $100 stipend for each user and $20 for three hours of videotape).
• What happens afterwards - Each observer writes one page of notes the day of the test. The development team can debrief the same day.

[ ‹ ]

What to do if you're the facilitator

• Try the test yourself first. The day before the test, try doing whatever you're going to ask the test participants to do and make sure that you can do it in the time allotted.
• Protect the participants. It's your responsibility to prevent any damage to your test users' self-esteem. Be nice to them. If they get stuck, don't let them get too frustrated.
• Be empathetic. Be kind, patient, and reassuring. Make it clear to them that you know they're not stupid.
• Try to see the thought balloons forming over their heads. Whenever you're not sure what they're thinking, ask them. You're trying to see what their expectation is at each step and how close the site comes to matching that expectation.
• Don't give them hints about what to do. If they say, "I'm not sure what to do next," you should say, "What do you think you should do?" or "What would you do if you were at home?"
• Keep your instructions simple. Don't be afraid to keep repeating them; it will be more boring to you than to the user.
  
  • "Look around the page and tell me what you think everything is and what you would be likely to click on."
  • "Tell me what you would click on next and what you expect you would see then."
  • "Try to think out loud as much as possible."
• Probe, probe, probe. You have to walk a delicate line between distracting or influencing the users and finding out what they're really thinking, which they may not know themselves.
• Don't be afraid to improvise. For instance, if the first two users get hopelessly stuck at the same point and it's obvious what the problem is and how to fix it, don't make the third user struggle with it needlessly.
• Don't be disappointed if a user turns out to be inexperienced or completely befuddled. You can often learn more by watching a user who doesn't get it than one who does. Because more experienced users have better coping strategies for "muddling through," you may not even notice that they don't get it.
• Make some notes after each session. Always take a few minutes right after each test session to jot down the main things that struck you. If you don't do it before you start the next test, it will be very hard to remember what they were.

[ ‹ ]

What to do if you're observing

• Do they get it? Without any help, can the users figure out what the site or the page is, what it does, and where to start?
• Can they find their way around? Do they notice and understand the site's navigation?
• Head slappers. [T]he user will do something, or not do something, and suddenly everyone who's observing the session will slap his or her forehead and say, "Why didn't we think of that?" or "Why didn't we ever notice that?"
• Shocks. [Y]ou'll say, "How could she [the user] not notice that?" or "How could she not understand that?" For instance, you might be shocked when someone doesn't notice that there is a menu bar at the top of each page.... [T]he solution to shocks won't always be obvious and they may send you back to the drawing board.
• Inspiration. Users will often suggest a solution or the germ of a solution to a problem that you've struggled with for a long time. Very often the solution will be something you'd already thought about and rejected, but just watching someone actually encounter the problem will let you see it in a whole new light.
• Passion. What are the elements of the site that users really connect with? Be careful not to mistake mere enthusiasm for passion, though. You're looking for phrases like "This is exactly what I've been looking for!" or "When can I start using this?"

Here are some things to keep in mind when you're observing:

• Brace yourself. It can be emotionally wrenching to watch someone have a negative reaction to something you've poured your soul into. The mantra you want to have in your head is not "It's not working!" but, rather, "What will take to fix it?"
• Don't panic. Try to resist the temptation to jump to any conclusions until you've seen at least two users, preferably three.
• Be quiet. There's nothing more disconcerting for a test participant than the sound of laughter—or groans—coming from an adjoining room when she's having trouble using the site.
• Remember that you're grading on a curve. When a participant who uses the Internet two hours a day doesn't know how to type a URL, don't think, "Sheesh What a dolt." Think, "How many people are there just like that out there? Can we afford to lose all of them as users?"
• Remember that you're seeing their best behavior. When you're watching a test you need to remember that people will tend to read Web pages much more thoroughly and put more effort into figuring things out in a test situation than they will in real life.... So when they can't figure something out, you have to realize that they're trying much harder than most people will and they still can't get it.
• Pay more attention to actions and explanations than opinions. Opinions expressed during user tests are notoriously unreliable. People will often exaggerate their opinions—positive and negative—because they think you want them to express strong opinions.

[ ‹ ]

Review the results right away

The important things that you learn from usability testing usually just make sense. They tend to be obvious to anyone who watches the sessions.

• Always consider tweaking first. [T]here's a tendency to be spooked by any bad user reactions, particularly after the first round of testing. Before scrapping anything, always stop and think, "What's the least we could do that might fix the problems we're seeing?" If it seems like a particular tweak has a reasonable chance of working, mock it up and test it as soon as possible.
• Focus on specifics. Try to avoid sweeping statements and focus on the precise points where people seemed to go astray.
• Tweak, but verify. Remember, this is a cyclic process. Since you'll be doing another test soon, you can afford to try some tweaks before scrapping anything.
• If the problem is deep, bite the bullet. If you've got a deep-seated problem, a post-test debriefing meeting is a good place to finally face it. Once it's out on the table, it usually turns out to be more manageable than it seems when you're not talking about it.
• Remember: It's almost never too late to challenge basic assumptions. Rethinking the basics doesn't necessarily mean changing everything, and it often turns out that the solution to the problem is simpler than you've feared.

[ ‹ ]

Typical problems

• Users are unclear on the concept. They just don't get it. They look at the site or a page and they either don't know what to make of it, or they think they do but they're wrong.
• The words they're looking for aren't there. This usually means that either (a) the categories you've used to organize your content aren't the ones they would use, or (b) the categories are what they expect, but you're just not using the names they expect.
• There's too much going on. Sometimes what they're looking for is right there on the page, but they're just not seeing it.

[ ‹ ]

Resist the impulse to add things. When it's obvious in testing that users aren't getting something, most people's first reaction is to add something, like an explanation or some instructions. Very often, the right solution is to take something (or things) away that are obscuring the meaning, rather than adding yet another distraction.

[ ‹ ]