21st Century User Testing: Test It, Cheap Fast and Often


Published on

It’s time for a new approach to usability testing.

White Paper By Dan Willis, Associate Creative Director: Government Services

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

21st Century User Testing: Test It, Cheap Fast and Often

  1. 1. 21st Century User Testing: Test It Cheap, Fast and Often 21st Century User Testing: Test It Cheap, Fast and Often1 © Sapient Corporation 2011
  2. 2. 21st Century User Testing: Test It Cheap, Fast and Often 21st Century User Testing: Test It Cheap, Fast and Often Four decades of usability testing has done little to make our believing that we can somehow create reports that are so world any more usable. This is a shame because usability effective that stakeholders find them irresistible. But that’s testing is such a unique and powerful development tool. like believing there’s an umbrella so effective that it can By watching people work through specific tasks using the stop rain from falling. interfaces, applications and services we build, we avoid our own biased views of what people want and need. It provides We kill ourselves to catalogue every possible usability issue, a much greater chance for us to meet users’ requirements but we need to step back and ask, how many issues from and align with their expectations. such an exhaustive list actually get resolved in a typical project? 20 percent? If we got to even half of the issues You would expect the use of such an impressive tool for 40 raised by testing, it would be a miracle. years to have a positive influence on the overall usability of all solutions. So why hasn’t this been the case? For years, we had long debates about the appropriate number of usability test participants until finally the For one thing, clients may listen intently when we present our research determined that 85 percent of problems surfaced findings, but the data doesn’t seem to stick. In organizations with five test participants, and we all nodded our heads and with weak project management, the data doesn’t stick felt secure in our methods. But what practical value have because no one is directly responsible for acting on we seen in 85 percent discovery when we have only been usability testing results. In organizations with strong project able to address a handful of issues? management, the data doesn’t stick because—whether or not it fits the project manager’s job description—everyone Even with all of its flawed elements, traditional practices assumes that the project manager will integrate research have given us glimpses of usability testing’s true potential. results into projects. This means that no one other than the User Interface Engineering (UIE), a leading user research project manager takes personal responsibility. firm, tested an e-commerce Web site for a major corporate client where first-time buyers encountered a screen similar Another influence on usability testing’s lack of long-term to the one in Figure 1. impact is the way we’ve traditionally reported results. We craft the most comprehensive analysis and communication of our research possible, but despite our efforts people rarely read the report. Many of us have taken this personally, seeing it as a reflection on the quality of our work and2 © Sapient Corporation 2011
  3. 3. 21st Century User Testing: Test It Cheap, Fast and Often There is no magic that will make traditional usability testing data suddenly stick within an organization, no secret formula for creating reports that stakeholders will actually read, and no new process that will allow projects to address a majority of issues raised. The only real answer is to radically change how we do usability testing, with the most radical part being that we actually need to be doing less. THE WAY WE’VE ALWAYS DONE USABILITY Figure 1. The Login Screen with a Register Button TESTING Participants resented being forced to register before making a purchase. As one person said, “I’m not here to This is how it typically works. You pay a recruiter to find enter into a relationship, I just want to buy something.” The people similar to your users. You rent a facility and invite client’s intent had been to make shopping easier, but their key stakeholders to observe the test. Your test participants solution actually made it much more difficult. Based on the work their way through a series of tasks using a fully- or usability testing results, the client changed the “Register” semi-functional version of the Web site or application your button to a “Continue” button (as shown in Figure 2). This team is building. made it more obvious that registering was optional. You review the video of each session again and again until you’ve gleaned every possible data point about every possible usability issue. You combine, compare, and classify all of the data and; issues are prioritized and placed into a comprehensive catalog of concerns. You take screenshots from the interface and snapshots of the testing and combine them with your insightful, exhaustive analysis. Then you deliver it all to the client in one giant, glorious report. Figure 2. The Login Screen with a Continue Button Starting the day they changed the button, the number In a meeting you present your data and watch as your client of customers purchasing jumped 45 percent. In the first scrawls notes on the report cover. After skimming the month, the client made $15 million more than it had in the executive summary, the client tosses your shiny, full-color, month before the button was changed. In the first year, it indexed report on a shelf with a stack of others, and she made $300 million more in sales than it had the year before. buries the digital version deep in her hard drive. Your keen insights and analysis eventually blend in with the existing Despite hundreds of thousands of usability tests and white noise of her organization’s developmental mythology. projects as successful as the $300,000 button example, the quality of usability across all products and interfaces If you’ve presented well and stayed within your budget— hasn’t improved. It certainly wasn’t the result of a lack of usually between $20,000 and $40,000, depending on the effort. We strove to find the right number of participants to number of types of users and your team’s travel costs—you surface the greatest number of issues. We carefully crafted get the chance to do it all over again for a later phase or for comprehensive analysis and presented exhaustive details a different project. to justify the substantial investments of our clients. But we ignored or sacrificed the practical in our dedication to a highly principled approach.3 © Sapient Corporation 2011
  4. 4. 21st Century User Testing: Test It Cheap, Fast and Often DOING LESS Test with three people and recruit loosely When a client needs usability testing, the package above Instead of trying to find all possible issues with no concern is what we typically suggest. Depending on the client, for how many we can actually address, our revised, more project and budget, we might get asked for a less ambitious practical goal is to concentrate on the most serious approach and the cheaper we need to go, the more guerilla usability problems. These tend to be about navigation, page we get with our tactics. We find ways to get usability testing layout, and other issues that don’t require participants with done on-site and we find ways to do things on a smaller domain knowledge, so recruitment can be less precise. scale. We assure our client that the reduced scope will still Three participants a month will surface the significant result in credible data, but we’re almost apologetic about problems and running so few sessions makes it possible to our methods. have both the testing and the debriefing on the same day. But to unleash usability testing’s true potential, we must As with traditional methods, testing is one-on-one with now adopt our guerilla tactics as the norm rather than sessions lasting up to an hour. A facilitator in the same mumble about them in embarrassment. Steve Krug, room follows a script, asking each participant to work author of Don’t Make Me Think and Rocket Surgery Made through the same series of tasks and asking them to think Easy, provides a blueprint for a more practical approach to aloud as they do so. usability testing. Conduct usability testing on-site and make it Test whatever you have at the same time an all-inclusive spectator sport every month Testing at off-site facilities automatically limits the number Testing should start as early in the project as possible, even of people who can attend. By moving it to the client’s offices earlier than you think makes sense. Organizations have (or to space extremely convenient to the client) we can historically done usability testing in a single, exhaustive involve every member of the team, every stakeholder and round at the end of the development cycle. While this everybody in the client’s organization with an interest in the approach generates a high volume of data, little of it is project. Watching users struggle with tasks transforms the useful because it arrives too late in the development cycle. people who see it and will generate the support we always By committing to monthly testing, we can change the expected (but never got) from old-school reports. culture of the project and its relationship to research. Testing takes place in one room as the audience watches It’s always tempting to wait for the thing you’re building in another. The two rooms should be far enough apart so to get to critical mass before testing it. But with testing that the test participants remain unaware of their audience. materials required in advance, the data can already be There are no special requirements for the rooms (they can weeks old before it’s available to the team (assuming that be regular offices or conference rooms). The room where testing takes at least a week and analysis takes at least the testing takes place should have a computer with Internet another week). Also, research results served in one or two access, a monitor, a mouse, and a keyboard. The computer bulky servings (as it is with traditional methods) is harder should have screen recording software (e.g., Camtasia) and for an organization to digest than data in regular, smaller screen sharing software (e.g., WebEx) so the audience in servings delivered over the course of the project. the second room can watch in real time what’s happening on the test participant’s screen. An audio solution as simple as a telephone set on speakerphone will be needed so the audience can hear the test participant (although the test participant should never hear the audience).4 © Sapient Corporation 2011
  5. 5. 21st Century User Testing: Test It Cheap, Fast and Often Get everybody who watches the sessions to Improving usability almost always requires taking things come to the debriefing away rather than adding things. The solutions we are looking for should be as simple and unambitious as After completing the three sessions and sending the test possible, because we have to implement them before the participants away, the team discusses what they saw over a next month’s regularly scheduled testing. lunch that can be as simple as pizza and sodas. Each team member brings with them a list of the three most important usability problems they saw for each session. By socializing the research, each member of the team becomes a source CONCLUSION of data. Let people talk about whether they thought it was Instead of being embarrassed by our guerilla tactics, important if a test participant did or did not click a button. we should be recommending their use as a regular part Let them have that argument along with any others that of every engagement. If a client is determined to pay big come up. If you don’t have the facilitation skills to maintain money for traditional usability testing, we should do it and a healthy and productive conversation, bring the talent do it well. We provide usability testing as a service, but it’s in. (There is usually someone on your team that can fill not what we’re selling when we make our pitch. What we’re in as a facilitator.) By making usability testing a spectator selling, what we deliver, is our client’s success. sport, research results can be internalized by the whole organization. Regularly scheduled, small-scale, highly practical usability testing will deliver success and change our clients’ Focus ruthlessly on only the most serious organizations. The dirty little secret of 21st Century usability problems. testing is that if we can’t talk the client into paying for it, we A successful debriefing session will result in a list of the should do it anyway and Sapient should pick up the tab. It’s most serious usability problems encountered, along with a smart and cheap and it greatly increases our ability to create list of the problems that will be fixed before the next month’s wildly successful interfaces, applications, and services. round of testing. Severity will be gauged by considering how many people will experience an issue and whether it will It’s time for a new approach to usability testing. By focusing cause a serious problem for the people who experience it on the practical application of user testing data rather than or if it will just be an inconvenience. clinging to principles that impractically aspire to perfection, the practice of usability testing will finally reach its potential Failing to deal with a serious problem ruthlessly guarantees in the first part of the this new century. that the problem will show up in future testing. It takes a severe ruthlessness to identify a serious problem and make it go away. As a follow-up to the debriefing, we send a short e-mail that briefly covers what was tested, the list of tasks the participants performed, and the list of problems to be fixed before the next test. The e-mail should take less than two minutes to read (making it more likely that most stakeholders will read it). In addition, it should be delivered in a format readable from any device (which means stakeholders can read it wherever and whenever they prefer). This e-mail replaces the voluminous report delivered and presented at the end of a traditional usability testing engagement.5 © Sapient Corporation 2011
  6. 6. 21st Century User Testing: Test It Cheap, Fast and Often Subject Matter Expert: Dan Willis Dan Willis is an Associate Creative Director for Sapient Government Services. He has been designing web sites since the mid-1990s. Dan is prominent in the user experience community in the U.S., having presented at the last two South by Southwest Interactive Festivals as well as numerous Usability Professionals’ Association, Information Architecture Institute, and Interaction Design Association conferences. He is also the creator of uxcrank.com, a resource for UX professionals.6 © Sapient Corporation 2011