How to Master UX Testing in an        Agile Design Process                           Jon Innes                           J...
Introductions                       www.uxinnovation.com                        Twitter @innes_jon                        ...
Overview  • Agile and UX       • Jon Innes, approx 35 min  • Agile Metrics An Example       • Jeff Sauro, approx 15 min  •...
Myths I Hope to Dispel Today  • Agile and user testing timelines are incompatible  • You need working code to test with us...
Things ChangeUX INNOVATION LLC                    5
Factors Influencing User Research  • Competitive forces make user experience far    more critical than the past  • Softwar...
The Old Way: Too Big & Too LateUX INNOVATION LLC                                    7
The 3x3 Way: Test Multiple Concepts Early•   3 concepts, 3                 Design              Refine    pages deep•   3 i...
The RITE Way: Refine One Concept Faster                                              REFINE           REFINE              ...
The A/B Way: Refine Details At End                      Build &    Design A                             Test A            ...
The Challenge: Testing in an Agile Way  • Traditional testing does not fit well into agile:       • Recruiting can take up...
What is Agile?  A collection of best practices for developing software  including:  • Holding regular status meetings in a...
What is Scrum?                                             Daily Scrum                                                    ...
What is Lean?  The western interpretation of Taiichi Ohno’s set of  values and practices for removing three types of  inef...
What is Lean UX?  As response to the overall trend of using agile and lean  methods, Lean UX applies the lean principles t...
Where’s The Waste?  • Large formal reports that nobody reads  • Studies that find big problems after it’s too late  • Gath...
What’s Changed?  • Product development cycles are getting faster  • Businesses are now more UX savvy  • In the US 79% of t...
What is Remote Automated User Testing?  • Not just site analytics or surveys  • Task oriented vs. impressions and time on ...
Remote Automated Testing Supports• Task completion rates vs. just  click counts• Time on tasks vs. simple time  on site• C...
Advantages of Automated Testing  Automation of moderation allows you to:       • Conduct large scale studies before having...
Tips for More Agile Testing  • Have personas agreed to by team before sprinting  • Develop an appropriate recruiting strat...
Tips for Remote Automated Testing  • Always run small pilots before large scale tests, in    many cases this can provide v...
Some Things Stay The Same  • Questions to ask on your team       • Have we validated these stories and personas?       • D...
Agile Metrics: An Example                        Jeff Sauro                    Measuring UsabilityUX INNOVATION LLC       ...
Design Wireframes: Don’t Guess Test • Which Design Comp is better ?      • Operationalize into MetricsUX INNOVATION LLC   ...
Trick Test: Tree Test + Click Test• Use Photoshop Mockups with Core Nav/Content• Create a set of realistic task scenarios ...
The Metrics  • Time to locate (task)  • Confidence  • Task Difficulty  • PreferenceUX INNOVATION LLC                      ...
3 Design Comps: 2 Navigation CompsUX INNOVATION LLC                                      28
Add CSS & HTML to Create Clickable HotspotsUX INNOVATION LLC                                               29
Audio & Video With YouEye.com + UserZoom• UserZoom ensures tasks and comps appear in counterbalanced order• UserZoom recor...
Click Map from UserZoom• 73% click on top  navigation• 90% confidence  interval (50% - 93%)UX INNOVATION LLC              ...
Which Design / Tasks Are Faster? (13 Users)             Comp A is Easier   About The Same   Comp B is EasierUX INNOVATION ...
Which Design is Preferred ?Strongly Prefer A    Neutral                   Strongly Prefer B                              M...
Recommendations Iterate Next Designs with Comp B and Nav 2UX INNOVATION LLC                                              34
Find out More                                                  www.LeanUXDenver.com                                       ...
Q&AUX INNOVATION LLC                          36
Thank you for your time!UX INNOVATION LLC                                               37
Thank you for your time!      Stay tuned for new webinars!        Measuring Usability
Upcoming SlideShare
Loading in...5
×

How to Master UX Testing in an Agile Design Process

2,890

Published on

0 Comments
8 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,890
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
39
Comments
0
Likes
8
Embeds 0
No embeds

No notes for slide
  • Today’s market’s, the cost of switching to an alternative is very low to the consumer, it’s just a click a way for an e-Commerce site. With social media it’s also much easier to hear about alternative offerings. These factor make it more critical that your customers are satisfied compared to products of the past. In addition software processes have changed, we have things like Scrum and XP. Some companies release updates as often as daily or weekly. Finally technology lets us do things we couldn’t before, like run studies with remote users, or gather click stream data and create heat maps highlighting user’s mouse clicks.
  • Over the past 10 years, I’ve heard a lot of people who were excellent user researchers complain about how hard it is to run a traditional usability test on an agile project. Most of them had lots of experience at big companies where user research was an established function and had been successful in those environments. That’s because they’re not adapting to the agile mindset.
  • 3x3 is a way of coming up with a conceptual design based on testing early low fidelity prototypes of the alternatives to select the best one. Like Mendel, breeding a better pea. It can save you a lot of rework later doing what Alan Cooper used to call “trying to sand a table into a chair” by ensuring your starting down the right path. It’s ideally suited for new projects or site redesign work. It can be extended to greater numbers of alternatives and more iterations as needed. Key to this type of formative study is recruiting the right participants. Biased results can send you off a cliff.
  • RITE or rapid iterative test and evaluation where you run many small sample tests and instead of waiting until the end to fix issues, you fix issues as you go and continue testing. While compatible with agile, scaling over the long term can be challenging as it’s done traditionally. Keep in mind it’s purpose is to rapidly refine an existing design. While it can be adapted it’s best suited for refining a single design.
  • A/B tests are very efficient at refining final details such as button size, color or placement for an existing site. But back to what Alan Cooper said, you can’t sand a table into a chair. A/B testing is well not suited to fixing major design problems. It’s a power sander. If you rely solely on intercept testing, you may never expand your market beyond the early adopters who might be giving you the wrong feedback for your ideal market.
  • The challenge with most traditional usability methods is they are very resource intensive and often take too long for agile timelines. We need to adapt…
  • These techniques were identified in the 90’s by various people who all met at a ski resort in Utah 11 years ago to coin the term Agile, and wisely to define a set of values and principles to guide future like minded individuals…
  • Ever write a usability report nobody read? That’s an example of muda. Did the report note UI inconsistencies due to a lack of communication on the team. That’s mura.Did you stay up all night to write that report. That’s muri.
  • Janice Fraser coined the term, others like Jeff Gothelf are popularizing it. Today, we’re going to focus on how user testing changes as you become more lean.
  • Sound familiar?
  • Many things, but one is that there are now better tools, like remote and automated user research tools.
  • There are many things that you can front load to make your life easier. It’s like having gas in the car before driving to work. There’s nothing non-agile about that.Take personas. If you work with the product owner or marketing staff to define these, it makes recruiting much easier. The budgets for recruiting from panels should come from marketing budgets. Get the whole team involved in defining tests and metrics, including setting goals for releases to improve metrics.
  • With large scale automation comes the potential for large scale error. Always pilot, or you’ll end up paying participants for nothing. Remember that you could be introducing a bias if you only test with users who respond to your requests in a very short time. Take advantage of the low cost of intercept recruiting when refining for existing users, and save the cost of panels for what they are best suited for. Avoid trying to run 2 hour studies, you’ll end up with lots of drop outs. Beware of non-automated methods that could create lots of work, plan for open ended questionnaires or other analyses that might be bigger than you bargained for.
  • Change only starts when people embrace new values. Tracking objective progress towards the desired outcome by making it visible is key. I’ll assume you care about UX and design, or you wouldn’t be here. The secret sauce in UX is user involvement. The more user involvement throughout the process the more effective your iterations will be.
  • How to Master UX Testing in an Agile Design Process

    1. 1. How to Master UX Testing in an Agile Design Process Jon Innes Jeff Sauro March 29, 2012 Measuring Usability
    2. 2. Introductions www.uxinnovation.com Twitter @innes_jon ‘The Agile UX GURU’ Measuring Usability MeasuringUsability.com Founder at UserZoom alfonso@userzoom.com jeff@measuringusability.com @userzoom Twitter @MsrUsability ‘The Moderator’ ‘The UX Measurement MASTER’UX INNOVATION LLC 2
    3. 3. Overview • Agile and UX • Jon Innes, approx 35 min • Agile Metrics An Example • Jeff Sauro, approx 15 min • Q&A session • Everyone, approx 10 min • Twitter hashtag #uzwebinarUX INNOVATION LLC 3
    4. 4. Myths I Hope to Dispel Today • Agile and user testing timelines are incompatible • You need working code to test with users • It’s hard to scale the usability testing process • Automating usability testing is not possibleUX INNOVATION LLC 4
    5. 5. Things ChangeUX INNOVATION LLC 5
    6. 6. Factors Influencing User Research • Competitive forces make user experience far more critical than the past • Software development processes have changed • Product development cycles are faster, yearly release cycles are almost extinct • Modern technology allows us to do things we couldn’t do beforeUX INNOVATION LLC 6
    7. 7. The Old Way: Too Big & Too LateUX INNOVATION LLC 7
    8. 8. The 3x3 Way: Test Multiple Concepts Early• 3 concepts, 3 Design Refine pages deep• 3 iterations of design & test Combine• 3 or more users per iteration• Test low fidelity first, then high• Improve concept each iteration Refine• Combine best elements into 1 Design concept & iteratehttp://www.carolrighi.com/documents/Righi%203x3.pdfUX INNOVATION LLC 8
    9. 9. The RITE Way: Refine One Concept Faster REFINE REFINE REFINE DESIGN DESIGN DESIGNSchedule image from Brian Keith Sullivanhttp://bigdesignevents.com/2011/08/5-ways-to-improve-your-next-agile-project-with-the-rite-method/RITE: Method http://en.wikipedia.org/wiki/RITE_MethodUX INNOVATION LLC 9
    10. 10. The A/B Way: Refine Details At End Build & Design A Test A Launch A Update Build & Design B Test B Launch B Repeat • Design & build multiple designs and compare details • By nature less subject to test condition biases • Easy to get large sample sizes & statistically reliable metrics • Not so great at providing deeper insights on “why”UX INNOVATION LLC 10
    11. 11. The Challenge: Testing in an Agile Way • Traditional testing does not fit well into agile: • Recruiting can take up to two weeks • Reports and test plans take too long • You often don’t have a stable UI until too late • What can we do? Adapt existing methods? • Use old methods planning outside of agile timeframes? • 3x3 solves the “waiting till the end” for working UI problem • RITE is a step in the right direction, it speeds up iterations • A/B split tests are effective for tweaking final detailsUX INNOVATION LLC 11
    12. 12. What is Agile? A collection of best practices for developing software including: • Holding regular status meetings in a standup format • Requirements in the form of user stories or use cases • Defining tests for code before writing the code itself • Working in short iterations to create incremental improvements Common variations include Scrum & XP and all have a mindset that is best described as “lean”UX INNOVATION LLC 12
    13. 13. What is Scrum? Daily Scrum Meeting 24 Potentially Product Sprint hrs Shippable Backlog Backlog Product 2-4 Increment week SprintImage from: http://www.mountaingoatsoftware.com/UX INNOVATION LLC 13
    14. 14. What is Lean? The western interpretation of Taiichi Ohno’s set of values and practices for removing three types of inefficiencies: • 無駄 = muda wasted outputs • 斑 = mura unnecessary inconsistencies • 無理 = muri wasted efforts Lean Startups and Lean UX are based on these values and practicesUX INNOVATION LLC 14
    15. 15. What is Lean UX? As response to the overall trend of using agile and lean methods, Lean UX applies the lean principles to UX: • Breaking deliverables down to fit into agile timelines so they are “just in time” • Reducing wasted effort associated with internal deliverables • Reducing the variability in UX deliverables that create waste • Collaborating more closely with non-UX team members • Getting user feedback earlier and on a more regular basisFor more on Lean UX see:http://www.slideshare.net/balancedteam/02-sat-janice-lean-ux-landscapehttp://joshuaseiden.com/blog/2011/09/what-makes-it-lean/http://uxdesign.smashingmagazine.com/2011/03/07/lean-ux-getting-out-of-the-deliverables-business/UX INNOVATION LLC 15
    16. 16. Where’s The Waste? • Large formal reports that nobody reads • Studies that find big problems after it’s too late • Gathering unnecessarily large sample sizes • Retesting stuff over and over the manual way • Developing and launching real stuff just to learn what you could learned from simpler prototypesUX INNOVATION LLC 16
    17. 17. What’s Changed? • Product development cycles are getting faster • Businesses are now more UX savvy • In the US 79% of the population is now online • Remote automated usability testing solutions have become more: • More capable • Less costly • Easier to useUX INNOVATION LLC 17
    18. 18. What is Remote Automated User Testing? • Not just site analytics or surveys • Task oriented vs. impressions and time on site • Not limited to existing users of your site • Can be used on prototypes or working code • No moderator required • Combines the benefits of lab testing with the scalability of A/B testingUX INNOVATION LLC 18
    19. 19. Remote Automated Testing Supports• Task completion rates vs. just click counts• Time on tasks vs. simple time on site• Click path data by task• Heat maps showing where users click on a page• Task specific and overall comments• Structured survey data (such as SUS)• Video highlights (of user’s screens)UX INNOVATION LLC 19
    20. 20. Advantages of Automated Testing Automation of moderation allows you to: • Conduct large scale studies before having working code • Easily test competitive sites against your site • Compare functionality from past designs with minimal effort including changes of any size with ease • Gather large sample sizes like A/B testing and actionable insights comparable to traditional usability data • Gather data from users who do not currently visit your site • Reduce both testing costs and associated developmentUX INNOVATION LLC 20
    21. 21. Tips for More Agile Testing • Have personas agreed to by team before sprinting • Develop an appropriate recruiting strategy early • Define hypotheses & target metrics for stories in the backlog working with your product owner • Track usability metrics in a public place • Test early design concepts not just final details • Automate user testing for stories as you go & retest • Waiting until the end is the waterfall wayUX INNOVATION LLC 21
    22. 22. Tips for Remote Automated Testing • Always run small pilots before large scale tests, in many cases this can provide valid data anyway • Allow at least a week for tests with panel participants • Combine intercept recruiting with panels to minimize costs while reaching both new and existing users • Keep the participant tasks to a small number to minimize drop out rates • Use content analysis techniques to analyze open ended survey dataUX INNOVATION LLC 22
    23. 23. Some Things Stay The Same • Questions to ask on your team • Have we validated these stories and personas? • Did the iteration incorporate user feedback? • Can they use it? • Do they like it? • Would they recommend it to a friend? • Did we make it measurably better? • UX principles remain the same in agile • Iterations without user tests are a lost learning opportunity • Waiting until the end to user test is the waterfall way • The definition of “done” can only be determined by usersUX INNOVATION LLC 23
    24. 24. Agile Metrics: An Example Jeff Sauro Measuring UsabilityUX INNOVATION LLC 24
    25. 25. Design Wireframes: Don’t Guess Test • Which Design Comp is better ? • Operationalize into MetricsUX INNOVATION LLC 25
    26. 26. Trick Test: Tree Test + Click Test• Use Photoshop Mockups with Core Nav/Content• Create a set of realistic task scenarios • For example, where would click to download drivers for your printer?• Recruit participants online to try and have them find the information.UX INNOVATION LLC 26
    27. 27. The Metrics • Time to locate (task) • Confidence • Task Difficulty • PreferenceUX INNOVATION LLC 27
    28. 28. 3 Design Comps: 2 Navigation CompsUX INNOVATION LLC 28
    29. 29. Add CSS & HTML to Create Clickable HotspotsUX INNOVATION LLC 29
    30. 30. Audio & Video With YouEye.com + UserZoom• UserZoom ensures tasks and comps appear in counterbalanced order• UserZoom records time and responses to questionsUX INNOVATION LLC 30
    31. 31. Click Map from UserZoom• 73% click on top navigation• 90% confidence interval (50% - 93%)UX INNOVATION LLC 31
    32. 32. Which Design / Tasks Are Faster? (13 Users) Comp A is Easier About The Same Comp B is EasierUX INNOVATION LLC 32
    33. 33. Which Design is Preferred ?Strongly Prefer A Neutral Strongly Prefer B Mean + 80% Confidence Int. UX INNOVATION LLC 33
    34. 34. Recommendations Iterate Next Designs with Comp B and Nav 2UX INNOVATION LLC 34
    35. 35. Find out More www.LeanUXDenver.com Sept 19-21st 2012 Hyatt Convention Center www.uxinnovation.com Denver, CO Twitter @innes_jon @LeanUXDenver Measuring Usability MeasuringUsability.com jeff@measuringusability.com Twitter @MsrUsabilityUX INNOVATION LLC 35
    36. 36. Q&AUX INNOVATION LLC 36
    37. 37. Thank you for your time!UX INNOVATION LLC 37
    38. 38. Thank you for your time! Stay tuned for new webinars! Measuring Usability
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×