Faster Usability Testing in an Agile World - Agile UX Virtual Summit 2017 by UXPin

UXPin Agile UX
Virtual Summit
June 15, 2017
Carol Smith
@carologic
Faster Usability Testing
in an Agile World

IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Design at IBM Watson
Design Thinking & Agile at IBM with Vera Rhoads
16th June 2017, 2:00-3:00 PM PDT

• Researcher and interaction designer on each squad
• Visual designer often shared
• Designers Consult on other teams as needed
Watson - Squads include UX Designers

• Iterative
• Repeatable
• Incremental
• Get to 80% confidence
Light Design - Lean UX

• Just enough
• Identify and solve problems
• Use each study to pick up information
• Think ahead (not make ahead)
– Concept Cars (vision)
– Know the North Star
No BDUF* at Watson
*Big Design Up Front

• Fit testing into Agile process
• Never too early - whatever is ready
• Regular testing
Image of sign, created by IBM’er Thomas J. Watson
http://www.newyorker.com/cartoons/bob-mankoff/computer-games

• Representative users
• Doing real tasks
• Being observed
• Using prototypes
or live products
Usability Testing is…
http://creativecommons.org/licenses/by-sa/2.0/
http://www.flickr.com/photos/raphaelquinet/513351385/sizes/l/in/photostream/
http://www.flickr.com/photos/raphaelquinet/

• There is a difference
• User testing infers testing people
– Can the participant figure out our design work?
• Usability testing puts emphasis on what is being tested
– Reduces the focus on the participant’s performance
– Is the product successful?
Usability NOT User Testing

• All interfaces have usability problems
• Limited resources to fix them
• More problems than resources
• Less serious problems distract
• Intense focus on fixing most serious problems first
True Statements
Adapted from: Rocket Surgery Made Easy:
The Do-It-Yourself Guide to Finding and Fixing Usability
Problems, by Steve Krug

Parallel-Track
Workflows
(How to Think Ahead)
a.k.a. Staggered Sprints

• Developer Track:
– Focus is on production
– Whole team (including UX)
• UX Track (Interaction Designer Track)
– Focus is on user contact
Agile Design Timing: Parallel Tracks
“Adapting Usability Investigations for Agile User-Centered Design” by Desirée Sy.
Journal of Usability Studies, Volume 2, Issue 3 http://www.uxpajournal.org/
UX

Iteration 1
UX

Iteration 2

And so on…

1) Development work
– Maximize velocity to make working software
– Goal: Working software
2) Discovery work
– Learning velocity - fast learning and validation
– Goal: Learning
Reality Check – 2 Types of Work
Dual Track Development is not Duel Track, May 10, 2017 By Jeff Patton
http://jpattonassociates.com/dual-track-development/

Dual Track Development is not Duel Track
Dual Track Development is not Duel Track, May 10, 2017 By Jeff Patton
http://jpattonassociates.com/dual-track-development/

• Between the two tracks
• Essential for success
• Not just hand-offs
• All at once – two tracks, not two teams
• All work will not move from discovery to development
Constant Communication
From Tutorial: Making UX Agile and Lean at UXPA 2016
by John Schrag, Carol Smith and Thyra Rauch
http://uxpa2016.org/event/making-ux-agile-and-lean

• Pros
– Space is made for discovery work
• Cons
– Can lead to BDUF and Waterfall
– Not a solo sport - need a team
Pros and Cons - Parallel/Staggered work

Balancing Act
• Understand users, context, etc.
• Create personas, mental models, etc.
• Prepare for story mapping
and other sessions thoroughly
Strive for
UX Best
Practices
• Engineers need designs
to develop
• Research cannot continue forever
Meet
Production
Needs
Agile requires leaner methods
Time box and focus on outcomes
Slide from Tutorial: Making UX Agile and Lean at UXPA 2016
By John Schrag, Carol Smith and Thyra Rauch
http://uxpa2016.org/event/making-ux-agile-and-lean

• Continuous improvement
• Continue to prioritize Dev work
• Start small with Discovery work in next sprint
• May need a Design Spike
Existing products/teams

What to Test

Learn from whatever is ready…
• Concepts, rough ideas,
brainstorming
• Work in Progress
• Prototypes - make to learn
– Clickable (code is optional)
• Websites, mobile, products,
services
• Competing designs (A/B
testing)
• Comparative studies
• Conduct user research

• Clickable
– Easy and quick to create
– Easy to change
• Real Code
– Risky – more time invested
– Great if it’s the right solution
• Not paper - participants may misunderstand
Clickable or Code?

When should I test?
• Early! Any Stage of process
• Anywhere (conference room,
remotely)
• Anytime (un-moderated)
• Realistic test environment
Photo by Roebot at http://www.flickr.com/photos/roebot/2964156413/

Testing Methods

• What do we need to know?
• What gets us to 80% confidence/disproves?
• Bare minimum of effort?
• Place higher value on repeatable formative studies
Adapting Usability Testing Methods

• Consider budget, resources
• Time
– Recruiting
– Facilitating
– Analyzing
Scope Effort

Small Scope is Better
• 5-7 participants uncover
~80% of major usability problems
• Don’t try to be statistically
significant
• More participants
- diminishing returns
Jakob Nielsen’s Alertbox. Why You Only Need to Test with 5 Users.
March 19, 2000. http://www.useit.com/alertbox/20000319.html
Number of Test Participants
Participants are representative of specific user group and are doing representative tasks
More participants take more time, cost more money - significantly diminishing returns.

• Moderated
– In Person
– Remote (Web Meeting)
• Un-moderated (online tools)
Testing Options

• 3 + participants (tie breaker)
• Short sessions
– Moderated: 30 minutes to 1 hour each
– Unmoderated: Under 20 minutes
Make it Quick

• Pros
– Get to “why” by asking follow-up questions
– Work with them longer
– Save or end session as appropriate
• Cons
– Harder to recruit
– Takes longer to plan
Moderated

• Pros
– Great for quick answers - complements moderated testing
– Easy recruiting
• Cons
– Recruiting very relaxed
– Professional participants
• Participants camera and internet only
Unmoderated

Remote Testing Options
Moderated Un-Moderated
Researcher is present Researcher not needed during study
Same effort as in-person Minimal effort
Limited number of participants
per day (3-6)
High number of participants per day
Immediate feedback Test data compiled usually within
2 days
Get to the “Why?” by asking
questions
Reduced or no ability to ask
questions

• No travel
• Easy to set up
• Conducted from almost anywhere
• Option to see participants actual environment
• Drawback: Not as high quality as in-person
Benefits of Remote

• Robust computer
• Screen sharing software
• Fast Internet on both ends
– May require separate phone line if no VOIP
• Participants camera
– People are becoming more comfortable
– I still do not use
Remote – Moderated Requirements

More on Remote Testing
• Getting the most out of remote
research and testing
– http://interactions.acm.org/archive/view/march-april-
2017/getting-the-most-out-of-remote-research-and-testing
– ACM Interactions Magazine
by Carol Smith
Getting the most out of remote research and testing. By Carol J. Smith. 2017.
Interactions, 24, 2 (February 2017), 82-84. DOI: https://doi.org/10.1145/3038225
http://interactions.acm.org/archive/view/march-april-2017/getting-the-most-out-of-
remote-research-and-testing

• In-person (office or participant location) or remote?
• Prototype/organization limitations (Online? Clickable?)
• Number of participants, observers?
• Logging and video editing (time on task, highlights)?
• Surveys before or after?
• Eye tracking?
Software Tool Considerations

Software Tool Considerations
Meeting
Software
ModeratedUn-Moderated
Complex
Simple

• Nice
– Are your users local?
– Opt for large screen TVs/Cameras vs. two way glass
• Recommend laptop/portable lab
– Can be semi-permanent
– Ready when you are (minimal setup)
Formal Lab?

Make it Repeatable

“Teams should stretch
to get work into that day’s test
and use the cadence to drive
productivity.”
- Jeff Gothelf
http://blog.usabilla.com/5-effective-ways-for-usability-testing-to-play-nice-
with-agile/

Regular and Early Testing
• Reduces risk and chance
of major issues later
• Priority needs and problems
get priority
• Don’t sweat the small stuff

• Small focused tests
• Reduced recruitment time
• Once per Sprint
• Same day(s) mid-week (avoid Mondays and Fridays)
Bring it On!

• Team becomes:
– accustomed to steady stream of qualitative insight
– insight encourages quick decisions…line up with business
and user goals
Why Regular?
Adapted from Jeff Gothelf - http://blog.usabilla.com/5-
effective-ways-for-usability-testing-to-play-nice-with-agile/

• Problems we will find while usability testing
• Research to inform personas
• Design/UX debt
• Advocate to get prioritized (UX rep should be in planning)
Design Stories

Managing Testing

• Select what to test and/or additional work
• Create guide
• Determine recruit and automate where possible
Planning is Tight

• Screener
– Technology use/experience
– Knowledge of topic
• Guides/Scripts
• Consent Forms
• Data Collection
Create Reusable Templates

• Test & Observation Rooms
– Use repeat calendar feature
• Any location will do
– Conference rooms
– Offices
– Quiet corner of cafeteria
Pre-Book Rooms Quarterly

Recruiting

Hire a Recruiter
• Allows you to focus
• Good recruiters:
– find right participants.
– give regular updates.
– take care of directions,
confirmations, incentives, etc.

• Email, phone, etc.
• Create a panel (large recruits, less often)
– Customer lists, user groups
– Schedule as needed
– Use cautiously
• Internal resource who REALLY understands who you need
Do it Yourself

Use Site to Recruit
Ethnio: https://ethn.io/

• “We are all only temporarily able-bodied.
Accessibility is good for us all.”
-@mollydotcom at #stirtrek 2011
• Know the law, build for the spirit of accessibility
– U.S. Section 508 - https://www.section508.gov/content/learn
– WCAG 2.0 - https://www.w3.org/WAI/intro/wcag.php
Include people with disabilities

• Keep it easy
– In Person: Gift cards from retail stores
– Remote: Online Gift Cards
• More complex
– Product credits
– Swag (don’t bother)
Honorariums

• Make team aware
• Invite everyone
– Can watch remotely
– Recurring meeting invites for stakeholders
Usability Testing Day!

Observing Studies

• Stakeholders and decision makers
– PO, PM, Dev, etc.
• Greet participants (in person sessions)
• Take notes and share them
– Create a template if helpful
• Help operate software/recording equipment
Team Members Are Active Observers

• Pro’s
– Gain empathy for users
– Understand struggles and what works
– Quick decisions can be made
• Cons
– Time is limited
– Not all sessions are great (but most are!)
Observers

Notetaking
• Keep it simple
• Paper or Computer
– Ask participant if bothered
by typing
– Consider if you can pay attention
(turn off all distractions)
Issue P1 P2 P3
Search Used Yes No No
Widget 1 Used N/A Used –
unsure
about
Task 1 Notes 3 – easy 2 – needed
help
3 – easy
Task 2 Notes 2 – needed
help
2 – easy 2 – needed
help
help
3 – easy Ran out of
time
help
3 – easy Ran out of
time

Analysis, Debrief
and Useful & Usable
Recommendations

• Occurs without full analysis when stakeholders have watched
tests
• If not: Quick analysis to quick decisions
Debrief with Team

• All decision makers MUST be present
• Stay on Topic
• Be Constructive
• Don’t get distracted by small problems
• Intense focus on fixing most serious
problems first
Guidelines for Debrief

• Identify top 5 or 10 most serious issues
– Top 3 from each observer’s list
– Prioritize from lists
– Commit resources for next sprint
– Stop
Goal
Adapted from: Rocket Surgery Made Easy:
The Do-It-Yourself Guide to Finding and Fixing Usability
Problems, by Steve Krug

Excel - Find Patterns in Data
Learn more: I’m Sorry. I Can’t. Don’t Hate Me:
The Post-It Breakup by Kyle Soucy at IA Summit 2017
https://www.slideshare.net/usableinterface/im-sorry-i-
cant-dont-hate-me-the-postit-breakup

• Look for patterns
• Read “between the lines”
• Know what you’ve got
– Sort, reorganize, review, repeat
– What refutes your expectations?
– Surprises?
– Outliers?
Transform Data

Don’t be a tree killer
Image of stack of paper:
https://commons.wikimedia.org/wiki/File:Stack_of_Copy_Paper.jpg
X Marks the Spot https://openclipart.org/detail/193050/x-marks-the-spot

• Bulleted list
– Who will use information and how?
– Share via common tools (wiki, Box, etc.)
• Include
– Goal of study
– High level findings and recommendations
– Link to guide and analysis (tasks attempted, actual
performance, etc.)
Just enough

• Identify stories related to findings
• Add findings/evidence/recommendations in comments
on related stories in repo
• Tangential findings/evidence/recommendations - also note
in comments
Tie to Stories in Repo

• Small iterative changes
– Make it better now
– Don’t break something else
• Take something away
– Reduce distractions
– Don’t just add – question it
Tweak, Don’t Redesign
Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding
and Fixing Usability Problems, by Steve Krug

• Fit testing into Agile process
• Never too early –
whatever is ready
• Regular testing
https://www.flickr.com/photos/fireatwillrva/5823369990

Contact Carol Smith
in/CarolJSmith
@Carologic
slideshare.net/carologic

Resources, References, etc.

Faster Usability Testing in an Agile World - Agile UX Virtual Summit 2017 by UXPin

More Related Content

What's hot

Viewers also liked

Similar to Faster Usability Testing in an Agile World - Agile UX Virtual Summit 2017 by UXPin

More from Carol Smith

Recently uploaded

Faster Usability Testing in an Agile World - Agile UX Virtual Summit 2017 by UXPin