The document summarizes 10 common pitfalls in user research. It begins by introducing the speaker, Dan Arra, and providing an overview of what user research entails, including qualitative and quantitative methods. It then details 10 frequent mistakes in user research: assuming personal expertise is enough, not understanding users are not designers, jumping to solutions too quickly, talking to the wrong customers or asking questions incorrectly, overstating features without validation, misusing numbers, treating focus groups and usability testing as interchangeable, relying on insufficient sample sizes, failing to translate data into requirements, and not involving the right people with research expertise. The document emphasizes the importance of asking the right questions using the proper methods and analyses.

1. Top 10 User Research Pitfalls
Dan Arra
Managing Director

2. Agenda
PART 1
About the Speaker
PART 2
What is Research?
PART 3
10 Research Pitfalls
2

3. ABOUT THE SPEAKER
3

4. About The Speaker: Dan Arra
• Began career as an engineer in the aerospace industry.
• Spent 10 years at software companies in systems engineering,
product management and professional services.
• Started software & technology companies and was responsible
for leading numerous successful projects. Another 10 years.
• Joined Macadamian, a software design & development studio.
Work closely with design and software engineering teams to
improve and create new products.
4

5. WHAT IS CUSTOMER RESEARCH?
5

6. User-centered Design
6

7. UCD: understanding users and tasks
Good design is built on a solid understanding of your audience: specific goals, tasks and
context of use.
Confidential 3/22/13
7

8. Research Methods: Qualitative versus Quantitative
QUALITATIVE QUANTITATIVE
attitude about Focus groups Surveys (online, telephone)
your product and Interviews
competition
Behavior Ethnographic research A/B Testing
related to your (Observation, shadowing)
product offering
usability of Usability testing Usability testing
product
Confidential 3/22/13
8

9. 10 RESEARCH PITFALLS
9

10. The most dangerous things about research!
Myth:
Being Smart means you
can do good UX Research
& Design.
Anybody can ask a
question… but relevant,
appropriate, constructive?
10

11. Design research IS real science!
Sci-ence [sahy-uh ns] noun
• systematic knowledge of the physical or material world gained through
observation and experimentation.
11

12. Reality!
Observe Think Design Test
! Rinse, repeat.
12

13. Our users don’t tell us what they need
Myth:
We asked users what they wanted, so we re
covered.
13

14. Users are NOT designers
• Users are focused on solutions
• Don’t consider context and other mitigating factors
• Why? Why? Why?
14

15. Avoid jumping to Solutions too quickly
15

16. Staying in problem space leads to INNOVATION
Go to: http://www.macadamian.com/ for full story.
16

17. Staying in problem space leads to INNOVATION
Go to: http://www.macadamian.com/ for full story.
17

18. Staying in problem space leads to INNOVATION
Go to: http://www.macadamian.com/ for full story.
18

19. Staying in problem space leads to INNOVATION
Go to: http://www.macadamian.com/ for full story.
19

20. Talking to customers " Research
Who to ask...
AND
How to “ask”.
20

21. Consider Triangulating Research !
21

22. Stating, without knowing! “We must have this feature”!
can be very to EXPENSIVE
Go to: http://www.macadamian.com/ for full story.
22

23. Numbers can be deceiving!
23

24. Sometimes numbers are not appropriate!!
Qualitative Quantitative
Attitude • Focus Groups • Surveys
(Preference) • Interviews • A/B Testing
Behavior • Usability Walkthrough • Usability Test
(Performance) • Ethnographic Research • Benchmark Test
• Job Shadowing
24

25. Focus Groups & Usability Testing!NOT the same!
What people say, what
people do, and what they
say they do are entirely
different things.
Photo credit: http://www.webusability.co.uk/what-we-do/mobile-testing/
25

26. Tale of Two methods
Usability Testing Focus Groups
Method One-on-one testing; observing representative Group discussion/exploration with up to
samples of end users using the product and 10 participants per group
measuring their performance on pre- (n=1)
determined tasks
(n= # of individuals being tested)
When to use Throughout development phase and during Pre-development; early in program
deployment verification
Key focus What will make the product easy for end What will motivate potential customers to
users to learn and use? buy the product? Determining market
need.
Key How well can end users use the What are the customers' wants and
functionality a product provides? Can they desires ?
motivation realize the value promised?
26

27. Eight is sometimes enough
27

28. It just depends
Faulkner, Laura (2003) Beyond the Five-user
Assumption: Benefits of Increased Sample
sizes in Usability Testing Behavior Research
Methods, Instruments and Computers Volume
35 (3) 379-383
28

29. Raw data is NOT enough
29

30. Translate data into meaningful requirements
30

31. Don’t rely only on the Spec!
31

32. More than words can say!
32

33. Not just anyone can do research
33

34. UX Researcher – Prevents “heard what I wanted to hear.”
• Consider bringing someone on board with experience, a
Degree in Human factors, HCI and/or soft sciences (e.g.,
Psychology, Sociology).
• If conducting true user research, you’ll want:
− someone with knowledge and experience in research methods
and/or experimental practices.
− someone with the expertise to turn insights into design direction
• Partner user researcher with SMEs, business analysts, and
product managers for maximum success!
34

35. A Good Researcher!
! can identify the best methods to get the right information
! is “familiar enough” with the subject material to determine the
right questions
! avoids leading questions
! has strong listening skills
! follows the the natural flow of the conversation
! can steer the conversation in the case of digression and guide
user to discuss new topics when one is exhausted
! remains neutral and puts the interviewee at ease
! focuses on the most recent experiences
! looks for ways to visualize and summarize results to be
actionable
35

36. Remember
Ask the right questions!
Use the right method(s)
Analyze appropriately
Use the right people!
36

37. Remember, also that you are!
Improving AND growing business.
Developing relationships = gold.
Establishing credibility = save time.
Learning/refining your craft.
37

38. Thank-You
darra@macadamian.com
For more information, visit: www.macadamian.com
Follow Macadamian on Twitter: @macadamianlabs
38

39. Design and Research Resources
”Innumeracy: Mathematical Illiteracy and Its Consequences”,
by John Allen Paulos
“Ambient Findability: What We Find Changes Who We Become”, by Peter
Morville
“Communicating Design: Developing Web Site Documentation for Design
and Planning,” by Dan Brown
The Design of Everyday Things Don Norman
Designing Interfaces Jennifer Tidwell
Do You Matter? How Great Design Will Make People Love Your
Company Robert Brunner
“Don't Make Me Think: A Common Sense Approach to Web Usability,” by
Steve Krug
Confidential 3/23/13 39

40. Design and Research Resources
“The Elements of User Experience: User-Centered Design for the Web,”
by Jesse James Garrett
Emotional Design Don Norman
Mental Models: Aligning Design Strategy With Human Behavior by Indi
Young
“Observing the User Experience: A Practitioner’s Guide to User
Research,” by Mike Kuniavsky
“Presentation Zen Garr Reynolds
Sketching User Experiences: Getting the Design Right and the Right
Design,” by Bill Buxton
Universal Principles of Design William Lidwell
“Visual Explanations: Images and Quantities, Evidence and Narrative,” by
Edward Tufte
Confidential 3/22/13 40

41. About Macadamian
We create compelling user experiences for:
• Consumer and enterprise desktop applications
• Interfaces embedded into physical products
• Mobile interfaces and applications on multiple platforms
• Web applications and cloud-based applications
Confidential 3/22/13 41

42. Design + Development!
Confidential 3/22/13
42

43. • Over 200 employees!
• 15 + years in business!
• Integrated design & development teams!
Ottawa!
Yerevan!
San Mateo!
Cluj, Romania!
Confidential 3/22/13
43

44. What We Do: UX services
• UX Research • UX Design
− Stakeholder Interviews − Information Architecture
− Product Strategy Workshops − Content Modeling
− Task Analysis − Interaction Design
− On-line Surveys − Visual Design
− Usability Testing
• Software Development • Software Quality Assurance-QA
− Stand alone or combined with design − QA Review: performance or process
− PoC or market ready − QA/QC for customer lead development
− Technologies: Web, Cloud, Mobile, − QA/QC within our agile process
Desktop − Localization QA
− Agile methodology − QA Script/Automation
− Integrated with customer teams
44