Notes on conducting qualitative observation for user requirements, presented to Baobab Health Trust, Lilongwe, Malawi, March 2014

1. Boabab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Human-Computer Interaction -
Observations
Harry Hochheiser
University of Pittsburgh
Department of Biomedical Informatics
harryh@pitt.edu
+1 412 648 9300
Attribution-ShareAlike
CC BY-SA

2. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Research Setup
●
User(s) with clinical problems
●
Current approach not optimal
●
How do we know what to build?

3. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Important Claim #1
●
Understanding work in context – goals, motivations,
priorities, behavior, difficulties, etc. is necessary for
building better systems
●
Most go beyond just talking about computer systems to
address bigger picture questions
●
What if we build a pharmacy prescription tracking system…
●
..but the pharmacy never has any medication…

4. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Important Claim #2
●
Successful implementations may require work redesign
●
Translating the same old methods and procedures to computers
may not help much
●
Must go beyond what we are doing?
●
Instead, ask why?
●
In-depth qualitative research needed to inform these
efforts
●
Not numbers. Rather, goals, methods, activities..

5. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Important Claim #3
●
In-depth qualitative research needed to inform these
efforts
●
Not numbers. Rather, goals, methods, activities..
●
Watch, listen , interpret

6. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
What are the factors that influence
usability?
Is a given tool usable for all users?
For some users?
Is an interface full of text usable… for someone who can’t
read?

7. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Determinants of Usability
Context: social, organizational, etc.
Tool
Task
Data analysis?
Writing?
Graphing?
usability= f(user,tool, task, context,system)
System:
desktop,
mobile,
tablet, etc.

8. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Systems Engineering for Patient Safety
Lowry, et al. Integrating Electronic Health Records into Clinical Workflow: An Application of Human
Factors Modeling Methods to Ambulatory Care National Institute of Standards and Technology
NISTIR 7988 http://dx.doi.org/10.6028/NIST.IR.7988

9. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Broad Goal
• Use study of users to understand how to build usable
systems
• What is their work?
• What struggles are they facing?
• What information do they need?
• Why do they do what they do?
• How can IT help? Where can’t it help?
• What will be required for an IT tool to work?

10. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Approaches
• Engage with users to see how work is done
• Interview
• Observe
• Collaborate
• Build models describing work
• Validate models with users
• Use models to design appropriate interactions.

11. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Some Methodologies
Participatory
Design
Contextual
Design
Scenario-Based
Design
Cooperative
Inquiry
Participatory
Action
Research
Methodologies

12. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Key Questions & Tradeoffs
● Who to involve?
● When to involve users?
● How to collect information?
● How to interpret?
● How to inform design?
● How to evaluate success?
● Usual tradeoffs apply: Never enough time or money

13. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Stakeholder Analysis
Rosson & Carroll 2002
●
Identify stakeholder groups
●
Background
●
Expectations
●
Needs
●
Preferences
●
Concerns
●
Values
●
An important, but often overlooked step

14. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Stakeholders
●
Anyone who has an interest in the outcome of a system
●
Work, play, or some other aspect of life
●
Customer – those who pay for the work
●
User – those who work with the system
●
Others – perhaps those who are described by data in the
system

15. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Stakeholders - Challenges
●
Defined by roles, not by person
●
Billing clerk for the hospital system is likely also a health-care
consumer
●
Must identify people who can speak to different roles?
●
How can we meaningfully integrate understanding of
needs of diverse users?
●
Patients, practitioners, financial people, bureaucrats?

16. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
A Spectrum of Possibilities for Engaging
Stakeholders
Focus Groups
Traditional Written
Requirements
Interviews
Observation
Contextual Interviews
Ethnography/
Participatory Design
Low Cost, Low Fidelity
High Cost, High Fidelity
Surveys
Diaries/ActivityRecording

17. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Observation: When
• Understanding work in complex environments
• Often used when interviews with users are not possible
• busy clinical environments
• Even if possible, interviews might not be optimal
• Ecological validity: observations must be as close a match
to reality as is practical.
• Least demanding on users and context

18. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Observation
• Watch users as they complete tasks
• Who is involved
• What they are trying to accomplish
• How they meet their goals
• Look for
• struggles
• breakdowns
• workarounds
• contextual factors
• improvisation

19. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Methodology
• Take notes - lots
• Pictures
• Diagrams
• Audio recordings (if privacy concerns allow)
• No interruption
• possibly debrief with user afterwards
• Multiple methodologies - triangulation
• Multiple observers - validation

20. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Guidelines
• Get out of the way
• particularly in busy clinics
• Respect participant privacy
• no recording or photos in clinic
• Consider observing from multiple perspectives
• clinician and patient

21. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
After the observation
• Debrief
• Summarize notes
• Transcribe any recordings
• Analyze
• build models (more on this later)
• identify new questions - what did you not ask
• ask these at subsequent interviews
• Do this as soon as possible!

22. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Iterative process
Data Collection
Analysis New Questions
Models/
Theories Convergence

23. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Drawbacks of observations
• Receptive only
• Potential for misinterpretation
• Observation of work as it is done
• not as management or others think it is done.
• Solutions:
• multiple observers
• Debrief/review with participants post fact
• “I think this is what I saw…”

24. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Iterative process
Data Collection
Analysis New Questions
Models/
Theories Convergence
Validation
with Users

25. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
“All models are wrong…”
• Many possible ways to try to model user work
• Remember the goal:
• to do develop rich and valid understanding of work
processes
• to use this understanding to drive design of appropriate
software
• The particular methods may matter less than the process
• iteration
• validation

26. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Key Points
• Understanding how work is done is vital for building tools
to support that work.
• Try to understand goals
• Redesign work if necessary
• Observations can provide useful data for driving these
processes
• Need appropriate participants, triangulation, and
validation.

27. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu
Tomorrow: Bwaila ANC
• Meet at Bwaila at 8:15
• Observe ANC clinic
• Return to Baobab: 9:45
• Discuss/review observations: 10:15
• Contextual inquiry presentation: 10:45
• Lunch: 11:45
• Arrive at Bwaila for contextual interviews with ANC
clinicians and mock patients: 1:15