Presented by: Brooke Dunagan and Jenna Weiss

1. How Ethnography Can
Save You Money
Brooke Dunagan &Jenna Weiss

2. Overview
• Ethnography 101
• Case Study
• What we did
• Where we went wrong
• Learning from our mistakes
• Findings
• Financials
• Cost of Features
• Ethnography costs
• Return on Investment
• Questions

3. What is an Ethnography?
That is, in the traditional sense...
“The systematic study of people and cultures. It is
designed to explore cultural phenomena where
the researcher observes society from the point of
view of the subject of the study”
Gerhard Friedrich Müller
(The only known portrait)

4. Ethnography in UX
Now we’re talking...
• See first hand how users interact with
their environment
• Make design decisions based on
qualitative/quantitative facts rather than
opinions
• Learn the unspeakable
“Ethnography reveals how digital and
physical processes work together to
help businesses address gaps and
focus on the entire consumer
experience”
- (Jessica Weber & John Cheng,
2013)

5. How to do an Ethnography
…no, no one needs to go and live in a Siberia for a year
Who, What, and
Where
Collect Data
Analyze Data
APPLY THE FINDINGS!
Share the Results
Determine Research
Questions

6. How to do an Ethnography
Who
• Who is the subject of the ethnography?
• Who is using your product?
What
• What is the goal of the project?
• What is your role in the ethnography?
Where
• Where are the observations occurring?
• How will you observe?
• Open observations
• Closed observations
Setting up the study

7. How to do an Ethnography
More about determining your role in the ethnography
INVOLVEMENT DETACHMENT
Source: Qualitative Methods I: Observation and Field Reserach (Ethnography), Sam Ladner
Complete Participant Complete Observer
Participant-
as-Observer
Observer-as-
Participant

8. How to do an Ethnography
• Determine research questions
• What do you hope to learn?
• How will the findings help the
project?
• Is there a qualitative and a
quantitative portion to the study?
Setting up the study

9. How to do an Ethnography
Collect the data
• Put yourself in the shoes of the person you are observing
• Standardize data collection
Analyze the data
Share the results
• Tailor results to your audience – no data dumps
• Pull key results for a quick reference cheat sheet
Apply the findings
• Take what was learned and apply it back to the design
Execute and Share

10. Case Study
Applying Ethnography and Design

11. Our Project
• LogistiCare provides scheduling for non-emergency medical
transportation
• Today, most reservations/scheduling, occur over the phone
through an outdated call center application manned by Customer
Service Representatives (CSRs)
A tale of a call center application redesign

12. Our Goal
Create an experience that
supports the CSR and meets
business goals

13. Current System
We call it LCAD for short

14. What We Did
• Met with business partners and gathered requirements
• Conducted multiple interviews with CSRs
• Understand their needs
• See how they work
• Experience the environment
• Frequent meetings with development to understand their technical needs
and concerns
• Data constraints
• Architectural limitations
• Data synchronization between new and old system
• Designed, tested, iterated, and tested again
• Include as many call centers as possible
…and what we didn’t do

15. LCAD 2.0
Where we landed

16. What We Didn’t Do
...and why
Ethnographic Research!!!

17. What Happened Next
…and why
Ethnographic Research!!!

18. Our Ethnography
…again not Siberia, but we did go to Phoenix in July and Maine in January
Who, What, and
Where
Who
• We sat with and observed 116 CSRs
What
• Time on task
• Culture observations
• Member/CSR interactions
• Call Center processes
Where
• On the call center floor at four different
locations

19. Our Ethnography
Research
Questions
• How long does it take to complete a standard reservation?
• What type of calls do CSRs receive?
• Primary and secondary tasks
• What is the interaction like between the member and the CSR?
• What “job hacks” has the CSR developed in order to do their job
effectively

20. Our Ethnography
Collect Data
• Visited four call centers
• Rhode Island, Maine, Las Vegas, Phoenix
• Observed 3012 calls
• Collected qualitative and quantitative data
• Time on task
• Frequency of events
• General observations

21. Our Ethnography
• Most of our calls were reservations
• The system was old and out dated but CSRs made it work
• Business processes were cumbersome and full of layers of bureaucracy
• And…
Analyze Data

22. Our design wasn’t going to work as
well as we hoped

24. Back to the drawing board…

26. But… did it really make a difference?

27. Measuring Impact of the Design Change
• Participants
• 22 participants (11 for each version)
• Our participants were CSRs who use the current system in their
day to day job
• What we did
• Every participant received 15 minutes of training on the new system
• Completed two “calls” with the CSRs on prototyped versions of the new
designs

28. Time to Complete a Reservation
N = 825
Data collected during
benchmarking using
actual calls with
members.
Current LCAD
5 min 59 sec
4 min 09 sec
3 min 08 sec
N = 11
LCAD 2.0
(Pre-ethnography)
N = 11
LCAD 3.0
(Post-ethnography)

29. Limitations and Next Steps
• Limitations
• The follow up timing data was collected in a highly controlled setting
• Not enough data was collected to make statistical conclusions
• Next Steps
• After the new application launches we will collect timing data and
compare it to the original benchmarking study
This data will allow us to see...

30. How much money we saved the company.
...which leads us to part two of this presentation,
how ethnography can save you money

31. The Financials
Money, money, money…

32. UX Timeline
Key milestones along the project
Designs were due to development with
the goal of having the product finished by
end of 2016
November 2016
Delivered Designs to Dev
After the ethnography, the UX was
redesign to address issues and findings
and create a better overall experience
March 2017
UX Redesigns
After site visits, ethnography and time on
task data collection, we realized we had a
problem
February 2017
UX Issues Arise
Initial interviews, light observations and
simple features were tackled first
June 2016
Project Begins
With a prototype, exploratory data were
collected to estimate the ROI (Return on
Investment) of the new UX
May 2017
Preliminary UX Data

33. UX rushing to meet the November 2016
date, caused considerable rework.
...using some UX methods but skipping key practices
like ethnographies was impacting the project

34. UX Team of 4
4 UX team members
Avg $ 50/hour
40 Hour weeks
4 weeks
$ 32,000
Dev Team of 6
6 Developers
Avg $ 80/hour
40 hour weeks
8 weeks
$154,000
Total Cost
Wireframes
Interaction flows &
functionality
Usability testing
UI Designs
Front end templates
Middleware and back end
$186,000
Average Cost of a Feature
The cost of an individual feature for our project

35. $32,000 $154,000 $186,000
What features should have cost...
Cost of Redesign
$64,000 $308,000 $372,000
What features actually cost with ”rework/redesign”...

36. Each time a feature was redesigned it cost
us $ 372,000 and 16 weeks or more.
...not conducting the ethnography was costing us money
and time and we had very little to lose of either

37. What is the actual cost of an
ethnography?
...so little money for so much value

38. UX Team
5 members
UX Director
UX Lead
UX Researcher
UX Designer (2)
During the week the team was conducting the ethnography, very little work was able to be conducted on the project deliverables
Travel
$10,100
Flight - $ 2500
Hotel - $ 4500
Food - $ 4000
Car - $ 1300
Productivity
$14,500
40 hours a week
Team of 5
Loss of productivity
Total Cost
$26,800
Travel costs
Productivity Loss
Cost of a week of Ethnography
How much it cost us to conduct the ethnography

39. Bottom line, was the cost worth it?
...what is the cost savings to the company?

40. Cost Savings
For each minute cut from
average call handle time the
company saves $16 million
dollars/year
1 min = $16
mil
Current basic
reservations call
handle time (time
CSRs are on the call)
is 6 minutes
6 min
Initial design, basic reservation
reduced by 2 min. will save the
company $32 million/year
Design 1
2 min = $32 mil
Improved ethnography design,
basic reservation will save the
company $48 million/year
Ethnography Design
3 min = $48 mil

41. By spending $26,800 and 1 week conducting ethnographies, we were able to reduce the time the CSR is on the phone by another
minute. Each minute saves the company $16 million a year. The C level executives were very satisfied by the value of the study.
$27k
Spent on an
ethnography
1min
Reduction of
CSR call
handle time
$16mil
Cost savings
to the
company
100%
Very satisfied
executives
Was the Ethnography Worth It?
What was our ROI?
= =

42. SAVING MONEY
Every project team, stake
holder & executive wants
to show their leadership
success. Financial
success is a quantitative,
easily measureable goal.
No interpretations, just
hard facts.
BUY IN
Business doesn’t always
understand UX but they
understand dollars and
numbers. Use the language
of your audience to help
them understand the
purpose.
CONFIDENCE
Having data, numbers, facts
and dollars allows you to
debate intelligently.
Qualitative is hard to argue,
without the data to support it.
Make your life easier, use
the data to support your
point of view.
The Importance of ROI
ROI – Return On Investment

43. Carmen Parks
UX Designer Extraordinaire
Mitch Berg
UX Designer/
Reservation Brainchild
Daniel Nutter
UX Designer/
Data Collecting Champion
Special Thanks
...seriously couldn’t have done it with out these folks
Brad O’Donnell
UX Engineer/
Personal Tech Support
Namon Huddleston
Product Owner/
Business Support/
UX Dad

44. Questions?

45. Sources
Making the Most of Ethnographic Research (Jessica Weber, John Cheng)
http://uxmag.com/articles/making-the-most-of-ethnographic-research
Qual Methods I: Observation and Field Research (Ethnography) (Sam Ladner)
https://www.slideshare.net/sladner/week04-ethnography-and-field-research-presentation
Ethnography in UX (Nathanael Boehm)
http://www.uxmatters.com/mt/archives/2010/06/ethnography-in-ux.php