How might we marry quantitative and qualitative methods to see the big picture? Generating knowledge through big and deep data and why and how you can use mixed methods.

1. MIXED METHODS AND THE
UNKNOWN UNKNOWNS OF UX
JUDITH MÜHLENHOFF
@judithmp
HOW MIGHT WE MARRY QUANT AND QUAL METHODS
TO SEE THE BIG PICTURE?
UXCE15, BERLIN

2. User and Innovation Researcher
Sociologist
PhD candidate, “Culture-Driven Innovation”
Owner of the User Research Lab: enhancing audience engagement with qualitative
insights
@judithmp
Judith
HELLO, MY NAME IS…

3. “There are known knowns; there are things we know
we know.
We also know there are known unknowns; that is to
say we know there are some things we do not know.
But there are also unknown unknowns – the ones we
don't know we don't know.”
Donald Rumsfeld, 2002
THE UNKNOWN UNKNOWNS
R. D. Ward [Public domain], via Wikimedia Commons
https://commons.wikimedia.org/wiki/File%3ADefense.gov_News_Photo_020304-D-9880W-013.jpg
Donald Rumsfeld

4. 1. GENERATING KNOWLEDGE THROUGH DIFFERENT DATA
2. MIXED METHODS
2.1 WHY & HOW
2.2 CASES
3. TAKE AWAYS
AGENDA

5. Type of Knowns and Unknowns
GENERATING KNOWLEDGE THROUGH DIFFERENT DATA
Knowns Unknowns
known
Known Knowns
Things we are aware of and
understand
Known Unknowns
Things we are aware of but
don‘t understand
unknown
Unknown Knowns
Things we understand but are
not aware of
Unknown Unknowns
Things we are neither aware
of nor understand

6. “With enough data, the numbers
speak for themselves.”
Chris Anderson, 2008
Big Data – The Holy Grail of Customer Experience?
GENERATING KNOWLEDGE THROUGH DIFFERENT DATA
Anderson pic by James Duncan (Flickr)
https://commons.wikimedia.org/wiki/File%3AEtech05_Chris.jpg

7. Numbers do not speak for themselves, they have to be interpreted within their context.
Big Data Needs Deep Data
GENERATING KNOWLEDGE THROUGH DIFFERENT DATA

8. Big Data are not objective, they inhibit high risks of hidden biases and faulty data.
Big Data Needs Deep Data
GENERATING KNOWLEDGE THROUGH DIFFERENT DATA

9. Big Data tell us about “how many/how often”, but not about the “why” and “how”.
Big Data Needs Deep Data
GENERATING KNOWLEDGE THROUGH DIFFERENT DATA

10. Qualitative methods, like ethnography, provide us with “Deep Data”
Tell us about the why and how of human behaviour
Put data into the context of the messy social world
Explore neglected research fields in depth and breadth
Big Data Needs Deep Data
GENERATING KNOWLEDGE THROUGH DIFFERENT DATA

11. Types of Knowns and Unknowns and Data
GENERATING KNOWLEDGE THROUGH DIFFERENT DATA
Knowns Unknowns
known
Known Knowns
Things we are aware of and
understand
• Known Data
• To be updated and refined
Known Unknowns
Things we are aware of but
don‘t understand
• Insufficient Big Data
 Deep Data
• To be explored in depth
unknown
Unknown Knowns
Things we understand but are
not aware of
• Esp. Big Data
• To be made aware of
Unknown Unknowns
Things we are neither aware
of nor understand
• Deep Data + Big Data
• To be explored in depth
and breadth

12. Mixed Methods

13. Qualitative and quantitative methods each have its strengths and weaknesses.
With mixed methods, the strength of a single method outweighs the weakness of the
other.
Why Mixed Methods?
MIXED METHODS – WHY & HOW
https://www.flickr.com/photos/archeon/

14. Combining methods provide a bigger picture, like different angles in a movie.
These can lead to corroboration, verification…
… or might spur complete new thinking and theories.  Unknown Unkowns
Why Mixed Methods?
MIXED METHODS – WHY & HOW

15. At which stage do you combine methods?
Do you choose an identical or non-identical sample?
What is your overall goal of mixing methods (e.g. corroboration, exploring)?
Will you focus on one method?
How to Conduct Mixed Methods?
MIXED METHODS – WHY & HOW

16. How to Conduct Mixed Methods?
MIXED METHODS – WHY & HOW
Sequential Design
Embedded Design
Parallel Design
Fully Integrated Design
Conversion Design

17. Help newsrooms to measure the quantitative and qualitative impact of their stories.
Two tasks:
Understanding what it means for a story to “do well” and what happened to cause
that.
Managing an event-tracking workflow while juggling other responsibilities
MIXED METHODS – CASES
NewsLynx (Columbia Journalism School/Tow Center)

18. Proposal of a taxonomy to measure impact beyond metrics
MIXED METHODS – CASES
NewsLynx Columbia Journalism School/Tow Center)

19. Ch
References
Sources

20. MIXED METHODS – CASES
Self-Observation: Diary Studies and Experience Sampling Method (ESM)
www.user-research-lab.org
Digital Media Diary (User Research Lab)
Self-observation of news consumption

21. Experience Sampling Method (e.g. Google)
Track experiences in the moment
Qualitative data at larger scale
MIXED METHODS – CASES
Self-Observation: Diary Studies and Experience Sampling Method (ESM)
www.pacoapp.com

22. Exploration of computer usage and temporality through tracking…
MIXED METHODS – CASES
Ethno-Mining (Intel)
Anderson et al. 2009: Numbers Have Qualities Too: Experiences with Ethno-Mining

23. …and participatory data interpretation
MIXED METHODS – CASES
Ethno-Mining (Intel)
Anderson et al. 2009: Numbers Have Qualities Too: Experiences with Ethno-Mining

24. Take Aways

25. 1. Leave your method comfort zone.
2. Aim for highly integrated research designs with different angles.
3. Take advantage of participatory data interpretation.
4. Plan and decide deliberately.
5. It takes a lot of effort, but may lead you to your unknown
unknowns.
TAKE AWAYS

26. THANK YOU!
Judith Mühlenhoff
@judithmp
www.user-research-lab.org

27. References
Newslynx: http://newslynx.org/
Big Data:
Kate Crawford: The hidden biases in data
https://hbr.org/2013/04/the-hidden-biases-in-big-data/
Mikkel Krenchel and Christian Madsbjerg: Your big data is worthless if you don’t bring it into the real world
http://www.wired.com/2014/04/your-big-data-is-worthless-if-you-dont-bring-it-into-the-real-world/
Experience Sampling:
Kathy Baxter, You too can collect big data!
https://www.epicpeople.org/pblog10/
Overview of programmes and apps: Experience Sampling and Ecological Momentary Assessment with Mobile Phones
http://www.otago.ac.nz/psychology/otago047475.pdf
Ethno-Mining:
Anderson et al. 2009: Numbers Have Qualities Too: Experiences with Ethno-Mining
Mixed Methods Research Design:
Miles MB, Huberman AM (1994) Qualitative Data Analysis: An Expanded Sourcebook. Sage Publications, Inc, Thousand
Oaks
Tashakkori A, Teddlie C (2009) Integrating qualitative and quantitative approaches to research. In: Bickman L, Rog DJ (eds)
The SAGE handbook of social research methods, 2d ed. Sage, Los Angeles, pp 283 – 317
Teddlie C, Tashakkori A (2006) A general typology of research designs featuring mixed methods. Res Sch 13:12–28.