Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Mixed methods and the unknown unknowns of ux uxce15

2,547 views

Published on

How might we marry quantitative and qualitative methods to see the big picture?
Generating knowledge through big and deep data and why and how you can use mixed methods.

Published in: Data & Analytics

Mixed methods and the unknown unknowns of ux uxce15

  1. 1. MIXED METHODS AND THE UNKNOWN UNKNOWNS OF UX JUDITH MÜHLENHOFF @judithmp HOW MIGHT WE MARRY QUANT AND QUAL METHODS TO SEE THE BIG PICTURE? UXCE15, BERLIN
  2. 2. User and Innovation Researcher Sociologist PhD candidate, “Culture-Driven Innovation” Owner of the User Research Lab: enhancing audience engagement with qualitative insights @judithmp Judith HELLO, MY NAME IS…
  3. 3. “There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns – the ones we don't know we don't know.” Donald Rumsfeld, 2002 THE UNKNOWN UNKNOWNS R. D. Ward [Public domain], via Wikimedia Commons https://commons.wikimedia.org/wiki/File%3ADefense.gov_News_Photo_020304-D-9880W-013.jpg Donald Rumsfeld
  4. 4. 1. GENERATING KNOWLEDGE THROUGH DIFFERENT DATA 2. MIXED METHODS 2.1 WHY & HOW 2.2 CASES 3. TAKE AWAYS AGENDA
  5. 5. Type of Knowns and Unknowns GENERATING KNOWLEDGE THROUGH DIFFERENT DATA Knowns Unknowns known Known Knowns Things we are aware of and understand Known Unknowns Things we are aware of but don‘t understand unknown Unknown Knowns Things we understand but are not aware of Unknown Unknowns Things we are neither aware of nor understand
  6. 6. “With enough data, the numbers speak for themselves.” Chris Anderson, 2008 Big Data – The Holy Grail of Customer Experience? GENERATING KNOWLEDGE THROUGH DIFFERENT DATA Anderson pic by James Duncan (Flickr) https://commons.wikimedia.org/wiki/File%3AEtech05_Chris.jpg
  7. 7. Numbers do not speak for themselves, they have to be interpreted within their context. Big Data Needs Deep Data GENERATING KNOWLEDGE THROUGH DIFFERENT DATA
  8. 8. Big Data are not objective, they inhibit high risks of hidden biases and faulty data. Big Data Needs Deep Data GENERATING KNOWLEDGE THROUGH DIFFERENT DATA
  9. 9. Big Data tell us about “how many/how often”, but not about the “why” and “how”. Big Data Needs Deep Data GENERATING KNOWLEDGE THROUGH DIFFERENT DATA
  10. 10. Qualitative methods, like ethnography, provide us with “Deep Data” Tell us about the why and how of human behaviour Put data into the context of the messy social world Explore neglected research fields in depth and breadth Big Data Needs Deep Data GENERATING KNOWLEDGE THROUGH DIFFERENT DATA
  11. 11. Types of Knowns and Unknowns and Data GENERATING KNOWLEDGE THROUGH DIFFERENT DATA Knowns Unknowns known Known Knowns Things we are aware of and understand • Known Data • To be updated and refined Known Unknowns Things we are aware of but don‘t understand • Insufficient Big Data  Deep Data • To be explored in depth unknown Unknown Knowns Things we understand but are not aware of • Esp. Big Data • To be made aware of Unknown Unknowns Things we are neither aware of nor understand • Deep Data + Big Data • To be explored in depth and breadth
  12. 12. Mixed Methods
  13. 13. Qualitative and quantitative methods each have its strengths and weaknesses. With mixed methods, the strength of a single method outweighs the weakness of the other. Why Mixed Methods? MIXED METHODS – WHY & HOW https://www.flickr.com/photos/archeon/
  14. 14. Combining methods provide a bigger picture, like different angles in a movie. These can lead to corroboration, verification… … or might spur complete new thinking and theories.  Unknown Unkowns Why Mixed Methods? MIXED METHODS – WHY & HOW
  15. 15. At which stage do you combine methods? Do you choose an identical or non-identical sample? What is your overall goal of mixing methods (e.g. corroboration, exploring)? Will you focus on one method? How to Conduct Mixed Methods? MIXED METHODS – WHY & HOW
  16. 16. How to Conduct Mixed Methods? MIXED METHODS – WHY & HOW Sequential Design Embedded Design Parallel Design Fully Integrated Design Conversion Design
  17. 17. Help newsrooms to measure the quantitative and qualitative impact of their stories. Two tasks: Understanding what it means for a story to “do well” and what happened to cause that. Managing an event-tracking workflow while juggling other responsibilities MIXED METHODS – CASES NewsLynx (Columbia Journalism School/Tow Center)
  18. 18. Proposal of a taxonomy to measure impact beyond metrics MIXED METHODS – CASES NewsLynx Columbia Journalism School/Tow Center)
  19. 19. Ch References Sources
  20. 20. MIXED METHODS – CASES Self-Observation: Diary Studies and Experience Sampling Method (ESM) www.user-research-lab.org Digital Media Diary (User Research Lab) Self-observation of news consumption
  21. 21. Experience Sampling Method (e.g. Google) Track experiences in the moment Qualitative data at larger scale MIXED METHODS – CASES Self-Observation: Diary Studies and Experience Sampling Method (ESM) www.pacoapp.com
  22. 22. Exploration of computer usage and temporality through tracking… MIXED METHODS – CASES Ethno-Mining (Intel) Anderson et al. 2009: Numbers Have Qualities Too: Experiences with Ethno-Mining
  23. 23. …and participatory data interpretation MIXED METHODS – CASES Ethno-Mining (Intel) Anderson et al. 2009: Numbers Have Qualities Too: Experiences with Ethno-Mining
  24. 24. Take Aways
  25. 25. 1. Leave your method comfort zone. 2. Aim for highly integrated research designs with different angles. 3. Take advantage of participatory data interpretation. 4. Plan and decide deliberately. 5. It takes a lot of effort, but may lead you to your unknown unknowns. TAKE AWAYS
  26. 26. THANK YOU! Judith Mühlenhoff @judithmp www.user-research-lab.org
  27. 27. References Newslynx: http://newslynx.org/ Big Data: Kate Crawford: The hidden biases in data https://hbr.org/2013/04/the-hidden-biases-in-big-data/ Mikkel Krenchel and Christian Madsbjerg: Your big data is worthless if you don’t bring it into the real world http://www.wired.com/2014/04/your-big-data-is-worthless-if-you-dont-bring-it-into-the-real-world/ Experience Sampling: Kathy Baxter, You too can collect big data! https://www.epicpeople.org/pblog10/ Overview of programmes and apps: Experience Sampling and Ecological Momentary Assessment with Mobile Phones http://www.otago.ac.nz/psychology/otago047475.pdf Ethno-Mining: Anderson et al. 2009: Numbers Have Qualities Too: Experiences with Ethno-Mining Mixed Methods Research Design: Miles MB, Huberman AM (1994) Qualitative Data Analysis: An Expanded Sourcebook. Sage Publications, Inc, Thousand Oaks Tashakkori A, Teddlie C (2009) Integrating qualitative and quantitative approaches to research. In: Bickman L, Rog DJ (eds) The SAGE handbook of social research methods, 2d ed. Sage, Los Angeles, pp 283 – 317 Teddlie C, Tashakkori A (2006) A general typology of research designs featuring mixed methods. Res Sch 13:12–28.

×