Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (10)
Similar to UXPA 2016: Mixed Methods Research in the Age of Big Data
Similar to UXPA 2016: Mixed Methods Research in the Age of Big Data (20)
Recently uploaded
Recently uploaded (20)
UXPA 2016: Mixed Methods Research in the Age of Big Data
- 1. Mixed Methods Research in the Age of Big Data A Primer for UX Professionals
- 2. Zachary Sam Zaiss UX Data Scientist | Microsoft Cloud @zszaiss 2006 2012 2016 UX Researcher UX DS Berkeley MIDS
- 3. Gartner Hype Cycle for Emerging Technologies: 2014 http://www.gartner.com/newsroom/id/2819918
- 4. Gartner Hype Cycle for Emerging Technologies: 2015 http://www.gartner.com/newsroom/id/3114217
- 5. The Education Perspective https://whatsthebigdata.com/2012/08/09/graduate-programs-in-big-data-and-data-science/ http://uxmastery.com/resources/ux-degrees/ 84 78Graduate Degree Programs in Data Science Graduate Degree Programs in UX
- 7. We need to make collaboration with Data Scientists a priority… … and it starts with a conversation
- 8. Tip #1 Stake Your Claim
- 9. Qualitative Evaluation Criteria Talking Points Quantitative basis for n values https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ http://www.measuringu.com/blog/five-history.php http://www.measuringu.com/blog/five-for-five.php
- 10. Qualitative Evaluation Criteria Talking Points Quantitative basis for n values Existence Proof https://www.youtube.com/watch?v=3uqZPnxG4_w
- 11. Qualitative Evaluation Criteria Talking Points Quantitative basis for n values Existence Proof Grounded Theory Inductive vs. Deductive Reasoning http://www.slideshare.net/traincroft/hcic-muller-guha-davis-geyer-shami-2015-0629 Theory from Data Data from Theory
- 12. Qualitative Evaluation Criteria Talking Points Quantitative basis for n values Existence Proof Grounded Theory Inductive vs. Deductive Reasoning Constructivism vs. Determinism https://us.sagepub.com/en-us/nam/research-design/book237357
- 13. Discussing evaluation criteria for qualitative research needs to be second nature.
- 15. What is your mic drop moment?
- 16. Tip #1 Stake Your Claim
- 17. Tip #2 Speak the Language
- 18. vs
- 19. Models + Key Aspects of Analysis Descriptive Model Descriptive Statistics Statistical Significance What is the probability of obtaining this result given the null hypothesis is true? Practical Significance Is the effect on the outcome large enough to be considered relevant?
- 20. http://fivethirtyeight.com/features/statisticians-found-one-thing-they-can-agree-on-its-time-to-stop-misusing-p-values/ The statement process was lengthier and more controversial than anticipated.
- 21. 6 Principles for p-values from ASA’s Statement 1. P-values can indicate how incompatible the data are with a specified statistical model. 2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone. 3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value crosses a specific threshold. 4. Proper inference requires full reporting and transparency. 5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result. 6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis. http://amstat.tandfonline.com/doi/abs/10.1080/00031305.2016.1154108
- 22. Models + Key Aspects of Analysis Descriptive Model Descriptive Statistics Statistical Significance What is the probability of obtaining this result given the null hypothesis is true? Practical Significance Is the effect on the outcome large enough to be considered relevant? Predictive Model Super vised Machine Learning Accuracy How well does the model predict the outcome for new data cases?
- 24. Models + Key Aspects of Analysis Descriptive Model Descriptive Statistics Statistical Significance What is the probability of obtaining this result given the null hypothesis is true? Practical Significance Is the effect on the outcome large enough to be considered relevant? Predictive Model Super vised Machine Learning Accuracy How well does the model predict the outcome for new data cases? Representation Model Unsuper vised Machine Learning Optimization Criteria How will we determine that we’ve built a reasonable and appropriate representation model for our data?
- 25. vs
- 26. None of these measures get at the contextual meaning behind the model.
- 27. A Diagram for Product Manager… Source: Martin Eriksson, Mind the Product. http://www.mindtheproduct.com/2011/10/what-exactly-is-a-product-manager/
- 28. … And a Framework for Attributes UX Business Tech Experience Attributes Customer attributes that can explain how that customer will experience a product. Technology Attributes Customer attributes that can explain whether customers will have technical issues with a product. Business Attributes Customer attributes that can explain the extent to which the customer will contribute to business outcomes.
- 29. Example: Developer Tools X B T Prog Language Target Platform Project Complexity Project Audience Type of App Educational Background Keyboard Proclivity Project Complexity
- 30. Example: Freemium Games X B T Platform Used Facebook Connected Whale Status Completionist Tendencies Game Session Time
- 31. Example: Fitness Bands X B T Connected Devices Type / Version Frequency of Exercise Friends with Same Band Finger Shape (Fat Fingers) Farsightedness Skin Irritation
- 32. We are uniquely qualified to articulate the experience attributes of our products.
- 33. Tip #2 Speak the Language
- 35. A Metaphor for A/B Experiments
- 36. A Better Metaphor for A/B Experiments
- 37. How can we provide greater context to A/B test findings?
- 38. Heterogeneous Treatment Effects control treatment some kpi 0.71 0.72 product experts product novices control treatment converted didn‘t convert converted didn‘t convert control treatment converted didn‘t convert converted didn‘t convert Heterogeneous Treatment Effect
- 40. Tip #1: Stake Your Claim Tip #2: Speak the Language Tip #3: Get Involved
- 41. Mixed Methods Research in the Age of Big Data A Primer for UX Professionals http://www.uxpa.org/sessionsurvey?sessionid=113
Editor's Notes
- How many have engaged in Mixed Methods Research before – either individually or more broadly, on your UX team? How many of you have a Data Science group within your organization that is starting to look at customer outcomes? Keep your hand up if you’re completely satisfied with your working relationship with those Data Scientists, otherwise put your hand down. I’ve gotten a handle on three groups in the audience. For some of you, Data Science hasn’t entered into the space of customer understanding in your organizations. I’ll speak to that briefly – but as a forecast, that day is coming up fast. A bunch of you do have Data Scientists in your organization working on customer outcomes, but you aren’t satisfied with your working relationship. That’s what the bulk of this presentation will be about. The future of Mixed Methods Research requires the best of what both disciplines bring to the table, so we need to start building stronger relationships with each other. Before I dive in, I’ll give a nod to the folks who are happy with their connection with Data Scientists. This is a new space, and what may seem like second nature to you is a challenging space for many. It’s a challenging space for me, frankly. The way forward is to have multiple points of view about how we collaborate, but I’m consistently struck by the scarcity of Data Science topics at UX conferences and vice versa. So today I’m going to be adding my perspective to the discussion, and I hope you’ll all consider doing the same at future events.
- Introduction – UX Data Scientist Not going in depth into my title – showing this viz to communicate that I consider myself a qualitative researcher at heart. Made the switch – not because I was tired of qual research, or because I think Data Science is better. I’m just passionate about how these two disciplines collaborate, and formalizing a hybrid role seemed like the best position for me to start bringing that passion to life. So I’ve been talking about “Data Science” for a couple minutes now, but I want to spend a bit more time unpacking what’s happening in the Data Science space. And to tee up this discussion, I want to reference a comment that Susan made during the past, present, future panel on Wednesday – there’s a lot of buzz around UX, and we need to make sure that UX work is being represented well, and that people who lack the rigor of the discipline aren’t defining what it means for everyone. That point really resonated with me, though in terms of buzz right now, my perception is that Data Science has a lot more of it.
- Coursera: 302 matches for User Experience, first page has developer courses (presumably with a UX lesson). Compare that to 400+ Data Science matches – and they are all Data Science for the first many pages – and UIUC is even hosting a master’s degree completely on Coursera. There’s an interesting dynamic with these programs, because the draw for them – the promise for someone who is already quantitatively minded – is that there’s so much data, you’ll be able to answer any question you have. The appropriate nuance shows up later, in the details of the course, if you’re paying attention. So these graduates will be exploding onto the scene, ready to tackle every question in their way, and quite frankly most of them may not think that we have a role to play in the questions that they’re asking. So what do we do about that?
- kv: Sam, that all sounds great, but sometimes it literally feels like we’re speaking different languages.
- Remember my note that Data Scientists may be coming onto the scene without an understanding of what we do. That’s not entirely true, unfortunately – in more than a few cases, they may have a negative impression of what we do. While I don’t think it’s meant to be antagonistic, many Data Scientists frame qualitative research as dealing with “small data,” “anecdotal data,” or not even calling it data at all. (When I told someone about my background and what I’m doing now, his response was, “Oh cool – I’m excited to see someone in the UX space starting to work with the ‘real data.’”) The good news is that this role of educating stakeholders about what we do is very familiar to us – and I know this has already been discussed at length. My goal here isn’t to educate about THE education approach, but more to discuss this process from the Data Science perspective, and how I’ve addressed this education challenge while I was a qualitative user researcher.
- Jen’s Wednesday keynote
- It’s fine if your story is different than mine, but fundamentally, we need to be comfortable – and proactive – in sharing this story.
- Reinforce that this won’t necessarily be an easy conversation, and you may meet with some resistance. There’s no silver bullet to take someone from believing they have all the answers to accepting that there are different approaches of thinking and knowing. The good news is that, there are folks like me on the Data Science side – we’re chipping away at this communication problem from both ends. But even if the conversation starts with a negative outlook on what Uxers do, the act of having that conversation moves the collaboration potential forward for everyone.
- “The statement process was lengthier and more controversial than anticipated.”
- Principles are super approachable. They clearly articulate what a p-value can be used for and then spend the other 5 principles trying to address common misuse of this measure. Check out Principle #6 – what is this saying? If your Data Scientist is using the p-value by itself as a mic drop moment, that’s an opportunity for you to hook in. There’s a lot of context missing if all you’re working off of is the p-value, so don’t be afraid to jump in there. Let’s tie some thoughts together. Remember what I said up front about the recent hype around citizen data science – the idea that anyone can adapt a model and make it work for their context. These principles were a reaction to seasoned practitioners abusing some of the fundamental concepts of the discipline. What do you think will happen when there are folks engaging who don’t even have a rigorous background in the space? The bottom line is that statistics is hard. Most statistics classes have a unit up front that emphasizes how bad our brains are at thinking in probabilistic and statistical terms. And to reinforce the point – it took the foremost experts in Statistics over a year to release a statement about this fundamental point in the discipline. What can we do about this? I really appreciate Principle #4 – offering full reporting and transparency.
- Partially the long way around to say that qualitative research excels at answering the Why. But there’s more to it than that. We also need to be aware of the context of the data – what the data is representing and what it isn’t. The best machine learning model can only perform on the basis of the data that it’s given, and making sure the right data is included is a tough problem that we need to be a part of. There’s a growing body of research on how machine learning models can be biased, and it stems back to this point – the person building the model has intrinsic biases that prevent him or her from pursuing the type of data that’s appropriate to speak about a particular space.
- If collaboration is happening today, I tend to see that it’s centered on results sharing. This is great – but I’ve found more than a few times that my ability to influence strong research outcomes is pretty limited if I’m only engaging on the results end. A/B testing is a common example, so I’ll close out with some ideas around how to start a conversation around that.
- “The statement process was lengthier and more controversial than anticipated.”