The Report is Dead, Long Live the Report ! Communicating Usability Research Findings for Maximum Impact
•
0 likes•25 views
The best way to improve products is to have people use them, but UX researchers struggle to share what they’ve learned in a way that has immediate and long-lasting impact. How do we keep the design process moving while grounding it thoroughly in research? This talk presents evidence for and against reports, and explores characteristics of reports that make them more and less successful at effecting change. We describe where approaches like debriefs, co-design, and video have succeeded and fallen short. Based on survey data from UX practitioners and experiences in the field, we address these questions: Is it worth it to write a report? Are there quicker, more engaging alternatives? What makes a compelling report? How do we make usability research usable? We offer a framework for choosing the best reporting approach, and share best practices for determining what to communicate, and how.
Recommended
More Related Content
Similar to The Report is Dead, Long Live the Report ! Communicating Usability Research Findings for Maximum Impact
Similar to The Report is Dead, Long Live the Report ! Communicating Usability Research Findings for Maximum Impact (20)
More from Centralis
More from Centralis (12)
Recently uploaded
Recently uploaded (20)
The Report is Dead, Long Live the Report ! Communicating Usability Research Findings for Maximum Impact
- 1. The Report is Dead, Long Live the Report! How to Communicate Usability Research Findings for Maximum Impact Kathi Kaiser Centralis @kathikaiser www.linkedin.com/in/kathikaiser
- 2. • Co-founder of Centralis • 20+ years making products easier, mostly through user research • Lots of training/mentoring of new UXers along the way • Observer and creator of various reporting approaches Hello! Hi! I’m Kathi Kaiser.
- 3. Back in the day… When I began in UX in the late 1990s, few people knew what UX research was.
- 4. Nowadays… In 25 years, UX researchers have grown in strength and numbers.
- 5. And so… Given our range of approaches, we should think critically about what we do, and why. • How do we share research results? • What seems to work best? • What challenges are we facing? • How should we adapt to different contexts? • How can we get beyond “It depends…”?
- 6. We researched… We solicited input from ~100 UX practitioners, via two surveys and follow-up interviews.
- 7. The greater the distance between the researcher and the stakeholders, the more formal results sharing tends to be. And we learned... 1. Distance demands documentation.
- 8. Gaining buy-in for UX research results is more about a collaborative process than the artifacts produced. And we learned... 2. Process outperforms artifacts.
- 9. The best way to package results depends on the researcher’s available time and the stakeholders’ available attention. And we learned... 3. Optimize time and attention.
- 11. Distance Demands Documentation Researchers share results in a wide range of ways, usually by creating an “artifact”.
- 12. Distance Demands Documentation Product Execs Product Owner Designers Devs Imagine a sphere…
- 13. Employee Vendor Not on the Product Team On the Product Team Distance Demands Documentation Product Team Execs Product Owner Designers Devs Where do UX researchers fit in?
- 14. Where do UX researchers fit in? The Integrated Model (Employee) Product Team Execs Product Owner Designers Devs Product Team Execs Product Owner Embedded UX Researcher Designers Devs Employee Vendor Not on the Product Team On the Product Team Distance Demands Documentation
- 15. Product Execs Product Owner Designers Devs Employee Vendor Not on the Product Team On the Product Team Product Execs Product Owner Embedded UX Researcher Designers Devs Distance Demands Documentation The Integrated Model (Contractor) Product Team Execs Embedded UX Researcher Product Owner Designers Devs
- 16. Product Team Execs Product Owner Designers Devs Employee Vendor Not on the Product Team On the Product Team Distance Demands Documentation The Service Model In-House UX Research Consultant
- 17. Product Team Execs Product Owner Designers Devs Employee Vendor Not on the Product Team On the Product Team Distance Demands Documentation The Outsourced Model External UX Research Consultant
- 18. Product Team Execs Product Owner Designers Devs Employee Vendor Not on the Product Team On the Product Team Distance Demands Documentation The Outsourced Model In-House UX Research Consultant External UX Research Consultant
- 19. Product Team Execs Product Owner Designers Devs Employee Vendor Not on the Product Team On the Product Team Distance Demands Documentation Research must span different distances to reach the stakeholders in each model. Product Team Execs Product Owner Integrated Model Designers Devs Service Model Outsourced Model
- 20. Distance Demands Documentation Scope of Research Question Range of Audience Text vs. Visual “I'll write up, here's what I think the main findings were in basically a one- page kind of thing.” Time to Create Shelf Life “Everybody knows that producing written documents in most organizational cultures is not the way to affect change.” “Usually, we're bumping up at the time when the product manager has to hand [the product] over to engineering for development.” Sharing in the Integrated Model “It's mostly just the bare bones: what did we observe, what did we learn from it, what do we need to do now.”
- 21. Distance Demands Documentation Scope of Research Question Range of Audience Text vs. Visual “I sit within our Consumer Insights Department, which is an overarching research center of excellence – that area supports the entire enterprise.” Time to Create Shelf Life “There's like 15 years of research…some of the insights, drivers, motivations, those things are still the same. And so they're still relevant. And so it's helpful.” Sharing in the Service Model “The way we write our reports is having that executive summary up front, because we are thinking about the executives… tell me what you're going to do. What needs to be fixed and why?
- 22. Distance Demands Documentation Scope of Research Question Range of Audience Text vs. Visual A: “Maybe less than 20 hours for a straightforward usability study…a big user research project, like could be like 40 or 50 or more hours.” Time to Create Shelf Life “I always walk through the whole report. So usually I book an hour, which is not always enough.” “I had a client reach out to me a couple of months ago who was like, we found your report from three years ago and it's incredible…we'd like you to do a new project with us.” Sharing in the Outsourced Model Q: “How long does it take you to write a usability report?” A: “Oh, forever. It takes forever.”
- 23. Distance Demands Documentation Scope of Research Question Range of Audience Text vs. Visual Time to Create Shelf Life Sharing in the Outsourced Model Scope of Research Question Range of Audience Text vs. Visual Time to Create Shelf Life Sharing in the Service Model Scope of Research Question Range of Audience Text vs. Visual Time to Create Shelf Life Sharing in the Integrated Model
- 24. Distance Demands Documentation “You want your stakeholders to DO something with your report; otherwise, your report is useless. Why are you doing this work? It’s because you want change to happen. The way to make change happen is to involve the stakeholders from the beginning.”
- 26. “Making the results easy to comprehend and showing visual examples” “Including an exec summary of key insights/ recommendations so they can quickly grasp the "greatest hits".” “Sharing participant quotes/anecdotes from testing” “…video clips showing participants failing (and succeeding).” Process Over Artifacts Q: What have you found most effective for sharing usability findings?(2020) Q: What have you found most effective for getting buy-in to the results of usability testing?(2023) “Building relationships and asking questions about the plans and roadmap” “Require that stakeholders observe usability testing.” “Collaboration on the goals and planning of the test” “Review the test plan with PM and UX and Engineering key stakeholders” “Including responsible stakeholders in planning and analysis.” “Having stakeholders watch… take notes on a shared sticky note space and ‘affinitizing’ the findings in a group workshop.” “Give stakeholders the opportunity to discuss/ask questions” “Cross-functional, collaborative analysis.” “Socializing findings ahead of the formal report also helps, as do clear recommendations.”
- 27. Process Over Artifacts A Common Usability Research Process
- 28. Process Over Artifacts Level of Involvement R Level of Involvement R R R R
- 29. Process Over Artifacts Level of Involvement: Integrated Model R Level of Involvement R R R R S S S S S Integrated Model “I know the stakeholders I'm working with and I know what problem we're focused on at that time.”
- 30. Process Over Artifacts Level of Involvement: Service Model R Level of Involvement R R R R S S S S S “If we're going to do research for you, you as a business owner need to participate. You cannot just wait for the report to be thrown over the wall.” Service Model
- 31. Process Over Artifacts Level of Involvement: Outsourced Model R Level of Involvement R R R R S S S S S Outsourced Model “And then there's, you know, an analysis phase…usually it's just me, where I try to make sense of all the stuff that we saw in the usability testing.”
- 32. Process Over Artifacts Level of Involvement R Level of Involvement R R R R S S S S S S S S S S S S S S S Integrated Model Service Model Outsourced Model
- 33. R S S Process Over Artifacts Boosting Stakeholder Involvement Level of Involvement R R R S S S S S S S S S S R I S S R I S S R S S S Integrated Model Service Model Outsourced Model
- 34. R S S Process Over Artifacts Boosting Stakeholder Involvement Level of Involvement R R R S S S S S S S S S S R I S S R I S S R S S S Integrated Model Service Model Outsourced Model ?
- 35. Process Over Artifacts 35 • The debrief session told us what we need to know • No one has time to write a report • No one will read a report • The design has already moved on • The debrief is for the team; the report is for the executives • The report is our memory • The report is our roadmap • The report is our product The Report is Dead! Long Live the Report! Integrated Model Service Model Outsourced Model
- 36. Optimize your time and their attention.
- 37. Optimizing for Time and Attention A Taxonomy of Reports (Cooley & Barnum, 2019) Cooley, D., Barnum, C. (2019). How Do Other People Do It? A Comparative Review of Usability Study Reports. User Experience Magazine, 19(1). Retrieved from https://uxpamagazine.org/how-do-other-people-do-it-a-comparative-review-of-usability-study-reports/ Visuals from DialogDesign: https://www.dialogdesign.dk/cue-studies/ ‘Traditional’ Reports ‘Visual’ Reports ‘Lean’ Reports
- 38. Optimizing for Time and Attention Report Building Blocks 38 Key Findings Product Images Quotes Video Clips Data Viz Detailed Findings Severity Ratings Session Metrics Recos Method What we did How to illustrate it What to do about it What we learned
- 39. Optimizing for Time and Attention Report Building Blocks 39 Key Findings Detailed Findings Severity Ratings Recos Method What we did How to illustrate it What to do about it Traditional Focus What we learned Product Images Quotes Video Clips Data Viz Session Metrics
- 40. Optimizing for Time and Attention Report Building Blocks 40 Key Findings Detailed Findings Severity Ratings Recos Method What we did How to illustrate it What to do about it Visual Focus What we learned Product Images Quotes Video Clips Data Viz Session Metrics
- 41. Lean Focus Optimizing for Time and Attention Report Building Blocks 41 Key Findings Detailed Findings Severity Ratings Recos Method What we did How to illustrate it What to do about it Lean Focus What we learned Product Images Quotes Video Clips Data Viz Session Metrics
- 42. Optimizing for Time and Attention Report Building Blocks Key Findings Product Images Quotes Video Clips Data Viz Detailed Findings Severity Ratings Session Metrics Recos Method Stakeholder Attention Researcher Time
- 43. Optimizing for Time and Attention The Attention Paradox: An Exercise 43 N P R F D R C I A P D Q B M W J F K
- 44. Optimizing for Time and Attention The Attention Paradox: An Exercise 44 N P R F D R C I A P D Q B M W J F K
- 45. Optimizing for Time and Attention The Attention Paradox: An Exercise 45 N P R F D R C I A P D Q B M W J F K N P R F D R C IA P D Q B M W J F K
- 46. Optimizing for Time and Attention The Attention Paradox 46 Our attention is limited. The less attention needed to process something, the more attention available for processing it. We capture more attention by requiring less attention.
- 47. Optimizing for Time and Attention Report Building Blocks Key Findings Product Images Quotes Video Clips Data Viz Detailed Findings Severity Ratings Session Metrics Recos Method Stakeholder Attention Researcher Time
- 48. Optimizing for Time and Attention Report Building Blocks Product Images Quotes Video Clips Data Viz Detailed Findings Severity Ratings Session Metrics Recos Method Stakeholder Attention Researcher Time Key Findings
- 49. Optimizing for Time and Attention The Elevator Pitch Game Claire Slattery, https://claireslatterycoaching.com/
- 50. Optimizing for Time and Attention Report Building Blocks Quotes Video Clips Data Viz Detailed Findings Severity Ratings Session Metrics Recos Method Stakeholder Attention Researcher Time Key Findings Key Findings Product Images
- 51. Optimizing for Time and Attention Report Building Blocks Video Clips Data Viz Detailed Findings Severity Ratings Session Metrics Recos Method Stakeholder Attention Researcher Time Key Findings Key Findings Product Images Key Findings Product Images Quotes
- 52. Optimizing for Time and Attention Otter.ai 52
- 53. Optimizing for Time and Attention Report Building Blocks Data Viz Detailed Findings Severity Ratings Session Metrics Recos Method Stakeholder Attention Researcher Time Key Findings Key Findings Product Images Key Findings Product Images Quotes Key Findings Product Images Quotes Video Clips
- 54. Optimizing for Time and Attention Reduct.video 54
- 55. Optimizing for Time and Attention Report Building Blocks Detailed Findings Severity Ratings Session Metrics Recos Method Stakeholder Attention Researcher Time Key Findings Key Findings Product Images Key Findings Product Images Quotes Key Findings Product Images Quotes Video Clips Key Findings Product Images Quotes Video Clips Data Viz
- 56. Method Method Optimizing for Time and Attention Report Building Blocks Detailed Findings Severity Ratings Session Metrics Recos Method Stakeholder Attention Researcher Time Key Findings Key Findings Product Images Key Findings Product Images Quotes Key Findings Product Images Quotes Video Clips Key Findings Product Images Quotes Video Clips Data Viz Severity Ratings Session Metrics Move to Appendix Move to Debrief & Preview Exclude Severity Ratings
- 57. Optimizing for Time and Attention Report Building Blocks Detailed Findings Recos Stakeholder Attention Researcher Time Key Findings Key Findings Product Images Key Findings Product Images Key Findings Product Images Key Findings Product Images Stakeholder Attention Researcher Time Detailed Findings Detailed Findings Product Images Detailed Findings Product Images Quotes Detailed Findings Product Images Quotes Video Clips Detailed Findings Product Images Quotes Video Clips Data Viz Quotes Quotes Video Clips Quotes Video Clips Data Viz Quotes Quotes Video Clips Quotes Video Clips Data Viz
- 58. Optimizing for Time and Attention Report Building Blocks 58 Finding = Behavior Why did they do that? How did they react? What happened next? + Cause + Mindset + Impact
- 59. Optimizing for Time and Attention Story Arcs 59
- 60. Optimizing for Time and Attention Progressive Specificity Navigation to Screen Screen Layout Individual Screen Elements Call to Action Next Screen >
- 61. Optimizing for Time and Attention Should researchers make recommendations? No Yes Recos
- 62. Optimizing for Time and Attention Should researchers make recommendations? 62 “That can create a silo effect and potentially, antagonism. Research has come here to tell you all the things that are wrong and all the things you should have done under the guise of ‘recommendations’.” “You don't tell me what to do and I won't tell you what to do because obviously you're a professional who has skills and knowledge that I don’t.” No Yes
- 63. Optimizing for Time and Attention Should researchers make recommendations? 63 No Yes “Some product managers were like, who do you think you are, trying to drive these next steps. That was when I was new. Now, I’m like, hey, we're having next steps meeting from the research and we're going to all discuss together what we're doing next.”
- 64. Optimizing for Time and Attention Should researchers make recommendations? 64 “Recommendations are a key part of what we do. Why be the one who holds all of the knowledge on the findings and not tell people what they should, ideas for things they should do about that?” No Yes
- 65. Optimizing for Time and Attention Should researchers make recommendations? 65 “I usually just do it because I need it to make the next steps in an iteration.” No Yes
- 66. Let’s Review
- 67. The greater the distance between the researcher and the stakeholders, the more formal results sharing tends to be. Choose your reporting approach in part based on whether you work in an integrated, service, or outsourced model. In Summary 1. Distance demands documentation.
- 68. Gaining buy-in for UX research results is more about a collaborative process than the artifacts produced. Loop in stakeholders early and often. In Summary 2. Process outperforms artifacts.
- 69. The best way to package results depends on the researcher’s available time and the stakeholders’ available attention. Choose your components wisely, and learn how to lighten their load on everyone. In Summary 3. Optimize time and attention.
- 70. In Summary The Report is Dead!
- 71. In Summary The Report is Dead! Long Live the Report! (and also collaborate along the way…)