Seeing the Elephant: Defragmenting User Research

66,443 views

Published on

Presented at Interact, October 2016 (London, UK); earlier versions in 2014 at UXPA Boston (Boston, MA, USA); in 2013 at Interaction S.A. (Recife, Brasil), Intuit (Mountain View, CA, USA), Designers + Geeks (New York, USA); in 2012 at UX Russia (Moscow, Russia), UX Hong Kong (Hong Kong, China), WebVisions NYC (New York, NY, USA); in 2011 at the IA Summit (Denver, CO, USA), UX-LX (Lisbon, Portugal), Love at First Website (Portland, OR, USA).

This is something of a successor to my talk "Marrying Web Analytics and User Experience" (http://is.gd/vK34zS)

Published in: Design, Business, Technology
14 Comments
194 Likes
Statistics
Notes
No Downloads
Views
Total views
66,443
On SlideShare
0
From Embeds
0
Number of Embeds
7,711
Actions
Shares
0
Downloads
1,029
Comments
14
Likes
194
Embeds 0
No embeds

No notes for slide
  • Image from http://assets.mytopfbcover.com/2012/11/08/3814/128730/funny-face-of-the-elephant_facebook_timeline_cover.jpg
  • http://www.youtube.com/watch?v=XRs7BJP6Ky4&feature=youtu.be
  • http://www.city-data.com/forum/members/johndbaumgardner-637750-albums-beautiful-cleveland-ohio-pic38486-standing-base-beautiful-rather-imposing-keycorp.jpg
  • http://dell.com
  • http://lukewalsh.co.uk/blog/uploaded_images/call-center-738087.jpg
  • http://www.cpasitesolutions.com/youget/cpa-website-marketing/google-analytics-for-accountants.php
  • http://community.acstechnologies.com/wp-content/uploads/2010/08/megaphone-stickman.jpg
  • http://www.crm-reviews.com/vendor-review/salesforce-com-crm-review/
  • http://www.research.ibm.com/images/about/labs/wat_outside.jpg
  • http://www.boxesandarrows.com/files/banda/where-is-your-mental/indiyoung.mentalmodel.large.png
  • http://www.blackcoffee.com/blog/wp-content/uploads/2009/10/brand-architecture.jpg
  • http://www.renps.com/images/NPSpic.png
  • http://pisspoordesign.wordpress.com/page/2/
    “Piss poor design”
  • http://1.bp.blogspot.com/-3buwQCBdO8E/Tkis7YLgTBI/AAAAAAAAAUg/qV_44w9gKUw/s1600/Blind+men+and+elephant.jpg
    Vive l’difference! (differences are a source of strength--if recognized/exploited)
  • http://blog.ideaworks.com/wp-content/uploads/2010/06/Quantitative-vs.-Qualitative1.jpg
  • http://www.writeforhr.com/wp-content/uploads/2010/04/Key-Performance-Indicators.jpg
    http://graffletopia.com/stencils/644
  • http://www.planetperplex.com/en/item/the-mysterious-island/
    TWISTY COURSE OF STARTUPS (AND THEIR ABILITY TO PIVOT ON DATA) SHOWS THE WORLD WE DON’T KNOW
  • http://www.quantshare.com/Images/tutorials/tutorial_statistical_data_analysis_1.gif
    http://www.thetechherald.com/media/images/200819/PostIt_16.jpg
  • http://www.pentagonpost.com/wp-content/uploads/2013/10/balanced_diet.jpg
    largely a diagnostic process to help us determine:* we don’t know what we don’t know (helps w/diagnostics)* we don’t know when to use what
  • http://www.xdstrategy.com/blog/
  • Can a persona• Be data-enriched?• Borrow from analytics segments?• ...and vice versa?
    Adapted from an Adaptive Path persona
  • http://2.bp.blogspot.com/-6jg9sv4lX8o/T05IsomD01I/AAAAAAAAB1g/UrdljvHNS5Y/s1600/music-clipart4.jpg
    understand/make sense of research in time (as opposed to balance, which maps it in space)
  • http://3.bp.blogspot.com/-pdziO1-SUQ0/TiylnsXmwcI/AAAAAAAAAnU/z_4Ctf9YK-4/s1600/128787933020784313.jpg
  • Dave Gray’s article/diagram: http://www.gogamestorm.com/?p=58
  • http://www.politicususa.com/wp-content/uploads/Angry-Palin1-300x225.jpg
  • http://atomic-candy.com/wp-content/uploads/2012/11/candy.jpg
  • moving from maps to containers--from seeing to doing
    http://julieanimation.blogspot.ca/2010/12/3-point-perspective-4-point-perspective.html
  • http://marion.sanap.org.za/MapPrinceEdward3d.jpg
  • http://www.xdstrategy.com/blog/
  • http://www.gatekeeperusainc.com/
  • http://www.inetsoft.com/images/screenshots/an_executive_dashboard.png
    ...but beware dashboards; the metaphor will only take you so far.
  • Wikipedia image from http://en.wikipedia.org/wiki/File:Viegas-UserActivityonWikipedia.gif
  • http://2.bp.blogspot.com/_wb8bAl1P-N0/TOFKkD6iQII/AAAAAAAARyQ/mmGzMbjvVrk/s1600/blue-sky.jpeg
    IT’S NO ONE’S FAULT THAT IT ENDED UP THIS WAY...
  • http://4.bp.blogspot.com/_cAFRZohUKig/TJd3XS2t-pI/AAAAAAAAAMQ/kWwHpDic1K4/s1600/600px-ConferenceBike.jpg
  • Seeing the Elephant: Defragmenting User Research

    1. Beyond User Research Lou Rosenfeld •  lou@rosenfeldmedia.com Interact • 19 October 2016
    2. November 14, 2013: Barack Obama, Commander-in-Chief
    3. November 14, 2013: Barack Obama, Commander-in-Chief
    4. November 14, 2013: Barack Obama, User Researcher-in-Chief
    5. What does victory look like?
    6. User research in today’s organization
    7. Reports from the user research group
    8. Query data gleaned from site search team XXX.XXX.X.104 - - [10/Jul/2013:10:25:46 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date %3AD%3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www& oe=UTF-8&proxystylesheet=www&q=lincense +plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02 XXX.XXX.X.104 - - [10/Jul/2013:10:25:48 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date %3AD%3AL%3Ad1&ie=UTF-8&client=www&q=license +plate

    9. Logs from the call center
    10. Reports from analytics applications
    11. Insights from Voice of the Customer research
    12. Reports from CRM applications
    13. Surveys behind Net Promoter Score
    14. Papers from the research center
    15. One agency’s user mental model
    16. Another agency’s brand architecture research
    17. So why does so much design still SUCK?
    18. The blind men and the elephant
    19. What Why
    20. Methods employed: quantitative versus qualitative
    21. Goals: help org or users Organizational goals Users’ goals
    22. How they use data: measur world we know versus wo we don’t Measuring the world we know 
 Exploring the world we don’t
    23. ...I Descriptive data Statistical data
    24. This is true. The problem with dichotomies And so is this.
    25. This is true. The problem with dichotomies And so is this.
    26. Lou’s TABLE OF OVERGENERALIZED DICHOTOMIES Web Analytics User Experience What they analyze Users' behaviors (what's happening) Users' intentions and motives (why those things happen) What methods they employ Quantitative methods to determine what's happening Qualitative methods for explaining why things happen What they're trying to achieve Helps the organization meet goals (expressed as KPI) Helps users achieve goals (expressed as tasks or topics of interest) How they use data Measure performance (goal- driven analysis) Uncover patterns and surprises (emergent analysis) What kind of data they use Statistical data ("real" data in large volumes, full of errors) Descriptive data (in small volumes, generated in lab environment, full of errors)
    27. Four themes for getting to synthesis and insight 1. Balance 2. Cadence 3. Conversation 4. Perspective
    28. 1. Balance
    29. XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET / search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD %3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&prox ystylesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/ 1.1" 200 971 0 0.02 XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET / search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD %3AL%3Ad1&ie=UTF-8&client=www&q=license+plate
 &ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&i p=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16 Web analytics pro’s question:
 “Are we converting license plate renewals?” User researcher’s question:
 “What are people searching the most?” Balanced analysis
    30. Thanks example courtesy to Angel Brown, Ogilvy DigitalHealth Balance within methods
    31. Thanks example courtesy to Angel Brown, Ogilvy DigitalHealth Balance within methods
    32. Rohrer’s user research landscape Text Christian Rohrer: http://www.nngroup.com/articles/which-ux-research-methods/ Balance within practice
    33. Rohrer’s user research landscape Text
    34. 2. Cadence
    35. A research cadence from Whitney Quesenbery
    36. Cadence Weekly Call center data trend analysis 2 – 4 hours behavioral/quantitative Task analysis 4 – 6 hours behavioral/quantitative Quarterly Exploratory analysis of site analytics data 8 – 10 hours behavioral/qualitative User survey 16 – 24 hours attitudinal/quantitative Annually Net Promoter Score study 3 – 4 days attitudinal/quantitative Field study 4 – 5 days behavioral/qualitative
    37. Cadence Weekly Call center data trend analysis 2 – 4 hours behavioral/quantitative Task analysis 4 – 6 hours behavioral/quantitative Quarterly Exploratory analysis of site analytics data 8 – 10 hours behavioral/qualitative User survey 16 – 24 hours attitudinal/quantitative Annually Net Promoter Score study 3 – 4 days attitudinal/quantitative Field study 4 – 5 days behavioral/qualitative Cadence + Balance
    38. Cadence Weekly Call center data trend analysis 2 – 4 hours behavioral/quantitative Task analysis 4 – 6 hours behavioral/quantitative Quarterly Exploratory analysis of site analytics data 8 – 10 hours behavioral/qualitative User survey 16 – 24 hours attitudinal/quantitative Annually Net Promoter Score study 3 – 4 days attitudinal/quantitative Field study 4 – 5 days behavioral/qualitative Cadence + Balance
    39. 3. Conversation
    40. Develop a pidgin Dave Gray’s boundary matrix: http://bit.ly/gWoZQm KPI goals segments personas
    41. Ban words that impede conversations • Product names: Omniture,, SharePoint... • Methods: focus group,, usability test... • Departments: market research,, analytics... • Disciplines: business analysis,, information architecture... • Outcomes: portal, social media layer...
    42. Tell Stories
    43. Tell Stories SKU: #39072-2AH1
    44. Buy Candy for Strangers
    45. 4. Perspective
    46. Maps help us make sense by seeing things in new ways
    47. Rohrer’s user research landscape Text
    48. Avinash Kaushik’s visualization (from Web Analytics 2.0)
    49. Avinash Kaushik’s visualization (from Web Analytics 2.0)
    50. Avinash Kaushik’s visualization (from Web Analytics 2.0) “...while I have a bucket for ‘Voice of Customer,’ in hindsight I should have worked harder still to paint the full qual and quant picture....”
    51. Containers help us make sense by putting things together in new ways
    52. MailChimp’s UX team: drowning in data
    53. MailChimp + Evernote • Shared bucket of buckets (60 notebooks) • Email is the API • OCR’d (nice for SurveyMonkey reports) • Searchable! • Led to “regular data nerd lunches” MailChimp: on the threshold of synthesis
    54. MailChimp + Evernote • Shared bucket of buckets (60 notebooks) • Email is the API • OCR’d (nice for SurveyMonkey reports) • Searchable! • Led to “regular data nerd lunches” MailChimp: on the threshold of synthesis was getting closer to
    55. WeWork’s UX team: 
 working with a blank slate
    56. WeWork + AirTable Challenges • “Orgs don’t have good research memory” • Reports get in way of insights Approach • Atomic units smaller than reports—nuggets (350 interviews yield @3400 nuggets) • LOTS of metadata • Researcher:curator ratio—3:1
    57. WeWork: "nuggetization" + metadata
    58. WeWork: "nuggetization" + metadata nuggets metadata
    59. WeWork: "nuggetization" + metadata
    60. WeWork: "nuggetization" + metadata
    61. WeWork: filter/search this stuff
    62. WeWork: an insight
    63. Map + Container
    64. Map + Container = Dashboard
    65. WELCOME TO OPERATIONS
    66. DevOps => DecisionOps
    67. DevOps => DecisionOps DevOps
    68. DevOps => DecisionOps DevOps DesignOps
    69. DevOps => DecisionOps DevOps DesignOps ResearchOps
    70. DevOps => DecisionOps DevOps DesignOps ResearchOps DecisionOps
    71. How will you get 
 beyond user research?
    72. How will you get 
 beyond user research? How to move your organization from micro-decisions to major insights?
    73. How will you get 
 beyond user research? How to move your organization from micro-decisions to major insights? Who needs to be talking with whom?
    74. How will you get 
 beyond user research? How to move your organization from micro-decisions to major insights? Who needs to be talking with whom? Can balance, cadence, 
 conversation, and perspective help?
    75. How will you get 
 beyond user research? How to move your organization from micro-decisions to major insights? Who needs to be talking with whom? Can balance, cadence, 
 conversation, and perspective help? What kind of operations 
 will get you there?
    76. How will you get 
 beyond user research? How to move your organization from micro-decisions to major insights? Who needs to be talking with whom? Can balance, cadence, 
 conversation, and perspective help? What kind of operations 
 will get you there?
    77. Thanks! slides: 
 http://rfld.me/11FrI3o article:
 http://rfld.me/145ZccP Lou Rosenfeld @louisrosenfeld
 www.rosenfeldmedia.com • @rosenfeldmedia

    ×