Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Beyond User Research

87,076 views

Published on

Presented at EuroIA17, September 2017; World IA Day NYC, February 2017; Interact, October 2016 (London, UK); earlier versions in 2014 at UXPA Boston (Boston, MA, USA); in 2013 at Interaction S.A. (Recife, Brasil), Intuit (Mountain View, CA, USA), Designers + Geeks (New York, USA); in 2012 at UX Russia (Moscow, Russia), UX Hong Kong (Hong Kong, China), WebVisions NYC (New York, NY, USA); in 2011 at the IA Summit (Denver, CO, USA), UX-LX (Lisbon, Portugal), Love at First Website (Portland, OR, USA).

This is something of a successor to my talk "Marrying Web Analytics and User Experience" (http://is.gd/vK34zS)

Published in: Design, Business, Technology

Beyond User Research

  1. Beyond User Research Lou Rosenfeld •  lou@rosenfeldmedia.com #EuroIA17 • 30 September 2017
  2. (a talk has no name) Lou Rosenfeld •  lou@rosenfeldmedia.com EuroIA 2017 • 30 September 2017
  3. What might we become?
  4. What does victory look like?
  5. User research in today’s organization
  6. Reports from the user research group
  7. Query data from the search team XXX.XXX.X.104 - - [10/Jul/2013:10:25:46 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date% 3AD%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&clie nt=www&oe=UTF-8&proxystylesheet=www&q=lincens e+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02 XXX.XXX.X.104 - - [10/Jul/2013:10:25:48 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date% 3AD%3AL%3Ad1&ie=UTF-8&client=www&q=license+pl
  8. Logs from the call center
  9. Reports from analytics applications
  10. “Learnings” from Voice of the Customer research
  11. Reports from CRM applications
  12. Surveys behind Net Promoter Score
  13. Studies from the research center
  14. Analysis of social media
  15. Research on brand architecture
  16. Even users’ mental models
  17. Yet why does so much design still SUCK?
  18. Yet why does so much design still SUCK?
  19. Problem It’s only going to get worse
  20. The blind men and the elephant
  21. Problem It’s only going to get worse Opportunity How might we support 
 synthesis and insight?
  22. What Why
  23. Methods employed: quantitative versus qualitative
  24. Goals: help org or users Organizational goals Users’ goals
  25. How they use data: measur world we know versus wo we don’t Measuring the world we know 
 Exploring the world we don’t
  26. ...I Descriptive data Statistical data
  27. This is true. And so is this.
  28. This is true. And so is this.
  29. Lou’s TABLE OF OVERGENERALIZED DICHOTOMIES Web Analytics User Experience What they analyze Users' behaviors (what's happening) Users' intentions and motives (why those things happen) What methods they employ Quantitative methods to determine what's happening Qualitative methods for explaining why things happen What they're trying to achieve Helps the organization meet goals (expressed as KPI) Helps users achieve goals (expressed as tasks or topics of interest) How they use data Measure performance (goal- driven analysis) Uncover patterns and surprises (emergent analysis) What kind of data they use Statistical data ("real" data in large volumes, full of errors) Descriptive data (in small volumes, generated in lab environment, full of errors)
  30. Five themes for getting to synthesis and insight 1. Balance 2. Cadence 3. Conversation 4. Perspective 5. Operations
  31. 1. Balance
  32. XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET / search? access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1& ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystyle sheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02 XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET / search? access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1& ie=UTF-8&client=www&q=license+plate
 &ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&i p=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16 Web analytics asks:
 “Are we converting license plate renewals?” User researcher asks:
 “What are people searching the most?” Balanced analysis
  33. Thanks Balance within methods example courtesy to Angel Brown, Ogilvy DigitalHealth
  34. Thanks Balance within methods example courtesy to Angel Brown, Ogilvy DigitalHealth
  35. Rohrer’s user research landscape Text Christian Rohrer: http://www.nngroup.com/articles/which-ux-research-methods/ Balance within practice
  36. Rohrer’s user research landscape Text
  37. 2. Cadence
  38. A research cadence from Whitney Quesenbery
  39. Cadence Weekly Call center data trend analysis 2 – 4 hours behavioral + quantitative Task analysis 4 – 6 hours behavioral + quantitative Quarterly Exploratory analysis of site analytics data 8 – 10 hours behavioral + qualitative User survey 16 – 24 hours attitudinal + quantitative Annually Net Promoter Score study 3 – 4 days attitudinal + quantitative Field study 4 – 5 days behavioral + qualitative
  40. Cadence Weekly Call center data trend analysis 2 – 4 hours behavioral + quantitative Task analysis 4 – 6 hours behavioral + quantitative Quarterly Exploratory analysis of site analytics data 8 – 10 hours behavioral + qualitative User survey 16 – 24 hours attitudinal + quantitative Annually Net Promoter Score study 3 – 4 days attitudinal + quantitative Field study 4 – 5 days behavioral + qualitative Cadence + Balance
  41. Cadence Weekly Call center data trend analysis 2 – 4 hours behavioral + quantitative Task analysis 4 – 6 hours behavioral + quantitative Quarterly Exploratory analysis of site analytics data 8 – 10 hours behavioral + qualitative User survey 16 – 24 hours attitudinal + quantitative Annually Net Promoter Score study 3 – 4 days attitudinal + quantitative Field study 4 – 5 days behavioral + qualitative Cadence + Balance
  42. 3. Conversation
  43. Buy Candy for Strangers
  44. Tell Stories
  45. Tell Stories SKU: #39072-2AH1
  46. Ban words that impede conversations • Product names: Omniture,, SharePoint... • Methods: focus group,, usability test... • Departments: market research,, analytics... • Disciplines: business analysis,, information architecture... • Outcomes: portal, social media layer...
  47. Consider framing
  48. Consider framing Big Data
  49. Consider framing Big Data THICk Data
  50. Consider framing Big Data THICk Data SeeTriciaWang’s “Why Big Data 
 NeedsThick Data”: http://bit.ly/23E9qlv
  51. Develop a pidgin Dave Gray’s boundary matrix: http://bit.ly/gWoZQm KPI goals segments personas
  52. 4. Perspective
  53. Maps help us make sense by seeing things in new ways
  54. Rohrer’s user research landscape Text
  55. Kaushik’s Trinity Strategy
  56. Kaushik’s Trinity Strategy
  57. Kaushik’s Trinity Strategy Avinash Kaushik’s “Trinity:A Mindset & Strategic Approach“: http://bit.ly/2yKA9CC “...while I have a bucket for ‘Voice of Customer,’ in hindsight I should have worked harder still to paint the full qual and quant picture….” —Kaushik (in private email)
  58. Containers help us make sense by putting things together in new ways
  59. MailChimp’s UX team: drowning in data
  60. MailChimp + Evernote • Shared bucket of buckets (60 notebooks) • Email is the API • OCR’d (nice for SurveyMonkey reports) • Searchable! • Led to “regular data nerd lunches” MailChimp: on the threshold of synthesis
  61. MailChimp + Evernote • Shared bucket of buckets (60 notebooks) • Email is the API • OCR’d (nice for SurveyMonkey reports) • Searchable! • Led to “regular data nerd lunches” MailChimp: on the threshold of synthesis was getting closer to
  62. 5. Operations
  63. WeWork’s Polaris: 
 working with a blank slate
  64. Research challenges at WeWork 1. Siloed research 2. Gaps in research memory 3. Reports instead of insight
  65. WeWork: "nuggetization" + metadata
  66. WeWork: "nuggetization" + metadata nuggets metadata
  67. WeWork: "nuggetization" + metadata
  68. WeWork: "nuggetization" + metadata
  69. WeWork: filter/search this stuff •
  70. WeWork: an insight(nuggets)
  71. WeWork: "nuggetization" + metadata nuggets metadata
  72. WeWork’s approach • Atomic units smaller than reports—nuggets (350 interviews yield @3400 nuggets) • LOTS of metadata • Findability improves organizational research memory • Researcher:curator ratio is 3:1
  73. WeWork’s approach • Atomic units smaller than reports—nuggets (350 interviews yield @3400 nuggets) • LOTS of metadata • Findability improves organizational research memory • Researcher:curator ratio is 3:1 …is an IA approach…
  74. WeWork’s approach • Atomic units smaller than reports—nuggets (350 interviews yield @3400 nuggets) • LOTS of metadata • Findability improves organizational research memory • Researcher:curator ratio is 3:1 …is an IA approach… …and an Ops approach
  75. Early efforts to operationalize insight
  76. Early efforts to operationalize insight
  77. Early efforts to operationalize insight
  78. Early efforts to operationalize insight
  79. Early efforts to operationalize insight
  80. Early efforts to operationalize insight information architects in Wellington interaction designers in London industrial designers in Seattle market researchers in London
  81. from DevOps: https://devops.com/2014/04/07/evolve-devops/ DevOps => DecisionOps DevOps
  82. from DevOps: https://devops.com/2014/04/07/evolve-devops/ DevOps => DecisionOps DevOps DesignOps
  83. from DevOps: https://devops.com/2014/04/07/evolve-devops/ DevOps => DecisionOps DevOps DesignOps ResearchOps
  84. from DevOps: https://devops.com/2014/04/07/evolve-devops/ DevOps => DecisionOps DevOps DesignOps ResearchOps InsightOps
  85. from DevOps: https://devops.com/2014/04/07/evolve-devops/ DevOps => DecisionOps DevOps DesignOps ResearchOps InsightOps Operations is nascent CreativeOps, Social Media Ops, DataOps…
  86. Operations is…
  87. Operations is… a platform
 (systems • infrastructure • processes • tooling • principles)
  88. Operations is… a platform
 (systems • infrastructure • processes • tooling • principles) that enables and amplifies the “talent”
  89. Operations is… a platform
 (systems • infrastructure • processes • tooling • principles) that enables and amplifies the “talent” maximizes efficiency
  90. Operations is… a platform
 (systems • infrastructure • processes • tooling • principles) that enables and amplifies the “talent” maximizes efficiency and makes sense of the unknown
  91. Operations is… a platform
 (systems • infrastructure • processes • tooling • principles) that enables and amplifies the “talent” maximizes efficiency and makes sense of the unknown
  92. Operations requires IA to enable
  93. Operations requires IA to enable balance
  94. Operations requires IA to enable balance cadence
  95. Operations requires IA to enable balance cadence conversation
  96. Operations requires IA to enable balance cadence conversation perspective
  97. Operations requires IA to enable balance cadence conversation perspective
  98. What might we become?
  99. Is Information Architecture more Operations than anything else?
  100. Architecting for insight Lou Rosenfeld •  lou@rosenfeldmedia.com #EuroIA17 • 30 September 2017
  101. Thanks! slides: 
 http://rfld.me/11FrI3o related article:
 http://rfld.me/145ZccP Lou Rosenfeld @louisrosenfeld
 www.rosenfeldmedia.com • @rosenfeldmedia

×