Seeing the Elephant: Defragmenting User Research

51,017
-1

Published on

Presented at UXPA Boston May 2014, Interaction S.A. (Recife, Brasil) November 2013 and Intuit (Mountain View, CA, USA) October 2013; earlier version given in 2013 in NYC at Designers + Geeks. Given in 2012 at UX Russia (http://uxrussia.com/), UX Hong Kong (http://www.uxhongkong.com/) and WebVisions NYC (http://www.webvisionsevent.com/new-york/). Given in 2011 at the IA Summit (http://2011.iasummit.org/), UX-Lisbon (http://ux-lx.com), and Love at First Website (http://www.isitedesign.com/love/).

This is something of a successor to my talk "Marrying Web Analytics and User Experience" (http://is.gd/vK34zS)

Published in: Design, Business, Technology
14 Comments
187 Likes
Statistics
Notes
No Downloads
Views
Total Views
51,017
On Slideshare
0
From Embeds
0
Number of Embeds
28
Actions
Shares
0
Downloads
1,002
Comments
14
Likes
187
Embeds 0
No embeds

No notes for slide
  • Image from http://assets.mytopfbcover.com/2012/11/08/3814/128730/funny-face-of-the-elephant_facebook_timeline_cover.jpg
  • http://www.youtube.com/watch?v=XRs7BJP6Ky4&feature=youtu.be
  • http://www.city-data.com/forum/members/johndbaumgardner-637750-albums-beautiful-cleveland-ohio-pic38486-standing-base-beautiful-rather-imposing-keycorp.jpg
  • http://dell.com
  • http://lukewalsh.co.uk/blog/uploaded_images/call-center-738087.jpg
  • http://www.cpasitesolutions.com/youget/cpa-website-marketing/google-analytics-for-accountants.php
  • http://community.acstechnologies.com/wp-content/uploads/2010/08/megaphone-stickman.jpg
  • http://www.crm-reviews.com/vendor-review/salesforce-com-crm-review/
  • http://www.research.ibm.com/images/about/labs/wat_outside.jpg
  • http://www.boxesandarrows.com/files/banda/where-is-your-mental/indiyoung.mentalmodel.large.png
  • http://www.blackcoffee.com/blog/wp-content/uploads/2009/10/brand-architecture.jpg
  • http://www.renps.com/images/NPSpic.png
  • http://pisspoordesign.wordpress.com/page/2/
    “Piss poor design”
  • http://1.bp.blogspot.com/-3buwQCBdO8E/Tkis7YLgTBI/AAAAAAAAAUg/qV_44w9gKUw/s1600/Blind+men+and+elephant.jpg
    Vive l’difference! (differences are a source of strength--if recognized/exploited)
  • http://blog.ideaworks.com/wp-content/uploads/2010/06/Quantitative-vs.-Qualitative1.jpg
  • http://www.writeforhr.com/wp-content/uploads/2010/04/Key-Performance-Indicators.jpg
    http://graffletopia.com/stencils/644
  • http://www.planetperplex.com/en/item/the-mysterious-island/
    TWISTY COURSE OF STARTUPS (AND THEIR ABILITY TO PIVOT ON DATA) SHOWS THE WORLD WE DON’T KNOW
  • http://www.quantshare.com/Images/tutorials/tutorial_statistical_data_analysis_1.gif
    http://www.thetechherald.com/media/images/200819/PostIt_16.jpg
  • http://www.pentagonpost.com/wp-content/uploads/2013/10/balanced_diet.jpg
    largely a diagnostic process to help us determine:* we don’t know what we don’t know (helps w/diagnostics)* we don’t know when to use what
  • http://www.xdstrategy.com/blog/
  • Can a persona• Be data-enriched?• Borrow from analytics segments?• ...and vice versa?
    Adapted from an Adaptive Path persona
  • http://2.bp.blogspot.com/-6jg9sv4lX8o/T05IsomD01I/AAAAAAAAB1g/UrdljvHNS5Y/s1600/music-clipart4.jpg
    understand/make sense of research in time (as opposed to balance, which maps it in space)
  • http://3.bp.blogspot.com/-pdziO1-SUQ0/TiylnsXmwcI/AAAAAAAAAnU/z_4Ctf9YK-4/s1600/128787933020784313.jpg
  • Dave Gray’s article/diagram: http://www.gogamestorm.com/?p=58
  • http://www.politicususa.com/wp-content/uploads/Angry-Palin1-300x225.jpg
  • http://atomic-candy.com/wp-content/uploads/2012/11/candy.jpg
  • moving from maps to containers--from seeing to doing
    http://julieanimation.blogspot.ca/2010/12/3-point-perspective-4-point-perspective.html
  • http://marion.sanap.org.za/MapPrinceEdward3d.jpg
  • http://www.xdstrategy.com/blog/
  • http://www.gatekeeperusainc.com/
  • http://www.inetsoft.com/images/screenshots/an_executive_dashboard.png
    ...but beware dashboards; the metaphor will only take you so far.
  • Wikipedia image from http://en.wikipedia.org/wiki/File:Viegas-UserActivityonWikipedia.gif
  • http://2.bp.blogspot.com/_wb8bAl1P-N0/TOFKkD6iQII/AAAAAAAARyQ/mmGzMbjvVrk/s1600/blue-sky.jpeg
    IT’S NO ONE’S FAULT THAT IT ENDED UP THIS WAY...
  • http://4.bp.blogspot.com/_cAFRZohUKig/TJd3XS2t-pI/AAAAAAAAAMQ/kWwHpDic1K4/s1600/600px-ConferenceBike.jpg
  • Seeing the Elephant: Defragmenting User Research

    1. Seeing the Elephant Defragmenting User Research Lou Rosenfeld •  lou@rosenfeldmedia.com UXPA Boston • May 15, 2014
    2. November 14, 2013: User Researcher-in-Chief Barack Obama
    3. November 14, 2013: User Researcher-in-Chief Barack Obama
    4. What does victory look like?
    5. User research in today’s organization
    6. Reports from the user research group
    7. Query data gleaned from site search team XXX.XXX.X.104 - - [10/Jul/2013:10:25:46 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date %3AD%3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www& oe=UTF-8&proxystylesheet=www&q=lincense +plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02 XXX.XXX.X.104 - - [10/Jul/2013:10:25:48 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date %3AD%3AL%3Ad1&ie=UTF-8&client=www&q=license +plate
    8. Logs from the call center
    9. Reports from analytics applications
    10. Insights from Voice of the Customer research
    11. Reports from CRM applications
    12. Papers from the research center
    13. One agency’s user mental model
    14. Another agency’s brand architecture research
    15. Surveys behind Net Promoter Score
    16. So why does so much design still SUCK?
    17. The blind men and the elephant
    18. What Why
    19. Methods employed: quantitative versus qualitative
    20. Goals: help org or users Organizational goals Users’ goals
    21. How they use data: measur world we know versus wo we don’t Measuring the world we know Exploring the world we don’t
    22. Kind of data they use: statistical vs. descriptive Descriptive data Statistical data
    23. Lou’s TABLE OF OVERGENERALIZED DICHOTOMIES Web Analytics User Experience What they analyze Users' behaviors (what's happening) Users' intentions and motives (why those things happen) What methods they employ Quantitative methods to determine what's happening Qualitative methods for explaining why things happen What they're trying to achieve Helps the organization meet goals (expressed as KPI) Helps users achieve goals (expressed as tasks or topics of interest) How they use data Measure performance (goal- driven analysis) Uncover patterns and surprises (emergent analysis) What kind of data they use Statistical data ("real" data in large volumes, full of errors) Descriptive data (in small volumes, generated in lab environment, full of errors)
    24. Four themes for getting to synthesis 1. Balance 2. Cadence 3. Conversation 4. Perspective
    25. 1. Balance
    26. Rohrer/Mulder/Yaar’s map Qualitative (direct) Quantitative (indirect) Attitudinal Behavioral © 2008 Christian Rohrer Approach DataSource mix mix Scripted (often lab-based) use of product Natural use of product De-contextualized / not using product Key for Context of Product Use during data collection Combination / hybrid Focus Groups Phone Interviews Ethnographic Field Studies Cardsorting Diary/Camera Study Intercept Surveys Usability Lab Studies Eyetracking Usability Benchmarking (in lab) A/B (Live) Testing Online User Experience Assessments (“Vividence-like” studies) Desirability studies Data Mining/Analysis Email Surveys Message Board Mining Participatory Design Customer feedback via email / / 20 Landscape of User Research Methods  Text Christian Rohrer: http://bit.ly/eAlbe2 / Steve Mulder & Ziv Yaar, The User Is Always Right
    27. XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET / search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD %3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&prox ystylesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/ 1.1" 200 971 0 0.02 XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET / search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD %3AL%3Ad1&ie=UTF-8&client=www&q=license+plate &ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&i p=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16 Web analytics’ question: “Are we converting license plate renewals?” UX practitioner’s question: “What are people searching the most?” Balanced analysis
    28. Balance within a method
    29. Balance within a method
    30. 2. Cadence
    31. A research cadence from Whitney Quesenbery
    32. Cadence WeeklyWeeklyWeekly Call center data trend analysis 2 – 4 hours behavioral/quantitative Task analysis 4 – 6 hours behavioral/quantitative QuarterlyQuarterlyQuarterly Exploratory analysis of site analytics data 8 – 10 hours behavioral/qualitative User survey 16 – 24 hours attitudinal/quantitative AnnuallyAnnuallyAnnually Net Promoter Score study 3 – 4 days attitudinal/quantitative Field study 4 – 5 days behavioral/qualitative
    33. Cadence WeeklyWeeklyWeekly Call center data trend analysis 2 – 4 hours behavioral/quantitative Task analysis 4 – 6 hours behavioral/quantitative QuarterlyQuarterlyQuarterly Exploratory analysis of site analytics data 8 – 10 hours behavioral/qualitative User survey 16 – 24 hours attitudinal/quantitative AnnuallyAnnuallyAnnually Net Promoter Score study 3 – 4 days attitudinal/quantitative Field study 4 – 5 days behavioral/qualitative Cadence + Balance
    34. Cadence WeeklyWeeklyWeekly Call center data trend analysis 2 – 4 hours behavioral/quantitative Task analysis 4 – 6 hours behavioral/quantitative QuarterlyQuarterlyQuarterly Exploratory analysis of site analytics data 8 – 10 hours behavioral/qualitative User survey 16 – 24 hours attitudinal/quantitative AnnuallyAnnuallyAnnually Net Promoter Score study 3 – 4 days attitudinal/quantitative Field study 4 – 5 days behavioral/qualitative Cadence + Balance
    35. 3. Conversation
    36. Develop a pidgin Dave Gray’s boundary matrix: http://bit.ly/gWoZQm KPI goals segments personas
    37. Ban words that impede conversations • Product names: Omniture,, SharePoint... • Methods: focus group,, usability test... • Departments: market research,, analytics... • Disciplines: business analysis,, information architecture... • Outcomes: portal, social media layer...
    38. Tell Stories
    39. Tell Stories SKU: #39072-2AH1
    40. Buy Candy for Strangers
    41. 4. Perspective
    42. Maps help us make sense by seeing things in new ways
    43. Rohrer/Mulder/Yaar’s map Qualitative (direct) Quantitative (indirect) Attitudinal Behavioral © 2008 Christian Rohrer Approach DataSource mix mix Scripted (often lab-based) use of product Natural use of product De-contextualized / not using product Key for Context of Product Use during data collection Combination / hybrid Focus Groups Phone Interviews Ethnographic Field Studies Cardsorting Diary/Camera Study Intercept Surveys Usability Lab Studies Eyetracking Usability Benchmarking (in lab) A/B (Live) Testing Online User Experience Assessments (“Vividence-like” studies) Desirability studies Data Mining/Analysis Email Surveys Message Board Mining Participatory Design Customer feedback via email / / 20 Landscape of User Research Methods  Text Christian Rohrer: http://bit.ly/eAlbe2 / Steve Mulder & Ziv Yaar, The User Is Always Right
    44. Avinash Kaushik’s visualization (from Web Analytics 2.0)
    45. Avinash Kaushik’s visualization (from Web Analytics 2.0)
    46. Avinash Kaushik’s visualization (from Web Analytics 2.0) “...while I have a bucket for ‘Voice of Customer,’ in hindsight I should have worked harder still to paint the full qual and quant picture....”
    47. Containers help us make sense by doing things in new ways
    48. MailChimp’s UX team: drowning in data • Analytics • Account closing surveys • Blog comments • Competitor news • Delivery stats • Industry research • Release notes • Support data
    49. MailChimp + Evernote • Shared bucket of buckets (60 notebooks) • Email is the API • OCR’d (nice for SurveyMonkey reports) • Searchable! • Led to “regular data nerd lunches” MailChimp is on the threshold of synthesis
    50. Map + Container = Dashboard
    51. Map + Container = Dashboard?
    52. A very helpful book Helps decision-makers understand that silos are your problem— and theirs too
    53. A parting question If you were going to build your organization’s brain— its decision-making capability —from scratch... What would it look like?
    54. Thanks! slides: http://rfld.me/11FrI3o article: http://rfld.me/145ZccP Lou Rosenfeld Rosenfeld Media  www.louisrosenfeld.com • @louisrosenfeld www.rosenfeldmedia.com • @rosenfeldmedia

    ×