Your SlideShare is downloading. ×
0
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Seeing the Elephant: Defragmenting User Research
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Seeing the Elephant: Defragmenting User Research

48,053

Published on

Presented at UXPA Boston May 2014, Interaction S.A. (Recife, Brasil) November 2013 and Intuit (Mountain View, CA, USA) October 2013; earlier version given in 2013 in NYC at Designers + Geeks. Given in …

Presented at UXPA Boston May 2014, Interaction S.A. (Recife, Brasil) November 2013 and Intuit (Mountain View, CA, USA) October 2013; earlier version given in 2013 in NYC at Designers + Geeks. Given in 2012 at UX Russia (http://uxrussia.com/), UX Hong Kong (http://www.uxhongkong.com/) and WebVisions NYC (http://www.webvisionsevent.com/new-york/). Given in 2011 at the IA Summit (http://2011.iasummit.org/), UX-Lisbon (http://ux-lx.com), and Love at First Website (http://www.isitedesign.com/love/).

This is something of a successor to my talk "Marrying Web Analytics and User Experience" (http://is.gd/vK34zS)

Published in: Design, Business, Technology
14 Comments
182 Likes
Statistics
Notes
No Downloads
Views
Total Views
48,053
On Slideshare
0
From Embeds
0
Number of Embeds
24
Actions
Shares
0
Downloads
968
Comments
14
Likes
182
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Image from http://assets.mytopfbcover.com/2012/11/08/3814/128730/funny-face-of-the-elephant_facebook_timeline_cover.jpg
  • http://www.youtube.com/watch?v=XRs7BJP6Ky4&feature=youtu.be
  • http://www.city-data.com/forum/members/johndbaumgardner-637750-albums-beautiful-cleveland-ohio-pic38486-standing-base-beautiful-rather-imposing-keycorp.jpg
  • http://dell.com
  • http://lukewalsh.co.uk/blog/uploaded_images/call-center-738087.jpg
  • http://www.cpasitesolutions.com/youget/cpa-website-marketing/google-analytics-for-accountants.php
  • http://community.acstechnologies.com/wp-content/uploads/2010/08/megaphone-stickman.jpg
  • http://www.crm-reviews.com/vendor-review/salesforce-com-crm-review/
  • http://www.research.ibm.com/images/about/labs/wat_outside.jpg
  • http://www.boxesandarrows.com/files/banda/where-is-your-mental/indiyoung.mentalmodel.large.png
  • http://www.blackcoffee.com/blog/wp-content/uploads/2009/10/brand-architecture.jpg
  • http://www.renps.com/images/NPSpic.png
  • http://pisspoordesign.wordpress.com/page/2/
    “Piss poor design”
  • http://1.bp.blogspot.com/-3buwQCBdO8E/Tkis7YLgTBI/AAAAAAAAAUg/qV_44w9gKUw/s1600/Blind+men+and+elephant.jpg
    Vive l’difference! (differences are a source of strength--if recognized/exploited)
  • http://blog.ideaworks.com/wp-content/uploads/2010/06/Quantitative-vs.-Qualitative1.jpg
  • http://www.writeforhr.com/wp-content/uploads/2010/04/Key-Performance-Indicators.jpg
    http://graffletopia.com/stencils/644
  • http://www.planetperplex.com/en/item/the-mysterious-island/
    TWISTY COURSE OF STARTUPS (AND THEIR ABILITY TO PIVOT ON DATA) SHOWS THE WORLD WE DON’T KNOW
  • http://www.quantshare.com/Images/tutorials/tutorial_statistical_data_analysis_1.gif
    http://www.thetechherald.com/media/images/200819/PostIt_16.jpg
  • http://www.pentagonpost.com/wp-content/uploads/2013/10/balanced_diet.jpg
    largely a diagnostic process to help us determine:* we don’t know what we don’t know (helps w/diagnostics)* we don’t know when to use what
  • http://www.xdstrategy.com/blog/
  • Can a persona• Be data-enriched?• Borrow from analytics segments?• ...and vice versa?
    Adapted from an Adaptive Path persona
  • http://2.bp.blogspot.com/-6jg9sv4lX8o/T05IsomD01I/AAAAAAAAB1g/UrdljvHNS5Y/s1600/music-clipart4.jpg
    understand/make sense of research in time (as opposed to balance, which maps it in space)
  • http://3.bp.blogspot.com/-pdziO1-SUQ0/TiylnsXmwcI/AAAAAAAAAnU/z_4Ctf9YK-4/s1600/128787933020784313.jpg
  • Dave Gray’s article/diagram: http://www.gogamestorm.com/?p=58
  • http://www.politicususa.com/wp-content/uploads/Angry-Palin1-300x225.jpg
  • http://atomic-candy.com/wp-content/uploads/2012/11/candy.jpg
  • moving from maps to containers--from seeing to doing
    http://julieanimation.blogspot.ca/2010/12/3-point-perspective-4-point-perspective.html
  • http://marion.sanap.org.za/MapPrinceEdward3d.jpg
  • http://www.xdstrategy.com/blog/
  • http://www.gatekeeperusainc.com/
  • http://www.inetsoft.com/images/screenshots/an_executive_dashboard.png
    ...but beware dashboards; the metaphor will only take you so far.
  • Wikipedia image from http://en.wikipedia.org/wiki/File:Viegas-UserActivityonWikipedia.gif
  • http://2.bp.blogspot.com/_wb8bAl1P-N0/TOFKkD6iQII/AAAAAAAARyQ/mmGzMbjvVrk/s1600/blue-sky.jpeg
    IT’S NO ONE’S FAULT THAT IT ENDED UP THIS WAY...
  • http://4.bp.blogspot.com/_cAFRZohUKig/TJd3XS2t-pI/AAAAAAAAAMQ/kWwHpDic1K4/s1600/600px-ConferenceBike.jpg
  • Transcript

    • 1. Seeing the Elephant Defragmenting User Research Lou Rosenfeld •  lou@rosenfeldmedia.com UXPA Boston • May 15, 2014
    • 2. November 14, 2013: User Researcher-in-Chief Barack Obama
    • 3. November 14, 2013: User Researcher-in-Chief Barack Obama
    • 4. What does victory look like?
    • 5. User research in today’s organization
    • 6. Reports from the user research group
    • 7. Query data gleaned from site search team XXX.XXX.X.104 - - [10/Jul/2013:10:25:46 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date %3AD%3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www& oe=UTF-8&proxystylesheet=www&q=lincense +plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02 XXX.XXX.X.104 - - [10/Jul/2013:10:25:48 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date %3AD%3AL%3Ad1&ie=UTF-8&client=www&q=license +plate
    • 8. Logs from the call center
    • 9. Reports from analytics applications
    • 10. Insights from Voice of the Customer research
    • 11. Reports from CRM applications
    • 12. Papers from the research center
    • 13. One agency’s user mental model
    • 14. Another agency’s brand architecture research
    • 15. Surveys behind Net Promoter Score
    • 16. So why does so much design still SUCK?
    • 17. The blind men and the elephant
    • 18. What Why
    • 19. Methods employed: quantitative versus qualitative
    • 20. Goals: help org or users Organizational goals Users’ goals
    • 21. How they use data: measur world we know versus wo we don’t Measuring the world we know Exploring the world we don’t
    • 22. Kind of data they use: statistical vs. descriptive Descriptive data Statistical data
    • 23. Lou’s TABLE OF OVERGENERALIZED DICHOTOMIES Web Analytics User Experience What they analyze Users' behaviors (what's happening) Users' intentions and motives (why those things happen) What methods they employ Quantitative methods to determine what's happening Qualitative methods for explaining why things happen What they're trying to achieve Helps the organization meet goals (expressed as KPI) Helps users achieve goals (expressed as tasks or topics of interest) How they use data Measure performance (goal- driven analysis) Uncover patterns and surprises (emergent analysis) What kind of data they use Statistical data ("real" data in large volumes, full of errors) Descriptive data (in small volumes, generated in lab environment, full of errors)
    • 24. Four themes for getting to synthesis 1. Balance 2. Cadence 3. Conversation 4. Perspective
    • 25. 1. Balance
    • 26. Rohrer/Mulder/Yaar’s map Qualitative (direct) Quantitative (indirect) Attitudinal Behavioral © 2008 Christian Rohrer Approach DataSource mix mix Scripted (often lab-based) use of product Natural use of product De-contextualized / not using product Key for Context of Product Use during data collection Combination / hybrid Focus Groups Phone Interviews Ethnographic Field Studies Cardsorting Diary/Camera Study Intercept Surveys Usability Lab Studies Eyetracking Usability Benchmarking (in lab) A/B (Live) Testing Online User Experience Assessments (“Vividence-like” studies) Desirability studies Data Mining/Analysis Email Surveys Message Board Mining Participatory Design Customer feedback via email / / 20 Landscape of User Research Methods  Text Christian Rohrer: http://bit.ly/eAlbe2 / Steve Mulder & Ziv Yaar, The User Is Always Right
    • 27. XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET / search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD %3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&prox ystylesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/ 1.1" 200 971 0 0.02 XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET / search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD %3AL%3Ad1&ie=UTF-8&client=www&q=license+plate &ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&i p=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16 Web analytics’ question: “Are we converting license plate renewals?” UX practitioner’s question: “What are people searching the most?” Balanced analysis
    • 28. Balance within a method
    • 29. Balance within a method
    • 30. 2. Cadence
    • 31. A research cadence from Whitney Quesenbery
    • 32. Cadence WeeklyWeeklyWeekly Call center data trend analysis 2 – 4 hours behavioral/quantitative Task analysis 4 – 6 hours behavioral/quantitative QuarterlyQuarterlyQuarterly Exploratory analysis of site analytics data 8 – 10 hours behavioral/qualitative User survey 16 – 24 hours attitudinal/quantitative AnnuallyAnnuallyAnnually Net Promoter Score study 3 – 4 days attitudinal/quantitative Field study 4 – 5 days behavioral/qualitative
    • 33. Cadence WeeklyWeeklyWeekly Call center data trend analysis 2 – 4 hours behavioral/quantitative Task analysis 4 – 6 hours behavioral/quantitative QuarterlyQuarterlyQuarterly Exploratory analysis of site analytics data 8 – 10 hours behavioral/qualitative User survey 16 – 24 hours attitudinal/quantitative AnnuallyAnnuallyAnnually Net Promoter Score study 3 – 4 days attitudinal/quantitative Field study 4 – 5 days behavioral/qualitative Cadence + Balance
    • 34. Cadence WeeklyWeeklyWeekly Call center data trend analysis 2 – 4 hours behavioral/quantitative Task analysis 4 – 6 hours behavioral/quantitative QuarterlyQuarterlyQuarterly Exploratory analysis of site analytics data 8 – 10 hours behavioral/qualitative User survey 16 – 24 hours attitudinal/quantitative AnnuallyAnnuallyAnnually Net Promoter Score study 3 – 4 days attitudinal/quantitative Field study 4 – 5 days behavioral/qualitative Cadence + Balance
    • 35. 3. Conversation
    • 36. Develop a pidgin Dave Gray’s boundary matrix: http://bit.ly/gWoZQm KPI goals segments personas
    • 37. Ban words that impede conversations • Product names: Omniture,, SharePoint... • Methods: focus group,, usability test... • Departments: market research,, analytics... • Disciplines: business analysis,, information architecture... • Outcomes: portal, social media layer...
    • 38. Tell Stories
    • 39. Tell Stories SKU: #39072-2AH1
    • 40. Buy Candy for Strangers
    • 41. 4. Perspective
    • 42. Maps help us make sense by seeing things in new ways
    • 43. Rohrer/Mulder/Yaar’s map Qualitative (direct) Quantitative (indirect) Attitudinal Behavioral © 2008 Christian Rohrer Approach DataSource mix mix Scripted (often lab-based) use of product Natural use of product De-contextualized / not using product Key for Context of Product Use during data collection Combination / hybrid Focus Groups Phone Interviews Ethnographic Field Studies Cardsorting Diary/Camera Study Intercept Surveys Usability Lab Studies Eyetracking Usability Benchmarking (in lab) A/B (Live) Testing Online User Experience Assessments (“Vividence-like” studies) Desirability studies Data Mining/Analysis Email Surveys Message Board Mining Participatory Design Customer feedback via email / / 20 Landscape of User Research Methods  Text Christian Rohrer: http://bit.ly/eAlbe2 / Steve Mulder & Ziv Yaar, The User Is Always Right
    • 44. Avinash Kaushik’s visualization (from Web Analytics 2.0)
    • 45. Avinash Kaushik’s visualization (from Web Analytics 2.0)
    • 46. Avinash Kaushik’s visualization (from Web Analytics 2.0) “...while I have a bucket for ‘Voice of Customer,’ in hindsight I should have worked harder still to paint the full qual and quant picture....”
    • 47. Containers help us make sense by doing things in new ways
    • 48. MailChimp’s UX team: drowning in data • Analytics • Account closing surveys • Blog comments • Competitor news • Delivery stats • Industry research • Release notes • Support data
    • 49. MailChimp + Evernote • Shared bucket of buckets (60 notebooks) • Email is the API • OCR’d (nice for SurveyMonkey reports) • Searchable! • Led to “regular data nerd lunches” MailChimp is on the threshold of synthesis
    • 50. Map + Container = Dashboard
    • 51. Map + Container = Dashboard?
    • 52. A very helpful book Helps decision-makers understand that silos are your problem— and theirs too
    • 53. A parting question If you were going to build your organization’s brain— its decision-making capability —from scratch... What would it look like?
    • 54. Thanks! slides: http://rfld.me/11FrI3o article: http://rfld.me/145ZccP Lou Rosenfeld Rosenfeld Media  www.louisrosenfeld.com • @louisrosenfeld www.rosenfeldmedia.com • @rosenfeldmedia

    ×