Seeing the Elephant: Defragmenting User Research
Upcoming SlideShare
Loading in...5
×
 

Seeing the Elephant: Defragmenting User Research

on

  • 44,945 views

Presented at UXPA Boston May 2014, Interaction S.A. (Recife, Brasil) November 2013 and Intuit (Mountain View, CA, USA) October 2013; earlier version given in 2013 in NYC at Designers + Geeks. Given in ...

Presented at UXPA Boston May 2014, Interaction S.A. (Recife, Brasil) November 2013 and Intuit (Mountain View, CA, USA) October 2013; earlier version given in 2013 in NYC at Designers + Geeks. Given in 2012 at UX Russia (http://uxrussia.com/), UX Hong Kong (http://www.uxhongkong.com/) and WebVisions NYC (http://www.webvisionsevent.com/new-york/). Given in 2011 at the IA Summit (http://2011.iasummit.org/), UX-Lisbon (http://ux-lx.com), and Love at First Website (http://www.isitedesign.com/love/).

This is something of a successor to my talk "Marrying Web Analytics and User Experience" (http://is.gd/vK34zS)

Statistics

Views

Total Views
44,945
Views on SlideShare
38,568
Embed Views
6,377

Actions

Likes
166
Downloads
901
Comments
14

50 Embeds 6,377

http://louisrosenfeld.com 3021
http://ixdabh.org 1410
http://www.scoop.it 791
http://www.uxd.cl 393
http://catarinasdesign.com.br 199
http://www.louisrosenfeld.com 120
http://paper.li 89
https://twitter.com 64
http://humansindesign.com 45
http://sprmario.tumblr.com 26
http://www.borjasantaolalla.com 25
http://sprmario.posterous.com 22
http://library.iasummit.org 19
http://flavors.me 16
http://lanyrd.com 16
http://translate.googleusercontent.com 15
http://lieveld.nl 14
http://jbaeck.posterous.com 9
http://humansindesign.tumblr.com 9
http://a0.twimg.com 8
http://feeds.feedburner.com 6
http://www.slideshare.net 5
http://twitter.com 4
http://www.stereoprototype.com 4
https://si0.twimg.com 3
http://feedly.com 3
https://reader.aol.com 3
http://xss.yandex.net 3
http://us-w1.rockmelt.com 3
http://www.twylah.com 3
http://webcache.googleusercontent.com 3
url_unknown 3
http://www.tumblr.com 2
https://internal.autodesk360beta.com 2
http://www.ofertadeviaje.org 2
http://www.linkedin.com 2
http://www.pinterest.com 2
http://www.mongodb.org 1
http://www.hanrss.com 1
http://digg.com 1
http://tweetedtimes.com 1
http://www.instacurate.com 1
https://www.google.co.uk 1
https://mercy.blackboard.com 1
http://www.ofertadeviajes.eu 1
https://bb-mercy.blackboard.com 1
http://fanyi.youdao.com 1
http://www.keytobiz.com 1
http://dashboard.bloglines.com 1
http://fabricioteixeira.wordpress.com 1
More...

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

110 of 14 Post a comment

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • @Mary Ann, that's so nice of you; thanks much for the link!
    Are you sure you want to
    Your message goes here
    Processing…
  • Hi Louis, I saw your presentation in Boston and really learned a lot about integration in UX research that I was able to apply directly to my work. I particularly liked the clip of Obama's nod to a UX team working on the FAFSA. I searched high and low for a good clip to include in one of my own presentations and I found it in very good quality. I figured I would share in case it's helpful to you. Clip is around the 36 min mark. http://www.c-span.org/video/?316233-1/pres-obama-announces-health-care-policy-changes
    Are you sure you want to
    Your message goes here
    Processing…
  • Nice presentation
    Are you sure you want to
    Your message goes here
    Processing…
  • @SeanVT Thanks man! Definitely enjoyed putting this together (over, frankly, about three years!).
    Are you sure you want to
    Your message goes here
    Processing…
  • Frickin' Awesome, Lou! The Holy Grail of Analytics and UX
    Are you sure you want to
    Your message goes here
    Processing…

110 of 14

Post Comment
Edit your comment
  • Image from http://assets.mytopfbcover.com/2012/11/08/3814/128730/funny-face-of-the-elephant_facebook_timeline_cover.jpg <br />
  • http://www.youtube.com/watch?v=XRs7BJP6Ky4&feature=youtu.be <br />
  • http://www.city-data.com/forum/members/johndbaumgardner-637750-albums-beautiful-cleveland-ohio-pic38486-standing-base-beautiful-rather-imposing-keycorp.jpg <br />
  • http://dell.com <br />
  • http://lukewalsh.co.uk/blog/uploaded_images/call-center-738087.jpg <br />
  • http://www.cpasitesolutions.com/youget/cpa-website-marketing/google-analytics-for-accountants.php <br />
  • http://community.acstechnologies.com/wp-content/uploads/2010/08/megaphone-stickman.jpg <br />
  • http://www.crm-reviews.com/vendor-review/salesforce-com-crm-review/ <br />
  • http://www.research.ibm.com/images/about/labs/wat_outside.jpg <br />
  • http://www.boxesandarrows.com/files/banda/where-is-your-mental/indiyoung.mentalmodel.large.png <br />
  • http://www.blackcoffee.com/blog/wp-content/uploads/2009/10/brand-architecture.jpg <br />
  • http://www.renps.com/images/NPSpic.png <br />
  • http://pisspoordesign.wordpress.com/page/2/ <br /> “Piss poor design” <br />
  • http://1.bp.blogspot.com/-3buwQCBdO8E/Tkis7YLgTBI/AAAAAAAAAUg/qV_44w9gKUw/s1600/Blind+men+and+elephant.jpg <br /> Vive l’difference! (differences are a source of strength--if recognized/exploited) <br />
  • http://blog.ideaworks.com/wp-content/uploads/2010/06/Quantitative-vs.-Qualitative1.jpg <br />
  • http://www.writeforhr.com/wp-content/uploads/2010/04/Key-Performance-Indicators.jpg <br /> http://graffletopia.com/stencils/644 <br />
  • http://www.planetperplex.com/en/item/the-mysterious-island/ <br /> TWISTY COURSE OF STARTUPS (AND THEIR ABILITY TO PIVOT ON DATA) SHOWS THE WORLD WE DON’T KNOW <br />
  • http://www.quantshare.com/Images/tutorials/tutorial_statistical_data_analysis_1.gif <br /> http://www.thetechherald.com/media/images/200819/PostIt_16.jpg <br />
  • http://www.pentagonpost.com/wp-content/uploads/2013/10/balanced_diet.jpg <br /> largely a diagnostic process to help us determine:* we don’t know what we don’t know (helps w/diagnostics)* we don’t know when to use what <br />
  • http://www.xdstrategy.com/blog/ <br />
  • Can a persona• Be data-enriched?• Borrow from analytics segments?• ...and vice versa? <br /> Adapted from an Adaptive Path persona <br />
  • http://2.bp.blogspot.com/-6jg9sv4lX8o/T05IsomD01I/AAAAAAAAB1g/UrdljvHNS5Y/s1600/music-clipart4.jpg <br /> understand/make sense of research in time (as opposed to balance, which maps it in space) <br />
  • http://3.bp.blogspot.com/-pdziO1-SUQ0/TiylnsXmwcI/AAAAAAAAAnU/z_4Ctf9YK-4/s1600/128787933020784313.jpg <br />
  • Dave Gray’s article/diagram: http://www.gogamestorm.com/?p=58 <br />
  • http://www.politicususa.com/wp-content/uploads/Angry-Palin1-300x225.jpg <br />
  • http://atomic-candy.com/wp-content/uploads/2012/11/candy.jpg <br />
  • moving from maps to containers--from seeing to doing <br /> http://julieanimation.blogspot.ca/2010/12/3-point-perspective-4-point-perspective.html <br />
  • http://marion.sanap.org.za/MapPrinceEdward3d.jpg <br />
  • http://www.xdstrategy.com/blog/ <br />
  • http://www.gatekeeperusainc.com/ <br />
  • http://www.inetsoft.com/images/screenshots/an_executive_dashboard.png <br /> ...but beware dashboards; the metaphor will only take you so far. <br />
  • Wikipedia image from http://en.wikipedia.org/wiki/File:Viegas-UserActivityonWikipedia.gif <br />
  • http://2.bp.blogspot.com/_wb8bAl1P-N0/TOFKkD6iQII/AAAAAAAARyQ/mmGzMbjvVrk/s1600/blue-sky.jpeg <br /> IT’S NO ONE’S FAULT THAT IT ENDED UP THIS WAY... <br />
  • http://4.bp.blogspot.com/_cAFRZohUKig/TJd3XS2t-pI/AAAAAAAAAMQ/kWwHpDic1K4/s1600/600px-ConferenceBike.jpg <br />

Seeing the Elephant: Defragmenting User Research Seeing the Elephant: Defragmenting User Research Presentation Transcript

  • Seeing the Elephant Defragmenting User Research Lou Rosenfeld •  lou@rosenfeldmedia.com UXPA Boston • May 15, 2014
  • November 14, 2013: User Researcher-in-Chief Barack Obama
  • November 14, 2013: User Researcher-in-Chief Barack Obama
  • What does victory look like?
  • User research in today’s organization
  • Reports from the user research group
  • Query data gleaned from site search team XXX.XXX.X.104 - - [10/Jul/2013:10:25:46 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date %3AD%3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www& oe=UTF-8&proxystylesheet=www&q=lincense +plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02 XXX.XXX.X.104 - - [10/Jul/2013:10:25:48 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date %3AD%3AL%3Ad1&ie=UTF-8&client=www&q=license +plate
  • Logs from the call center
  • Reports from analytics applications
  • Insights from Voice of the Customer research
  • Reports from CRM applications
  • Papers from the research center
  • One agency’s user mental model
  • Another agency’s brand architecture research
  • Surveys behind Net Promoter Score
  • So why does so much design still SUCK?
  • The blind men and the elephant
  • What Why
  • Methods employed: quantitative versus qualitative
  • Goals: help org or users Organizational goals Users’ goals
  • How they use data: measur world we know versus wo we don’t Measuring the world we know Exploring the world we don’t
  • Kind of data they use: statistical vs. descriptive Descriptive data Statistical data
  • Lou’s TABLE OF OVERGENERALIZED DICHOTOMIES Web Analytics User Experience What they analyze Users' behaviors (what's happening) Users' intentions and motives (why those things happen) What methods they employ Quantitative methods to determine what's happening Qualitative methods for explaining why things happen What they're trying to achieve Helps the organization meet goals (expressed as KPI) Helps users achieve goals (expressed as tasks or topics of interest) How they use data Measure performance (goal- driven analysis) Uncover patterns and surprises (emergent analysis) What kind of data they use Statistical data ("real" data in large volumes, full of errors) Descriptive data (in small volumes, generated in lab environment, full of errors)
  • Four themes for getting to synthesis 1. Balance 2. Cadence 3. Conversation 4. Perspective
  • 1. Balance
  • Rohrer/Mulder/Yaar’s map Qualitative (direct) Quantitative (indirect) Attitudinal Behavioral © 2008 Christian Rohrer Approach DataSource mix mix Scripted (often lab-based) use of product Natural use of product De-contextualized / not using product Key for Context of Product Use during data collection Combination / hybrid Focus Groups Phone Interviews Ethnographic Field Studies Cardsorting Diary/Camera Study Intercept Surveys Usability Lab Studies Eyetracking Usability Benchmarking (in lab) A/B (Live) Testing Online User Experience Assessments (“Vividence-like” studies) Desirability studies Data Mining/Analysis Email Surveys Message Board Mining Participatory Design Customer feedback via email / / 20 Landscape of User Research Methods  Text Christian Rohrer: http://bit.ly/eAlbe2 / Steve Mulder & Ziv Yaar, The User Is Always Right
  • XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET / search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD %3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&prox ystylesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/ 1.1" 200 971 0 0.02 XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET / search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD %3AL%3Ad1&ie=UTF-8&client=www&q=license+plate &ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&i p=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16 Web analytics’ question: “Are we converting license plate renewals?” UX practitioner’s question: “What are people searching the most?” Balanced analysis
  • Balance within a method
  • Balance within a method
  • 2. Cadence
  • A research cadence from Whitney Quesenbery
  • Cadence WeeklyWeeklyWeekly Call center data trend analysis 2 – 4 hours behavioral/quantitative Task analysis 4 – 6 hours behavioral/quantitative QuarterlyQuarterlyQuarterly Exploratory analysis of site analytics data 8 – 10 hours behavioral/qualitative User survey 16 – 24 hours attitudinal/quantitative AnnuallyAnnuallyAnnually Net Promoter Score study 3 – 4 days attitudinal/quantitative Field study 4 – 5 days behavioral/qualitative
  • Cadence WeeklyWeeklyWeekly Call center data trend analysis 2 – 4 hours behavioral/quantitative Task analysis 4 – 6 hours behavioral/quantitative QuarterlyQuarterlyQuarterly Exploratory analysis of site analytics data 8 – 10 hours behavioral/qualitative User survey 16 – 24 hours attitudinal/quantitative AnnuallyAnnuallyAnnually Net Promoter Score study 3 – 4 days attitudinal/quantitative Field study 4 – 5 days behavioral/qualitative Cadence + Balance
  • Cadence WeeklyWeeklyWeekly Call center data trend analysis 2 – 4 hours behavioral/quantitative Task analysis 4 – 6 hours behavioral/quantitative QuarterlyQuarterlyQuarterly Exploratory analysis of site analytics data 8 – 10 hours behavioral/qualitative User survey 16 – 24 hours attitudinal/quantitative AnnuallyAnnuallyAnnually Net Promoter Score study 3 – 4 days attitudinal/quantitative Field study 4 – 5 days behavioral/qualitative Cadence + Balance
  • 3. Conversation
  • Develop a pidgin Dave Gray’s boundary matrix: http://bit.ly/gWoZQm KPI goals segments personas
  • Ban words that impede conversations • Product names: Omniture,, SharePoint... • Methods: focus group,, usability test... • Departments: market research,, analytics... • Disciplines: business analysis,, information architecture... • Outcomes: portal, social media layer...
  • Tell Stories
  • Tell Stories SKU: #39072-2AH1
  • Buy Candy for Strangers
  • 4. Perspective
  • Maps help us make sense by seeing things in new ways
  • Rohrer/Mulder/Yaar’s map Qualitative (direct) Quantitative (indirect) Attitudinal Behavioral © 2008 Christian Rohrer Approach DataSource mix mix Scripted (often lab-based) use of product Natural use of product De-contextualized / not using product Key for Context of Product Use during data collection Combination / hybrid Focus Groups Phone Interviews Ethnographic Field Studies Cardsorting Diary/Camera Study Intercept Surveys Usability Lab Studies Eyetracking Usability Benchmarking (in lab) A/B (Live) Testing Online User Experience Assessments (“Vividence-like” studies) Desirability studies Data Mining/Analysis Email Surveys Message Board Mining Participatory Design Customer feedback via email / / 20 Landscape of User Research Methods  Text Christian Rohrer: http://bit.ly/eAlbe2 / Steve Mulder & Ziv Yaar, The User Is Always Right
  • Avinash Kaushik’s visualization (from Web Analytics 2.0)
  • Avinash Kaushik’s visualization (from Web Analytics 2.0)
  • Avinash Kaushik’s visualization (from Web Analytics 2.0) “...while I have a bucket for ‘Voice of Customer,’ in hindsight I should have worked harder still to paint the full qual and quant picture....”
  • Containers help us make sense by doing things in new ways
  • MailChimp’s UX team: drowning in data • Analytics • Account closing surveys • Blog comments • Competitor news • Delivery stats • Industry research • Release notes • Support data
  • MailChimp + Evernote • Shared bucket of buckets (60 notebooks) • Email is the API • OCR’d (nice for SurveyMonkey reports) • Searchable! • Led to “regular data nerd lunches” MailChimp is on the threshold of synthesis
  • Map + Container = Dashboard
  • Map + Container = Dashboard?
  • A very helpful book Helps decision-makers understand that silos are your problem— and theirs too
  • A parting question If you were going to build your organization’s brain— its decision-making capability —from scratch... What would it look like?
  • Thanks! slides: http://rfld.me/11FrI3o article: http://rfld.me/145ZccP Lou Rosenfeld Rosenfeld Media  www.louisrosenfeld.com • @louisrosenfeld www.rosenfeldmedia.com • @rosenfeldmedia