Rethinking User Research forthe Social WebDana Chisnelldana@usabilityworks.net   @danachisIA Summit 2011            #rethi...
Shelly         2
The state of theart is ageneration oldTime for a new generation ofuser research methods.                               3
Usability testing isn’t telling us what weneed to know for SxD                                   Me, Inc.                 ...
We don’t know how to find out about thingswe don’t know about                                             5
People don’t live in the world doing one taskwith one device out of context                        WebEx - meetings       ...
7
H+C=I        7
7
H <== (computer) ==> H                         7
8
Methods and measures were probablyalways imperfect.                                     8
Methods and measures were probablyalways imperfect.But they used to be enough.                                     8
Methods and measures were probablyalways imperfect.But they used to be enough.No more.                                     8
Everything isdifferent now.But not different.All this has happened before.                                9
The nature ofbeing online issocialIt always has been.                      email                      calendaring         ...
Scale is a gamechangerCaution: large sample sizes                     20,000 googlers didn’t                     expose re...
Tasks aren’twhat you thinkActivities = goals that emerge andchange                              Effectiveness             ...
Satisfaction iscorrelated withtask completionInstead: control, engagement   But how do you measure                        ...
Users                     How do you design effective tasks in                          this case?                        ...
And we have to becareful not to violateeither of those things:Etsy CirclesFacebook engagementring story                   ...
Social is about context.                           And we have to be                           careful not to violate     ...
Social is about context.And relationships.                           And we have to be                           careful n...
Methods are not robust for understandingcontext and relationships.                                           16
Cultivate polymaths Decision sciences       Communication  Anthropology                         sciences  Psychologies    ...
The most successful userresearch methods for SxDcombine field and testingtechniques                            18
19
Methods          19
MethodsReviewing online profiles, connections in groups ininterviews                                                      19
MethodsReviewing online profiles, connections in groups ininterviewsStories of how you met                                ...
MethodsReviewing online profiles, connections in groups ininterviewsStories of how you metMulti-user sessions with people ...
MethodsReviewing online profiles, connections in groups ininterviewsStories of how you metMulti-user sessions with people ...
MethodsReviewing online profiles, connections in groups ininterviewsStories of how you metMulti-user sessions with people ...
Takes more timerecruiting, prep, session, analysis                                  20
Takes strong research design  rather than templated, commodity usability testing                                          ...
Takes deep thinkingabout questions to answer, and what the evidence will be that                                          ...
Takes study of cohesivenesscollaboration, connections, networks, relationships                                  Strong and...
Where are you with planning?    what we asked answered a business question                                             24
Who do you ask when you don’t know?How do you make financial decisions was what we wanted to           know - which is abo...
Would have told us Who is part of planning and deciding Who is trusted Why these people are important Why they are trusted...
To build a model that supports                             27
Rethinking userresearch                  28
Rethinking userresearchWe’re not getting the answers we need.                                         28
Rethinking userresearchWe’re not getting the answers we need.Experimenting is limited because we’repressured to go to mark...
Rethinking userresearchWe’re not getting the answers we need.Experimenting is limited because we’repressured to go to mark...
We’re missing things we don’t know about.                                            29
It’s time to rethink user research forthe new (old)   social web                  30
Dana Chisnelldana@usabilityworks.nettwitter: @danachiswww.usabilityworks.com415.519.1148                          31
Upcoming SlideShare
Loading in...5
×

Rethinking user research for social design

4,185

Published on

This is a talk I gave at IA Summit 11 about how the methods we've been using for a generation to evaluate UIs are imperfect for social interfaces.

Published in: Design
0 Comments
17 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,185
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
69
Comments
0
Likes
17
Embeds 0
No embeds

No notes for slide
  • \n
  • \nThe story of the session. \nEnding: she and her father would have done this together. \n\nThis tool is a piece of the social web \n\nNothing unusual about this test. \none person, one computer\npretend to do a task \n\n
  • In use at least 20 years \n\nMethod failed Shelly desperately \n\nStarted thinking about this with Buzz\nFTC\n\n
  • \nBuzz testing \n\nAfter I blogged about it... \n\n
  • We&amp;#x2019;re not sure how to do user research that will tell us what we need to know about relationships online\n Something&amp;#x2019;s not working, and I think it&amp;#x2019;s this: \n
  • People simply do operate in real life the way we ask them to behave in usability tests. \n\nWebEx - observe meetings? No \n\nMarriott.com - take into account the others who decide what hotel? No \n\nRetirement planning tool - husbands + household expenses; wives + children about healthcare decisions \n\nEgypt and Libya, using eHarmony and Facebook to pass coded messages: Who knew a social network would *ever* be used that way? \n
  • \n(talk) \n
  • \n(talk) \n
  • \n(talk) \n
  • The methods were probably *always* less than perfect. \n Considering SxD magnifies the imperfections of user research methods\n \n
  • The methods were probably *always* less than perfect. \n Considering SxD magnifies the imperfections of user research methods\n \n
  • The methods were probably *always* less than perfect. \n Considering SxD magnifies the imperfections of user research methods\n \n
  • \nSeems that with ambient technology, everything has changed. \n\nBut it hasn&amp;#x2019;t. \n\nThere are 5 factors to user research that I&amp;#x2019;m convinced we should rework, rethink to avoid things like the Buzz and Etsy privacy violations. \n
  • \nAnything one person does to change the behavior of another is social. \n\nName one thing you can do online that isn&amp;#x2019;t social. \n
  • \n From one person + one computer... \n Large sample sizes won&amp;#x2019;t tell you everything you need to know. The Google 20,000 didn&amp;#x2019;t reveal the privacy problem. Why not? \n Looking at scale another way, let&amp;#x2019;s think about Quora and it&amp;#x2019;s massively user-generated IA. How do you design for that? How do you test for that? How do you do user research that will tell you how to go? \n
  • Usability testing and user research methods use tasks - but users don&amp;#x2019;t come to technology with task goals in mind. \n ephemeral, changeable + context, timing, space, what just happened, what happens next\n \n Tasks are probably activities where goals emerge and change depending on context, timing, space, etc. \n
  • This is probably the wrong measure. \n We should look at control by the user and engagement. People want to be the boss of their data. Buzz and Facebook&amp;#x2019;s Beacon, and now Etsy&amp;#x2019;s changes take that control away. \n We need to take a hard look at conventional metrics. \n What is effectiveness and efficiency with Twitter and Facebook open all day? What&amp;#x2019;s the task? \n Measures of success don&amp;#x2019;t translate to SxD\n time?\n task completion? \n success? \n
  • People use your UI in ways you didn&amp;#x2019;t intend, but that help them do what they want to do.\n Individual workarounds now surface to everyone in a network. \n Etiquette and social norms adjust over time and scale. \n Design is effectively open sourced to the user community\n
  • Onlineness, ambient technology is about context and\n\nwho is zooming whom \n\nWe must not violate the context of the relationships. Otherwise, problems. \n\nClients learn embarrassing purchases - Etsy\nGirlfriend learns source of engagement ring - Facebook\n
  • Onlineness, ambient technology is about context and\n\nwho is zooming whom \n\nWe must not violate the context of the relationships. Otherwise, problems. \n\nClients learn embarrassing purchases - Etsy\nGirlfriend learns source of engagement ring - Facebook\n
  • Something built for HCI doesn&amp;#x2019;t cover the expanded view of this one person interacting online and the implications of that. \n\nLab testing &amp; structured field visits are lacking. \n
  • \n
  • Borrow, borrow from, combine. \n\nI interviewed smart people.... \n\nThis is what they told me: The key is combiningmethods. \n
  • \nMet: Twitter road trip\n
  • \nMet: Twitter road trip\n
  • \nMet: Twitter road trip\n
  • \nMet: Twitter road trip\n
  • \nMet: Twitter road trip\n
  • \nMet: Twitter road trip\n
  • \n
  • \n
  • \n
  • \n
  • And so, we&amp;#x2019;re back in the lab in Waltham. \n\nShelly is crying as she looks out at her future self. \n\nWe asked a business question. \n
  • \nThe business would have been better served by our asking more about Shelly. \n\nWhat&amp;#x2019;s the right question? \n
  • \n
  • \n
  • With our generation old methods, we really only know how to find things we&amp;#x2019;re looking for. \n\n\n
  • With our generation old methods, we really only know how to find things we&amp;#x2019;re looking for. \n\n\n
  • With our generation old methods, we really only know how to find things we&amp;#x2019;re looking for. \n\n\n
  • We don&amp;#x2019;t have good ways of learning about things we don&amp;#x2019;t know about. \n
  • \nTime to rework the old standard methods into new remixes. \n
  • Thank you. \n
  • Rethinking user research for social design

    1. 1. Rethinking User Research forthe Social WebDana Chisnelldana@usabilityworks.net @danachisIA Summit 2011 #rethinking 1
    2. 2. Shelly 2
    3. 3. The state of theart is ageneration oldTime for a new generation ofuser research methods. 3
    4. 4. Usability testing isn’t telling us what weneed to know for SxD Me, Inc. 4
    5. 5. We don’t know how to find out about thingswe don’t know about 5
    6. 6. People don’t live in the world doing one taskwith one device out of context WebEx - meetings Marriott.com - reserving rooms Me, Inc. - retirement planning eHarmony - coded messages in Libya Facebook & Twitter - organizing demonstrations in Egypt 6
    7. 7. 7
    8. 8. H+C=I 7
    9. 9. 7
    10. 10. H <== (computer) ==> H 7
    11. 11. 8
    12. 12. Methods and measures were probablyalways imperfect. 8
    13. 13. Methods and measures were probablyalways imperfect.But they used to be enough. 8
    14. 14. Methods and measures were probablyalways imperfect.But they used to be enough.No more. 8
    15. 15. Everything isdifferent now.But not different.All this has happened before. 9
    16. 16. The nature ofbeing online issocialIt always has been. email calendaring sharing documents --> Name some online transaction you think is not social. http://www.flickr.com/photos/holeymoon/ 10
    17. 17. Scale is a gamechangerCaution: large sample sizes 20,000 googlers didn’t expose real issues. Quora moderating overburdened Google Buzz http://www.flickr.com/photos/daguerreotyped/ 11
    18. 18. Tasks aren’twhat you thinkActivities = goals that emerge andchange Effectiveness Efficiency ... aren’t meaningful http://www.flickr.com/photos/jeremybrooks/ 12
    19. 19. Satisfaction iscorrelated withtask completionInstead: control, engagement But how do you measure control and engagement? What does engagement mean? 13 http://www.flickr.com/photos/litebriteneonstudio/
    20. 20. Users How do you design effective tasks in this case? /cc @continuously product reviews that turn into literary projectsdesign your UI Using comments in Facebook for a conversation Hey, where did everybody go? - user-created help page to describein real time a hack to get around a feature they don’t like Coded messages in Egypt on dating sitesWorkarounds = hacksEtiquette and norms http://www.flickr.com/photos/juleshabib/ 14
    21. 21. And we have to becareful not to violateeither of those things:Etsy CirclesFacebook engagementring story 15
    22. 22. Social is about context. And we have to be careful not to violate either of those things: Etsy Circles Facebook engagement ring story 15
    23. 23. Social is about context.And relationships. And we have to be careful not to violate either of those things: Etsy Circles Facebook engagement ring story 15
    24. 24. Methods are not robust for understandingcontext and relationships. 16
    25. 25. Cultivate polymaths Decision sciences Communication Anthropology sciences Psychologies Sociolinguistics Behavioral economics Organizational behavior Psychobiology Sociology Social neuroscience Social networks 17
    26. 26. The most successful userresearch methods for SxDcombine field and testingtechniques 18
    27. 27. 19
    28. 28. Methods 19
    29. 29. MethodsReviewing online profiles, connections in groups ininterviews 19
    30. 30. MethodsReviewing online profiles, connections in groups ininterviewsStories of how you met 19
    31. 31. MethodsReviewing online profiles, connections in groups ininterviewsStories of how you metMulti-user sessions with people who have strong andloose ties 19
    32. 32. MethodsReviewing online profiles, connections in groups ininterviewsStories of how you metMulti-user sessions with people who have strong andloose tiesVideo diaries with retrospective review 19
    33. 33. MethodsReviewing online profiles, connections in groups ininterviewsStories of how you metMulti-user sessions with people who have strong andloose tiesVideo diaries with retrospective reviewExperience sampling with SMS 19
    34. 34. Takes more timerecruiting, prep, session, analysis 20
    35. 35. Takes strong research design rather than templated, commodity usability testing 21
    36. 36. Takes deep thinkingabout questions to answer, and what the evidence will be that you have answers 22
    37. 37. Takes study of cohesivenesscollaboration, connections, networks, relationships Strong and weak ties Dunbar 23
    38. 38. Where are you with planning? what we asked answered a business question 24
    39. 39. Who do you ask when you don’t know?How do you make financial decisions was what we wanted to know - which is about context and relationships 25
    40. 40. Would have told us Who is part of planning and deciding Who is trusted Why these people are important Why they are trusted How planning happens 26
    41. 41. To build a model that supports 27
    42. 42. Rethinking userresearch 28
    43. 43. Rethinking userresearchWe’re not getting the answers we need. 28
    44. 44. Rethinking userresearchWe’re not getting the answers we need.Experimenting is limited because we’repressured to go to market. 28
    45. 45. Rethinking userresearchWe’re not getting the answers we need.Experimenting is limited because we’repressured to go to market.We’re looking for things we know about. 28
    46. 46. We’re missing things we don’t know about. 29
    47. 47. It’s time to rethink user research forthe new (old) social web 30
    48. 48. Dana Chisnelldana@usabilityworks.nettwitter: @danachiswww.usabilityworks.com415.519.1148 31
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×