UX Sofia 2012 - DIY Mobile Usability Testing Workshop
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

UX Sofia 2012 - DIY Mobile Usability Testing Workshop

on

  • 2,376 views

These are the slides for our 4-hour workshop in UX Sofia 2012.

These are the slides for our 4-hour workshop in UX Sofia 2012.

Statistics

Views

Total Views
2,376
Views on SlideShare
2,161
Embed Views
215

Actions

Likes
4
Downloads
31
Comments
0

1 Embed 215

http://www.diymobileusabilitytesting.net 215

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-ShareAlike LicenseCC Attribution-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

UX Sofia 2012 - DIY Mobile Usability Testing Workshop Presentation Transcript

  • 1. Bernard, packet coreengineer at NSN
  • 2. Belén, interactiondesigner at Intel’s OTC
  • 3. how to plan and run usabilitytests with mobile devices in alaboratory setting
  • 4. assuming ...1. you have experience planning and running usability tests with desktop computers2. you are testing software or an interactive prototype running on a mobile device3. you are testing in a usability laboratory or similar environment
  • 5. we will not focus on1. the fundamentals of the usability testing technique2. how to prototype for mobile devices3. how to test in the field
  • 6. these arethe goldenrules youneed toremember
  • 7. let us begin ...
  • 8. usability testinga process that employs people as testingparticipants who are representative of thetarget audience to evaluate the degree towhich a product meets specific usabilitycriteria.Handbook of usability testing 2nd Ed., J. Rubin and D. Chisnell
  • 9. usability testingwatching people try to use what you’recreating/designing/building (...) with theintention of (a)making it easier for people touse or (b)proving that it is easy to use.Rocket Surgery Made Easy, S. Krug
  • 10. dut vs mut
  • 11. Desktop MobileProduct software software MobileDevice Computer phoneContext Predictable Unpredictable Mobile phoneConnection DSL / Cable network
  • 12. which phone?which context?which connection?
  • 13. which phone?which context?which connection?
  • 14. web task success rates feature phones 38% smartphones 55% touch phones 75%Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html
  • 15. web task success rates feature phones 38% smartphones 55% touch phones 75%Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html
  • 16. web task success rates feature phones 38% smartphones 55% touch phones 75%Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html
  • 17. web task success rates feature phones 38% smartphones 55% touch phones 75%Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html
  • 18. web task success rates feature phones 38% smartphones 55% touch phones 75%Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html
  • 19. web task success rates feature phones 38% smartphones 55% touch phones 75%Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html
  • 20. web task success rates feature phones 38% smartphones 55% touch phones 75%Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html
  • 21. handset usabilityaffects test results
  • 22. test withparticipants’own phones
  • 23. when not possible,include training and warm-up tasks
  • 24. handset becomes ascreening criteria whenrecruiting test subjects
  • 25. http://goo.gl/hmp8t
  • 26. mobile phone subscriptions per 100 peoplehttp://data.worldbank.org
  • 27. http://gs.statcounter.com/
  • 28. http://gs.statcounter.com/
  • 29. WP iOS Android Blackberry
  • 30. http://developer.android.com/resources/dashboard/platform-versions.html
  • 31. http://us.blackberry.com/developers/choosingtargetos.jsp
  • 32. do notconfusetargetplatformwith targetuser
  • 33. which phone would you use for testing?who are the target users?handset screening criteriawill you need warm-up tasks? (yes / no)if yes, which ones?
  • 34. BNT are considering amobile version of theirwebsite.Before investing in theproject, they would like toknow how their existingdesktop site performs onmobile devices.
  • 35. You are working on anAndroid application forSofia Airport.Together with yourdevelopers, you havecreated a first version ofthe app. This ‘alpha’version is not available inGoogle Play, it is not fullytested and might bebuggy.
  • 36. St. Ekaterina UniversityHospital has developed aBlackberry applicationthat helps nurses anddoctors access andupdate patients recordsduring their daily shift.They have asked you tohelp them test theapplication.
  • 37. Evernote wants touncover the main usabilityissues in their existingmobile applications foriOS, Android, WindowsPhone and Blackberry.Help them organise thenecessary testing.
  • 38. what about emulators?mas_fsx_ by Massimiliano Silipigni at http://www.flickr.com/photos/silipigni/6055803785/
  • 39. good for ...using an emulator enables testers tothoroughly capture user behaviour (...) Thecaptured data are generally informative anduseful for analysing user performanceChallenges, Methodologies, and Issues in the Usability Testing of Mobile Applications,D. Zhang, B. Adipat, International Journal of Human Computer Studies (2005)
  • 40. good for ...emulators are more suitable to be used forimproving the interface design of applicationssuch as the layout of menu structures duringthe development process.Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications,D. Zhang, B. Adipat, International Journal of Human Computer Studies (2005)
  • 41. but bad for ...using emulators alleviates the problems oflatency, inefficient input mechanisms, and thechanging wireless environment, potentiallyleading to untruthful user satisfactionChallenges, Methodologies, and Issues in the Usability Testing of Mobile Applications,D. Zhang, B. Adipat, International Journal of Human Computer Studies (2005)
  • 42. which phone?which context?which connection?
  • 43. field vs. labIt’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
  • 44. field vs. lab 0 0It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
  • 45. The results show that the added value ofconducting usability evaluations in the field isvery little and that recreating central aspects ofthe use context in a laboratory setting enablesthe identification of the same usability problemlist.Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-AwareMobile Systems in the FieldJ. Kjeldskov, M. B. Skov, B. S. Als, R. T. Høegh, 2004
  • 46. 0 1The results show that the added value ofconducting usability evaluations in the field isvery little and that recreating central aspects ofthe use context in a laboratory setting enablesthe identification of the same usability problemlist. Field LabIs it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-AwareMobile Systems in the FieldJ. Kjeldskov, M. B. Skov, B. S. Als, R. T. Høegh, 2004
  • 47. according to our study there was no differencein the number of problems that occurred in thetwo test settings. Our hypothesis that moreproblems would be found in the field was notsupportedUsability Testing of Mobile Applications: A Comparison between Laboratory and Field TestingA. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005
  • 48. 0 2according to our study there was no differencein the number of problems that occurred in thetwo test settings. Our hypothesis that moreproblems would be found in the field was notsupported Field LabUsability Testing of Mobile Applications: A Comparison between Laboratory and Field TestingA. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005
  • 49. evaluations conducted in field settings can revealproblems not otherwise identified in laboratoryevaluationsIt’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
  • 50. 1 2evaluations conducted in field settings can revealproblems not otherwise identified in laboratoryevaluationsIt’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006 Field Lab
  • 51. The analyses of the comparison betweenusability testing done in two different settingsrevealed that there were many more types andoccurrences of usability problems found in thefield than in the laboratory. Those problemsdiscovered tend to be critical issues.Usability Evaluation of Mobile Device: a Comparison of Laboratory and Field TestsH.B Duh, G. C. B. Tan, V. H. Chen, MobileHCI 2006
  • 52. 2 2The analyses of the comparison betweenusability testing done in two different settingsrevealed that there were many more types andoccurrences of usability problems found in thefield than in the laboratory. Those problemsdiscovered tend to be critical issues. Field LabUsability Evaluation of Mobile Device: a Comparison of Laboratory and Field TestsH.B Duh, G. C. B. Tan, V. H. Chen, MobileHCI 2006
  • 53. field vs. lab D ISA GR EE EXPE RTSIt’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
  • 54. ... but they all agree testing in the field requires double the time in comparison to the laboratory Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing A. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005
  • 55. ... but they all agree evaluations in the field (are) more complex and time-consuming It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
  • 56. ... but they all agree field-based usability studies are not easy to conduct. They are time consuming and the added value is questionable. Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field, J. Kjeldskov, M. B. Skov, B. S. Als, R. T. Høegh, 2004
  • 57. testing inthe lab isbetter thanno testing
  • 58. can you think of anexample of mobilesoftware that youmust test in the field?
  • 59. if doing field testing ... do it late plan an run pilot tests be prepared (like the Scouts)
  • 60. if doing field testing ... have I chosen an appropriate location? which external factors could impact results? is the equipment working as planned? do I have enough time to run the tests? does the protocol work? Is it well understood?
  • 61. If we cannot go to the field,the field must come to us
  • 62. Bringing the field into the lab Simulations Scenarios Take the lab out
  • 63. simulationsUse of techniques to “reflect or recreate amobile use situation” (...) their level ofsophistication is measured by the extent towhich the environmental and ergonomicfactors of the actual use context arerecreated.Emerging Research Methods for Understanding Mobile Technology Use,P. Hagen, T. Robertson, M. Kan and K. Sadler, Proceedings of OZCHI 2005, Canberra (Australia)
  • 64. Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware MobileSystems in the Field, J. Kjeldskov, M.B. Skov, B.S. Als and R.T.Høegh, MobileHCI 2004
  • 65. Experimental Evaluation of Techniques for Usability Testing of Mobile Systems in a Laboratory Setting, E.T.Beck, M.K. Christiansen, J. Kjedskov, N. Kolbe and J. Stage, OZCHI 2003, Brisbane (Australia)
  • 66. scenariosFirst you choose the tasks to test (...) Thenyou expand these tasks into scenarios (...) Ascenario provides some context andsupplies information the user needs to knowRocket Surgery Made Easy, S. Krug
  • 67. a task:Find a hotela desktop scenario:You are planning a surprise weekend trip toParis. You’ll be leaving on Friday and comingback on Sunday evening. Your partner’sfavourite Paris area is Le Marais. Find a placeto stay he/she will like.
  • 68. a task:Find a hotela mobile scenario:You were late to Orly airport and havemissed your flight home. There are no moreflights from Paris today, so you’ll have tospend the night. Since it’s already 10 pm, useyour phone to look for a place to stay closeto the airport.
  • 69. portable labs can bedeployed anywhere document camera webcam laptop
  • 70. Jyske Bank in Roskilde by king_david_uk at http://www.flickr.com/photos/kingdavid/2926673466/Vodafone, Croydon, London CR0 by Kate Pugh at http://www.flickr.com/photos/kake_pugh/6545344321/
  • 71. write a mobilescenario for your group’s task
  • 72. You are testing:http://www.tfl.gov.ukTask:Find a public transport routeA bit of context:From Heathrow Terminal 5 to the Olympic Stadium.User must arrive on time for the Olympics openingceremony (27th July at 7:30pm)
  • 73. You are testing:http://www.toptable.co.ukTask:Find a restaurantA bit of context:Lebanese cuisineAround Shoreditch (an East London area)For tomorrow at 8:30pmFor 5 people
  • 74. You are testing:An Android app called ‘CycleStreets’, which helps youfind cycling routes across LondonTask:Calculate a route distanceA bit of context:From King’s Cross station to Tate ModernRoute with least traffic
  • 75. You are testing:A task management app called Remember the MilkTask:Create a taskA bit of context:The task is to buy flowers for a friend (make up a name)Next Tuesday morningIn her favourite flower shop, called Blooming
  • 76. which phone?which context?which connection?
  • 77. how often are mobile users connecting their devices to WiFi networks instead of mobile networks?Towering by Rachel Strohm at http://www.flickr.com/photos/rachelstrohm/4693016049/
  • 78. 25% of households worldwide have WiFi networks set up Globally, smartphones account for 36% of all public WiFi hotspot connections, compared to 48% for laptops and 10% for tablets Out of 4 billion unique mobile phone users, 1 billion uses WiFi to access the Internet (there are only about 900 million smartphones)http://www.lukew.com/ff/entry.asp?1537http://communities-dominate.blogs.com/brands/2012/05/mobile-services-beyond-messaging-excellent-tns-global-survey-reveals-tons.html
  • 79. http://databank.worldbank.org
  • 80. do not testover WiFi
  • 81. cover yourparticipants’data costs
  • 82. and on the big day ...
  • 83. remote?Remote Mania by Dramagirl at http://www.flickr.com/photos/dramagirl/2268235917/
  • 84. http://www.usertesting.com/mobile
  • 85. but ...I’d say remote testing gives you about 80% ofthe benefits of a live test with about 70% ofthe effort. You do lose 20% of something notbeing in the same room as the participant.The in-person experience is just richer,somehow.Rocket Surgery Made Easy, S. Krug
  • 86. to record or not to record ...To be or not to be, that is the question by Mister_Jack at http://www.flickr.com/photos/machineman22/5130901135/
  • 87. it is difficult ...We have used Morae in the past also forwebsite testing, but could not easily use thesystem for mobile (...) Obviously it’s difficultto video something like that so we just sitnext to them and observe.http://belenpena.posterous.com/user-research-for-mobile-software-a-short-sur
  • 88. ... but video is both a memory aid powerful communication tool
  • 89. actions reactions
  • 90. reactionsactions
  • 91. which phone?which context?which connection?which recording setup?
  • 92. Mobiola Screen Capture for Blackberry 4.2+ and Symbian S60 v3http://www.shapeservices.com/en/products/details.php?product=capture&platform=none#
  • 93. USB webcam records document camera participant’s face records mobile screen both cameras connect to a document camera range laptop running screen is marked with tape recording software
  • 94. Mobile Device Camera by Noldus Ovo Studios device camerahttp://www.noldus.com/human-behavior-research/ http://www.ovostudios.com/devicecamera.aspaccessories/mobile-device-camera-mdc#
  • 95. Little Springs Designhttp://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/ Google Towards the Perfect Infrastructure for Usability Testing on Mobile Devices, R. Schusteritsch, C.Y. Wei, M. LaRosa - Google (CHI 2007) Usability Sciences http://www.usabilitysciences.com/services/lab- based-usability-testing/mobile-usability-testing Nick Bowmast http://www.bowmast.com/mob-device-cam/
  • 96. are blooming!!Little Springs Design Nick Bowmast http://www.bowmast.com/mob-device-cam/http://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/ Google Towards the Perfect Infrastructure for Usability Testing on Mobile Usability Sciences Devices, R. Schusteritsch, C.Y. Wei, http://www.usabilitysciences.com/services/lab- M. LaRosa - Google (CHI 2007) based-usability-testing/mobile-usability-testing
  • 97. the observation roomscreen audioSkype SkypeScreen sharing Telephone conferencing systemMirroring
  • 98. MirroringScreen sharing Telephone conferencing system
  • 99. screen capture applicationspros cons participants won’t appreciate you installing stuff on their phonesprovides high qualityscreen recording no application will support all platforms in touchscreen phones, finger interactions are not captured
  • 100. Ovo Studios screen capture application for iOShttp://www.ovostudios.com/
  • 101. Obviously, think aloud is critical because I cannot see how the participant is interacting with his fingers on the touch screen. ... and that is a big I can see using the mirroring in a lab setting to get the signal from the iPad to the observation room though you still won’t see the physical interaction deal with the device, like you would with a device mounted camera (e.g. Noldus).Why AirPlay mirroring is the Biggest Thing to Happen to User Research in 2011 - http://www.remoteresear.ch/airplay/
  • 102. Recently, we’ve questioned the value of capturing the device screen. It does after all end up being a video of the screen changing but with no sense of the participant interacting with it. ... and that is a big dealiPad usability testing - our equipment - http://www.cxpartners.co.uk/cxblog/ipad_usability_testing_-_our_equipment/
  • 103. document cameraspros cons document camerasgood recording quality bring up the price participants must keep withineasy to set up the camera range phone must lay on a desk or be hold at a flat angle
  • 104. mounted devicespros cons ready-made devices will bring up the priceallow natural interaction with if bulky, they can preventthe phone single-hand use if heavy, they can become tiring during long tests
  • 105. the ideal recording setupeasy to put togethercheaprepeatableallows holding the deviceallows one-handed usesupports all form factorsruns tests with participants’ phonescaptures screen, face & fingersgives enough video quality
  • 106. screen mountedthe ideal recording setup capture applications document cameras devices ready-made mounted devices DIYeasy to put togethercheaprepeatableallows holding the deviceallows one-handed usesupports all form factorsruns tests with participants’ phonescaptures screen, face & fingersgives enough video quality
  • 107. no recordingsetup isperfect: justuse the besttool for thejob
  • 108. ?
  • 109. test with participants’ own phonesdo not confuse target platform with target usertesting in the lab is better than no testingdo not test over WiFicover your participant’s data costsno recording setup is perfect: just use the besttool for the job
  • 110. thanks!!!www.DIYMobileUsabilityTesting.net