Do it yourself mobile usability testing at EuroIA 2010

31,885 views
30,790 views

Published on

Published in: Technology
1 Comment
90 Likes
Statistics
Notes
No Downloads
Views
Total views
31,885
On SlideShare
0
From Embeds
0
Number of Embeds
5,197
Actions
Shares
0
Downloads
0
Comments
1
Likes
90
Embeds 0
No embeds

No notes for slide

Do it yourself mobile usability testing at EuroIA 2010

  1. we are ...
  2. Bernard, packet core engineer at NSN
  3. Belén, interaction designer at LBi UK
  4. the proverbial poll
  5. how many have observed a mobile usability testing session?
  6. how many have planned or moderated mobile usability testing?
  7. usability testing a process that employs people as testing participants who are representative of the target audience to evaluate the degree to which a product meets specific usability criteria. Handbook of usability testing 2nd Ed., J. Rubin and D. Chisnell
  8. why recording? no need to rely on notes and memory a powerful communication means
  9. actions reactions
  10. dut = mut
  11. dut = mut + afec where: dut = desktop usability testing mut = mobile usability testing afec = a few extra challenges
  12. which phone? which context? which connection?
  13. which phone? which context? which connection?
  14. web task success rates feature phones 38% smartphones 55% touch phones 75% Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009 http://www.useit.com/alertbox/mobile-usability.html
  15. web task success rates feature phones 38% smartphones 55% touch phones 75% Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009 http://www.useit.com/alertbox/mobile-usability.html
  16. web task success rates feature phones 38% smartphones 55% touch phones 75% Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009 http://www.useit.com/alertbox/mobile-usability.html
  17. web task success rates feature phones 38% smartphones 55% touch phones 75% Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009 http://www.useit.com/alertbox/mobile-usability.html
  18. handset usability affects test results
  19. remember... test with participants’ own phones if not possible, include training and warm-up tasks
  20. which phone? which context? which connection?
  21. field vs. lab It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
  22. experts disagree according to our study there was no difference in the number of problems that occurred in the two test settings. Our hypothesis that more problems would be found in the field was not supported Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing A. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005
  23. experts disagree evaluations conducted in field settings can reveal problems not otherwise identified in laboratory evaluations It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
  24. but they all agree evaluations in the field (are) more complex and time-consuming It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006 testing in the field requires double the time in comparison to the laboratory Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing A. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005
  25. testing in the lab is better than no testing
  26. remember... for most software, lab testing is fine if you must do field testing... do it late plan and run pilot tests be prepared (like the Scouts)
  27. which phone? which context? which connection?
  28. http://www.coveragemaps.com/gsmposter_europe.htm
  29. state of coverage: UK mobile coverage for 2G is extensive at 98%. The picture for 3G coverage is different. Figures are high overall (87%), but vary markedly in the nations, with only 43% availability in Northern Ireland Mobile Evolution - Ofcom’s mobile sector assessment - 17 December 2009 (p35) http://stakeholders.ofcom.org.uk/binaries/consultations/msa/statement/MSA_statement.pdf
  30. Map of UK 3G geographic coverage by number of operators Ofcom / GSM Association / Europa Technologies, Q1 2009
  31. disable 3G and see what happens
  32. remember... do not test over wi-fi cover participants’ data costs
  33. tsdohoeaygtrtwt dut = (mut + afec) where: dut = desktop usability testing mut = mobile usability testing afec = a few extra challenges tsdohoeaygtrtwt = the small detail of how on earth are you going to record the whole thing
  34. why recording? no need to rely on notes and memory a powerful communication means
  35. 4 approaches to the small detail of how on earth are you going to record the whole thing
  36. 1. wearable equipment Methods and techniques for field-based usability testing of mobile geo-applications, I. Delikostidis (2007) International Institute for Geo-Information Science and Earth Observation (Enschede, The Netherlands)
  37. allows testing in the field
  38. but... difficult and time-consuming set up intrusive, uncomfortable and heavy
  39. 2. screen-capture applications UsabilityExpo by Digia for S60 3rd and 5th ed, WinMo 6.1+ and Maemo OS2008 http://www.digia.com/C2256FEF0043E9C1/0/405001348
  40. provides high quality screen recording
  41. but no app will support all platforms http://www.digia.com/C2256FEF0043E9C1/0/405001348
  42. it’s not cheap https://shop.digia.com
  43. and participants won’t appreciate you installing stuff on their phones
  44. 3. document cameras Handheld Usability, S. Weiss (page 174)
  45. Google Towards the Perfect Infrastructure for Usability Testing on Mobile Devices, R. Schusteritsch, C.Y. Wei, M. LaRosa - Google (CHI 2007)
  46. Nielsen Norman Elmo TT-02RX Teachers Tool Photo from Nielsen Norman Mobile Usability workshop handout (London, 22 May 2009)
  47. Nielsen Norman document camera with autofocus controlled remotely from a laptop records screen and fingers webcam records both video streams participant’s face fed onto a laptop and recorded with Morae Photo from Nielsen Norman Mobile Usability workshop handout (London, 22 May 2009)
  48. Foolproof http://www.foolproof.co.uk/our-services/user-experience/facilities/
  49. popular and well documented
  50. but it’s not cheap, and... http://www.bhphotovideo.com/c/product/644591-REG/Elmo_1304.html
  51. participants must keep within the camera range
  52. phone must lay on a desk or be hold at a flat angle http://www.flickr.com/photos/zabriskiepoint/2806511301/sizes/m/in/photostream/
  53. 4. mounted devices (a) ready made Mobile Device Camera by Tracksys http://www.tracksys.co.uk/product- details.php?id=9 Mobile Device Camera by Noldus http://www.noldus.com/human-behavior- research/accessories/mobile-device-camera- mdc#
  54. 4. mounted devices (b) DIY Nick Bowmast http://www.bowmast.com/mob-device-cam/ Little Springs Design http://www.littlespringsdesign.com/blog/2008/Jun/usability-testing- for-mobile-devices-2/
  55. 4. mounted devices (b) DIY Google Towards the Perfect Infrastructure Usability Sciences for Usability Testing on Mobile http://www.usabilitysciences.com/services/lab- Devices, R. Schusteritsch, C.Y. Wei, based-usability-testing/mobile-usability-testing M. LaRosa - Google (CHI 2007)
  56. natural interaction with the phone
  57. but not cheap to buy http://www.90percentofeverything.com/2010/05/07/quick-tip-make-your-own-iphone-usability-testing-sled-for-5/
  58. messy to build http://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/
  59. and... if bulky they can prevent single- hand use if heavy they can become tiring during long tests
  60. easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  61. 1 2 3 4a 4b easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  62. 1 2 3 4a 4b easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  63. 1 2 3 4a 4b easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  64. the spirit http://www.wired.com/gadgetlab/2010/07/the-pencil-ipad-stand-smart-enough-to-impress-a-new-yorker/
  65. the ingredients
  66. two meccano trunions (part no. 126)
  67. two 5 and 6-hole meccano strips (part nos. 5 & 4)
  68. one 11-hole meccano strip (part no. 2)
  69. six meccano screws & nuts (part nos. 69 & 37h)
  70. a 13-20mm jubilee clip
  71. one HUE HD webcam
  72. a second USB webcam
  73. a USB male to female extension cable
  74. blu tack
  75. an allen key
  76. a meccano wrench
  77. a screwdriver
  78. a Windows computer (or virtual machine)
  79. CamStudio (free and open source)
  80. a fake (but realistic) impromptu usability test
  81. task 1 Your friend from Dublin is flying to London tonight. He is a cheap dude, so he must be flying Ryanair. He hasn’t told you which airport he is flying to, but you know he should be landing at 12:45 in the afternoon. Go to www.ryanair.com and find out if his flight is on time.
  82. task 2 Your friend from Dublin is flying to London tonight. He is a posh dude, so he must be flying BA. He hasn’t told you which airport he is flying to, but you know he should be landing at 11:10 in the morning. Go to www.ba.com and find out if his flight is on time.
  83. how was that?
  84. b&b’s easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  85. b&b’s easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  86. 15-model meccano set 18.33 Hue HD webcam 28.94 Philips webcam 52.08 additional meccano parts 4.90 blu tack 0.98 jubilee clips (x2) 1.55 USB cable 8.99 screwdriver 5.99 CamStudio 0.00 total (in GBP) 121.76
  87. in Euro: 143.429
  88. b&b’s easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  89. b&b’s easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  90. weight: 125 grams an iPhone weighs 137 grams an iPad weighs 680 grams I weigh 55,000 grams a blue whale weighs 136,400,000 grams
  91. b&b’s easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  92. Nokia E66 slider with 12-key numeric keypad
  93. Nokia E71 BB Bold candybar with qwerty keyboard
  94. iPhone 3G touchscreen candybar
  95. Nokia N900 slider touchscreen with qwerty keyboard
  96. b&b’s easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  97. b&b’s easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  98. b&b’s easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  99. b&b’s easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  100. a call to arms an Irish unit of the International Brigade http://www.communistpartyofireland.ie/spainn.jpg
  101. thanks! @belenpena @chistorra http://belenpena.posterous.com http://www.runningwithbulls.com

×