we are ...
Bernard, packet core engineer at NSN
Belén, interaction designer at LBi UK
the proverbial poll
how many have
   observed a mobile
usability testing session?
how many have
planned or moderated
mobile usability testing?
usability testing
a process that employs people as testing
participants who are representative of the
target audience to e...
why recording?

no need to rely on notes
and memory

a powerful communication
means
actions

          reactions
dut = mut
dut = mut + afec

where:

dut = desktop usability testing
mut = mobile usability testing
afec = a few extra challenges
which phone?

which context?

which connection?
which phone?

which context?

which connection?
web task success rates
         feature phones                               38%

           smartphones                  ...
web task success rates
         feature phones                               38%

           smartphones                  ...
web task success rates
         feature phones                               38%

           smartphones                  ...
web task success rates
         feature phones                               38%

           smartphones                  ...
handset usability
affects test results
remember...

test with participants’ own
phones

if not possible, include
training and warm-up tasks
which phone?

which context?

which connection?
field vs. lab




It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Ni...
experts disagree
according to our study there was no
difference in the number of problems that
occurred in the two test se...
experts disagree

evaluations conducted in field settings can
reveal problems not otherwise identified
in laboratory evaluat...
but they all agree
evaluations in the field (are) more
complex and time-consuming
It’s Worth the Hassle! The Added Value of...
testing in the lab is
better than no testing
remember...

for most software, lab testing is fine
if you must do field testing...
     do it late
     plan and run pilot ...
which phone?

which context?

which connection?
http://www.coveragemaps.com/gsmposter_europe.htm
state of coverage: UK

mobile coverage for 2G is extensive at
98%. The picture for 3G coverage is
different. Figures are h...
Map of UK 3G geographic coverage by number of operators
Ofcom / GSM Association / Europa Technologies, Q1 2009
disable 3G and see
   what happens
remember...

do not test over wi-fi

cover participants’ data costs
tsdohoeaygtrtwt
dut = (mut + afec)

where:

dut = desktop usability testing
mut = mobile usability testing
afec = a few ex...
why recording?

no need to rely on notes
and memory

a powerful communication
means
4 approaches to the
small detail of how on earth
   are you   going to   record the   whole   thing
1. wearable equipment




Methods and techniques for field-based usability testing of mobile geo-applications, I. Delikosti...
allows testing in
    the field
but...


difficult and time-consuming set up
intrusive, uncomfortable and heavy
2. screen-capture applications




UsabilityExpo by Digia for S60 3rd and 5th ed, WinMo 6.1+ and Maemo OS2008
http://www.d...
provides high quality
  screen recording
but no app will support
               all platforms




http://www.digia.com/C2256FEF0043E9C1/0/405001348
it’s not cheap




https://shop.digia.com
and participants
won’t appreciate
you installing
stuff on their
phones
3. document cameras




Handheld Usability, S. Weiss (page 174)
Google




Towards the Perfect Infrastructure for Usability Testing on Mobile Devices, R. Schusteritsch, C.Y. Wei, M. LaRo...
Nielsen Norman



                                                                     Elmo TT-02RX Teachers Tool




    ...
Nielsen Norman

                                   document camera with autofocus
                                   contr...
Foolproof




            http://www.foolproof.co.uk/our-services/user-experience/facilities/
popular and well
  documented
but it’s not cheap, and...




http://www.bhphotovideo.com/c/product/644591-REG/Elmo_1304.html
participants must
keep within the
camera range
phone must lay
         on a desk or be
         hold at a flat angle




http://www.flickr.com/photos/zabriskiepoint/280651...
4. mounted devices (a) ready made




                                                  Mobile Device Camera by Tracksys
 ...
4. mounted devices (b) DIY




                                                             Nick Bowmast http://www.bowmas...
4. mounted devices (b) DIY




     Google
     Towards the Perfect Infrastructure
                                       ...
natural interaction with
       the phone
but not cheap to buy




http://www.90percentofeverything.com/2010/05/07/quick-tip-make-your-own-iphone-usability-testing-...
messy to build




http://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/
and...

if bulky they can prevent single-
hand use
if heavy they can become tiring
during long tests
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures ...
1   2   3   4a   4b
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all fo...
1   2   3   4a   4b
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all fo...
1   2   3   4a   4b
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all fo...
the spirit




http://www.wired.com/gadgetlab/2010/07/the-pencil-ipad-stand-smart-enough-to-impress-a-new-yorker/
the
ingredients
two meccano trunions (part no. 126)
two 5 and 6-hole meccano strips (part nos. 5 & 4)
one 11-hole meccano strip (part no. 2)
six meccano screws & nuts (part nos. 69 & 37h)
a 13-20mm jubilee clip
one HUE HD webcam
a second USB webcam
a USB male to female extension cable
blu tack
an allen key
a meccano wrench
a screwdriver
a Windows computer (or virtual machine)
CamStudio (free and open source)
a fake (but realistic)
impromptu usability test
task 1

Your friend from Dublin is flying to
London tonight. He is a cheap dude, so he
must be flying Ryanair. He hasn’t tol...
task 2

Your friend from Dublin is flying to
London tonight. He is a posh dude, so he
must be flying BA. He hasn’t told you
...
how was that?
b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
cap...
b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
cap...
15-model meccano set                 18.33
Hue HD webcam                        28.94
Philips webcam                      ...
in Euro: 143.429
b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
cap...
b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
cap...
weight: 125 grams
 an iPhone weighs 137 grams
 an iPad weighs 680 grams
 I weigh 55,000 grams
 a blue whale weighs 136,400...
b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
cap...
Nokia E66
slider with 12-key
numeric keypad
Nokia E71

      BB Bold
      candybar with
      qwerty keyboard
iPhone 3G
touchscreen
candybar
Nokia N900

             slider
             touchscreen
             with qwerty
             keyboard
b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
cap...
b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
cap...
b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
cap...
b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
cap...
a call to arms




an Irish unit of the International Brigade   http://www.communistpartyofireland.ie/spainn.jpg
thanks!

   @belenpena @chistorra

 http://belenpena.posterous.com
http://www.runningwithbulls.com
Do it yourself mobile usability testing at EuroIA 2010
Do it yourself mobile usability testing at EuroIA 2010
Do it yourself mobile usability testing at EuroIA 2010
Do it yourself mobile usability testing at EuroIA 2010
Do it yourself mobile usability testing at EuroIA 2010
Do it yourself mobile usability testing at EuroIA 2010
Do it yourself mobile usability testing at EuroIA 2010
Do it yourself mobile usability testing at EuroIA 2010
Do it yourself mobile usability testing at EuroIA 2010
Do it yourself mobile usability testing at EuroIA 2010
Do it yourself mobile usability testing at EuroIA 2010
Do it yourself mobile usability testing at EuroIA 2010
Do it yourself mobile usability testing at EuroIA 2010
Do it yourself mobile usability testing at EuroIA 2010
Upcoming SlideShare
Loading in...5
×

Do it yourself mobile usability testing at EuroIA 2010

20,285

Published on

Published in: Technology
1 Comment
87 Likes
Statistics
Notes
No Downloads
Views
Total Views
20,285
On Slideshare
0
From Embeds
0
Number of Embeds
14
Actions
Shares
0
Downloads
0
Comments
1
Likes
87
Embeds 0
No embeds

No notes for slide

Transcript of "Do it yourself mobile usability testing at EuroIA 2010"

  1. 1. we are ...
  2. 2. Bernard, packet core engineer at NSN
  3. 3. Belén, interaction designer at LBi UK
  4. 4. the proverbial poll
  5. 5. how many have observed a mobile usability testing session?
  6. 6. how many have planned or moderated mobile usability testing?
  7. 7. usability testing a process that employs people as testing participants who are representative of the target audience to evaluate the degree to which a product meets specific usability criteria. Handbook of usability testing 2nd Ed., J. Rubin and D. Chisnell
  8. 8. why recording? no need to rely on notes and memory a powerful communication means
  9. 9. actions reactions
  10. 10. dut = mut
  11. 11. dut = mut + afec where: dut = desktop usability testing mut = mobile usability testing afec = a few extra challenges
  12. 12. which phone? which context? which connection?
  13. 13. which phone? which context? which connection?
  14. 14. web task success rates feature phones 38% smartphones 55% touch phones 75% Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009 http://www.useit.com/alertbox/mobile-usability.html
  15. 15. web task success rates feature phones 38% smartphones 55% touch phones 75% Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009 http://www.useit.com/alertbox/mobile-usability.html
  16. 16. web task success rates feature phones 38% smartphones 55% touch phones 75% Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009 http://www.useit.com/alertbox/mobile-usability.html
  17. 17. web task success rates feature phones 38% smartphones 55% touch phones 75% Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009 http://www.useit.com/alertbox/mobile-usability.html
  18. 18. handset usability affects test results
  19. 19. remember... test with participants’ own phones if not possible, include training and warm-up tasks
  20. 20. which phone? which context? which connection?
  21. 21. field vs. lab It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
  22. 22. experts disagree according to our study there was no difference in the number of problems that occurred in the two test settings. Our hypothesis that more problems would be found in the field was not supported Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing A. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005
  23. 23. experts disagree evaluations conducted in field settings can reveal problems not otherwise identified in laboratory evaluations It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
  24. 24. but they all agree evaluations in the field (are) more complex and time-consuming It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006 testing in the field requires double the time in comparison to the laboratory Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing A. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005
  25. 25. testing in the lab is better than no testing
  26. 26. remember... for most software, lab testing is fine if you must do field testing... do it late plan and run pilot tests be prepared (like the Scouts)
  27. 27. which phone? which context? which connection?
  28. 28. http://www.coveragemaps.com/gsmposter_europe.htm
  29. 29. state of coverage: UK mobile coverage for 2G is extensive at 98%. The picture for 3G coverage is different. Figures are high overall (87%), but vary markedly in the nations, with only 43% availability in Northern Ireland Mobile Evolution - Ofcom’s mobile sector assessment - 17 December 2009 (p35) http://stakeholders.ofcom.org.uk/binaries/consultations/msa/statement/MSA_statement.pdf
  30. 30. Map of UK 3G geographic coverage by number of operators Ofcom / GSM Association / Europa Technologies, Q1 2009
  31. 31. disable 3G and see what happens
  32. 32. remember... do not test over wi-fi cover participants’ data costs
  33. 33. tsdohoeaygtrtwt dut = (mut + afec) where: dut = desktop usability testing mut = mobile usability testing afec = a few extra challenges tsdohoeaygtrtwt = the small detail of how on earth are you going to record the whole thing
  34. 34. why recording? no need to rely on notes and memory a powerful communication means
  35. 35. 4 approaches to the small detail of how on earth are you going to record the whole thing
  36. 36. 1. wearable equipment Methods and techniques for field-based usability testing of mobile geo-applications, I. Delikostidis (2007) International Institute for Geo-Information Science and Earth Observation (Enschede, The Netherlands)
  37. 37. allows testing in the field
  38. 38. but... difficult and time-consuming set up intrusive, uncomfortable and heavy
  39. 39. 2. screen-capture applications UsabilityExpo by Digia for S60 3rd and 5th ed, WinMo 6.1+ and Maemo OS2008 http://www.digia.com/C2256FEF0043E9C1/0/405001348
  40. 40. provides high quality screen recording
  41. 41. but no app will support all platforms http://www.digia.com/C2256FEF0043E9C1/0/405001348
  42. 42. it’s not cheap https://shop.digia.com
  43. 43. and participants won’t appreciate you installing stuff on their phones
  44. 44. 3. document cameras Handheld Usability, S. Weiss (page 174)
  45. 45. Google Towards the Perfect Infrastructure for Usability Testing on Mobile Devices, R. Schusteritsch, C.Y. Wei, M. LaRosa - Google (CHI 2007)
  46. 46. Nielsen Norman Elmo TT-02RX Teachers Tool Photo from Nielsen Norman Mobile Usability workshop handout (London, 22 May 2009)
  47. 47. Nielsen Norman document camera with autofocus controlled remotely from a laptop records screen and fingers webcam records both video streams participant’s face fed onto a laptop and recorded with Morae Photo from Nielsen Norman Mobile Usability workshop handout (London, 22 May 2009)
  48. 48. Foolproof http://www.foolproof.co.uk/our-services/user-experience/facilities/
  49. 49. popular and well documented
  50. 50. but it’s not cheap, and... http://www.bhphotovideo.com/c/product/644591-REG/Elmo_1304.html
  51. 51. participants must keep within the camera range
  52. 52. phone must lay on a desk or be hold at a flat angle http://www.flickr.com/photos/zabriskiepoint/2806511301/sizes/m/in/photostream/
  53. 53. 4. mounted devices (a) ready made Mobile Device Camera by Tracksys http://www.tracksys.co.uk/product- details.php?id=9 Mobile Device Camera by Noldus http://www.noldus.com/human-behavior- research/accessories/mobile-device-camera- mdc#
  54. 54. 4. mounted devices (b) DIY Nick Bowmast http://www.bowmast.com/mob-device-cam/ Little Springs Design http://www.littlespringsdesign.com/blog/2008/Jun/usability-testing- for-mobile-devices-2/
  55. 55. 4. mounted devices (b) DIY Google Towards the Perfect Infrastructure Usability Sciences for Usability Testing on Mobile http://www.usabilitysciences.com/services/lab- Devices, R. Schusteritsch, C.Y. Wei, based-usability-testing/mobile-usability-testing M. LaRosa - Google (CHI 2007)
  56. 56. natural interaction with the phone
  57. 57. but not cheap to buy http://www.90percentofeverything.com/2010/05/07/quick-tip-make-your-own-iphone-usability-testing-sled-for-5/
  58. 58. messy to build http://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/
  59. 59. and... if bulky they can prevent single- hand use if heavy they can become tiring during long tests
  60. 60. easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  61. 61. 1 2 3 4a 4b easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  62. 62. 1 2 3 4a 4b easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  63. 63. 1 2 3 4a 4b easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  64. 64. the spirit http://www.wired.com/gadgetlab/2010/07/the-pencil-ipad-stand-smart-enough-to-impress-a-new-yorker/
  65. 65. the ingredients
  66. 66. two meccano trunions (part no. 126)
  67. 67. two 5 and 6-hole meccano strips (part nos. 5 & 4)
  68. 68. one 11-hole meccano strip (part no. 2)
  69. 69. six meccano screws & nuts (part nos. 69 & 37h)
  70. 70. a 13-20mm jubilee clip
  71. 71. one HUE HD webcam
  72. 72. a second USB webcam
  73. 73. a USB male to female extension cable
  74. 74. blu tack
  75. 75. an allen key
  76. 76. a meccano wrench
  77. 77. a screwdriver
  78. 78. a Windows computer (or virtual machine)
  79. 79. CamStudio (free and open source)
  80. 80. a fake (but realistic) impromptu usability test
  81. 81. task 1 Your friend from Dublin is flying to London tonight. He is a cheap dude, so he must be flying Ryanair. He hasn’t told you which airport he is flying to, but you know he should be landing at 12:45 in the afternoon. Go to www.ryanair.com and find out if his flight is on time.
  82. 82. task 2 Your friend from Dublin is flying to London tonight. He is a posh dude, so he must be flying BA. He hasn’t told you which airport he is flying to, but you know he should be landing at 11:10 in the morning. Go to www.ba.com and find out if his flight is on time.
  83. 83. how was that?
  84. 84. b&b’s easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  85. 85. b&b’s easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  86. 86. 15-model meccano set 18.33 Hue HD webcam 28.94 Philips webcam 52.08 additional meccano parts 4.90 blu tack 0.98 jubilee clips (x2) 1.55 USB cable 8.99 screwdriver 5.99 CamStudio 0.00 total (in GBP) 121.76
  87. 87. in Euro: 143.429
  88. 88. b&b’s easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  89. 89. b&b’s easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  90. 90. weight: 125 grams an iPhone weighs 137 grams an iPad weighs 680 grams I weigh 55,000 grams a blue whale weighs 136,400,000 grams
  91. 91. b&b’s easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  92. 92. Nokia E66 slider with 12-key numeric keypad
  93. 93. Nokia E71 BB Bold candybar with qwerty keyboard
  94. 94. iPhone 3G touchscreen candybar
  95. 95. Nokia N900 slider touchscreen with qwerty keyboard
  96. 96. b&b’s easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  97. 97. b&b’s easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  98. 98. b&b’s easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  99. 99. b&b’s easy to put together cheap repeatable allows holding the device allows one-handed hold supports all form factors captures screen, face & fingers runs tests with participants’ phones gives enough video quality
  100. 100. a call to arms an Irish unit of the International Brigade http://www.communistpartyofireland.ie/spainn.jpg
  101. 101. thanks! @belenpena @chistorra http://belenpena.posterous.com http://www.runningwithbulls.com

×