Our presentation for the IA Summit 2011 in Denver.

2. we are ...

3. Bernard, packet core engineer at NSN

4. Belén, interaction designer at Intel

5. usability testing
a process that employs people as testing
participants who are representative of the
target audience to evaluate the degree to
which a product meets specific usability
criteria.
Handbook of usability testing 2nd Ed., J. Rubin and D. Chisnell

6. please, stand up
everyone

7. sit down if you don’t
have a US cellphone
with a data plan

8. sit down if you don’t
like beer

9. sit down if you are
absolutely terrified by
the idea of being our
test subject

11. why recording?
memory aid
powerful communication tool

12. actions
reactions

13. dut = mut

14. dut = mut + afec
where:
dut = desktop usability testing
mut = mobile usability testing
afec = a few extra challenges

15. which phone?
which context?
which connection?

16. which phone?
which context?
which connection?

17. web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009
http://www.useit.com/alertbox/mobile-usability.html

18. web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009
http://www.useit.com/alertbox/mobile-usability.html

19. web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009
http://www.useit.com/alertbox/mobile-usability.html

20. web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009
http://www.useit.com/alertbox/mobile-usability.html

21. handset usability
affects test results

22. remember ...
test with participants’ own phones
if not possible, include training and
warm-up tasks

23. which phone?
which context?
which connection?

24. field vs. lab
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006

25. experts disagree
according to our study there was no
difference in the number of problems that
occurred in the two test settings. Our
hypothesis that more problems would be
found in the field was not supported
Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing
A. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005

26. experts disagree
evaluations conducted in field settings can
reveal problems not otherwise identified
in laboratory evaluations
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006

27. but they all agree
evaluations in the field (are) more
complex and time-consuming
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
testing in the field requires double the
time in comparison to the laboratory
Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing
A. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005

28. testing in the lab is
better than no testing

29. remember ...
for most software, lab testing is fine
if you must do field testing
do it late
plan and run pilot tests
be prepared (like the Scouts)

30. which phone?
which context?
which connection?

31. !"#$%& '&()"*+,-*.//$0,1*")(2
US Mobile Data Market Update Q4 2010 and 2010, Chetan Sharma Technology & Strategy Consulting
http://www.chetansharma.com/usmarketupdate2010.htm
"$$89::;;;<="#$%&,"%)*%<=+* 67 !"!"#$%&'("%)*%'!+&,-.$/&01' 2..'3/0"$,'3#,#)4#5 #$%&"'())

32. remember ...
do not test over wi-fi
cover participants’ data costs

33. tsdohoeaygtrtwt
dut = (mut + afec)
where:
dut = desktop usability testing
mut = mobile usability testing
afec = a few extra challenges
tsdohoeaygtrtwt = the small
detail of how on earth are you
going to record the whole thing

34. why recording?
memory aid
powerful communication tool

35. 4 approaches to the
small detail of how on earth
are you going to record the whole thing

36. 1. wearable equipment
.
Methods and techniques for field-based usability testing of mobile geo-applications, I. Delikostidis (2007)
International Institute for Geo-Information Science and Earth Observation (Enschede, The Netherlands)

37. allows testing in
the field

38. but ...
difficult and time-consuming to set up
intrusive, uncomfortable and heavy

39. 2. screen-capture applications
UsabilityExpo by Digia for S60 3rd and 5th ed, WinMo 6.1+ and Maemo 5
http://www.digia.com/C2256FEF0043E9C1/0/405001348

40. provides high quality
screen recording

41. but no application will
support all platforms
http://www.digia.com/C2256FEF0043E9C1/0/405001348

42. it’s not cheap
https://shop.digia.com

43. and participants won’t appreciate
you installing stuff on their phones

44. 3. document cameras
Handheld Usability, S. Weiss (page 174)

45. Google
Towards the Perfect Infrastructure for Usability Testing on Mobile Devices
R. Schusteritsch, C.Y. Wei, M. LaRosa - Google (CHI 2007)

46. Nielsen Norman
Elmo TT-02RX Teachers Tool
document camera with autofocus
controlled remotely from a laptop
records screen and fingers
webcam records both video streams
participant’s face fed onto a laptop and
recorded with Morae
Photo from Nielsen Norman Mobile Usability workshop handout (London, 22 May 2009)

47. popular and well
documented

48. it’s not cheap
http://www.bhphotovideo.com/c/product/644591-REG/Elmo_1304.html

49. participants must
keep within the
camera range

50. phone must lay on
a desk or be hold
at a flat angle
http://www.flickr.com/photos/zabriskiepoint/2806511301/sizes/m/in/photostream/

51. 4. mounted devices

52. 4a. ready-made mounted devices
Mobile Device Camera by Tracksys
http://www.tracksys.co.uk/product-
details.php?id=9
Mobile Device Camera by Noldus
http://www.noldus.com/human-behavior-
research/accessories/mobile-device-camera-
mdc#

53. 4b. DIY mounted devices
Little Springs Design
http://www.littlespringsdesign.com/blog/2008/Jun/usability-
testing-for-mobile-devices-2/

54. 4b. DIY mounted devices
Little Springs Design Nick Bowmast http://www.bowmast.com/mob-device-cam/
http://www.littlespringsdesign.com/blog/2008/Jun/usability-
testing-for-mobile-devices-2/

55. 4b. DIY mounted devices
Little Springs Design Nick Bowmast http://www.bowmast.com/mob-device-cam/
http://www.littlespringsdesign.com/blog/2008/Jun/usability-
testing-for-mobile-devices-2/
Usability Sciences
http://www.usabilitysciences.com/services/lab-
based-usability-testing/mobile-usability-testing

56. 4b. DIY mounted devices
Little Springs Design Nick Bowmast http://www.bowmast.com/mob-device-cam/
http://www.littlespringsdesign.com/blog/2008/Jun/usability-
testing-for-mobile-devices-2/
Google
Towards the Perfect Infrastructure
for Usability Testing on Mobile Usability Sciences
Devices, R. Schusteritsch, C.Y. Wei, http://www.usabilitysciences.com/services/lab-
M. LaRosa - Google (CHI 2007) based-usability-testing/mobile-usability-testing

57. natural interaction with
the phone

58. but it’s not cheap
http://www.90percentofeverything.com/2010/05/07/quick-tip-make-your-own-iphone-usability-testing-sled-for-5/

59. messy to build
http://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/

60. and ...
if bulky they can prevent single-hand
use
if heavy they can become tiring during
long tests

61. easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

62. easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face & fingers
gives enough video quality

63. easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face & fingers
gives enough video quality

64. the spirit
http://www.wired.com/gadgetlab/2010/07/the-pencil-ipad-stand-smart-enough-to-impress-a-new-yorker/

65. the
ingredients

66. two meccano trunions (part no. A126)

67. two 5 and 6-hole meccano strips (part nos. 5 & 4)

68. one 11-hole meccano strip (part no. 2)

69. six meccano screws & nuts (part nos. 69 & 37h)

70. one 13-20mm jubilee clip

71. one HUE HD webcam

72. a second USB webcam

73. a USB male to female extension cable

74. blu tack

75. an allen key

76. a meccano wrench

77. a screwdriver

78. a Windows computer

79. screen recording software

81. a task
You are planning a visit to the Denver
Zoo after this (great) session.
Go to www.denverzoo.org and find out
which bus number will bring you to the
zoo from the Colorado Convention
Centre.

82. how was that?

84. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

85. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

86. 15-model meccano set 18.33
Hue HD webcam 28.94
Philips webcam 52.08
additional meccano parts 4.90
blu tack 0.98
jubilee clips (x2) 1.55
USB cable 8.99
screwdriver 5.99
CamStudio 0.00
total (in GBP) 121.76

87. in USD: 196.40

88. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

94. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

95. weight: 125 grams
an iPhone weighs 137 grams
an iPad weighs 680 grams
I weigh 55,000 grams
a blue whale weighs 136,400,000 grams

96. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

97. Nokia E66
slider with 12-key
numeric keypad

98. Nokia E71
BB Bold
candybar with
qwerty keyboard

99. iPhone 3G
touchscreen
candybar

100. Nokia N900
slider
touchscreen
with qwerty
keyboard

102. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

106. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality

107. we expect much of our buildings: they
need to have firm foundations, solid
structures, pleasing aesthetics. We should
expect the same of emerging mobile
systems.
Mobile Interaction Design, M. Jones and G. Marsden (2005)

108. thanks!
belenbarrospena@gmail.com
b@runningwithbulls.com
@belenpena @bernardtyers
http://belenpena.posterous.com
http://www.runningwithbulls.com