8. usability testing
a process that employs people as testing
participants who are representative of the
target audience to evaluate the degree to
which a product meets specific usability
criteria.
Handbook of usability testing 2nd Ed., J. Rubin and D. Chisnell
23. field vs. lab
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
24. experts disagree
according to our study there was no
difference in the number of problems that
occurred in the two test settings. Our
hypothesis that more problems would be
found in the field was not supported
Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing
A. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005
25. experts disagree
evaluations conducted in field settings can
reveal problems not otherwise identified
in laboratory evaluations
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
26. but they all agree
evaluations in the field (are) more
complex and time-consuming
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
testing in the field requires double the
time in comparison to the laboratory
Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing
A. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005
31. state of coverage: UK
mobile coverage for 2G is extensive at
98%. The picture for 3G coverage is
different. Figures are high overall (87%),
but vary markedly in the nations, with
only 43% availability in Northern Ireland
Mobile Evolution - Ofcom’s mobile sector assessment - 17 December 2009 (p35)
http://stakeholders.ofcom.org.uk/binaries/consultations/msa/statement/MSA_statement.pdf
32. Map of UK 3G geographic coverage by number of operators
Ofcom / GSM Association / Europa Technologies, Q1 2009
35. tsdohoeaygtrtwt
dut = (mut + afec)
where:
dut = desktop usability testing
mut = mobile usability testing
afec = a few extra challenges
tsdohoeaygtrtwt = the small
detail of how on earth are you
going to record the whole thing
37. 4 approaches to the
small detail of how on earth
are you going to record the whole thing
38. 1. wearable equipment
Methods and techniques for field-based usability testing of mobile geo-applications, I. Delikostidis (2007)
International Institute for Geo-Information Science and Earth Observation (Enschede, The Netherlands)
47. Google
Towards the Perfect Infrastructure for Usability Testing on Mobile Devices, R. Schusteritsch, C.Y. Wei, M. LaRosa - Google (CHI 2007)
48. Nielsen Norman
Elmo TT-02RX Teachers Tool
Photo from Nielsen Norman Mobile Usability workshop handout (London, 22 May 2009)
49. Nielsen Norman
document camera with autofocus
controlled remotely from a laptop
records screen and fingers
webcam records both video streams
participant’s face fed onto a laptop and
recorded with Morae
Photo from Nielsen Norman Mobile Usability workshop handout (London, 22 May 2009)
54. phone must lay
on a desk or be
hold at a flat angle
http://www.flickr.com/photos/zabriskiepoint/2806511301/sizes/m/in/photostream/
55. 4. mounted devices (a) ready made
Mobile Device Camera by Tracksys
http://www.tracksys.co.uk/product-
details.php?id=9
Mobile Device Camera by Noldus
http://www.noldus.com/human-behavior-
research/accessories/mobile-device-camera-
mdc#
56. 4. mounted devices (b) DIY
Nick Bowmast http://www.bowmast.com/mob-device-cam/
Little Springs Design
http://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-
for-mobile-devices-2/
57. 4. mounted devices (b) DIY
Google
Towards the Perfect Infrastructure
Usability Sciences
for Usability Testing on Mobile
http://www.usabilitysciences.com/services/lab-
Devices, R. Schusteritsch, C.Y. Wei,
based-usability-testing/mobile-usability-testing
M. LaRosa - Google (CHI 2007)
61. and...
if bulky they can prevent single-
hand use
if heavy they can become tiring
during long tests
62. easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & fingers
runs tests with participants’ phones
gives enough video quality
63. 1 2 3 4a 4b
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & fingers
runs tests with participants’ phones
gives enough video quality
64. 1 2 3 4a 4b
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & fingers
runs tests with participants’ phones
gives enough video quality
65. 1 2 3 4a 4b
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & fingers
runs tests with participants’ phones
gives enough video quality
85. a fake (but realistic)
impromptu usability test
86. task 1
Your friend from Dublin is flying to
London tonight. He is a cheap dude, so he
must be flying Ryanair. He hasn’t told you
which airport he is flying to, but you know
he should be landing at 12:45 in the
afternoon. Go to www.ryanair.com and
find out if his flight is on time.
87. task 2
Your friend from Dublin is flying to
London tonight. He is a posh dude, so he
must be flying BA. He hasn’t told you
which airport he is flying to, but you know
he should be landing at 11:10 in the
morning. Go to www.ba.com and find out
if his flight is on time.
90. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & fingers
runs tests with participants’ phones
gives enough video quality
91. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & fingers
runs tests with participants’ phones
gives enough video quality
92. 15-model meccano set 18.33
Hue HD webcam 28.94
Philips webcam 52.08
additional meccano parts 4.90
blu tack 0.98
jubilee clips (x2) 1.55
USB cable 8.99
screwdriver 5.99
CamStudio 0.00
total (in GBP) 121.76
94. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & fingers
runs tests with participants’ phones
gives enough video quality
95.
96.
97.
98.
99.
100. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & fingers
runs tests with participants’ phones
gives enough video quality
101. weight: 125 grams
an iPhone weighs 137 grams
an iPad weighs 680 grams
I weigh 55,000 grams
a blue whale weighs 136,400,000 grams
102. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & fingers
runs tests with participants’ phones
gives enough video quality
106. Nokia N900
slider
touchscreen
with qwerty
keyboard
107. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & fingers
runs tests with participants’ phones
gives enough video quality
108.
109.
110. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & fingers
runs tests with participants’ phones
gives enough video quality
111. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & fingers
runs tests with participants’ phones
gives enough video quality
112.
113. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed hold
supports all form factors
captures screen, face & fingers
runs tests with participants’ phones
gives enough video quality
114. a call to arms
an Irish unit of the International Brigade http://www.communistpartyofireland.ie/spainn.jpg