5. usability testing
a process that employs people as testing
participants who are representative of the
target audience to evaluate the degree to
which a product meets specific usability
criteria.
Handbook of usability testing 2nd Ed., J. Rubin and D. Chisnell
6. usability testing
a process that employs people as testing
participants who are representative of the
target audience to evaluate the degree to
which a product meets specific usability
criteria.
Handbook of usability testing 2nd Ed., J. Rubin and D. Chisnell
33. field vs. lab
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
34. field vs. lab
0 0
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
35. The results show that the added value of
conducting usability evaluations in the field is
very little and that recreating central aspects
of the use context in a laboratory setting
enables the identification of the same usability
problem list.
Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware
Mobile Systems in the Field, J. Kjeldskov, M. B. Skov, B. S. Als, R. T. Høegh, 2004
36. 0 1
The results show that the added value of
conducting usability evaluations in the field is
very little and that recreating central aspects
of the use context in a laboratory setting
enables the identification of the same usability
problem list.
Field Lab
Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware
Mobile Systems in the Field, J. Kjeldskov, M. B. Skov, B. S. Als, R. T. Høegh, 2004
37. according to our study there was no
difference in the number of problems that
occurred in the two test settings. Our
hypothesis that more problems would be
found in the field was not supported
Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing
A. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005
38. 0 2
according to our study there was no
difference in the number of problems that
occurred in the two test settings. Our
hypothesis that more problems would be
found in the field was not supported
Field Lab
Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing
A. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005
39. evaluations conducted in field settings can
reveal problems not otherwise identified in
laboratory evaluations
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
40. 1
evaluations conducted in field settings can
reveal problems not otherwise identified in
laboratory evaluations 2
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
Field Lab
41. The analyses of the comparison between
usability testing done in two different settings
revealed that there were many more types
and occurrences of usability problems found
in the field than in the laboratory. Those
problems discovered tend to be critical issues.
Usability Evaluation of Mobile Device: a Comparison of Laboratory and Field Tests
H.B Duh, G. C. B. Tan,V. H. Chen, MobileHCI 2006
42. The analyses of the comparison between
2 2
usability testing done in two different settings
revealed that there were many more types
and occurrences of usability problems found
in the field than in the laboratory. Those
problems discovered tend to be critical issues.
Field Lab
Usability Evaluation of Mobile Device: a Comparison of Laboratory and Field Tests
H.B Duh, G. C. B. Tan,V. H. Chen, MobileHCI 2006
43. field vs. lab
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
44. field vs. lab
DI SA GR EE
EXP ER TS
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
45. ... but they all agree
evaluations in the field (are) more
complex and time-consuming
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
46. ... but they all agree
testing in the field requires double the
time in comparison to the laboratory
Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing
A. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005
47. ... but they all agree
field-based usability studies are not easy
to conduct. They are time consuming and
the added value is questionable.
Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware
Mobile Systems in the Field, J. Kjeldskov, M. B. Skov, B. S. Als, R. T. Høegh, 2004
51. remember ...
do not test over wi-fi
cover participants’ data costs
52. dut = mut + afec
where:
dut = desktop usability testing
mut = mobile usability testing
afec = a few extra challenges
53. tsdohoeaygtrtwt
dut = (mut + afec)
where:
dut = desktop usability testing
mut = mobile usability testing
afec = a few extra challenges
54. tsdohoeaygtrtwt
dut = (mut + afec)
where:
dut = desktop usability testing
mut = mobile usability testing
afec = a few extra challenges
tsdohoeaygtrtwt = the small
detail of how on earth are you
going to record the whole thing
56. 4 approaches to the
small detail of how on earth
are you going to record the whole thing
57. 1. wearable equipment
.
Methods and techniques for field-based usability testing of mobile geo-applications, I. Delikostidis (2007)
International Institute for Geo-Information Science and Earth Observation (Enschede, The Netherlands)
58. 1. wearable equipment
.
Methods and techniques for field-based usability testing of mobile geo-applications, I. Delikostidis (2007)
International Institute for Geo-Information Science and Earth Observation (Enschede, The Netherlands)
59. 1. wearable equipment
.
Methods and techniques for field-based usability testing of mobile geo-applications, I. Delikostidis (2007)
International Institute for Geo-Information Science and Earth Observation (Enschede, The Netherlands)
60. 1. wearable equipment
.
Methods and techniques for field-based usability testing of mobile geo-applications, I. Delikostidis (2007)
International Institute for Geo-Information Science and Earth Observation (Enschede, The Netherlands)
61. Figure 10. Video recording with third-person view of participants and close-up view of PDA. Note that
1. wearable equipment
the camera focused on the device screen is turned 90 degrees to optimize use of the Picture-in-Picture
view.
cm—containing video and audio receivers, Picture-in-Picture unit, hard disk recorder, and battery.
e vulnerableas described above, but now weights only reliability, device. Asofwireless video attached to to get any me
capture to unexpected events,the mobile and size the cameras technology
make it impossible
Will Laboratory Test Results be Valid making it Contexts?, A. Kaikkonen, A. Kekäläinen, M. Cankar, T. Kallio, A.
2 kg and measures only 18x14x25 cm, in Mobile
us schedules. These risks shouldgenceR.T.cheap high-endthe M.video cameras 2011
Kankainen and A Field Laboratory for Evaluating in Situ, of Høegh, J. Kjeldskov, test. J. Stage,
from wireless B. Skov,
matures and becomes more widespread, an emer-
for longer periods of time. Powered by only one matching the professional standard of the wireless
74. Ovo Studios screen capture application for iOS
http://www.ovostudios.com/
75. fingers are not captured ...
Ovo Studios screen capture application for iOS
http://www.ovostudios.com/
76. .. and that is a big deal
Why AirPlay mirroring is the Biggest Thing to Happen to User Research in 2011
http://www.remoteresear.ch/airplay/
77. .. and that is a big deal
Obviously,
think aloud is critical because I cannot see how the participant is interacting
with his fingers on the touch screen.
I can see using the mirroring in a lab setting to get the signal from the iPad
to the observation room though you still won’t see the physical interaction
with the device, like you would with a device mounted camera (e.g. Noldus).
Why AirPlay mirroring is the Biggest Thing to Happen to User Research in 2011
http://www.remoteresear.ch/airplay/
78. .. and that is a big deal
iPad usability testing - our equipment
http://www.cxpartners.co.uk/cxblog/ipad_usability_testing_-_our_equipment/
79. .. and that is a big deal
Recently, we’ve questioned the value of capturing the device screen. It does
after all end up being a video of the screen changing but with no sense
of the participant interacting with it.
iPad usability testing - our equipment
http://www.cxpartners.co.uk/cxblog/ipad_usability_testing_-_our_equipment/
81. Google
Towards the Perfect Infrastructure for Usability Testing on Mobile Devices
R. Schusteritsch, C.Y. Wei, M. LaRosa - Google (CHI 2007)
82. Nielsen Norman
Elmo TT-02RX Teachers Tool
document camera with autofocus
controlled remotely from a laptop
records screen and fingers
webcam records both video streams
participant’s face fed onto a laptop and
recorded with Morae
Photo from Nielsen Norman Mobile Usability workshop handout (London, 22 May 2009)
83. usertesting.com
m o te
http://www.usertesting.com/mobile/
Re
85. but it’s not particularly cheap
Elmo TT-12 Document Camera (accessed March 4th 2012)
http://www.bhphotovideo.com/c/product/843500-REG/Elmo_1331_TT_12_Interactive_Document_Camera.html
http://www.bhphotovideo.com/c/product/644591-REG/Elmo_1304.html
86. ea p
IPEVO Point 2 View USB Document Cam (accessed March 4th 2012)
http://www.ipevo.com/prods/Point-2-View-USB-Camera
http://www.bhphotovideo.com/c/product/644591-REG/Elmo_1304.html C h
92. 4a. ready-made mounted devices
Mobile Device Camera by Tracksys
http://www.tracksys.co.uk/product-
details.php?id=9
Mobile Device Camera by Noldus
http://www.noldus.com/human-behavior-research/
accessories/mobile-device-camera-mdc#
Ovo Studios device camera
http://www.ovostudios.com/devicecamera.asp
93. 4b. DIY mounted devices
Little Springs Design
http://www.littlespringsdesign.com/blog/2008/Jun/usability-
testing-for-mobile-devices-2/
94. 4b. DIY mounted devices
Little Springs Design Nick Bowmast http://www.bowmast.com/mob-device-cam/
http://www.littlespringsdesign.com/blog/2008/Jun/usability-
testing-for-mobile-devices-2/
Google
Towards the Perfect Infrastructure
for Usability Testing on Mobile Usability Sciences
Devices, R. Schusteritsch, C.Y. Wei, http://www.usabilitysciences.com/services/lab-
M. LaRosa - Google (CHI 2007) based-usability-testing/mobile-usability-testing
95. 4b. DIY mounted devices are blooming!!
Little Springs Design Nick Bowmast http://www.bowmast.com/mob-device-cam/
http://www.littlespringsdesign.com/blog/2008/Jun/usability-
testing-for-mobile-devices-2/
by curiouslee
Google
Towards the Perfect Infrastructure
for Usability Testing on Mobile Usability Sciences
Devices, R. Schusteritsch, C.Y. Wei, http://www.usabilitysciences.com/services/lab-
M. LaRosa - Google (CHI 2007) based-usability-testing/mobile-usability-testing
100. and ...
if bulky they can prevent single-hand
use
if heavy they can become tiring during
long tests
101. easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
102. screen mounted mounted
wearable document
capture devices devices
equipment applications cameras ready-made DIY
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face & fingers
gives enough video quality
103. screen mounted mounted
wearable document
capture devices devices
equipment applications cameras ready-made DIY
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face & fingers
gives enough video quality
104. screen mounted mounted
wearable document
capture devices devices
equipment applications cameras ready-made DIY
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face & fingers
gives enough video quality
122. a task
You just moved to Austin, to an old, big
house in Pemberton Heights.You love it,
but there is a problem: mice and rats. The
house is infested!
Go to www.austintexas.gov and find out
how to let the local authorities know
about the infestation.
125. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
126. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
127. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
128. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
129. 15-model Meccano set 14.99
Hue HD webcam 34.95
Philips webcam 24.32
additional Meccano parts 4.01
blu tack 0.98
jubilee clips (x2) 1.99
USB cable 8.99
screwdriver 6.29
CamStudio 0.00
total (in GBP) 96.52
131. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
132. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
133.
134.
135.
136.
137.
138. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
139. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
141. weight: 125 grams
an iPhone weighs 137 grams
an iPad weighs 680 grams
I weigh 55,000 grams
a blue whale weighs 136,400,000 grams
142. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
143. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
144.
145.
146. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
147. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
148. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
149. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
150.
151.
152.
153. b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
154. we expect much of our buildings: they
need to have firm foundations, solid
structures, pleasing aesthetics. We should
expect the same of emerging mobile
systems.
Mobile Interaction Design, M. Jones and G. Marsden (2005)