Remote Testing
Prague
June 2006

Remote Testing
\"History\" (I)
§ Usability testing for Otto
• Only specific areas / processes in the shop
• Not the \"whole customer experience\"
§ Development of \"seasonal testings\"
• Recruit participants that are about to buy something on otto.de
• Observe them just doing that in the labs
§ Open issues
• Testing is still not \"real\" enough
- Lab situation
- Perform tasks with possibly low involvement
- Not all relevant tasks in the scope of the lab test
Remote Testing page 2

Remote Testing
\"History\" (II)
§ Idea: Remote-Testing
• At home (\"natural\" environment)
• Real involvement
• \"Natural\" tasks (just what people want to do)
§ Challenges
• Tool must work without installation on PCs of users
• Tool must work without having to change anything on the server of
the client
• We must be able to \"observe\" and \"measure\" (not only \"ask\")
Remote Testing page 3

Remote Testing
How does Remote Testing work?
Participants Proxy Server Web Site
Natural use of the target Passes inquiries on to
web site
Recalls all recorded
actions of the
respondents
Give free and SirValUse
scaled feedback Usability
Expert
Survey
Server Retrieves the
participants'
comments and
evaluations
Remote Testing page 4

Remote Testing
Variants of Remote Testing
§ Entirely free and realistic utilisation of the site.
Reported are \"critical incidents\" that …
Critical • … trigger positive or negative sensations.
Incidents
• … prove to be crucial in reaching the target, e.g.
successful use of the search function or failure
due to complicated texts.
§ The respondents are confronted with tasks that
they have to master either with or without a time
limit.
Task-oriented
§ The participants themselves may assess whether they
have completed a task or not; the system is able to
recognise multiple target conditions.
Remote Testing page 5

Remote Testing
What does the respondent see? Critical Incident
Call up feedback Go back to
sessions questionnaire
positive feedback negative feedback end visit
Remote Testing page 6

Remote Testing
What does the respondent see? Critical Incident – positive feedback
You want to give us positive feedback?
Please describe the event or your observation!
What was it that you wanted to do just before the event you want to tell us about?
Please describe short and
precisely in your own words
what you wanted to do.
Example
Please describe the event or the observation and tell us what you liked about it.
Please describe the event or
your observation in a way
that a good friend of yours
would understand it.
Example
Does the event help you in doing what you wanted to do - as described in (1)?
very helpful somehow helpful not helpful at all
How much do you like this event?
very much somehow not at all
cancel submit … and continue surfing.
Remote Testing page 7

Remote Testing
What does the respondent see? Critical Incident – negative feedback
You want to give us positive feedback?
Please describe the event or your observation!
What was it that you wanted to do just before the event you want to tell us about?
Please describe short and
precisely in your own words
what you wanted to do.
Example
What happened? Please describe the event or the observation as detailed as possible.
Please describe the event or
your observation in a way
that a good friend of yours
would understand it.
Example
What were your expectations? Do you have any ideas what we could improve?
What did you expect from
T-Online? What or how
could we do better?
Example
Does the event prevent you in doing what you wanted to do - as described in (1)?
very much somehow not at all
How much do you dislike this event?
very much somehow not at all
cancel submit … and continue surfing.
Remote Testing page 8

Remote Testing
What does the respondent see? Task-oriented
Go back to
Task questionnaire
You're looking for a solution from T-Online in order to protect your PC from viruses, worms or other dangers that may be task done
connected with the Internet. Please look for some information regarding the \"SecurityPackage\".
cancel task
Remote Testing page 9

Remote Testing
Was does the test head see? Screen shot, mouse traces, click points
Remote Testing page 10

Remote Testing
Example 1: Critical Incidents on otto.de
§ How is otto.de \"really\" being used? How do the
respondents proceed while browsing through,
searching or purchasing?
Questions § Which elements or rather functions stimulate or
obstruct purchase processes?
§ Which optimisation measures can be derived from
these insights?
§ 60 respondents were recruited offline.
• They were requested to use otto.de \"in a normal
way\" for 6 weeks, but only via the Remote
system.
Method and
• While surfing they could report \"critical
Implementa- incidents.\"
tion
§ The remote system recorded all of the participants'
actions as well as the pages they visited.
• As a result valuable clues regarding optimisation
measures could be generated.
Remote Testing page 11

Remote Testing
Example 2: Task-oriented benchmarking
§ As a means of further developing the search engine,
one of our clients asked us to conduct a bench-
marking that covered 7 search interfaces:
• Where are the strengths and weaknesses of the
Questions individual searches?
• Which search options are employed for which
search?
• How long does it take to find a result and how
satisfied are the participants with it?
§ Altogether 175 User were recruited via a panel.
§ 75 of them employed 3 search interfaces each
Method and (rotated presentation) through the remote system.
Implementa- • Certain search tasks had to be completed (e.g.
tion \"Please search for a picture of the Cologne
Cathedral\"), while the remote system recorded
the times, the options employed, etc.
Remote Testing page 12

Remote Testing
Example 2: Task-oriented benchmarking: excerpted results
Question: Did you find anything that would meet your requirements?
Search G 94%
Search C 88%
Search F 87%
Search
Success Search A 87%
All Tasks Search B 83%
Search D 82%
Search E 81%
Basis: persons who tested the respective search engine (N=75)
Remote Testing page 13

Remote Testing
Example 2: Task-oriented benchmarking: excerpted results
Average search duration(in seconds)
Search F (N=65)
Frequent Users
Search D (N=62) Customers
Total
Search B (N=62)
Search
Duration Search C (N=66)
All Tasks Search A (N=65)
Search G (N=71)
Search E (N=61)
0
100 20 40 50 60 80 100
0
Basis: persons who successfully searched with the respective search engines
Remote Testing page 14

Remote Testing
Example 2: Task-oriented benchmarking: excerpted results
When was the image search used in searching for a picture?
Search D
Search E Successful Search
Customers
Search C Total
Use of Image Search A
Search
Search G
Search F
Search B
0%
0 20%
20 40%
40 60%
60 80%
80
Basis: persons who tested the respective search engines (N=75)
Remote Testing page 15

Remote Testing
Example 2: Task-oriented benchmarking: excerpted results
How often were search sections employed?
4 - 6 Times
6%
3 Times
15%
Never
Use of 44%
Search
Sections
2 Times
21%
Once
14%
Basis: total, per person during all searches (N=525)
Remote Testing page 16

Remote Testing
Example 3: Click Tracking
Remote Testing page 17

Remote Testing
Example 4: Testing of Variants
§ Four variants of a navigation scheme within an
online shop were to be tested:
• Which navigation scheme leads to satisfactory
results (users find products they would like to
Questions
buy)?
• Which navigation scheme can be used effectively
and efficiently (less clicks, less time, less
problems)?
§ Online recruitment of 1,000 users per variant.
• Users were asked to solve a task (same task for
Method and
each variant)
Implementa-
• Remote tool to measure effectiveness and
tion
efficiency
• Short online questionnaire to assess satisfaction
Remote Testing page 18

Remote Testing
Example 5: Control Quality of Portal
§ One of the biggest internet service providers in
Germany wants to control the quality of their
portals continuously:
• How do users assess content, usability, design
Questions etc. while using the portal?
• Is it possible for users to use the portal efficiently
and effectively?
• How can the \"total user experience\" be improved
so that users intensify their usage?
§ Online recruitment of 2,500 users per month.
• 2,000 users solve 40 tasks to generate valid
measurements.
Method and
• 500 users surf the portal via the \"critical
Implementa-
incident\" technique to generate ideas for
tion
optimization.
• Short reports every months, big reports every six
months.
Remote Testing page 19

Remote Testing
Research Questions
§ Critical incidents:
• Capturing actual user behaviour
• Adding \"reasons\" and \"qualitative feedback\" to your log files
§ Task oriented:
• \"Hard figures\" needed
• Variant testing
• Benchmarking with competitors
§ General:
• Generate log files
• Quality control
• Target groups (e.g. B2B) difficult to recruit
• Target groups geographically wide spread
Remote Testing page 20

Remote Testing
Why Remote Testing?
§ The test setting provides the utmost validity.
§ Many respondents can participate within a short time.
§ Participants stemming from very diverse geographical
Advantages regions can be recruited.
over Lab Tests
§ Studies that require several appointments per
respondent can be realised without much effort.
§ Participants are in their usual surroundings;
they are not bound by fixed test dates.
§ \"This tool made it possible for the first time to
conduct a test directly in the living room of
Otto.de users, i.e. in the environment in which
Customer
buying experiences are created and buying
Feedback
decisions are made.\"
(Steffen Kehr, Customer Experience Management,
www.otto.de)
Remote Testing page 21

Remote Testing
Does Remote Testing have any disadvantages over lab tests?
§ Motivations and problems cannot be explored in
depth.
§ No opportunity to boost specific feedback.
Disadvantages
over Lab Tests § Sessions cannot be adapted to the wishes and
capacity of the participants during an interview.
§ No video tapes of the respondents' facial expressions
or gestures.
§ Remote Testing is complementary to qualitative
sessions.
Remote
Testing and § Problems identified by means of Remote Testing can
Lab Tests: be thoroughly investigated during a lab test.
a Useful
Combination § Problems identified in lab tests or by expert reviews
assist in further interpreting the results of task-based
Remote Tests.
Remote Testing page 22

Remote Testing
Automated Remote-Testing: User Experience Scorecard
Remote Testing page 23

Remote Testing
Automated Remote-Testing: User Experience Scorecard
Remote Testing page 24

Thank you for your attention!
SirValUse Consulting GmbH
Schlossstraße 8g
22041 Hamburg
Phone: +49 40 68 28 27 - 20
Fax: +49 40 68 28 27 - 20
www.sirvaluse.de
info@sirvaluse.de
Remote Testing page 25