Your SlideShare is downloading. ×
Remote Usability Testing
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Remote Usability Testing

873

Published on

Prague ACM SIGCHI presentation from 6/26/2006 about Remote usability testing by Karsten Skuppin

Prague ACM SIGCHI presentation from 6/26/2006 about Remote usability testing by Karsten Skuppin

Published in: Design, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
873
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
45
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Remote Testing Prague June 2006
  • 2. Remote Testing quot;Historyquot; (I) § Usability testing for Otto • Only specific areas / processes in the shop • Not the quot;whole customer experiencequot; § Development of quot;seasonal testingsquot; • Recruit participants that are about to buy something on otto.de • Observe them just doing that in the labs § Open issues • Testing is still not quot;realquot; enough - Lab situation - Perform tasks with possibly low involvement - Not all relevant tasks in the scope of the lab test Remote Testing page 2
  • 3. Remote Testing quot;Historyquot; (II) § Idea: Remote-Testing • At home (quot;naturalquot; environment) • Real involvement • quot;Naturalquot; tasks (just what people want to do) § Challenges • Tool must work without installation on PCs of users • Tool must work without having to change anything on the server of the client • We must be able to quot;observequot; and quot;measurequot; (not only quot;askquot;) Remote Testing page 3
  • 4. Remote Testing How does Remote Testing work? Participants Proxy Server Web Site Natural use of the target Passes inquiries on to web site Recalls all recorded actions of the respondents Give free and SirValUse scaled feedback Usability Expert Survey Server Retrieves the participants' comments and evaluations Remote Testing page 4
  • 5. Remote Testing Variants of Remote Testing § Entirely free and realistic utilisation of the site. Reported are quot;critical incidentsquot; that … Critical • … trigger positive or negative sensations. Incidents • … prove to be crucial in reaching the target, e.g. successful use of the search function or failure due to complicated texts. § The respondents are confronted with tasks that they have to master either with or without a time limit. Task-oriented § The participants themselves may assess whether they have completed a task or not; the system is able to recognise multiple target conditions. Remote Testing page 5
  • 6. Remote Testing What does the respondent see? Critical Incident Call up feedback Go back to sessions questionnaire positive feedback negative feedback end visit Remote Testing page 6
  • 7. Remote Testing What does the respondent see? Critical Incident – positive feedback You want to give us positive feedback? Please describe the event or your observation! What was it that you wanted to do just before the event you want to tell us about? Please describe short and precisely in your own words what you wanted to do. Example Please describe the event or the observation and tell us what you liked about it. Please describe the event or your observation in a way that a good friend of yours would understand it. Example Does the event help you in doing what you wanted to do - as described in (1)? very helpful somehow helpful not helpful at all How much do you like this event? very much somehow not at all cancel submit … and continue surfing. Remote Testing page 7
  • 8. Remote Testing What does the respondent see? Critical Incident – negative feedback You want to give us positive feedback? Please describe the event or your observation! What was it that you wanted to do just before the event you want to tell us about? Please describe short and precisely in your own words what you wanted to do. Example What happened? Please describe the event or the observation as detailed as possible. Please describe the event or your observation in a way that a good friend of yours would understand it. Example What were your expectations? Do you have any ideas what we could improve? What did you expect from T-Online? What or how could we do better? Example Does the event prevent you in doing what you wanted to do - as described in (1)? very much somehow not at all How much do you dislike this event? very much somehow not at all cancel submit … and continue surfing. Remote Testing page 8
  • 9. Remote Testing What does the respondent see? Task-oriented Go back to Task questionnaire You're looking for a solution from T-Online in order to protect your PC from viruses, worms or other dangers that may be task done connected with the Internet. Please look for some information regarding the quot;SecurityPackagequot;. cancel task Remote Testing page 9
  • 10. Remote Testing Was does the test head see? Screen shot, mouse traces, click points Remote Testing page 10
  • 11. Remote Testing Example 1: Critical Incidents on otto.de § How is otto.de quot;reallyquot; being used? How do the respondents proceed while browsing through, searching or purchasing? Questions § Which elements or rather functions stimulate or obstruct purchase processes? § Which optimisation measures can be derived from these insights? § 60 respondents were recruited offline. • They were requested to use otto.de quot;in a normal wayquot; for 6 weeks, but only via the Remote system. Method and • While surfing they could report quot;critical Implementa- incidents.quot; tion § The remote system recorded all of the participants' actions as well as the pages they visited. • As a result valuable clues regarding optimisation measures could be generated. Remote Testing page 11
  • 12. Remote Testing Example 2: Task-oriented benchmarking § As a means of further developing the search engine, one of our clients asked us to conduct a bench- marking that covered 7 search interfaces: • Where are the strengths and weaknesses of the Questions individual searches? • Which search options are employed for which search? • How long does it take to find a result and how satisfied are the participants with it? § Altogether 175 User were recruited via a panel. § 75 of them employed 3 search interfaces each Method and (rotated presentation) through the remote system. Implementa- • Certain search tasks had to be completed (e.g. tion quot;Please search for a picture of the Cologne Cathedralquot;), while the remote system recorded the times, the options employed, etc. Remote Testing page 12
  • 13. Remote Testing Example 2: Task-oriented benchmarking: excerpted results Question: Did you find anything that would meet your requirements? Search G 94% Search C 88% Search F 87% Search Success Search A 87% All Tasks Search B 83% Search D 82% Search E 81% Basis: persons who tested the respective search engine (N=75) Remote Testing page 13
  • 14. Remote Testing Example 2: Task-oriented benchmarking: excerpted results Average search duration(in seconds) Search F (N=65) Frequent Users Search D (N=62) Customers Total Search B (N=62) Search Duration Search C (N=66) All Tasks Search A (N=65) Search G (N=71) Search E (N=61) 0 100 20 40 50 60 80 100 0 Basis: persons who successfully searched with the respective search engines Remote Testing page 14
  • 15. Remote Testing Example 2: Task-oriented benchmarking: excerpted results When was the image search used in searching for a picture? Search D Search E Successful Search Customers Search C Total Use of Image Search A Search Search G Search F Search B 0% 0 20% 20 40% 40 60% 60 80% 80 Basis: persons who tested the respective search engines (N=75) Remote Testing page 15
  • 16. Remote Testing Example 2: Task-oriented benchmarking: excerpted results How often were search sections employed? 4 - 6 Times 6% 3 Times 15% Never Use of 44% Search Sections 2 Times 21% Once 14% Basis: total, per person during all searches (N=525) Remote Testing page 16
  • 17. Remote Testing Example 3: Click Tracking Remote Testing page 17
  • 18. Remote Testing Example 4: Testing of Variants § Four variants of a navigation scheme within an online shop were to be tested: • Which navigation scheme leads to satisfactory results (users find products they would like to Questions buy)? • Which navigation scheme can be used effectively and efficiently (less clicks, less time, less problems)? § Online recruitment of 1,000 users per variant. • Users were asked to solve a task (same task for Method and each variant) Implementa- • Remote tool to measure effectiveness and tion efficiency • Short online questionnaire to assess satisfaction Remote Testing page 18
  • 19. Remote Testing Example 5: Control Quality of Portal § One of the biggest internet service providers in Germany wants to control the quality of their portals continuously: • How do users assess content, usability, design Questions etc. while using the portal? • Is it possible for users to use the portal efficiently and effectively? • How can the quot;total user experiencequot; be improved so that users intensify their usage? § Online recruitment of 2,500 users per month. • 2,000 users solve 40 tasks to generate valid measurements. Method and • 500 users surf the portal via the quot;critical Implementa- incidentquot; technique to generate ideas for tion optimization. • Short reports every months, big reports every six months. Remote Testing page 19
  • 20. Remote Testing Research Questions § Critical incidents: • Capturing actual user behaviour • Adding quot;reasonsquot; and quot;qualitative feedbackquot; to your log files § Task oriented: • quot;Hard figuresquot; needed • Variant testing • Benchmarking with competitors § General: • Generate log files • Quality control • Target groups (e.g. B2B) difficult to recruit • Target groups geographically wide spread Remote Testing page 20
  • 21. Remote Testing Why Remote Testing? § The test setting provides the utmost validity. § Many respondents can participate within a short time. § Participants stemming from very diverse geographical Advantages regions can be recruited. over Lab Tests § Studies that require several appointments per respondent can be realised without much effort. § Participants are in their usual surroundings; they are not bound by fixed test dates. § quot;This tool made it possible for the first time to conduct a test directly in the living room of Otto.de users, i.e. in the environment in which Customer buying experiences are created and buying Feedback decisions are made.quot; (Steffen Kehr, Customer Experience Management, www.otto.de) Remote Testing page 21
  • 22. Remote Testing Does Remote Testing have any disadvantages over lab tests? § Motivations and problems cannot be explored in depth. § No opportunity to boost specific feedback. Disadvantages over Lab Tests § Sessions cannot be adapted to the wishes and capacity of the participants during an interview. § No video tapes of the respondents' facial expressions or gestures. § Remote Testing is complementary to qualitative sessions. Remote Testing and § Problems identified by means of Remote Testing can Lab Tests: be thoroughly investigated during a lab test. a Useful Combination § Problems identified in lab tests or by expert reviews assist in further interpreting the results of task-based Remote Tests. Remote Testing page 22
  • 23. Remote Testing Automated Remote-Testing: User Experience Scorecard Remote Testing page 23
  • 24. Remote Testing Automated Remote-Testing: User Experience Scorecard Remote Testing page 24
  • 25. Thank you for your attention! SirValUse Consulting GmbH Schlossstraße 8g 22041 Hamburg Phone: +49 40 68 28 27 - 20 Fax: +49 40 68 28 27 - 20 www.sirvaluse.de info@sirvaluse.de Remote Testing page 25

×