Measuring UX by Tim Bosenick, SirValUse

Measuring User Experience Combining the “Qualitative” with the “Quantitative” Some Examples from the Web Tim Bosenick, SirValUse Consulting GmbH Bild

What do I know about my site and the users? I have some data of Sometimes a users given during apply surveys I know that some order. on my site. users order. I have a Web Analytics System I know the and collect loads serverlogs, PIs and of data. sessions. I run regularely But... qualitative User Experience Tests.

But still there are many questions… Who are my most important Are my competitos competitors? better? Why? Do I reach my (scope, CR, ..) target groups? How does the Who exactly are my business users? Do they act competition effectively on my develop? website? Where is my target group on Does my marketing the way in internet? reach the people I want Where can I appeal to them? to reach?

Conclusion: User experience, usability and marketing (the whole “experience chain”) come closer together We need a 360 degree view on our costumers We often know the “what happens”, we sometimes know the “why” – a combination would be great

The Idea WebWerte 5

Combining qualitative and quantitative observational and survey data. Further requirements: Quantitative Qualitative » Single source data collection for high validity Detailed Observation User Tracking analysis of » Scalable to large user sessions numbers of participants » Possibility of true- intent and task-based Standardised Free Survey Data user experience tests, Surveys User Feedback also experiments » Measurement without cooperation of website for one specific website, owners or publishers site centric, short time frame (only Add-On)

Whole internet, user centric Features of the LEOtrace® Browser Add-Ons Features: Deliverable data: » Single source measurement of reactive » Non-reactive data ("Tracking") and non-reactive data » Session information (e.g. duration) » Visited URL’s » Data collection without cooperation of » Search queries (Google, ...) website owners or publishers » Precise ad impressions » Allows data collection on third party sites » Screenshots » Flexible setup » Clicks (and mouse movements) » Customising of design and branding » Reactive data » Remote control of all functions » Standardised surveys (also event- » Extended research designs possible: triggered) with any survey software » Task-based designs » Free ad-hoc feedback » Experimental designs (manipulation of » Client information client-side HTML code) » Operating system, web browser, … » Data collection in SSL-encrypted areas » User structure data provided from possible (usually disabled) access panel

Whole internet, user centric Installation process Download 1 Add-On Optional: Identification on every browser start 4 Install 2 Add-On Easy setup – just like any other Optional: browser add-on 3 Registration on first browser start

Specific website, site centric The user are invited on your website … » To elevate the problems occuring while using the website, the LEOtrace® Remote Test has been developed: Preliminary interview Follow-Up Interview Invitation by a layer Remote Session Online questionnaire Online questionnaire • "What is your today main • "Please surf the website as you • "Did you achieve everything reason of visit?" would do it normally." today you planned to?" • "Please evaluate the website on the basis of following • User-Behaviour items." • Clickway- Analysis • User-Feedback … and use the website according to your natural performance.

Specific website, site centric The Technology Participants Website 2 1 Utility target-website Content enquieries Proxy-Server through Give free and scaled 3 feedback Recalls monitored actions of participants Usability Expert Surveyserver Recalls comments and evaluations of participants Participants and website-providers don´t need to install software . Only the Invitationlayer has to be linked. Adjustments have only to be neccessary in the LEOtrace®-System itself

Specific website, site centric The Feedback-Bar Call up feedback- To the finaly survey sessions Via feedback-buttons the users are always able to call up a short survey or to give a positive or negative feedback .

Types of Studies WebWerte 12

Typical Studies Tasked based Benchmarking 2 3 Initial survey Task (e.g. favourite search with detailed recording of engines, …) user behaviour (clicks, screenshots) as well as free feedback and event-triggered questionnaires 1 4 Recruitment Final survey from online access panel (e.g. rating of websites used); (screening according to Deinstallation of Add-On target group criteria); Installation of LEOtrace® Browser-Add-On

Typical Studies Tasked based Benchmarking & Usage Monitoring 2 3 Initial survey Task (e.g. favourite search with detailed recording of engines, …) user behaviour (clicks, 3a screenshots) as well as free feedback and event-triggered questionnaires Non-reactive monitoring of online usage during several weeks (before receiving task) 1 4 Recruitment Final survey from online access panel (e.g. rating of websites used); (screening according to Deinstallation of Add-On target group criteria); Installation of LEOtrace® Browser-Add-On

Typical Studies Digital Behaviour Studies 2 Initial survey (e.g. favourite search engines, …) 3 Non-reactive monitoring of online usage during several weeks (before receiving task) 1 4 Recruitment Final survey from online access panel (e.g. rating of websites used); (screening according to Deinstallation of Add-On target group criteria); Installation of LEOtrace® Browser-Add-On

Typical Studies Digital Behavior Studies with Event based Surveys 2 Initial survey Event based surveys, (e.g. favourite search e.g. having used a engines, …) certain function, 3 leaving a certain website, leaving a certain page Non-reactive monitoring of online usage during several weeks (before receiving task) 1 4 Recruitment Final survey from online access panel (e.g. rating of websites used); (screening according to Deinstallation of Add-On target group criteria); Installation of LEOtrace® Browser-Add-On

Typical Studies Web Efficiency Panel Generating single Fusion & source consumer & Browser Add-On sends analysis Panellist Internet usage data:  URLs  Ad contacts scans  Search queries purchases » Purchases (Consumer Scan) » Internet usage Internet user buys online  PageImpressions Internet user Internet user buys offline  Visits in WEP Offliner  AdImpressions buys offline  AdClicks  Queries with relevant search engines » Structural data Offliner in GfK Consumer Scan panel

Typical Studies Industry Benchmarking » We use the data from the Web Efficiency Panel to calculate benchmarking KPIs for certain industries (e.g. e-commerce, automobile, …) » Possible analyses:  Target group  Cross usage  Previous and following websites  Usage of functions / areas  Conversion rates  …

Typical Studies True Intent Experience Reports 2 Initial survey (e.g. reason for the visit, …) 3 Non-reactive monitoring of website usage during this session optional: free feedback 1 4 Recruitment Final survey online (real users) (e.g. rating of website, (screening according to satisfaction, reason for target group criteria); leaving, …) no installation needed

The Importance of Measurements

Quantitative studies are an ideal enhancement of qualitative studies Qualitative Approach Quantitative Approach Focus on “usability” Focus on “user experience” Derive recommendations Measure key performance indicators Focus on product management / UX Focus on market research / general departments management Sometimes “artificial” lab situation, “True intent” studies possible, no “real” behavior can be observed observation of “real behavior” Combination with qualitative Combination with marketing KPIs marketing research possible possible Testing of “offline” applications Test object must be “online” possible

Why is quantitative measurement important? » Users tend to misjudge their behaviour  Social request, known brands are over-rated, problem to correctly quantify their own behavior  Example: User were asked: What kind of websites did you use while looking for a product? Answer: 88% Google. In the actual measurement phase, only 68% really used Google. » “User Experience” gets more and more important  “Usability” still is an important factor of the overall user experience, but e.g. “joy of use” and “design” come into the focus of testing – also in the early development phase.  With this, classical market research methodologies and “real measurement” become central for the usability testing practice – or else market research companies will gain bigger parts of the market. » “Usability” and Marketing come closer together  It is nowadays not only important to get as much traffic as possible to a website (online marketing) and to optimize the conversion rate (usability), it is also important to combine both views so that the whole shopper experience can be optimized.

Automobile Industry Monitor 2009 WebWerte 23

Methods To satisfactorily address the questions at hand, we have employed all of the following methods: » Permanent data collection regarding all surfing behaviour by means of the Web Efficiency Panel (WEP) designed by GfK during Q1 2009 (01/01 - 31/03). Internet Tracking » Quantitative evaluation of internet use with regard to relevant automotive websites. » Determination of indicators and modelling of navigation behaviour. » Surveys completed by the WEP panelists regarding the Surveys topics automotive ownership, automotive purchase, brand affinity and advertising awareness. » Expert analysis of the most important manufacturer Market Analysis websites with regard to user experience.  Identification of best practices.

In Q1 2009, one-fifth of internet users visited at least one automotive manufacturer website. Internet users in Germany: 42,540,000 Total users of "Auto-Websites": 16,380,824 Users of manufacturers' websites: 8,308,536 Users of sales portals: 7,692,277 Users info portals: 5,423,271 Users of auto club websites: 4,153,947 Users of online community websites: 2,678,939 Users of media websites: 2,352,151 Users of auto group websites: 223,034 1% Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009

In the first quarter, manufacturer websites were visited around 41 million times, and on average, 9.5 PIs were generated. 9.5 11.1 5.0 8.9 5.9 3.8 4.6 PIs per Visit (in mil.) Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009

The VW website has by far the most users, followed by Opel, Ford and Audi. Users of manufacturer websites (in thousands) Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009 Page 1 of 2

The Mercedes and VW websites have the highest rate of repeat visits. Mercedes 1.9 VW Opel Audi Skoda Fiat Citroën Toyota Mitsubishi Mazda Seat Porsche Visits per user of the given Renault manufacturer websites (mean) Jaguar Suzuki Smart Honda Dacia Ford Peugeot BMW Nissan Alfa Romeo Daihatsu Hyundai Subaru Kia Volvo Data base: Internet users in Chevrolet WEP designed by GfK between 01/01.–31/03/2009

Overview of the most significant indicators for manufacturer websites User Proportion User Proportion Proportion of User Proportion User Proportion User Proportion Unique Users PIs per Visits per for for Showroom ended for Financing Financing for Dealer (in thousands) Visit User Configurator use configurations pages Calculator Search use Mean Value* 899 9.6 2.2 52% 28% 36% 11% 6% 13% Audi 1,051 10.6 2.3 52% 23% 45% 1% 1% 0% BMW 615 6.1 1.5 43% 26% n. a. 5% 5% 4% Chevrolet 306 15.7 1.3 70% 23% 37% 14% n. a. 16% Citroën 692 9.0 2.0 62% 29% 59% 8% 0% 8% Dacia 756 9.3 1.6 89% n. a. n. a. 29% n. a. 31% Fiat 545 14.9 2.1 40% 45% 45% 0% 13% 13% Ford 1,053 6.4 1.6 51% 27% n. a. 9% 3% 20% Honda 895 6.4 1.6 24% 12% 50% 4% 0% 11% Mercedes 771 7.8 5.0 39% 24% 24% 5% 8% 31% Mitsubishi 274 5.7 2.0 54% 27% 6% 0% 6% 18% Opel 1,109 16.6 2.5 59% 48% 50% 3% 16% 10% Peugeot 923 5.0 1.5 37% 14% n. a. 27% n. a. 11% Renault 882 9.4 1.8 61% 43% 27% 3% 1% 10% Skoda 722 14.6 2.2 40% 36% n. a. 4% 5% 4% Toyota 860 6.5 2.0 75% 24% n. a. 10% 12% 16% VW 2,933 9.2 3.7 34% 20% 12% 59% 2% 5% * Refers to the 16 manufacturers represented Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009 Info: The best three providers are marked in green, the worst three in red.

Showrooms, configurators and information about financing are of particular interest to users who intend to purchase – the dealer search is somewhat less used by those interested in purchasing. Info: In the calculation of the mean, only those manufacturer websites are represented for which use data are available for the given area. Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009

Fewer than half of manufacturer website users visit showrooms – if one is visited, then others are likely to be also. Showroom users (Q1): 3,680,249 1 Showroom visited: 1,373,005 2 Showrooms visited: 626,044 3 Showrooms visited: 438,454 4 Showrooms visited: 330,935 5 Showrooms visited: 190,848 More than 5 Showrooms visited: 720,963 Number of showrooms visited for all users of manufacturer websites (mean): 1.0 Number of showrooms visited for all users of showrooms (mean): 4.2 Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009

Overview of the most significant indicators for manufacturer websites User Proportion User Proportion Proportion of User Proportion User Proportion User Proportion Unique Users PIs per Visits per for for Showroom ended for Financing Financing for Dealer (in thousands) Visit User Configurator use configurations pages Calculator Search use Mean Value* 899 9.6 2.2 52% 28% 36% 11% 6% 13% Audi 1,051 10.6 2.3 52% 23% 45% 1% 1% 0% BMW 615 6.1 1.5 43% 26% n. a. 5% 5% 4% Chevrolet 306 15.7 1.3 70% 23% 37% 14% n. a. 16% Citroën 692 9.0 2.0 62% 29% 59% 8% 0% 8% Dacia 756 9.3 1.6 89% n. a. n. a. 29% n. a. 31% Fiat 545 14.9 2.1 40% 45% 45% 0% 13% 13% Ford 1,053 6.4 1.6 51% 27% n. a. 9% 3% 20% Honda 895 6.4 1.6 24% 12% 50% 4% 0% 11% Mercedes 771 7.8 5.0 39% 24% 24% 5% 8% 31% Mitsubishi 274 5.7 2.0 54% 27% 6% 0% 6% 18% Opel 1,109 16.6 2.5 59% 48% 50% 3% 16% 10% Peugeot 923 5.0 1.5 37% 14% n. a. 27% n. a. 11% Renault 882 9.4 1.8 61% 43% 27% 3% 1% 10% Skoda 722 14.6 2.2 40% 36% n. a. 4% 5% 4% Toyota 860 6.5 2.0 75% 24% n. a. 10% 12% 16% VW 2,933 9.2 3.7 34% 20% 12% 59% 2% 5% * Refers to the 16 manufacturers represented Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009 Info: The best three providers are marked in green, the worst three in red.

The showrooms on the websites of Dacia, Toyota and Chevrolet are visited especially often. Number of users of a website who visited the showroom on that website. Proportion of total users who visited the showroom on any 52% manufacturer website (mean): Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009

Images of the models displayed on the start page spur users on to visit the showrooms. » The first-ranked websites of Dacia, Toyota and Chevrolet feature links to the showroom directly on the start page, including images of the models. Toyota Dacia » Websites that only offer a text link (e.g. Peugeot, VW) create less interest for showrooms.  If the links to the models are not prominently placed, the showrooms are used significantly less often (see Honda). Honda Showrooms with a high degree of multimedia concentration often may not exploit their potential because they are less able to be located.

eCommerce Branchenmonitor 2008 WebWerte 35

Analysis question » Why does a website sells good – and better than the competition? » Why do customers decide for one special online-shop? » Where (also online/offline) and how do customers inform themselves? » Which products are bought out of what reasons? Focus: Consuming goods in low to middle pricesegment, which are bought regularly

Method- Mixing 1. LEOtrace® Behaviour Tracking of online-shopper » Multi-Method Study for wholistic mapping and analysis of the onlineshopping-process, from wording the shoppingintention until delivery. » Monitoring of users during productresearch, measuring of detailled data- characteristics and different interviews before, during and after the monitoring phase. 2. Expert assessment by the SirValUse E-Commerce-Team » Focus: Best Practice-Analysis.

Fact sheet LEOtrace® Behaviour Tracking » Several weeks long Add-On-Study for Internet Explorer and Method Firefox. » 440 Users were recruited via an Online-Access-Panel for the study. – All participants planned to purchase online-products during Sample field time. – 59% Women, 41% Men. – 52% under 30 years, 26% 30-39 years, 22% 40 years & older. » Testing of all websites which could be important for research- and shoppingprocess. Tested websites – Focus on onlineshops. – Furthermore: price comparing-websites, searchengines and portals, manufacturer-websites and community-websites. Field time » From Juli 1st until September 8th 2008.

Expiry of Behaviour Trackings for Users Final survey: First survey: Field time:  Afterexploratio  Screening  Handle the n  Installation of AddOn shoppinglist  Imagemeasure  Questions to online-  Monitoring of ment of shopping surf behaviour onlineshops  Event-Interviews  Deinstallation of AddOn 2 weeks 7 weeks 1 week

Technologie Browser Extension LEOtrace® AddOn

The Shoppinglist » Origin- and endpoint of all productresearch. » The Users were told to keep following aspects up to date: – Productcategory – Productname – (contemplated) price of product – State of research » Furthermore they were told before a research session, what product had to been researched.

Devolution of a research-session Eventinterview "general" Eventinterview "detailled product website Shoppinglist: Shoppinglist: Post survey: Instruction for If so matching of Research-session Evaluation of product to be price and research-session researched for research-status Eventinterview "searchfunction Eventinterview "ordering process"

Example: Research- & Shoppingsession (1) Product for which exists a buying interest: Nintendo Wii Research-sessions:  July 20th (21:12 - 21:57) User: 244756  July 21st (18:20 - 18:41) male, 29 years On July 20th following websites were selected:  Geizkragen.de  Search for "Nintendo Wii"  cyberport.com  discount24.de  amazon.de

Example: Research- & Shoppingsession (2) On July 21st amazon.de was accessed:  search for "Wii"  Game console put into shopping basket  Search for "Mark Medlock"  CD "Cloud Dancer" by Mark Medlock put into shopping basket User: 244756 male, 29 years Analysis of research: Both products were purchased online at amazon.de Comment about order: "Great that I could bought the CD here as well"

Research-Status at end of field time Cases Cases Quota per user Still in research-phase 800 36% 1,8 Bought online 920 42% 2,1 Bought offline 189 9% 0,4 No shopping interest 289 13% 0,7 anymore Total 2.198 5,0 Basis: Total (N=440).

Used Onlineshops during field time Basic: Total (N=440).

Rating of Onlineshops Amazon (N=342) eBay (N=281) Bonprix (N=50) Weltbild (N=57) Neckermann (N=53) Tchibo (N=65) Question: How helpful were following Onlineshops during research and purchase of the varied Buecher.de (N=46) products? Esprit (N=35) Basic = User of each website Rating on a scale of 6: Ikea (N=33) 1=Very helpful 6=Not helpful at all. Quelle (N=80) Only a section of the answering scale is shown . Otto (N=81) Buch.de (N=42) H&M (N=36)

Used shops per product following productcategories Videogames 2,3 Tickets Sports goods / -tools CDs / DVDs housewares / -tools Consumer electronics Fashion / Shoes Toys Jewellery/Watches/Accessoiries Fitments Perfume / Cosmetiques Telecommunicationproducts Healthproducts Books Computerhardware / -equipment Other products Basic: products being researched for (N=1.766).

Rating of experiences during research-phase How do you rate the experiences you made during the last 6 weeks according to research and shopping in internet? Incomplete delivery Couldn´t find anything fitting Unsufficient productdescription/-mapping No wanted payment options no response on E-Mail-request Complicated navigation Und Delivery problems pricing not transparent zwar: Product not available unsufficient searchfunction Oversupply Speedtransfer Too expensive Tag Cloud View Basic: Total (N=440).

True Intent Study WebWerte 50

Area: Shop – 'Handys & Shop' Success in percent successful not successful Buy something (N=50) 37 63 Buy something online (N=50) 52 48 Question 10040: Did you succeed in what you wanted to do on the website today?

Area: Shop – 'Handys & Shop' Structure of navigation in percent "Tarife & Optionen" * 10 30 8 28 12 6 8 Handset "Handys & 9 "Aktions- "Vertrags- 24 details Datengeräte" Angebote" Verlängerung" 24 5 Login area "Prepaid- 27 ("https") Angebote" * First page of the area (Overview page) Navigation Cancel of remote-session Results generated from log data. Participants who started from the page "Handys & Shop": N=100. Only navigation steps with N > 1.

Area: Shop – 'Handys & Shop' Structure of navigation – Buyers in percent "Handys & 11 Shop" * 37 23 Homepage 25 "Handys & "Aktions- 18 18 Datengeräte" angebote" 9 Price 35 pop-up 20 Handset Login-area 20 20 details ("https") 20 23 * First page of the area (Overview page) Navigation Cancel of visit Results generated from log data. Participants with buying intention: N=100. Only navigation steps with N > 3.

Area: Shop – 'Handys & Shop' General  The homepage of the section 'Handys & Shop' was praised as clearly arranged and pleasantly designed.  The range of devices in 'Handys & Shop' was well liked.  The choice of mobile phones was stated as complete and up-to-date.  Particularly the phone offers for 1€ were liked by the participants.  "Great, that there are mobile phones for one Euro, which look o.k. and have many features."  If the preferred phone model was not offered on the website, the range of products was often criticized as too small.  In this case the choice was also described as out of date.  Participants always expect that the latest models are available on the website.  "I searched for the Samsung E 840, thinking that XXX as market leader always offers the most up-to-date phone models, even before other providers offer them."  " 'Top-Angebote' (top-offers) are on offer - latest mobile phones are missing."

Area: Shop – 'Handys & Shop' Mobile phones detail-page ? (radio)  Understanding the meaning of the several "function-icons" was a problem.  "Icons completely inoperative!!! They don't work for a comparison and are not understandable without explanations. Very, very bad."  The icons did not suggest that further explanations were available and participants could therefore not find them.  The offered alt-tags were often not clearly understandable (e.g. Voice, Orga).  The explanations were sometimes very long and it was necessary to read carefully through them in order to understand which function was described.  Most users only wanted to know which function was included, not its purpose.

Time for Questions WebWerte 56

www.sirvaluse.de SirValUse Consulting GmbH | Schlossstraße 8g | 22041 Hamburg Tel. +49 (40) 68 28 27-0 | Fax +49 (40) 68 28 27-20 WebWerte

http://marathon.uidesign.ru	173
http://marafon.uidesign.ru	69
http://www.scoop.it	27
http://www.webvalueindex.de	15
http://www.slideshare.net	3
http://www.marathon.uidesign.ru	2
http://webvalueindex.de	1

Measuring UX by Tim Bosenick, SirValUse

by UIDesign Group on Oct 26, 2009

Accessibility

Tags

Upload Details

Usage Rights

7 Embeds 290

Statistics

Measuring UX by Tim Bosenick, SirValUse — Presentation Transcript