Measurement options,
measurement error, and usability
in mobile surveys

In cooperation with:




Dr. Tanja Pferdekämper, ...
Agenda



1   Current Situation and Objectives


2   Subject of Research Study


3   Fact Sheet of Research Study


4   Fi...
Agenda



1   Current Situation and Objectives


2   Subject of Research Study


3   Fact Sheet of Research Study


4   Fi...
Current Situation and Objectives
Percentage of mobile users in county




                                       4
Current Situation and Objectives
Market Research Challenge

      Mobility & Flexibility
      - Instant Feedback at the P...
Agenda



1   Current Situation and Objectives


2   Subject of Research Study


3   Fact Sheet of Research Study


4   Fi...
Research Questions:



                      How is Content
                                                   ?
    ?    ...
Study Focus


Mobile Surveys here are defined as:
Self-administered surveys with mobile devices

       Not topic of the s...
Agenda



1   Current Situation and Objectives


2   Subject of Research Study


3   Fact Sheet of Research Study


4   Fi...
Research Summary:
Joint project by Globalpark AG, FU Bozen & YOC AG

                      Two survey waves to analyse res...
Display of different question types
We used 5 prototypical mobile question types in the survey:
                        Ho...
Display of different question types
We used 5 prototypical mobile question types in the survey:
                          ...
Agenda



1   Current Situation and Objectives


2   Subject of Research Study


3   Fact Sheet of Research Study


4   Fi...
Technical Requirements and Restrictions
Input devices and types of connections (Own Study: Germany)


                    ...
Technical Requirements and Restrictions
Traffic, Manufacturers and Devices (Benchmark: Western Europe)


                 ...
Display of different question types
Drop-out Rates – objective Measure
                                                   ...
Scrolling 1 (2)
                                                             “While answering, I
Significant Difference be...
Scrolling 2 (2)
                                                             “While answering, I
Significant Difference be...
Ease of answering
                                                                  “With this question
Significant Differ...
Unintentional input
Significant Difference between                                      “During completion,
Mean Values on...
Fluent answering
                                                                           “I could use this
Significant ...
Ease of Use
                                                                “This question type
Significant Difference bet...
Display of different question types

    During the post-survey every respondent was asked about the 5
    question types....
Display of different question types:
Fit for Purpose - Summary


       Overall, the difference in the subjective user eva...
Summary
    Technical Requirements & Restrictions
    - Efficiency of Design- and Implementation Technology for Mobile
   ...
Outlook

     Future research topics:
1    Robustness of results for a variety of topics?

     Influence of participation...
Thank you!


Visit our mobile survey with your cell phone at:

Mobile.opst.de/uc/main/157f
There you can directly particip...
Globalpark mit Sitz bei Köln, in London, New York und Wien gehört zu den weltweit führenden Anbietern von
                ...
Nutzungshinweise


© 2009 – Globalpark AG, Globalpark UK Ltd., Globalpark USA, Globalpark Österreich GmbH


Die in dieser ...
Upcoming SlideShare
Loading in …5
×

Measurement options, measurement error, and usability in mobile surveys

1,331 views
1,240 views

Published on

Measurement options, measurement error, and usability in mobile surveys. Presentation held at General Online Research 2009 (GOR09) in Vienna

Published in: Business, Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,331
On SlideShare
0
From Embeds
0
Number of Embeds
26
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Measurement options, measurement error, and usability in mobile surveys

  1. 1. Measurement options, measurement error, and usability in mobile surveys In cooperation with: Dr. Tanja Pferdekämper, Globalpark AG Prof. Dr. Michael Bosnjak, Free University of Bozen-Bolzabo 1 Gottfried Metzger, University of Mannheim
  2. 2. Agenda 1 Current Situation and Objectives 2 Subject of Research Study 3 Fact Sheet of Research Study 4 Findings 5 Summary & Outlook 2
  3. 3. Agenda 1 Current Situation and Objectives 2 Subject of Research Study 3 Fact Sheet of Research Study 4 Findings 5 Summary & Outlook 3
  4. 4. Current Situation and Objectives Percentage of mobile users in county 4
  5. 5. Current Situation and Objectives Market Research Challenge Mobility & Flexibility - Instant Feedback at the Point of Sale - Reaching high mobility target group - Location independent surveys Speed - Contemporary, event triggered surveys - Fast responses Reporting - Results available in real-time - Faster decision making/evaluation 5
  6. 6. Agenda 1 Current Situation and Objectives 2 Subject of Research Study 3 Fact Sheet of Research Study 4 Findings 5 Summary & Outlook 6
  7. 7. Research Questions: How is Content ? ? displayed? ? ? How is the usability of ? different question types ? ? rated? ? ? ? ? ? ? 7
  8. 8. Study Focus Mobile Surveys here are defined as: Self-administered surveys with mobile devices Not topic of the study: - Types of interviewer-administered surveys - PDA-based and similar surveys with Interviewer - CATI-Surveys conducted with cell phone users
  9. 9. Agenda 1 Current Situation and Objectives 2 Subject of Research Study 3 Fact Sheet of Research Study 4 Findings 5 Summary & Outlook 9
  10. 10. Research Summary: Joint project by Globalpark AG, FU Bozen & YOC AG Two survey waves to analyse respondent intention , behavior, usability and the acceptance of mobile Aim surveys 2 surveys from 18.08. – 01.09.2008 Field work 1. Mobile survey Olympia 2008 (YOC Mobile-Panel, N = 171) Sample source and 2. Post web based survey to evaluate usability from a content user perspective (YOC Mobile-Panel, N = 413) Had not taken part in mobile survey before; 33.9% had Sample used mobile internet over a year; 49.7% had never accessed the internet with their mobile phone 10
  11. 11. Display of different question types We used 5 prototypical mobile question types in the survey: How interested are Did you watch the Where do you look you in the Olympic 2008 Olympic up information & Summer Games of Opening Ceremony results? 2008? on TV? Single response vertical Multi response vertical Question with image 11
  12. 12. Display of different question types We used 5 prototypical mobile question types in the survey: Which of the 28 How many hours sporting disciplines will you watch the interests you Olympic most? competitions? Single line text input Closed response list 12
  13. 13. Agenda 1 Current Situation and Objectives 2 Subject of Research Study 3 Fact Sheet of Research Study 4 Findings 5 Summary & Outlook 13
  14. 14. Technical Requirements and Restrictions Input devices and types of connections (Own Study: Germany) The Connection is Although Data mostly GPRS (30%), Tarifs are often UMTS (11%) and GSM calculated based on (10%). 40% do not know transfer volume (29%); their type of 36% of respondents do connection. not know how much it costs. Most cell phones were manufactured by Nokia (39%) and Sony Ericsson (25%). (Own Study in 2008, N = 170) 14
  15. 15. Technical Requirements and Restrictions Traffic, Manufacturers and Devices (Benchmark: Western Europe) Apple leads with 31% share of overall requests in Western Europe in the AdMob Network. ( AdMob Mobile Metrics Report, January 2009) 15
  16. 16. Display of different question types Drop-out Rates – objective Measure Question with Multi response Single text During the survey, 23% of image vertical input respondents cancelled: Mostly at Question with Image (5%), followed by Multi response Cancellations and Text input (each with (N=150) 2% 2% 2%), all other pages without 5% questions (14%). Single text Single input response vertical Item-Nonresponse is high Item - Non with Text input (26%) and response low with Single response (N=115) 26% 2% (2%). 16
  17. 17. Scrolling 1 (2) “While answering, I Significant Difference between had to move the Mean Values on 5%-Level content around (‘scrolling‘).” Text input Closed Question Single response Multi response single line response list with picture vertical vertical 2,65 2,86 3,42 3,52 3,93 Mean Values from a scale of ‘never’ [1] to ‘often’ [7] 17
  18. 18. Scrolling 2 (2) “While answering, I Significant Difference between had to move the Mean Values on 5%-Level content around (‘scrolling‘).” Text input Closed Question Single response Multi response single line response list with picture vertical vertical 2,65 2,86 3,42 3,52 3,93 Mean Values from a scale of ‘never’ [1] to ‘often’ [7] 18
  19. 19. Ease of answering “With this question Significant Difference between type, I found it easy Mean Values on 5%-Level to select an answer.” Question Single response Multi response Closed Text input with picture vertical vertical response list single line 6,39 6,36 6,10 5,82 5,17 Mean Values from a scale of ‘Strongly disagree’ [1] to ‘Strongly agree’ [7] 19
  20. 20. Unintentional input Significant Difference between “During completion, Mean Values on 5%-Level I made unintended selections.” Single response Question Text input Multi response Closed vertical with picture single line vertical response list 1,54 1,67 1,67 2,01 2,24 Mean Values from a Scale of ‘never’ [1] to ‘extremely often’ [7] 20
  21. 21. Fluent answering “I could use this Significant Difference between question type Mean Values on 5%-Level fluently while answering.” Question Multi response Single response Closed Text input with picture vertical vertical response list single line 6,35 6,28 6,26 6.03 5,55 Mean Values from a scale of ‘Strongly disagree’ [1] to ‘Strongly agree’ [7] 21
  22. 22. Ease of Use “This question type Significant Difference between was ease to answer Mean Values on 5%-Level on my mobile phone.” Question Single response Multi response Closed Text input with picture vertical vertical response list single line 6,46 6,31 6,25 6.00 5,44 Mean Values from a scale of ‘Strongly disagree’ [1] to ‘Strongly agree’ [7] 22
  23. 23. Display of different question types During the post-survey every respondent was asked about the 5 question types. Additionally, we also looked for non-obtrusive indicators for measuring usability Highly Scores given by Respondents Single response correlated 89.2 vertical attributes of Multi response vertical 87.3 Question type the five 82.7 Closed list question types 74.7 Single line text input were 87.9 Question with Image summarised 0 25 50 75 100 into an Index: 'Fit for Purpose' (0-100 Points) 23
  24. 24. Display of different question types: Fit for Purpose - Summary Overall, the difference in the subjective user evaluation + are small The disadvantages of Text input fields are notable - Still, overall, respondents rated the question types + positively Limitations - Retrospective Survey - General assumptions regarding question content and questionnaire length should be examined further 24
  25. 25. Summary Technical Requirements & Restrictions - Efficiency of Design- and Implementation Technology for Mobile surveys - Diversity on the part of network providers and handset manufacturers (Data transfer rates, Display of content): Pre tests required! Display of content - All common question types: Positive ratings by respondents Usability Rating - Different perception with regard to scrolling effort, ease of use, unintended answers and fluency - Relative disadvantage of text input questions and partly with Drop- Down-Menus - Variation of respondents subjective perceptions and objective usability properties particular with regard to question type with pictures 25
  26. 26. Outlook Future research topics: 1 Robustness of results for a variety of topics? Influence of participation context on mobile survey 2 error dimensions largely unexplored. Development of perceived barriers in light of 3 technological developments (more advanced devices, such as smartphones; flatrates): Cost concerns may disappear, usability ratings may go up, broader applicability possible. 26
  27. 27. Thank you! Visit our mobile survey with your cell phone at: Mobile.opst.de/uc/main/157f There you can directly participate in the mobile survey. Tanja Pferdekämper Michael Bosnjak Gottfried Metzger tanja.pferdekaemper@globalpark.com http://contact.bosnjak.eu MetzgerGottfried@aol.com 27
  28. 28. Globalpark mit Sitz bei Köln, in London, New York und Wien gehört zu den weltweit führenden Anbietern von Online-Feedback-Software für Marktforschung, Personalwesen, Marketing und Kundenbeziehungsmanagement. Im deutschsprachigen Raum ist Globalpark in diesen Segmenten Marktführer. Über Globalpark Mehr als 1.000 Kunden arbeiten weltweit erfolgreich mit Globalpark-Software. Über 350 davon sind internationale Konzerne, führende Marktforschungsinstitute und Beratungsunternehmen. Zu den Kunden von Globalpark zählen namhafte Unternehmen wie Continental, Daimler, die Deutsche Lufthansa, die Deutsche Telekom, die GfK Gruppe, Siemens, Warner Music und Wrigley. Globalpark USA Globalpark AG Globalpark UK Ltd. 405 Lexington Ave. Kalscheurener Str. 19a 5 Archie Street New York, NY 10174 50354 Hürth London SE1 3JT Vereinigte Staaten von Amerika Deutschland (Hauptsitz) Großbritannien Tel.: +1 888 2999422 Tel.: +49 2233 7933 6 Tel.: +44 207 4033900 Standorte Globalpark Österreich GmbH Wassergasse 25 1030 Wien Österreich Tel.: +43 1715 028911 Dr. Tanja Pferdekämper Head of Consulting Kontakt Globalpark AG Phone: +49 2233 7933 721 E-Mail: tanja.pferdekaemper@globalpark.com
  29. 29. Nutzungshinweise © 2009 – Globalpark AG, Globalpark UK Ltd., Globalpark USA, Globalpark Österreich GmbH Die in dieser Publikation enthaltene Information ist Eigentum der Globalpark AG und ihrer Unternehmenstöchter. Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus ist nur mit ausdrücklicher schriftlicher Genehmigung durch die Globalpark AG oder ihre Unternehmenstöchter gestattet. Diese Publikation wird ohne jegliche Gewähr bereitgestellt. Enthaltene Informationen können ohne vorherige Ankündigung geändert werden. Zudem übernimmt Globalpark keine Garantie für die Exaktheit oder Vollständigkeit der Informationen, Texte, Grafiken, Links und sonstigen in dieser Publikation enthaltenen Elemente. Bei dieser Publikation handelt es sich um eine vorläufige Version, die gültigen Lizenzverträgen oder anderen Vereinbarungen mit Globalpark nicht unterliegt. Die von der Globalpark AG, ihren Unternehmenstöchtern und/oder ihren Vertriebsfirmen angebotenen Softwareprodukte können Softwarekomponenten anderer Softwarehersteller enthalten. Alle Produkte können länderspezifische Unterschiede aufweisen. Globalpark, Enterprise Feedback Suite, EFS, EFS Survey, EFS Panel, EFS Employee und EFS Leadership, weitere im Text erwähnte Globalpark-Produkte und -Dienstleistungen sowie die entsprechenden Logos sind Marken und können eingetragene Marken der Globalpark AG und/oder ihrer Unternehmenstöchter in Deutschland und anderen Ländern weltweit sein. Alle anderen Namen von Produkten und Dienstleistungen sind Marken der jeweiligen Firmen.

×