Mobile research -it’s just the same right-or is it?


Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Mobile research -it’s just the same right-or is it?

  1. 1. March 28, 2012MOBILE RESEARCH: IT’S JUST THESAME, RIGHT? OR IS IT?Dispatches from the cutting edge of research on mobile research
  2. 2. AgendaLessons learned about who wants to domobile research, when and whyDo you get the same data with mobile anddesktop?Touch vs. non-touch: is BlackBerrydifferent than an Android or iPhone? 2
  3. 3. Drawing on 3 RoR StudiesAll studies conducted in US, UK and CanadaStudy 1: 4 arms 500 respondents each, ShortMobile (11Q), Medium Mobile (16Q), Long Mobile(21Q), Desktop (21Q), Aug. 2011. Allsmartphone (iPhone, Android) ownersStudy 2: 2025 smartphone owners, given thechoice of mobile or desktop, Oct 2011Study 3: 1116 BlackBerry users, 902 iPhone orAndroid users, completing on their mobileMarch 2012 3
  4. 4. March 28, 2012Lessons learned about who wants todo mobile research, when and why
  5. 5. Response Rates amongst smartphone ownerswhere mobile was required for 3 of the 4 arms The response to the invite was basically the same across all arms but the actual completion rate was much lower for the mobile arms Responded  to  Invite  but  Dropped  Out   Successfully  completed  survey   50%   45%   41% 38% 38% 39% 40%   35%   30%   16%   15%   16%   25%   20%   39%   15%   10%   22%   23%   23%   5%   0%   2%   Short   Medium   Long   Desktop   Mobile Mobile Mobile
  6. 6. Where they fell by the waysideOnly 4 in 10 of those who responded to the invite to do the study on a mobiledevice actually completed it. Many in the mobile arms tried to do it on adesktop, despite our instructions, and then gave up. Most of the others whowere non compliant just gave up after being asked to go mobile. 100%   90%   80%   41%   40%   40%   Completed  survey   70%   Dropped  out  when  asked  to   60%   go  mobile   50%   17%   17%   17%   96%   Dropped  out  in  survey   40%   30%   Tried  to  do  survey  on  wrong   device  and  gave  up   20%   40%   41%   41%   10%   3%   2%   0%   Short   Medium   Long   Desktop  
  7. 7. Length and Dropping Out during the SurveyLength had no effect on the % of people who dropped outmid-survey, at least not up to the 9 ½ minute mark %  dropped  out  in  survey   4%   5%   4%   0%   Short   Medium   Long   Desktop   (4 min) (6 min) (9.5 min) (6 min) Mobile
  8. 8. In a subsequent study of smartphone owners, we gave them a choice of mobile or desktop•  Within the invitation respondents were informed that they could use their smart phone to complete. 11% of respondents used their mobile phone 11% 89% Desktop Mobile 8
  9. 9. Where they answered, given the choiceAmongst those smartphone owners who choosedesktop, 97% answered the survey at home or atwork.Amongst those who choose mobile, 15% wereanswering either from transit or somewhere otherthan home or work. 9
  10. 10. When they answered, given the choiceAmongst those smartphone owners who choosedesktop, 58% answered the survey the first day itlaunchedAmongst those who choose mobile, 88%answered it the first day. 10
  11. 11. Who answered mobile, when given the choice Those age 18-34, and women, were more likely to choose mobile Mobile Desktop 55% 55% 48% 47% 48%45% 45% 42% 42% 35% 35% 33% 32% 31% 27% 28% 29% 26% 25% 17% 16% 10% 11
  12. 12. Who answered mobile, given the choiceThose smartphone owners who completed thestudy on a mobile device were more likely to beiPhone users and people who used their mobilefor all sorts of things, like music, photos, surfingthe internet, and games.In short, they are mobile aficionados. 12
  13. 13. It takes longer on mobile, but…Study 2 took an average of 10 minutes tocomplete on a desktop and 15 minutes tocomplete on a mobile device.But it did not have any negative effect on drop-out rate or enjoyment of the survey. 13
  14. 14. “This survey was fun to complete” Agree strongly 50% 38% Mobile Desktop 14
  15. 15. Why mobile?People chose to do research on theirmobile when it is reaching them wherethey want, when they want and using themedium they want.Mobile should be a choice, not an either/orrequirement. But for it to be a choice, thedata you collect needs to be equivalent… 15
  16. 16. March 28, 2012Do you get the same data with mobileand desktop?
  17. 17. Interface MattersWe know that the type of interfaceyou use can make a difference.And the mobile interface isdefinitely different.We have previously proven, in a number ofstudies, that our visual question types are moreengaging. The result is better quality data,compared to traditional “radio button” questions.Depending upon the question type, we may alsosee better use of the entire scale, less flat liningand fewer “don’t know”. 17
  18. 18. Where they answered, given the choiceIn Study 2, we gave people the option to choosemobile or desktop. And we asked a series ofquestions about the environment, poverty andother social issues.We used a variety of different question types inthe survey, to see if each question type wasproviding the same answers.Because we know that those who choose mobileare different in some ways, in our analysis wecontrolled for differences in age, gender and typeof mobile device owned between those whoanswered desktop vs. mobile. 18
  19. 19. Single choice QuestionsThe responses to thesesimple single choice questionswere the same, despite adifference in orientation. Decrease 32% 36%Remain the same 30% Desktop 29% Mobile Increase 38% 35% 19
  20. 20. Multi-choice QuestionsThe responses to this multi-choicequestion were the same, despite theneed for scrolling on the mobileversion.Take public transit, walk or bicycle, 75% to reduce pollution 75% Reduce the amount of electricity 68% you use 76% Recycle paper and cardboard, 74% when you are not at home 70% Recycle bottles and cans, when 69% you are not at home 67% Put food scraps in the compost or 37% green bin 37% Desktop Do not flush the toilet every time 34% Mobile you use it 33%Bring your own reusable bag when 18% going shopping 19% Bring your own cup when 4% purchasing a beverage 2% 20
  21. 21. AllocationsDespite having quite differentinterfaces, there were nostatistically significant differences inpeople’s allocations $26.90 $27.50 Improving Public Education $26.20 $28.00 Reducing Poverty $17.00 $18.00 Preserving the Environment $25.60 $26.50 Reducing Crime Mobile Desktop 21
  22. 22. Numeric QuestionsThere were no statisticallysignificant differences betweeninterfaces, in people’s estimates ofthe percent of people living inpoverty Mobile Desktop 28 26 21 20 15 15 16 16 Australia Canada UK USA 22
  23. 23. Visual Grid Single Choice QuestionsThere were no statisticallysignificant differences betweeninterfaces, in people’s attitudestoward poverty.This is notable because theinterfaces are quite different, withone being a grid with a visualpattern reminder of previousanswers and the other providing awritten record. 23
  24. 24. Visual Grid Multi-Choice QuestionsThere were significant differences withthe multi-choice grid question type.What tended to happen was thatrespondents choose more items withthe mobile version. This possibledifference needs to be exploredfurther, as it could implications for useand data interpretation. Descriptors for the UK Mobile Desktop 25%17% 17% 9% 11% 11%9% 8% 8% 5% Greedy Resource Protect Wasteful Enviromental Rich Environment Innovators 24
  25. 25. March 28, 2012Touch vs. non-touch: is BlackBerrydifferent than an iPhone?
  26. 26. BlackBerry vs. iPhone There is something almost tribal about the BlackBerry vs. iPhone debate. We know that the profile of the users of each device differ. But what about how they answer surveys? We have seen in Study 1 and 2 that, with our mobile interface, that touchscreen smart phones can deliver an enjoyable way to complete a survey. A survey that produces data that, with the probable exception of the multi-choice grid, is identical to that we gather from a computer- based survey. But what about a non-touch BlackBerry? Again, the interface is different… 26
  27. 27. When required in the survey invite to completethe study on their mobile, those using non-touch BlackBerries were less likely to go aheadand complete the study The 43% completion rate we see with the touch device owners is very similar to the 40% we saw in Study 1. Once they started the study, there were no differences in drop-out rate between touch and non-touch devices Tried  desktop/dropped  out  on  mobile   Completed  on  Mobile   100%   80%   30%   43%   60%   40%   70%   57%   20%   0%   iPhone/Android   BlackBerry  Base is all those mobile device owners who opened the survey invite for Study 3 27
  28. 28. Those who completed the survey on a non-touch BlackBerry tended to be less delightedthan those who did it on a Touch device “This survey was hard to complete.” “This survey was easy to complete.” % agree % agree 100%   100%   95%   89%   50%   50%   12%   16%   0%   0%   Touch   Non-­‐touch   Touch   Non-­‐touch   “This survey was fun to complete.” “This survey was more enjoyable than most.” 100%   82%   100%   83%   74%   72%   50%   50%   0%   0%   Touch     Non-­‐touch   Touch   Non-­‐touch  
  29. 29. One reason for the slightly lower engagementand easy of use is likely that it took longer tocomplete the survey on a non-touchBlackBerry than on a Touch device Median length of survey, in minutes Touch  Device   Non-­‐Touch  BlackBerry   10.5   14.6   29
  30. 30. Interface: Touch vs. TypeThe fundamental interface differencethat we are concerned with here istouch vs. typing.There are touch BlackBerries, but our research hasshown the interface behaves the same as other touchdevices, be it iPhone or Android. So our focus is ontouch vs. typing, because that’s where the interfacesdiffer.We know that there are significant differences in the age,gender and education of touch users vs. users ofBlackBerries with a physical keyboard—differences thatinfluence people’s answers. So we controlled for theseage, gender and education differences in our analysis. 30
  31. 31. Touch vs. Type: For the most part, theinterface looks fairly similar. It is how younavigate it that differs more 31
  32. 32. Touch vs. Type: Our analysis revealed thatthere was no pattern of significant differencesbetween touch and non-touch smartphonesfor…Single Choice QuestionsMulti-choice QuestionsAllocationsNumeric QuestionsVisual Grid Single Choice Questions 32
  33. 33. Where we did see a difference between Touchand Non-Touch was the Multi-choice GridThere was, however, no clear pattern of difference.One confounding factor may be the fact thesequestions were about people’s image of differentcountries. There is variation in ownership of thetypes of touch and non-touch mobile devices bycountry. We attempted to control for that statisticallybut may not have been fully successful.Or there may be real differences by device. Furtherresearch is needed to confirm or deny thesehypotheses. 33
  34. 34. Conclusions 34
  35. 35. March 28, 2012Lessons learned about who wants todo mobile research, when and why People chose to do research on their mobile when it is reaching them where they want, when they want and using the medium they want. Mobile should be a choice, not an either/or requirement. But for it to be a choice, the data you collect needs to be equivalent…
  36. 36. March 28, 2012Do you get the same data with mobileand desktop? We have proven that our mobile touch-screen interface delivers the same data as our desktop computer interface, and is enjoyable and engaging. The one question type that may be different is the multi- choice grid. Further study is required, but until we clarify that, it should be kept in mind that the mobile interface for multi-choice questions may generate slightly more associations.
  37. 37. March 28, 2012Touch vs. non-touch: is BlackBerrydifferent than an iPhone? While the BlackBerry non-touch interface performed well, people were slightly less enamored with it, confirming that iPhones are simply more fun.* The data collected was equivalent to the Touch interface and the desktop interface, with the possible exception of the multi-choice grid. Further study is needed on this point. * Yes, I have an iPhone, and used to be a BlackBerry user
  38. 38. ConclusionMobile research is a new frontier and we need toexplore it carefully before plunging ahead.Our research suggests that well designed interfacescan provide an engaging experience for respondentsand allow them to do surveys where they want andwhen they want.We have seen that you cannot just assumeequivalence of data when using different interfaces—you need proof.Further research will be done as we explore thefrontiers of mobile research 38
  39. 39. Questions? 39