Making The Leap From Web To Mobile


Published on

Best Practices in Mobile User Experience Research

Amy Buckner, AnswerLab
Kris Mihalic, Yahoo!
UPA 2009, June 12, Portland

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Making The Leap From Web To Mobile

  1. 1. Making the Leap from Web to Mobile <br />Best Practices in Mobile User Experience Research<br />Amy Buckner, AnswerLab<br />Kris Mihalic, Yahoo!<br />UPA 2009, June 12, Portland<br />
  2. 2. Agenda<br />
  3. 3. Introductions<br />
  4. 4. State of Mobile<br />
  5. 5. There are mobile phone subscribers worldwide<br />4.1 Billion<br />Only of US mobile phones are iPhones (~5 Million)<br />2%<br />Social networking on mobile phones is growing Y/Y<br />196%<br />There are more mobile IM users than mobile business email users in the US<br />1.4x<br />
  6. 6. Mobile Market Summary<br />Globally, there are 800 million cars, 850 million PCs, 1.3 billion fixed landline phones, 1.4 billion credit cards, 1.5 billion TV sets – and 2.7 billion mobile phones – in use. (Experian 2008) <br />eMarketer projects that worldwide spending on mobile advertising will reach a total of $19.1 billion in 2012 – up from 4.6 billion in 2008.<br />Source: Experian 2008, eMarketer 2008<br />
  7. 7. U.S. Mobile Web Users Growing Rapidly<br />Source: m:metrics/comScore, 2009<br />
  8. 8. Mobile Audience Size (# of users)<br />In the U.S., the size of the Mobile Audience is 250 Million<br />Source: m:metrics/comScore, 2009<br />
  9. 9. U.S. Mobile Internet Usage<br />Mobile Internet users are heavy users<br />60% access the Internet from their phone at least once/day<br />48% access the Internet more than once per day<br />Source: TSM|TargetProfile 2007<br />
  10. 10. Device Traffic<br />Mobile phones dominate mobile Web traffic with 66%market share (US).<br />BlackBerry and Smartphones (Windows Mobile, Palm, etc.) are heavy users of mobile Internet – combined 26% of traffic<br />iPhone/iPod Touch has shown tremendous growth, capturing 8% of the mobile Internet market in the US<br />Source: m:metrics/comScore, 2009<br />
  11. 11. Widgets Improve Mobile Internet Experience<br />Source: ‘Widgets Improve the Convenience of the Mobile Internet,’ Forrester, 2008<br />
  12. 12. Group Exercise<br />
  13. 13. Take out your cell phone and search for times that ‘Star Trek’ is playing in your neighborhood. <br />Raise your hand when you have finished the task.<br />
  14. 14. Discussion Points<br /> How long did it take?<br />How did you do it? (web, app, SMS?) <br />What problems, if any, did you have?<br />
  15. 15. Mobile User Research Methods<br />
  16. 16. Overview of Research Methods<br />Quick and Dirty feedback<br />Lab Studies<br />Field Studies<br />Online Surveys<br />Basic Survey<br />Survey with Behavioral Tracking (Keynote on iPhone only)<br />Quantitative Behavioral Analysis<br />
  17. 17. Quick and Dirty – What are obvious problems?<br />Advantages<br />Informal setting with Lo-Fi prototypes<br />Fast iteration cycles<br />Brain-damage check<br />Finds critical usability issues in shortest time<br />Challenges<br />Examples<br />Insider/stakeholder view<br />Anecdotal, unstructured results<br />Small sample size<br />Difficult to convey real usage scenarios<br />Cross-functional team feedback<br />Internal alpha/beta<br />
  18. 18. Lab Studies – Can they use it?<br />Advantages<br />Allows for probing by moderator<br />Can record interactions<br />Best for prototype testing<br />Allows for real-time viewing of interaction – team engagement<br />Challenges<br />Examples<br />Artificial environment<br />Moderator bias; participant ‘pleasing’<br />Small sample size<br />Difficult to test all hardware / network scenarios<br />Usability study<br />Participatory design sessions<br />
  19. 19. Field Studies – What is the context of usage?<br />Advantages<br />Assess usage under real conditions<br />Candid user feedback<br />Delivers unknown use-cases<br />Challenges<br />Examples<br />Small sample size<br />Difficult to observe and probe<br />Participant engagement difficult<br />Requires robust product<br />Diary study<br />Community study<br />
  20. 20. Online Surveys – What do they think of it?<br />Advantages<br />Can deliver statistically valid results<br />Provides qualitative and/or quantitative data<br />Geographical diversity (remote)<br />Broad representation of devices<br />Challenges<br />Examples<br />Difficult to observe and probe<br />Out-of-context interaction, esp. with desktop survey<br />Technical constraints<br />Desktop surveys<br />On-device survey<br />Survey with behavioral tracking<br />
  21. 21. Behavioral Analysis – How do they use it?<br />Advantages<br />Reflects what people do, not say<br />Statistically valid results<br />Shows all behaviors, rather than those confined to a single task<br />Challenges<br />Examples<br />Probing / deep-dive difficult<br />Context and intent unknown<br />Attitudes and perceptions unknown<br />Log data mining<br />
  22. 22. Types of Prototypes<br />Paper prototypes<br />Mocks<br />Paper and pencil<br />Interactive prototypes<br />Apps<br />Flash<br />Native, e.g. iPhone<br />Browser-based<br />Simple HTML prototype<br />High-fidelity prototype (with session variables)<br />
  23. 23. Team Exercise & Case Study<br />
  24. 24. Yahoo! Go<br />Mobile Application<br />Search<br />News<br />Sports<br />Weather<br />Other Topics<br />
  25. 25. Y! oneSearch integrated into Y! Go<br />
  26. 26. Business Issue<br />Drive mobile search uptake – Improved user experience can help accelerate growth of the service<br />Improve relevancy – Understanding search intent can contribute to delivering more relevant results<br />Product differentiation – New approach to search results page can deliver higher value to customers<br />
  27. 27. Research Objectives<br />Understand mobile search behavior in users’ daily lives<br />Identify content users seekwhen conducting mobile searches<br />Assess the context surrounding mobile searches<br />Evaluate effectiveness of new product<br />Identify opportunities to improve the user experience<br />
  28. 28. Form into teams of 4.<br />Develop a research plan for Yahoo! Go.<br />You have 10 minutes.<br />
  29. 29. Research Objectives<br />Team Activity<br />Understand mobile search behavior in users’ daily lives<br />Identify content users seekwhen conducting mobile searches<br />Assess the context surrounding mobile searches<br />Evaluate effectiveness of new product<br />Identify opportunities to improve the user experience<br />Develop a research plan for Yahoo! Go<br />Include: <br /><ul><li>Recommended method(s)
  30. 30. Number of participants
  31. 31. Profile of participants</li></li></ul><li>Discussion Points<br /> What methods do you recommend & why?<br /> What are your anticipated concerns?<br /> Any potential limitations?<br />
  32. 32. Our Solution: One-Month Field Study (1)<br />Daily Mobile Survey<br />Digital Photos<br />Voicemails & Pocket Card<br />Daily SMS mini-survey <br />Linked to phone number for voicemail <br />Users sent photos of themselves or surroundings in the context of using oneSearch<br />9 users <br />Daily voicemails<br />Laminated pocket card with key questions<br />
  33. 33. ‘Pocket Card’ Questionnaire<br />
  34. 34. Sample Voicemail Diary Messages -- Winchelle<br />Friday 7:39pm<br />I used oneSearch today, was looking for a hospital – directions and address to it. I was getting off work and sitting in my car, trying to find directions. It was about 4:30. I liked that it came up with the website and with information about that. And the directions on how to go.<br />I didn’t like that it could not find my work address – which is saved on my Yahoo! directions (recently gone) saved places. <br />I think using the actual internet through the PC will help me better because it knows where the address is. I’m not sure why the address is not coming up in my oneSearch and that sucked.<br />Saturday 8:06pm<br />I didn’t use oneSearch today. I looked for info, looked for a store and I was at home. I wanted to know if the store was open STORE HOURS (9:45am). I used the internet with my phone and used Google to search. It was not very helpful because I wanted to see what time they opened and I couldn’t find it. I’ve used oneSearch in the past and I guess the results were the same but I guess it’d be nice to go to the site for that particular store.<br />Monday 10:48pm<br />I used oneSearch to find Expedia, was at work on my lunch break. It was about 12:20. I liked that I can find it quickly. I guess the website didn’t support my mobile phone though, so I didn’t like that. Would rather book a flight/hotel at home because its supported.<br />
  35. 35. Our Solution: One-Month Field Study (2)<br />Pre-Interview<br />Mid-Check Interview<br />Wrap-up Interview<br />An initial 45-minute interview <br />Verified technical capabilities of phone<br />Explained research program<br />20-minute check-in phone interview<br />Initial feedback<br />Q&A<br />Final in-person interview of 75 minutes <br />Clarified voicemail reports<br />Overall impressions of the product from the month-long usage<br />
  36. 36. Screening Criteria: 9 Experienced Mobile Users <br />Have an unlimited data plan for WAP services through their mobile carrier<br />Use their mobile for text messages (SMS) daily and use mobile more than three times per week for activities that require the transmission of online data<br />Have Cingular, Sprint, T-Mobile, or Verizon as their mobile carrier<br />Have a primary mobile device capable of downloading and using Y! Go<br />Live or work within San Francisco<br />Mix of gender, age (23-46+),and typical method of commuting to work (4 users primarily drive, 5 primarily take mass transit) <br />
  37. 37. Sliding Scale for Participant Incentives<br />$100 for the initial in-person interview<br />$50 for the mid-way phone interview<br />$150 for the final in-person interview<br />$3 each day a report is given<br />
  38. 38. Natural environment key to context<br /> All research activities on the mobile device<br /> Mix of methods <br /> Engage users in program<br /> Breadth of insights – context, usability, impressions<br />Why did we choose this method?<br />
  39. 39. What did we learn? (1)<br />Perceived value increased with usage<br />Usage typically driven by lack of computer availability<br />However, certain scenarios drove phone choice over computer: <br />Social setting <br />Privacy <br />Convenience <br />
  40. 40. What did we learn? (2)<br />Mental model based on computer usage<br />Speed and relevance of search results highest area of frustration<br />Lack of understanding that results were customized by widget<br />
  41. 41. Daily SMS Survey Results<br />*Website not adapted for mobile device<br />
  42. 42. What did we learn? (3)<br />
  43. 43. Summary of Use Case Frequency<br />Note: The 166 searches were conducted by 9 users<br />
  44. 44. Shopping<br />Business’ number, location, directions, and hours of operation <br />“While driving home from work, I did a search for Fry’s electronics on the local search and it came up with all the different Fry’s in the area, their numbers, as well as directions and all that. The results were all helpful and fast in helping me find the one closest.”<br />“Since we were already out shopping, I wanted to find where the closest Citibank was to where we were.”<br />Pricing/Product comparisons<br />“I was at Costco and comparing TV prices. I wanted to know how much these same TV’s would be at the Circuit City down the street.”<br />
  45. 45. Dining<br />A restaurant’s number, location, directions, and hours of operation <br />“I just woke up and didn’t want to boot my computer up but I wanted to find out what time this breakfast place opened so I could know how soon I could get there.”<br />“I was in my car and wanted their number so I could call and place an order to pick it up on my way home from work.”<br />Ideas of places to eat<br />“I was with my wife in the Haight and we wanted to find nearby restaurants so we could decide what we wanted to eat that was within walking distance.”<br />“I was in the South Bay with some friends and we wanted to find a good sushi place nearby.”<br />Reservations & Reviews<br />“I was hoping to make reservations straight from the phone, or at least call them to place them.”<br />“I was out with some friends and curious about whether the place nearby was any good.”<br />
  46. 46. Traveling<br />Transit schedules and numbers <br />“I wanted to find out when the next MUNI but the MUNI’s site wouldn’t work on my phone.”<br />“I wanted the number to call a taxi. I was outside and tired of waiting for the bus.”<br />Tourist activities<br />“I wanted to see what kind of ferry tours we could do before looking for another museum that would be fun for my daughter.”<br />“I was looking for fun things to do while at Hermosa beach.”<br />Maps and directions<br />“I was trying to get a map of the Mt. Shasta area and I wanted to plan which route to take. It would be nice if the driving directions had live updates on traffic and road conditions.”<br />Flight status / check-in<br />“I was in the taxi to the airport and wanted to check my flight’s status.”<br />
  47. 47. Business Implications<br />Drive mobile search uptake – Distribution through partnerships, marketing campaigns, opening to third party services and developers<br />Improve relevancy – Focused on improving results for specific use cases (e.g. local businesses, flights, etc.)<br />Product differentiation – Federated search results, integrated user experience across multiple services (e.g. search results + maps)<br />
  48. 48. Mobile Usability Research Challenges & Tips<br />
  49. 49. Variety of Mobile Devices<br />Challenges<br />Tips<br />Difficult to know and understand user experience across all devices<br />Devices vary by model, browser type, carrier, and input type<br />Moderator may be unable to help users through usability tasks – resetting prototypes, navigating – if interface is not well-known<br />Design team should have a variety of devices simulators / devices on hand for testing<br />Prototype designs should be tested across as many devices as possible<br />Limit recruiting to only devices that have been tested<br />Have a back-up plan with either a basic device or paper prototypes<br />
  50. 50. Challenges<br />Tips<br />Mobile Devices are Small and . . . Well, Very Mobile <br />Create a hot zone on the table, encouraging users to keep the device within a narrow frame <br />Consider light source and potential reflection on the phone; continually adjust mid testing; turn-off lights<br />Use remote-controlled video camera (with technician in back room) or have a second technician available in interview room <br />Project the mobile screen on a larger monitor in interview room and back room<br />Creatively screen out participants whose finger size may distract from findings (if non-touch screen)<br />Users move the device around while interacting and explaining, often moving video display out of focus<br />Reflectors on screens and smudges can make video all reflection<br />Small device screens make it difficult to see what users are doing<br />Large fingers and long fingernails can cause unintended device responses<br />
  51. 51. Challenges<br />Tips<br />Varied Locations and Use Cases <br />Mobile device usage occurs in a variety of places<br />59% of Americans check email while in the bathroom*<br />Difficult to recreate the true experience in a lab setting <br />Create opportunities for feedback in context of usage<br />Conduct field studies on beta products<br />MOBILE DEVICE ACCESS<br /> Source: Nielsen Mobile 2008<br />
  52. 52. Challenges<br />Tips<br />Varied Carrier Network Coverage<br />Network coverage is inconsistent; difficult to predict accessibility during lab testing<br />Difficult to validate that out-of-town lab truly has adequate coverage<br />Visit facility with colleagues who have various carriers to confirm network availability in lab<br />From out-of-town labs, request:<br />Reference clients from last mobile studies<br />List of carriers confirmed to have consistent coverage<br />At a minimum, check carrier coverage in various cities online<br />
  53. 53. Screen during recruiting for:<br />Unlimited data plans<br />Willingness to send / receive text messages during study<br />Rescreen again upon arrival for study<br />Challenges<br />Tips<br />Calling / Data Plans Vary Widely<br />Participants may be charged for mobile web access or sample texts sent during testing<br />Participants may change data plan between recruiting interview and date of study<br />
  54. 54. Challenges<br />Tips<br />Varied User Experience<br />Most mobile device owners only utilize a small percentage of total device capabilities<br />43% of mobile subscribers do not use text messaging on a regular basis*<br />Users may have no experience typing in a web address to access a site or with sending SMS messages (which may be critical to your study)<br />If using a prototype phone for downloadable app, users may not be comfortable with it<br />Pre-test participants to ensure appropriate device experience<br />Mobile address typing <br />Send SMS to recruiter<br />Spend 2-5 minutes explaining how prototype phone works<br />* Source: Nielsen 2008 ‘The Short Code Marketing Opportunity’<br />
  55. 55. Other Helpful Tips<br />Tell participants to bring recharger<br />Simplify language (e.g., SMS vs. text message)<br />Have a prototype strategy<br />Back-end SMS text simulation<br />Easy URL for access<br />Back-up paper prototypes<br />Index page for easy access to alternative flows<br />Have plan for interruption from phone call or text (resetting session variables)<br />Plan for getting screenshots for your report<br />iPhone: Press hold button and home button at same time (saves into photo gallery)<br />Use documented comps<br />
  56. 56. Q&A<br />
  57. 57. Thank You.<br />Amy Buckner<br />Managing Partner & Co-Founder, AnswerLab<br />Kris Mihalic<br />Head of Mobile User Experience and Design Research, Yahoo!<br />Twitter: suikris<br />