Your SlideShare is downloading. ×
Making The Leap From Web To Mobile
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Making The Leap From Web To Mobile


Published on

Best Practices in Mobile User Experience Research …

Best Practices in Mobile User Experience Research

Amy Buckner, AnswerLab
Kris Mihalic, Yahoo!
UPA 2009, June 12, Portland

Published in: Technology, Business

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Making the Leap from Web to Mobile
    Best Practices in Mobile User Experience Research
    Amy Buckner, AnswerLab
    Kris Mihalic, Yahoo!
    UPA 2009, June 12, Portland
  • 2. Agenda
  • 3. Introductions
  • 4. State of Mobile
  • 5. There are mobile phone subscribers worldwide
    4.1 Billion
    Only of US mobile phones are iPhones (~5 Million)
    Social networking on mobile phones is growing Y/Y
    There are more mobile IM users than mobile business email users in the US
  • 6. Mobile Market Summary
    Globally, there are 800 million cars, 850 million PCs, 1.3 billion fixed landline phones, 1.4 billion credit cards, 1.5 billion TV sets – and 2.7 billion mobile phones – in use. (Experian 2008)
    eMarketer projects that worldwide spending on mobile advertising will reach a total of $19.1 billion in 2012 – up from 4.6 billion in 2008.
    Source: Experian 2008, eMarketer 2008
  • 7. U.S. Mobile Web Users Growing Rapidly
    Source: m:metrics/comScore, 2009
  • 8. Mobile Audience Size (# of users)
    In the U.S., the size of the Mobile Audience is 250 Million
    Source: m:metrics/comScore, 2009
  • 9. U.S. Mobile Internet Usage
    Mobile Internet users are heavy users
    60% access the Internet from their phone at least once/day
    48% access the Internet more than once per day
    Source: TSM|TargetProfile 2007
  • 10. Device Traffic
    Mobile phones dominate mobile Web traffic with 66%market share (US).
    BlackBerry and Smartphones (Windows Mobile, Palm, etc.) are heavy users of mobile Internet – combined 26% of traffic
    iPhone/iPod Touch has shown tremendous growth, capturing 8% of the mobile Internet market in the US
    Source: m:metrics/comScore, 2009
  • 11. Widgets Improve Mobile Internet Experience
    Source: ‘Widgets Improve the Convenience of the Mobile Internet,’ Forrester, 2008
  • 12. Group Exercise
  • 13. Take out your cell phone and search for times that ‘Star Trek’ is playing in your neighborhood.
    Raise your hand when you have finished the task.
  • 14. Discussion Points
    How long did it take?
    How did you do it? (web, app, SMS?)
    What problems, if any, did you have?
  • 15. Mobile User Research Methods
  • 16. Overview of Research Methods
    Quick and Dirty feedback
    Lab Studies
    Field Studies
    Online Surveys
    Basic Survey
    Survey with Behavioral Tracking (Keynote on iPhone only)
    Quantitative Behavioral Analysis
  • 17. Quick and Dirty – What are obvious problems?
    Informal setting with Lo-Fi prototypes
    Fast iteration cycles
    Brain-damage check
    Finds critical usability issues in shortest time
    Insider/stakeholder view
    Anecdotal, unstructured results
    Small sample size
    Difficult to convey real usage scenarios
    Cross-functional team feedback
    Internal alpha/beta
  • 18. Lab Studies – Can they use it?
    Allows for probing by moderator
    Can record interactions
    Best for prototype testing
    Allows for real-time viewing of interaction – team engagement
    Artificial environment
    Moderator bias; participant ‘pleasing’
    Small sample size
    Difficult to test all hardware / network scenarios
    Usability study
    Participatory design sessions
  • 19. Field Studies – What is the context of usage?
    Assess usage under real conditions
    Candid user feedback
    Delivers unknown use-cases
    Small sample size
    Difficult to observe and probe
    Participant engagement difficult
    Requires robust product
    Diary study
    Community study
  • 20. Online Surveys – What do they think of it?
    Can deliver statistically valid results
    Provides qualitative and/or quantitative data
    Geographical diversity (remote)
    Broad representation of devices
    Difficult to observe and probe
    Out-of-context interaction, esp. with desktop survey
    Technical constraints
    Desktop surveys
    On-device survey
    Survey with behavioral tracking
  • 21. Behavioral Analysis – How do they use it?
    Reflects what people do, not say
    Statistically valid results
    Shows all behaviors, rather than those confined to a single task
    Probing / deep-dive difficult
    Context and intent unknown
    Attitudes and perceptions unknown
    Log data mining
  • 22. Types of Prototypes
    Paper prototypes
    Paper and pencil
    Interactive prototypes
    Native, e.g. iPhone
    Simple HTML prototype
    High-fidelity prototype (with session variables)
  • 23. Team Exercise & Case Study
  • 24. Yahoo! Go
    Mobile Application
    Other Topics
  • 25. Y! oneSearch integrated into Y! Go
  • 26. Business Issue
    Drive mobile search uptake – Improved user experience can help accelerate growth of the service
    Improve relevancy – Understanding search intent can contribute to delivering more relevant results
    Product differentiation – New approach to search results page can deliver higher value to customers
  • 27. Research Objectives
    Understand mobile search behavior in users’ daily lives
    Identify content users seekwhen conducting mobile searches
    Assess the context surrounding mobile searches
    Evaluate effectiveness of new product
    Identify opportunities to improve the user experience
  • 28. Form into teams of 4.
    Develop a research plan for Yahoo! Go.
    You have 10 minutes.
  • 29. Research Objectives
    Team Activity
    Understand mobile search behavior in users’ daily lives
    Identify content users seekwhen conducting mobile searches
    Assess the context surrounding mobile searches
    Evaluate effectiveness of new product
    Identify opportunities to improve the user experience
    Develop a research plan for Yahoo! Go
    • Recommended method(s)
    • 30. Number of participants
    • 31. Profile of participants
  • Discussion Points
    What methods do you recommend & why?
    What are your anticipated concerns?
    Any potential limitations?
  • 32. Our Solution: One-Month Field Study (1)
    Daily Mobile Survey
    Digital Photos
    Voicemails & Pocket Card
    Daily SMS mini-survey
    Linked to phone number for voicemail
    Users sent photos of themselves or surroundings in the context of using oneSearch
    9 users
    Daily voicemails
    Laminated pocket card with key questions
  • 33. ‘Pocket Card’ Questionnaire
  • 34. Sample Voicemail Diary Messages -- Winchelle
    Friday 7:39pm
    I used oneSearch today, was looking for a hospital – directions and address to it. I was getting off work and sitting in my car, trying to find directions. It was about 4:30. I liked that it came up with the website and with information about that. And the directions on how to go.
    I didn’t like that it could not find my work address – which is saved on my Yahoo! directions (recently gone) saved places.
    I think using the actual internet through the PC will help me better because it knows where the address is. I’m not sure why the address is not coming up in my oneSearch and that sucked.
    Saturday 8:06pm
    I didn’t use oneSearch today. I looked for info, looked for a store and I was at home. I wanted to know if the store was open STORE HOURS (9:45am). I used the internet with my phone and used Google to search. It was not very helpful because I wanted to see what time they opened and I couldn’t find it. I’ve used oneSearch in the past and I guess the results were the same but I guess it’d be nice to go to the site for that particular store.
    Monday 10:48pm
    I used oneSearch to find Expedia, was at work on my lunch break. It was about 12:20. I liked that I can find it quickly. I guess the website didn’t support my mobile phone though, so I didn’t like that. Would rather book a flight/hotel at home because its supported.
  • 35. Our Solution: One-Month Field Study (2)
    Mid-Check Interview
    Wrap-up Interview
    An initial 45-minute interview
    Verified technical capabilities of phone
    Explained research program
    20-minute check-in phone interview
    Initial feedback
    Final in-person interview of 75 minutes
    Clarified voicemail reports
    Overall impressions of the product from the month-long usage
  • 36. Screening Criteria: 9 Experienced Mobile Users
    Have an unlimited data plan for WAP services through their mobile carrier
    Use their mobile for text messages (SMS) daily and use mobile more than three times per week for activities that require the transmission of online data
    Have Cingular, Sprint, T-Mobile, or Verizon as their mobile carrier
    Have a primary mobile device capable of downloading and using Y! Go
    Live or work within San Francisco
    Mix of gender, age (23-46+),and typical method of commuting to work (4 users primarily drive, 5 primarily take mass transit)
  • 37. Sliding Scale for Participant Incentives
    $100 for the initial in-person interview
    $50 for the mid-way phone interview
    $150 for the final in-person interview
    $3 each day a report is given
  • 38. Natural environment key to context
    All research activities on the mobile device
    Mix of methods
    Engage users in program
    Breadth of insights – context, usability, impressions
    Why did we choose this method?
  • 39. What did we learn? (1)
    Perceived value increased with usage
    Usage typically driven by lack of computer availability
    However, certain scenarios drove phone choice over computer:
    Social setting
  • 40. What did we learn? (2)
    Mental model based on computer usage
    Speed and relevance of search results highest area of frustration
    Lack of understanding that results were customized by widget
  • 41. Daily SMS Survey Results
    *Website not adapted for mobile device
  • 42. What did we learn? (3)
  • 43. Summary of Use Case Frequency
    Note: The 166 searches were conducted by 9 users
  • 44. Shopping
    Business’ number, location, directions, and hours of operation
    “While driving home from work, I did a search for Fry’s electronics on the local search and it came up with all the different Fry’s in the area, their numbers, as well as directions and all that. The results were all helpful and fast in helping me find the one closest.”
    “Since we were already out shopping, I wanted to find where the closest Citibank was to where we were.”
    Pricing/Product comparisons
    “I was at Costco and comparing TV prices. I wanted to know how much these same TV’s would be at the Circuit City down the street.”
  • 45. Dining
    A restaurant’s number, location, directions, and hours of operation
    “I just woke up and didn’t want to boot my computer up but I wanted to find out what time this breakfast place opened so I could know how soon I could get there.”
    “I was in my car and wanted their number so I could call and place an order to pick it up on my way home from work.”
    Ideas of places to eat
    “I was with my wife in the Haight and we wanted to find nearby restaurants so we could decide what we wanted to eat that was within walking distance.”
    “I was in the South Bay with some friends and we wanted to find a good sushi place nearby.”
    Reservations & Reviews
    “I was hoping to make reservations straight from the phone, or at least call them to place them.”
    “I was out with some friends and curious about whether the place nearby was any good.”
  • 46. Traveling
    Transit schedules and numbers
    “I wanted to find out when the next MUNI but the MUNI’s site wouldn’t work on my phone.”
    “I wanted the number to call a taxi. I was outside and tired of waiting for the bus.”
    Tourist activities
    “I wanted to see what kind of ferry tours we could do before looking for another museum that would be fun for my daughter.”
    “I was looking for fun things to do while at Hermosa beach.”
    Maps and directions
    “I was trying to get a map of the Mt. Shasta area and I wanted to plan which route to take. It would be nice if the driving directions had live updates on traffic and road conditions.”
    Flight status / check-in
    “I was in the taxi to the airport and wanted to check my flight’s status.”
  • 47. Business Implications
    Drive mobile search uptake – Distribution through partnerships, marketing campaigns, opening to third party services and developers
    Improve relevancy – Focused on improving results for specific use cases (e.g. local businesses, flights, etc.)
    Product differentiation – Federated search results, integrated user experience across multiple services (e.g. search results + maps)
  • 48. Mobile Usability Research Challenges & Tips
  • 49. Variety of Mobile Devices
    Difficult to know and understand user experience across all devices
    Devices vary by model, browser type, carrier, and input type
    Moderator may be unable to help users through usability tasks – resetting prototypes, navigating – if interface is not well-known
    Design team should have a variety of devices simulators / devices on hand for testing
    Prototype designs should be tested across as many devices as possible
    Limit recruiting to only devices that have been tested
    Have a back-up plan with either a basic device or paper prototypes
  • 50. Challenges
    Mobile Devices are Small and . . . Well, Very Mobile
    Create a hot zone on the table, encouraging users to keep the device within a narrow frame
    Consider light source and potential reflection on the phone; continually adjust mid testing; turn-off lights
    Use remote-controlled video camera (with technician in back room) or have a second technician available in interview room
    Project the mobile screen on a larger monitor in interview room and back room
    Creatively screen out participants whose finger size may distract from findings (if non-touch screen)
    Users move the device around while interacting and explaining, often moving video display out of focus
    Reflectors on screens and smudges can make video all reflection
    Small device screens make it difficult to see what users are doing
    Large fingers and long fingernails can cause unintended device responses
  • 51. Challenges
    Varied Locations and Use Cases
    Mobile device usage occurs in a variety of places
    59% of Americans check email while in the bathroom*
    Difficult to recreate the true experience in a lab setting
    Create opportunities for feedback in context of usage
    Conduct field studies on beta products
    Source: Nielsen Mobile 2008
  • 52. Challenges
    Varied Carrier Network Coverage
    Network coverage is inconsistent; difficult to predict accessibility during lab testing
    Difficult to validate that out-of-town lab truly has adequate coverage
    Visit facility with colleagues who have various carriers to confirm network availability in lab
    From out-of-town labs, request:
    Reference clients from last mobile studies
    List of carriers confirmed to have consistent coverage
    At a minimum, check carrier coverage in various cities online
  • 53. Screen during recruiting for:
    Unlimited data plans
    Willingness to send / receive text messages during study
    Rescreen again upon arrival for study
    Calling / Data Plans Vary Widely
    Participants may be charged for mobile web access or sample texts sent during testing
    Participants may change data plan between recruiting interview and date of study
  • 54. Challenges
    Varied User Experience
    Most mobile device owners only utilize a small percentage of total device capabilities
    43% of mobile subscribers do not use text messaging on a regular basis*
    Users may have no experience typing in a web address to access a site or with sending SMS messages (which may be critical to your study)
    If using a prototype phone for downloadable app, users may not be comfortable with it
    Pre-test participants to ensure appropriate device experience
    Mobile address typing
    Send SMS to recruiter
    Spend 2-5 minutes explaining how prototype phone works
    * Source: Nielsen 2008 ‘The Short Code Marketing Opportunity’
  • 55. Other Helpful Tips
    Tell participants to bring recharger
    Simplify language (e.g., SMS vs. text message)
    Have a prototype strategy
    Back-end SMS text simulation
    Easy URL for access
    Back-up paper prototypes
    Index page for easy access to alternative flows
    Have plan for interruption from phone call or text (resetting session variables)
    Plan for getting screenshots for your report
    iPhone: Press hold button and home button at same time (saves into photo gallery)
    Use documented comps
  • 56. Q&A
  • 57. Thank You.
    Amy Buckner
    Managing Partner & Co-Founder, AnswerLab
    Kris Mihalic
    Head of Mobile User Experience and Design Research, Yahoo!
    Twitter: suikris