Handalian shelkey mobile-presentation_upa_2010


Published on

Considerations to keep in mind when embarking upon mobile usability testing, either on-device, or on-computer. Presented at the Boston UPA Mini-conference 2010.

Published in: Technology
1 Comment
  • Hi Lisa, enjoyed the presentation on Wednesday. Are you able to provide any insight into the specific methods (i.e. software emulator, sled mounted cam, stationary cam, etc.) BofA used to test their mobile applications? Thanks, John
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • What gave birth to this talk: Participating in a lot of the mobile meet up activities that are included in the appendices, I have noticed that developers of the apps are in the dark ages of usability --
  • Keep repeating the word ‘consideration’ bec the goal of this presentation is not to tell you how, but to present potential issues that may come up and need to be factored in when planning a test on mobile
  • Add note about Morae “click” markers. Lots of moving parts to consider
  • Could be remote or in lab
  • Take out emulator, code on iphone is faked out but happens on website. Dropbox. Learnin about browser – don’t take out too much chrome so it takes away overall app-ness
  • Loaded on desktop and iphone, create connection – can tether. Dependin on functionality, can control desktop via iphone or vice versa. And/or it sees the iphone on the desktop.
  • Handalian shelkey mobile-presentation_upa_2010

    1. 1. Yes, but is there an app for THAT? UPA Boston Mini-Conference Bentley University June 9, 2010 Lisa Renery Handalian & Brent Shelkey Usability Engineers Bank of America User Experience An Overview of User Research & Testing Methods for Smart Phone / Mobile Devices
    2. 2. Setting Expectations <ul><li>This presentation does contain: </li></ul><ul><ul><li>Suggestions on how to adapt existing evaluation practices to smart phone testing </li></ul></ul><ul><ul><li>Handheld-specific issues and challenges to consider as you plan your research (a checklist of sorts) </li></ul></ul><ul><ul><li>Practical tools and technology resources for prototyping and testing on computer and on device </li></ul></ul><ul><ul><li>Unbalanced focus on iPhone, with far less attention to other devices and platforms </li></ul></ul><ul><ul><li>Focus on app implementations, rather than mobile web. </li></ul></ul><ul><li>But it doesn’t include: </li></ul><ul><ul><li>Usability heuristics or design best practices for mobile applications </li></ul></ul><ul><ul><li>Endorsement of any products or methods. </li></ul></ul>
    3. 3. Non- & Pre-Test Research Tools & Methods <ul><li>Manufacturer design guidelines / heuristics See Appendices for iPhone, Blackberry and Droid “versions” </li></ul><ul><li>Secondary and Market Research Forrester, Nielsen, CHI and UPA Proceedings, etc. See links at the end of this presentation. </li></ul><ul><li>Competitive Landscape and Feature Analysis Product descriptions, customer ratings/reviews in App Store and iTunes. </li></ul><ul><li>Expert Reviews with mobile screen captures “Sleep + Home” on the iPhone to create jpegs </li></ul>
    4. 4. Test Planning Considerations <ul><li>Recruiting </li></ul><ul><li>Devices </li></ul><ul><li>Test Approaches </li></ul><ul><li>Recording & Sharing </li></ul><ul><li>Prototype Considerations: On Computer (Emulation) </li></ul><ul><ul><li>Test Set Up (Remote and In Lab) </li></ul></ul><ul><li>Prototype Considerations: On Device </li></ul><ul><ul><li>Test Set Up (Attached Camera, In Lab) </li></ul></ul>
    5. 5. <ul><li>Android Fragmentation: What’s the impact of having a user familiar with the “Nexus One” participate in a test based on the “Incredible”? You may want to ask participants to email a picture of their phone as part of the screening process. ANDROID OS DIVERSITY </li></ul>1. Recruiting Screening for mobile tests can pose specific challenges: Do users differentiate between phone make, model, carrier, and OS? <ul><li>User Tech Savvy-ness: If familiarity is an important aspect of your test, this may point to a certain level of technical knowledge on the part of your user. Where does someone who knows their phone’s specs inside and out fall on your study’s “Geekiness Spectrum”! </li></ul>Motorola “Droid” HTC EVO HTC Incredible Verizon Sprint Verizon
    6. 6. 2. Devices <ul><li>Prototypes should reflect the diversity among devices: iPhone, Blackberry, Palm Pre, Windows Mobile, and the myriad Droids. </li></ul><ul><ul><li>For HTML click-through tests (aka computer-based simulations ), specifically designed prototypes and conditions will yield most realistic results. Be prepared to budget resources to create different prototypes to accommodate Blackberry’s smaller screen and device-based scrolling mechanisms, vs. the iPhone’s gestural navigation and on-screen tap type keyboard, etc. </li></ul></ul><ul><ul><li>If possible, plan ahead for multiple test formats through CSS. JQuery, HTML and CSS can take you far. Ideas at http://tinyurl.com/mbmvv6 . </li></ul></ul><ul><ul><li>And speaking of planning ahead , designing in vector-based images from the start will help you resize and repurpose your visuals from computer mockups to different devices to iPad interfaces. Desktop to iPhone size specs are provided later in this presentation. </li></ul></ul>
    7. 7. 3. Test Approaches Will your test be on paper,* a computer-based simulation, or an on-device “app” emulation – remote, in a Lab, or in the field? On the user’s own phone or using Lab equipment? http://judykeeley.com/portfolio/UPA-2009/Mobile_Testing_Methods.ppt At last year’s Mini-UPA Boston, Yang & Keeley (2009) detailed the pros and cons of in-person/remote testing, based on secondary sources as well as their own experiences at Verizon. <ul><li>In-person Testing on Device: Lab tests on mobile phone with cameras are cheaper and easier to set up than field studies. </li></ul><ul><li>Remote, on Computer Testing : Using simulations found to be easier and more cost-effective than using user’s own phone (and recording his/her actions). Yield a large percentage of usability problems despite less authentic context. </li></ul>* Link to paper prototype stencils provided in Appendices.
    8. 8. 4. Recording & Sharing <ul><li>What options do you have to view and record the user interactions? Will remote observers be able to watch along? </li></ul><ul><ul><li>Screen Sharing (Morae, WebEx, MS LiveMeeting) are typically used to conduct remote tests while broadcasting user interactions out to stakeholders. </li></ul></ul><ul><ul><li>Camera Systems involve many different ways to: </li></ul></ul><ul><ul><ul><li>Capture interaction with the phone </li></ul></ul></ul><ul><ul><ul><li>Isolate/stabilize the input so that cameras may accurately view the interactions </li></ul></ul></ul><ul><ul><ul><li>Angling mechanisms to allow cameras to view the phone’s input </li></ul></ul></ul><ul><ul><li>Murphy’s Law dictates that the more complex the technology set up, the more that can (and will!) go wrong. Be sure to budget for extra time pre-study -- and between session – to ensure that all these moving parts play nicely together. </li></ul></ul><ul><ul><li>Note: Don’t be fooled by the WebEx iPhone app … iPhone user can view the meeting but not interact on the screen! </li></ul></ul>
    9. 9. 5. Prototype Considerations: On Computer <ul><ul><li>Computer-based simulations are prototypes accessed on traditional computers (e.g. HTML click-throughs). As Keeley’s presentation indicated, most usability issues will be exposed through a computer-based test. </li></ul></ul><ul><li>Gestural navigation doesn’t translate well to keyboard/monitor. Mouse clicks in lieu of taps and full keyboard typing skew the experience. Will provide useful input on app structure, task flow and visual language, and are easier to modify on the spot. </li></ul><ul><li>Flash to the rescue? Flash may provide realistic interactions, but it won’t run on an on-device test (iPhone, mainly, though Blackberry is also a stretch) </li></ul><ul><li>Desktop icon pixels vs. iPhone: Home screen icon (72hx72w vs. 57hx57w) and app once entered (1004hx768w vs. 460hx320w). </li></ul>Part of the issue with mocking up ‘apps’ for test drives is that native coding is typically outside the in-house skill set (Objective C/Cocoa, and SDKs). If you opt for a computer-based test (stand-alone or in conjunction with an on-device test), here are some things to think about (SDKs notwithstanding):
    10. 10. 5A. On Computer Test Set Up Computer Simulation / Click-thru (loaded onto computer, or accessible via website if remote) Morae, WebEx, LiveMeeting, etc. Web cam on face Participant Stakeholders UE Record Report
    11. 11. 6. Prototype Considerations: On Device <ul><ul><li>If you have devices in the Lab, you may want to test your prototype in the actual mobile context. If your app is being outsourced to a vendor, ensure that user testing is built in to their development cycle. </li></ul></ul><ul><ul><li>Device-based prototypes combine screenshots and code to provide a realistic, seemingly interactive user experience. </li></ul></ul><ul><ul><li>The prototype is actually accessed via a text link (SMS or emailed), which will launch the experience on a mobile web browser . If all goes well, the user is none the wiser. </li></ul></ul><ul><ul><li>Goal is for prototype to honor the look and behavior of the native application. See Appendices for resources. </li></ul></ul><ul><ul><li>The DeviceAnywhere Test Center is a licensed solution that provides test environments across every handset built. See Appendices. </li></ul></ul>
    12. 12. 6. On-device Attached Camera Set Up LAB Stakeholders Morae, WebEx, LiveMeeting, etc. Record Report Simulation accessed on device from browser via Lab wi-fi Attached camera Morae, Camtasia will provide picture in picture and best recording tools In-room camera face Participant UE
    13. 13. Appendices <ul><li>Screen sharing and remote access matrices </li></ul><ul><li>Cameras and projection devices </li></ul><ul><li>Templates and prototyping tools </li></ul><ul><li>Publications </li></ul><ul><li>Web resources </li></ul><ul><li>Twitter feeds </li></ul><ul><li>Boston Mobile Community </li></ul>
    14. 14. A. Screen Sharing Applications <ul><li>Here are some other screen sharing apps that we haven’t tried out, but that are touted (at least) to allow you to viewing, interacting with, controlling, and remotely viewing the device from another computer. </li></ul>Name Platform Features/ Issues Cost Jaadu VNC Mac/Windows Supports touch screen gestures to interact with other OSes (scrolling webpage, etc.) $26 Mocha VNC Lite Mac/Windows Can interact with current screen sharing apps loaded on client computers (Apple Remote management, etc.) Free- $6 VNSea Mac/Windows Open source project may have additional bugs Free RealVNC Mac/Windows May be used with other phone types as well. (Blackberry, Symbian, Win Mobile) $30-50 WebEx PCNow Mac/Windows Can join WebEx meetings via iPhone/Blackberry and some other smart phones. Cost of WebEx
    15. 15. Wireless camera Movable “Sled” with fixed camera Head/Helmet Cams Wired Camera with “sled” Wired camera, fixed surface B. Cameras: Device Screen Cameras
    16. 16. Little Spring Design Noldus Camera Netherlands Usability Lab Usability Sciences Gotomobile (LampCam) B. Cameras: Device Screen Cameras
    17. 17. B. Cameras: Web Cams / Cameras <ul><li>HUE HD </li></ul><ul><li>$32.99 </li></ul><ul><li>1.3 Megapixel </li></ul><ul><li>640x480 Resolution </li></ul><ul><li>USBOutput </li></ul><ul><li>X10 Camera </li></ul><ul><li>$100.00 </li></ul><ul><li>Needs receiver as well </li></ul><ul><li>Logitech Pro9000 </li></ul><ul><li>$70-90 </li></ul><ul><li>2 Megapixels HD Video </li></ul><ul><li>720P HD Resolution </li></ul><ul><li>Carl Zeiss Lens </li></ul><ul><li>USB Output </li></ul>
    18. 18. B. Cameras: Projectors / Document Cameras <ul><li>Buhl Projector </li></ul><ul><li>$309 </li></ul><ul><li>¼” CCD, 470 TV Lines </li></ul><ul><li>RCA & USB Output </li></ul><ul><li>Avermedia Document Camera </li></ul><ul><li>$470 </li></ul><ul><li>1/2” CCD, 470 TV Lines </li></ul><ul><li>2x Optical Zoom, 8x Digital </li></ul><ul><li>RCA, USB, VGA, DVI, S Video Output </li></ul><ul><li>Ken-A-Vision </li></ul><ul><li>$344.99 </li></ul><ul><li>1/3” CCD, 1280x960 </li></ul><ul><li>USB Output </li></ul>
    19. 19. C. Templates and Prototyping Tools <ul><li>Funky Space Monkey iPhone Stencils for Paper Prototypes http://bit.ly/52MINM </li></ul><ul><li>Fireworks toolkit for creating iPhone UI mockups http://blog.metaspark.com/2009/02/fireworks-toolkit-for-creating-iphone-ui-mockups/ </li></ul><ul><li>“ We’re Full of It” (blog) http://fullofdesign.com/posts/iphone-app-wireframe-template/ Portrait and landscape oriented iPhone templates. Pretty basic. </li></ul><ul><li>Prototshare: Secure prototype sharing application (online, Mac and PC) http://www.protoshare.com/2009/06/17/native-iphone-prototypes-with-protoshare/ Link points to the demo of a container in which you build your wireframes to look and behave like a native app when viewed through Safari on your iPhone. $29/month after one month trial. </li></ul><ul><li>MobiOne: iPhone and Palm Pre (!!) Emulator for Windows http://www.genuitech.com/mobile - Free download. “Easy to get mobile app prototypes up and running in minutes – even without technical know-how!” </li></ul><ul><li>iPhone-ized – General iPhone Resource site http://iphoneized.com/2009/11/21-prototyping-mockup-wireframing-tools-iphone-app-development/ </li></ul>
    20. 20. C. Templates and Prototyping Tools ( 2) <ul><li>Interface: “The Mock Up and Prototyping Tool for the iPhone” http://lesscode.co.nz/interface?ref=fusion “You can create and preview mockup screens for your iPhone prototype using native iPhone OS widgets” (at least according to their site). Demo at this site. Can buy tool for $9.99. </li></ul><ul><li>HTML - CSSP - JQUERY on-device Prototype Simulator http://www.webresourcesdepot.com/create-mobile-applications-with-just-html-css-jquery-jqtouch/ Wrap the application pages in the XCode shell. Tell the iPhone/iPad to honor the motion detection for native applications. Tell the iPhone/iPad to discard the Chrome navigation. All other behaviors work normally. Magic! </li></ul><ul><li>MobiGenie: Mobile Application Development Resource Site http://www.mobigeni.com/2010/02/10/iphone-mockup-wireframe-and-prototyping-tools-for-iphone-developers/ iPhone prototyping tools - - site contains iPad and other resources as well. </li></ul>
    21. 21. Device Anywhere Test Center http://www.deviceanywhere.com/mobile-application-testing-web.html (From the site): Now you can easily test your mobile websites across all your target handsets and mobile networks – from the convenience of your desktop computer, without owning a single device! Provides access to 2000+ real handsets connected to over 30 live networks, all over the internet. DeviceAnywhere Test Center. With remote access you can: C. Templates and Prototyping Tools ( 3) <ul><li>Perform any phone operation as if the mobile handset was physically in your hand </li></ul><ul><li>Observe all outputs from the handset, including display, video, and audio </li></ul><ul><li>Test mobile website display across any mobile device screen-size with accurate pixel-by-pixel screen images </li></ul><ul><li>Perform full mobile tests of data application, and integrated data including video, downloads, or messaging applications </li></ul>LiveView for iPhone (and iPad): Screen Caster / Emulator www.zambetti.com/projects/liveview/ Really ingenious (we have tried it) tool for “quick and dirty” iterative prototyping. Requires downloading on both Mac (no PC), and from the App store onto your iPhone. Frame the emulator ‘casing’ that appears on your monitor around the prototype. With the iPhone and Mac “tethered” over wi-fi, you can interact with the prototype on the iPhone, while it actually runs on the desktop.
    22. 22. D. Publications <ul><li>Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing. Kaikkonen, et al. Journal of Usability Studies ( November, 2005). </li></ul><ul><li>It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field. Nielsen, et al. NordiCHI Proceedings (2006). </li></ul><ul><li>Handbook of Research on User Interface Design and Evaluation for Mobile Technology . Lumsden, ed. University of Salzburg (2009). </li></ul><ul><li>Mobile Testing Methods. Yang & Keeley, UPA Conference Paper (2009). </li></ul><ul><li>Usability Testing of Mobile Devices: A Comparison of Three Approaches . Betiol & Cybis, INTERACT Proceedings (2005). </li></ul><ul><li>CHI 2009 Workshop: Mobile User Experience Research: Challenges, Methods & Tools. </li></ul>
    23. 23. D. Publications, cont. <ul><li>Capturing Mobile Phone Usage: Research Methods for Mobile Studies. Wei - Google, Inc., IEEE (2007). </li></ul><ul><li>Flexible Hardware Configurations for Studying Mobile Usability , Oulasvirta & Nyyssönen, Journal of Usability Studies (February 2009). </li></ul><ul><li>My Phone Is My Castle: Facing the Special Challenges of Mobile Usability Studies. Duda, CHI Proceedings (2009). </li></ul><ul><li>Overcoming Mobile Usability Testing Hurdles: A Mobile Payments Case Study, PayPal . Griffin, UPA Proceedings (2007). </li></ul>
    24. 24. E. Web Resources <ul><li>iPhone Usability: How People Really Use the iPhone http://www.uselog.com/2008/11/usability-test-how-people-really-use.html </li></ul><ul><li>iPhone and iPad UX Reviews http://www.iphoneuxreviews.com/ </li></ul><ul><li>Mobile Design / Testing Resources http://www.deyalexander.com.au/resources/uxd/mobile.html </li></ul><ul><li>Small Surfaces: A site about design for mobile technology http://www.smallsurfaces.com </li></ul><ul><li>Apple’s Human Interface Guidelines (HIG) for the iPhone: http://developer.apple.com/iphone/library/documentation/userexperience/conceptual/mobilehig/Introduction/Introduction.html </li></ul><ul><li>Blackberry’s: http://docs.blackberry.com/en/ </li></ul><ul><li>Droid’s: http://developer.android.com/guide/practices/ui_guidelines/index.html </li></ul>
    25. 25. E. Web Resources, cont. <ul><li>Insightful Slideshare presentation highlighting UX issues in mobile http://www.slideshare.net/Rachel_Hinman/mobile-user-experience-what-web-designers-need-to-know </li></ul><ul><li>Mobile Best Practices for Accessibility from the WC3 http://www.w3.org/TR/mobile-bp/ </li></ul><ul><li>App Store Secrets http://www.pinchmedia.com/appstore-secrets/ Data / analytics drawn from thousands of downloads. Learn how users actually interact with the app store! </li></ul>
    26. 26. F. Twitter Crowd Sourcing <ul><li>Tweeps </li></ul><ul><li>@Appleinsider </li></ul><ul><li>@DevAnywhere </li></ul><ul><li>@Iphoneized </li></ul><ul><li>@Iphonetastic </li></ul><ul><li>@keysoze </li></ul><ul><li>@Mobimarketnews </li></ul><ul><li>@mrudnick </li></ul>Hash Tags #iPhone #mobile #droid #tuaw And so forth…
    27. 27. G. Boston Mobile Community <ul><li>Boston Meet Ups for Developers, Designers, and more … </li></ul><ul><ul><li>Mobile Mondays: www.momoboston.com </li></ul></ul><ul><ul><li>Drinks on Tap: http://www.drinksontap.org/ </li></ul></ul><ul><ul><li>“ Future of Mobile” meet up group: http://www.meetup.com/Boston-Mobile-App-Developers-iPhone-Droid-iPad/ </li></ul></ul><ul><ul><li>“ New England Blackberry Developers” meet up group: http://www.meetup.com/New-England-BlackBerry-Developers/ </li></ul></ul>