Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

To Fly or Not to Fly? How to Use Remote Techniques for Moderated Research on Mobile Devices to Stay Off Airplanes and Save Clients' Money


Published on

Online screen sharing tools have changed our research toolkit. Now we can conduct research faster and more cost effectively using screen sharing tools and webcams.

And then came mobile devices. To see people interact with their smartphones and tablets, we had to be in person. Back on planes!

Now it's possible to conduct multi-channel research remotely Cash- and time-strapped clients are hungry for this affordable, fast solution. It's not easy (and it's not right for every project), but you should know how to do it for projects where it's a good fit.

In this session, we'll discuss

pros and cons of each approach,
lessons learned,
when remote multi-channel research is a good idea (& when it's not), &
hot tips on how to effectively conduct research remotely on mobile devices.

  • Be the first to comment

To Fly or Not to Fly? How to Use Remote Techniques for Moderated Research on Mobile Devices to Stay Off Airplanes and Save Clients' Money

  1. 1. 1BellaVia Research ©2015 Julie Francis, UX Researcher & Founder / BellaVia Research Alex Berg, Director of Strategy & Analytics / Fell Swoop (formerly VP Product / Rebecca West, Global Vice President, Research / Civicom® June 29, 2015 (Abridged) Moderated Remote Research on Mobile Devices (aka, Stay off planes & save client money!) UXPA 2015
  2. 2. 2BellaVia Research ©2015 Who we are Julie / Founder, BellaVia Research • 20 years in research / Focus on UX Research since 2000 • Launched BellaVia in 2001, post Creative Good • Santa Cruz, CA /, 831.454.8217, @BellaVia Alex / Director, Strategy & Analytics, Fell Swoop • Former VP Product • 20 years in Product Mgmt. Leadership roles with, Blue Nile, Expedia. • Seattle, WA /, 206.310.9959, @Fell_Swoop Rebecca / Global Vice President, Research, Civicom® • 25+ years in research, technology support and product/service development • Investor in Civicom and Member, Board of Directors • New York/Greenwich CT / 203.413.2423, @civicom
  3. 3. 3BellaVia Research ©2015 So many UX research platforms! Today’s focus: Moderated •Not unmoderated, and Remote •Not in-person, and Real-time •Not after-the-fact recordings, and  Visibility to mobile devices •Not just computers
  4. 4. 4BellaVia Research ©2015 Our viewpoints Alex Client perspective: What needed & Why8 20 minutes Julie Researcher perspective: Options, Pros & Cons, Tips & Tricks 30 minutes Rebecca Technical perspective Demo 10 minutes
  5. 5. 5BellaVia Research ©2015 Why was research so essential to Buying a ring is a very unique purchase. They are expensive. There are a lot of options in stores and online. It’s not like buying a t-shirt! Understanding the dynamics at play were critical – particularly with Ritani’s unique offering.
  6. 6. 6BellaVia Research ©2015 Why was research so essential? It’s an emotional experience. For everyone.
  7. 7. 7BellaVia Research ©2015 Why was research so essential? There’s lots to do. And a lot of advice. And that takes time. GENTLEMEN'S ENGAGEMENT RING BUYING GUIDE Now that you're ready to get engaged, it's important to ensure that you're going to get the most value for your money. Doing research, looking online, talking to friends to see how much they spent on rings or how much diamonds cost and talking to a jeweler you trust are all great starts towards becoming an informed buyer. When we're talking about buying wise, we also recommend considering all the components that go into a ring purchase. For example, as part of setting a budget, you might want to set separate budgets for the diamond, for the engagement ring setting, for your intended's wedding band, and potentially for your own future wedding band. Visit a reputable jewelry store, and have the associate guide you through each step. Have them explain the difference in the service, have them explain the difference between platinum and gold, have them talk you through every question that you have. Buying wise means working with a jeweler you trust, who is going to stand behind your purchase(s) for the lifetime of the piece so that you know you have the very best quality that your money can buy. Koerbers Fine Jewelry
  8. 8. 8BellaVia Research ©2015 Why was research so essential? The experience occurs over time & across channels, & it is very hard to measure. There’s no tidy purchase funnel. wanted to better understand the role of mobile devices for ring research and purchasing. Decision & purchase computer tablet smartphone store
  9. 9. 9BellaVia Research ©2015 Why not in-person research? Money.  Hotels and airplanes are expensive.  Longitudinal nature of ring- buying required multiple interviews (multiple trips).  Multi-city for regional sensibilities, in-store visit partners (long trips).  Start-up budget
  10. 10. 10BellaVia Research ©2015 Why not in-person research? Observation.  Remote research is easy to share. “Hidden” research offers less value to the team.  Each team member got two users for the entire journey • 3 interviews over a few months, 4.5 hours per participant  Active participation was key.
  11. 11. 11BellaVia Research ©2015 Why not unmoderated remote research?  It’s a longitudinal purchase. People are forgetful.  It’s emotional. Not all guys readily share their feelings.  It’s not linear. Many factors involved. We needed the ability to easily go off script and go deep. How do I pick a diamond? Where do I start? Who can I trust? What will she like? Depth of insights.
  12. 12. 12BellaVia Research ©2015 needed: Remote, webcam interviews • To include geographically distributed participants (nationwide) Real-time visibility to mobile devices • Computer + tablet + smartphone Moderated, in-depth exploration • Not unmoderated, not task-based research, not self- reported diary ? But how?
  13. 13. 13BellaVia Research ©2015 The problem * Moderated and real-time
  14. 14. 14BellaVia Research ©2015 The problem We need to observe it remotely & real-time and talk about it
  15. 15. 15BellaVia Research ©2015 What we did 3 rounds of research (longitudinal purchase), •>4 hours of interviewing each 14 ring shoppers, early in funnel •Wide geographic mix And we found an approach for remote moderated research with visibility to mobile devices!
  16. 16. 16BellaVia Research ©2015 Landscape: Remote Research Computer online Mobile devices moderated Un- moderated If they can do it on mobile, they can do it on desktop, too
  17. 17. 17BellaVia Research ©2015 Considerations when choosing a solution Interactions? • How important is it to see fingers on screen? Non-verbals? • Do you need to see the user’s face? Stimuli? • Mobile web? • Apps? Platform? • Android? • iOS? Who are your users? • Tech savvy? Not?
  18. 18. 18BellaVia Research ©2015 Option 1: IT Support Tool: e.g., Bomgar How it works: Participant installs Bomgar app. Enters session number provided by researcher. Uses Bomgar browser to access site or Android app. Very clear view of device Affordable if regularly used: single- user license: >$1100 per year Cannot test iOS apps: only Android apps (and mobile web) Slow screen refresh makes it hard to see interactions Not designed for research (odd verbiage for participant) Similar option: TeamViewer Researcher view Thanks to Nyle Feist of UX Guys for the demo!
  19. 19. 19BellaVia Research ©2015 Option 1: IT Support Tool: e.g., Bomgar Cannot see finger interaction Cannot see user’s face Bomgar browser eats screen real estate (address bar at top, action buttons at bottom) via Bomgar app Participant view
  20. 20. 20BellaVia Research ©2015 Option 2: document camera, e.g., Ipevo How it works: Participant plugs in external USB- Document camera to their computer. Starts screen- sharing tool. Great if you want to see the interaction (fingers touching screen) Very easy to switch between devices BUT…you can’t see the user’s face. Can’t use a document camera and webcam simultaneously  Note: According to UXPA 2015 session attendees, you can see the user’s face if the document camera is used in conjunction with a webcam and Morae Observer. A Morae license is $2,000 per year. Focus issues Logistics / fulfillment hassle: Need to send each participant a document camera (budget: $100 per device with shipping). Vendors like can help & have a panel. Photo from Thanks to session attendees for the info about document cameras used in conjunction with Morae Observer!!
  21. 21. 21BellaVia Research ©2015 Option 3: laptop love! How it works: Participant turns their laptop around & “hugs” it. Use webcam & screen sharing tool to show their mobile device. Free! (if user has a webcam) Can see the interaction (fingers touching screen) Very easy to switch between devices BUT…you can’t see the person’s face when they are snuggling their laptop Challenging for participant to keep device in webcam view (participant can’t see it, so researcher has to remind them) Image quality can be an issue (focus issues, low quality webcams)
  22. 22. 22BellaVia Research ©2015 Option 4: Lookback’s How it works: Participant installs on Mac. Signs in via Twitter credentials. Connects device via USB cord. Clicks “stream.” Shares link with researcher. Free download! (currently) Very clear image Streams the video & audio If used with screen sharing tool, can also see the participant’s face
  23. 23. 23BellaVia Research ©2015 Option 4: Lookback’s Only works for participants who have Apple devices and a Mac running recent software •only iOS 8+ (80%) •only Mac OSX 10.10+ (Yosemite) (60%) Participant must have Twitter account Cannot see finger interaction - Note: A similar option was volunteered by a UXPA 2015 session attendee. X-Mirage is another iOs- only option that he feels works well. He recommended Zoom for screen sharing.* Thanks to the session attendee who recommended x-mirage and zoom!
  24. 24. 24BellaVia Research ©2015 Option 5: AirServer + AirPlay How it works: Participant installs & launches AirServer on their computer. Swipe up on iOS device to start AirPlay. Device is wirelessly mirrored to computer screen. Starts screen sharing tool and webcam. Shares screen. Affordable: $15 per download Very clear image Can see the participant’s face via webcam (screen sharing tool) User can easily toggle between PC, iPhone, iPad
  25. 25. 25BellaVia Research ©2015 Option 5: AirServer + AirPlay Only works for iOS Only works on iPhone 4S+ and iPad 2+ Users may need a second phone (some iPhones cannot mirror & be used as a phone simultaneously)
  26. 26. 26BellaVia Research ©2015 Option 6: Reflector How it works: Participant installs & launches the Reflector app on their computer. Ensures computer and device are on same WiFi network. Shares screen via screen sharing. Starts webcam. Android: launch Screencast app. (older Android devices require ChromeCast) iOS: Swipes up to launch AirPlay Wirelessly mirrors device to computer screen.
  27. 27. 27BellaVia Research ©2015 Option 6: Reflector Very clear view of device Free! 7 days free per user Visibility to everything (apps, mobile websites, native phone function) Ability to see user’s face via webcam & screen sharing Cannot see finger interaction on screen Users may need a second phone (some phones cannot mirror & talk simultaneously)
  28. 28. 28BellaVia Research ©2015 Reflector: In the field  has been in the field using this technique this week. It’s going great! They aren’t crazy about the Reflector “watermark” – probably because they are using the free version? Thanks to Marianne Connor and Amaya Becvar Weddle from Mind Body Online for this participant quote! It was fine, I’m really, really nervous with tech stuff, so the fact that this worked is impressive. That never happens for me. Ever. Participants say it’s easy!
  29. 29. 29BellaVia Research ©2015 Pros & Cons: Things are changing rapidly! IT Support tool Document camera Laptop hug AirServer Reflector See face?   If used with Morae observer     See interaction?       Apps?  Not on iOS      Android & iOS?     iOS only  iOS only  iOS + Android Clarity Clear, but slow refresh Focus can be an issue Focus can be an issue Excellent Excellent Excellent Cost $1100 single license / year <$100 per camera Free Free (for now) $15 per license Free for user for 7 days What else? Tech support tool Can’t see full screen Logistics (sending cameras) Morae is expensive! $2,000 license Awkward! Only works with recent iOS & Mac updates Requires twitter account iPhone users may need second phone for voice iPhone users may need second phone for voice
  30. 30. 30BellaVia Research ©2015 Lessons Learned & Tips for Success Schedule a “tech check” before the interview • Before the interview, help the participant get the technology set up. Do it before the interview, so you don’t waste the time of client observers. • Consider bringing in tech help. Make time for tech checks • Our projects move at Mach speed, but add 2-3 days before interviewing to make sure all your participants are ready on the technology front. • If all goes well, each tech check will take 10-20 minutes. Incentivize for tech check • What would you pay for a 30-minute interview? Add that to the incentive. Plan for attrition • You may lose more than normal due to tech issues / frustrations. • Over-recruit! e.g., For 10 interviews, recruit 12-14
  31. 31. 31BellaVia Research ©2015 Lessons Learned & Tips for Success Ask users to have a second phone available • You’ll know from the tech check if the user can talk & mirror simultaneously. Don’t be caught off guard! Not a suitable methodology for all participants • Works great for tech savvy participants • Didn’t work as well for a general population study about smartphone app abandonment. Less tech-savvy participants were frustrated and we lost several during the tech checks. Solutions are getting easier, but...still may not be ideal for less tech savvy participants. Use it as part of a hybrid, mixed-method approach • Be careful that this methodology doesn’t skew your sample towards tech savvy participants. • Consider using as part of a hybrid methodology: some remote interviews for geographic mix + in-person interviews to ensure tech savvy mix.
  32. 32. 32BellaVia Research ©2015 Technical Perspective: Reflector During the Recruiting Process:  Confirm Mobile Device Meets Specs: IOS 7 or higher Android, 4.4.2 or higher  Confirm Computer Meets Specs: Apple: OS X 10.7.5 or Later PC: Windows Vista, Windows 7 or Windows 8 (Windows 8 RT is not supported)  Confirm Webcam: If it is built in or external, and if one needs to be sent  Confirm Network Connection: Wireless 802.11n or Ethernet highly recommended Both computer and mobile device must be on the same connection No use of free WiFi systems Beware of office firewalls Disconnect from VPNs (eats up too much bandwidth)
  33. 33. 33BellaVia Research ©2015 Technical Perspective: Reflector Conduct the Tech Check: • Send email to respondent with link tech check room • Schedule tech check session • Technician walks respondent through steps involved in sharing mobile device on computer screen (turn on webcam, start screen sharing, download Reflector, enable Reflector, connect mobile device) • Respondents test should occur on the actual equipment they will use for the interview computer with webcam (detachable or built-in) plus their mobile device/s • Make sure respondent understands they will need to replicate equipment and set up at actual interview • Confirm interview date and time • If recruit is accurate, tech check is 10 to 20 minutes per respondent Tech Check Tips : • Variations in mobile device drives ability to connect to Reflector: iOS users use Airplay to connect to Reflector Android 5.0 Lollipop device users cast screens to Reflector Devices running Android 4.4.2 use the Chromecast App to connect to Reflector • Recruiter must be tech savvy and thorough in recruiting respondents that meet the tech specs or attrition rate will be high during the tech check and respondents will have to be re-recruited. • Unforeseen circumstances that require rescheduling include sudden loss of internet connection, computer crash, weather, stolen or lost device, etc.
  34. 34. 34BellaVia Research ©2015 Methodology Benefits • Deep learnings without breaking the bank with travel Affordable • Savvy population was not afraid of technology and trying new things. They were curious about the approach and invested in making it work. • Ring shopping is (often) secret! Easier to hide phone interview than in-home visit + shopalong. Good fit for affluent ring-buyers • Wide audience at could observe sessions real-time. Built empathy for customer among a diverse set of employees. • Empowered the team with actionable information, created appetite for findings and recommendations. • Seeing is believing - direct exposure to *how* customers use devices on our site and competitors surfaced lots of insights for the team. Research observation helped with buy-in
  35. 35. 35BellaVia Research ©2015 Video Demo! Reflector Mirroring iPhone If you’d like to see the video demo, please contact Rebecca at Civicom
  36. 36. 36BellaVia Research ©2015 Questions? Alex client perspective Julie Researcher perspective julie@ Rebecca Technical perspective