How to Run User Testing Sessions on Mobile

How to Test on Mobile
Moderated and Unmoderated

Takeaways
1) Methods for user testing on Mobile (moderated and unmoderated)
2) Tradeoffs associated with each
3) Some of the free tools you can use
4) Trends in UX and implications moving forward
Analytics tells you what, Userlytics tell you why.

Who is Userlytics?
User Testing Platform founded in 2009 with a Proprietary International Panel of Participants
We’ve run tens of thousands of usability and user testing studies that have been:
- Moderated & Unmoderated
- Remote & In Person
- Mobile & Desktop & Tablet
- Designed by our clients or co-designed with our clients or entirely designed by us
Some of our clients:

Who Am I?
Mad researcher
Masters in Market Research and Consumer Behavior at IE Business School
Webinar Lead at CareerFoundry, an online web and UX boot camp
VP of European Business Development at Userlytics

What Can Be Done With Mobile Testing?
What is the importance?
Is it possible to test mobile devices?
How do you run these tests?
Possible to do moderated testing on mobile devices?
What methods are there?
What limitations and challenges are there?

Why Test on Mobile Devices?
Between 52.2%1 and 57%2 of all website traffic worldwide was generated through mobile phones
Some estimates are as high as 70%3
By 2020 Forrester estimates that 41%4 of in store sales will be influenced by the internet, and Statista
estimates that by the same year online sales + in store sales influenced by the internet will make up
roughly 50%5 of all purchases in the US.
eMarketer reports that “global [mobile] commerce sales rose 40.3% last year to $1.357 trillion,
representing 6.0% of total retail expenditures” and that “[mobile] commerce accounted for 58.9% of
digital sales.”
1 Statista, 2018 4 Forrester, 2017
2 Brightedge, 2017 5 Statista, 2016
3 CIODIVE, 2018 6 eMarketer, 2017

However, Mobile Testing Isn’t Keeping Up
In Userlytics, 43.26% of all tests are mobile
2019 to date, 46.98% of all tests have been mobile
While we just saw that 52% of traffic (or more) and 58% of purchases are on mobile and that online
searching habits will affect roughly 50% of all sales (online or not)

And It Shows
Reddit - 18th largest site in the world1 (6th in the US2)
1 Wikipedia
2 Alexa
Trying to watch a videoLogin - nothing loaded Video pauses every second

So, How Do We Start Testing Mobile More?
Well, there are many ways to test mobile, including:
- In-person testing in a lab setting
- In-person testing in random or neutral meeting spot
- Online platforms for unmoderated sessions
- Online platforms for moderated sessions
- Online conferencing tools
But, which one is best?

Depends On What You Need to Test
Things to take into account:
- Device: Mobile, desktop, tablet or mix? (we’ll focus on mobile here)
- Moderated or unmoderated
- In Person or remote
- Recording or no recording
- Webcam or no webcam
- Screen view or no screen view (recommend screen view for sure)
- Full solution platform or some assembly required
- iOS or Android (also, Mac or Windows)
- How will you track notes?
- Multiple participants or moderators?
- Will participants have disabilities?
- Will translations be needed?
- …. Other unique test specific actors…..

So Let’s Start Looking at the Choices

1) In Person Moderated
Moderator sitting side by side with a participant as they go through the user testing study
There can be many variations of this:
- Lab setting or natural setting
- Passive or active moderator
- All notes taken live or results monitored later by team
No tools NEEDED but can have:
- Notepad (digital or pen and paper)
- Screen recorder
- Webcam recorder
- Finger tap recorder
- Screen sharing
- User testing platform running in parallel

1) In Person Moderated
Advantages:
- High adaptability inside and between tests
- Test all mobile device types
- Greatest ability to ask follow up questions
- View face and body language and device at once
Disadvantages:
- Does not scale
- Very time consuming
- Small risk no shows
- Need to track answers and calculate results by hand (or excel)
Other:
- Can be free, but is often the most expensive methodology when sourcing participants

2) Online Remote Unmoderated Platforms
Qualitative and Quantitative digital tools that automate the:
- Sourcing of participants
- Recording of results
- Displaying of results for researchers
Platforms such as:
- User Testing
- UserZoom
- Userlytics
- Validately
- Lookback
- … and a slew of other players

2) Online Remote Unmoderated Platform
Challenge:
- Learning curve
Advantages:
- Easy to recruit worldwide
- Only methodology that scales very easily
- Fastest method to get results back
- Faster method of analyzing results
- Participants in own environment
- Many advanced features
- Aggregate SUS, NPS, SUPRQ, SEQ, Success/Failure first - then watch videos after
- Shareable highlight/lowlight reels, notes, and full projects
- Full audio transcriptions
- Tree testing and Card sorting

2) Online Remote Unmoderated Platform
Disadvantages:
- Can’t see the participant’s full device
- Need well thought out tests
- No live follow up questions (unless using branching logic - but still limited)
- Technical issues apparent when results come in
Other:
- Depending on participant profile, costs can vary widely
- Setup fees, monthly fees, yearly fees on some platforms can add up (esp on slow months)

3) Online Remote Moderated Platform
Pretty much all of the tools listed before have moderated capabilities or support moderated sessions
Advantages (same as unmoderated platforms):
- Easy to recruit worldwide
- Faster method of analyzing results
- Many advanced features
- Participants in own environment
Advantages (new):
- Strong adaptability inside tests
- Cheaper than most other methodologies (except unmoderated platforms)

Disadvantages (same as unmoderated platforms):
Disadvantages (new):
- Setup can sometimes be difficult
- Technical issues can slow down or stop a session
- Does not scale easily
- No shows can waste time

Challenge (same as unmoderated platforms):
- Learning curve
Limitations:
- Watch out for RAM usage
- Live audio + screen sharing + recording + platform + app/site/prototype (+ live webcam)
Other:
- Overall timing is faster than in person, but slower than unmoderated sessions on a platform
- Some platforms don’t allow webcam recording

4) Free or Paid Conferencing Tools (Moderated)
A conferencing tool is needed for this, such as:
- GoToMeeting
- Zoom
- Google Hangouts
- Skype
- etc….
How it is done?
- Schedule time to meet
- Get everyone to join at the right time
- Moderator on a desktop, participant on their mobile device
- Participant to share their mobile screen with moderator
- Hit record and begin testing

Advantages (when compared to in person):
- Can scale slightly faster
- Typically lower cost (user can join from their home)
- Easily add as many moderators or participants as you want (if tool allows)
Disadvantages (when compared to in person):
- Potential for disconnections
- Can’t see webcam and screen together

Advantages (when compared to remote unmoderated):
- Probe deeper and more on the fly
- Can help if there are technical issues
Disadvantages (when compared to remote unmoderated):
- More time consuming
- Slightly higher expense
- All analysis will need to be done outside the platform
- Need to track answers and calculate results by hand (or excel)
- Can’t see webcam and screen together

Notes:
- Be sure to hit record!
- Screen and audio sharing capabilities for mobile devices
- Double check the number of seats you have
Limitations:
- Watch out for RAM usage
- Live audio + screen sharing + recording + conferencing tool + app/site/prototype

5) Conferencing Tool Moderated with Webcam
How?
- Mirror mobile device to computer and run computer to computer moderated session
- Webcam recorded from the computer
- User would interact with the site/app/prototype through their mobile device

4 different variations to display mobile screen on computer:
1) iOS + Mac (easiest)
a) Connect iOS device to Mac with cable
b) Open Quicktime player on Mac
c) Start recording mobile screen
2) iOS + Windows
a) Install AirServer in Windows
b) Enable functionality of AirPlay in iOS
c) Have AirPlay look for AirServer server name
3) Android + Mac
a) Make sure Android device and computer are on same WiFi
b) Install Screen Mirror on Android device
c) Start broadcasting Android device
d) Open browser and scan QR code
4) Android + Windows
a) Same as step 3

Advantages:
- Can do a moderated session with webcam
- Less risk of crashing (due to RAM on mobile devices)
Disadvantages:
- Very technically challenging for a participant

6) Physical Device Recording (Mr. Tappy)
Great if you need to record full interaction (physical device included) and typically done in lab or with
professional testers
Advantages:
- Records full physical device (webcam can be added)
- From the users perspective
- All device types/makes/models
- Helpful for testing people with disabilities
Disadvantages:
- Expensive
- Does not scale
- Either needs to be done in lab or spend a bit per participant to send them the device
- $400+ per participant if sending new devices or $30+ per participant if reusing (shipping costs)

Extra - How to Source Participants (Cheap)

Source Participants from Facebook?
Our recommendation is to avoid for most cases
People seem to not trust ads on facebook that say “make $X today in 30 minutes”
And when a result does come in, they’re typically low quality
Participant probably just learned about user testing through the ad
New Facebook Non Discrimination Policy - need to be an approved vendor

Source Participants from Survey Providers?
Our recommendation is to avoid survey providers for the most part
Participants from survey panels are used to doing surveys, not user testing sessions. In fact, all of
these elements are barriers:
- Not used to downloading software or creating an account
- Not used to webcam, screen, or audio recordings
- Not used to speaking out loud
- Not used to having full interaction recorded
- Participants are typically just trying to speed through results

Source Participants by Creating Own Panel?
We would moderately recommend creating your own panel IF you have an initial large sample you could
pull from
How to do:
- Email blast to all (or top) customers
- Feedback popups on key pages (ex: loyalty pages)
Advantages
- Low incentives seem to work (as they feel they are helping a brand they already like)
- Appreciate recognition and sometimes don’t require an incentive
Disadvantages
- Not all testing can be done with this group (this group is already familiar with you)
- Could take a long time to build a large enough panel (so start now if you can)

Source Participants from Qualitative Providers?
We would highly recommend using qualitative providers to source participants
Several recently have been purchased, however, still a couple valuable ones out there
Advantages:
- Their participants are used to doing user testing sessions and do them well
- Fast turn around
- Multiple countries, backgrounds, profiles, etc.
- Fair price

Need It to Be Free?
Get a free conferencing tool and invite people you know to join on you through the tool
Advantages:
- Free
- Very flexible
Disadvantages:
- Very hard to get users
- Very time consuming
When to do this method:
- When you have 0 budget
- Concept validation / MVP phase
- If you are a large company with a lot of users to pool from

User Testing on a Budget?
1) Sourcing platform and conferencing tool:
- 3rd party sourcing tool
- Free conferencing tool (with recording)
- Notepad (notes)
- Excel (calculations)
2) User testing platform:
- Lot of advanced features for small additional cost
- Handles sourcing participants, note taking, and advanced features

Need to Move Fast?
User testing platforms would be the way to go:
- They have sourcing included
- Results typically come in within hours or a day
- Costs typically aren’t much higher than if you did everything yourself
Large Testing Volume?
Moderated to unmoderated:
- Start with moderated for a few results
- Make sure test script and processes are good (all branching logic follow up questions thought up)
- Then run rest of sessions as unmoderated
Kind of best of both worlds

Research Is Moving Faster and Faster
Unmoderated testing is ramping up
Researchers can use expertise in the test creation and analysis phases and get more insights faster with
help of technology
Some things that platforms help with:
- Branching Logic in test creation to probe deeper when needed in unmoderated test environments
- Watching videos at different speeds (2x, 5x, 10x)
- AI language processing for survey portions of studies
- Audio transcription searches with search functionalities
- Deep linking to key moments of videos from other pages (ex: task 5 or failed session)

In Store (Multi Channel) Testing
New digital ethnographic tools in market
User testing platforms are also starting to use
their mobile apps for in store research. No
longer just restricted to testing online
purchasing habits.
View Highlight Reel

Device Type Increasingly Important
Less Important:
- Different screen sizes (esp for responsive sites)
- We’re finding more themes and developers are able to take this into account
Becoming more relevant:
- Differences in iOS and Android makeup
- Ex: Sign up processes on iOS or Android
- Ex: Adblocker prevalence on iOS or Android
- Ex: App use permissions on iOS or Android
- Differences between models
- Ex: AR now available in iOS 12+

Stress Testing
Is the GDPR popup coming up when an IP is in Europe?
Are location services accurate?
Run a test with many people (ex: 100 to 400) and just test Success/Failure rates at end
- Deeper dive into results if needed
Testing Sooner
More and more of our clients are using our platform for concept testing
- before an MVP is even begun
- Feedback from their target persona within a day

Other Types of Testing
Some new things we’re seeing:
- Advertising research (for images or videos)
- SEO (search habits and display results)
- Product market fit (why a product might not be working in a new area // if advertising spend should
be increased in a new area)
- Diaries

Takeaways
1) Methods for user testing on Mobile (moderated and unmoderated)
2) Tradeoffs associated with each and recommendations
3) Some of the free tools you can use
Extra) Sourcing participants tips
4) Trends in UX and implications moving forward

How to Run User Testing Sessions on Mobile

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

How to Run User Testing Sessions on Mobile

Editor's Notes