Click below to see best practices for using automated user testing to get quick, efficient user input. You'll learn: -How to use automated testing with agile design and development sprints -How to set up automated studies for optimal performance -When to use moderated vs. un-moderated testing -Benefits and comparisons of top un-moderated testing tools: UserZoom, Usertesting.com, Userlytics, and Loop11.

1. 4149 Pennsylvania, Suite 202
Kansas City, MO 64111
Automated User Testing
Presented by
Chrys Sullivan
Founder and CEO, Useagility

2. When We’re Finished, You Should:
2
Understand how unmoderated user testing
works and how you could use it in your
organization.
01
Be able to identify when unmoderated
testing is and is not the best testing
methodology.
02
Know our tips for avoiding the most common
pitfalls in automated testing.
03
Understand primary differences between four top
automated testing tools.
04

3. About Useagility
3

4. Our team of experienced professionals draw from a wide range of skills, processes and tools to across the spectrum of user
experience research and design services. Each client project uses different design and research tools and process depending
on the project needs
Our Services
User
Research
Experience
Strategy
Design
Front-End
Development
Tap into customers and
end users to inform
strategy and design
Envision the future of your
digital business strategy
Dive into complex UI
challenges; simplify,
streamline and get results
Bring design to life across
all devices

5. Evolution of Usability Testing Technology
5
Lab-based testing
- Moderated user testing, in lab
setting
Remote, facilitated testing
- Moderated testing conducted
remotely via web-conference and
screen sharing technology
Automated, un-facilitated testing
- Unmoderated studies conducted
online via web-based applications

6. Benefits of Unmoderated
6
 Easier recruitment process
 Recruiting
 Scheduling
 No worries about no-shows – or
replacements
 Automated incentive payments
 Simultaneously test large number of participants
 Less expensive per-study costs than moderated
testing
 Get results quicker than moderated
 Built-in metrics (time on task, rating scale, etc.)

7. Limitations & Pitfalls of Unmoderated
7
 User panels and “professional testers”
 Stakeholders are not involved as observers during
the study
 Can’t revise tasks / questions once the study has
been launched
 Can’t adjust study approach as sessions progress
 Lacks ability to probe unexpected insights
 No real-time support for users who get significantly
off-track
 Greater likelihood of prototype-related issues
causing task error

8. 8

9. 9

10. 10

11. 11

12. 12

13. Selecting Right Methodology
13
Moderated or Un-moderated?
Choosing the right methodology has the greatest impact on your ability to meet
expectations with the outcomes of the study.

14. First Step: Define Goals for Testing
14
Hard to get business stakeholders to
define their goals. Ask them
questions:
• What is the primary purpose of the
study?
• What do you need to know coming
out of this study?
• What are you hoping to do with the
study findings?

15. Second Step: Define Additional Details
15
After goals are defined, understand
other key components:
• What is the scope? (Big, small, how
focused?)
• What are we testing with?
(Prototype, wireframes, live site)
• Who are the target users?
• Which kinds of user scenarios and
tasks will help achieve study goals?

16. Step 3: Moderated or Unmoderated?
16
 Are follow-up questions necessary?
 Are you in concept/strategy phase?
 Are you dealing with a limited-functionality prototype?
 Will it take longer than 20 – 30 minutes to complete
tasks?
Yes = Moderated

17. Step 3: Moderated or Unmoderated?
17
 Is the scope focused?
 Can testing tasks be accomplished in 30 min or less?
 Are there clear, specific tasks that can be tested?
 Is testing of task success and usability a primary objective?
 Is there minimal reliance on attitudinal feedback that requires
follow up probing?
Yes = Unmoderated

18. Which Tool is Right for My Needs?
18

19. Which Tool is Right for My Needs?
19
Features
Audio/Video Capture   
Metrics and Data    
Mobile    
Native App   
Provider has User Panel    
Use My Users  Not easily  
Automated Incentive    
Flexible Incentives  PayPal only PayPal only 
Consulting Services  
Pricing $350 per project - up to 1,000
participants per project
$50 per study, minimum of
150 purchased upfront
($7,500)
$350 set - $49 per project $12,000 + minimum per year

20. 20
Pros
• Most economic for
conversion testing on
large sample size
• Reliable, time-tested
platform
• Webcam to see user’s
face
• Lower entry-level
pricing
• Ability to share videos
easily
• Flexible incentives
• Full-service consulting
support
Cons
• No video
• Reliant on self-reported
task success and
analytics
• Inability to easily recruit
from intercepts or own
user group
• Set incentive amounts
• Panel size may be
smaller than UT.com
• Expensive
• May be overkill for
some companies who
just need task-based
testing at affordable
price point
BottomLine
• Best option for high-
volume conversion flow
testing
• No video; only metrics
• Least expensive
• Easy study set-up
• Large panel
• Good for “do-it-
yourself” testing
• Newer, user friendly
interface
• Lower pricing point to
get started
• Good for “do-it-
yourself” testing
• Easy to share clips and
full video files
• The “Cadillac”
• Highest entry price
• Not best fit for “do-it-
yourself”
• Push the consulting
services

21. Which Tool is Right for My Needs?
21
• Best option for high-
volume conversion
flow testing
• No video; only
metrics
• Least expensive
Pricing $350 per project - up to 1,000 participants per project
What’s captured? Metrics and data, no video
Features Clickstream analysis, heat mapping, task success, abandonment
analysis
Mobile? Mobile browser enabled, not able to test native apps
Recruitment options Panel, intercept from your site, email distribution
Automated
incentives?
Yes, if their panel is used
Incentive types Incentives can be set for any amount; Gift cards, Lotteries, bank
transfers, PayPal transfers, and more.
Pros Most economic for conversion testing on large sample size
Cons No video, reliant on self-reported task success and analytics

22. Which Tool is Right for My Needs?
22
• Easy study set-up
• Large panel
• Good for “do-it-
yourself” testing
Pricing $50 per study, minimum of 150 purchased up front ($7,500)
What’s captured? Metrics and video/audio
Features Audio/video capture, exit survey, net-promoter scores, metrics, time-
on-task, consulting services to run studies
Mobile? Desktop, tablet and phone in browser
App must be downloaded for native app testing (ios and Android)
Recruitment options Panel - Can’t use your own participants without them having to be
added to the UserTesting panel
Automated incentives? Yes, if their panel is used
Incentive types A set incentive of $10 for every 20 minutes; PayPal
Pros Reliable, time-tested platform
Cons • Inability to recruit from intercepts or own user group without
users being added to their panel and going through “training”
• Set incentive amounts

23. Which Tool is Right for My Needs?
23
• Newer, more user
friendly interface
compared to
UT.com
• Lower pricing point
to get started
• Good for “do-it-
yourself” testing
• Video share feature
makes easy to share
clips and full video
files
Pricing $350 set up - $49 per user session
What’s captured? Metrics and video/audio
Features Audio/video capture, webcam to see participant’s face, exit survey,
metrics, time-on-task, consulting services to run studies
Mobile? Desktop, tablet and phone in browser
App must be downloaded for native app testing (ios and Android)
Recruitment options Panel or use own users from intercept, email distribution, etc.
Automated incentives? Yes, if their panel is used
Incentive types $1 - $100 via PayPal
Pros Webcam to see user’s face, lower entry-level pricing, ability to share
videos with stakeholders and clients easily through links.
Cons • Panel size may be smaller than UT.com

24. Which Tool is Right for My Needs?
24
• The “Cadillac”
• Highest entry price
• Not best fit for “do-
it-yourself”
• Push the consulting
services
Pricing $12,000 + minimum per year
What’s captured? Metrics and video/audio
Features Audio/video capture, UX benchmarking, card sorts, tree testing,
metrics, heat mapping, click test, consulting services to run studies
Mobile? Desktop, tablet and phone in browser
App must be downloaded for native app testing (iOS and Android)
Recruitment options Panel, or use own users from intercept, email distribution, etc.
Automated incentives? Yes, if their panel is used
Incentive types Any amount, many different payment method options (Paypal, gift
certificates, bitoins
Pros Flexible incentives, full-service consulting support
Cons Expensive, may be overkill for some companies who just need task-
based testing at affordable price point

25. Conduct Pilot
Session
Tips for Avoiding Pitfalls of Unmoderated Testing
01 02 03
04
06
1. Conduct one
session as pilot
2. Watch results &
adjust tasks
and flow
25
Verify Participant
Profiles
1. Use screener
questions
2. Ask users to give
you verbal
answers to
questions that
confirm their
profile
Ask follow ups
1. Ask probing
questions after
tasks
2. Use task metrics to
prompt follow up
questions
03
Question self-
reported success
1. Self—
reported task
success is
unreliable.
Users don’t
want to
“mess up”
2. Measure
actual success
vs. reported
success
04 05 Keep tasks
simple
1. Keep tasks
simple; users
skim task
instructions

26. Our Favorite Ways to Use Unmoderated Testing
26
 As quick way to test usability in detailed design phase
 Use Axure, InVision prototypes
 Can also use static wireframes, videos and other design files as stimulus
 During agile design & development sprints to test wireframes and prototypes with
users before development begins
 To test effectiveness of one design approach vs. another
 To obtain user preference for one design over another
 To validate suspected usability issues or support Expert Review findings

27. 27
Wrapping It Up

28. Now You Know
28
How unmoderated user testing works and
how you could use it in your organization.
01
Identify when unmoderated testing is and is
not the best testing methodology.
02
Quick tips for avoiding the most common pitfalls in
automated testing.
03
The primary differences between four top automated
testing tools.
04

29. Thank You
29
Thank you for joining us today
We would love to work with you on
your next project
Connect with us
www.Useagility.com
info@useagility.com
Watch for our newsletter to learn about upcoming webinar topics