A look at why user testing is so important to developing great products and how you can conduct them properly.

1. User Testing Business
Strategy & Framework

2. THE OBJECTIVES &
OUTCOMES OF USER TESTING
01
2© 2017 SYMBILITY SOLUTIONS INC.

3. PRODUCT
STRATEGY
Upsell &
Revenue
Referrals
Customer
Acquisition
Costs
Customer
Lifetime
Value
Positive
Experience
THE BUSINESS OUTCOMES OF USER TESTING
KEY TAKEAWAY
User testing can be tied back to direct and measurable business objectives.
THE BUSINESS OUTCOMES WE SEE WITH USER TESTING:
User testing should be structured in a way to ensure we are living up to objectives
set out for the product, which, in turn, results in positive business outcomes.
“86% of customers will pay more for a better customer experience1”
1. https://www.usertesting.com/blog/2016/04/12/customer-experience-roi/

4. PRODUCT
STRATEGY
USER TESTING BUILDS THE STRONGEST PRODUCT
KEY TAKEAWAY
Building a strong product & experience means understanding your product can always be better.
OUR PRODUCT IS STRONG IF WE CAN VALIDATE:
Products strategy should always be based on customer insights. User testing then
validates we’ve properly translated these needs into a strong & valuable product.
Can customers navigate our
experience effectively and use our
features successfully?
Design / UX
Does the product have the
information customers are looking
for? Does the content used improve
& add to the experience?
Content
Does the product do what customers
expect it to? Did we get the MVP
right? Should our roadmap change?
Capability

5. PRODUCT
STRATEGY
USING TESTING TO REVEAL PRODUCT OPPORTUNITIES
KEY TAKEAWAY
User testing is a critical stress-test to ensure our product is achieving the objectives we’ve set for it.
USER TESTING HELPS US IDENTIFY:
User testing will always lead to more insight. Every time we test, we understand where our
experience is successful, and where there are potential gaps or opportunities we’ve missed.
What
doesn’t need
to be there?
What could
be better?
What’s
working?
What’s not?
What’s
missing?
What’s our
next priority?
How do
people react
to it?
Are we
achieving our
product
objectives?

6. PRODUCT
STRATEGY
THE RETURN ON INVESTMENT WITH USER TESTING
KEY TAKEAWAY
User testing is a manageable and relatively simple process with valuable results.
THE BUSINESS OUTCOMES WE SEE WITH USER TESTING:
The return gained through quantitative user testing can be quite large,
without necessarily needing a huge base of customers to test against.
%oftotalproblemsencountered
0%
25%
50%
75%
100%
# of Test Group Users
0 1 2 3 4 5 6 8 12
2.https://www.invisionapp.com/blog/ux-usability-research-testing/
If done right, speaking to
12 customers can be enough to
catch 99% of interface problems2

7. PRODUCT
STRATEGY
KEY TAKEAWAYKEY TAKEAWAY
The product strategy becomes the baseline to help direct user testing.
USING PRODUCT OBJECTIVES TO GUIDE OUR GOALS & OBJECTIVES
Product objectives determined in Strategy Discovery are used as a guideline for the goals
& expected outcomes of user testing. All these should be in line with the product’s goal.
Product Goal: An aspirational vision that aligns everyone on what this product is trying to accomplish. User
testing should clarify if the product is meeting its goal & strategies and where it’s falling short.
Product Strategy #1
(i.e. Transparent & Digestable
Information)
Product Strategy #2 Product Strategy #3
Testing Objectives for Major Features/ Tactics
(i.e. For Overview Dashboard - Objective is to determine if there is enough strong content to provide a useful summary?)
LINING UP USER TESTING WITH PRODUCT STRATEGY:
Product Features/ Tactics
(i.e.Overview Dashboard)
Product Features/ Tactics Product Features/ Tactics

8. 18© 2017 SYMBILITY SOLUTIONS INC.
WHAT KIND OF TESTING
TO DO & WHEN
8
02

9. PRODUCT
STRATEGY
KEY TAKEAWAYKEY TAKEAWAY
Both user testing techniques can add value - it’s best to have a mix of data to act on.
USER TESTING CAN TAKE ON MANY FORMS
User testing comes down to better understanding how your customer interacts with
your product. It can vary from quantitative & impersonal to qualitative & personal.
MAPPING OUT SOME TYPES OF USER TESTING:
Impersonal &
Quantitative
Personal &
Qualitative
Analytics &
A/B Testing
Mouse Tracking
Remote
Interview
In-Person
Interview

10. PRODUCT
STRATEGY
BOTH TYPES OF TESTING SUPPORT EACH OTHER
KEY TAKEAWAY
User testing can be particularly powerful when different testing methods are used to support each other.
HOW QUANTITATIVE & QUALITATIVE TESTING SUPPORT EACH OTHER:
Once in market, analytics become a pillar for recognizing possible usage patterns (i.e.
drop off, feature interaction). These patterns can then be clarified in qualitative interviews.
Analytics
InterviewsA/B Testing

11. PRODUCT
STRATEGY
KEY TAKEAWAYKEY TAKEAWAY
In-person interviews are recommended to recognize the full value of user testing.
THE BENEFITS OF IN-PERSON INTERVIEWS
In-person interviews give you insights that are not possible with remote testing.
In-person interviews are conducted by 84% of UX professionals3.
IN-PERSON INTERVIEWS, NOT REMOTE, LET US:
Create a Comfortable
Environment
Observe Hand
Movements & Facial
Expressions
Ensure the Right State
of Mind & Attention
https://www.nngroup.com/articles/ux-research-cheat-sheet/

12. PRODUCT
STRATEGY
WHEN TO DO WHAT KIND OF TESTING
KEY TAKEAWAY
Test through every major iteration of the product, starting with interviews and adding in analytics when available.
USER TESTING IS CRITICAL THROUGHOUT THE WHOLE DEVELOPMENT CYCLE:
User testing is best done early and often in order to guide decisions before executing a large
change. The earlier the testing, the more the risk of moving in the wrong direction is reduced.
Design
Prototype
MVP
Additional
Changes
Interviews Interviews Interviews
AnalyticsAnalytics
A/B Testing
HTML
Prototype
Interviews

13. © 2017 SYMBILITY SOLUTIONS INC.
WHO TO DO USER
TESTING WITH
03
1313

14. PRODUCT
STRATEGY
KEY TAKEAWAY
We want to find the segments of people that could have the most impact during testing.
DIFFERENT AUDIENCES THAT COULD BE A PART OF USER TESTING:
FINDING THE RIGHT PEOPLE TO HELP GUIDE US
In order to understand who offers the most insight, we have to think about
every different group that could be interacting with this product. Don’t
forget that customers can be broken down into different personas.
Target
Customers
Current
Customers
Not Previously
Interviewed
Customers
Who Have
Cancelled
Customers
Interviewed
during Strategy
Discovery
Internal Team

15. PRODUCT
STRATEGY
PROPOSED BREAKDOWN OF USERS TO INVOLVE
KEY TAKEAWAY
We want to gain insight into every audience to use the app, while keeping the sample size small.
EXAMPLE OUTLINE OF WHO TO INTERVIEW:
To help your company to gain a holistic understanding of the product,
a good rule of thumb is to test with 2-3 people per audience.
Customers interviewed during
Strategy Discovery
New Customers
Current Customers Not Interviewed
Cancelled Customers
Total 8-12
2-3
2-3
2-3
2-3

16. © 2017 SYMBILITY SOLUTIONS INC.
SETTING UP A USER
TESTING PLAN
04
1616

17. PRODUCT
STRATEGY
KEY TAKEAWAYKEY TAKEAWAY
Mapping out user testing at a high level paints a clear picture of what needs to be accomplished.
HOW TO SET UP A USER TESTING PLAN
Building a user testing plan helps flesh out the details around user
testing, and helps to ensure everyone is on the same page.
SECTIONS INCLUDED IN A USER TESTING PLAN:
The Goal, Underlying
Objectives & Tactics
The Basics Measuring Success
Detailing who, what, where
when and how
The ‘why’ of user testing;
what is the goal & what
tasks can users perform
How do we take these
objectives and create
metrics to quantify success
Possible Scripting &
Questions/
Tasks to Ask
How we frame the situation,
make customers comfortable &
ask the right questions

18. PRODUCT
STRATEGY
EXAMPLE OF ‘THE BASICS’ FOR YOUR COMPANY
The high-level overview of user testing, including the basics of ‘who’,
‘what’, ‘where’, ‘when’, and ‘how’ could look similar to the outline below.
AN EXAMPLE OF ‘THE BASICS’ FOR YOUR COMPANY:
WHO WHAT WHERE WHEN HOW
Customers interviewed
during Strategy Discovery
(2-3)
New Customers (2-3)
Current Customers Not
Interviewed (2-3)
Cancelled Customers(2-3)
Conducting user testing to
validate our product is
meeting our goals,
strategies, & expectations
in the market
Details like participants
will receive compensation
An agreed upon location
between user &
company (i.e. customer’s
home, office, cafe)
Testing will take place
over the course of X
number of weeks
starting in X month
Time slots of 30 minutes
scheduled with each
user to be listed
In-person interviews will be held
and conducted at respective
locations
Details on device provided to
user & record screen
Company to complete 1-2 user
tests & regroup to adjust
possible issues
Scripting & questions to ask to
be listed

19. PRODUCT
STRATEGY
KEY TAKEAWAYKEY TAKEAWAY
Objectives should be set for all major features or processes in the platform.
DIVING INTO AN EXAMPLE USER TESTING OBJECTIVE
Our objectives tie back to our strategy pillars, and are the ‘why’ of testing.
Below is an example of what an objective for a feature could look like.
Feature &
Strategy
Objective 1
Task
Evaluative
Question
Example:
Dashboard - Transparent &
Digestible Information
Example:
Determine if this dashboard offers
the right information hierarchy
Example: Get on the dashboard and find
credit score over time period
Example: How user feels about clarity of
information? Above or below expectations?