The Missing Voice:
The Role of (Good) Customer Research in
Optimization Programs
Presented by AJ Davis, Founder of Experiment Zone

Agenda
What we’ll cover
We’ll discuss how to advance your CRO practice using user research.
▪ Intro to user research methods
▪ How & where to infuse usability research
▪ Examples of the impact of research on A/B testing programs
▪ How to make the case for better research

Intro to AJ Davis
Experiment Zone
AJ loves helping brands improve their conversion rates
by improving customer experiences.
Expertise:
- Finding user experience issues & solving them
- Identifying small and large changes that improve
conversion rates
- User research, analytics, and conversion rate
optimization
Reach me
aj@experimentzone.com
LinkedIn

Customer Research + CRO
Current State
In 2019, CRO research was the top process issue
for optimizers.
Bigger than test prioritization or test velocity.
Results as seen in the 2019 CXL State of Conversion report: https://cxl.com/blog/2019-conversion-optimization-report/

Customer Research + CRO
Current State
One-third of optimizers had no process in place for
user research or test prioritization.
Many CRO experts focus on research that doesn’t include talking with a
person.
Results as seen in the 2019 CXL State of Conversion report: https://cxl.com/blog/2019-conversion-optimization-report/

Intro to User Research
What is it?
User research focuses on understanding,
user behaviors, needs, and motivations.
Through observation techniques, task analysis, and other
feedback methodologies.
Definition adapted from Usability.gov (https://www.usability.gov/what-and-why/user-research.html)

CRO Process
Balancing act
Good decision-making needs a balance of
qualitative and quantitative insights.

CRO Toolkit
How do we find answers?
Adapted from NNGroup’s A Landscape of User Research Methods
https://www.nngroup.com/articles/which-ux-research-methods/
A/B Testing
Usability Testing
Analytics Analysis
Email SurveysInterviews
Intercept SurveysFirst Impression Testing
Card Sorting &
Tree Testing
Click Tracking Analysis
Heuristic Review
Expert Review
Why
Behaviors

Definition: evaluating a product or service by
testing it with representative users.
▪ Participants complete core tasks
▪ Observers watch, listen, and take notes
The outcome?
▪ Uncover usability problems
▪ Understand participant’s understanding & satisfaction with
the product
Usability Research
What is it?
Definition adapted from Usability.gov (https://www.usability.gov/how-to-and-tools/methods/usability-testing.html)

Usability Research and A/B Testing
When to research
Before Testing Test Planning After a Test

Usability Research and A/B Testing
When to research
Workout 4 You 👟
Goal
Increase online revenue
from DTC business
Testing Maturity
We run a test or two per
month

Usability Research and A/B Testing
When to research
Before Testing Test Planning After a Test

Usability Testing in CRO
Before Testing
Why now?
▪ Direct our testing to the highest impact areas.
▪ Focus attention to the pages with the most pain points
for visitors
Outcome
Improve test prioritization
Illuminate new problems that testing can solve.

Usability Testing in CRO
Before Testing
At this stage, you can answer:
▪ What are the biggest pain points in our customer journey?
▪ Where do participants get confused?
▪ What’s unclear to participants about our product or service?
▪ What tasks do participants fail to complete? Struggle?

Usability Testing in CRO
Before Testing
Workout 4 You 👟
Tasks
Browse and find the
right product for you.
Insight
Visitors struggled to use
the filters to narrow
their results.

Usability Research and A/B Testing
When to research
Before Testing Test Planning After a Test

Usability Testing in CRO
Test Planning
Why now?
▪ Improve the test variant before putting it into the world.
▪ Pinpoint small points of friction that could turn a winning
improvement into a losing test.
▪ Create more effective test variants.
Outcome
Increase the likelihood of a winning test.
Refine the variant prior to running the A/B test.
Decrease likelihood of custom service phone calls, big losses.

Usability Testing in CRO
Test Planning
At this stage, you can answer:
▪ What usability issues exist in this variant?
▪ How do participants respond to this new solution / design?
▪ What improvements can we make to this design before A/B
testing?

Usability Testing in CRO
Test Planning
Workout 4 You 👟
Test
Add Live Chat to the
website
Insight
Participants didn’t
notice the Live Chat
next to the Call Now.

Usability Research and A/B Testing
When to research
Before Testing Test Planning After a Test

Usability Testing in CRO
After a Test
Why now?
▪ Some A/B test outcomes are inconclusive or
confusing
▪ We need to know why something happened, to
update our knowledge of customers
Outcome
▪ Understand what happened in unexpected results.
▪ Unpack mixed or unexpected results.

Usability Testing in CRO
After a Test
At this stage, we can answer:
▪ Why did the test fail?
▪ Why did different visitor groups yield polarizing
results?
▪ What should we consider in the next round of testing?

Usability Testing in CRO
After a Test
Workout 4 You 👟
Test
Add track your order at
the top of each page.
Insight
Visitors ignored the
message initially and
eventually felt annoyed.

User Research
Tools & technology
Usability Testing Tools
Participant Recruitment Platforms Additional Tools

Usability Testing in CRO
4 Tips
#1 - Ask open ended questions.
#2 - Ask participants to think aloud. Remind them if they forget.
#3 - Avoid leading questions or responding with obvious interest to
certain comments.
#4 - Include 5-6 participants per persona or customer segment.

Usability Testing in CRO
Make the Case
Each usability study yields many A/B tests worth of insights.
▪ More focused test roadmap
▪ Less time spent to get to the best (and highest converting)
experience
▪ Risk Reduction - avoid putting poorly performing variants in
front of large numbers of visitors

Next Steps
Visit Us
Need help bringing usability testing to your CRO program?
▪ Find out about what we do at ExperimentZone.com
▪ Sign up for our next webinar at GrowthThroughEmpathy.com
▪ Join our Growth Through Empathy Facebook Group to have
regular advice at your fingertips.
▫ https://www.facebook.com/groups/growththroughempathy/