As per a survey by Zipdo, data-driven A/B testing can boost app engagement by a whopping 72%. It's true, and it's all about finding the right thing to test. But how do you come up with those genius A/B test ideas? That's exactly what our webinar is here for. We will show you how watching your users' moves (thanks to session replays) and seeing where they tap (hello, heatmaps) can spark those brilliant ideas for your tests. It's not just about guessing; it's about knowing what will make your app stickier. So, if you're all in for making your app more engaging, this is one you will want to take advantage of.

1. 1
A/B Testing with
Behavioral Insights
Building hypothesis and analysing experiments with behavioral insights
A Product Webinar by VWO

2. 2
What I do day in, day out?
“Look at session recordings &
heatmaps of mobile apps of our
customers and suggest them UX
improvement ideas”
Little about me
12+ years of managing and marketing
websites and apps, for B2B & B2C.
Piyush Sharma
Product Marketing, VWO

3. 3
Agenda
1
2
3
4
5
6
Mobile app experiment lifecycle
Collect data
Analyse test results
Popular use cases
Tour of session replays, heatmaps & funnels
Q & A

4. 4
Mobile app experiment lifecycle
Define Objective
Activation, Engagement, Retention,
Conversions
Develop Hypothesis
Building hypothesis for multiple variations
Analyse Test Results
Check movement in metrics. “And why?”
Collect Data
Quantitative Analysis, Qualitative Analysis
Run Experiment
Running A/B tests, multivariate or split tests
Deploy or Record Learnings
1
2
3
5
4
6
Repeat

5. 5
Collect data
5

6. 6
Collect Data
Quantitative Data Qualitative Data
What Users Say
VWO Insights - Mobile App
Session recordings
Heatmaps
Surveys
User Interviews
Reviews
Emails
What Users Do

7. 7
Collect Data - Quantitative Data
Limitations
Not easy to find the why behind the numbers.
60%
Completed app
onboarding
78%
Dropping off from the
checkout screen
9%
Users use the
new feature

8. 8
Search what users are
saying around the issue.
Only 1 in 25 unhappy
customers complain
directly to you.
Source: Esteban Kolsky Research
App store reviews
Support emails
8
Collect Data - Qualitative Data
Moreover, lot of to and fro with the
user to understand issue.

9. 9
Deal with conscious (biased)
responses.
App is used with subconscious
instincts
Gaps between what they say
and what they do
Difficult to deduce exact
actionables
9
“Onboarding form is long”
“Price is too high”
“Got confused between options”
Collect Data - Qualitative Data
Surveys
User interviews

10. 10
10
Collect Data - Qualitative Data
Session recordings
Heatmaps
Understand why something
might be happening
See exactly what the users see
Get team’s confidence into
your next A/B test with a
glaring insight you observed

11. 11
11
Quantitative Data
Collect Data
Example
Why is this user cohort
exiting the app?
Insight
The navigation of the app
is not clearly visible on
iOS so users don’t know
where to go.
Qualitative Data
Example
Unusual number of exits
from home screen.
Insight
Majority of iOS users
using the latest app
version are exiting the
home screen.
Power-packed Hypothesis
Example
Test with a new version of
navigation which is much more
visible.

12. 12
Analyse Test Results
12

13. 13
13
Analysing Test Results

14. 14
Analysing Test Results
Quantitative Data
Qualitative Data
(Session Recordings & Heatmaps)
Statistical significance and winning variant
Compare across multiple KPIs
How different segments performed?
External and internal factors?
Why there was an improvement?
How was the behavior different in
both the variations?
Insights that quantitative data might
miss

15. 15
Use cases
15

16. 16
Quantitative Insight - Low use of the search
function within the app.
Behavioral Insight - “Users try the search feature
but quickly give up. They seem to prefer
browsing through categories.”
A/B Test Hypothesis - Improve search
functionality with auto-suggestions and filters to
make it more user-friendly.
Analyzing Results - Monitor if there's an
increase in the use of the search function without
users getting confused about what to search.
Use cases
16
Feature adoption

17. 17
Quantitative Insight - Only 20% women
completed app onboarding.
Behavioral Insight - “Users want to skip the DOB
option but they can’t, hence dropping off”
A/B test Hypothesis - Make the DOB optional in a
new version.
Analysing Results - More women completing the
onboarding. Verify if the users are indeed skipping
the DOB and going past next screens.
Use cases
17
Onboarding completion

18. 18
Use cases
18
Quantitative Insight - Product screen showing
unusual drop-offs
Behavioral Insight - “Users tapping on the image
to magnify it but failing. Rage taps on the image.”
A/B test Hypothesis - Provide option to see a
magnified view with multiple images, so that user
can see product in detail
Analyse Results - Check if enough users are
actually tapping and checking out the product the
way you expected.
Improved conversions

19. 19
Use cases
More abandons in new app version How
users compare between new app
versions?
New users not sticking - How new vs
returning users compare?
UK users churning - How users from
different geographies behave?
Check it out →
19

20. 20
More events?
Add to cart? Checkout?
Upgrade plan?
20
Use Cases

21. 21
Use Cases
21
Which are important areas?
Which areas are ignored?
Test UX elements
CTA placements
Eliminate unnecessary elements
Identify new clickable areas

22. 22
22
Let’s see
Session Recordings &
Heatmaps
in action

23. 23
23
Open for any questions.

24. 24
Want to understand how it could help your
app and explore features in detail?
Try VWO free
24