To build an optimized website funnel, you need to capture the pulse of your users. What is that they’re looking for? What’s stopping them from taking the desired action? What could you do better to improve their experience on your website? We answer all such questions, and more in this webinar on how to successfully conduct Visitor Research. Both these brands (BrainShark & Quandoo) have been leveraging data to build strong hypotheses for their optimization program.

1. Cutting Guesswork About User
Behaviour with Data-driven
Conversion Optimization
Erica Schroder
Web Manager, BrainShark
Pablo Paternain
Sr. Product Owner, Quandoo
Rahul Jain
Sr. Product Manager, VWO
Utkarsh Rai
Growth Marketer, VWO

2. vwo.com
Today’s Agenda
● Methods of visitor research
● Tools & techniques for gathering visitor data
● Research Blueprint: The Sketch
● Conversion Optimization Learnings By BrainShark
● Conversion Optimization Learnings By Quandoo

3. vwo.com
Ask Questions on Twitter
#AskVWO

4. vwo.com
A step-by-step approach to
increase conversions
Conversion Rate Optimization Process
Research Phase
Tracking metrics and identifying what
parts of conversion funnel need fixing
Hypothesis Phase
Constructing educated hypothesis,
based on your research
Prioritization Phase
Planning and prioritizing your hypothesis
Testing Phase
Testing your hypothesis against the
existing version of the website
Learning Phase
Deploying the winning hypothesis and/or
gathering learning for subsequent tests

5. vwo.com
What are the different
methods to do research?

6. vwo.com
What are the
different
methods to do
research ?

7. vwo.com
• Web Analytics
• Conversion Funnel Tracking
• Form Analytics
Quantitative
Methods
Collecting metrics about users on
how they interact and engage with
your web properties

8. vwo.com
Quantitative data shows you what is
happening, but why it is happening is
discovered by qualitative tools.

9. vwo.com
• Heatmaps and Clickmaps
• Customer Interviews
• Surveys
• Session Replays
• Click Testing
• Customer Support Interactions
...and so on
Qualitative
Methods

10. vwo.com
Technical Research Heuristic Analysis
Discovering broken
functionality and user
experience
• Browsers
• Devices
• Page load speed
Evaluation of a website by a
usability expert against a set
heuristics to identify
potential conversion
barriers.

11. vwo.com
Research Tools
Which tools to choose?
1) Web analytics
2) Surveys
3) Heatmaps and Clickmaps
4) Visitor Recordings (Session Replays)
5) User Testing
6) Form Analysis
7) Heuristic Analysis (Expert Reviews)
8) Customer Interviews
9) Click testing
10) 5 second test

12. vwo.com

13. vwo.com
Putting Visitor Research
Into Action
Setting Up a Research Blueprint

14. vwo.com
GOALS
FUNNELS
VISITOR PROFILES

15. vwo.com
Keeping close eye on business
metrics
1) Define business objectives
2) Create goals in terms of metrics of these objectives
3) Start with Macro conversions
4) Setup micro conversions that form part of macro
conversions
5) Create funnels of most important pathways
Start Baselining measurements
GOALS

16. vwo.com
Find Out Drop-Offs at Each Step of the Conversion
Funnel
VWO’s Plan layer

17. vwo.com
VISITOR PROFILES
Characteristics Confused Visitor Average Visitor
Session time 30 sec 15 mins
Number of pages viewed 1 5
Characteristics Loyal Visitor Average Visitor
Time before first click 30 sec 45 sec
Time spent per page 40 sec 1 min

18. vwo.com
VISITOR PROFILES
Characteristics Frustrated
Visitor
Average Visitor
Session Time 8 mins 15 mins
Number of clicks per page 10 5
Characteristics Lazy Visitor Average Visitor
Average scroll rate 30% 50%
Number of clicks per page 5 3

19. vwo.com
Defining Your Visitor Journey
A visitor journey is the complete sum of experiences that a visitor go through when
interacting with your website.
Instead of looking at just a part of a transaction or experience, a visitor journey
documents the full experience of being a website visitor.

20. vwo.com
Typical Visitor Journey on an eCommerce Website

21. vwo.com
Sketch Your Visitor Journey
Right Now!
In-session activity:
Take a paper & a pen and draw any one of the many visitor journey
of your website

22. vwo.com
IDENTIFYING THE “WHAT” AND ”WHY” FROM BEHAVIORAL DATA
What Products are
visitors looking for?
What step in the
funnel are visitors
dropping off the most?
Why are visitors abandoning their carts?
Why are visitors
bouncing off the
category pages?

23. vwo.com
SHAPING YOUR RESEARCH BLUEPRINT
Outline Your
Visitor Journey
Identify
Opportunities
in Your Visitor
Flow
Address The
What & Whys
Run
Experiments
and Measure
Impact
● Define Goals
● Build Funnels
● Profile your
visitors
Know WHAT your
visitor is doing using
Quantitative Research
Use Qualitative
Research to know WHY
they behave in a
certain way
Collect and analyse
your research data.
Build strong hypothesis
to test inferences you
drew in your research.
Break your conversion
goal into several micro
goals to analyse true
impact of your
experiments.

25. vwo.com
Using the Website as a Demand-Gen Tool

26. vwo.com
Identifying the Top Performers
- See a Demo
- Contact Sales
• Provide more qualified leads than any
other Brainshark resource or asset
• Salesforce campaign reporting shows the
number of conversions
• Salesforce wasn’t able to provide any
insight into user behavior on the pages
• The 2 top performing pages on the
website are:

27. vwo.com
#1 Question We Asked Ourselves…

28. vwo.com
We’ve Already Done So Much…What Else Can We Do?
•
•
•
•
•

29. vwo.com
Next Step: Investigate User Behavior

30. vwo.com
Heatmaps & Form Analysis – The Gold Mines
of User Information

31. vwo.com
Heatmaps Help Identify Distractions
We want users to click here…
(which they are!)
…but, they are also clicking here and here
(these two links simply reload the page!)

32. vwo.com
Next Step: A/B Testing
A/B test hypothesis that removing these
’distractions’ will increase conversions
Variation removes button & footer link

33. vwo.com
A/B Testing Gives You Data To Make Informed Decisions

34. vwo.com
Implement Changes Based on Results of A/B Testing
-
-

35. vwo.com
Form Analysis Shows How Users Interact with Forms
•
•
•

36. vwo.com
Examine User Interaction with Form Fields & Take Action
•
•
•

37. vwo.com
Added Bonus: Good Data Makes Difficult Conversations
Easier!
…
…

38. Data-Driven A/B Testing
at Quandoo
www.quandoo.com
Pablo Paternain, Sr. Product Owner B2C Web

39. vwo.com
The Fastest growing Restaurant Reservation Platform
✓ 12 COUNTRIES
✓ >110 MILLION SEATED GUESTS
✓ 18,352 PARTNER RESTAURANTS
✓ 35 MICHELIN STAR RESTAURANTS
✓ 374 EMPLOYEES WORLDWIDE
by

40. vwo.com
Agenda
1) Quandoo Website Visitor Research
2) Successful A/B Tests examples
3) Our A/B Testing journey so far at Quandoo

41. vwo.com
DATA DATA DATA - Obsess about Data
1) Where’s the biggest opportunity?
- Google Analytics + Device Segmentation
- DB reports
2) What can we improve?
- Polls/Surveys
- Heatmaps/Scrollmaps
- Form Analysis
- Visitor Recordings
Quandoo Website Visitor Research - Customer Insights
Customer
Insights
Hypotheses
Ideate
Prioritize
Learnings
Execute

42. vwo.com
Quandoo Website Visitor Research - Where’s the biggest Opportunity?
Homepage
Popular Flow: Users come directly to Quandoo to find a Restaurant and make a reservation
Results Page Restaurant
Page
Booking
Widget
Checkout
Form
Success

43. vwo.com
Quandoo Website Visitor Research - Where’s the biggest Opportunity?
Popular Flow: Users know the city/district where they want to find a restaurant.
Results Page Restaurant
Page
Booking
Widget
Checkout
Form
Success

44. vwo.com
Quandoo Website Visitor Research - Where’s the biggest Opportunity?
Popular Flow: Users know the name of the Restaurant and want to make a reservation.
Restaurant
Page
Booking
Widget
Checkout
Form
Success

45. vwo.com
At Quandoo, we generally set 2 types of polls/surveys:
1) Broad questions, with open answer:
- e.g. ‘What are you trying to achieve?’
These type of questions help us identify potential areas of
opportunity, so that we can ideate for potential solutions to
problems which our users face.
2) More narrowed questions, providing choices:
- e.g. ‘What information is more important? A/B/C…?’
These type of questions help us quickly verify our hypothesis
with qualitative feedback backed by numbers.
Quandoo Website Visitor Research - What can we improve?

46. vwo.com
Quandoo Website Visitor Research - What can we improve?
Set up Heatmaps / Scrollmaps / Form analysis / Recordings to gather data throughout your most
important funnel.

47. vwo.com
Quandoo Website Visitor Research - What can we improve?
Ensure Only Important Elements
Ensure Only Secondary Elements
*TIP -> After running the first tests, Track EVERYTHING
you can Track, to paint the best possible picture.

48. vwo.com
Agenda
1) Quandoo Website Visitor Research
2) Successful A/B Tests examples
3) Our A/B Testing journey so far at Quandoo

49. vwo.com
Successful A/B Tests Examples - Example 1
Menu is crucial to our visitors, but in spite of being
on the restaurant page, some users cannot find it.
- Heatmaps show high interaction with Menu Tab
- Recordings show high interaction with Menu Tab
- Surveys complain they cannot find the Menu
“If we rename the Chef’s Choice to Menu Highlights,
visitors will feel that they’ve found the Menu and
make the decision of booking afterwards.”
Pure Copy Tests → Topic = Information

50. vwo.com
Successful A/B Tests Examples - Example 1
Speak the language
of our Users.
+ 1,8% relative
increase
in reservation CVR
Control Variation

51. vwo.com
Successful A/B Tests Examples - Example 2
When a visitor has decided to proceed to the
checkout, we expect no distractions to change
their mind about making a reservation.
- Checkout CVR could be higher
- Form Analysis shows problems with drop-off
after interacting with Prefix field
“If we provide country flags and add the full
country name to the prefix, visitors will find
their country prefix more easily and thus
proceed with the reservation.”
Ease of Use → Topic = Information

52. vwo.com
Successful A/B Tests Examples - Example 2
The less the users
have to think, the
better.
+ 4,6% relative
increase
in reservation CVR
Control Variation Variation

53. vwo.com
Successful A/B Tests Examples - Example 3
When a visitor has decided to proceed to the checkout, we
expect no distractions to change their mind about making a
reservation.
- Clickmap shows lots of interactions with SSO options, but not
all clicks end up with log in
- Scrollmap shows difference between users reaching SSO &
users reaching Phone number, and form end for mandatory
fields is not seen
“If we prioritize the simple form over the SSO for non-logged
in users, anonymous visitors will see that’s the only
information required to create a reservation and thus will
proceed with the form.”
Restructuring → Topic = Information Architecture

54. vwo.com
Successful A/B Tests Examples - Example 3
Checkout form length
has to be clear on
viewport.
+0,8% relative increase
in reservation CVR
Control Variation

55. vwo.com
Agenda
1) Quandoo Website Visitor Research
2) Successful A/B Tests examples
3) Our A/B Testing journey so far at Quandoo

56. vwo.com
10AB Tests running at some point simultaneously
+15,3 %Relative Increase in Avg. Reservation CVR from H2
2017 - H1 2018
+101 %Growth in *AB Speed from H2 2017 - H1 2018
*AB Speed = # Tests during a specific time-frame
+151 %Growth in *AB Effectiveness from H2 2017 - H1 2018
*AB Effectiveness = Successful Tests / Total # Tests
Our A/B Testing journey so far at Quandoo

57. vwo.com
+275%
Growth in Quandoo Team Members involved in AB Testing from H2 2017 - H1
2018
If you would like to Join our Quandoo Team, please reach out!
www.linkedin.com/in/pablo-paternain-soler-77841653
Our A/B Testing journey so far at Quandoo

58. Thank You!
Q&A