The primary challenge for eCommerce businesses is to get people on their website. The next challenge is to get them to purchase products. Digital marketers work tirelessly to understand the entire customer journey from discovery to purchase and find ways to influence customer intent. If the website does not offer a great experience to users (from the beginning), these efforts will not translate into a commensurate growth in revenue. In this session, Narayan Keshavan from Dell Technologies will focus on the importance of UX Analytics and Experimentation as key enablers for eCommerce revenue growth. Based on his experience in this area, he will outline some key principles that are necessary to effectively leverage experimentation to offer a superior experience to users and enhance the business KPIs. He will also use specific A/B tests to articulate the significance of these principles.

1. UX Analytics and Experimentation
for eCommerce Growth
Narayan Keshavan
Head – Digital Analytics
(Design Analytics)

2. About Me
• Head of Digital Analytics in the Design Analytics org., a
global team that owns the experimentation charter for
Dell.com
• Over 13 years of experience spanning eCommerce, Telecom
and Insurance sectors
• Have managed marketing with P&L responsibilities for over
USD 45 Mn in the hyper-competitive telecom sector
• Have managed teams working in the area of marketing and
digital marketing analytics including conversion rate
optimisation, campaign analytics, SEO and social media
listening.
• Lead the Digital Analytics Community (DAC) initiative to
build and expand a thriving community of digital analytics
professionals DigitalAnalyticsCommunity.wordpress.com

3. Focus on the user
EDD EDDand all else will follow

4. Dell.com - an optimization journey with
experimentation at its core
1996 2009 2020
Narayan Keshavan | Design Analytics | Dell Digital Design

5. A “great idea” can come from anywhere
Narayan Keshavan | Design Analytics | Dell Digital Design
Quantitative Qualitative
UX
Analytics
Heatmaps/
Session replays
VOC
Test
iterations
Business
inputs

6. Building your experimentation roadmap
Experimentation
Roadmap
Business
inputs
Bench-
marking
Voice of
users
Product
roadmap
Test
Insights
Analytics
UX
Research
Product
managers
Design/ UX
Marketing
eCommerce
Segment/
Element targeted
KPI Projections

7. Iterative experimentation
success metricshypothesis
analysis performance check

8. Key principles for an effective
experimentation strategy

9. Less is ‘More’?
Please note: the screenshots have been simplified for illustration and do not completely resemble the actual version

10. Nothing is OBVIOUS
Experimentation is the real ‘Moment of Truth’
and can, many times, SURPRISE us!
Image courtesy: Fabian Reitmeier | pexels

11. Reducing the clutter had unintended consequences
Please note: the screenshots have been simplified for illustration and do not completely resemble the actual version

12. “I have not failed.
I've just found 10,000 ways that won't
work.”
― Thomas A. Edison

13. There is no such thing
as a ‘failed experiment’!
Every test offers insights which feed
into subsequent iterations

14. We tried to simplify the checkout pages by collapsing
optional fields
optional
fields
Please note: the screenshots have been simplified for illustration and do not completely resemble the actual version

15. Interestingly, more users dropped off later in checkout
optional fields
collapsed
Please note: the screenshots have been simplified for illustration and do not completely resemble the actual version

16. The insights gained helped us iterate with a better experience
optional
fields at the
bottom
(not collapsed)
Please note: the screenshots have been simplified for illustration and do not completely resemble the actual version
mandator
y fields

17. Beware of ‘Testing Myopia’Avoid ‘Testing Myopia’
Short-sightedness in interpreting test results
may prove detrimental to business
Image courtesy: David Travis | unsplash

18. Let’s consider an A/B test for a banner
Variation A Variation B

19. Which variation is better? Do you say ‘B’?
A B
Metric Variation A Variation B
Visits 1000 1000
Users 200 100
Conversion Rate
Users 10% 12%
Please Note: I have taken random numbers for convenience of explanation. These are not from any real test

20. Which variation is better? Is it that obvious?
A B
Metric Variation A Variation B
Visits 1000 1000
Users 200 100
Orders
Users 20 12
Conversion Rate
Users 10% 12%
Please Note: I have taken random numbers for convenience of explanation. These are not from any real test

21. What do you think now? Is ‘A’ better?
A B
Metric Variation A Variation B
Visits 1000 1000
Users 200 100
Non-users 800 900
Orders 36 30
Users 20 12
Non-users 16 18
Conversion Rate 3.6% 3.0%
Users 10% 12%
Non-users 2% 2%
Please Note: I have taken random numbers for convenience of explanation. These are not from any real test

22. What do you think now? Is ‘A’ better?
A B
Metric Variation A Variation B
Visits 1000 1000
Users 200 100
Non-users 800 900
Orders 36 30
Users 20 12
Non-users 16 18
Conversion Rate 3.6% 3.0%
Users 10% 12%
Non-users 2% 2%
Please Note: I have taken random numbers for convenience of explanation. These are not from any real test

23. Users are on your team
Image courtesy: Malte Wingen | unsplash
Keeping your ears open to what
users are saying is often rewarding

24. Users were not sure about the payment options available
till they reached the payment page
Please note: the screenshots have been simplified for illustration and do not completely resemble the actual version

25. Simply adding this information on the cart page improved
cart conversion
Please note: the screenshots have been simplified for illustration and do not completely resemble the actual version

26. Competition has it; do we need it too?
Please note: the screenshots have been simplified for illustration and do not completely resemble the actual version

27. Every website is unique
‘Competitive Benchmarking’ trap - just because a
leading website has it, doesn’t mean it will work for you

28. We added merchandising banners in the masthead

29. Only to realise that not many users clicked on it

30. Don’t lean towards
‘your’ idea
Aim to ‘TEST' if the idea is effective;
not ‘PROVE' that the idea is effective
Image courtesy: Marco Ceschi | unsplash

31. What is the best way to sort search results?
Please note: the screenshots have been simplified for illustration and do not completely resemble the actual version

32. Conversion improved but not for the reasons expected
Please note: the screenshots have been simplified for illustration and do not completely resemble the actual version

33. Did we implement this then?
Please note: the screenshots have been simplified for illustration and do not completely resemble the actual version

34. Every website
is unique
Don’t lean towards
‘your’ idea
Nothing is
obvious
There is no such thing
as a ‘failed experiment’
Users are on
your team
In a nutshell…
Avoid ‘Testing Myopia’

35. Thank you Thank you

36. QnA

37. [Next up]
An Interview with Amazon Prime Video’s
Sr. Product Lead Karan Peri
Date & Time - 22 July | 11 AM ET / 8 AM PT
Details - vwo.com/webcast