1.
Driving Customer Experience
Through Testing, Research &
Analytics
Vab Dwivedi
Dell | Director, Testing & Research
@vabdwivedi
linkedin.com/in/vabdwivedi

2.
2 of Y
What’s our approach?

3.
3 of Y
Our tenets
W1
A2
D3
G4
S5
B6

4.
4 of Y
Our tenets
We Are Customer Obsessed1
A2
D3
G4
S5
B6

5.
5 of Y
Our tenets
W1
Act as Subject Matter Experts2
D3
G4
S5
B6

6.
6 of Y
Our tenets
W1
A2
Deliver Advanced Analytics In A Consumable Manner3
G4
S5
B6

7.
7 of Y
Our tenets
W1
A2
D3
Guide Decision Making Through Insights And Experimentation4
S5
B6

8.
8 of Y
Our tenets
W1
A2
D3
G4
Serve As A Channel of Knowledge5
B6

9.
9 of Y
Our tenets
W1
A2
D3
G4
S5
Be Agile6

10.
10 of Y
Our tenets
W1
A2
D3
G4
S5
B6

11.
11 of Y
Holding ourselves accountable
In1
D2
D3
M4
E5
D6

12.
12 of Y
Holding ourselves accountable
Influencing Agenda/Roadmap1
D2
D3
M4
E5
D6

13.
13 of Y
Holding ourselves accountable
In1
Delivering Actionable Insights2
D3
M4
E5
D6

14.
14 of Y
Holding ourselves accountable
In1
D2
Delighting Our Customers3
M4
E5
D6

15.
15 of Y
Holding ourselves accountable
In1
D2
D3
Managing Stakeholders4
E5
D6

16.
16 of Y
Holding ourselves accountable
In1
D2
D3
M4
Executing With Operational Excellence5
D6

17.
17 of Y
Holding ourselves accountable
In1
D2
D3
M4
E5
Driving An Innovative Vision6

18.
18 of Y
Holding ourselves accountable
In1
D2
D3
M4
E5
D6

19.
Setting our team up for
success

20.
20 of Y

21.
21 of Y

22.
22 of Y

23.
23 of Y

24.
24 of Y
That was ‘how’…let’s
look at some projects…

25.
Dell.com Home Page

26.
26 of Y

27.
27 of Y

28.
28 of Y

29.
29 of Y

30.
30 of Y
Iterative testing over 18 months…

31.
31 of Y
Iterative testing over 18 months…

32.
32 of Y
We measured EVERYTHING!

33.
33 of Y
We measured EVERYTHING!

34.
34 of Y
We measured EVERYTHING!

35.
35 of Y
We measured EVERYTHING!

36.
36 of Y
We measured EVERYTHING!

37.
37 of Y
We measured EVERYTHING!

38.
38 of Y
…And then some more…

39.
39 of Y
…And then some more…

40.
40 of Y
…And then some more…

41.
41 of Y
…And then some more…

42.
42 of Y
…And then some more…

43.
43 of Y
…Look into device type performance…

44.
44 of Y
…Look into device type performance…

45.
45 of Y
…Look into device type performance…

46.
46 of Y
…Look into device type performance…

47.
Drive Big Change:
A new Dell.com

48.
48 of Y

49.
49 of Y

50.
50 of Y
Not just a new visual identity…

51.
51 of Y
Not just a new visual identity…
The new Dell.com was new in every way….platform,
architecture, back end services

52.
52 of Y
Not just a new visual identity…
The new Dell.com was new in every way….platform,
architecture, back end services
Before going live, test it to ensure a win for customers
and the business

53.
53 of Y
Not just a new visual identity…
The new Dell.com was new in every way….platform,
architecture, back end services
Before going live, test it to ensure a win for customers
and the business
Expecting a quick win….

54.
54 of Y
Not just a new visual identity…
The new Dell.com was new in every way….platform,
architecture, back end services
Before going live, test it to ensure a win for customers
and the business
Expecting a quick win….
… -33% revenue/visit

55.
55 of Y
OK, no big deal…we know what to do

56.
56 of Y
OK, no big deal…we know what to do
Let’s introduce UI enhancements

57.
57 of Y
OK, no big deal…we know what to do
Let’s introduce UI enhancements
24 of them…through various parts in the site

58.
58 of Y
OK, no big deal…we know what to do
Let’s introduce UI enhancements
24 of them…through various parts in the site
Recipe C will be our super recipe that will bridge the
gap when we re-run the test

59.
59 of Y
OK, no big deal…we know what to do
Let’s introduce UI enhancements
24 of them…through various parts in the site
Recipe C will be our super recipe that will bridge the
gap when we re-run the test
… -45% revenue/visit

60.
60 of Y
Maybe its not the UI…

61.
61 of Y
Maybe its not the UI…
It is a new platform…

62.
62 of Y
Maybe its not the UI…
It is a new platform…
We need re-run and this time look at the metrics that
can point to any issues with the platform

63.
63 of Y
Performance metrics

64.
64 of Y
Performance metrics

65.
65 of Y
Performance metrics

66.
66 of Y
Performance metrics

67.
67 of Y
Performance metrics

68.
68 of Y
Performance metrics

69.
69 of Y
Performance metrics

70.
70 of Y
Well that explains it…

71.
71 of Y
Well that explains it…
Now we can work with engineering and provide
guidance on where the issues are

72.
72 of Y
Well that explains it…
Now we can work with engineering and provide
guidance on where the issues are
Emergency sprints scheduled to address live issues

73.
73 of Y
Well that explains it…
Now we can work with engineering and provide
guidance on where the issues are
Emergency sprints scheduled to address live issues
Re-test now….

74.
74 of Y
Well that explains it…
Now we can work with engineering and provide
guidance on where the issues are
Emergency sprints scheduled to address live issues
Re-test now….
… +3% revenue/visit!!!

75.
Find something small that
can make an impact:
Continue shopping

76.
76 of Y
Should ‘Continue Shopping’ be smarter?

77.
77 of Y
Should ‘Continue Shopping’ be smarter?

78.
78 of Y
Should ‘Continue Shopping’ be smarter?

79.
79 of Y
Should ‘Continue Shopping’ be smarter?

80.
80 of Y
Did it work?

81.
81 of Y
Did it work?

82.
82 of Y
Are you sure?

83.
83 of Y
Are you sure?

84.
84 of Y
Are you sure?

85.
85 of Y
Are you sure?

86.
86 of Y
Now iterate, globalize and future proof

87.
87 of Y
Now iterate, globalize and future proof

88.
88 of Y
Now iterate, globalize and future proof

89.
89 of Y
Now iterate, globalize and future proof

90.
90 of Y
Now iterate, globalize and future proof

91.
91 of Y
Now iterate, globalize and future proof

92.
92 of Y
Now iterate, globalize and future proof

93.
93 of Y
Now iterate, globalize and future proof

94.
Rapid fire: Everything doesn’t
have to be a ‘project’…

95.
95 of Y
Cycling through order codes
vs

96.
96 of Y
Cycling through order codes
vs
+9.1% Conversion
+11.9% Revenue/Visit
+91.6% Button Engagement
-1.49% Exit rates
+.6% Compare usage

97.
97 of Y
Adding payment methods to cart
vs

98.
98 of Y
Adding payment methods to cart
vs
+4.3% Conversion
+10.5% Revenue/Visit
+2.2% Payment Pg > Confirm Page

99.
99 of Y
Results sorting method
vs

100.
100 of Y
Results sorting method
vs
+6.7% Conversion
+15% Revenue/Visit
+51% Conversion for highest price
band
-64% Engagement with sort option

101.
101 of Y
Category page filter layout
vs

102.
102 of Y
Category page filter layout
vs
-7.5% Conversion
-6.4% Revenue/Visit
-40% Engagement
-17% Conversion for
engagers
-8% Next page

103.
103 of Y
Sequencing options for systems
Based on business rules
and availability
Hardware, Software,
Services, Misc.
vs

104.
104 of Y
Sequencing options for systems
Based on business rules
and availability
Hardware, Software,
Services, Misc.
vs
+13.8% Conversion
+15.5% Revenue/Visit
+12.7% Margin/Visit

105.
105 of Y
Clickable tabs and ‘Add to Cart’ CTA
vs

106.
106 of Y
Clickable tabs and ‘Add to Cart’ CTA
vs
+11% Conversion
+52% Engagement w/tabs & CTAs
+13% Movement through funnel
-2% Exit rate

107.
107 of Y
Sign in page titles
vs

108.
108 of Y
Sign in page titles
vs
+3% Conversion
-4% Guest checkout
+3.7% Movement through funnel
+1% Authentication

109.
109 of Y
Navigation title change
vs

110.
110 of Y
Navigation title change
-3.5% Conversion
-5.8% Revenue/Visit
-3% Engagement
No Change Movement to
browse
vs

111.
If you only remember
three things from
today…

112.
112 of Y
Our daily mantra

113.
113 of Y
Our daily mantra
You want iterative testing & a LOT of data1
Winners and Losers are good2
Keep it simple for your customer3

114.
114 of Y

115.
Driving Customer Experience
Through Testing, Research &
Analytics
Vab Dwivedi
Dell | Director, Testing & Research
@vabdwivedi
linkedin.com/in/vabdwivedi