1. How I learned to hate
‘engagement’
(and love subjective feedback)

2. Title text
Arthur Etchells
Director of Product Management
Imaging Resource

3. Our product looked like
this

5. We added a cool product
tour shortly after launch

7. People loved it!

8. ‘Engagement’ was through the roof!
People loved it!

9. “Lorem ipsum dolor sit
amet, consectetur
adipiscing elit, set eiusmod
tempor incidunt et labore
et dolore magna aliquam.”
– First Name Last Name, Attribution
Or did they?

10. “Lorem ipsum dolor sit
amet, consectetur
adipiscing elit, set eiusmod
tempor incidunt et labore
et dolore magna aliquam.”
– First Name Last Name, Attribution
Or did they?
Was ‘engagement’ what we wanted?

11. • Yes, engagement was great (Optimizely)

12. • Yes, engagement was great (Optimizely)
• Bounce rates were not (Google Analytics)

13. • Yes, engagement was great (Optimizely)
• Bounce rates were not (Google Analytics)
• Months pass

14. • Yes, engagement was great (Optimizely)
• Bounce rates were not (Google Analytics)
• Months pass
• Some internal feedback (that tour is annoying!) …

15. • Yes, engagement was great (Optimizely)
• Bounce rates were not (Google Analytics)
• Months pass
• Some internal feedback (that tour is annoying!) …
• Led to further research…

16. Title Text
• Crazy Egg
• In-page click analysis

17. Showed us something
interesting…

19. • Bullets only
• Up to five items
• One to four words
• First letter is capitalized
• Periods are not used

20. • Bullets only
• Up to five items
• One to four words
• First letter is capitalized
• Periods are not used

21. • Bullets only
• Up to five items
• One to four words
• First letter is capitalized
• Periods are not used

22. The Question

23. Do users interact more with our
content* after viewing a product
tour?

24. *non-product tour content
Do users interact more with our
content* after viewing a product
tour?

25. The Experiment

26. Two Variations

29. Two Metrics

30. Engagement*

31. Engagement*
*But not as the primary goal

32. Clicks on non-product tour content

33. Results

34. Title Text
• Engagement

35. Title Text
• Engagement
• Dropped by 20.4%

36. Title Text
• Clicks on non-product tour
links

37. Title Text
• Clicks on non-product tour
links
• Increased by 36.3%

38. Lessons Learned

39. 1

40. Be careful what you wish for

41. Be careful what you wish optimize
for

42. Engagement may not be
the best metric to
optimize

47. • Yes, engagement was great (Optimizely)
• Bounce rates were not (Google Analytics)

48. • Yes, engagement was great (Optimizely)
• Bounce rates were not (Google Analytics)
Any click

49. • Yes, engagement was great (Optimizely)
• Bounce rates were not (Google Analytics)
Pageviews and events
Any click

52. Optimizely: Not a fan of the engagement metric either

53. Engagement may not be
the best metric to
optimize

54. Engagement may not be is
not the best metric to
optimize

55. 2

56. Make sure product tours are
actually doing what you hope
they are

57. Make sure product tours are
actually doing what you hope
they areThere should be a high bar for anything you put between users and your
content

58. 3

59. Take a holistic approach
to optimization efforts

60. Take a holistic approach
to optimization efforts
Subjective feedback can let you uncover issues earlier

61. User
feedback
sessions
In-page
feedback
prompt
HeatmapsAnalytics
Biz goals:
revenue,
lead gen
Objective Subjective

62. User
feedback
sessions
In-page
feedback
prompt
HeatmapsAnalytics
Biz goals:
revenue,
lead gen
Objective Subjective

63. User
feedback
sessions
In-page
feedback
prompt
HeatmapsAnalytics
Biz goals:
revenue,
lead gen
Objective Subjective

64. • Those were the tools we used, but there are lots of
options…

65. • Those were the tools we used, but there are lots of
options…
• Analytics (Omniture/Google Analytics/Mixpanel/Heap)

66. • Those were the tools we used, but there are lots of
options…
• Analytics (Omniture/Google Analytics/Mixpanel/Heap)
• Heatmaps (ClickTale/CrazyEgg/HotJar)

67. • Those were the tools we used, but there are lots of
options…
• Analytics (Omniture/Google Analytics/Mixpanel/Heap)
• Heatmaps (ClickTale/CrazyEgg/HotJar)
• In-page feedback forms (Qualaroo/HotJar)

68. • Those were the tools we used, but there are lots of
options…
• Analytics (Omniture/Google Analytics/Mixpanel/Heap)
• Heatmaps (ClickTale/CrazyEgg/HotJar)
• In-page feedback forms (Qualaroo/HotJar)
• User feedback sessions (In-person/UserTesting)

69. Subjective feedback can allow
you to uncover usability problems
earlier

70. Subjective feedback can allow
you to uncover usability problems
earlierWhat does that look like in practice?

71. UserTesting/HotJar/Optimizely
Discussion…

73. “Lorem ipsum dolor sit
amet, consectetur
adipiscing elit, set eiusmod
tempor incidunt et labore
et dolore magna aliquam.”
– First Name Last Name, Attribution
Questions?