The document emphasizes that both data and customer feedback can be misleading and do not provide a complete picture of user needs and behaviors. It advocates for the combination of quantitative data and qualitative customer stories to guide better product decisions and acknowledges the necessity of understanding the context behind customer requests. Ultimately, it encourages avoiding minor optimizations and suggests being bold in testing new ideas to discover significant improvements.

1. Everyone’s Customers Are Wrong and
Their Data Is Lying
Evan Hamilton
Head of Community, UserVoice

2. DATA!
Big Data! Social Data! Mobile Data!
It’s the future!

3. Actually...
Data doesn’t tell the whole story.

4. Data doesn’t tell the whole story.
A/B test: will this carousel increase clicks?
^ incredibly ugly

5. Data doesn’t tell the whole story.
“Hey, look! The version of the page with the carousel is
making more people click through!”

6. Data doesn’t tell the whole story.
“Oh wait...it’s making more people click, not actually
convert.”
Tricky data at work!

7. Data doesn’t tell the whole story.
We created confusion, not conversions.

8. Data doesn’t tell the whole story.

9. Data doesn’t tell the whole story.
Hey, that looks good!

10. Data doesn’t tell the whole story.
Oops. They “forgot” to report refunds (amongst other things.)

11. PEOPLE!
Feelings! Empathy! Gut!
It’s the only way!

12. Actually...
People don’t tell the whole story.

13. People don’t tell the whole story.
“Wow, maybe we should build this. Let’s ask for more
feedback first...”

14. People don’t tell the whole story.
Oh. So they want this feature because they’re not
responding to customers. Next.

15. There’s always more context.
• What is the pain point?
• Are these paying users? Freeloaders?
• Are these people using your product in the
intended way?
• (We had someone using our forum to name their baby.
Cute, but not our target market.)
• Are they using your main features?
• Early “adopters”?
• (Robert Scoble is not your target market. He is fickle.)

16. People don’t tell the whole story.
“Makes sense, let’s build it!”

17. People don’t tell the whole story.
But customers found this confusing. (And it made us throw
up in our mouths a little bit.)

18. People don’t tell the whole story.
“Yet another online identity” is what we should have
focused on.

19. People don’t tell the whole story.
v2
No password. No signup. No registration.
Solved the problem without a complicated sign-in process.

20. Customers often give you bad solutions to
real problems.
(Don’t misinterpret this as good solutions
or not real problems.)

21. Everything is wrong.
Now what?

22. Data still has to be interpreted.

23. Data still has to be interpreted.
Requests can lead you astray.

24. Data still has to be interpreted.
Requests can lead you astray.
Combine data + customer stories.

25. Combine data + customer stories.
Back when we just had UserVoice Feedback, we’d lose
lots of customers after 1 year. So we asked them why
they were leaving.

26. Turns out they were growing up. New staff, new
department heads...& feedback forums didn’t have a
definitive place in the organization.

27. So we built a customer service tool into our product. Not
what we necessarily planned to build...but what customers
wanted.

28. It increased retention immensely. We now have a place in
the organization.
We got here because data alerted us to something, & user
stories filled out the details.

29. Don’t get lost in the data rat-hole.

30. Don’t get lost in the data rat-hole.
Don’t chase 1% when you can get 15%.

31. Don’t get lost in the data rat-hole.
Don’t chase 1% when you can get 15%.
Don’t get lost in minor homepage
optimization.

32. You’ll never know what you could get unless
you try something crazy.

33. Tried this yet? If not, go do that
before trying another shade of blue.

34. Tips, tricks, and tools
• (At least) Google Analytics
• Hell, even Bit.ly links give you a basis to work from
• Mixpanel
• The ol’ fake button
• How many people click a (fake) “calendar” button in your app? If 40% of
users do, you should build it!
• Test with segments
• Try tests on nonpaying users if you don’t want to scare off paying users
• Talk to customers (constantly)
• UserVoice

35. Thank you!
Evan Hamilton
Head of Community, UserVoice
@evanhamilton