This document discusses the debate between design decisions based on intuition versus data-driven analytics. It outlines a conversation between a designer and product manager where the PM wanted decisions based on quantitative data from A/B tests, while the designer preferred qualitative user research and past experience. However, A/B tests have limitations and can produce inconclusive or misleading results if not used correctly. The document argues that designers should view data as one input among many, and look at the broader context, systemic effects, and motivations behind any changes. It encourages finding ways to combine qualitative and quantitative research to make the most informed decisions.

1. Design vs. Data:
Battle Royale
March 6, 2019

2. Kim Moy
Capital One
Columbia University
Bill Prickett
Pre/Post Producer
merged with UX Design
Hello!
Matthew Weber
Development Guild
Columbia University

6. Who is hiring?
Raise your hand

7. Design vs. Data:
Battle Royale
with Jess Dale

8. PM
DESIGN

9. a conversation between
me and my product
manager
basically went like this...

11. like many designers, my
answers come from:
qualitative input from users, feedback from team, “best
practices”, past experience, material design specs...

12. PMs want to get answers
from numbers and data
the stuff that *MATTERS*

13. should designers use
data* to make decisions?
*in this case quantitative data (numbers)

14. battleground:
a/b tests
intuition vs. analytics

15. the first blow
feeling forced to choose between revenue gains or
usability wins

17. drinking the kool-aid
showing numbers to people who need to see numbers

18. start asking too much of
the a/b test
should we change our CTA from red to green?
should we test our design system?

20. you think you’re getting:
- neutral, unbiased results
- move from a hunch to a sure thing
- prove a ‘winner’

21. you actually get:
- numbers stand in for rigor and objectivity
- a false or inconclusive “conclusion"
- set up for confirmation bias

22. what a/b tests ignore
- larger context
- systemic effects
- WHY

24. the whole thing can go
sideways quick
when we aren’t using a/b tests correctly

25. why im calling bullsh*t
- test can only be as good as your assumptions
- results can be easily manipulated
- completely ignore “why”

26. a/b tests cannot give you
answers!
they’re a tool for measuring performance against a metric

27. be skeptical!
you’ll get a result, but what is it actually telling you?

29. if not a/b tests, then
what?
gonna have to figure out how to use data, while also
keeping track of the whole dynamic system

30. maybe designers could
reframe how they think
about data*
*again, in this case quantitative data (numbers)

31. maybe there’s a way to
combine quant + qual
and to bring your PM along for this ride!

32. let’s make it actionable
thinking about data as a building block

33. Concept by: Stephen Fraser;
Poor graphic design skillz by: me
Layers of
Decision-making

34. what does this mean in
practice?
with data is our starting point, let’s zoom out and look at
the context

35. do your research
I cracked open a book, literally
written on this very topic!
buy it.

36. design what you want to
validate
think about the data capture, analysis and questions as
part of the process

37. get the framing right
it has a compounding effect on subsequent work

38. everything takes
investigation
though there’s an element of instinct, can’t be solely
based on intuition

39. you need to be able to
evaluate solutions
both input and data help form a testable hypothesis

40. steer clear of the illusion
of “winning” results
insights alone won’t give you the answers

41. above all else, you need to
understand the WHY
that’s your source of truth

43. key takeaways:
data can’t make
decisions
data can’t tell us what the problem is, but it can help us
expose them and provide more information

44. key takeaways:
go with your gut,
but back it up
design is an exercise in using empathy and intuition to
come up with solutions to a problem

45. key takeaways:
partner with your PM
winning as a team!

46. See my next talk entitled:
why do we still have
carts? lolwtf
fin.