1. All About the User Experience:
Data Science at MailChimp
John Foreman
September 2013

2. OR guy
Former consultant
I heart large, slow organizations

19. Most companies are
not in the business of
analytics

21. Improve the product
Improve the user experience

28. Improve the product
Improve the user experience

31. Internal External
Insights
Capabilities
Customers

32. Internal External
20%
80%
Insights
Capabilities

33. Internal External
Insights
Capabilities

36. Internal External
Insights
Capabilities

38. “Send email, all
the time, as
much as you
can…Obama.”

39. 0.0%
1.0%
2.0%
3.0%
4.0%
5.0%
6.0%
7.0%
8.0%
0 5 10 15
CampaignClickRate
Sends per Month
User X: Sends per month vs
campaign click rate

40. 0.0%
0.5%
1.0%
1.5%
2.0%
2.5%
3.0%
3.5%
4.0%
4.5%
5.0%
0 10 20 30
CampaignClickRate
Sends per Month
User Y: Sends per month vs
campaign click rate

41. 0.0%
0.5%
1.0%
1.5%
2.0%
2.5%
3.0%
3.5%
4.0%
4.5%
5.0%
0 10 20 30
CampaignClickRate
Sends per Month
User Y: Sends per month vs
campaign click rate

42. 0
50
100
150
200
250
0 10 20 30
Totalmonthlyclicks
Sends per Month
User Y: Sends per month vs
total monthly clicks (list size 1000)

43. Internal External
Insights
Capabilities

44. Compliance

45. Sends campaign w/
high bounce rate
Sends campaign w/
high unsub rate
Sends campaign w/
high abuse rate
Warning (Still
Enabled)
Disable/Review
Replies
Ignores/Waits/A
cknowledges
Disabled
Enabled
Review
(can still send
at current level)
User Action Consequence User Action Final State
Replies
Ignores/Waits
Disabled
Enabled
High bounce rate
High unsub rate
High abuse rate
Warning (Still
Enabled)
Disable/Review
Review
(can still send
at current level)
Replies
Ignores/Waits
Disabled
Enabled
Warning
Disable
Review
(can still send
at current
level)
Direct complaint
Hardcoded spam rule
Causes IP blacklisting
Hits URL blacklist
Multiple accounts
No unsub link
TOS violation
Large List Upload
Permission reminder
Tries to buy $X+
Requests HV approval
33%
17%
2%
1%
3%
19%
3%
1%
3%
2%
14%
2%

46. Sends campaign w/
high bounce rate
Sends campaign w/
high unsub rate
Sends campaign w/
high abuse rate
Warning (Still
Enabled)
Disable/Review
Replies
Ignores/Waits/A
cknowledges
Disabled
Enabled
Review
(can still send
at current level)
Replies
Ignores/Waits
Disabled
Enabled
High bounce rate
High unsub rate
High abuse rate
Warning (Still
Enabled)
Disable/Review
Review
(can still send
at current level)
Replies
Ignores/Waits
Disabled
Enabled
Warning
Disable
Review
(can still send
at current
level)
33%
17%
2%
User Action Consequence User Action Final State

47. Warning (Still
Enabled)
Disable/Review
Replies
Ignores/Waits/A
cknowledges
Disabled
Enabled
Review
(can still send
at current level)
Replies
Ignores/Waits
Disabled
Enabled
Warning (Still
Enabled)
Disable/Review
Review
(can still send
at current level)
Replies
Ignores/Waits
Disabled
Enabled
Warning
Disable
Review
(can still send
at current
level)
Large List Upload
Tries to buy $X+
Requests HV approval
3%
2%
14%
User Action Consequence User Action Final State

48. “Account Review
takes too much
time!
My needs is
gone!”

49. Stop reviewing
accounts every
time they make a
change. We paid
you for
a service…your
service is too
restrictive…

50. “Bullshit review process...I'm sending out high Press Releases about this
time wasting process...Google me and you would know I'm a serious
marketer with influence. Ill definitely make sure everyone I know in this
industry knows what a pain in the ass this was.”

51. Permission
Hygiene

52. Permission
Hygiene
Purchased
Hard Bounce

53. Final
modeling
set
Inter-
mediate
Modeling
Data
Collection
Category
User
Acct
metadata
Acct
Metadata
List
Address
history Bounce
Model
List scores
Campaign
Campaign
history
Prev. camp.
perf.
Content
Content
Model
Content
scores
Predictions
FinalModel
Public Lists

54. Design to eliminate
risk/complexity
Function Tool
Slow Storage Sharded PostgreSQL
Fast Storage Redis
Modeling Language R
Machine Learning Model Random Forest
API + Background Processes Python

55. Sends campaign w/
high bounce rate
Sends campaign w/
high unsub rate
Sends campaign w/
high abuse rate
Warning (Still
Enabled)
Disable/Review
Replies
Ignores/Waits/A
cknowledges
Disabled
Enabled
Review
(can still send
at current level)
Replies
Ignores/Waits
Disabled
Enabled
High bounce rate
High unsub rate
High abuse rate
Warning (Still
Enabled)
Disable/Review
Review
(can still send
at current level)
Replies
Ignores/Waits
Disabled
Enabled
Warning
Disable
Review
(can still send
at current
level)
Large List Upload
Tries to buy $X+
Requests HV approval
33%
17%
2%
3%
2%
14%
70+% tickets automatable

56. Internal External
Insights
Capabilities

57. If I am to speak 10 minutes, I need a week for prep;
If 15 minutes, three days;
If half an hour, two days;
If an hour, I am ready now.
-Woodrow Wilson

60. Internal External
Insights
Capabilities
Customers
Communicating
Technical
HARD

61. A product is only as powerful
as it is easy to use

62. Example: Discovered Segments

63. Example: Discovered Segments

74. Make Data More Valuable
Inside
the Product than
Outside

75. Align Yourself & Serve

76. Align Yourself & Serve
No Special Treatment

77. Align Yourself & Serve
No Special Treatment
1) Goal 2) Toys

78. Align Yourself & Serve
No Special Treatment
1) Goal 2) Toys
Avoid Complexity

79. Thanks!
@John4man