Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
All About the User Experience:
Data Science at MailChimp
John Foreman
September 2013
OR guy
Former consultant
I heart large, slow organizations
Most companies are
not in the business of
analytics
Improve the product
Improve the user experience
Improve the product
Improve the user experience
Internal External
Insights
Capabilities
Customers
Internal External
20%
80%
Insights
Capabilities
Internal External
Insights
Capabilities
Internal External
Insights
Capabilities
“Send email, all
the time, as
much as you
can…Obama.”
0.0%
1.0%
2.0%
3.0%
4.0%
5.0%
6.0%
7.0%
8.0%
0 5 10 15
CampaignClickRate
Sends per Month
User X: Sends per month vs
campai...
0.0%
0.5%
1.0%
1.5%
2.0%
2.5%
3.0%
3.5%
4.0%
4.5%
5.0%
0 10 20 30
CampaignClickRate
Sends per Month
User Y: Sends per mont...
0.0%
0.5%
1.0%
1.5%
2.0%
2.5%
3.0%
3.5%
4.0%
4.5%
5.0%
0 10 20 30
CampaignClickRate
Sends per Month
User Y: Sends per mont...
0
50
100
150
200
250
0 10 20 30
Totalmonthlyclicks
Sends per Month
User Y: Sends per month vs
total monthly clicks (list s...
Internal External
Insights
Capabilities
Compliance
Sends campaign w/
high bounce rate
Sends campaign w/
high unsub rate
Sends campaign w/
high abuse rate
Warning (Still
Enab...
Sends campaign w/
high bounce rate
Sends campaign w/
high unsub rate
Sends campaign w/
high abuse rate
Warning (Still
Enab...
Warning (Still
Enabled)
Disable/Review
Replies
Ignores/Waits/A
cknowledges
Disabled
Enabled
Review
(can still send
at curr...
“Account Review
takes too much
time!
My needs is
gone!”
Stop reviewing
accounts every
time they make a
change. We paid
you for
a service…your
service is too
restrictive…
“Bullshit review process...I'm sending out high Press Releases about this
time wasting process...Google me and you would k...
Permission
Hygiene
Permission
Hygiene
Purchased
Hard Bounce
Final
modeling
set
Inter-
mediate
Modeling
Data
Collection
Category
User
Acct
metadata
Acct
Metadata
List
Address
history ...
Design to eliminate
risk/complexity
Function Tool
Slow Storage Sharded PostgreSQL
Fast Storage Redis
Modeling Language R
M...
Sends campaign w/
high bounce rate
Sends campaign w/
high unsub rate
Sends campaign w/
high abuse rate
Warning (Still
Enab...
Internal External
Insights
Capabilities
If I am to speak 10 minutes, I need a week for prep;
If 15 minutes, three days;
If half an hour, two days;
If an hour, I a...
Internal External
Insights
Capabilities
Customers
Communicating
Technical
HARD
A product is only as powerful
as it is easy to use
Example: Discovered Segments
Example: Discovered Segments
Make Data More Valuable
Inside
the Product than
Outside
Align Yourself & Serve
Align Yourself & Serve
No Special Treatment
Align Yourself & Serve
No Special Treatment
1) Goal 2) Toys
Align Yourself & Serve
No Special Treatment
1) Goal 2) Toys
Avoid Complexity
Thanks!
@John4man
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Ona
Upcoming SlideShare
Loading in …5
×

Ona

438 views

Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Ona

  1. 1. All About the User Experience: Data Science at MailChimp John Foreman September 2013
  2. 2. OR guy Former consultant I heart large, slow organizations
  3. 3. Most companies are not in the business of analytics
  4. 4. Improve the product Improve the user experience
  5. 5. Improve the product Improve the user experience
  6. 6. Internal External Insights Capabilities Customers
  7. 7. Internal External 20% 80% Insights Capabilities
  8. 8. Internal External Insights Capabilities
  9. 9. Internal External Insights Capabilities
  10. 10. “Send email, all the time, as much as you can…Obama.”
  11. 11. 0.0% 1.0% 2.0% 3.0% 4.0% 5.0% 6.0% 7.0% 8.0% 0 5 10 15 CampaignClickRate Sends per Month User X: Sends per month vs campaign click rate
  12. 12. 0.0% 0.5% 1.0% 1.5% 2.0% 2.5% 3.0% 3.5% 4.0% 4.5% 5.0% 0 10 20 30 CampaignClickRate Sends per Month User Y: Sends per month vs campaign click rate
  13. 13. 0.0% 0.5% 1.0% 1.5% 2.0% 2.5% 3.0% 3.5% 4.0% 4.5% 5.0% 0 10 20 30 CampaignClickRate Sends per Month User Y: Sends per month vs campaign click rate
  14. 14. 0 50 100 150 200 250 0 10 20 30 Totalmonthlyclicks Sends per Month User Y: Sends per month vs total monthly clicks (list size 1000)
  15. 15. Internal External Insights Capabilities
  16. 16. Compliance
  17. 17. Sends campaign w/ high bounce rate Sends campaign w/ high unsub rate Sends campaign w/ high abuse rate Warning (Still Enabled) Disable/Review Replies Ignores/Waits/A cknowledges Disabled Enabled Review (can still send at current level) User Action Consequence User Action Final State Replies Ignores/Waits Disabled Enabled High bounce rate High unsub rate High abuse rate Warning (Still Enabled) Disable/Review Review (can still send at current level) Replies Ignores/Waits Disabled Enabled Warning Disable Review (can still send at current level) Direct complaint Hardcoded spam rule Causes IP blacklisting Hits URL blacklist Multiple accounts No unsub link TOS violation Large List Upload Permission reminder Tries to buy $X+ Requests HV approval 33% 17% 2% 1% 3% 19% 3% 1% 3% 2% 14% 2%
  18. 18. Sends campaign w/ high bounce rate Sends campaign w/ high unsub rate Sends campaign w/ high abuse rate Warning (Still Enabled) Disable/Review Replies Ignores/Waits/A cknowledges Disabled Enabled Review (can still send at current level) Replies Ignores/Waits Disabled Enabled High bounce rate High unsub rate High abuse rate Warning (Still Enabled) Disable/Review Review (can still send at current level) Replies Ignores/Waits Disabled Enabled Warning Disable Review (can still send at current level) 33% 17% 2% User Action Consequence User Action Final State
  19. 19. Warning (Still Enabled) Disable/Review Replies Ignores/Waits/A cknowledges Disabled Enabled Review (can still send at current level) Replies Ignores/Waits Disabled Enabled Warning (Still Enabled) Disable/Review Review (can still send at current level) Replies Ignores/Waits Disabled Enabled Warning Disable Review (can still send at current level) Large List Upload Tries to buy $X+ Requests HV approval 3% 2% 14% User Action Consequence User Action Final State
  20. 20. “Account Review takes too much time! My needs is gone!”
  21. 21. Stop reviewing accounts every time they make a change. We paid you for a service…your service is too restrictive…
  22. 22. “Bullshit review process...I'm sending out high Press Releases about this time wasting process...Google me and you would know I'm a serious marketer with influence. Ill definitely make sure everyone I know in this industry knows what a pain in the ass this was.”
  23. 23. Permission Hygiene
  24. 24. Permission Hygiene Purchased Hard Bounce
  25. 25. Final modeling set Inter- mediate Modeling Data Collection Category User Acct metadata Acct Metadata List Address history Bounce Model List scores Campaign Campaign history Prev. camp. perf. Content Content Model Content scores Predictions FinalModel Public Lists
  26. 26. Design to eliminate risk/complexity Function Tool Slow Storage Sharded PostgreSQL Fast Storage Redis Modeling Language R Machine Learning Model Random Forest API + Background Processes Python
  27. 27. Sends campaign w/ high bounce rate Sends campaign w/ high unsub rate Sends campaign w/ high abuse rate Warning (Still Enabled) Disable/Review Replies Ignores/Waits/A cknowledges Disabled Enabled Review (can still send at current level) Replies Ignores/Waits Disabled Enabled High bounce rate High unsub rate High abuse rate Warning (Still Enabled) Disable/Review Review (can still send at current level) Replies Ignores/Waits Disabled Enabled Warning Disable Review (can still send at current level) Large List Upload Tries to buy $X+ Requests HV approval 33% 17% 2% 3% 2% 14% 70+% tickets automatable
  28. 28. Internal External Insights Capabilities
  29. 29. If I am to speak 10 minutes, I need a week for prep; If 15 minutes, three days; If half an hour, two days; If an hour, I am ready now. -Woodrow Wilson
  30. 30. Internal External Insights Capabilities Customers Communicating Technical HARD
  31. 31. A product is only as powerful as it is easy to use
  32. 32. Example: Discovered Segments
  33. 33. Example: Discovered Segments
  34. 34. Make Data More Valuable Inside the Product than Outside
  35. 35. Align Yourself & Serve
  36. 36. Align Yourself & Serve No Special Treatment
  37. 37. Align Yourself & Serve No Special Treatment 1) Goal 2) Toys
  38. 38. Align Yourself & Serve No Special Treatment 1) Goal 2) Toys Avoid Complexity
  39. 39. Thanks! @John4man

×