Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Krishna Gade
Why everything is an A/B test at
Pinterest
Confidentia
l
2
What is Pinterest?
A visual bookmarking tool
Discover an inspiring idea
Save it to a board
Go do it
Pinterest is a data product
Data Flow
App
events
Kafka
Secor
Skyline
Pinball
Redshift
Pinalytics
Features
Qubole
(Hadoop)
Singer
Confidentia
l
Metrics
7
Confidentia
l
User Lifecycle
New
Users
Active Users
Dormant Users
Acquisition
Activation
Churn
Churn Resurrection
Confidentia
l
Terminology
● MAU – Monthly Active User
● WAU – Weekly Active User
● DAU – Daily Active User
● WAR – Weekly ...
Confidentia
l
MAUs
● Current progress to quarterly goal
● MAU forecast
● MAUs by app/gender/country
● MAU Accounting
Confidentia
l
MAU Accounting
Confidentia
l
Acquisition Metrics
● Total Signups
● Signups by App
● Signups by Referrer
● Invites Sent
● Unique Invite Se...
Confidentia
l
Activation Metrics
Activation Rate = 1rc7/signups
● Activation Rate by App/Gender/Referrer
● Overall signups...
Confidentia
l
Engagement Metrics
● *AU ratios
● # of Emails sent
● # of Emails opened
● # of Emails clicked
● # of Notific...
Confidentia
l
Resurrection Metrics
● # of Users who were
dormant for 28+ days
but came back, split by
platform, referrer e...
Confidentia
l
Takeaways
● Emphasize on activation (process of converting
a new user to MAU)
● Activation critical to long-...
Confidentia
l
Automated Metrics
17
Confidentia
l
Confidentia
l
Chart creation
Confidentia
l
Reuse existing query result
Confidentia
l
Metrics creation
Confidentia
l
A/B Experimentation
22
Confidentia
l
Categories of Experiments
● A/B testing new features
● Copy testing
● SEO changes
● Infrastructure updates
Confidentia
l
A/B Testing Framework
Confidentia
l
Confidentia
l
Confidentia
l
Confidentia
l
A/B Testing Dashboard
Confidentia
l
Confidentia
l
p-value (via chi-squared test)
Statistical tests for validation
Confidentia
l
Impact of Badging
Confidentia
l
Impact of Badging
● 7% lift in DAU
● 2.5% lift over long term
● Effective at increasing
engagement levels
Confidentia
l
Copy Testing
Definition: Testing different variations of text we
display to the user.
● A/B Testing copy was...
Confidentia
l
New Approach
● Ability to run a copy test w/o changing a line of
code.
● Support for translating text dynami...
Confidentia
l
Multi Armed Bandits
Confidentia
l
SEO
36
50+ Billion Pins
categorized by people
into more than
1 Billion Boards
Confidentia
l
37
What makes SEO so challenging?
What is the
Impact of the
new SEO
feature?
Will this new
change affect
SEO...
Confidentia
l
KEEP
CALM
and
RUN AN
SEO EXPERIMENT
Confidentia
l
What is an SEO experiment?
39
Regular AB Experiment SEO Experiment
Metrics User behavior Search engine refer...
Confidentia
l
What is an SEO experiment?
Launch
● Measure the gap
● Before vs After launch
Confidentia
l
Dashboard
41
DIFF = enabled - control
Traffic Gain
Confidentia
l
42
Decision Making
1 2
3
4
SHIP!
SHIP!
SHUT DOWN!
VALIDATE
Confidentia
l
43
Experiment Driven SEO
What is the
Impact of the
new SEO
feature?
We got XX% of
traffic growth from it!
Ca...
Confidentia
l
Why you should be A/B testing
even your infrastructure
Confidentia
l
Infrastructure Changes
● 2013: A new web framework
● 2014: Pyapns
● 2015: User service
Confidentia
l
Experiment results for replacing PyAPNs
Confidentia
l
Conclusions
● Figure out what success means
● Define your metrics
● Automate metric computation
● A/B test e...
Confidentia
l
Thank You!
Why everything is an A/B Test at Pinterest
Upcoming SlideShare
Loading in …5
×

Why everything is an A/B Test at Pinterest

15,218 views

Published on

A/B Experimentation at Pinterest

Published in: Data & Analytics
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Doctor's 2-Minute Ritual For Shocking Daily Belly Fat Loss! Watch This Video ♣♣♣ http://ishbv.com/bkfitness3/pdf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Doctor's 2-Minute Ritual For Shocking Daily Belly Fat Loss! Watch This Video ◆◆◆ http://scamcb.com/bkfitness3/pdf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Download The Complete Lean Belly Breakthrough Program with Special Discount. ■■■ https://tinyurl.com/bkfitness4u
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Why everything is an A/B Test at Pinterest

  1. 1. Krishna Gade Why everything is an A/B test at Pinterest
  2. 2. Confidentia l 2
  3. 3. What is Pinterest? A visual bookmarking tool Discover an inspiring idea Save it to a board Go do it
  4. 4. Pinterest is a data product
  5. 5. Data Flow App events Kafka Secor Skyline Pinball Redshift Pinalytics Features Qubole (Hadoop) Singer
  6. 6. Confidentia l Metrics 7
  7. 7. Confidentia l User Lifecycle New Users Active Users Dormant Users Acquisition Activation Churn Churn Resurrection
  8. 8. Confidentia l Terminology ● MAU – Monthly Active User ● WAU – Weekly Active User ● DAU – Daily Active User ● WAR – Weekly Active Repinner ● WAC – Weekly Active Clickthrougher ● WARC – Weekly Active Repinner or Clickthrougher ● 1d7 – User who comes back at-least once in 7 days after signup ● 1rc7 - User who repins/clicks at-least once in 7 days after signup
  9. 9. Confidentia l MAUs ● Current progress to quarterly goal ● MAU forecast ● MAUs by app/gender/country ● MAU Accounting
  10. 10. Confidentia l MAU Accounting
  11. 11. Confidentia l Acquisition Metrics ● Total Signups ● Signups by App ● Signups by Referrer ● Invites Sent ● Unique Invite Senders ● Invite Signups
  12. 12. Confidentia l Activation Metrics Activation Rate = 1rc7/signups ● Activation Rate by App/Gender/Referrer ● Overall signups to 1rc7 ● Signup engagement funnel ● Signups to WAU 35 days after signup
  13. 13. Confidentia l Engagement Metrics ● *AU ratios ● # of Emails sent ● # of Emails opened ● # of Emails clicked ● # of Notifications sent ● # of Notifications clicked
  14. 14. Confidentia l Resurrection Metrics ● # of Users who were dormant for 28+ days but came back, split by platform, referrer etc
  15. 15. Confidentia l Takeaways ● Emphasize on activation (process of converting a new user to MAU) ● Activation critical to long-term growth ● Deeply understand how different segments of users interact
  16. 16. Confidentia l Automated Metrics 17
  17. 17. Confidentia l
  18. 18. Confidentia l Chart creation
  19. 19. Confidentia l Reuse existing query result
  20. 20. Confidentia l Metrics creation
  21. 21. Confidentia l A/B Experimentation 22
  22. 22. Confidentia l Categories of Experiments ● A/B testing new features ● Copy testing ● SEO changes ● Infrastructure updates
  23. 23. Confidentia l A/B Testing Framework
  24. 24. Confidentia l
  25. 25. Confidentia l
  26. 26. Confidentia l
  27. 27. Confidentia l A/B Testing Dashboard
  28. 28. Confidentia l
  29. 29. Confidentia l p-value (via chi-squared test) Statistical tests for validation
  30. 30. Confidentia l Impact of Badging
  31. 31. Confidentia l Impact of Badging ● 7% lift in DAU ● 2.5% lift over long term ● Effective at increasing engagement levels
  32. 32. Confidentia l Copy Testing Definition: Testing different variations of text we display to the user. ● A/B Testing copy was a tedious process ● Even when we did this we were optimizing only for English
  33. 33. Confidentia l New Approach ● Ability to run a copy test w/o changing a line of code. ● Support for translating text dynamically ● Use Multi-armed bandits to converge to showing best performing text
  34. 34. Confidentia l Multi Armed Bandits
  35. 35. Confidentia l SEO 36 50+ Billion Pins categorized by people into more than 1 Billion Boards
  36. 36. Confidentia l 37 What makes SEO so challenging? What is the Impact of the new SEO feature? Will this new change affect SEO? Uhhh... Dunno... Uhhh... Ask Google.
  37. 37. Confidentia l KEEP CALM and RUN AN SEO EXPERIMENT
  38. 38. Confidentia l What is an SEO experiment? 39 Regular AB Experiment SEO Experiment Metrics User behavior Search engine referred traffic Unique id for hashing userid / sessionid page url
  39. 39. Confidentia l What is an SEO experiment? Launch ● Measure the gap ● Before vs After launch
  40. 40. Confidentia l Dashboard 41 DIFF = enabled - control Traffic Gain
  41. 41. Confidentia l 42 Decision Making 1 2 3 4 SHIP! SHIP! SHUT DOWN! VALIDATE
  42. 42. Confidentia l 43 Experiment Driven SEO What is the Impact of the new SEO feature? We got XX% of traffic growth from it! Can we use more resources to make bigger impact? No, the experiment result is flat, so let’s ship it. SEO experiment SEO experiment Is my UI change affecting SEO negatively?
  43. 43. Confidentia l Why you should be A/B testing even your infrastructure
  44. 44. Confidentia l Infrastructure Changes ● 2013: A new web framework ● 2014: Pyapns ● 2015: User service
  45. 45. Confidentia l Experiment results for replacing PyAPNs
  46. 46. Confidentia l Conclusions ● Figure out what success means ● Define your metrics ● Automate metric computation ● A/B test everything!
  47. 47. Confidentia l Thank You!

×