Dan siroker data testing and business business of software conference 2013
Upcoming SlideShare
Loading in...5
×
 

Dan siroker data testing and business business of software conference 2013

on

  • 250 views

Dan Siroker, CEO, Optimizely talks about Data Testing and Business Business of Software Conference 2013

Dan Siroker, CEO, Optimizely talks about Data Testing and Business Business of Software Conference 2013

Statistics

Views

Total Views
250
Views on SlideShare
250
Embed Views
0

Actions

Likes
0
Downloads
9
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Dan siroker data testing and business business of software conference 2013 Dan siroker data testing and business business of software conference 2013 Presentation Transcript

  • A/B Testing If data can help win elections, what can it help you do for your business? Dan Siroker Co-Founder & CEO
  • ‘05 ‘06 ‘07 ‘08 ’09 ’10
  • ‘05 ‘06 ‘07 ‘08 ’09 ’10
  • Millions $7 $6 $5 $4 $3 $2 $1 $0 2010 Q2 Q1 Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Q4
  • Millions $7 Team: 4 Customers: 644 Annual Run Rate: $0.1m $6 $5 $4 $3 $2 $1 Dan Pete Eric $0 2010 Q2 Q1 Elliot Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Q4
  • Millions $7 Team: 10 (+150%) Customers: 1,508 (+134%) Annual Run Rate: $1.2m(+982%) $6 $5 $4 $3 $2 Julio Ryan M $1 Ricky Meng Xiao Dan Mat Jeff P Pete Eric $0 2010 Q2 Q1 Elliot Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Q4
  • Millions $7 Team: 42 (+320%) Customers: 3,460 (+129%) Annual Run Rate: $7.6m (+533%) $6 $5 Steve Tyle Jeff Z Arthu Amina Greg Sandy $4 Brendan Sophie Dogan Sonesh Shanann Tommy $3 Dan G Cara Byron Mik Alan Eddie Jodie $2 Dae-Ho Cyrus Julio Ryan M $1 Lauren Ashley Ricky Meng Xiao Dan Mat Cristian Travis Jeff P Pete Eric $0 2010 Q2 Q1 Elliot Katy Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Ryan L Ian Jonathan Q4 Joanna Jon
  • Millions $7 $6 $5 $4 $3 $2 $1 $0 2010 Q2 Q1 Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Q4
  • Millions $7 $6 $5 $4 $3 $2 $1 $0 2010 Q1 Q2 Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Q4
  • Millions $28 $27 $26 $25 $24 $23 $22 $21 $20 $19 $18 $17 $16 $15 $14 $13 $12 $11 $10 $9 $8 $7 $6 $5 $4 $3 $2 $1 $0 2010 Q2 Q1 Q3 Q4 2011 Q2 Q1 Q3 Actual Q4 2012 Q2 Q1 Projection Q3 Q4 2013 Q2 Q1 Q3 Q4
  • Millions $28 $27 $26 $25 $24 $23 $22 $21 $20 $19 $18 $17 $16 $15 $14 $13 $12 $11 $10 $9 $8 $7 $6 $5 $4 $3 $2 $1 $0 Steve Team: 42 Tyle Jeff Z Arthu Amina Greg Sophie Dogan Cara Alan Dan G Shanann Byron Eddie Sandy Dae-Ho Cyrus Sonesh Brendan Cristian Mik Ricky Dan Pete Eric 2010 Q2 Q1 Elliot Q3 Jeff P Lauren Ashley Tommy Ryan M Jodie Mat Meng Xiao Julio Katy Travis Jonathan Q4 2011 Q2 Q1 Q3 Actual Q4 2012 Q2 Q1 Projection Ryan L Q3 Q4 2013 Q2 Joanna Q1 Ian Jon Q3 Q4
  • Millions $28 $27 $26 $25 $24 $23 $22 $21 $20 $19 $18 $17 $16 $15 $14 $13 $12 $11 $10 $9 $8 $7 $6 $5 $4 $3 $2 $1 $0 Steve Steve Team: 130 (+210%) Tyle Marius Adam Bo Evan Junan Ash Grace Anshu Lucas Benson Bryce Darwish Jeff Z Arthu David Jaron Richar Tammie Khattaab Laura Asia Amanda Amina Pamela Brian Greg Nikhil Martjin Sophie Amelia Dogan Gary Marco Natarajan Ben Kasra Katie Andrea Daniel SG Dae-Ho Omari Andy Eric Andy Mustafa Ranjit Chris Waylon Pedro Bradley Neha Lauren Adrian Chrix Cristian Mik Ricky Robin Dan Pete Eric 2010 Q2 Q1 Elliot Q3 Jeff P Ashley Tommy Ryan M Mat Julio Katy Travis Lisa Jamie Jonathan Q4 2011 Q2 Q1 Q3 Actual Q4 2012 Q2 Q1 Projection Nick Miyo Jodie Meng Xiao Ryan L Q3 Becca Oren Cyrus Brendan Megan Byron Eddie Sonesh Hemant Shanann Brett Sandy Ryan Michael James Etienne Alan Lacy Dan G Luke Cara Westly Aaron Adam Q4 2013 Q2 Helen Joanna Q1 Jen Ian Jon Nick Fabian Saum Courtney Dan Yonat Kent Ural Eduar Toby Oliver Grant Q3 Cate Christop Julia Q4 Danielle Tai Jonathan Chris Chris Rama
  • How?
  • Limited Life Experience
  • Limited Life Experience + Overgeneralization
  • Limited Life Experience + Overgeneralization = Advice -Paul Buchheit
  • Lesson #1: Define success.
  • Lesson #1: Define success .
  • Lesson #1: Define quantifiable success metrics.
  • visitors per day Visit sign up rate $ per recipient Email $
  • visitors per day Visit sign up rate $ per recipient Email $
  • Media Button
  • Splash page experiment Variations: Button: 1. Sign Up Media: 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
  • Button: “Sign Up”
  • Button: “Learn More”
  • Button: “Join Us Now”
  • Button: “Sign Up Now”
  • Splash page experiment Variations: Button: 1. Sign Up Media: 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
  • Media: “Get Involved”
  • Media: “Family”
  • Media: “Change”
  • Media: “Barack’s Video”
  • Media: “Springfield Video”
  • Media: “Sam’s Video”
  • Splash page experiment Variations: Button: 1. Sign Up Media: 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
  • Splash page experiment
  • Splash page experiment
  • Splash page experiment
  • Splash page experiment Email Volunteers Subscriptions 712,000 Amount Raised Original: 7,120,000 $143,000,000 +40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000
  • Splash page experiment Email Volunteers Subscriptions 712,000 Amount Raised Original: 7,120,000 $143,000,000 +40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000
  • Splash page experiment Email Volunteers Subscriptions 712,000 Amount Raised Original: 7,120,000 $143,000,000 +40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000
  • Lesson #1: Define quantifiable success metrics.
  • Lesson #2: Less is more. Reduce choices.
  • vs. +15% conversion rate +10% $ per visitor
  • vs. +20% submissions
  • vs. +16.8% $ per visitor
  • vs. +43.8% purchases
  • Lesson #2: Less is more. Reduce choices.
  • Lesson #3: Words matter. Focus on your call to action.
  • Donation button experiment Variations
  • Donation button experiment Variations Never Signed Up Signed Up, Never Donated Previously Donated 0.0% 0.0% 0.0% +2.3% +27.8% +16.3% -27.8% N/A N/A +15.2% -24.6% +11.9% +8.5% +2.9% +18.4%
  • vs. +14.6% clicks
  • vs. +15.75% $ per visitor
  • Lesson #3: Words matter. Focus on your call to action.
  • Lesson #4: Seek the global maximum.
  • Source: http://blog.intercom.io/criticism-and-two-way-streets/
  • vs.
  • Source: http://blog.netflix.com/2011/06/new-look-and-feel-for-netflix-website.html
  • vs. +0.5% retention +1.4% engagement
  • vs. +600% engagement
  • vs. +49% conversion rate from 240 A/B tests
  • Lesson #4: Seek the global maximum.
  • Lesson #5: Start today.
  • Source: http://techcrunch.com/2013/04/10/optimizely-series-a/
  • Source: http://techcrunch.com/2013/04/10/optimizely-series-a/
  • 5,000+ Happy Customers!
  • Lesson #5: Start today.
  • Lessons Learned 1. Define quantifiable success metrics. 2. Less is more. Reduce choices. 3. Words matter. Focus on your call to action. 4. Seek the global maximum. 5. Start today.
  • One more thing...
  • A/B Testing If data can help win elections, what can it help you do for your business? Dan Siroker Co-Founder & CEO