SlideShare a Scribd company logo
1 of 14
Download to read offline
A/B Testing for Apps
How I Learned to Stop Worrying and Love
Data-Driven Decisions

Prepared for:
Coca-Cola Mobile Innovation Workshop
September 24-26, 2013
Shanghai, China

A/B Testing for Apps
User Base

Publish two different versions
of your app...
50% sees version A

...and see which one is driving
desirable user behavior.

50% sees version B
Which Version Won?
Version A

Version B

Source: WhichTestWon
Version B: 114% Improvement

✗

Version A

Version B

Marketer’s Surprise: ‘FREE’ Loses

✔
Multivariate Testing

• Every screen has X components (ex: Marilyn’s hair)
• For each, we can test Y variations (ex.: Green)
• In total, we have [Y1 x Y2 x Y3] combinations
How Obama Raised $60 Million

Source: Optimizely
Four Button Variations
Six Media Variations
24 Combinations!
And the Winner is...
+40%
increase in conversion
rate

2.9 million
additional donators

$60 million
value of additional
donations
Obamalytics
• Original Conversion Rate:
• New Conversion Rate:

8.3%
11.6%

• 10 million signups from New Version would have been
7.12 million signups with the Original Version
• +2.88 million additional signups
• $21 average donation per signup
• Approximately $60 million in additional donations
Testing to
Increase Virality
Headline
Call to Action
Workflow
Data-Driven App Marketing
• App marketing and analytics is in early stages
• App stores are opaque, approval processes slow down
iteration cycles
•

Few options for making decisions
• HiPPO: Highest Paid Person’s Opinion
• IHOP: In-House Optimization

• Scalable, reliable app measurement tools are emerging
Thank You!

www.splitforce.com
For more information:
Zac Aghion, CEO & Co-Founder
zac@splitforce.com
China: (+86)1592-1631-924
USA: (+1)617-750-6684

More Related Content

Viewers also liked

The Response Rates of Personalized Cross-Media Marketing Campaigns
The Response Rates of Personalized Cross-Media Marketing CampaignsThe Response Rates of Personalized Cross-Media Marketing Campaigns
The Response Rates of Personalized Cross-Media Marketing Campaigns
Aaron Corson
 
Putting the Fun in Functional
Putting the Fun in FunctionalPutting the Fun in Functional
Putting the Fun in Functional
Amy Jo Kim
 

Viewers also liked (8)

The Response Rates of Personalized Cross-Media Marketing Campaigns
The Response Rates of Personalized Cross-Media Marketing CampaignsThe Response Rates of Personalized Cross-Media Marketing Campaigns
The Response Rates of Personalized Cross-Media Marketing Campaigns
 
Facebook Application Marketing
Facebook Application MarketingFacebook Application Marketing
Facebook Application Marketing
 
Facebook Connect 5 Things You Wish You Had Known Before You Started
Facebook Connect   5 Things You Wish You Had Known Before You StartedFacebook Connect   5 Things You Wish You Had Known Before You Started
Facebook Connect 5 Things You Wish You Had Known Before You Started
 
Inactive Email Subscribers Reactivation Strategies
Inactive Email Subscribers Reactivation StrategiesInactive Email Subscribers Reactivation Strategies
Inactive Email Subscribers Reactivation Strategies
 
Buddymedia: Strategies for effective facebook wall posts
Buddymedia: Strategies for effective facebook wall postsBuddymedia: Strategies for effective facebook wall posts
Buddymedia: Strategies for effective facebook wall posts
 
Designing Your Reputation System
Designing Your Reputation SystemDesigning Your Reputation System
Designing Your Reputation System
 
Putting the Fun in Functional
Putting the Fun in FunctionalPutting the Fun in Functional
Putting the Fun in Functional
 
Naked startups - Lessons from the startup economy
Naked startups - Lessons from the startup economyNaked startups - Lessons from the startup economy
Naked startups - Lessons from the startup economy
 

More from Zac Aghion

Intro ab-taguchi
Intro ab-taguchiIntro ab-taguchi
Intro ab-taguchi
Zac Aghion
 
Splitforce at Barcamp Beijing
Splitforce at Barcamp BeijingSplitforce at Barcamp Beijing
Splitforce at Barcamp Beijing
Zac Aghion
 

More from Zac Aghion (6)

Retention is Every App's Game to Lose
Retention is Every App's Game to LoseRetention is Every App's Game to Lose
Retention is Every App's Game to Lose
 
Localytics - Predictive Retention Strategy
Localytics - Predictive Retention StrategyLocalytics - Predictive Retention Strategy
Localytics - Predictive Retention Strategy
 
Big Data, Little Devices: Mobile A/B Testing
Big Data, Little Devices: Mobile A/B TestingBig Data, Little Devices: Mobile A/B Testing
Big Data, Little Devices: Mobile A/B Testing
 
Intro ab-taguchi
Intro ab-taguchiIntro ab-taguchi
Intro ab-taguchi
 
Splitforce at Barcamp Beijing
Splitforce at Barcamp BeijingSplitforce at Barcamp Beijing
Splitforce at Barcamp Beijing
 
Conversion Optimization: Drive More Sales from Existing Traffic and Increase ...
Conversion Optimization: Drive More Sales from Existing Traffic and Increase ...Conversion Optimization: Drive More Sales from Existing Traffic and Increase ...
Conversion Optimization: Drive More Sales from Existing Traffic and Increase ...
 

Recently uploaded

Call Girls Navi Mumbai Just Call 9907093804 Top Class Call Girl Service Avail...
Call Girls Navi Mumbai Just Call 9907093804 Top Class Call Girl Service Avail...Call Girls Navi Mumbai Just Call 9907093804 Top Class Call Girl Service Avail...
Call Girls Navi Mumbai Just Call 9907093804 Top Class Call Girl Service Avail...
Dipal Arora
 
Call Girls Hebbal Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore
Call Girls Hebbal Just Call 👗 7737669865 👗 Top Class Call Girl Service BangaloreCall Girls Hebbal Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore
Call Girls Hebbal Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore
amitlee9823
 
Quick Doctor In Kuwait +2773`7758`557 Kuwait Doha Qatar Dubai Abu Dhabi Sharj...
Quick Doctor In Kuwait +2773`7758`557 Kuwait Doha Qatar Dubai Abu Dhabi Sharj...Quick Doctor In Kuwait +2773`7758`557 Kuwait Doha Qatar Dubai Abu Dhabi Sharj...
Quick Doctor In Kuwait +2773`7758`557 Kuwait Doha Qatar Dubai Abu Dhabi Sharj...
daisycvs
 
Chandigarh Escorts Service 📞8868886958📞 Just📲 Call Nihal Chandigarh Call Girl...
Chandigarh Escorts Service 📞8868886958📞 Just📲 Call Nihal Chandigarh Call Girl...Chandigarh Escorts Service 📞8868886958📞 Just📲 Call Nihal Chandigarh Call Girl...
Chandigarh Escorts Service 📞8868886958📞 Just📲 Call Nihal Chandigarh Call Girl...
Sheetaleventcompany
 
0183760ssssssssssssssssssssssssssss00101011 (27).pdf
0183760ssssssssssssssssssssssssssss00101011 (27).pdf0183760ssssssssssssssssssssssssssss00101011 (27).pdf
0183760ssssssssssssssssssssssssssss00101011 (27).pdf
Renandantas16
 
Call Girls From Pari Chowk Greater Noida ❤️8448577510 ⊹Best Escorts Service I...
Call Girls From Pari Chowk Greater Noida ❤️8448577510 ⊹Best Escorts Service I...Call Girls From Pari Chowk Greater Noida ❤️8448577510 ⊹Best Escorts Service I...
Call Girls From Pari Chowk Greater Noida ❤️8448577510 ⊹Best Escorts Service I...
lizamodels9
 

Recently uploaded (20)

How to Get Started in Social Media for Art League City
How to Get Started in Social Media for Art League CityHow to Get Started in Social Media for Art League City
How to Get Started in Social Media for Art League City
 
Call Girls Navi Mumbai Just Call 9907093804 Top Class Call Girl Service Avail...
Call Girls Navi Mumbai Just Call 9907093804 Top Class Call Girl Service Avail...Call Girls Navi Mumbai Just Call 9907093804 Top Class Call Girl Service Avail...
Call Girls Navi Mumbai Just Call 9907093804 Top Class Call Girl Service Avail...
 
VVVIP Call Girls In Greater Kailash ➡️ Delhi ➡️ 9999965857 🚀 No Advance 24HRS...
VVVIP Call Girls In Greater Kailash ➡️ Delhi ➡️ 9999965857 🚀 No Advance 24HRS...VVVIP Call Girls In Greater Kailash ➡️ Delhi ➡️ 9999965857 🚀 No Advance 24HRS...
VVVIP Call Girls In Greater Kailash ➡️ Delhi ➡️ 9999965857 🚀 No Advance 24HRS...
 
Call Girls Hebbal Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore
Call Girls Hebbal Just Call 👗 7737669865 👗 Top Class Call Girl Service BangaloreCall Girls Hebbal Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore
Call Girls Hebbal Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore
 
Quick Doctor In Kuwait +2773`7758`557 Kuwait Doha Qatar Dubai Abu Dhabi Sharj...
Quick Doctor In Kuwait +2773`7758`557 Kuwait Doha Qatar Dubai Abu Dhabi Sharj...Quick Doctor In Kuwait +2773`7758`557 Kuwait Doha Qatar Dubai Abu Dhabi Sharj...
Quick Doctor In Kuwait +2773`7758`557 Kuwait Doha Qatar Dubai Abu Dhabi Sharj...
 
Call Girls Service In Old Town Dubai ((0551707352)) Old Town Dubai Call Girl ...
Call Girls Service In Old Town Dubai ((0551707352)) Old Town Dubai Call Girl ...Call Girls Service In Old Town Dubai ((0551707352)) Old Town Dubai Call Girl ...
Call Girls Service In Old Town Dubai ((0551707352)) Old Town Dubai Call Girl ...
 
Chandigarh Escorts Service 📞8868886958📞 Just📲 Call Nihal Chandigarh Call Girl...
Chandigarh Escorts Service 📞8868886958📞 Just📲 Call Nihal Chandigarh Call Girl...Chandigarh Escorts Service 📞8868886958📞 Just📲 Call Nihal Chandigarh Call Girl...
Chandigarh Escorts Service 📞8868886958📞 Just📲 Call Nihal Chandigarh Call Girl...
 
Falcon Invoice Discounting platform in india
Falcon Invoice Discounting platform in indiaFalcon Invoice Discounting platform in india
Falcon Invoice Discounting platform in india
 
The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Commun...
The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Commun...The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Commun...
The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Commun...
 
MONA 98765-12871 CALL GIRLS IN LUDHIANA LUDHIANA CALL GIRL
MONA 98765-12871 CALL GIRLS IN LUDHIANA LUDHIANA CALL GIRLMONA 98765-12871 CALL GIRLS IN LUDHIANA LUDHIANA CALL GIRL
MONA 98765-12871 CALL GIRLS IN LUDHIANA LUDHIANA CALL GIRL
 
Forklift Operations: Safety through Cartoons
Forklift Operations: Safety through CartoonsForklift Operations: Safety through Cartoons
Forklift Operations: Safety through Cartoons
 
Organizational Transformation Lead with Culture
Organizational Transformation Lead with CultureOrganizational Transformation Lead with Culture
Organizational Transformation Lead with Culture
 
RSA Conference Exhibitor List 2024 - Exhibitors Data
RSA Conference Exhibitor List 2024 - Exhibitors DataRSA Conference Exhibitor List 2024 - Exhibitors Data
RSA Conference Exhibitor List 2024 - Exhibitors Data
 
Mysore Call Girls 8617370543 WhatsApp Number 24x7 Best Services
Mysore Call Girls 8617370543 WhatsApp Number 24x7 Best ServicesMysore Call Girls 8617370543 WhatsApp Number 24x7 Best Services
Mysore Call Girls 8617370543 WhatsApp Number 24x7 Best Services
 
Call Girls Pune Just Call 9907093804 Top Class Call Girl Service Available
Call Girls Pune Just Call 9907093804 Top Class Call Girl Service AvailableCall Girls Pune Just Call 9907093804 Top Class Call Girl Service Available
Call Girls Pune Just Call 9907093804 Top Class Call Girl Service Available
 
0183760ssssssssssssssssssssssssssss00101011 (27).pdf
0183760ssssssssssssssssssssssssssss00101011 (27).pdf0183760ssssssssssssssssssssssssssss00101011 (27).pdf
0183760ssssssssssssssssssssssssssss00101011 (27).pdf
 
Call Girls From Pari Chowk Greater Noida ❤️8448577510 ⊹Best Escorts Service I...
Call Girls From Pari Chowk Greater Noida ❤️8448577510 ⊹Best Escorts Service I...Call Girls From Pari Chowk Greater Noida ❤️8448577510 ⊹Best Escorts Service I...
Call Girls From Pari Chowk Greater Noida ❤️8448577510 ⊹Best Escorts Service I...
 
Dr. Admir Softic_ presentation_Green Club_ENG.pdf
Dr. Admir Softic_ presentation_Green Club_ENG.pdfDr. Admir Softic_ presentation_Green Club_ENG.pdf
Dr. Admir Softic_ presentation_Green Club_ENG.pdf
 
Call Girls In Panjim North Goa 9971646499 Genuine Service
Call Girls In Panjim North Goa 9971646499 Genuine ServiceCall Girls In Panjim North Goa 9971646499 Genuine Service
Call Girls In Panjim North Goa 9971646499 Genuine Service
 
John Halpern sued for sexual assault.pdf
John Halpern sued for sexual assault.pdfJohn Halpern sued for sexual assault.pdf
John Halpern sued for sexual assault.pdf
 

A/B Testing for Apps (Coca-Cola Mobile Innovation Workshop)

  • 1. A/B Testing for Apps How I Learned to Stop Worrying and Love Data-Driven Decisions Prepared for: Coca-Cola Mobile Innovation Workshop September 24-26, 2013 Shanghai, China A/B Testing for Apps
  • 2. User Base Publish two different versions of your app... 50% sees version A ...and see which one is driving desirable user behavior. 50% sees version B
  • 3. Which Version Won? Version A Version B Source: WhichTestWon
  • 4. Version B: 114% Improvement ✗ Version A Version B Marketer’s Surprise: ‘FREE’ Loses ✔
  • 5. Multivariate Testing • Every screen has X components (ex: Marilyn’s hair) • For each, we can test Y variations (ex.: Green) • In total, we have [Y1 x Y2 x Y3] combinations
  • 6. How Obama Raised $60 Million Source: Optimizely
  • 10. And the Winner is... +40% increase in conversion rate 2.9 million additional donators $60 million value of additional donations
  • 11. Obamalytics • Original Conversion Rate: • New Conversion Rate: 8.3% 11.6% • 10 million signups from New Version would have been 7.12 million signups with the Original Version • +2.88 million additional signups • $21 average donation per signup • Approximately $60 million in additional donations
  • 13. Data-Driven App Marketing • App marketing and analytics is in early stages • App stores are opaque, approval processes slow down iteration cycles • Few options for making decisions • HiPPO: Highest Paid Person’s Opinion • IHOP: In-House Optimization • Scalable, reliable app measurement tools are emerging
  • 14. Thank You! www.splitforce.com For more information: Zac Aghion, CEO & Co-Founder zac@splitforce.com China: (+86)1592-1631-924 USA: (+1)617-750-6684