Getting Started with
Optimizely for Mobile
Apps
Suneet Shah
Senior Product Manager,
Optimizely
@suneets
suneet@optimizely.com
#opticon2015
#opticon2015
Suneet Shah
In The Next 45 Minutes
1. Customer Stories
2. Demos
3. Test Ideas
4. Best Practices for getting started
5. Q & A
• iOS Developer Preview
Launched
• A/B testing a new concept for
mobile apps
Last Year
Today
• iOS and Android
• New Visual Editor
• Tons of customer feedback
• Hundreds of apps on iOS and
Android
Time spent per adult user per day (USA)
Source: Mary Meeker internet trends report 2015
0
1.5
3
4.5
6
2012 2013 2014 2015 (YTD)
0.40.3
0.3
0.3
2.82.6
2.3
1.6
2.42.42.32.5
Desktop/Laptop Mobile Other Connected Devices
} 51%
} 42%
HoursPerDay
Another 1b people will come online through mobile
0
1
2
3
4
1995 2000 2014 2020
People Online Smartphones
Source: a16z Benedict Evans “Mobile is eating the world”
BillionPeopleOnline
Cost Per Install (CPI) rising
April 2014 April 2015
Cost Per Install (iOS) $1.38 $2.13
Cost Per Install
(Android)
$1.31 $2.08
Source: FiKSU mobile marketing index https://www.fiksu.com/resources/fiksu-indexes#analysis
+54%
On average, apps lose 80% of their new users
Source: Andrew Chen’s blog, http://andrewchen.co/
The top apps follow the same pattern
Source: Andrew Chen’s blog, http://andrewchen.co/
If you’re not testing your
app yet, you should be!
Customer Stories
French Girls Optimizes Activation
• Activation is the difference
between 1-2 days of use and 7+
days of use
• Activation rate of 25%
Baseline activation rate of 25%
Baseline activation rate of 25%
Winning variation doubled activation to 50%
Activation
50%
Copy changes
+16%
“Optimization is no longer an
afterthought, but a necessary
part of every feature”
- Jeff Farkas, French Girls
French Girls
• Used code blocks to increase activation
from 25% to 50%
• Subsequent visual changes found an
additional 16.5% improvement
• Total improvement: from 25% to 66.5%
Secret Escapes Optimizes Acquisition
• To force sign up, or not to force
sign up
• Looking to optimize LTV, sign up
rates, and justify mobile ad spend
Experiment Setup
Version 1 Version 2
Sign up gate with
mandatory sign in on first
user experience
Sign up not required,
added a “skip” button to
close the sign up form
Results
• Defying all expectations, a mandatory
signup gate was by far the optimal
experience
• Sign up rates doubled, LTV average
increased, and LTV:CAC ratio was
positive
“If the skip button was part of
the onboarding flow, we
would not be able to do paid
marketing on mobile.”
– Sebastian Fallert, Secret
Escapes
Optimizely for iOS and Android
Demo
Demo
• SDK
• Visual Editor
• Live Variables
• Code Blocks
SDK Installation
• Optimizely SDK needs
to be integrated and
deployed
• 1 line integration
Visual Changes
• Copy changes, calls to
action, images and layout
changes
• Create tests without
releasing a new app
• No code required
Live Variables & Code Blocks
• Have to be built into the
app before release
• Easy setup for developers
• Allow you to test anything
Body text goes here.
Live Variables
Regular Variable
private static final String ratingsPromptText = "Do you love our app?”;
alertDialog.setText(ratingsPromptText);
Optimizely Live Variable
private static LiveVariable<String> ratingsPromptText =
Optimizely.stringVariable("RatingsPrompt1","Do you love our app?”);
alertDialog.setText(ratingsPromptText.get())
Body text goes here.
Live Variables
Regular Variable
private static final String ratingsPromptText = "Do you love our app?”;
alertDialog.setText(ratingsPromptText);
Optimizely Live Variable
private static LiveVariable<String> ratingsPromptText =
Optimizely.stringVariable("RatingsPrompt1","Do you love our app?”);
alertDialog.setText(ratingsPromptText.get())
OptimizelyCodeBlocksKeyWithBlockNames(CheckoutFunnel,
@"Default Checkout”,
@"One Stage Checkout”,
@"Paypal Checkout”);
Code Blocks
[Optimizely codeBlocksWithKey:CheckoutFunnel
blockOne:^{
[self showOneStageCheckout];
}
blockTwo:^{
[self showPaypalCheckout];
}
defaultBlock:^{
[self showDefaultCheckout];
}];
Mobile Testing Ideas
Framework for Mobile Apps
Acquisition Engagement
Retention Monetization
Framework for Mobile Apps
Acquisition Engagement
Retention Monetization
Framework for Mobile Apps
Acquisition Engagement
Retention Monetization
Engagement
• Layouts and navigation
patterns
• Discoverability and
placement of key features
• Test every new feature
Facebook layout test
Source: LukeW’s Design blog, http://www.lukew.com
LukeW’s Polar App
Source: LukeW’s Design blog, http://www.lukew.com
Test every new feature
Phased Rollouts
Test key feature placements
1. Date selector

2. Repeat map CTA as floating button

3. Narrower focus + information
hierarchy

2
3
1
Framework for Mobile Apps
Acquisition Engagement
Retention Monetization
Monetization
• Checkout flows and purchase
funnels
• Ad networks
• Virality flows and sharing
features
Calls to action in your purchase flow
Payment Options
Payment Options
Ad Placements
Social/sharing
Framework for Mobile Apps
Acquisition Engagement
Retention Monetization
Retention
• Permissions - Push
Notifications, Location,
Camera
• Reviews
Permissions
Only one chance
Asking for push notification permissions
When and how you ask
Reviews
Source: Circa’s blog article, https://medium.com/circa/the-right-way-to-ask-users-to-review-your-app-9a32fd604fca
Optimizing Reviews
3 Things They Optimized:
Threshold (Timing)
of Request Location of Request Flow
of Request
10 Sessions
3x in 1
Week
Retention for top apps
The best way to improve
retention is to improve
onboarding and
activation
Framework for Mobile Apps
Acquisition Engagement
Retention Monetization
Best Practices
5 Key Steps To Get Started
Integrate the SDK
Start using in development
Plan Testing Roadmap
What metrics are you improving?
Implement First 3 Tests
Before going to the app store
QA Experiments
Verify metrics
Submit to the App Store
And start your experiments
1
2
3
4
5
Plan Ahead
• Make testing part of your release cycle
• Plan tests upfront
• Keep an experiment backlog
Start With 3 Tests
• Get familiar with Optimizely
• Visual Change
• Code Blocks
• Live Variables
• QA your experiments the way your users will see them
• Verify conversions and visitors
• Use SDKs’ developer features in QA process
• Use Preview feature
Experiment QA
#opticon2015
Suneet Shah
Senior Product Manager,
Optimizely
@suneets
suneet@optimizely.com
Thank you!
#opticon2015
10 Secrets to Building an Amazing Mobile Testing Roadmap
Thurs 2:30 - 3:15 | Coit Tower Room
Optimizely Product Feedback Roundtable Discussions
Thurs 11:25 - 12:10 | Expo Hall
Roundtable - Optimizing for Mobile Retention
Thurs 2:35 – 3:12 | Expo Hall

Option 2015- Getting Started with Optimizely for Mobile

  • 1.
    Getting Started with Optimizelyfor Mobile Apps Suneet Shah Senior Product Manager, Optimizely @suneets suneet@optimizely.com #opticon2015
  • 2.
  • 3.
    In The Next45 Minutes 1. Customer Stories 2. Demos 3. Test Ideas 4. Best Practices for getting started 5. Q & A
  • 4.
    • iOS DeveloperPreview Launched • A/B testing a new concept for mobile apps Last Year
  • 5.
    Today • iOS andAndroid • New Visual Editor • Tons of customer feedback • Hundreds of apps on iOS and Android
  • 6.
    Time spent peradult user per day (USA) Source: Mary Meeker internet trends report 2015 0 1.5 3 4.5 6 2012 2013 2014 2015 (YTD) 0.40.3 0.3 0.3 2.82.6 2.3 1.6 2.42.42.32.5 Desktop/Laptop Mobile Other Connected Devices } 51% } 42% HoursPerDay
  • 7.
    Another 1b peoplewill come online through mobile 0 1 2 3 4 1995 2000 2014 2020 People Online Smartphones Source: a16z Benedict Evans “Mobile is eating the world” BillionPeopleOnline
  • 8.
    Cost Per Install(CPI) rising April 2014 April 2015 Cost Per Install (iOS) $1.38 $2.13 Cost Per Install (Android) $1.31 $2.08 Source: FiKSU mobile marketing index https://www.fiksu.com/resources/fiksu-indexes#analysis +54%
  • 9.
    On average, appslose 80% of their new users Source: Andrew Chen’s blog, http://andrewchen.co/
  • 10.
    The top appsfollow the same pattern Source: Andrew Chen’s blog, http://andrewchen.co/
  • 11.
    If you’re nottesting your app yet, you should be!
  • 12.
  • 14.
    French Girls OptimizesActivation • Activation is the difference between 1-2 days of use and 7+ days of use • Activation rate of 25%
  • 15.
  • 16.
  • 17.
    Winning variation doubledactivation to 50% Activation 50% Copy changes +16%
  • 18.
    “Optimization is nolonger an afterthought, but a necessary part of every feature” - Jeff Farkas, French Girls
  • 19.
    French Girls • Usedcode blocks to increase activation from 25% to 50% • Subsequent visual changes found an additional 16.5% improvement • Total improvement: from 25% to 66.5%
  • 21.
    Secret Escapes OptimizesAcquisition • To force sign up, or not to force sign up • Looking to optimize LTV, sign up rates, and justify mobile ad spend
  • 22.
    Experiment Setup Version 1Version 2 Sign up gate with mandatory sign in on first user experience Sign up not required, added a “skip” button to close the sign up form
  • 23.
    Results • Defying allexpectations, a mandatory signup gate was by far the optimal experience • Sign up rates doubled, LTV average increased, and LTV:CAC ratio was positive
  • 24.
    “If the skipbutton was part of the onboarding flow, we would not be able to do paid marketing on mobile.” – Sebastian Fallert, Secret Escapes
  • 25.
    Optimizely for iOSand Android Demo
  • 26.
    Demo • SDK • VisualEditor • Live Variables • Code Blocks
  • 27.
    SDK Installation • OptimizelySDK needs to be integrated and deployed • 1 line integration
  • 29.
    Visual Changes • Copychanges, calls to action, images and layout changes • Create tests without releasing a new app • No code required
  • 30.
    Live Variables &Code Blocks • Have to be built into the app before release • Easy setup for developers • Allow you to test anything
  • 31.
    Body text goeshere. Live Variables Regular Variable private static final String ratingsPromptText = "Do you love our app?”; alertDialog.setText(ratingsPromptText); Optimizely Live Variable private static LiveVariable<String> ratingsPromptText = Optimizely.stringVariable("RatingsPrompt1","Do you love our app?”); alertDialog.setText(ratingsPromptText.get())
  • 32.
    Body text goeshere. Live Variables Regular Variable private static final String ratingsPromptText = "Do you love our app?”; alertDialog.setText(ratingsPromptText); Optimizely Live Variable private static LiveVariable<String> ratingsPromptText = Optimizely.stringVariable("RatingsPrompt1","Do you love our app?”); alertDialog.setText(ratingsPromptText.get())
  • 33.
    OptimizelyCodeBlocksKeyWithBlockNames(CheckoutFunnel, @"Default Checkout”, @"One StageCheckout”, @"Paypal Checkout”); Code Blocks [Optimizely codeBlocksWithKey:CheckoutFunnel blockOne:^{ [self showOneStageCheckout]; } blockTwo:^{ [self showPaypalCheckout]; } defaultBlock:^{ [self showDefaultCheckout]; }];
  • 34.
  • 35.
    Framework for MobileApps Acquisition Engagement Retention Monetization
  • 36.
    Framework for MobileApps Acquisition Engagement Retention Monetization
  • 37.
    Framework for MobileApps Acquisition Engagement Retention Monetization
  • 38.
    Engagement • Layouts andnavigation patterns • Discoverability and placement of key features • Test every new feature
  • 39.
    Facebook layout test Source:LukeW’s Design blog, http://www.lukew.com
  • 40.
    LukeW’s Polar App Source:LukeW’s Design blog, http://www.lukew.com
  • 41.
    Test every newfeature Phased Rollouts
  • 42.
    Test key featureplacements 1. Date selector
 2. Repeat map CTA as floating button
 3. Narrower focus + information hierarchy
 2 3 1
  • 43.
    Framework for MobileApps Acquisition Engagement Retention Monetization
  • 44.
    Monetization • Checkout flowsand purchase funnels • Ad networks • Virality flows and sharing features
  • 45.
    Calls to actionin your purchase flow
  • 46.
  • 47.
  • 48.
  • 49.
  • 50.
    Framework for MobileApps Acquisition Engagement Retention Monetization
  • 51.
    Retention • Permissions -Push Notifications, Location, Camera • Reviews
  • 52.
  • 53.
  • 54.
    Asking for pushnotification permissions
  • 55.
    When and howyou ask
  • 56.
    Reviews Source: Circa’s blogarticle, https://medium.com/circa/the-right-way-to-ask-users-to-review-your-app-9a32fd604fca
  • 57.
    Optimizing Reviews 3 ThingsThey Optimized: Threshold (Timing) of Request Location of Request Flow of Request 10 Sessions 3x in 1 Week
  • 58.
  • 60.
    The best wayto improve retention is to improve onboarding and activation
  • 61.
    Framework for MobileApps Acquisition Engagement Retention Monetization
  • 62.
  • 63.
    5 Key StepsTo Get Started Integrate the SDK Start using in development Plan Testing Roadmap What metrics are you improving? Implement First 3 Tests Before going to the app store QA Experiments Verify metrics Submit to the App Store And start your experiments 1 2 3 4 5
  • 64.
    Plan Ahead • Maketesting part of your release cycle • Plan tests upfront • Keep an experiment backlog
  • 65.
    Start With 3Tests • Get familiar with Optimizely • Visual Change • Code Blocks • Live Variables
  • 66.
    • QA yourexperiments the way your users will see them • Verify conversions and visitors • Use SDKs’ developer features in QA process • Use Preview feature Experiment QA
  • 67.
    #opticon2015 Suneet Shah Senior ProductManager, Optimizely @suneets suneet@optimizely.com Thank you!
  • 68.
    #opticon2015 10 Secrets toBuilding an Amazing Mobile Testing Roadmap Thurs 2:30 - 3:15 | Coit Tower Room Optimizely Product Feedback Roundtable Discussions Thurs 11:25 - 12:10 | Expo Hall Roundtable - Optimizing for Mobile Retention Thurs 2:35 – 3:12 | Expo Hall