TEXT

SAIF Ignition
Bangalore
20th Sep 2013

A/B testing for mobile apps
WHAT IS A/B TESTING?

Version ‘A’ vs Version ‘B’

That’s how the name A/B testing came about.
How does it work?
WEB A/B TESTING - Etsy

•

In-house A/B testing solution

•

Results are sometimes surprising!

INFINITE SCROLL

•

Infini...
WEB A/B TESTING
INFINITE SCROLL

•

What went wrong?

•

Changed multiple things at once

•

They should have broken-down ...
WEB A/B TESTING
SEARCH DROPDOWN

•

Broke-down the process into testing multiple short measurable tests
•

Decrease the nu...
WEB A/B TESTING
SEARCH DROPDOWN

•

Most people did not notice the major feature changes

•

Search dropdown was subsequen...
MOBILE A/B TESTING – Why is it needed?

• Increase conversion, revenue, engagement

metrics
• User-segmented and targeted ...
MOBILE A/B TESTING – Challenges

• Complex coding involved
• Delays in app store updates • slowing the pace of innovation
...
WHY A/B TEST MOBILE APPS?

•

Everyone A/B tests in real-life:
•

People try two or more pieces of clothing before buying
...
WHAT TO A/B TEST IN NATIVE MOBILE APPS?

If it is there in your app, it can be A/B tested!
Example use cases:
• Call to Ac...
CASE STUDY: PRICE COMPARISON APP

The app lets users compare prices of
multiple cross-category products from
online stores...
values

in-code test
(normal)

more

•Integrate SDK
•Pre-specify tests
•Pre-specify attribute

PRE SETUP TIME

less

COMPA...
COMPARING TEST TYPES (2/3)

•Integrate SDK
•Pre-specify tests
•Change attribute values

values

data driven test

in-code ...
COMPARING TEST TYPES (3/3)

•Integrate SDK
•Pre-specify tests
•Change attribute values

PRE SETUP TIME

less
values

wysiw...
NEW FEATURES

Live Tracking of User’s Interaction
with the App

Content Management System

Feature Launches and Segmented
...
www.appiterate.com

“If genius is, as Edison said, 1% inspiration and 99% perspiration, then perhaps
innovation is 1% inve...
Upcoming SlideShare
Loading in …5
×

Appiterate - SAIF Ignition

470 views

Published on

Appiterate held a workshop at SAIF Ignition (held at Bangalore on 20th Sep 2013) on A/B testing for mobile apps. It had a look at pre-existing practices in web A/B testing and also extensively covered mobile A/B testing.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
470
On SlideShare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Appiterate - SAIF Ignition

  1. 1. TEXT SAIF Ignition Bangalore 20th Sep 2013 A/B testing for mobile apps
  2. 2. WHAT IS A/B TESTING? Version ‘A’ vs Version ‘B’ That’s how the name A/B testing came about. How does it work?
  3. 3. WEB A/B TESTING - Etsy • In-house A/B testing solution • Results are sometimes surprising! INFINITE SCROLL • Infinite scroll is presumed to be a better experience as you see more items faster • Visitors clicked on • fewer results (-11%) • favourited less (-8%) • fewer items purchased (-22%) Source – Design for continuous experimentation, http://www.slideshare.net/danmckinley/design-for-continuous-experimentation Infinite Scroll
  4. 4. WEB A/B TESTING INFINITE SCROLL • What went wrong? • Changed multiple things at once • They should have broken-down the feature change into two hypothesis testing validations: • More items is better • Faster is better
  5. 5. WEB A/B TESTING SEARCH DROPDOWN • Broke-down the process into testing multiple short measurable tests • Decrease the number of items in dropdown • Default to ‘all items’ • Search bars for in-shop and in-page search • Rich auto-suggest • .. and many more such tests!
  6. 6. WEB A/B TESTING SEARCH DROPDOWN • Most people did not notice the major feature changes • Search dropdown was subsequently withdrawn • Iterative ‘Design – Develop – Measure’ Infinite Scroll Design Search Dropdown Design Develop Measure Develop Measure
  7. 7. MOBILE A/B TESTING – Why is it needed? • Increase conversion, revenue, engagement metrics • User-segmented and targeted app feature launches • Most mobile apps struggle to make money - faster iterations will genuinely help developers in monetizing their apps
  8. 8. MOBILE A/B TESTING – Challenges • Complex coding involved • Delays in app store updates • slowing the pace of innovation • Simplicity and ease of web A/B testing not prevalent on mobiles • thus devoid of 'iterative improvements‘ • key growth drivers on the web
  9. 9. WHY A/B TEST MOBILE APPS? • Everyone A/B tests in real-life: • People try two or more pieces of clothing before buying • You have a test drive of two shortlisted cars • The HiPPO (the Highest Paid Person’s Opinion) method of decision making • Decisions based on such opinions, hunches, guesses and conjectures • Use data in decision making • A/B testing on mobile apps is a great way to increase conversions and improving your key mobile metrics.
  10. 10. WHAT TO A/B TEST IN NATIVE MOBILE APPS? If it is there in your app, it can be A/B tested! Example use cases: • Call to Action buttons (these include buttons like ‘buy now’, ‘add to cart’, ‘sign up’) • Set any value to the text, size, colors, fonts or its position • App features – Test engagement metrics by having different number of free levels in games • User experience and app navigation sequence tests • Entire screen themes/layouts • Different page opens on tap of a button • Different ways of social media credentialbased sign-ins
  11. 11. CASE STUDY: PRICE COMPARISON APP The app lets users compare prices of multiple cross-category products from online stores. User engagement metric ‘review’ button •Price link – Increases element (arrow) beside the price Variation A review button with longer text No visual element (arrow) beside the price A review button with shorter text A small arrow beside the price DISTRIBUTION: 50% 50% Price link clicks on 18.1% of screen opens Price link clicks on 22.4% of screen opens Hypotheses: •Review button – Control DESCRIPTION: Aim was to increase conversion on the following: VARIATIONS: revenue in the form of referral fees. •A short or a long text on a •Impact of a small visual RESULTS: Longer text button got Shorter text button got clicked clicked 1.4% times of screen 1.1% times of screen opens opens Price link with a visual element resulted in an increase of 23.4% in revenues for the app! The longer text on the review button is better at driving engagement by 23.1%
  12. 12. values in-code test (normal) more •Integrate SDK •Pre-specify tests •Pre-specify attribute PRE SETUP TIME less COMPARING TEST TYPES (1/3) slow TIME FOR DEPLOYMENT & ITERATION fast
  13. 13. COMPARING TEST TYPES (2/3) •Integrate SDK •Pre-specify tests •Change attribute values values data driven test in-code test (normal) more •Integrate SDK •Pre-specify tests •Pre-specify attribute PRE SETUP TIME less live slow TIME FOR DEPLOYMENT & ITERATION fast
  14. 14. COMPARING TEST TYPES (3/3) •Integrate SDK •Pre-specify tests •Change attribute values PRE SETUP TIME less values wysiwyg data driven test in-code test (normal) more •Integrate SDK •Pre-specify tests •Pre-specify attribute •Bypass app store updates •Drag-n-drop visual interface •Fast and easy deployment live slow TIME FOR DEPLOYMENT & ITERATION fast
  15. 15. NEW FEATURES Live Tracking of User’s Interaction with the App Content Management System Feature Launches and Segmented Rollouts
  16. 16. www.appiterate.com “If genius is, as Edison said, 1% inspiration and 99% perspiration, then perhaps innovation is 1% invention and 99% iteration.”

×