SlideShare a Scribd company logo
1 of 74
Download to read offline
Feature Configuration
and A/B Experiments
in native and hybrid mobile apps

Hello
Feature Configuration
and A/B Experiments
in native and hybrid mobile apps
Breaking Development Nashville
October 21 2013
lacy@etsy.com
Mostly at Etsy we work on greenfield development, new products or features from
scratch. However as our mobile platform matures and our apps start to mature, we
are finding ourselves wanting to take the principals we find most fruitful in other,
older parts of Etsy and apply them in the mobile realm. TLDR this is about the fun
stuff.
When I talk to people about working at Etsy I expect they want to talk about
deploying web applications on tons of servers..
Or working with industry leaders like Rasmus Lerdorf..
But in fact typically people want to talk about the unique and sometimes strange
things you can find on Etsy.
Often little do these people know that interesting oddities can be purchased on their
phone or tablet.
One might wander what kind of business it is selling unique items over the internet,
but as you can see we do pretty well. In recent months we’ve seen as much as half of
our business happen on mobile devices. This makes mobile experimentation a really
exciting place to be.
Feature Flags

http://www.flickr.com/photos/mig/15964697/

The first of the concepts that we are trying to adapt from the old web world to mobile
is the notion of feature flags. We find wrapping things up as features is incredibly
useful. Another of these principals of engineering is something we call Continuous
Experimentation.
Continuous Experimentation

When we talk about Experimentation though we’re talking about A/B experiments. If
you’ve ever been using an app or a site and noticed that someone sitting RIGHT NEXT
to you has a different looking app or site, this is probably part of an experiment.
Continuous Experimentation
• make small changes
• stay honest
• don't break the [product]

The name we give it comes from what we call “Continuous Deployment”. Instead this
“Continuous Experimentation” describes how we try to continue to develop apps and
mature app features after they’ve been working out in the world and proven as largely
a good idea.
Real People
Brenda

https://www.etsy.com/shop/CattailsWoodwork

Why is it important to be honest and to not break the product? Because we’re talking
about the livelihood of real people.
Matthew

Real People

https://www.etsy.com/shop/PretentiousBeerGlass

People who make cool stuff and depend on Etsy to let them sell that cool stuff.
Continuous Experimentation
• make small changes
• stay honest
• don't break the [product]

We use this to develop our products iteratively using real world feedback and data
about our users and how their experience is.
Continuous Experimentation
• make small changes
• stay honest
• don't break the [product]

I love building things, making things. You can’t, for lack of a better analogy, you can’t
build an automobile or a vacuum cleaner or whatever thing, sell that thing, have
someone use that thing every day and then expect that you could change it, change
how the car works and look at how your changes effect the experience or the
performance of that thing from back at the factory.
http://www.flickr.com/photos/usfws_pacificsw/6391185103/

Steve Jobs had this quote about how the computer was a bicycle for the mind. I feel
like software today allows us to use computers as a bicycle for industry.
Disclaimer
• Analytics / Big Data
• Experimental Analysis
• Exploratory Analysis

I’m going to talk mostly about design and development that will keep us from making
bad data or picking poor audiences. There are other elements here but let’s assume
we have some way to gather analytics and some way to analyze the data we have
gathered.
Continuous Experimentation
• Everyone does experiments
• But not everything works this way
• Rarely seller tools, usually public stuff
• The develop / release / cheer cycle
• Mobile Apps

Everyone at Etsy dabbles in experiments, at least a little. But really not every line and
every product gets measured by experiment. Sometimes products are just made
because they’re needed. Like making a mobile website.
Launch Day

Story time, we have a big announcement to make one day at Etsy..
Introducing
the New Thing

We are introducing a new thing, a new feature or whatever..
The new product exists in a ton of places. These are all DIFFERENT teams mind you!
iPhone

Android
Web

Email

iPad
This is all to say that this New Thing exists in an ecosystem across all these different
mediums. This is kind of the mindset you have to be in with cross-platform features and
their a/b tests. A feature now becomes a dimension in and of itself. It is not just part of your
website. Features can transcend an individual app.
Launch Day

Fortunately in this story, on launch day, QA has already seen the new product in operation,
ahead of the world. The App Store review process is already done. We had a release go out
last week. The code that will show this New Thing lies “dormant” out in the wild, on the
website, in the web views in the app and even in the app binary itself.
if ($cfg[‘cool-dog-pics’]) { ...

Z

Z
Z
index.php

in the PHP world..

Z

Z

index2.php
if ($cfg[‘cool-dog-pics’]) { ...

Z

Z
Z

Z

index.php

Z

index2.php

When we flip the switch, the code can branch to the new code.
if (coolDogsIsEnabled()) { ...

Z

Z
Z
base.js

in the DOM..

Z

Z

base_new.js
if ([Config isEnabled@”CoolDogs”])
{ ...

Z

Z
Z

Z

Z

SomeViewController.h SomeNewViewController.h

and in native apps..
One Line!

So the launch starts on a engineer’s macbook. We push, for sake of illustration, as
little as one line of web code out to our web servers and API servers. This one line
mentions The New Thing by name, saying simply “turn on The New Thing” and within
the granularity of 20-30 minutes you have the Newness throughout the product to
millions of users, across the ecosystem of all types, across the world.
Launch Day

So this story is really just like a parlor trick. It’s to get your attention. It’s fun to talk
about. The real value here is the illustration of this unique dimension, these
“features” which we can use to divide up our product that is spanning out in so many
different mediums.
Measure Everything
• Feature Configuration
• Benefits of Features
• Configuration as Tests
• How Flags Get to the Code
• Examples of Experiments
• Making Sensible Tests

The roadmap to making our mobile a/b experiments a reality..
Feature Configuration

So first some of the things we take for granted. Four years ago -four years is a really
long time- there was a blog article from Flickr I saw floating around on the internet. It
was about what they called feature flags. We all thought it was pretty cool stuff. I had
no idea why. Then at some point I started working at Etsy and now I have to preface
any technical discussion of interest with how feature configuration shapes the way we
make software.
Flags, Flippers, Flickr
http://code.flickr.net/2009/12/02/flipping-out/
$cfg = array();
$cfg[‘cool-dog-pics’];
$cfg[‘cool-dog-pics’] =
array(‘enabled’=>‘off’);
$cfg[‘cool-dog-pics’] =
array(‘enabled’=>‘on’);

This idea starts with a configuration array that names things on your site. Maybe cool
dog pics.
Flags, Flippers, Flickr
if ($cfg[‘cool-dog-pics’) {
echo Dogs::getCoolPics();
}

You basically push out all the code all the time. Even if it doesn’t work, you just keep
it turned off.
Feature Configuration

The original benefit here, the exciting thing about this from the Flickr article is that
you don’t have to merge code or do gigantic atomic (and painful) releases. We push
code 30, 40, 50 times a day. We push code that doesn’t even work, on purpose,
because pushing tons of little pieces of resilient code is a lot more predictable than
trying to do a big release.
More Cool Stuff
$cfg = array(
‘enabled’=>array(‘users’=>‘lacyrhoades’)
);
$cfg = array(
‘enabled’=>array(‘admin’=>true)
);
$cfg = array(
‘enabled’=>array(‘groups’=>54321)
);

You could maybe do something like only enable that branch of code to be active for
one person, just some people you work with or a group of users you know have a
particular interest in the feature.
Measuring Things

Another principal is measuring things in a real-time way
Like here we’ve got what we call dashboards, so as we flip these switches on or off we
can look for anomalies and make sure nothing is blowing up.
Benefits of Features

Another trick we have is what we call “Slow Rampups”. Things don’t have to be just
“on” or “off”.
Benefits of Features

Launches can now happen on a schedule. There’s no sense in trying to make a
software deadline be punctuated by releasing that software. The software
development is going to take too long. You are going to need to be able to QA it,
change it and make it right.
Requisites
• No backwards incompatible changes
• Uniquely identified users

There are some limitations or requirements.. you’ll need to make sure the branches
in the code are not drastically different. The data schema for example has to be
backwards compatible. You’re going to need a way to uniquely identify users in order
to put them into groups and give them consistent experiences.
Features in the
New World

Feature flags were made for websites. But your website is not a website anymore. It
hasn’t been a website for a while now. Your website probably employs at least one
mobile developer who doesn’t even look at your “website” every day. Similarly feature
flags have to adapt to this new world.
Features as Tests

A/B Tests put people in buckets, giving them different values for one flag. Watch
what they do over time, this means attaching information about their test buckets to
analytics events. Make your app better based on the data you get back.
An Example
Feature

On the main screen. The individual facets you see there are powered by different parts
of the infrastructure.
The personal activity feed is fed from an map-reduce stack in PHP. The curated
panels are fed from a Java-based search stack. One of these might mess up and you
want the app to go on without it. This is a great place to start for dividing up features
in your design.
An Example
Prototype

For a prototype we might want to gather feedback from users, a select group, for
example the photo-editor in our app started as a prototype group and grew with
feedback we received.
This was a photo editing interface in the Etsy app. We wanted to know some
particularly qualitative things about it, like was it easy to use and understand. It’s not
exactly like we can study really dry analytics to get at this sort of answer. So we
invited some interested sellers to join a group on Etsy. Those sellers could see the
new tools and we could at that point gather feedback from them.
This was a redesign of the activity feed on our mobile web. We used a feedback group
to preview the features and make sure we got it right.
An Example
Experiment
One of our recent experiments.. mobile templates vs. desktop templates. It turns out
from the measurements we could make that users were quantifiably more satisfied
with the desktop “look” of Etsy on tablet devices.
We wanted to really go down the road of experiments as the web side of Etsy has
before us. We took this listing view as a place to start. Here’s a reasonably priced
shadow puppet. Here it is (right) with the experiment enabled.
Here was the flow before. The last three steps were all web views on the server side.
Here’s the flow afterwards. The idea was reducing the number of steps would
significantly reduce friction in the checkout flow.
Experiment Steps

• Set up a feature flag

First we needed to look at eligibility, or who can take part in this experiment. If the
eligible audience is VERY small compared the general audience of all users, our total
number of "people who have used this feature" is going to be small, and so then the
numerator of "people who bought an item with this feature" will also be small. Then
there’s the feature flag. We make this and start to write code against it, so we’re
pushing code from day 1.
Experiment Steps
$config['mobile']['iphone']
['BuyItNow'] = array(
'enabled' => 0,
'group' => 54321,
'admin' => 0
)
];

The config might look something like this. This goes into the one config file that’s at
the center of everything.
Experiment Steps

• Set up a feature flag
• Determine eligibility programmatically

Next we need to determine eligibility programatically. We’ve already determined our
eligible audience is of considerable size at this point. We did that as part of
exploratory analysis. This is more about being able to say in the code, something
like..
Eligibility
$eligible = isEligible($user,
$listing);
if ($eligible) { ... }

We want to answer questions like, do we have this user’s billing information on file?
Can this listing even be bought using that credit card? Can the seller who’s selling
this item ship the item to the country the user lives in? If the answers to any of these
questions are not good, we need to not be including this buyer in the experiment.
We’ll dilute our results quickly, since we know there are a bunch of combinations of
items and buyers who can’t take part in the experiment.
Experiment Steps

• Set up a feature flag
• Determine eligibility
• Start hacking away

At this point we start coding up the native elements and the web elements, all the
while hiding them behind the feature configuration we chose before. Generally being
careful if we need to add code to shared libraries or shared files.
Experiment Steps
• Set up a feature flag
• Determine eligibility
• Start hacking away
• Begin testing

When things are working the way we expect, we begin testing with a small internal
group, usually people in QA or just staff members. Also we begin to QA the app as a
whole for release.
Experiment Steps
• Set up a feature flag
• Determine eligibility
• Start hacking away
• Begin testing
• Put the product on the shelf

Once QA approves the way it works, most of the coding is done and we put the
feature on the shelf for a while. We’ll probably try to begin the app store review
process as soon as possible.
Eligibility

[EtsyConfig isEnabled:@”BuyItNow”];

The code in objective C will look something like this. The EtsyConfig class here is
going to be responsible for remembering “yes I did see this experiment, someone
asked about it, and the answer was: x” That answer, and the specific question we
came looking for need to be attached to analytics events the user is firing.
Experiment Steps

• Experiment group
• Up to a certain percentage
• Analytics events

When the app is live, we can implement an experiment group. We can ramp up slowly
so that we can kick the tires and know things are okay. When things are looking good
we’ll take the experiment up to a percentage we established beforehand. Analytics
events are capturing the state of this test as people see it. Typically you can ignore
the state of this feature on analytics events for people who are completely ineligible.
Looking at results

• Self Selection
• Refunds / Returns
• Visit-level vs. User-level
Our initial results were actually pretty good. There are other things to consider,
drawbacks and perhaps biased design of the experiment.
How the Config
Gets into the Device
plist

The code starts as the plist you probably recognize from any Xcode project.
+
server

We download a set of config values, things that are enabled or disabled, from the
server.
+
plist

=
server

At runtime we merge these values into a single dictionary.

runtime
Configuration Steps
• App launch
• Periodically later, login
• Merge downloaded config
• Post notification

The config is downloaded and merged whenever the app launches. It also is
downloaded when the user logs in or out, as their experiments might change. As a
final step in this merging we’ll post a notification in the app code so that UI elements
in the app which need to update based on any experimental code, can do so.
Bucketing Users

You’re going to need to know who individual users are if you can put 20% into one
bucket and ensure that they stay there, and ensure that no one from the 80% control
group, makes it in to that experiment bucket.
Bucketing Users

• Persistent Cookies
• Device UDID
• user_id (where available)

The best thing to use here would be the user_id. This is not always available, like if
the user is logged out. Typically websites use persistent cookies to bucket logged out
users. Unfortunately an API for mobile apps doesn't have this avenue of cookies.
You've got to have some way to bucket users. One approach is to make up a sort of
UDID. Something that is specific to a device, and is stored locally with the app. This is
truly only unique to each install of the app, but it seems to work fairly well. You need
to pass that UDID to any webviews so that those webviews can identify themselves as
part of those app sessions.
Sensible Testing

We mentioned before that the number of users, the percentage, was key to obtaining
significant results. If you don’t run the experiment long enough, you’re going to not
prove anything. If you take too long measuring something, you’re wasting time.
There are deadlines to meet for this or other things, etc. For our test results we follow
the equation used here:
ExperimentCalculator.com
via @mcfunley

• How many eligible visits per day?
• What percentage of visits will see the change?
• What is your current conversion rate?
• How will you change conversion?
• How confident do you want to be?
• How likely should you be to detect the change?

experimentcalculator.com This is something one of our engineers made from a paper
on statistics. I don’t pretend to understand exactly but the idea is it gets you to not
choose random numbers for the length of the experiment. You have to weigh the cost
of developing a new feature against the feature’s potential value.
Sensible Testing

Some shortcomings of these approaches include: lots more code. More permutations
of QA. For every test variant you add, you essentially add the need for another QA
user story. You risk introducing an unpredictable user experience. Changing minor
interactions are probably okay, changing the main navigation scheme in your
application probably isn’t. Analysis paralysis - At some point you’ve just got to make
decisions. A/B testing only goes so far in helping you choose direction for your
product. You can’t be creative in your product decisions by just piling A/B tests back
to back.
Things to
Watch Out For

Make sure default is "off" for predictable stable experiences and sanity of future
support. You don’t want to have a bunch of old flags you need to keep around for
un-updated apps. If you've got a good app and a good idea going, you've probably
not going to discover a breakaway victory by running an a/b test. They're too subtle
and often times only prove that your intuition about your users is heavily biased.
The Future

So get out there, do some experiments. Time is precious. If you're going to make
heads or tails of the numbers you see in an experiment, you'll probably need all the
time you can get.
Feature Configuration
and A/B Experiments
in native and hybrid mobile apps
Breaking Development Nashville
October 21 2013
lacy@etsy.com

More Related Content

Viewers also liked

описание пед опыта Махнач В.В.
описание пед опыта Махнач В.В.описание пед опыта Махнач В.В.
описание пед опыта Махнач В.В.Елена Исакова
 
μικρασιατική εκστρατεία
μικρασιατική εκστρατείαμικρασιατική εκστρατεία
μικρασιατική εκστρατείαTRPAGONI
 
Презентация опыта Махнач В.В.
Презентация опыта Махнач В.В.Презентация опыта Махнач В.В.
Презентация опыта Махнач В.В.Елена Исакова
 
Napojení Joomly! na rezervační systém Colosseum
Napojení Joomly! na rezervační systém ColosseumNapojení Joomly! na rezervační systém Colosseum
Napojení Joomly! na rezervační systém ColosseumKačka Kubová
 

Viewers also liked (8)

описание пед опыта Махнач В.В.
описание пед опыта Махнач В.В.описание пед опыта Махнач В.В.
описание пед опыта Махнач В.В.
 
программа леонович с
программа леонович спрограмма леонович с
программа леонович с
 
murang bahay for sale, house and lot for sale, ready for occupancy houses for...
murang bahay for sale, house and lot for sale, ready for occupancy houses for...murang bahay for sale, house and lot for sale, ready for occupancy houses for...
murang bahay for sale, house and lot for sale, ready for occupancy houses for...
 
μικρασιατική εκστρατεία
μικρασιατική εκστρατείαμικρασιατική εκστρατεία
μικρασιατική εκστρατεία
 
Урок 12
Урок 12Урок 12
Урок 12
 
статьи. выступления.
статьи. выступления.статьи. выступления.
статьи. выступления.
 
Презентация опыта Махнач В.В.
Презентация опыта Махнач В.В.Презентация опыта Махнач В.В.
Презентация опыта Махнач В.В.
 
Napojení Joomly! na rezervační systém Colosseum
Napojení Joomly! na rezervační systém ColosseumNapojení Joomly! na rezervační systém Colosseum
Napojení Joomly! na rezervační systém Colosseum
 

Similar to Mobile App Feature Configuration and A/B Experiments

The Open Commerce Conference - Premature Optimisation: The Root of All Evil
The Open Commerce Conference - Premature Optimisation: The Root of All EvilThe Open Commerce Conference - Premature Optimisation: The Root of All Evil
The Open Commerce Conference - Premature Optimisation: The Root of All EvilFabio Akita
 
Continuous (Production) Integration: Ruby on Rails Application Monitoring wit...
Continuous (Production) Integration: Ruby on Rails Application Monitoring wit...Continuous (Production) Integration: Ruby on Rails Application Monitoring wit...
Continuous (Production) Integration: Ruby on Rails Application Monitoring wit...jnewland
 
Continuous validation (incl speaker notes)
Continuous validation (incl speaker notes)Continuous validation (incl speaker notes)
Continuous validation (incl speaker notes)Niels Talens
 
Professional web development with libraries
Professional web development with librariesProfessional web development with libraries
Professional web development with librariesChristian Heilmann
 
The State of Front-end At CrowdTwist
The State of Front-end At CrowdTwistThe State of Front-end At CrowdTwist
The State of Front-end At CrowdTwistMark Fayngersh
 
Connecting with the enterprise - The how and why of connecting to Enterprise ...
Connecting with the enterprise - The how and why of connecting to Enterprise ...Connecting with the enterprise - The how and why of connecting to Enterprise ...
Connecting with the enterprise - The how and why of connecting to Enterprise ...Kevin Poorman
 
DevDay 2013 - Building Startups and Minimum Viable Products
DevDay 2013 - Building Startups and Minimum Viable ProductsDevDay 2013 - Building Startups and Minimum Viable Products
DevDay 2013 - Building Startups and Minimum Viable ProductsBen Hall
 
Automating Mobile Testing at Gilt with Appium
Automating Mobile Testing at Gilt with AppiumAutomating Mobile Testing at Gilt with Appium
Automating Mobile Testing at Gilt with AppiumSauce Labs
 
Introduction to GraphQL (or How I Learned to Stop Worrying about REST APIs)
Introduction to GraphQL (or How I Learned to Stop Worrying about REST APIs)Introduction to GraphQL (or How I Learned to Stop Worrying about REST APIs)
Introduction to GraphQL (or How I Learned to Stop Worrying about REST APIs)Hafiz Ismail
 
Stapling and patching the web of now - ForwardJS3, San Francisco
Stapling and patching the web of now - ForwardJS3, San FranciscoStapling and patching the web of now - ForwardJS3, San Francisco
Stapling and patching the web of now - ForwardJS3, San FranciscoChristian Heilmann
 
Microsoft Webday 2008 - Silverlight Experiences
Microsoft Webday 2008 - Silverlight ExperiencesMicrosoft Webday 2008 - Silverlight Experiences
Microsoft Webday 2008 - Silverlight ExperiencesNicklas Andersson
 
Resisting The Feature Creature
Resisting The Feature CreatureResisting The Feature Creature
Resisting The Feature CreatureChristian Heilmann
 
Grails Worst Practices
Grails Worst PracticesGrails Worst Practices
Grails Worst PracticesBurt Beckwith
 
Qalcwise Introduction to Designer v20160215
Qalcwise Introduction to Designer v20160215Qalcwise Introduction to Designer v20160215
Qalcwise Introduction to Designer v20160215qalcwise
 
Infrastructure is development
Infrastructure is developmentInfrastructure is development
Infrastructure is developmentstahnma
 
Getting Started with Test Automation: Introduction to Cucumber with Lapis Lazuli
Getting Started with Test Automation: Introduction to Cucumber with Lapis LazuliGetting Started with Test Automation: Introduction to Cucumber with Lapis Lazuli
Getting Started with Test Automation: Introduction to Cucumber with Lapis LazuliRebecca Eloise Hogg
 
Developing for LinkedIn's Application Platform
Developing for LinkedIn's Application PlatformDeveloping for LinkedIn's Application Platform
Developing for LinkedIn's Application PlatformTaylor Singletary
 
AtlasCamp 2013: ADG / Lean UX
AtlasCamp 2013: ADG / Lean UXAtlasCamp 2013: ADG / Lean UX
AtlasCamp 2013: ADG / Lean UXcolleenfry
 
User Story Mapping in Practice
User Story Mapping in PracticeUser Story Mapping in Practice
User Story Mapping in PracticeSteve Rogalsky
 

Similar to Mobile App Feature Configuration and A/B Experiments (20)

The Open Commerce Conference - Premature Optimisation: The Root of All Evil
The Open Commerce Conference - Premature Optimisation: The Root of All EvilThe Open Commerce Conference - Premature Optimisation: The Root of All Evil
The Open Commerce Conference - Premature Optimisation: The Root of All Evil
 
Continuous (Production) Integration: Ruby on Rails Application Monitoring wit...
Continuous (Production) Integration: Ruby on Rails Application Monitoring wit...Continuous (Production) Integration: Ruby on Rails Application Monitoring wit...
Continuous (Production) Integration: Ruby on Rails Application Monitoring wit...
 
Continuous validation (incl speaker notes)
Continuous validation (incl speaker notes)Continuous validation (incl speaker notes)
Continuous validation (incl speaker notes)
 
Professional web development with libraries
Professional web development with librariesProfessional web development with libraries
Professional web development with libraries
 
The State of Front-end At CrowdTwist
The State of Front-end At CrowdTwistThe State of Front-end At CrowdTwist
The State of Front-end At CrowdTwist
 
Connecting with the enterprise - The how and why of connecting to Enterprise ...
Connecting with the enterprise - The how and why of connecting to Enterprise ...Connecting with the enterprise - The how and why of connecting to Enterprise ...
Connecting with the enterprise - The how and why of connecting to Enterprise ...
 
DevDay 2013 - Building Startups and Minimum Viable Products
DevDay 2013 - Building Startups and Minimum Viable ProductsDevDay 2013 - Building Startups and Minimum Viable Products
DevDay 2013 - Building Startups and Minimum Viable Products
 
Automating Mobile Testing at Gilt with Appium
Automating Mobile Testing at Gilt with AppiumAutomating Mobile Testing at Gilt with Appium
Automating Mobile Testing at Gilt with Appium
 
Introduction to GraphQL (or How I Learned to Stop Worrying about REST APIs)
Introduction to GraphQL (or How I Learned to Stop Worrying about REST APIs)Introduction to GraphQL (or How I Learned to Stop Worrying about REST APIs)
Introduction to GraphQL (or How I Learned to Stop Worrying about REST APIs)
 
Stapling and patching the web of now - ForwardJS3, San Francisco
Stapling and patching the web of now - ForwardJS3, San FranciscoStapling and patching the web of now - ForwardJS3, San Francisco
Stapling and patching the web of now - ForwardJS3, San Francisco
 
12 top app testing tools
12 top app testing tools12 top app testing tools
12 top app testing tools
 
Microsoft Webday 2008 - Silverlight Experiences
Microsoft Webday 2008 - Silverlight ExperiencesMicrosoft Webday 2008 - Silverlight Experiences
Microsoft Webday 2008 - Silverlight Experiences
 
Resisting The Feature Creature
Resisting The Feature CreatureResisting The Feature Creature
Resisting The Feature Creature
 
Grails Worst Practices
Grails Worst PracticesGrails Worst Practices
Grails Worst Practices
 
Qalcwise Introduction to Designer v20160215
Qalcwise Introduction to Designer v20160215Qalcwise Introduction to Designer v20160215
Qalcwise Introduction to Designer v20160215
 
Infrastructure is development
Infrastructure is developmentInfrastructure is development
Infrastructure is development
 
Getting Started with Test Automation: Introduction to Cucumber with Lapis Lazuli
Getting Started with Test Automation: Introduction to Cucumber with Lapis LazuliGetting Started with Test Automation: Introduction to Cucumber with Lapis Lazuli
Getting Started with Test Automation: Introduction to Cucumber with Lapis Lazuli
 
Developing for LinkedIn's Application Platform
Developing for LinkedIn's Application PlatformDeveloping for LinkedIn's Application Platform
Developing for LinkedIn's Application Platform
 
AtlasCamp 2013: ADG / Lean UX
AtlasCamp 2013: ADG / Lean UXAtlasCamp 2013: ADG / Lean UX
AtlasCamp 2013: ADG / Lean UX
 
User Story Mapping in Practice
User Story Mapping in PracticeUser Story Mapping in Practice
User Story Mapping in Practice
 

Recently uploaded

Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda
 
QMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdfQMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdfROWELL MARQUINA
 
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...Karmanjay Verma
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
Infrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsInfrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsYoss Cohen
 
Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!Memoori
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Kaya Weers
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observabilityitnewsafrica
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxAna-Maria Mihalceanu
 
Accelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessAccelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessWSO2
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkPixlogix Infotech
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPathCommunity
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfNeo4j
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfpanagenda
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI AgeCprime
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA
 
Digital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentDigital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentMahmoud Rabie
 
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxGenerative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxfnnc6jmgwh
 
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Mark Simos
 

Recently uploaded (20)

Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
 
QMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdfQMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdf
 
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
Infrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsInfrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platforms
 
Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance Toolbox
 
Accelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessAccelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with Platformless
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App Framework
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to Hero
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdf
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI Age
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 
Digital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentDigital Tools & AI in Career Development
Digital Tools & AI in Career Development
 
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxGenerative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
 
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
 

Mobile App Feature Configuration and A/B Experiments

  • 1. Feature Configuration and A/B Experiments in native and hybrid mobile apps Hello
  • 2. Feature Configuration and A/B Experiments in native and hybrid mobile apps Breaking Development Nashville October 21 2013 lacy@etsy.com Mostly at Etsy we work on greenfield development, new products or features from scratch. However as our mobile platform matures and our apps start to mature, we are finding ourselves wanting to take the principals we find most fruitful in other, older parts of Etsy and apply them in the mobile realm. TLDR this is about the fun stuff.
  • 3. When I talk to people about working at Etsy I expect they want to talk about deploying web applications on tons of servers..
  • 4. Or working with industry leaders like Rasmus Lerdorf..
  • 5. But in fact typically people want to talk about the unique and sometimes strange things you can find on Etsy.
  • 6. Often little do these people know that interesting oddities can be purchased on their phone or tablet.
  • 7. One might wander what kind of business it is selling unique items over the internet, but as you can see we do pretty well. In recent months we’ve seen as much as half of our business happen on mobile devices. This makes mobile experimentation a really exciting place to be.
  • 8. Feature Flags http://www.flickr.com/photos/mig/15964697/ The first of the concepts that we are trying to adapt from the old web world to mobile is the notion of feature flags. We find wrapping things up as features is incredibly useful. Another of these principals of engineering is something we call Continuous Experimentation.
  • 9. Continuous Experimentation When we talk about Experimentation though we’re talking about A/B experiments. If you’ve ever been using an app or a site and noticed that someone sitting RIGHT NEXT to you has a different looking app or site, this is probably part of an experiment.
  • 10. Continuous Experimentation • make small changes • stay honest • don't break the [product] The name we give it comes from what we call “Continuous Deployment”. Instead this “Continuous Experimentation” describes how we try to continue to develop apps and mature app features after they’ve been working out in the world and proven as largely a good idea.
  • 11. Real People Brenda https://www.etsy.com/shop/CattailsWoodwork Why is it important to be honest and to not break the product? Because we’re talking about the livelihood of real people.
  • 12. Matthew Real People https://www.etsy.com/shop/PretentiousBeerGlass People who make cool stuff and depend on Etsy to let them sell that cool stuff.
  • 13. Continuous Experimentation • make small changes • stay honest • don't break the [product] We use this to develop our products iteratively using real world feedback and data about our users and how their experience is.
  • 14. Continuous Experimentation • make small changes • stay honest • don't break the [product] I love building things, making things. You can’t, for lack of a better analogy, you can’t build an automobile or a vacuum cleaner or whatever thing, sell that thing, have someone use that thing every day and then expect that you could change it, change how the car works and look at how your changes effect the experience or the performance of that thing from back at the factory.
  • 15. http://www.flickr.com/photos/usfws_pacificsw/6391185103/ Steve Jobs had this quote about how the computer was a bicycle for the mind. I feel like software today allows us to use computers as a bicycle for industry.
  • 16. Disclaimer • Analytics / Big Data • Experimental Analysis • Exploratory Analysis I’m going to talk mostly about design and development that will keep us from making bad data or picking poor audiences. There are other elements here but let’s assume we have some way to gather analytics and some way to analyze the data we have gathered.
  • 17. Continuous Experimentation • Everyone does experiments • But not everything works this way • Rarely seller tools, usually public stuff • The develop / release / cheer cycle • Mobile Apps Everyone at Etsy dabbles in experiments, at least a little. But really not every line and every product gets measured by experiment. Sometimes products are just made because they’re needed. Like making a mobile website.
  • 18. Launch Day Story time, we have a big announcement to make one day at Etsy..
  • 19. Introducing the New Thing We are introducing a new thing, a new feature or whatever..
  • 20. The new product exists in a ton of places. These are all DIFFERENT teams mind you!
  • 21. iPhone Android Web Email iPad This is all to say that this New Thing exists in an ecosystem across all these different mediums. This is kind of the mindset you have to be in with cross-platform features and their a/b tests. A feature now becomes a dimension in and of itself. It is not just part of your website. Features can transcend an individual app.
  • 22. Launch Day Fortunately in this story, on launch day, QA has already seen the new product in operation, ahead of the world. The App Store review process is already done. We had a release go out last week. The code that will show this New Thing lies “dormant” out in the wild, on the website, in the web views in the app and even in the app binary itself.
  • 23. if ($cfg[‘cool-dog-pics’]) { ... Z Z Z index.php in the PHP world.. Z Z index2.php
  • 24. if ($cfg[‘cool-dog-pics’]) { ... Z Z Z Z index.php Z index2.php When we flip the switch, the code can branch to the new code.
  • 25. if (coolDogsIsEnabled()) { ... Z Z Z base.js in the DOM.. Z Z base_new.js
  • 26. if ([Config isEnabled@”CoolDogs”]) { ... Z Z Z Z Z SomeViewController.h SomeNewViewController.h and in native apps..
  • 27. One Line! So the launch starts on a engineer’s macbook. We push, for sake of illustration, as little as one line of web code out to our web servers and API servers. This one line mentions The New Thing by name, saying simply “turn on The New Thing” and within the granularity of 20-30 minutes you have the Newness throughout the product to millions of users, across the ecosystem of all types, across the world.
  • 28. Launch Day So this story is really just like a parlor trick. It’s to get your attention. It’s fun to talk about. The real value here is the illustration of this unique dimension, these “features” which we can use to divide up our product that is spanning out in so many different mediums.
  • 29. Measure Everything • Feature Configuration • Benefits of Features • Configuration as Tests • How Flags Get to the Code • Examples of Experiments • Making Sensible Tests The roadmap to making our mobile a/b experiments a reality..
  • 30. Feature Configuration So first some of the things we take for granted. Four years ago -four years is a really long time- there was a blog article from Flickr I saw floating around on the internet. It was about what they called feature flags. We all thought it was pretty cool stuff. I had no idea why. Then at some point I started working at Etsy and now I have to preface any technical discussion of interest with how feature configuration shapes the way we make software.
  • 31. Flags, Flippers, Flickr http://code.flickr.net/2009/12/02/flipping-out/ $cfg = array(); $cfg[‘cool-dog-pics’]; $cfg[‘cool-dog-pics’] = array(‘enabled’=>‘off’); $cfg[‘cool-dog-pics’] = array(‘enabled’=>‘on’); This idea starts with a configuration array that names things on your site. Maybe cool dog pics.
  • 32. Flags, Flippers, Flickr if ($cfg[‘cool-dog-pics’) { echo Dogs::getCoolPics(); } You basically push out all the code all the time. Even if it doesn’t work, you just keep it turned off.
  • 33. Feature Configuration The original benefit here, the exciting thing about this from the Flickr article is that you don’t have to merge code or do gigantic atomic (and painful) releases. We push code 30, 40, 50 times a day. We push code that doesn’t even work, on purpose, because pushing tons of little pieces of resilient code is a lot more predictable than trying to do a big release.
  • 34. More Cool Stuff $cfg = array( ‘enabled’=>array(‘users’=>‘lacyrhoades’) ); $cfg = array( ‘enabled’=>array(‘admin’=>true) ); $cfg = array( ‘enabled’=>array(‘groups’=>54321) ); You could maybe do something like only enable that branch of code to be active for one person, just some people you work with or a group of users you know have a particular interest in the feature.
  • 35. Measuring Things Another principal is measuring things in a real-time way
  • 36. Like here we’ve got what we call dashboards, so as we flip these switches on or off we can look for anomalies and make sure nothing is blowing up.
  • 37. Benefits of Features Another trick we have is what we call “Slow Rampups”. Things don’t have to be just “on” or “off”.
  • 38. Benefits of Features Launches can now happen on a schedule. There’s no sense in trying to make a software deadline be punctuated by releasing that software. The software development is going to take too long. You are going to need to be able to QA it, change it and make it right.
  • 39. Requisites • No backwards incompatible changes • Uniquely identified users There are some limitations or requirements.. you’ll need to make sure the branches in the code are not drastically different. The data schema for example has to be backwards compatible. You’re going to need a way to uniquely identify users in order to put them into groups and give them consistent experiences.
  • 40. Features in the New World Feature flags were made for websites. But your website is not a website anymore. It hasn’t been a website for a while now. Your website probably employs at least one mobile developer who doesn’t even look at your “website” every day. Similarly feature flags have to adapt to this new world.
  • 41. Features as Tests A/B Tests put people in buckets, giving them different values for one flag. Watch what they do over time, this means attaching information about their test buckets to analytics events. Make your app better based on the data you get back.
  • 42. An Example Feature On the main screen. The individual facets you see there are powered by different parts of the infrastructure.
  • 43. The personal activity feed is fed from an map-reduce stack in PHP. The curated panels are fed from a Java-based search stack. One of these might mess up and you want the app to go on without it. This is a great place to start for dividing up features in your design.
  • 44. An Example Prototype For a prototype we might want to gather feedback from users, a select group, for example the photo-editor in our app started as a prototype group and grew with feedback we received.
  • 45. This was a photo editing interface in the Etsy app. We wanted to know some particularly qualitative things about it, like was it easy to use and understand. It’s not exactly like we can study really dry analytics to get at this sort of answer. So we invited some interested sellers to join a group on Etsy. Those sellers could see the new tools and we could at that point gather feedback from them.
  • 46. This was a redesign of the activity feed on our mobile web. We used a feedback group to preview the features and make sure we got it right.
  • 48. One of our recent experiments.. mobile templates vs. desktop templates. It turns out from the measurements we could make that users were quantifiably more satisfied with the desktop “look” of Etsy on tablet devices.
  • 49. We wanted to really go down the road of experiments as the web side of Etsy has before us. We took this listing view as a place to start. Here’s a reasonably priced shadow puppet. Here it is (right) with the experiment enabled.
  • 50. Here was the flow before. The last three steps were all web views on the server side.
  • 51. Here’s the flow afterwards. The idea was reducing the number of steps would significantly reduce friction in the checkout flow.
  • 52. Experiment Steps • Set up a feature flag First we needed to look at eligibility, or who can take part in this experiment. If the eligible audience is VERY small compared the general audience of all users, our total number of "people who have used this feature" is going to be small, and so then the numerator of "people who bought an item with this feature" will also be small. Then there’s the feature flag. We make this and start to write code against it, so we’re pushing code from day 1.
  • 53. Experiment Steps $config['mobile']['iphone'] ['BuyItNow'] = array( 'enabled' => 0, 'group' => 54321, 'admin' => 0 ) ]; The config might look something like this. This goes into the one config file that’s at the center of everything.
  • 54. Experiment Steps • Set up a feature flag • Determine eligibility programmatically Next we need to determine eligibility programatically. We’ve already determined our eligible audience is of considerable size at this point. We did that as part of exploratory analysis. This is more about being able to say in the code, something like..
  • 55. Eligibility $eligible = isEligible($user, $listing); if ($eligible) { ... } We want to answer questions like, do we have this user’s billing information on file? Can this listing even be bought using that credit card? Can the seller who’s selling this item ship the item to the country the user lives in? If the answers to any of these questions are not good, we need to not be including this buyer in the experiment. We’ll dilute our results quickly, since we know there are a bunch of combinations of items and buyers who can’t take part in the experiment.
  • 56. Experiment Steps • Set up a feature flag • Determine eligibility • Start hacking away At this point we start coding up the native elements and the web elements, all the while hiding them behind the feature configuration we chose before. Generally being careful if we need to add code to shared libraries or shared files.
  • 57. Experiment Steps • Set up a feature flag • Determine eligibility • Start hacking away • Begin testing When things are working the way we expect, we begin testing with a small internal group, usually people in QA or just staff members. Also we begin to QA the app as a whole for release.
  • 58. Experiment Steps • Set up a feature flag • Determine eligibility • Start hacking away • Begin testing • Put the product on the shelf Once QA approves the way it works, most of the coding is done and we put the feature on the shelf for a while. We’ll probably try to begin the app store review process as soon as possible.
  • 59. Eligibility [EtsyConfig isEnabled:@”BuyItNow”]; The code in objective C will look something like this. The EtsyConfig class here is going to be responsible for remembering “yes I did see this experiment, someone asked about it, and the answer was: x” That answer, and the specific question we came looking for need to be attached to analytics events the user is firing.
  • 60. Experiment Steps • Experiment group • Up to a certain percentage • Analytics events When the app is live, we can implement an experiment group. We can ramp up slowly so that we can kick the tires and know things are okay. When things are looking good we’ll take the experiment up to a percentage we established beforehand. Analytics events are capturing the state of this test as people see it. Typically you can ignore the state of this feature on analytics events for people who are completely ineligible.
  • 61. Looking at results • Self Selection • Refunds / Returns • Visit-level vs. User-level Our initial results were actually pretty good. There are other things to consider, drawbacks and perhaps biased design of the experiment.
  • 62. How the Config Gets into the Device
  • 63. plist The code starts as the plist you probably recognize from any Xcode project.
  • 64. + server We download a set of config values, things that are enabled or disabled, from the server.
  • 65. + plist = server At runtime we merge these values into a single dictionary. runtime
  • 66. Configuration Steps • App launch • Periodically later, login • Merge downloaded config • Post notification The config is downloaded and merged whenever the app launches. It also is downloaded when the user logs in or out, as their experiments might change. As a final step in this merging we’ll post a notification in the app code so that UI elements in the app which need to update based on any experimental code, can do so.
  • 67. Bucketing Users You’re going to need to know who individual users are if you can put 20% into one bucket and ensure that they stay there, and ensure that no one from the 80% control group, makes it in to that experiment bucket.
  • 68. Bucketing Users • Persistent Cookies • Device UDID • user_id (where available) The best thing to use here would be the user_id. This is not always available, like if the user is logged out. Typically websites use persistent cookies to bucket logged out users. Unfortunately an API for mobile apps doesn't have this avenue of cookies. You've got to have some way to bucket users. One approach is to make up a sort of UDID. Something that is specific to a device, and is stored locally with the app. This is truly only unique to each install of the app, but it seems to work fairly well. You need to pass that UDID to any webviews so that those webviews can identify themselves as part of those app sessions.
  • 69. Sensible Testing We mentioned before that the number of users, the percentage, was key to obtaining significant results. If you don’t run the experiment long enough, you’re going to not prove anything. If you take too long measuring something, you’re wasting time. There are deadlines to meet for this or other things, etc. For our test results we follow the equation used here:
  • 70. ExperimentCalculator.com via @mcfunley • How many eligible visits per day? • What percentage of visits will see the change? • What is your current conversion rate? • How will you change conversion? • How confident do you want to be? • How likely should you be to detect the change? experimentcalculator.com This is something one of our engineers made from a paper on statistics. I don’t pretend to understand exactly but the idea is it gets you to not choose random numbers for the length of the experiment. You have to weigh the cost of developing a new feature against the feature’s potential value.
  • 71. Sensible Testing Some shortcomings of these approaches include: lots more code. More permutations of QA. For every test variant you add, you essentially add the need for another QA user story. You risk introducing an unpredictable user experience. Changing minor interactions are probably okay, changing the main navigation scheme in your application probably isn’t. Analysis paralysis - At some point you’ve just got to make decisions. A/B testing only goes so far in helping you choose direction for your product. You can’t be creative in your product decisions by just piling A/B tests back to back.
  • 72. Things to Watch Out For Make sure default is "off" for predictable stable experiences and sanity of future support. You don’t want to have a bunch of old flags you need to keep around for un-updated apps. If you've got a good app and a good idea going, you've probably not going to discover a breakaway victory by running an a/b test. They're too subtle and often times only prove that your intuition about your users is heavily biased.
  • 73. The Future So get out there, do some experiments. Time is precious. If you're going to make heads or tails of the numbers you see in an experiment, you'll probably need all the time you can get.
  • 74. Feature Configuration and A/B Experiments in native and hybrid mobile apps Breaking Development Nashville October 21 2013 lacy@etsy.com