The document discusses feature configuration and A/B testing in mobile apps. It describes how Etsy uses feature flags and continuous experimentation to iteratively develop and test new features. Features can be enabled or disabled for certain users or groups. Experiments follow a process of setting up a feature flag, determining user eligibility, coding the feature, internal testing, then launching the feature to a percentage of users while collecting analytics. This allows gathering feedback to improve products and user experience.
Revamped Image Upload: Whys and Hows - Flavio Percoco - OpenStack Day Israel ...Cloud Native Day Tel Aviv
Glance's community has been discussing a new way for doing image uploads. If you've been following the evolution of Glance's API, you'd likely be wondering why is there a need for yet another API for image uploads.
Join me in this talk where I'll present the discussions the team has had in the last months, the needs for this new API and how these goals will be achieved.
Lean Startup + Story Mapping = Awesome Products FasterBrad Swanson
To deliver the right outcomes, you need to learn your customers needs and validate your assumptions as early as possible. This means getting an early version of your product completed to start testing, validating and improving. This session will demonstrate how to combine Lean Startup and User Story Mapping techniques to determine where to start and how to learn early and often.
Participants will start with a partially completed Lean Canvas to flesh out and then define a product roadmap by building a Story Map. We will use Lean Startup concepts of Minimal Viable Product (MVP) and validated learning to focus on outcome over output.
Learning objectives:
Understand the importance of accelerated learning and techniques to achieve it
How a Lean Canvas can help shape your product vision and MVP
How to build a story map to create a product roadmap
How to use a story map to validate your users' journey
Revamped Image Upload: Whys and Hows - Flavio Percoco - OpenStack Day Israel ...Cloud Native Day Tel Aviv
Glance's community has been discussing a new way for doing image uploads. If you've been following the evolution of Glance's API, you'd likely be wondering why is there a need for yet another API for image uploads.
Join me in this talk where I'll present the discussions the team has had in the last months, the needs for this new API and how these goals will be achieved.
Lean Startup + Story Mapping = Awesome Products FasterBrad Swanson
To deliver the right outcomes, you need to learn your customers needs and validate your assumptions as early as possible. This means getting an early version of your product completed to start testing, validating and improving. This session will demonstrate how to combine Lean Startup and User Story Mapping techniques to determine where to start and how to learn early and often.
Participants will start with a partially completed Lean Canvas to flesh out and then define a product roadmap by building a Story Map. We will use Lean Startup concepts of Minimal Viable Product (MVP) and validated learning to focus on outcome over output.
Learning objectives:
Understand the importance of accelerated learning and techniques to achieve it
How a Lean Canvas can help shape your product vision and MVP
How to build a story map to create a product roadmap
How to use a story map to validate your users' journey
The Open Commerce Conference - Premature Optimisation: The Root of All EvilFabio Akita
This is the talk I presented in NYC at the Spree Conference. It's about how we may be making bad decisions out of blindly following misleading pitches. To avoid it, we just need to go back to the basics of CS: Don't optimize prematurely. Here's how.
Continuous (Production) Integration: Ruby on Rails Application Monitoring wit...jnewland
Feature: Ruby on Rails Application Monitoring with Cucumber
In order to ensure continuous application availability
A developer should be able to assert the behavior of production apps
From the outside in
Without using antiquated monitoring tools
To protect revenue
We know the world changes and, in order to be competitive, so must your software. That’s why we invented continuous validation.
In this presentation we discuss how we came to the conclusion that continuous validation is urgently needed, where and when it should take place and how it works.
We think there is no need to have a discrepancy between building the right things and doing this the right way at the same time.
A brownbag presentation at IPC media in London about the need to use libraries to make web development much less random and more professional. Get the audio at: http://www.archive.org/details/ProfessionalWebDevelopmentWithLibraries
Connecting with the enterprise - The how and why of connecting to Enterprise ...Kevin Poorman
Not everyone plays games; fewer games really take off. But we all use email, we all have systems that we login to just for work. These Enterprise systems are part of a large and ever-growing category of spending for IT Managers. But Enterprise systems ... stink. RubyMotion developers are in a unique position to integrate with these Enterprise Software systems while providing beautiful, functional and elegant interfaces. Join in as we learn why we should be writing Enterprise connected apps, and walk through an example of connecting to Salesforce using their iOS SDK.
DevDay 2013 - Building Startups and Minimum Viable ProductsBen Hall
DevDay (http://devday.pl),
20th of September 2013, Kraków
Video at http://www.youtube.com/watch?v=L4eTOvq2WmM&feature=c4-overview-vl&list=PLBMFXMTB7U74NdDghygvBaDcp67owVUUF
Introduction to GraphQL (or How I Learned to Stop Worrying about REST APIs)Hafiz Ismail
Talk for FOSSASIA 2016 (http://2016.fossasia.org)
----
This talk will give a brief and enlightening look into how GraphQL can help you address common weaknesses that you, as a web / mobile developer, would normally face with using / building typical REST API systems.
Let's stop fighting about whether we should implement the strictest interpretation of REST or how pragmatic REST-ful design is the only way to go, or debate about what REST is or what it should be.
A couple of demos (In Golang! Yay!) will be shown that are guaranteed to open up your eyes and see that the dawn of liberation for product developers is finally here.
Background: GraphQL is a data query language and runtime designed and used at Facebook to request and deliver data to mobile and web apps since 2012.
Hafiz Ismail (@sogko) is a contributor to Go / Golang implementation of GraphQL server library (https://github.com/graphql-go/graphql) and is looking to encourage fellow developers to join in the collaborative effort.
The Open Commerce Conference - Premature Optimisation: The Root of All EvilFabio Akita
This is the talk I presented in NYC at the Spree Conference. It's about how we may be making bad decisions out of blindly following misleading pitches. To avoid it, we just need to go back to the basics of CS: Don't optimize prematurely. Here's how.
Continuous (Production) Integration: Ruby on Rails Application Monitoring wit...jnewland
Feature: Ruby on Rails Application Monitoring with Cucumber
In order to ensure continuous application availability
A developer should be able to assert the behavior of production apps
From the outside in
Without using antiquated monitoring tools
To protect revenue
We know the world changes and, in order to be competitive, so must your software. That’s why we invented continuous validation.
In this presentation we discuss how we came to the conclusion that continuous validation is urgently needed, where and when it should take place and how it works.
We think there is no need to have a discrepancy between building the right things and doing this the right way at the same time.
A brownbag presentation at IPC media in London about the need to use libraries to make web development much less random and more professional. Get the audio at: http://www.archive.org/details/ProfessionalWebDevelopmentWithLibraries
Connecting with the enterprise - The how and why of connecting to Enterprise ...Kevin Poorman
Not everyone plays games; fewer games really take off. But we all use email, we all have systems that we login to just for work. These Enterprise systems are part of a large and ever-growing category of spending for IT Managers. But Enterprise systems ... stink. RubyMotion developers are in a unique position to integrate with these Enterprise Software systems while providing beautiful, functional and elegant interfaces. Join in as we learn why we should be writing Enterprise connected apps, and walk through an example of connecting to Salesforce using their iOS SDK.
DevDay 2013 - Building Startups and Minimum Viable ProductsBen Hall
DevDay (http://devday.pl),
20th of September 2013, Kraków
Video at http://www.youtube.com/watch?v=L4eTOvq2WmM&feature=c4-overview-vl&list=PLBMFXMTB7U74NdDghygvBaDcp67owVUUF
Introduction to GraphQL (or How I Learned to Stop Worrying about REST APIs)Hafiz Ismail
Talk for FOSSASIA 2016 (http://2016.fossasia.org)
----
This talk will give a brief and enlightening look into how GraphQL can help you address common weaknesses that you, as a web / mobile developer, would normally face with using / building typical REST API systems.
Let's stop fighting about whether we should implement the strictest interpretation of REST or how pragmatic REST-ful design is the only way to go, or debate about what REST is or what it should be.
A couple of demos (In Golang! Yay!) will be shown that are guaranteed to open up your eyes and see that the dawn of liberation for product developers is finally here.
Background: GraphQL is a data query language and runtime designed and used at Facebook to request and deliver data to mobile and web apps since 2012.
Hafiz Ismail (@sogko) is a contributor to Go / Golang implementation of GraphQL server library (https://github.com/graphql-go/graphql) and is looking to encourage fellow developers to join in the collaborative effort.
Qalcwise Introduction to Designer v20160215qalcwise
Create your first Qalcwise Business App Tutorial.
Qalcwise Business Applications Cloud Platform.
This tutorial let's you create your first qalcwise business application. It's 2-4h teaching course. Have fun!!!
Getting Started with Test Automation: Introduction to Cucumber with Lapis LazuliRebecca Eloise Hogg
Introduction to web testing with Cucumber using spriteCloud's Lapis Lazuli Cucumber helper.
This presentation will take you through the steps needed to set up a test automation project using Cucumber - a software tool that runs automated tests in the BBD style - in combination with Lapis Lazuli, a gem that provides Cucumber helper functions and scaffolding for easier web test automation suite development.
To do this you will need to have installed Ruby with some drivers and libraries. You can find detailed notes on how to do this here: http://www.testautomation.info/Getting_Started
To successfully follow this tutorial, prior knowledge of test automation isn’t needed, but knowledge of scripting - especially Ruby - and HTML skills are recommended.
Lapus Lazuli's GitHub depository: https://github.com/spriteCloud/lapis-lazuli
For more information about everything test automation, whether you need someone to help you set up TA in your organisation or want to find out more about our Test Automation as a Service options, email: info@spritecloud.com.
Creating a backlog of user stories is pretty straight forward but it doesn't help you when it comes to decisions like what to build first, how to prioritize and groom the backlog, how to scope and plan the project, and how to visualize progress. The traditional backlog is simply too flat and often too long to help you see the bigger picture and make good decisions. User Story Mapping helps simplify all of these common project issues. By adding a third dimension to your backlog, your team will make better decisions about priorities, scope, and planning while improving your ability to visualize progress.
In this practical session I’ll cover the basics of user story mapping before walking you through case studies of how our teams are using this approach and the results we are achieving. I'll show you the before, during, and after pictures from several projects so that you can understand how our maps progress during the projects and how we use them to influence iterative development, promote good decision making, and visualize priorities, plans, scope and progress.
Similar to Mobile App Feature Configuration and A/B Experiments (20)
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
2. Feature Configuration
and A/B Experiments
in native and hybrid mobile apps
Breaking Development Nashville
October 21 2013
lacy@etsy.com
Mostly at Etsy we work on greenfield development, new products or features from
scratch. However as our mobile platform matures and our apps start to mature, we
are finding ourselves wanting to take the principals we find most fruitful in other,
older parts of Etsy and apply them in the mobile realm. TLDR this is about the fun
stuff.
3. When I talk to people about working at Etsy I expect they want to talk about
deploying web applications on tons of servers..
5. But in fact typically people want to talk about the unique and sometimes strange
things you can find on Etsy.
6. Often little do these people know that interesting oddities can be purchased on their
phone or tablet.
7. One might wander what kind of business it is selling unique items over the internet,
but as you can see we do pretty well. In recent months we’ve seen as much as half of
our business happen on mobile devices. This makes mobile experimentation a really
exciting place to be.
8. Feature Flags
http://www.flickr.com/photos/mig/15964697/
The first of the concepts that we are trying to adapt from the old web world to mobile
is the notion of feature flags. We find wrapping things up as features is incredibly
useful. Another of these principals of engineering is something we call Continuous
Experimentation.
9. Continuous Experimentation
When we talk about Experimentation though we’re talking about A/B experiments. If
you’ve ever been using an app or a site and noticed that someone sitting RIGHT NEXT
to you has a different looking app or site, this is probably part of an experiment.
10. Continuous Experimentation
• make small changes
• stay honest
• don't break the [product]
The name we give it comes from what we call “Continuous Deployment”. Instead this
“Continuous Experimentation” describes how we try to continue to develop apps and
mature app features after they’ve been working out in the world and proven as largely
a good idea.
13. Continuous Experimentation
• make small changes
• stay honest
• don't break the [product]
We use this to develop our products iteratively using real world feedback and data
about our users and how their experience is.
14. Continuous Experimentation
• make small changes
• stay honest
• don't break the [product]
I love building things, making things. You can’t, for lack of a better analogy, you can’t
build an automobile or a vacuum cleaner or whatever thing, sell that thing, have
someone use that thing every day and then expect that you could change it, change
how the car works and look at how your changes effect the experience or the
performance of that thing from back at the factory.
16. Disclaimer
• Analytics / Big Data
• Experimental Analysis
• Exploratory Analysis
I’m going to talk mostly about design and development that will keep us from making
bad data or picking poor audiences. There are other elements here but let’s assume
we have some way to gather analytics and some way to analyze the data we have
gathered.
17. Continuous Experimentation
• Everyone does experiments
• But not everything works this way
• Rarely seller tools, usually public stuff
• The develop / release / cheer cycle
• Mobile Apps
Everyone at Etsy dabbles in experiments, at least a little. But really not every line and
every product gets measured by experiment. Sometimes products are just made
because they’re needed. Like making a mobile website.
20. The new product exists in a ton of places. These are all DIFFERENT teams mind you!
21. iPhone
Android
Web
Email
iPad
This is all to say that this New Thing exists in an ecosystem across all these different
mediums. This is kind of the mindset you have to be in with cross-platform features and
their a/b tests. A feature now becomes a dimension in and of itself. It is not just part of your
website. Features can transcend an individual app.
22. Launch Day
Fortunately in this story, on launch day, QA has already seen the new product in operation,
ahead of the world. The App Store review process is already done. We had a release go out
last week. The code that will show this New Thing lies “dormant” out in the wild, on the
website, in the web views in the app and even in the app binary itself.
27. One Line!
So the launch starts on a engineer’s macbook. We push, for sake of illustration, as
little as one line of web code out to our web servers and API servers. This one line
mentions The New Thing by name, saying simply “turn on The New Thing” and within
the granularity of 20-30 minutes you have the Newness throughout the product to
millions of users, across the ecosystem of all types, across the world.
28. Launch Day
So this story is really just like a parlor trick. It’s to get your attention. It’s fun to talk
about. The real value here is the illustration of this unique dimension, these
“features” which we can use to divide up our product that is spanning out in so many
different mediums.
29. Measure Everything
• Feature Configuration
• Benefits of Features
• Configuration as Tests
• How Flags Get to the Code
• Examples of Experiments
• Making Sensible Tests
The roadmap to making our mobile a/b experiments a reality..
30. Feature Configuration
So first some of the things we take for granted. Four years ago -four years is a really
long time- there was a blog article from Flickr I saw floating around on the internet. It
was about what they called feature flags. We all thought it was pretty cool stuff. I had
no idea why. Then at some point I started working at Etsy and now I have to preface
any technical discussion of interest with how feature configuration shapes the way we
make software.
31. Flags, Flippers, Flickr
http://code.flickr.net/2009/12/02/flipping-out/
$cfg = array();
$cfg[‘cool-dog-pics’];
$cfg[‘cool-dog-pics’] =
array(‘enabled’=>‘off’);
$cfg[‘cool-dog-pics’] =
array(‘enabled’=>‘on’);
This idea starts with a configuration array that names things on your site. Maybe cool
dog pics.
32. Flags, Flippers, Flickr
if ($cfg[‘cool-dog-pics’) {
echo Dogs::getCoolPics();
}
You basically push out all the code all the time. Even if it doesn’t work, you just keep
it turned off.
33. Feature Configuration
The original benefit here, the exciting thing about this from the Flickr article is that
you don’t have to merge code or do gigantic atomic (and painful) releases. We push
code 30, 40, 50 times a day. We push code that doesn’t even work, on purpose,
because pushing tons of little pieces of resilient code is a lot more predictable than
trying to do a big release.
34. More Cool Stuff
$cfg = array(
‘enabled’=>array(‘users’=>‘lacyrhoades’)
);
$cfg = array(
‘enabled’=>array(‘admin’=>true)
);
$cfg = array(
‘enabled’=>array(‘groups’=>54321)
);
You could maybe do something like only enable that branch of code to be active for
one person, just some people you work with or a group of users you know have a
particular interest in the feature.
36. Like here we’ve got what we call dashboards, so as we flip these switches on or off we
can look for anomalies and make sure nothing is blowing up.
37. Benefits of Features
Another trick we have is what we call “Slow Rampups”. Things don’t have to be just
“on” or “off”.
38. Benefits of Features
Launches can now happen on a schedule. There’s no sense in trying to make a
software deadline be punctuated by releasing that software. The software
development is going to take too long. You are going to need to be able to QA it,
change it and make it right.
39. Requisites
• No backwards incompatible changes
• Uniquely identified users
There are some limitations or requirements.. you’ll need to make sure the branches
in the code are not drastically different. The data schema for example has to be
backwards compatible. You’re going to need a way to uniquely identify users in order
to put them into groups and give them consistent experiences.
40. Features in the
New World
Feature flags were made for websites. But your website is not a website anymore. It
hasn’t been a website for a while now. Your website probably employs at least one
mobile developer who doesn’t even look at your “website” every day. Similarly feature
flags have to adapt to this new world.
41. Features as Tests
A/B Tests put people in buckets, giving them different values for one flag. Watch
what they do over time, this means attaching information about their test buckets to
analytics events. Make your app better based on the data you get back.
42. An Example
Feature
On the main screen. The individual facets you see there are powered by different parts
of the infrastructure.
43. The personal activity feed is fed from an map-reduce stack in PHP. The curated
panels are fed from a Java-based search stack. One of these might mess up and you
want the app to go on without it. This is a great place to start for dividing up features
in your design.
44. An Example
Prototype
For a prototype we might want to gather feedback from users, a select group, for
example the photo-editor in our app started as a prototype group and grew with
feedback we received.
45. This was a photo editing interface in the Etsy app. We wanted to know some
particularly qualitative things about it, like was it easy to use and understand. It’s not
exactly like we can study really dry analytics to get at this sort of answer. So we
invited some interested sellers to join a group on Etsy. Those sellers could see the
new tools and we could at that point gather feedback from them.
46. This was a redesign of the activity feed on our mobile web. We used a feedback group
to preview the features and make sure we got it right.
48. One of our recent experiments.. mobile templates vs. desktop templates. It turns out
from the measurements we could make that users were quantifiably more satisfied
with the desktop “look” of Etsy on tablet devices.
49. We wanted to really go down the road of experiments as the web side of Etsy has
before us. We took this listing view as a place to start. Here’s a reasonably priced
shadow puppet. Here it is (right) with the experiment enabled.
50. Here was the flow before. The last three steps were all web views on the server side.
51. Here’s the flow afterwards. The idea was reducing the number of steps would
significantly reduce friction in the checkout flow.
52. Experiment Steps
• Set up a feature flag
First we needed to look at eligibility, or who can take part in this experiment. If the
eligible audience is VERY small compared the general audience of all users, our total
number of "people who have used this feature" is going to be small, and so then the
numerator of "people who bought an item with this feature" will also be small. Then
there’s the feature flag. We make this and start to write code against it, so we’re
pushing code from day 1.
54. Experiment Steps
• Set up a feature flag
• Determine eligibility programmatically
Next we need to determine eligibility programatically. We’ve already determined our
eligible audience is of considerable size at this point. We did that as part of
exploratory analysis. This is more about being able to say in the code, something
like..
55. Eligibility
$eligible = isEligible($user,
$listing);
if ($eligible) { ... }
We want to answer questions like, do we have this user’s billing information on file?
Can this listing even be bought using that credit card? Can the seller who’s selling
this item ship the item to the country the user lives in? If the answers to any of these
questions are not good, we need to not be including this buyer in the experiment.
We’ll dilute our results quickly, since we know there are a bunch of combinations of
items and buyers who can’t take part in the experiment.
56. Experiment Steps
• Set up a feature flag
• Determine eligibility
• Start hacking away
At this point we start coding up the native elements and the web elements, all the
while hiding them behind the feature configuration we chose before. Generally being
careful if we need to add code to shared libraries or shared files.
57. Experiment Steps
• Set up a feature flag
• Determine eligibility
• Start hacking away
• Begin testing
When things are working the way we expect, we begin testing with a small internal
group, usually people in QA or just staff members. Also we begin to QA the app as a
whole for release.
58. Experiment Steps
• Set up a feature flag
• Determine eligibility
• Start hacking away
• Begin testing
• Put the product on the shelf
Once QA approves the way it works, most of the coding is done and we put the
feature on the shelf for a while. We’ll probably try to begin the app store review
process as soon as possible.
59. Eligibility
[EtsyConfig isEnabled:@”BuyItNow”];
The code in objective C will look something like this. The EtsyConfig class here is
going to be responsible for remembering “yes I did see this experiment, someone
asked about it, and the answer was: x” That answer, and the specific question we
came looking for need to be attached to analytics events the user is firing.
60. Experiment Steps
• Experiment group
• Up to a certain percentage
• Analytics events
When the app is live, we can implement an experiment group. We can ramp up slowly
so that we can kick the tires and know things are okay. When things are looking good
we’ll take the experiment up to a percentage we established beforehand. Analytics
events are capturing the state of this test as people see it. Typically you can ignore
the state of this feature on analytics events for people who are completely ineligible.
61. Looking at results
• Self Selection
• Refunds / Returns
• Visit-level vs. User-level
Our initial results were actually pretty good. There are other things to consider,
drawbacks and perhaps biased design of the experiment.
66. Configuration Steps
• App launch
• Periodically later, login
• Merge downloaded config
• Post notification
The config is downloaded and merged whenever the app launches. It also is
downloaded when the user logs in or out, as their experiments might change. As a
final step in this merging we’ll post a notification in the app code so that UI elements
in the app which need to update based on any experimental code, can do so.
67. Bucketing Users
You’re going to need to know who individual users are if you can put 20% into one
bucket and ensure that they stay there, and ensure that no one from the 80% control
group, makes it in to that experiment bucket.
68. Bucketing Users
• Persistent Cookies
• Device UDID
• user_id (where available)
The best thing to use here would be the user_id. This is not always available, like if
the user is logged out. Typically websites use persistent cookies to bucket logged out
users. Unfortunately an API for mobile apps doesn't have this avenue of cookies.
You've got to have some way to bucket users. One approach is to make up a sort of
UDID. Something that is specific to a device, and is stored locally with the app. This is
truly only unique to each install of the app, but it seems to work fairly well. You need
to pass that UDID to any webviews so that those webviews can identify themselves as
part of those app sessions.
69. Sensible Testing
We mentioned before that the number of users, the percentage, was key to obtaining
significant results. If you don’t run the experiment long enough, you’re going to not
prove anything. If you take too long measuring something, you’re wasting time.
There are deadlines to meet for this or other things, etc. For our test results we follow
the equation used here:
70. ExperimentCalculator.com
via @mcfunley
• How many eligible visits per day?
• What percentage of visits will see the change?
• What is your current conversion rate?
• How will you change conversion?
• How confident do you want to be?
• How likely should you be to detect the change?
experimentcalculator.com This is something one of our engineers made from a paper
on statistics. I don’t pretend to understand exactly but the idea is it gets you to not
choose random numbers for the length of the experiment. You have to weigh the cost
of developing a new feature against the feature’s potential value.
71. Sensible Testing
Some shortcomings of these approaches include: lots more code. More permutations
of QA. For every test variant you add, you essentially add the need for another QA
user story. You risk introducing an unpredictable user experience. Changing minor
interactions are probably okay, changing the main navigation scheme in your
application probably isn’t. Analysis paralysis - At some point you’ve just got to make
decisions. A/B testing only goes so far in helping you choose direction for your
product. You can’t be creative in your product decisions by just piling A/B tests back
to back.
72. Things to
Watch Out For
Make sure default is "off" for predictable stable experiences and sanity of future
support. You don’t want to have a bunch of old flags you need to keep around for
un-updated apps. If you've got a good app and a good idea going, you've probably
not going to discover a breakaway victory by running an a/b test. They're too subtle
and often times only prove that your intuition about your users is heavily biased.
73. The Future
So get out there, do some experiments. Time is precious. If you're going to make
heads or tails of the numbers you see in an experiment, you'll probably need all the
time you can get.
74. Feature Configuration
and A/B Experiments
in native and hybrid mobile apps
Breaking Development Nashville
October 21 2013
lacy@etsy.com