Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building an Octopus


Published on

An introduction about A/B Testing.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Building an Octopus

  1. 1. How to build an Octopus - Introduction to our new A/B testing and personalization service August 13, 2015 Soroosh Sarabadani and Till Riffert
  2. 2. What is Octopus? A central service to enable A/B testing and personalization of all the things
  3. 3. What is A/B testing? Variant A: blue Variant B: orange 50% get variant A 50% get variant B 11 % Conversion 23 % Conversion ● A/B testing measures impact of a variant compared to another variant ● General principle is also used for more than two variants and e.g. on article level ● Breaking down results by population properties (age, gender, order history) forms the basis for personalization Baseline Δ = +12%
  4. 4. Why do I need a central solution for A/B testing? Scenario: two teams independently perform testing and steering First team: ● What is the impact of 10€ voucher? ○ Give voucher to half of the test-customers ○ Do not give voucher to the other half ○ Measure impact, e.g. profit up by 12€ per customer Second team: ● What is the impact of a 20€ voucher on top customers? ○ Interferes with the voucher test by first team ○ Top customers in the control group of first team get a voucher now ○ Impact measured in testing is wrong ○ Profitable treatment is not pursued due to interference ○ Centralizing all testings and steerings solves this issue!
  5. 5. Octopus-Call (decision): In: steering point id, environment, entity id Out: variant, correlation id Feedback: In: correl. id, steering point id, used variant, reason, environm. Out: - Registration: In: name, entity-type, variants, weights, environment Out: steering point id Octopus is a service for including A/B testing and personalization capabilities into your features Feature (built by engineer) Octopus (A/B testing & steering) SteeringPoint
  6. 6. steering_point = "ecbf6791-102b-4ce6-b777-2b472dbfb550" //provided at registration app_domain_id = 1 customer_id = "98ba7d91ed2df8eb0385fe7c1b093498" GET variants/$app_domain?steering_point_id=$steering_point&/customer=$customer_id result: { "offered_variant":$offered_variant, "correlation_id": $correlation_id } PUT /feedbacks/$correlation_id payload: { "selected_variants": $selected_variant, "reason": "Not applicable" //only if offered_variant not used } Incorporating Octopus in your code takes only a GET and a PUT
  7. 7. Octopus ● … offers you live quality control functionalities ○ Automatic stopping when a variant significantly underperforms ○ Automatic notification of sudden deviations from previous behaviour ● ... has the overview over all testings and personalizations ○ Examine all tests and steerings in one place ● ... automatically detects and resolves interference ○ Checks all running tests and personalizations for interference ○ Automatically applies layer system for resolving interference ● … provides state of the art analysis ○ Uses the analysis framework built by the testing centre ○ Incorporates dashboards for exploring your data Using Octopus gives you access to convenient features, simplifying monitoring and analysis
  8. 8. The Octopus MVP is built and ready to use Octopus status: August 13th current version: ● Most important APIs ready to use (registration, decision, feedback) ● GUI for registration and management of all of a team's steering points ● Integrated forced assignment of variant (black-/white-lists) ● QA integration so unit/integration-tests for steering points can be built ● Several steering points can be requested at once using batch mode Next steps (end of Q3): ● Integration of within variant monitoring (anomaly detection) ● Integration with existing analysis module
  9. 9. Sounds awesome! Where do I find this Octopus?
  10. 10. Octopus architecture AB Differentiation Domain Resolver Decision tree evaluator Decision tree builder AB-SteerageData Fetcher Decision tree repository Aggregated data Shop data Customer etc. Analysis RESTA PIs Analysis repository AB repository
  11. 11. Customer Customer Customer Customer Customer Customer Development and Production environment Admin Nodes Production environment Service Nodes Monitoring Nodes Cassandra Nodes Admin Node Development environment Service Node Monitoring NodeCassandra Node Version 0.1 Admin Node Service Node Monitoring NodeCassandra Node Version 0.2 Auto Scaling Group Auto Scaling Group Auto Scaling Group Auto Scaling Group
  12. 12. The Octopus technology stack
  13. 13. ● Any particular set of objects that can be split into multiple groups for A/B testing or personalization is called an entity type ● The available data is specific to the entity type ● A specific member of an entity type is called entity ● Related entity types form entity domains Article domain (e.g. pricing-tests) Model SKU Config SKU Simple SKU Customer domain (e.g. label-in-parcel test) Customer Order Shipment Pession Session What are entity-types and how are they related?
  14. 14. Octopus offers you live quality control functionalities (under development, scheduled for end of Q3) Between variant monitoring ● Automatically switch off variants which significantly underperform ● Able to detect errors in implementation of variant Within variant monitoring ● Alarms for significant deviation from previous behaviour ● Monitores KPIs to give hints on unexpected events taking place Switch off variant D Switch off variant C Time KPI Variant A Variant B Variant C Variant D Impact of variants over time Unexpected behaviour
  15. 15. Layer A Layer B A layer-system resolves and prevents interference (under development, integration scheduled for Q4) Checkout text color Black text (default) Orange text Checkout background color Orange background White background (d.) Steering points interfere Idea behind the layer-system: ● Each steering point starts out in new layer ● Interfering steering points are merged in one layer ● Each entity in a layer is assigned to one variant in one of the steering points and receives the default variant in all others
  16. 16. A layer-system resolves and prevents interference (under development, integration scheduled for Q4) Idea behind the layer-system: ● Each steering point starts out in new layer ● Interfering steering points are merged in one layer ● Each entity in a layer is assigned to one variant in one of the steering points and receives the default variant in all others Combined layer Checkout button Black checkout text (default) Orange checkout text Checkout background Orange background White background (d.)
  17. 17. Analysis dashboard Subgroup analysis Number of orders Customer Age Analysis dashboard Analysis over time Number of orders ● Delta KPI difference between treatment & control ● Analysis over time development of difference over time ● Subgroup analysis difference broken down by properties Octopus will include the analysis capabilities offered by the framework developed by the testing center @ DI ● Basic analysis capabilities will be migrated to Octopus by end of Q3 ● Migration of more complex analysis/scoring capabilities scheduled for Q4
  18. 18. Personalizing customer experience based on A/B testing results (part of analysis framework, integration to Octopus in Q4) Customer Female Male Top customer Non-top Customer >1 month to last order <1 month to last order + 5% orders (vs default) - 2% orders (vs default) + 1% orders (vs default) + 2% orders (vs default) ● Decision trees are built from test results for each variant ● Each leaf has a prediction on the impact of a treatment ● Traversing the tree for an entity leads to a prediction of the impact Female, Non-Top Customer, ... Use default variant
  19. 19. Steering ratio (#steerings / #entities) Controlling trade off between A/B testing and steering (research on topic scheduled for Q4) ● Usually a steering point starts with a steering ratio of 0% ● As soon as enough data is available the steering ratio can be adjusted ● Goal: option to automatically and optimally adjust the steering ratio 100% 0% Entitiesused forsteering Time Testing phase Steering phase (continuous learning)
  20. 20. ● Octopus is a centralized service for A/B testing and personalization ● Including octopus in your code requires only a GET and a PUT ● The MVP is ready to use and offers: ○ Registration/configuration of your own steering points ○ Requesting a decision on which variant to display ○ QA capabilities to ensure your code is working ● A lot of cool stuff is being built to make your life easier (~ end of Q3) ○ Integration of the analysis framework ○ Octopus integration of within variant monitoring solution ○ Prototype and octopus integration of between variant monitoring algorithm ○ Prototype of interference detection and automatic layer management tl;dl - Summary
  21. 21. ● Use Octopus and register your own steering point via its admin page ● Visit the Octopus tech wiki page for further information on octopus: https: // ● Join the A/B testing guild for Octopus updates & A/B testing info: https://groups.!forum/ab-testing ● Questions, ideas, feedback? Contact Octopus via hipchat or e-mail: hipchat: #octopus, email: You want to know even more about Octopus?
  22. 22. ● Team Octopus is hiring: ○ We are looking for engineers and business developers ○ If you know someone, please contact us We are hiring!
  23. 23. QUESTIONS?
  24. 24. Admin UI
  25. 25. Admin UI
  26. 26. Admin UI
  27. 27. Admin UI