ABBY - A Django app to document your A/B tests by Andy Goldschmidt PyData Berlin 2014


Published on

ABBY is a Django app that helps you manage your A/B tests. The main objective is to document all tests happening in your company, in order to better understand which measures work and which don't. Thereby leading to a better understanding of your product and your customer. ABBY offers a front-end that makes it easy to edit, delete or create tests and to add evaluation results. Further, it provides a RESTful API to integrate directly with our platform to easily handle A/B tests without touching the front-end. Another notable feature is the possibility to upload a CSV file and have the A/B test auto-evaluated, although this feature is considered highly experimental. At Jimdo, a do-it-yourself website builder, we have a team of about 180 people from different countries and with professional backgrounds just as diverse. Therefore it is crucial to have tools that allow having a common perspective on the tests. This facilitates having data informed discussions and to deduce effective solutions. In our opinion tools like ABBY are cornerstones to achieve the ultimate goal of being a data-driven company. It enables all our co-workers to review past and plan future tests to further improve our product and to raise the happiness of our customers. The proposed talk will give a detailed overview of ABBY, which eventually will be open-sourced, and its capabilities. I will further discuss the motivation behind the app and the influence it has on the way our company is becoming increasingly data driven.

Published in: Data & Analytics
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

ABBY - A Django app to document your A/B tests by Andy Goldschmidt PyData Berlin 2014

  1. 1. ABBY A Django app for A/B test documentation
  2. 2. Who? ● Data Scientist at Jimdo ● Jimdo ○ DIY website builder ○ founded in 2007 in Hamburg, Germany ○ > 180 employees in 3 countries (DE, US, JP) ○ > 10 million websites Twitter: @datenheini
  3. 3. Medicine: ● placebo = control group ● drug(s) = test group(s) A/B Testing Basics Internet company: = old version = new version(s)
  4. 4. A/B Testing Basics Rules: ● Frame your hypothesis. ● Keep it simple. ● Set a timeframe.
  5. 5. A/B Testing Basics Best practices: ● Be realistic. ● Grab low-hanging fruits. ● Don’t get frustrated.
  6. 6. A/B Testing Basics 2 minutes of science...
  7. 7. Significance (p-value): A/B Testing Basics → should be 0% (at most 5%) How often will failed tests lead to positive results?
  8. 8. Statistical power: A/B Testing Basics → should be 100% (at least 80%) How often will you recognize a successful test?
  9. 9. Evaluation metrics: ● p-value ● statistical power ● effect size ● confidence interval A/B Testing Basics
  10. 10. Let’s do a little A/B test And see what we need documentation for.
  11. 11. Why? Communicating the results
  12. 12. Why? Persisting the results → Knowledge Base
  13. 13. Why? Avoid duplicated tests
  14. 14. Why? Everybody has this problem, there needs to be a solution already! NO!
  15. 15. Why?
  16. 16. What?
  17. 17. What? Central place for A/B test documentation
  18. 18. What? Tests and results self-explaining
  19. 19. What? Keep track of evaluation metrics
  20. 20. What? Tries to encourage best practices
  21. 21. What? Productivity gain
  22. 22. What? Productivity gain! Reference for daily work Cross-functional effects Better understanding of customers
  23. 23. How?
  24. 24. How? CRUD app (create-read-update-delete) RESTful API (WIP) Test evaluation logic
  25. 25. Demo time!
  26. 26. It’s your turn! ABBY is open-source. Have a look:
  27. 27. Thank you!