ABBY - A Django app to document your A/B tests by Andy Goldschmidt PyData Berlin 2014

Uploaded on

ABBY is a Django app that helps you manage your A/B tests. The main objective is to document all tests happening in your company, in order to better understand which measures work and which don't. …

ABBY is a Django app that helps you manage your A/B tests. The main objective is to document all tests happening in your company, in order to better understand which measures work and which don't. Thereby leading to a better understanding of your product and your customer. ABBY offers a front-end that makes it easy to edit, delete or create tests and to add evaluation results. Further, it provides a RESTful API to integrate directly with our platform to easily handle A/B tests without touching the front-end. Another notable feature is the possibility to upload a CSV file and have the A/B test auto-evaluated, although this feature is considered highly experimental. At Jimdo, a do-it-yourself website builder, we have a team of about 180 people from different countries and with professional backgrounds just as diverse. Therefore it is crucial to have tools that allow having a common perspective on the tests. This facilitates having data informed discussions and to deduce effective solutions. In our opinion tools like ABBY are cornerstones to achieve the ultimate goal of being a data-driven company. It enables all our co-workers to review past and plan future tests to further improve our product and to raise the happiness of our customers. The proposed talk will give a detailed overview of ABBY, which eventually will be open-sourced, and its capabilities. I will further discuss the motivation behind the app and the influence it has on the way our company is becoming increasingly data driven.

More in: Data & Analytics
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. ABBY A Django app for A/B test documentation
  • 2. Who? ● Data Scientist at Jimdo ● Jimdo ○ DIY website builder ○ founded in 2007 in Hamburg, Germany ○ > 180 employees in 3 countries (DE, US, JP) ○ > 10 million websites Twitter: @datenheini
  • 3. Medicine: ● placebo = control group ● drug(s) = test group(s) A/B Testing Basics Internet company: = old version = new version(s)
  • 4. A/B Testing Basics Rules: ● Frame your hypothesis. ● Keep it simple. ● Set a timeframe.
  • 5. A/B Testing Basics Best practices: ● Be realistic. ● Grab low-hanging fruits. ● Don’t get frustrated.
  • 6. A/B Testing Basics 2 minutes of science...
  • 7. Significance (p-value): A/B Testing Basics → should be 0% (at most 5%) How often will failed tests lead to positive results?
  • 8. Statistical power: A/B Testing Basics → should be 100% (at least 80%) How often will you recognize a successful test?
  • 9. Evaluation metrics: ● p-value ● statistical power ● effect size ● confidence interval A/B Testing Basics
  • 10. Let’s do a little A/B test And see what we need documentation for.
  • 11. Why? Communicating the results
  • 12. Why? Persisting the results → Knowledge Base
  • 13. Why? Avoid duplicated tests
  • 14. Why? Everybody has this problem, there needs to be a solution already! NO!
  • 15. Why?
  • 16. What?
  • 17. What? Central place for A/B test documentation
  • 18. What? Tests and results self-explaining
  • 19. What? Keep track of evaluation metrics
  • 20. What? Tries to encourage best practices
  • 21. What? Productivity gain
  • 22. What? Productivity gain! Reference for daily work Cross-functional effects Better understanding of customers
  • 23. How?
  • 24. How? CRUD app (create-read-update-delete) RESTful API (WIP) Test evaluation logic
  • 25. Demo time!
  • 26. It’s your turn! ABBY is open-source. Have a look:
  • 27. Thank you!