django-lean<br />An open-source framework for performing and analyzing a/b split-test experiments in Django<br />
60 sec. intro to split testing<br />
What is split testing?<br />A technique for comparing two alternative implementations<br />Done properly, allows you to el...
Why do people split test?<br />Improve conversion, ARPU, or other key metrics systematically<br />Eliminate guesswork and ...
Introducing django-lean<br />
django-lean<br />Experiment<br />JavaScript<br />Django templates<br />Python<br />Measure<br />User-defined goals<br />Us...
Statistical significance
Start, stop, and promote experiments via django-admin</li></li></ul><li>Examples<br />
Python<br />
Django template<br />
JavaScript<br />In your HTML template:<br />In your JavaScript:<br />
Reports<br />
future<br />
Wishlist<br />Experimentation improvements<br />Multivariate testing<br />Arbitrary numbers of test cases<br />Tools for a...
Upcoming SlideShare
Loading in...5
×

django-lean, Akoha's open-source a/b experimentation framework @ Montreal Python 9

5,161

Published on

'django-lean' is an open source a/b split-test experimentation framework for Django.

'django-lean' was created by Akoha, a Montreal-based startup building the world's first social reality game.

This presentation was delivered on September 30th, 2009 at Montreal Python 9, by Erik Wright, Director of Engineering for Akoha.

For further details visit:

http://bitbucket.org/akoha/django-lean/
http://akoha.com
http://twitter.com/erikwright

Published in: Technology
0 Comments
8 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,161
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
8
Embeds 0
No embeds

No notes for slide
  • Erik Wright, Dir. Eng @ AkohaHere to present django-leanAkoha’s recently open-sourced split-testing framework for Django
  • A tool for measuring differences between alternative implementations* font sizes, texts, prices, images, and so on.Recently, use for measuring effectiveness of new features and enhancements is gainingSimultaneously run both implementations,randomly assign users to one or the other, compare performance
  • Rarely confident of the impact our changes will have on users / customersEven when confident, sometimes we are dead wrongSplit testing reduces the risk of wrong decisions / find, correct, and learn from mistakes quickly.
  • Trivially implement split-test experiments in Javascript, Django Templates, PythonMeasures user performance according to metrics you defineGenerates and publishes nightly reports of experiment resultsIncluding calculating statistical significance so that you know when you can stop testingThanks to django-admin, starting, stopping, and promoting tests is easy
  • Now that I’ve told you what django-lean can do, allow me to show you
  • django-lean, Akoha's open-source a/b experimentation framework @ Montreal Python 9

    1. 1. django-lean<br />An open-source framework for performing and analyzing a/b split-test experiments in Django<br />
    2. 2. 60 sec. intro to split testing<br />
    3. 3. What is split testing?<br />A technique for comparing two alternative implementations<br />Done properly, allows you to eliminate other variables<br />For the gory details, seehttp://elem.com/~btilly/effective-ab-testing/<br />
    4. 4. Why do people split test?<br />Improve conversion, ARPU, or other key metrics systematically<br />Eliminate guesswork and opinions<br />Determine, scientifically, which implementation is best<br />Test your hypotheses<br />
    5. 5. Introducing django-lean<br />
    6. 6. django-lean<br />Experiment<br />JavaScript<br />Django templates<br />Python<br />Measure<br />User-defined goals<br />User-defined metric<br />Analyze<br />Manage<br /><ul><li>Nightly reports
    7. 7. Statistical significance
    8. 8. Start, stop, and promote experiments via django-admin</li></li></ul><li>Examples<br />
    9. 9. Python<br />
    10. 10. Django template<br />
    11. 11. JavaScript<br />In your HTML template:<br />In your JavaScript:<br />
    12. 12. Reports<br />
    13. 13. future<br />
    14. 14. Wishlist<br />Experimentation improvements<br />Multivariate testing<br />Arbitrary numbers of test cases<br />Tools for automated (integration, unit, ...) testing<br />Support more test scenarios<br />Conversion funnels<br />Decoupling from Django<br />More “Lean Startup” tools<br />Cohort analysis<br />In-site surveys (net promoter score, etc.)<br />
    15. 15. Find out more:<br />http://bitbucket.org/akoha/django-lean/<br />http://akoha.com<br />erik@erikwright.com / @erikwright<br />

    ×