Launchpad
   The good, the bad, and
the OMG how does that work?

                                  Tim Penhey
            ...
Launchpad? WTF is Launchpad?
●   Python based web application
●   ~350k lines of python
●   ~25k tests
●   Uses Zope 3 and...
Kiwi PyCon 2009
Learn From Others
●   Been in development for over five years
●   Made lots of mistakes
●   Found many ways not to do thin...
Testing is Essential
●   All types of tests throughout the code
●   Unit tests
●   Doc tests
●   Acceptance level tests
● ...
Slow Tests are Bad
●   Complete test suite run takes around four hours
●   Cannot run all tests for every landing now
    ...
Code Reviews Are Good
●   All changes are reviewed by at least one other
    developer
●   New team members went through a...
Multiple Environments are Good
●   Production – updated every four weeks
    ●   https://launchpad.net
●   Edge – updated ...
Can lead to many branches
●   devel – primary development branch
●   stable – devel with all tests passed
    ●   Rolled t...
Bazaar makes it all workable
●   Bazaar is a distributed revision control system
    (DVCS)
●   Merging just works
●   Dev...
Object-Relational Mappers
●   Mixed blessing
●   Originally used SQLObject
●   Moved to Storm
    ●   http://storm.canonic...
Branches
●   Represent bazaar branches in Launchpad
●   Have owners
●   Most belong to a project or source package
●   Bra...
Branch Listings
●   Shows up to 100 branches on a page
●   Primary text is the “bazaar identity”
●   For any single branch...
Branch Listing Solution
●   Utility class – BranchListingQueryOptimiser
●   lazr.delegates
    ●   Wraps real object but c...
Branch Collection Idiom
●   Global utility to get all branches
      >>> collection = getUtility(IAllBranches)
●   Filter ...
Branch Collection Idiom
●   Accessor methods return result sets
        >>> branches = collection.getBranches()
        >>...
Interfaces are Good
●   zope.interface and zope.schema
    ●   schema contains field types


    class IFruit(Interface):
...
Interfaces are Good
●   Model classes implement interfaces

    class Fruit(object):
      implements(IFruit)
      def __...
Interfaces are Good
>>> apple = Fruit('apple')
>>> IFruit.providedBy(apple)
True
>>> from zope.interfaces.verify import (
...
Zope Views and Pages
●   Path traversal is mostly object traversal
●   Views are associated with objects through
    inter...
Adapters Supercharge Interfaces
●   Adapters allow an object to be converted to a
    different interface in a defined way...
Be Smart About Pages
●   Many different objects have branches
●   Simple branch listing registered against
    IHasBranche...
Launchpad API using lazr.restful
●   Annotations to the interface class allow the
    objects to be exposed over a ReST ba...
Modules Matter
●   Initially all of Launchpad was in the
    canonical.launchpad module
    ●   .browser – contained the v...
The Module Move Apocalypse
●   Each team was responsible for moving code
●   New base module “lp” chosen
●   lp.registry –...
ZCML Is Not Fun
●   Zope Configuration Markup Language is an
    XML document
●   Defines content classes, security, adapt...
Learn to use TAGS
●   TAGS are used describe the location of the
    function definition
●   Can be read by Emacs and Vi
●...
Databases Evolve
●   Your database schema will evolve
    ●   New tables and columns
    ●   Changing columns
    ●   With...
Share Common Components
●   lazr-js               ●   lazr.publisher
●   lazr.authentication   ●   lazr.restful
●   lazr.b...
Eat Your Own Dog Food




                        Kiwi PyCon 2009
That's It



Questions?




             Kiwi PyCon 2009
Lunch!




         Kiwi PyCon 2009
Upcoming SlideShare
Loading in...5
×

Launchpad: Lessons Learnt

1,823

Published on

Some lessons learnt from five years of Launchpad development.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,823
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Launchpad: Lessons Learnt

  1. 1. Launchpad The good, the bad, and the OMG how does that work? Tim Penhey tim@penhey.net IRC: thumper #nzpug Kiwi PyCon 2009
  2. 2. Launchpad? WTF is Launchpad? ● Python based web application ● ~350k lines of python ● ~25k tests ● Uses Zope 3 and Twisted libraries ● Developed primarily by Canonical ● AGPL3 now ● https://launchpad.net Kiwi PyCon 2009
  3. 3. Kiwi PyCon 2009
  4. 4. Learn From Others ● Been in development for over five years ● Made lots of mistakes ● Found many ways not to do things ● Nuggets hidden in the depths ● Here to share Kiwi PyCon 2009
  5. 5. Testing is Essential ● All types of tests throughout the code ● Unit tests ● Doc tests ● Acceptance level tests ● Windmill Javascript tests ● Run your tests automatically Kiwi PyCon 2009
  6. 6. Slow Tests are Bad ● Complete test suite run takes around four hours ● Cannot run all tests for every landing now ● Buildbot, devel and stable branches ● Sample data for tests causes problems ● tearDown needs to reset DB state ● Zope test layers proliferate ● Run the test in the lowest layer possible Kiwi PyCon 2009
  7. 7. Code Reviews Are Good ● All changes are reviewed by at least one other developer ● New team members went through a mentoring process to learn from other reviewers ● Extra eyes on code can spot issues that the developer misses ● Reviewer makes sure new code has tests ● Have a coding standard Kiwi PyCon 2009
  8. 8. Multiple Environments are Good ● Production – updated every four weeks ● https://launchpad.net ● Edge – updated nightly ● https://edge.launchpad.net ● Beta testers are automatically redirected here ● Staging – updated nightly with production copy ● https://staging.launchpad.net ● Test area for people to mess around with Kiwi PyCon 2009
  9. 9. Can lead to many branches ● devel – primary development branch ● stable – devel with all tests passed ● Rolled to edge nightly ● db-devel – database patches + devel ● db-stable – db-devel with all the tests passed ● Rolled to staging nightly ● production – what we rolled out + cherry picks Kiwi PyCon 2009
  10. 10. Bazaar makes it all workable ● Bazaar is a distributed revision control system (DVCS) ● Merging just works ● Develop in branches ● Merge into devel (or db-devel) Kiwi PyCon 2009
  11. 11. Object-Relational Mappers ● Mixed blessing ● Originally used SQLObject ● Moved to Storm ● http://storm.canonical.com Kiwi PyCon 2009
  12. 12. Branches ● Represent bazaar branches in Launchpad ● Have owners ● Most belong to a project or source package ● Branches can be linked to be “official” ● lp:bzr ● lp:ubunutu/jaunty/gwibber ● lp:~thumper/launchpad/fix-branch-layout Kiwi PyCon 2009
  13. 13. Branch Listings ● Shows up to 100 branches on a page ● Primary text is the “bazaar identity” ● For any single branch it can effectively traverse across 8 tables ● Naïve approach would mean 800 queries to show a listing Kiwi PyCon 2009
  14. 14. Branch Listing Solution ● Utility class – BranchListingQueryOptimiser ● lazr.delegates ● Wraps real object but can override specific method ● View class determines visible batch ● Single queries executed for the “set” of visible branches Kiwi PyCon 2009
  15. 15. Branch Collection Idiom ● Global utility to get all branches >>> collection = getUtility(IAllBranches) ● Filter methods return a new branch collection >>> collection = collection.inProject(foo) >>> collection = collection.ownedBy(eric) ● Subclasses handle visibility based on privacy >>> collection = collection.visibleBy(user) >>> collection = collection.visibleBy(None) Kiwi PyCon 2009
  16. 16. Branch Collection Idiom ● Accessor methods return result sets >>> branches = collection.getBranches() >>> branches.order_by(Branch.name) ● Look at the code: ● lib/lp/code/model/branchcollection.py ● lib/lp/code/interfaces/branchcollection.py Kiwi PyCon 2009
  17. 17. Interfaces are Good ● zope.interface and zope.schema ● schema contains field types class IFruit(Interface): name = Text(required=True) def throw(target): """No self defined for methods.""" Kiwi PyCon 2009
  18. 18. Interfaces are Good ● Model classes implement interfaces class Fruit(object): implements(IFruit) def __init__(self, name): self.name = name def throw(self, target): return target.hit_with(self) Kiwi PyCon 2009
  19. 19. Interfaces are Good >>> apple = Fruit('apple') >>> IFruit.providedBy(apple) True >>> from zope.interfaces.verify import ( ... verifyObject) >>> verifyObject(IFruit, apple) True Kiwi PyCon 2009
  20. 20. Zope Views and Pages ● Path traversal is mostly object traversal ● Views are associated with objects through interfaces ● Resulting web pages are rendered views ● Most views use a page template Kiwi PyCon 2009
  21. 21. Adapters Supercharge Interfaces ● Adapters allow an object to be converted to a different interface in a defined way def branch_collection_for_product(project): """Adapt a project to a branch collection.""" return getUtility(IAllBranches).inProject(project) >>> collection = IBranchCollection(project) Kiwi PyCon 2009
  22. 22. Be Smart About Pages ● Many different objects have branches ● Simple branch listing registered against IHasBranches instead of individual interfaces ● View adapts the objects to IBranchCollection ● Each object that implements IHasBranches also has an IBranchCollection adapter Kiwi PyCon 2009
  23. 23. Launchpad API using lazr.restful ● Annotations to the interface class allow the objects to be exposed over a ReST based API ● This is still magic to me ● lazr.restful and lazr.restfulclient are found on Launchpad ● launchpadlib is a Python client API Kiwi PyCon 2009
  24. 24. Modules Matter ● Initially all of Launchpad was in the canonical.launchpad module ● .browser – contained the views ● .interfaces – contained all the interfaces ● .database – contained all the model code ● .templates – contained all the page templates ● .doc – documentation including doc tests ● .ftests, .pagetests, .webapp, … ● Became just too cumbersome Kiwi PyCon 2009
  25. 25. The Module Move Apocalypse ● Each team was responsible for moving code ● New base module “lp” chosen ● lp.registry – people, projects, distributions ● lp.bugs – code related to the bug tracker ● lp.services – code used by other applications ● mail, jobs, testing, scripts ● lp.codehosting, lp.blueprints, lp.translations Kiwi PyCon 2009
  26. 26. ZCML Is Not Fun ● Zope Configuration Markup Language is an XML document ● Defines content classes, security, adapters, utilities, views ● Registration away from the actual code hinders discoverability ● Ideally we'd like to bring in some of the ideas from Grok and Martian Kiwi PyCon 2009
  27. 27. Learn to use TAGS ● TAGS are used describe the location of the function definition ● Can be read by Emacs and Vi ● Have a make target (or equivalent) to build your TAGS file Kiwi PyCon 2009
  28. 28. Databases Evolve ● Your database schema will evolve ● New tables and columns ● Changing columns ● With time comes complexity ● Have a way to evolve your database schema ● Launchpad uses SQL patch files to describe discrete changes ● Production database is updated every 4 weeks Kiwi PyCon 2009
  29. 29. Share Common Components ● lazr-js ● lazr.publisher ● lazr.authentication ● lazr.restful ● lazr.batchnavigator ● lazr.restfulclient ● lazr.canonicalurl ● lazr.smtptest ● lazr.config ● lazr.testing ● lazr.delegates ● lazr.uri ● lazr.enum ● storm ● lazr.exportedfolder ● wadllib ● lazr.lifecycle Kiwi PyCon 2009
  30. 30. Eat Your Own Dog Food Kiwi PyCon 2009
  31. 31. That's It Questions? Kiwi PyCon 2009
  32. 32. Lunch! Kiwi PyCon 2009

×