Launchpad: Lessons Learnt

  • 1,751 views
Uploaded on

Some lessons learnt from five years of Launchpad development.

Some lessons learnt from five years of Launchpad development.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,751
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
0
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Launchpad The good, the bad, and the OMG how does that work? Tim Penhey tim@penhey.net IRC: thumper #nzpug Kiwi PyCon 2009
  • 2. Launchpad? WTF is Launchpad? ● Python based web application ● ~350k lines of python ● ~25k tests ● Uses Zope 3 and Twisted libraries ● Developed primarily by Canonical ● AGPL3 now ● https://launchpad.net Kiwi PyCon 2009
  • 3. Kiwi PyCon 2009
  • 4. Learn From Others ● Been in development for over five years ● Made lots of mistakes ● Found many ways not to do things ● Nuggets hidden in the depths ● Here to share Kiwi PyCon 2009
  • 5. Testing is Essential ● All types of tests throughout the code ● Unit tests ● Doc tests ● Acceptance level tests ● Windmill Javascript tests ● Run your tests automatically Kiwi PyCon 2009
  • 6. Slow Tests are Bad ● Complete test suite run takes around four hours ● Cannot run all tests for every landing now ● Buildbot, devel and stable branches ● Sample data for tests causes problems ● tearDown needs to reset DB state ● Zope test layers proliferate ● Run the test in the lowest layer possible Kiwi PyCon 2009
  • 7. Code Reviews Are Good ● All changes are reviewed by at least one other developer ● New team members went through a mentoring process to learn from other reviewers ● Extra eyes on code can spot issues that the developer misses ● Reviewer makes sure new code has tests ● Have a coding standard Kiwi PyCon 2009
  • 8. Multiple Environments are Good ● Production – updated every four weeks ● https://launchpad.net ● Edge – updated nightly ● https://edge.launchpad.net ● Beta testers are automatically redirected here ● Staging – updated nightly with production copy ● https://staging.launchpad.net ● Test area for people to mess around with Kiwi PyCon 2009
  • 9. Can lead to many branches ● devel – primary development branch ● stable – devel with all tests passed ● Rolled to edge nightly ● db-devel – database patches + devel ● db-stable – db-devel with all the tests passed ● Rolled to staging nightly ● production – what we rolled out + cherry picks Kiwi PyCon 2009
  • 10. Bazaar makes it all workable ● Bazaar is a distributed revision control system (DVCS) ● Merging just works ● Develop in branches ● Merge into devel (or db-devel) Kiwi PyCon 2009
  • 11. Object-Relational Mappers ● Mixed blessing ● Originally used SQLObject ● Moved to Storm ● http://storm.canonical.com Kiwi PyCon 2009
  • 12. Branches ● Represent bazaar branches in Launchpad ● Have owners ● Most belong to a project or source package ● Branches can be linked to be “official” ● lp:bzr ● lp:ubunutu/jaunty/gwibber ● lp:~thumper/launchpad/fix-branch-layout Kiwi PyCon 2009
  • 13. Branch Listings ● Shows up to 100 branches on a page ● Primary text is the “bazaar identity” ● For any single branch it can effectively traverse across 8 tables ● Naïve approach would mean 800 queries to show a listing Kiwi PyCon 2009
  • 14. Branch Listing Solution ● Utility class – BranchListingQueryOptimiser ● lazr.delegates ● Wraps real object but can override specific method ● View class determines visible batch ● Single queries executed for the “set” of visible branches Kiwi PyCon 2009
  • 15. Branch Collection Idiom ● Global utility to get all branches >>> collection = getUtility(IAllBranches) ● Filter methods return a new branch collection >>> collection = collection.inProject(foo) >>> collection = collection.ownedBy(eric) ● Subclasses handle visibility based on privacy >>> collection = collection.visibleBy(user) >>> collection = collection.visibleBy(None) Kiwi PyCon 2009
  • 16. Branch Collection Idiom ● Accessor methods return result sets >>> branches = collection.getBranches() >>> branches.order_by(Branch.name) ● Look at the code: ● lib/lp/code/model/branchcollection.py ● lib/lp/code/interfaces/branchcollection.py Kiwi PyCon 2009
  • 17. Interfaces are Good ● zope.interface and zope.schema ● schema contains field types class IFruit(Interface): name = Text(required=True) def throw(target): """No self defined for methods.""" Kiwi PyCon 2009
  • 18. Interfaces are Good ● Model classes implement interfaces class Fruit(object): implements(IFruit) def __init__(self, name): self.name = name def throw(self, target): return target.hit_with(self) Kiwi PyCon 2009
  • 19. Interfaces are Good >>> apple = Fruit('apple') >>> IFruit.providedBy(apple) True >>> from zope.interfaces.verify import ( ... verifyObject) >>> verifyObject(IFruit, apple) True Kiwi PyCon 2009
  • 20. Zope Views and Pages ● Path traversal is mostly object traversal ● Views are associated with objects through interfaces ● Resulting web pages are rendered views ● Most views use a page template Kiwi PyCon 2009
  • 21. Adapters Supercharge Interfaces ● Adapters allow an object to be converted to a different interface in a defined way def branch_collection_for_product(project): """Adapt a project to a branch collection.""" return getUtility(IAllBranches).inProject(project) >>> collection = IBranchCollection(project) Kiwi PyCon 2009
  • 22. Be Smart About Pages ● Many different objects have branches ● Simple branch listing registered against IHasBranches instead of individual interfaces ● View adapts the objects to IBranchCollection ● Each object that implements IHasBranches also has an IBranchCollection adapter Kiwi PyCon 2009
  • 23. Launchpad API using lazr.restful ● Annotations to the interface class allow the objects to be exposed over a ReST based API ● This is still magic to me ● lazr.restful and lazr.restfulclient are found on Launchpad ● launchpadlib is a Python client API Kiwi PyCon 2009
  • 24. Modules Matter ● Initially all of Launchpad was in the canonical.launchpad module ● .browser – contained the views ● .interfaces – contained all the interfaces ● .database – contained all the model code ● .templates – contained all the page templates ● .doc – documentation including doc tests ● .ftests, .pagetests, .webapp, … ● Became just too cumbersome Kiwi PyCon 2009
  • 25. The Module Move Apocalypse ● Each team was responsible for moving code ● New base module “lp” chosen ● lp.registry – people, projects, distributions ● lp.bugs – code related to the bug tracker ● lp.services – code used by other applications ● mail, jobs, testing, scripts ● lp.codehosting, lp.blueprints, lp.translations Kiwi PyCon 2009
  • 26. ZCML Is Not Fun ● Zope Configuration Markup Language is an XML document ● Defines content classes, security, adapters, utilities, views ● Registration away from the actual code hinders discoverability ● Ideally we'd like to bring in some of the ideas from Grok and Martian Kiwi PyCon 2009
  • 27. Learn to use TAGS ● TAGS are used describe the location of the function definition ● Can be read by Emacs and Vi ● Have a make target (or equivalent) to build your TAGS file Kiwi PyCon 2009
  • 28. Databases Evolve ● Your database schema will evolve ● New tables and columns ● Changing columns ● With time comes complexity ● Have a way to evolve your database schema ● Launchpad uses SQL patch files to describe discrete changes ● Production database is updated every 4 weeks Kiwi PyCon 2009
  • 29. Share Common Components ● lazr-js ● lazr.publisher ● lazr.authentication ● lazr.restful ● lazr.batchnavigator ● lazr.restfulclient ● lazr.canonicalurl ● lazr.smtptest ● lazr.config ● lazr.testing ● lazr.delegates ● lazr.uri ● lazr.enum ● storm ● lazr.exportedfolder ● wadllib ● lazr.lifecycle Kiwi PyCon 2009
  • 30. Eat Your Own Dog Food Kiwi PyCon 2009
  • 31. That's It Questions? Kiwi PyCon 2009
  • 32. Lunch! Kiwi PyCon 2009