• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Launchpad: Lessons Learnt
 

Launchpad: Lessons Learnt

on

  • 2,600 views

Some lessons learnt from five years of Launchpad development.

Some lessons learnt from five years of Launchpad development.

Statistics

Views

Total Views
2,600
Views on SlideShare
2,584
Embed Views
16

Actions

Likes
2
Downloads
0
Comments
0

3 Embeds 16

http://www.slideshare.net 11
http://trunk.ly 4
http://us-w1.rockmelt.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Launchpad: Lessons Learnt Launchpad: Lessons Learnt Presentation Transcript

    • Launchpad The good, the bad, and the OMG how does that work? Tim Penhey tim@penhey.net IRC: thumper #nzpug Kiwi PyCon 2009
    • Launchpad? WTF is Launchpad? ● Python based web application ● ~350k lines of python ● ~25k tests ● Uses Zope 3 and Twisted libraries ● Developed primarily by Canonical ● AGPL3 now ● https://launchpad.net Kiwi PyCon 2009
    • Kiwi PyCon 2009
    • Learn From Others ● Been in development for over five years ● Made lots of mistakes ● Found many ways not to do things ● Nuggets hidden in the depths ● Here to share Kiwi PyCon 2009
    • Testing is Essential ● All types of tests throughout the code ● Unit tests ● Doc tests ● Acceptance level tests ● Windmill Javascript tests ● Run your tests automatically Kiwi PyCon 2009
    • Slow Tests are Bad ● Complete test suite run takes around four hours ● Cannot run all tests for every landing now ● Buildbot, devel and stable branches ● Sample data for tests causes problems ● tearDown needs to reset DB state ● Zope test layers proliferate ● Run the test in the lowest layer possible Kiwi PyCon 2009
    • Code Reviews Are Good ● All changes are reviewed by at least one other developer ● New team members went through a mentoring process to learn from other reviewers ● Extra eyes on code can spot issues that the developer misses ● Reviewer makes sure new code has tests ● Have a coding standard Kiwi PyCon 2009
    • Multiple Environments are Good ● Production – updated every four weeks ● https://launchpad.net ● Edge – updated nightly ● https://edge.launchpad.net ● Beta testers are automatically redirected here ● Staging – updated nightly with production copy ● https://staging.launchpad.net ● Test area for people to mess around with Kiwi PyCon 2009
    • Can lead to many branches ● devel – primary development branch ● stable – devel with all tests passed ● Rolled to edge nightly ● db-devel – database patches + devel ● db-stable – db-devel with all the tests passed ● Rolled to staging nightly ● production – what we rolled out + cherry picks Kiwi PyCon 2009
    • Bazaar makes it all workable ● Bazaar is a distributed revision control system (DVCS) ● Merging just works ● Develop in branches ● Merge into devel (or db-devel) Kiwi PyCon 2009
    • Object-Relational Mappers ● Mixed blessing ● Originally used SQLObject ● Moved to Storm ● http://storm.canonical.com Kiwi PyCon 2009
    • Branches ● Represent bazaar branches in Launchpad ● Have owners ● Most belong to a project or source package ● Branches can be linked to be “official” ● lp:bzr ● lp:ubunutu/jaunty/gwibber ● lp:~thumper/launchpad/fix-branch-layout Kiwi PyCon 2009
    • Branch Listings ● Shows up to 100 branches on a page ● Primary text is the “bazaar identity” ● For any single branch it can effectively traverse across 8 tables ● Naïve approach would mean 800 queries to show a listing Kiwi PyCon 2009
    • Branch Listing Solution ● Utility class – BranchListingQueryOptimiser ● lazr.delegates ● Wraps real object but can override specific method ● View class determines visible batch ● Single queries executed for the “set” of visible branches Kiwi PyCon 2009
    • Branch Collection Idiom ● Global utility to get all branches >>> collection = getUtility(IAllBranches) ● Filter methods return a new branch collection >>> collection = collection.inProject(foo) >>> collection = collection.ownedBy(eric) ● Subclasses handle visibility based on privacy >>> collection = collection.visibleBy(user) >>> collection = collection.visibleBy(None) Kiwi PyCon 2009
    • Branch Collection Idiom ● Accessor methods return result sets >>> branches = collection.getBranches() >>> branches.order_by(Branch.name) ● Look at the code: ● lib/lp/code/model/branchcollection.py ● lib/lp/code/interfaces/branchcollection.py Kiwi PyCon 2009
    • Interfaces are Good ● zope.interface and zope.schema ● schema contains field types class IFruit(Interface): name = Text(required=True) def throw(target): """No self defined for methods.""" Kiwi PyCon 2009
    • Interfaces are Good ● Model classes implement interfaces class Fruit(object): implements(IFruit) def __init__(self, name): self.name = name def throw(self, target): return target.hit_with(self) Kiwi PyCon 2009
    • Interfaces are Good >>> apple = Fruit('apple') >>> IFruit.providedBy(apple) True >>> from zope.interfaces.verify import ( ... verifyObject) >>> verifyObject(IFruit, apple) True Kiwi PyCon 2009
    • Zope Views and Pages ● Path traversal is mostly object traversal ● Views are associated with objects through interfaces ● Resulting web pages are rendered views ● Most views use a page template Kiwi PyCon 2009
    • Adapters Supercharge Interfaces ● Adapters allow an object to be converted to a different interface in a defined way def branch_collection_for_product(project): """Adapt a project to a branch collection.""" return getUtility(IAllBranches).inProject(project) >>> collection = IBranchCollection(project) Kiwi PyCon 2009
    • Be Smart About Pages ● Many different objects have branches ● Simple branch listing registered against IHasBranches instead of individual interfaces ● View adapts the objects to IBranchCollection ● Each object that implements IHasBranches also has an IBranchCollection adapter Kiwi PyCon 2009
    • Launchpad API using lazr.restful ● Annotations to the interface class allow the objects to be exposed over a ReST based API ● This is still magic to me ● lazr.restful and lazr.restfulclient are found on Launchpad ● launchpadlib is a Python client API Kiwi PyCon 2009
    • Modules Matter ● Initially all of Launchpad was in the canonical.launchpad module ● .browser – contained the views ● .interfaces – contained all the interfaces ● .database – contained all the model code ● .templates – contained all the page templates ● .doc – documentation including doc tests ● .ftests, .pagetests, .webapp, … ● Became just too cumbersome Kiwi PyCon 2009
    • The Module Move Apocalypse ● Each team was responsible for moving code ● New base module “lp” chosen ● lp.registry – people, projects, distributions ● lp.bugs – code related to the bug tracker ● lp.services – code used by other applications ● mail, jobs, testing, scripts ● lp.codehosting, lp.blueprints, lp.translations Kiwi PyCon 2009
    • ZCML Is Not Fun ● Zope Configuration Markup Language is an XML document ● Defines content classes, security, adapters, utilities, views ● Registration away from the actual code hinders discoverability ● Ideally we'd like to bring in some of the ideas from Grok and Martian Kiwi PyCon 2009
    • Learn to use TAGS ● TAGS are used describe the location of the function definition ● Can be read by Emacs and Vi ● Have a make target (or equivalent) to build your TAGS file Kiwi PyCon 2009
    • Databases Evolve ● Your database schema will evolve ● New tables and columns ● Changing columns ● With time comes complexity ● Have a way to evolve your database schema ● Launchpad uses SQL patch files to describe discrete changes ● Production database is updated every 4 weeks Kiwi PyCon 2009
    • Share Common Components ● lazr-js ● lazr.publisher ● lazr.authentication ● lazr.restful ● lazr.batchnavigator ● lazr.restfulclient ● lazr.canonicalurl ● lazr.smtptest ● lazr.config ● lazr.testing ● lazr.delegates ● lazr.uri ● lazr.enum ● storm ● lazr.exportedfolder ● wadllib ● lazr.lifecycle Kiwi PyCon 2009
    • Eat Your Own Dog Food Kiwi PyCon 2009
    • That's It Questions? Kiwi PyCon 2009
    • Lunch! Kiwi PyCon 2009