Launchpad
The good, the bad, and
the OMG how does that work?
Tim Penhey
tim@penhey.net
IRC: thumper
#nzpug
Kiwi PyCon 2009

Launchpad? WTF is Launchpad?
● Python based web application
● ~350k lines of python
● ~25k tests
● Uses Zope 3 and Twisted libraries
● Developed primarily by Canonical
● AGPL3 now
● https://launchpad.net
Kiwi PyCon 2009

Kiwi PyCon 2009

Learn From Others
● Been in development for over five years
● Made lots of mistakes
● Found many ways not to do things
● Nuggets hidden in the depths
● Here to share
Kiwi PyCon 2009

Testing is Essential
● All types of tests throughout the code
● Unit tests
● Doc tests
● Acceptance level tests
● Windmill Javascript tests
● Run your tests automatically
Kiwi PyCon 2009

Slow Tests are Bad
● Complete test suite run takes around four hours
● Cannot run all tests for every landing now
● Buildbot, devel and stable branches
● Sample data for tests causes problems
● tearDown needs to reset DB state
● Zope test layers proliferate
● Run the test in the lowest layer possible
Kiwi PyCon 2009

Code Reviews Are Good
● All changes are reviewed by at least one other
developer
● New team members went through a mentoring
process to learn from other reviewers
● Extra eyes on code can spot issues that the
developer misses
● Reviewer makes sure new code has tests
● Have a coding standard
Kiwi PyCon 2009

Multiple Environments are Good
● Production – updated every four weeks
● https://launchpad.net
● Edge – updated nightly
● https://edge.launchpad.net
● Beta testers are automatically redirected here
● Staging – updated nightly with production copy
● https://staging.launchpad.net
● Test area for people to mess around with
Kiwi PyCon 2009

Can lead to many branches
● devel – primary development branch
● stable – devel with all tests passed
● Rolled to edge nightly
● db-devel – database patches + devel
● db-stable – db-devel with all the tests passed
● Rolled to staging nightly
● production – what we rolled out + cherry picks
Kiwi PyCon 2009

Bazaar makes it all workable
● Bazaar is a distributed revision control system
(DVCS)
● Merging just works
● Develop in branches
● Merge into devel (or db-devel)
Kiwi PyCon 2009

Object-Relational Mappers
● Mixed blessing
● Originally used SQLObject
● Moved to Storm
● http://storm.canonical.com
Kiwi PyCon 2009

Branches
● Represent bazaar branches in Launchpad
● Have owners
● Most belong to a project or source package
● Branches can be linked to be “official”
● lp:bzr
● lp:ubunutu/jaunty/gwibber
● lp:~thumper/launchpad/fix-branch-layout
Kiwi PyCon 2009

Branch Listings
● Shows up to 100 branches on a page
● Primary text is the “bazaar identity”
● For any single branch it can effectively traverse
across 8 tables
● Naïve approach would mean 800 queries to
show a listing
Kiwi PyCon 2009

Branch Listing Solution
● Utility class – BranchListingQueryOptimiser
● lazr.delegates
● Wraps real object but can override specific method
● View class determines visible batch
● Single queries executed for the “set” of visible
branches
Kiwi PyCon 2009

Branch Collection Idiom
● Global utility to get all branches
>>> collection = getUtility(IAllBranches)
● Filter methods return a new branch collection
>>> collection = collection.inProject(foo)
>>> collection = collection.ownedBy(eric)
● Subclasses handle visibility based on privacy
>>> collection = collection.visibleBy(user)
>>> collection = collection.visibleBy(None)
Kiwi PyCon 2009

Branch Collection Idiom
● Accessor methods return result sets
>>> branches = collection.getBranches()
>>> branches.order_by(Branch.name)
● Look at the code:
● lib/lp/code/model/branchcollection.py
● lib/lp/code/interfaces/branchcollection.py
Kiwi PyCon 2009

Interfaces are Good
● zope.interface and zope.schema
● schema contains field types
class IFruit(Interface):
name = Text(required=True)
def throw(target):
"""No self defined for methods."""
Kiwi PyCon 2009

Interfaces are Good
● Model classes implement interfaces
class Fruit(object):
implements(IFruit)
def __init__(self, name):
self.name = name
def throw(self, target):
return target.hit_with(self)
Kiwi PyCon 2009

Interfaces are Good
>>> apple = Fruit('apple')
>>> IFruit.providedBy(apple)
True
>>> from zope.interfaces.verify import (
... verifyObject)
>>> verifyObject(IFruit, apple)
True
Kiwi PyCon 2009

Zope Views and Pages
● Path traversal is mostly object traversal
● Views are associated with objects through
interfaces
● Resulting web pages are rendered views
● Most views use a page template
Kiwi PyCon 2009

Adapters Supercharge Interfaces
● Adapters allow an object to be converted to a
different interface in a defined way
def branch_collection_for_product(project):
"""Adapt a project to a branch collection."""
return getUtility(IAllBranches).inProject(project)
>>> collection = IBranchCollection(project)
Kiwi PyCon 2009

Be Smart About Pages
● Many different objects have branches
● Simple branch listing registered against
IHasBranches instead of individual interfaces
● View adapts the objects to IBranchCollection
● Each object that implements IHasBranches
also has an IBranchCollection adapter
Kiwi PyCon 2009

Launchpad API using lazr.restful
● Annotations to the interface class allow the
objects to be exposed over a ReST based API
● This is still magic to me
● lazr.restful and lazr.restfulclient are found on
Launchpad
● launchpadlib is a Python client API
Kiwi PyCon 2009

Modules Matter
● Initially all of Launchpad was in the
canonical.launchpad module
● .browser – contained the views
● .interfaces – contained all the interfaces
● .database – contained all the model code
● .templates – contained all the page templates
● .doc – documentation including doc tests
● .ftests, .pagetests, .webapp, …
● Became just too cumbersome
Kiwi PyCon 2009

The Module Move Apocalypse
● Each team was responsible for moving code
● New base module “lp” chosen
● lp.registry – people, projects, distributions
● lp.bugs – code related to the bug tracker
● lp.services – code used by other applications
● mail, jobs, testing, scripts
● lp.codehosting, lp.blueprints, lp.translations
Kiwi PyCon 2009

ZCML Is Not Fun
● Zope Configuration Markup Language is an
XML document
● Defines content classes, security, adapters,
utilities, views
● Registration away from the actual code hinders
discoverability
● Ideally we'd like to bring in some of the ideas
from Grok and Martian
Kiwi PyCon 2009

Learn to use TAGS
● TAGS are used describe the location of the
function definition
● Can be read by Emacs and Vi
● Have a make target (or equivalent) to build your
TAGS file
Kiwi PyCon 2009

Databases Evolve
● Your database schema will evolve
● New tables and columns
● Changing columns
● With time comes complexity
● Have a way to evolve your database schema
● Launchpad uses SQL patch files to describe
discrete changes
● Production database is updated every 4 weeks
Kiwi PyCon 2009

Share Common Components
● lazr-js ● lazr.publisher
● lazr.authentication ● lazr.restful
● lazr.batchnavigator ● lazr.restfulclient
● lazr.canonicalurl ● lazr.smtptest
● lazr.config ● lazr.testing
● lazr.delegates ● lazr.uri
● lazr.enum ● storm
● lazr.exportedfolder ● wadllib
● lazr.lifecycle
Kiwi PyCon 2009

Eat Your Own Dog Food
Kiwi PyCon 2009

That's It
Questions?
Kiwi PyCon 2009

Lunch!
Kiwi PyCon 2009