Search

Motion Django Meetup

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

0 comments

Post a comment

    Post a comment
    Embed Video
    Edit your comment Cancel

    Notes on slide 1

    - The bottom layer is remoteobjects & batchhttp, we’ll talk about those later.
    - The Python TypePad API library is a pure Python client library for talking to the TypePad API
    - typepadapp is a Django application that provides basic functionality that’s generally useful for Django apps built on top of TypePad like user authentication, OAuth token management, session management, and session synchronization
    - typepad-motion is really just a thin layer on top of these building blocks - it’s a set of views, URLConfs, templates, and some static resources like CSS, images, and javascript

    So, to simplify things to a stupid degree, the Motion architecture looks something like this.

    PyPi, pip, distribute, setuptools, etc. help a lot. We could decompose Motion into separate apps and maintain a good design, but still make it really easy to install. It also let us leverage outside code like httplib2 without having to include all of that stuff in our distributions.

    We’re using a custom management command to install our own skeleton files for a typepad project. You don’t have to use it, but it will set up sane defaults for all the Motion settings. Obscure settings are in motion.settings, so our default settings.py imports * from there.

    Turns out Django’s Management commands in Django really easily. Django iterates over your installed apps and looks in management/commands/.py for a class called Command that subclasses management.base.BaseCommand. So once you have a project it’s easy to add commands. It’s a bit trickier for django-admin.py because you don’t have any installed apps by default. But you can specify a settings file to django-admin.py and it’ll load those settings before it tries to find the command. Score!

    Automatically creates directory structure for theming & static resources, creates urlpattern for static media. Stupid Django 1.1.1 “security patch” broke this stuff though, fixed in Motion trunk, but not in the PyPi version.

    All of our templates just extend motion/base/.html. Thus, you can create a template _with the same name_, also extend motion/base/.html, and override specific blocks. This is basically an alternative to letting you create templates that have different names and then pass them in as kwargs in your urls.py.

    We decomposed our templates a lot to make them more extensible. Problem is, whenever a template is rendered Django has to read it from disk, lex it, parse it, and then render it. It doesn’t do any caching. This was taking around 1ms per template. We were rendering hundreds of templates. It adds up.

    All Django requires for a “view” function is a “callable” that returns an HttpResponse. You can use a function to do this, but you can also use classes. We decided to use classes so we could provide hooks that make it easier to extend Motion - like batch HTTP stuff.

    Here’s an example of a RemoteObject class representing a Twitter user. The code for this comes with the remoteobjects package, by the way. There are a number of other examples that come with the package as well.

    - For context I’ll just summarize the resource model for the TypePad API

    Data API URLs always start with a noun and an id
    A bare URL like this retrieves a single object

    A third path level selects a sub-resource of a top-level object
    Many of these are lists

    List resources tend to support filters which constrain what items are returned.
    There are both parameterized filters...

    ...and boolean filters.
    and in some cases multiple filters can be combined together.

    Every page in Motion is constructed predominently from data retrieved from TypePad via the API.

    We need to get all of the required data out of TypePad as quickly as possible.

    At first, we just used parallel HTTP requests, which is pretty straightforward.
    This proved inefficient within TypePad’s infrastructure since we ended up with each request in its own Apache process.
    While other architectures are possible, completely reinventing our stack would be time-consuming.

    We briefly considered HTTP Pipelining as a purist solution,
    but in practice we found various problems.
    - Poor client library support
    - Poor infrastructure support (our load balancers don’t support pipelining)
    - Implementation would’ve been complicated due to having to maintain the request order.

    We eventually settled on having a special endpoint for submitting batch requests.

    The client submits a single request to the batch processor which contains a description of all of the constituent requests.
    The individual requests get handled, and are returned together as a single HTTP response.
    But we need to figure out what a description of a batch job looks like on the wire...

    We looked at some prior art here, but everything seemed to be specific to the API it was wrapping.
    Figuring that batch processing ought to be transparent, we wanted something that could exist as a separate infrastructure layer.

    Think of it as an HTTP proxy with a funny-looking frontend.
    We use a multipart MIME message with each part containing a HTTP message as our wire format.
    This allows HTTP messages to be wrapped with minimal overhead.

    Although our internal implementation actually does something more clever, the protocol presents the illusion that each request is being handled independently as if it were a proxied request to a data endpoint.

    We think this approach ought to work for other REST-based APIs, so we made a point of keeping it generic and we’ve released a draft specification and some sample implementations, including a Python client library which works with httplib2 and a Twisted-based proxy that can be used to put a batch processor frontend in front of some existing endpoints.

    8 Favorites

    Motion Django Meetup - Presentation Transcript

    1. Motion a application Tonight’s presentation brought to you by in association with The San Francisco Django Meetup Group
    2. The Us Ed Anuff Martin Atkins Leah Culver Nima Badiey Mike Malone Chris Hall Brad Choate
    3. A Live Demo
    4. Local DB Request Handler typepad-motion Django Motion App typepadapp python-typepad-api remoteobjects batchhttp TypePad
    5. Motion
    6. Motion
    7. It’s all BSD’d.
    8. Feel free to utilize our teqniks (a.k.a. steal our code)
    9. We Feedback But plz tell us if you make it better!
    10. Things we wanted • Easy to install • Easy to extend • Easy to integrate
    11. Easy to install $ easy_install -q typepad-motion $ django-admin.py typepadproject mymotion > --settings=motion.settings $ cat >> mymotion/local_settings.py OAUTH_CONSUMER_KEY = 'key' OAUTH_CONSUMER_SECRET = 'secret' OAUTH_GENERAL_PURPOSE_KEY = 'gp_key' OAUTH_GENERAL_PURPOSE_SECRET = 'gp_secret' $ cd mymotion $ ./manage.py syncdb $ ./manage.py runserver
    12. Easy to install • PyPi & pip (or setuptools or distribute) for package & dependency management • Custom management commands • typepadproject • typepadapp • Sane defaults
    13. Management Commands http://github.com/sixapart/typepadapp/blob/master/typepadapp/management/commands/typepadproject.py
    14. Easy to extend $ ./manage.py typepadapp myapp
    15. Base Templates $ cat motion/templates/motion/events.html {% extends "motion/base/events.html" %} $ http://github.com/sixapart/typepad-motion/tree/master/motion/templates/motion/
    16. Template Caching http://code.djangoproject.com/ticket/6262 http://github.com/sixapart/typepadapp/blob/master/typepadapp/cached_templates.py
    17. Class-based Views http://github.com/sixapart/typepadapp/blob/master/typepadapp/views/base.py#L61
    18. Easy to integrate It’s just a Django app. Not much more to say.
    19. Replaced auth.models.User http://github.com/sixapart/typepadapp/blob/master/typepadapp/models/users.py#L47 http://github.com/sixapart/typepadapp/blob/master/typepadapp/backends.py#L41
    20. Below the
    21. httplib2 • Developed by Joe Gregorio • Sports a number of features that are useful for talking to web services • Caching • Compression • Extensible authentication mechanism
    22. OAuth & httplib2 http://github.com/sixapart/python-typepad-api/blob/master/typepad/oauthclient.py#L106
    23. OAuth & httplib2 >>> import oauthclient >>> from oauth import oauth >>> consumer = oauth.OAuthConsumer('key', 'secret') >>> token = oauth.OAuthToken('key', 'secret') >>> client = oauthclient.OAuthHttp() >>> client.add_credentials(consumer, token, 'twitter.com') >>> client.request( 'http://twitter.com/account/verify_credentials.json') ... THE RESPONSE ...
    24. remoteobjects • We work a lot with RESTish web services at Six Apart • Mark Paschal (@markpasc) created an abstraction layer that allows us to create robust client libraries very easily • The remoteobjects library allows you to describe an API using a declarative syntax, like Django models • Works very well with RESTful APIs, but it’s generic enough to work with RPC APIs too
    25. remoteobjects class User(RemoteObject): id = fields.Field() name = fields.Field() screen_name = fields.Field() [ ... ] followers_count = fields.Field() status = fields.Object(‘status’) @classmethod def get_user(cls, http=None, **kwargs): url = ‘/usrs/show’ if ‘id’ in kwargs: url += ‘/%s.json’ % quote_plus(kwargs.pop(‘id’)) else: url += ‘.json’ query = urlencode(kwargs) url = urlunsplit((None, None, url, query, None)) return cls.get(urljoin(Twitter.endpoint, url), http=http)
    26. remoteobjects >>> t = twitter.Twitter() >>> t.add_credentials('mjmalone', 'thisisnotmypassword') >>> friends_timeline = t.friends_timeline() >>> [e.user.screen_name for e in friends.entries] ['ronaldheft', 'joestump', 'donttrythis', 'laughingsquid', 'FrankGruber', 'courtstarr', 'mbaratz', 'djchall', 'dozba', 'chrismessina', 'pop17', 'ijustine', 'calden', 'bryanveloso', 'jessenoller', 'pierre', 'optimarcusprime', 'snackfight', 'shiralazar']
    27. remoteobjects http://github.com/sixapart/remoteobjects
    28. Batch HTTP
    29. What? • A mechanism for retrieving several distinct resources in a single HTTP request.
    30. TypePad API URLs • The TypePad API is a REST API • Each distinct resource has its own URL which supports some subset of the operations GET, PUT, POST and DELETE.
    31. TypePad API URLs
    32. TypePad API URLs /users/apparentlymart noun id
    33. TypePad API URLs /users/apparentlymart/relationships noun id sub-resource
    34. TypePad API URLs /users/apparentlymart/relationships/@by-group/6p1234123412341234 noun id sub-resource parameterized filter
    35. TypePad API URLs /users/apparentlymart/relationships/@following noun id sub-resource boolean filter
    36. Motion’s Home Page All of the data here comes from the TypePad API
    37. Motion’s Home Page /users/apparentlymart /users/apparentlymart/relationships/@following /users/apparentlymart/relationships/@follower Requires 5 resources from api.typepad.com /groups/{id}/events /groups/{id}/memberships
    38. How can we retrieve those five resource from TypePad as quickly and efficiently as possible?
    39. Design Goals • Retrieve all required resources in parallel • Allow TypePad to optimize the multi-get to return a response as quickly as possible • Be consistent with how single resources are returned in the normal case • Invent as little as possible
    40. Parallel HTTP Requests • Reasonably easy from the client’s perspective, assuming that their HTTP client library doesn’t suck. • Difficult to optimize predictably on TypePad’s end because we can’t control the order and timing of the handling of individual requests.
    41. HTTP Pipelining • Poor support from libraries and infrastructure. • Responses must be returned in order of request, which complicates the API dispatcher.
    42. Batch Processing API
    43. Batch Processing API Client Batch Processor API Endpoint API Endpoint
    44. Batch Processing API • GData (ab?)uses Atom to define a list of operations. • AWS uses some AWS-specific query string arguments or SOAP-ish stuff. • Ideally we wanted batch processing to be an orthogonal layer.
    45. Batch Processing API • Long story short: we designed a batch processing mechanism that can hopefully apply to any REST-based API.
    46. Batch Processing API • Batch Processor is conceptually similar to an HTTP proxy. • The frontend accepts a POST request containing a MIME multipart message containing a bunch of requests, and returns a multipart message containing the corresponding response.
    47. Batch Processing API • Conceptually, the backend of the batch processor makes its own HTTP requests to the indicated resources like an HTTP proxy would. • In practice, TypePad’s implementation handles the entire batch job together and fakes up the separate responses.
    48. Batch Processing API • We’ve published a spec and some open source sample implementations in the hope that it’s useful to the community: http://martin.atkins.me.uk/specs/batchhttp http://github.com/sixapart/batchhttp http://github.com/sixapart/libhttp-request-multi-perl
    49. Python Batch Request def callback(url, subresponse, subcontent): self.subresponse = subresponse self.subcontent = subcontent client = BatchClient(endpoint= 'http://127.0.0.1:8000/batch-processor') client.batch_request() client.batch({'uri': 'http://example.com/moose'}, callback=callback) client.batch({'uri': 'http://example.com/mouse'}, callback=callback) client.complete_batch()
    50. That’s All! http://developer.typepad.com/ http://github.com/sixapart/typepad-motion/ Questions?

    + Mike MaloneMike Malone, 1 month ago

    custom

    1045 views, 8 favs, 1 embeds more stats

    TypePad Motion is a Django web application for crea more

    More info about this document

    © All Rights Reserved

    Go to text version

    • Total Views 1045
      • 1044 on SlideShare
      • 1 from embeds
    • Comments 0
    • Favorites 8
    • Downloads 22
    Most viewed embeds
    • 1 views on http://www.slideshare.net

    more

    All embeds
    • 1 views on http://www.slideshare.net

    less

    Flagged as inappropriate Flag as inappropriate
    Flag as inappropriate

    Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

    Cancel
    File a copyright complaint
    Having problems? Go to our helpdesk?

    Categories

    Tags

    -->
    Search
    RSS Feed
    What's new?

    © 2009 SlideShare Inc. All Rights Reserved