• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
How we build Vox
 

How we build Vox

on

  • 24,317 views

Slides about "How we build Vox" in Six Apart, presented by Benjamin Trott, the CTO and co-founder of the company in YAPC::Asia 2007 in Tokyo. The talk was ...

Slides about "How we build Vox" in Six Apart, presented by Benjamin Trott, the CTO and co-founder of the company in YAPC::Asia 2007 in Tokyo. The talk was done in English and interpreted by Tatsuhiko Miyagawa into Japanese.

Statistics

Views

Total Views
24,317
Views on SlideShare
24,041
Embed Views
276

Actions

Likes
47
Downloads
3
Comments
6

14 Embeds 276

http://poorbuthappy.com 220
http://www.poorbuthappy.com 15
http://www.slideshare.net 11
http://www.lonerunners.net 8
http://a4.vox-data.com 8
http://www.linkedin.com 3
http://a2.vox-data.com 2
http://www.filescon.com 2
http://whitehalladvisory.org 2
http://a5.vox-data.com 1
http://static.slideshare.net 1
http://www.oplahol.com 1
http://www.thoughtbag.com 1
http://209.85.173.104 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

16 of 6 previous next Post a comment

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • How nice to share such an experience with the rest of us
    Are you sure you want to
    Your message goes here
    Processing…
  • So nice of you to share this info. Thanks. I’m Ana Mui Stanley, working on my latest site on lyrics, www.lyrics-search.org/ . I enjoy reading the slide.
    Are you sure you want to
    Your message goes here
    Processing…
  • Very good discovering.

    John.
    www.freeringtones.ws/
    Are you sure you want to
    Your message goes here
    Processing…
  • Exceptional slideshow. Very clear and helpful

    Janie
    http://financejedi.com
    http://healthjedi.com
    Are you sure you want to
    Your message goes here
    Processing…
  • We support arbitrary HTML embed in the compose screen (beta feature), so we can just embed any slideshare slides on Vox, like seen on my blog: http://bulknews.vox.com/
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    How we build Vox How we build Vox Presentation Transcript

    • How we Build Vox April 4 2007
    • Six Apart
      • Movable Type
      • TypePad
      • LiveJournal
      • Vox
    • How we Build Vox: a Web 2.0, Large-scale, Fast, Internationalized website
    • How we Build Vox: a Web 2.0 , Large-scale, Fast, Internationalized website
    • Web 2.0
      • Overused…
      • But useful
    • Vox talks to web services
    • APIs: Tools
      • We use our own custom libraries
        • No Net::Amazon, Net::Flickr, etc
        • Why?
        • We don’t want to load 7 XML parsers
        • All of our tools use XML::LibXML
    • APIs: Open Media Profile
      • <entry>
      • <title>Foo bar baz</title>
      • <link href=&quot;http://example.com/show/video/123&quot; />
      • <link rel=&quot;alternate&quot; type=&quot;application/atom+xml&quot;
      • href=&quot;http://example.com/atom/123&quot; />
      • <id>tag:example.com,2006:video-123</id>
      • <updated>2003-12-13T18:30:02Z</updated>
      • <content type=&quot;text&quot;>
      • Vox rocks blah blah blah ...
      • </content>
      • <category term=&quot;Vox&quot; scheme=&quot;http://example.com/tags/Vox/&quot; label=&quot;Vox&quot; />
      • <category term=&quot;cat&quot; scheme=&quot;http://example.com/tags/Cat/&quot; label=&quot;cat&quot; />
      • <link rel=&quot;license&quot; type=&quot;text/html&quot;
      • href=&quot;http://creativecommons.org/licenses/by/2.5/&quot; />
      • <media:content url=&quot;http://example.com/data/123.flv&quot; fileSize=&quot;123456&quot;
      • type=&quot;video/x-flv&quot; />
      • <media:player url=&quot;http://example.com/data/123.swf&quot; height=&quot;200&quot; width=&quot;400&quot; />
      • <media:thumbnail url=&quot;http://example.com/thumb/1223.jpg&quot; width=&quot;75&quot; height=&quot;50&quot; />
      • </entry>
    • GData, OpenSearch, Media RSS… GData OpenSearch Media RSS
    • Open Media Profile GData OpenSearch Media RSS
    • APIs: Outbound
      • Atom Publishing Protocol
      • Everything is Atom/RSS
      • Cool URIs
        • /library/posts/atom.xml
        • /library/posts/2007/03/atom.xml
        • /library/posts/2007/03/tags/yapc/atom.xml
    • Ajax
      • JSON serialization
        • Lightweight
        • Normal data types (no need to invent syntax)
      • Catalyst + JSON-RPC
      • Everything is an API
      • Our own core JS libraries
      • http://search.cpan.org/~miyagawa/Catalyst-Plugin-JSONRPC-0.01/
      • http://code.sixapart.com/svn/js/trunk/
    • How we Build Vox: a Web 2.0, Large-scale , Fast, Internationalized website
    • Large-scale
      • We started with this:
    • We added some stuff.
    • Data::ObjectDriver
      • Movable Type and TypePad: custom ORM
      • We wanted more:
        • Built-in caching
        • Built-in partitioning
    • Data::ObjectDriver: Caching
      • Built-in support for memcached
      • All primary key data maintained for you
      • Completely automatic
    • One line of code (basically):
      • Data::ObjectDriver::Driver::Cache::Memcached->new(
      • cache => Cache::Memcached->new({ servers => [ ... ] }),
      • fallback => Data::ObjectDriver::Driver::DBI->new(
      • dsn => 'dbi:SQLite:dbname=global.db',
      • ),
    • Data::ObjectDriver: Partitioning
      • Sharded data
      • Based on arbitrary criteria
      • Completely transparent
    • One line of code:
      • Data::ObjectDriver::Driver::Cache::Cache->new(
      • cache => Cache::Memcached->new({ servers => [ ... ] }),
      • fallback => Data::ObjectDriver::Driver::SimplePartition->new(
      • using => 'Recipe',
      • ),
    • Partitioning: traffic
    • Example: loading user’s posts
      • my $user = ArcheType::M::User->lookup_by_email(
      • 'ben@sixapart.com’
      • );
      • my @assets = $user->assets({ type => 'Post' });
    • Example
      • Loading $user hits the global
      • Loading @assets then does:
        • Get the partition number of $user
        • Connect to that partition
        • Runs a query like:
      • SELECT user_id, asset_id
      • FROM asset
      • WHERE user_id = ?
      • AND type = 6
    • ID Allocation: Issues
      • Partitioned databases -> no more auto_increment
      • Master/master
      • Were UUIDs the answer? No.
    • ID Allocation: yuidd
      • IDs unique a datacenter
      • 64-bit integers (fit in a BIGINT column)
      • yuidd is the server
      • Data::YUID::Client is the client
      • asynchronous, non-blocking, simple, fast
    • Job Queueing
      • Offload processing from Apache
      • It’s big and heavy
    • Job Queueing: TheSchwartz
      • We’ll probably rename it.
      • asynchronous, reliable job queue
      • N databases
      • Pool of workers to handle the jobs
    • How we Build Vox: a Web 2.0, Large-scale, Fast , Internationalized website
    • Fast
      • Need both large-scale and fast
    • Catalyst
      • Vox uses Catalyst
      • Does what we want, allows us to do everything else
      • Want to use our own ORM, etc
    • Is Catalyst fast?
      • A common question on the mailing list!
      • It’s fast enough (more on that later).
    • Template Toolkit
      • Pretty fast…
      • But we’re probably overloading it.
    • Template Toolkit: profile
      • [info] Request took 0.244932s (4.083/s)
      • .----------------------------------------------------------------+-----------.
      • | Action | Time |
      • +----------------------------------------------------------------+-----------+
      • | /auto | 0.005569s |
      • | -> /set_locale | 0.000854s |
      • | -> /set_locale | 0.000648s |
      • | /home/root | 0.072194s |
      • | -> /home/home_loggedout | 0.071337s |
      • | -> /home/load_thisisgoods | 0.009077s |
      • | -> /home/load_specials | 0.014897s |
      • | -> /home/load_featured_voxers | 0.044168s |
      • | /end | 0.143675s |
      • | -> Vox::App::V::TT->process | 0.140877s |
      • '----------------------------------------------------------------+-----------'
    • Template Toolkit: profile
      • Wow! Template Toolkit takes 60% of the request time.
      • 4 times as long as 10-15 network requests.
      • Oh well.
    • Template Toolkit: versioned caching
      • On-disk cache
      • Versioned with application version
      • Automatic cache bust
    • Versioned caching
      • Template->new({
      • ...,
      • COMPILE_DIR => '/tmp/tt-cache-' . Vox->VERSION,
      • });
    • Template Toolkit: syscalls
      • Lots of syscalls for files that don’t exist!
      • But we patched it.
    • Caching
      • Data caching
      • Automatic caching using Data::ObjectDriver
      • Saves millions of lookups per day from reaching the database
    • Caching: lists
      • Lists of things: tags on an asset.
      • Tag objects are automatically cached
      • Cache asset => list of tag IDs
    • Like this:
      • asset<assetid>-tags => [ <tagid1>, <tagid2>, … ]
    • Caching: lists
      • Grab list of tag IDs from memcached
      • Use get_multi to get back the tags
    • Get back the tag objects:
      • get_multi <tagid1> <tagid2> …
    • That’s not all! In bulk:
      • get_multi asset<assetid1>-tags asset<assetid2>-tags …
    • Caching: lists
      • In a database: N one-to-many queries
      • In memcached: 2 queries
      • Use the right caching strategy
    • Perlbal
      • Reverse-proxy setup
      • Like Apache 2/mod_proxy in front of mod_perl…
      • But much better!
    • Perlbal: webserver mode
      • Serves CSS, images, JavaScript
      • Static stuff
      • Really fast
    • Perlbal: Serving JS and CSS
      • We use a lot of JS and CSS!
      • 20 JS files per page, 10 CSS files per page
      • SLOW
    • Perlbal: Serving JS and CSS
      • Added file concatenation support in a plugin
      • (it’s now core)
    • Used to be this:
      • <script type=&quot;text/javascript&quot; src=&quot;/.shared:v25.3:vox:en/js/Core.jsc&quot;></script>
      • <script type=&quot;text/javascript&quot; src=&quot;/.shared:v25.3:vox:en/js/DOM.jsc&quot;></script>
      • <script type=&quot;text/javascript&quot; src=&quot;/.shared:v25.3:vox:en/js/DOM/Proxy.jsc&quot;></script>
      • <script type=&quot;text/javascript&quot; src=&quot;/.shared:v25.3:vox:en/js/JSON.jsc&quot;></script>
      • <script type=&quot;text/javascript&quot; src=&quot;/.shared:v25.3:vox:en/js/Timer.jsc&quot;></script>
      • <script type=&quot;text/javascript&quot; src=&quot;/.shared:v25.3:vox:en/js/Observer.jsc&quot;></script>
      • <script type=&quot;text/javascript&quot; src=&quot;/.shared:v25.3:vox:en/js/Cache.jsc&quot;></script>
      • <script type=&quot;text/javascript&quot; src=&quot;/.shared:v25.3:vox:en/js/Client.jsc&quot;></script>
      • <script type=&quot;text/javascript&quot; src=&quot;/.shared:v25.3:vox:en/js/Template.jsc&quot;></script>
      • ...
    • And now it’s this!
      • <script type=&quot;text/javascript&quot; src=&quot;/.shared:v25.3:vox:en/js/Core.jsc?/js/DOM.jsc,/js/DOM/Proxy.jsc,/js/JSON.jsc,/js/Timer.jsc,/js/Observer.jsc,/js/Cache.jsc,/js/Client.jsc,/js/Template.jsc,/js/Autolayout.jsc,/js/Component.jsc,/js/Dialog.jsc,/js/App.jsc,/js/List.jsc,/js/ArcheType.jsc,/js/ArcheType/Client.jsc,/js/ArcheType/Controller.jsc,/js/ArcheType/Autocomplete.jsc&quot;></script>
    • Perlbal: Concatenation
      • Much faster
      • Much less latency
      • Perlbal handles Last-Modified/If-Modified-Since
    • Lots of good stuff!
      • Tools to play with:
        • Perlbal
        • MogileFS
        • Memcached
        • TheSchwartz
        • yuidd
        • JavaScript libraries
        • etc.
    • All available at…
      • http://code.sixapart.com/