How we build Vox

How we Build Vox April 4 2007

Six Apart

Movable Type

TypePad

LiveJournal

Vox

How we Build Vox: a Web 2.0, Large-scale, Fast, Internationalized website

How we Build Vox: a Web 2.0 , Large-scale, Fast, Internationalized website

Web 2.0

Overused…

But useful

Vox talks to web services

APIs: Tools

We use our own custom libraries

No Net::Amazon, Net::Flickr, etc

Why?

We don’t want to load 7 XML parsers

All of our tools use XML::LibXML

APIs: Open Media Profile

<entry>

<title>Foo bar baz</title>

<link href="http://example.com/show/video/123" />

<link rel="alternate" type="application/atom+xml"

href="http://example.com/atom/123" />

<id>tag:example.com,2006:video-123</id>

<updated>2003-12-13T18:30:02Z</updated>

<content type="text">

Vox rocks blah blah blah ...

</content>

<category term="Vox" scheme="http://example.com/tags/Vox/" label="Vox" />

<category term="cat" scheme="http://example.com/tags/Cat/" label="cat" />

<link rel="license" type="text/html"

href="http://creativecommons.org/licenses/by/2.5/" />

<media:content url="http://example.com/data/123.flv" fileSize="123456"

type="video/x-flv" />

<media:player url="http://example.com/data/123.swf" height="200" width="400" />

<media:thumbnail url="http://example.com/thumb/1223.jpg" width="75" height="50" />

</entry>

GData, OpenSearch, Media RSS… GData OpenSearch Media RSS

Open Media Profile GData OpenSearch Media RSS

APIs: Outbound

Atom Publishing Protocol

Everything is Atom/RSS

Cool URIs

/library/posts/atom.xml

/library/posts/2007/03/atom.xml

/library/posts/2007/03/tags/yapc/atom.xml

Ajax

JSON serialization

Lightweight

Normal data types (no need to invent syntax)

Catalyst + JSON-RPC

Everything is an API

Our own core JS libraries

http://search.cpan.org/~miyagawa/Catalyst-Plugin-JSONRPC-0.01/

http://code.sixapart.com/svn/js/trunk/

How we Build Vox: a Web 2.0, Large-scale , Fast, Internationalized website

Large-scale

We started with this:

We added some stuff.

Data::ObjectDriver

Movable Type and TypePad: custom ORM

We wanted more:

Built-in caching

Built-in partitioning

Data::ObjectDriver: Caching

Built-in support for memcached

All primary key data maintained for you

Completely automatic

One line of code (basically):

Data::ObjectDriver::Driver::Cache::Memcached->new(

cache => Cache::Memcached->new({ servers => [ ... ] }),

fallback => Data::ObjectDriver::Driver::DBI->new(

dsn => 'dbi:SQLite:dbname=global.db',

),

Data::ObjectDriver: Partitioning

Sharded data

Based on arbitrary criteria

Completely transparent

One line of code:

Data::ObjectDriver::Driver::Cache::Cache->new(

cache => Cache::Memcached->new({ servers => [ ... ] }),

fallback => Data::ObjectDriver::Driver::SimplePartition->new(

using => 'Recipe',

),

Partitioning: traffic

Example: loading user’s posts

my $user = ArcheType::M::User->lookup_by_email(

'ben@sixapart.com’

);

my @assets = $user->assets({ type => 'Post' });

Example

Loading $user hits the global

Loading @assets then does:

Get the partition number of $user

Connect to that partition

Runs a query like:

SELECT user_id, asset_id

FROM asset

WHERE user_id = ?

AND type = 6

ID Allocation: Issues

Partitioned databases -> no more auto_increment

Master/master

Were UUIDs the answer? No.

ID Allocation: yuidd

IDs unique a datacenter

64-bit integers (fit in a BIGINT column)

yuidd is the server

Data::YUID::Client is the client

asynchronous, non-blocking, simple, fast

Job Queueing

Offload processing from Apache

It’s big and heavy

Job Queueing: TheSchwartz

We’ll probably rename it.

asynchronous, reliable job queue

N databases

Pool of workers to handle the jobs

How we Build Vox: a Web 2.0, Large-scale, Fast , Internationalized website

Fast

Need both large-scale and fast

Catalyst

Vox uses Catalyst

Does what we want, allows us to do everything else

Want to use our own ORM, etc

Is Catalyst fast?

A common question on the mailing list!

It’s fast enough (more on that later).

Template Toolkit

Pretty fast…

But we’re probably overloading it.

Template Toolkit: profile

[info] Request took 0.244932s (4.083/s)

.----------------------------------------------------------------+-----------.

| Action | Time |

+----------------------------------------------------------------+-----------+

| /auto | 0.005569s |

| -> /set_locale | 0.000854s |

| -> /set_locale | 0.000648s |

| /home/root | 0.072194s |

| -> /home/home_loggedout | 0.071337s |

| -> /home/load_thisisgoods | 0.009077s |

| -> /home/load_specials | 0.014897s |

| -> /home/load_featured_voxers | 0.044168s |

| /end | 0.143675s |

| -> Vox::App::V::TT->process | 0.140877s |

'----------------------------------------------------------------+-----------'

Template Toolkit: profile

Wow! Template Toolkit takes 60% of the request time.

4 times as long as 10-15 network requests.

Oh well.

Template Toolkit: versioned caching

On-disk cache

Versioned with application version

Automatic cache bust

Versioned caching

Template->new({

...,

COMPILE_DIR => '/tmp/tt-cache-' . Vox->VERSION,

});

Template Toolkit: syscalls

Lots of syscalls for files that don’t exist!

But we patched it.

Caching

Data caching

Automatic caching using Data::ObjectDriver

Saves millions of lookups per day from reaching the database

Caching: lists

Lists of things: tags on an asset.

Tag objects are automatically cached

Cache asset => list of tag IDs

Like this:

asset<assetid>-tags => [ <tagid1>, <tagid2>, … ]

Caching: lists

Grab list of tag IDs from memcached

Use get_multi to get back the tags

Get back the tag objects:

get_multi <tagid1> <tagid2> …

That’s not all! In bulk:

get_multi asset<assetid1>-tags asset<assetid2>-tags …

Caching: lists

In a database: N one-to-many queries

In memcached: 2 queries

Use the right caching strategy

Perlbal

Reverse-proxy setup

Like Apache 2/mod_proxy in front of mod_perl…

But much better!

Perlbal: webserver mode

Serves CSS, images, JavaScript

Static stuff

Really fast

Perlbal: Serving JS and CSS

We use a lot of JS and CSS!

20 JS files per page, 10 CSS files per page

SLOW

Perlbal: Serving JS and CSS

Added file concatenation support in a plugin

(it’s now core)

Used to be this:

<script type="text/javascript" src="/.shared:v25.3:vox:en/js/Core.jsc"></script>

<script type="text/javascript" src="/.shared:v25.3:vox:en/js/DOM.jsc"></script>

<script type="text/javascript" src="/.shared:v25.3:vox:en/js/DOM/Proxy.jsc"></script>

<script type="text/javascript" src="/.shared:v25.3:vox:en/js/JSON.jsc"></script>

<script type="text/javascript" src="/.shared:v25.3:vox:en/js/Timer.jsc"></script>

<script type="text/javascript" src="/.shared:v25.3:vox:en/js/Observer.jsc"></script>

<script type="text/javascript" src="/.shared:v25.3:vox:en/js/Cache.jsc"></script>

<script type="text/javascript" src="/.shared:v25.3:vox:en/js/Client.jsc"></script>

<script type="text/javascript" src="/.shared:v25.3:vox:en/js/Template.jsc"></script>

...

And now it’s this!

<script type="text/javascript" src="/.shared:v25.3:vox:en/js/Core.jsc?/js/DOM.jsc,/js/DOM/Proxy.jsc,/js/JSON.jsc,/js/Timer.jsc,/js/Observer.jsc,/js/Cache.jsc,/js/Client.jsc,/js/Template.jsc,/js/Autolayout.jsc,/js/Component.jsc,/js/Dialog.jsc,/js/App.jsc,/js/List.jsc,/js/ArcheType.jsc,/js/ArcheType/Client.jsc,/js/ArcheType/Controller.jsc,/js/ArcheType/Autocomplete.jsc"></script>

Perlbal: Concatenation