How we build Vox

How we Build Vox April 4 2007

Six Apart
Movable Type
TypePad
LiveJournal
Vox

How we Build Vox: a Web 2.0, Large-scale, Fast, Internationalized website

How we Build Vox: a Web 2.0 , Large-scale, Fast, Internationalized website

Web 2.0
Overused…
But useful

Vox talks to web services

APIs: Tools
We use our own custom libraries
No Net::Amazon, Net::Flickr, etc
Why?
We don’t want to load 7 XML parsers
All of our tools use XML::LibXML

APIs: Open Media Profile
<entry>
<title>Foo bar baz</title>
<link href="http://example.com/show/video/123" />
<link rel="alternate" type="application/atom+xml"
href="http://example.com/atom/123" />
<id>tag:example.com,2006:video-123</id>
<updated>2003-12-13T18:30:02Z</updated>
<content type="text">
Vox rocks blah blah blah ...
</content>
<category term="Vox" scheme="http://example.com/tags/Vox/" label="Vox" />
<category term="cat" scheme="http://example.com/tags/Cat/" label="cat" />
<link rel="license" type="text/html"
href="http://creativecommons.org/licenses/by/2.5/" />
<media:content url="http://example.com/data/123.flv" fileSize="123456"
type="video/x-flv" />
<media:player url="http://example.com/data/123.swf" height="200" width="400" />
<media:thumbnail url="http://example.com/thumb/1223.jpg" width="75" height="50" />
</entry>

GData, OpenSearch, Media RSS… GData OpenSearch Media RSS

Open Media Profile GData OpenSearch Media RSS

APIs: Outbound
Atom Publishing Protocol
Everything is Atom/RSS
Cool URIs
/library/posts/atom.xml
/library/posts/2007/03/atom.xml
/library/posts/2007/03/tags/yapc/atom.xml

Ajax
JSON serialization
Lightweight
Normal data types (no need to invent syntax)
Catalyst + JSON-RPC
Everything is an API
Our own core JS libraries
http://search.cpan.org/~miyagawa/Catalyst-Plugin-JSONRPC-0.01/
http://code.sixapart.com/svn/js/trunk/

How we Build Vox: a Web 2.0, Large-scale , Fast, Internationalized website

Large-scale
We started with this:

We added some stuff.

Data::ObjectDriver
Movable Type and TypePad: custom ORM
We wanted more:
Built-in caching
Built-in partitioning

Data::ObjectDriver: Caching
Built-in support for memcached
All primary key data maintained for you
Completely automatic

One line of code (basically):
Data::ObjectDriver::Driver::Cache::Memcached->new(
cache => Cache::Memcached->new({ servers => [ ... ] }),
fallback => Data::ObjectDriver::Driver::DBI->new(
dsn => 'dbi:SQLite:dbname=global.db',
),

Data::ObjectDriver: Partitioning
Sharded data
Based on arbitrary criteria
Completely transparent

One line of code:
Data::ObjectDriver::Driver::Cache::Cache->new(
cache => Cache::Memcached->new({ servers => [ ... ] }),
fallback => Data::ObjectDriver::Driver::SimplePartition->new(
using => 'Recipe',
),

Partitioning: traffic

Example: loading user’s posts
my $user = ArcheType::M::User->lookup_by_email(
'ben@sixapart.com’
);
my @assets = $user->assets({ type => 'Post' });

Example
Loading $user hits the global
Loading @assets then does:
Get the partition number of $user
Connect to that partition
Runs a query like:
SELECT user_id, asset_id
FROM asset
WHERE user_id = ?
AND type = 6

ID Allocation: Issues
Partitioned databases -> no more auto_increment
Master/master
Were UUIDs the answer? No.

ID Allocation: yuidd
IDs unique a datacenter
64-bit integers (fit in a BIGINT column)
yuidd is the server
Data::YUID::Client is the client
asynchronous, non-blocking, simple, fast

Job Queueing
Offload processing from Apache
It’s big and heavy

Job Queueing: TheSchwartz
We’ll probably rename it.
asynchronous, reliable job queue
N databases
Pool of workers to handle the jobs

How we Build Vox: a Web 2.0, Large-scale, Fast , Internationalized website

Fast
Need both large-scale and fast

Catalyst
Vox uses Catalyst
Does what we want, allows us to do everything else
Want to use our own ORM, etc

Is Catalyst fast?
A common question on the mailing list!
It’s fast enough (more on that later).

Template Toolkit
Pretty fast…
But we’re probably overloading it.

Template Toolkit: profile
[info] Request took 0.244932s (4.083/s)
.----------------------------------------------------------------+-----------.
| Action | Time |
+----------------------------------------------------------------+-----------+
| /auto | 0.005569s |
| -> /set_locale | 0.000854s |
| -> /set_locale | 0.000648s |
| /home/root | 0.072194s |
| -> /home/home_loggedout | 0.071337s |
| -> /home/load_thisisgoods | 0.009077s |
| -> /home/load_specials | 0.014897s |
| -> /home/load_featured_voxers | 0.044168s |
| /end | 0.143675s |
| -> Vox::App::V::TT->process | 0.140877s |
'----------------------------------------------------------------+-----------'

Template Toolkit: profile
Wow! Template Toolkit takes 60% of the request time.
4 times as long as 10-15 network requests.
Oh well.

Template Toolkit: versioned caching
On-disk cache
Versioned with application version
Automatic cache bust

Versioned caching
Template->new({
...,
COMPILE_DIR => '/tmp/tt-cache-' . Vox->VERSION,
});

Template Toolkit: syscalls
Lots of syscalls for files that don’t exist!
But we patched it.

Caching
Data caching
Automatic caching using Data::ObjectDriver
Saves millions of lookups per day from reaching the database

Caching: lists
Lists of things: tags on an asset.
Tag objects are automatically cached
Cache asset => list of tag IDs

Like this:
asset<assetid>-tags => [ <tagid1>, <tagid2>, … ]

Caching: lists
Grab list of tag IDs from memcached
Use get_multi to get back the tags

Get back the tag objects:
get_multi <tagid1> <tagid2> …

That’s not all! In bulk:
get_multi asset<assetid1>-tags asset<assetid2>-tags …

Caching: lists
In a database: N one-to-many queries
In memcached: 2 queries
Use the right caching strategy

Perlbal
Reverse-proxy setup
Like Apache 2/mod_proxy in front of mod_perl…
But much better!

Perlbal: webserver mode
Serves CSS, images, JavaScript
Static stuff
Really fast

Perlbal: Serving JS and CSS
We use a lot of JS and CSS!
20 JS files per page, 10 CSS files per page
SLOW

Perlbal: Serving JS and CSS
Added file concatenation support in a plugin
(it’s now core)

Used to be this:
<script type="text/javascript" src="/.shared:v25.3:vox:en/js/Core.jsc"></script>
<script type="text/javascript" src="/.shared:v25.3:vox:en/js/DOM.jsc"></script>
<script type="text/javascript" src="/.shared:v25.3:vox:en/js/DOM/Proxy.jsc"></script>
<script type="text/javascript" src="/.shared:v25.3:vox:en/js/JSON.jsc"></script>
<script type="text/javascript" src="/.shared:v25.3:vox:en/js/Timer.jsc"></script>
<script type="text/javascript" src="/.shared:v25.3:vox:en/js/Observer.jsc"></script>
<script type="text/javascript" src="/.shared:v25.3:vox:en/js/Cache.jsc"></script>
<script type="text/javascript" src="/.shared:v25.3:vox:en/js/Client.jsc"></script>
<script type="text/javascript" src="/.shared:v25.3:vox:en/js/Template.jsc"></script>
...

And now it’s this!
<script type="text/javascript" src="/.shared:v25.3:vox:en/js/Core.jsc?/js/DOM.jsc,/js/DOM/Proxy.jsc,/js/JSON.jsc,/js/Timer.jsc,/js/Observer.jsc,/js/Cache.jsc,/js/Client.jsc,/js/Template.jsc,/js/Autolayout.jsc,/js/Component.jsc,/js/Dialog.jsc,/js/App.jsc,/js/List.jsc,/js/ArcheType.jsc,/js/ArcheType/Client.jsc,/js/ArcheType/Controller.jsc,/js/ArcheType/Autocomplete.jsc"></script>

Perlbal: Concatenation
Much faster
Much less latency
Perlbal handles Last-Modified/If-Modified-Since

Lots of good stuff!
Tools to play with:
Perlbal
MogileFS
Memcached
TheSchwartz
yuidd
JavaScript libraries
etc.

All available at…
http://code.sixapart.com/

http://poorbuthappy.com	220
http://www.poorbuthappy.com	15
http://www.slideshare.net	11
http://a4.vox-data.com	8
http://www.lonerunners.net	8
http://whitehalladvisory.org	2
http://a2.vox-data.com	2
http://www.filescon.com	2
http://www.oplahol.com	1
http://static.slideshare.net	1
http://209.85.173.104	1
http://a5.vox-data.com	1
http://www.thoughtbag.com	1

How we build Vox

by Tatsuhiko Miyagawa on Apr 08, 2007

Accessibility

Categories

Tags

Upload Details

Usage Rights

13 Embeds 273

Statistics

How we build Vox — Presentation Transcript