Large platform architecture in (mostly) perlPresentation Transcript
architecture in (mostly)
perl - an illustrated
Tomas (t0m) Doran
São Paulo.pm perl workshop 2010
YAPC::EU Pisa 2010
• Is mostly a ramble
• About what I do for a living
• Good bits
• and bad bits (probably mostly bad bits)
• And when I say ‘illustrated’, I’m not very
good at diagrams, sorry...
Making money from
• No, no it isn’t. But we’re very lucky to have
people who know the music industry
• A startup would tank
• Last.fm guys “keep losing less money”
The state51 conspiracy
Media Service Provider
• Several (largely proﬁtable) businesses based
on the same technology platform
• East London (Brick Lane), a warehouse.
• > 60% of UK independent content goes
through us somewhere
Being S3 on the cheap
• WAV ﬁles are big.Videos are bigger.
• Transcodes aren’t small, especially when
you have 15 of them.
• My music collection is several hundred
• Need to be able to serve this stuff fast and
• Is free.
• Runs on cheap hardware
• Cheaper then S3.
• Not so awesome if you aren’t Livejournal
Data center design
• 8 amp racks. Seriously, WTF!?!?!
• Electricity is more expensive than servers,
ergo rolling hardware upgrades trivially pay
• Transit is really, really expensive.
• Worth buying ﬁber to other locations to
peer if you need lots of bandwidth.
• App servers apache, apps FastCGI, port 81
• Varnish + ESI, caching, port 80
• 1 varnish per host, talks to all the apaches
• 1 VIP per host
• Host fail:VIP transfer
• Apache/app fail (or overload), varnish
Web architecture (cont)
• Varnish doesn’t cache media, just provides
• nginx sends the hit to FastCGI app.
• Returns X-Accel-Redirect.
• nginx talks to MogileFS, handles delivery.
• Lots of boxes with lots of disk.
• Many additional roles to storage. (Mogile
tracker, memcache node, metal encoding,
VMWare, SOAP Service)
• Not all the boxes do all the roles.
• All the roles can safely fall over and die.
• Which is good, as they do. Or the box falls
over. Or a, then b.
• WAV is a container format.
• Loosely deﬁned.
• You can stuff XML documents in WAV ﬁles
• Some encoders (oh hai ﬂac) very picky.
• ‘dirty’ and ‘clean’ WAV ﬁles.
• Lots of different formats
• WMA - GNARGGH%$@*&!!
• We’re running ActiveState for hysterical
• No XS modules
• Thin as possible
HTTP Nodes Encoding Service Uploading Service
Local Disk Encoder
Snakes On A Plane
• SOAP actually works ok here, as we
control both ends.
• Old version of SOAP::Lite
• Wouldn’t recommend interoperating
• Used to be terribly hard to debug
• Push logs into syslog
• Aggregate in splunk - time correlated from
encoding machines, web service machines,
• Much easier to work out what happened.
Hardware is shit
• When you have several 100 Tb, undetected
bit error rate of magnetic media is actually
• See also networks, memory, etc.
Things will always fail
• If you need reliability, you have to design it
in from the start.
• Not only will you have (a lot of) hardware
failures, all the software will break in
unexpected ways. Lets not talk about
• Maybe you don’t need this..
• We have work queues of different types of
media (e.g. mp3/wma/aac etc)
• In the database.
• Don’t do this.
• 1 type of JOIN
• No query rewriting
• Not enough stats for the planner to be
This can hurt
• File Transform table:
• Master (File)
• Result (File)
• Status (pending/complete/failed/running)
• TransformStep (from/to)
• Leads to bad join order, massive fail
How to fail
• SELECT all ﬁle transforms that lead to wma
• JOIN all ﬁles, ever (millions). Filter to ﬁnd
those in state ‘pending’
• All pending looks like a bad bet - cardinality
of ‘all wmas’ looks better than cardinality of
• JOIN in the wrong order, nested loop,
• Did I mention queues in the DB suck?
• Even if you’re not screwing it up.
• Get a Message Queue (or at least an async
• If your problem is simple - Gearman.
Harder or you need interop - RabbitMQ.
• Mutable state is the enemy
• Too many things rw.
• No idea how an object got to this state
Anemic domain model
Object-oriented programming (OOP) is a
programming paradigm that uses "objects" –
data structures consisting of data ﬁelds and
methods together with their
interactions – to design applications and
computer programs. Programming techniques
may include features such as data
abstraction, encapsulation, modularity,
polymorphism, and inheritance.
Anemic domain model
• Superset of too much mutable state
• Able to create invalid objects
• Able to make previously valid objects
• Violation of the encapsulation and
information hiding principles.
• Lots of our business logic was in scripts
that manipulated objects
• You need people to run scripts (in screen
• Ewwww, ewwwww.
• Moved to a job based approach
• Jobs started by ﬁle creation, or changing
state of something in a web app
• Jobs sent via message queuing.
• Results go via message queueing
• Jobs trigger other jobs
• Validate XLS ﬁle supplied with order.
• Valid ﬁles trigger another job to create
objects for each thing in the XLS
• This then triggers another job to create
transforms, which are then done...
• ... etc ...
• Can’t do this workﬂow in a web request.
• More automation of things people run
• Automatic job regeneration (you will lose
• Old (possibly unclean/invalid) data
• Old (unused/unmaintained) code
• “What harm does it do”
• Seems to be a pipe dream more often then
not in the real world.
• It’s not hard
• This should theoretically be the same thing
as relational integrity.
• In practice...
Mumble View Crap
• Too much logic in templates
• Copy & paste
• Business objects viewed as unchangeable
• Deleted 3000 lines from 2 simple
workﬂows. This ﬁxed a dozen bugs.
• No LEFT JOIN
• Displaying a product list becomes an x n
• Keep stupid - put the entire DB hot in
Don’t do web design
• You are a programmer
• Make people pay for a design/CSS/HTML
• Work with them
• Be happy
Love your sysadmins
• Help them out.
• Build packages, or local::libs or something
• Keep everything in revision control
• Allow things to be sensibly conﬁgured.
• DOCUMENT THE POSSIBLE SETTINGS
• Use systems management - Puppet?
Love your logs
• Active feedback
• Aggregate in splunk
• Actively prune useless stuff
• Actively add useful stuff after a production
• Is really awesome
• Make the pain go away
• PURGE requests
• Keep everything hot all the time
• Keep the entire database hot in memcache
• We mostly ask trivial questions, so just
cache those paths.
• 30 Gb of RAM isn’t actually much (3
• IS A CACHE
• Use sequential port numbers and CNAMES
• E.g. cache0:11210, cache1:11211,
• Run several per machine
• Allows you to scale capacity and rebalance
without entire cache ﬂush.
Don’t push bytes
• X-Sendﬁle and X-Accel-Redirect
• I already talked about ﬁle delivery like this
• Using 100Mb of RAM to proxy web
requests does not scale.
• Redundant systems need testing
• You’ll still die unexpectedly in production
• If you can manage it, make responsibility for
• Thanks for listening
Let LinkedIn power your SlideShare experience
Let LinkedIn power your SlideShare experience
Customize SlideShare content based on your interests
We will import your LinkedIn profile and you will be visible on SlideShare.
Keep up to date when your LinkedIn contacts post on SlideShare