Scaling Early

Scaling an early stage startup by Mark Maunder <mark@feedjit.com>

Why does performance and scaling quickly matter?
Slow performance could cost you 20% of your revenue according to Google.
Any reduction in hosting costs goes directly to your bottom line as profit or can accelerate growth.
In a viral business, slow performance can damage your viral growth.

My first missteps
Misconfiguration. Web server and DB configured to grab too much RAM.
As traffic builds, the server swaps and slows down drastically.
Easy to fix – just a quick config change on web server and/or DB.

Traffic at this stage
2 Widgets per second
10 HTTP requests per second.
1 Widget = 1 Pageview
We serve as many pages as our users do, combined.

Keepalive – Good for clients, bad for servers.
As http requests increased to 10 per second, I ran out of server threads to handle connections.
Keepalive was on and Keepalive Timeout was set to 300.
Turned Keepalive off.

20 HTTP requests per second

Cache as much DB data as possible
I used Perl’s Cache::FileCache to cache either DB data or rendered HTML on disk.
MemCacheD, developed for LiveJournal, caches across servers.
YMMV – How dynamic is your data?

MySQL not fast enough
High number of writes & deletes on a large single table caused severe slowness.
Writes blow away the query cache.
MySQL doesn’t support a large number of small tables (over 10,000).
MySQL is memory hungry if you want to cache large indexes.
I maxed out at about 200 concurrent read/write queries per second with over 1 million records (and that’s not large enough).

Perl’s Tie::File to the early rescue
Tie::File is a very simple flat-file API.
Lots of files/tables.
Faster – 500 to 1000 concurrent read/writes per second.
Prepending requires reading and rewriting the whole file.

BerkeleyDB is very very fast!
I’m also experimenting with BerkeleyDB for some small intensive tasks.
Data From Oracle who owns BDB: Just over 90,000 transactional writes per second.
Over 1 Million non-transactional writes per second in memory.
Oracle’s machine: Linux on an AMD Athlon™ 64 processor 3200+ at 1GHz system with 1GB of RAM. 7200RPM Drive with 8MB cache RAM.
Source: http://www.oracle.com/technology/products/berkeley-db/pdf/berkeley-db-perf.pdf

35 HTTP requests per second

Created a separate image and CSS server
Enabled Keepalive on the Image server to be nice to clients.
Static content requires very little memory per thread/process.
Kept Keepalive off on the App server to reduce memory.
Added benefit of higher browser concurrency with 2 hostnames.
Source: http://www.die.net/musings/page_load_time/

Now using Home Grown Fixed Length Records
A lot like ISAM or MyISAM
Fixed length records mean we seek directly to the data. No more file slurping.
Sequential records mean sequential reads which are fast.
Still using file level locking.
Benchmarked at 20,000+ concurrent reads/writes/deletes.

50 to 60 HTTP requests per second
Load average spiking to 12 or more about 3 times per day for unknown reason.

Blocking Content Thieves
Content thieves were aggressively crawling our site on pages that are CPU intensive.
Robots.txt is irrelevant.
Reverse DNS lookup with ‘dig –x’
Firewall the &^%$@’s with ‘iptables’

Moved to httpd.prefork
Httpd.worker consumes more memory than prefork because worker doesn’t share memory.
Tuning the number of Perl interpreters vs number of threads didn’t improve things.
Prefork with no keepalive on the app server uses less RAM and works well – for Mod_Perl.

The amazing Linux Filesystem Cache
Linux uses spare memory to cache files on disk.
Lots of spare memory == Much faster I/O.
Prefork freed lots of memory. 1.3 Gigs out of 2 Gigs is used as cache.
I’ve noticed a roughly 20% performance increase since using it.

Tools
httperf for benchmarking your server
Websitepulse.com for perf monitoring.

Summary
Make content as static as possible.
Cache as much of your dynamic content as possible.
Separate serving app requests and serving static content.
Don’t underestimate the speed of lightweight file access API’s .
Only serve real users and search engines you care about.

http://www.royans.net	1224
http://ionut-tudor.blogspot.com	90
http://www.ionuttudor.com	30
http://www.zhuaxia.com	4
http://www.ionut-tudor.blogspot.com	2
http://safe.tumblr.com	2
http://www.netvibes.com	2
http://web.archive.org	1
http://74.125.77.132	1
http://www.royans.net.sixxs.org	1
http://409frank.tumblr.com	1
http://209.85.173.104	1
http://209.85.165.104	1
http://royans.net	1
http://mschuette.name	1
http://stage.feedeachother.com	1
http://64.233.183.104	1
http://0.0.0.0	1

Scaling Early

by vishnu

Accessibility

Categories

Upload Details

Usage Rights

18 Embeds 1,365

Statistics

Scaling Early Presentation Transcript