Experience in Building Scalable Web Sites
through Infrastructure's Aspect
Barcamp Bangkok 3
May 23-24, 2009
Sripathum University

Gen#1
Web + DB
Cost effective for startup

Gen#2
Web
DB
After boom!!

Gen#3
Web Web Web
DB
More boom!!, add more web servers but DB is the
problem

Gen#4
Web Web Web
WRITE Query
DB
Replication
READ ONLY
DB
Replication could save your life for a while

Gen#...
Reverse Proxy
Web
Reverse Proxy
Web
Memcached WRITE Query
Memcached DB
Replication
READ ONLY
DB
Memcached: reduce READ query at DB
Reverse Proxy: serves static contents

Finally: Horizontal Scaling
Reverse Proxy
LB Storage
Reverse Proxy
Memcached Global Database - Multimaster
Web (dynamic) masterA masterB
LB
Replication
Web (dynamic)
Slave 1 Slave 2
UserDB1 UserDB2 UserDB3 UserDB..n

Behind the Scene
Hardware (LB, Server)
Software architecture
OS
Web Server
Caching
DB
Infra: dns, network, storage, firewall, security process, log,
backup, ntp, logging, source control, monitoring, ...

Good to Know
80% of the end-user response
time is spent on the front-end
Ref: developer.yahoo.com

YAHOO's “Best Practices for Speeding Up Your
Web Site”
Minimize HTTP requests
Enable GZIP
Add “Cache-Control” HTTP header
Put stylesheet at the top
Put scripts at the bottom
............... more at developer.yahoo.com

TIPS
Enable “keep-alive” at all termination hops
ISP Reverse Web
LB
(may do caching) Proxy Server

TIPS
Split STATIC contents server
Set up new top level domain to reduce cookie in
HTTP header

TIPS
No more cookie in HTTP request

Lot of Things to find out
Hardware (LB, Server)
Software architecture
OS
Web Server
Caching
DB
Infra: dns, network, storage, firewall, security process, log,
backup, ntp, logging, source control, monitoring, ...

Hardware: Server
Any class can be used (blade, rack, local brand)
DB Server: Raid 1+0, SAS, SCSI, FC (15K)
Prefer 64 bits
Battery-backed write cache could improve disk I/O
performance
We have HP BL20p, BL25p, BL460c, DL380 G3 G5,
DL360 G3 G5 … Dell PE, Dell R300
Out of band management is required

Hardware: Load Balancer
Evaluate first (L4 - L7)
Foundry ServerIron GTE, G4 (now Brocade)
We tried,
Citrix Netscalr
Radware
Modern box comes with HTTP accelerate
Crescendo

OS.
64 Bits
Choose any os. that you can fully control it
Monitor your servers
CPU, load average, mem, disk, network

Web Server
Apache 2.2
Still use prefork
Comment unused modules
Hardening php.ini
Lighttpd, Nginx are ok
Be careful about FastCGI
IIS for some applications

Web Server (cont.)
Keep monitor

Database
MySQL 5.x
Use memory table for unimportant data
MyISAM, table locking
InnoDB, difficult to backup
PostgreSQL is coming
Oracle just acquired SUN (April 2009)
Enterprise features

Database (cont.)
Examine all queries (if you can)
Index hitting for SELECT ?
Query monitoring

Database (cont.)
Avoid full text searching
Use Sphinx
Use Solr
Use EXPLAIN

Caching
Memcached (32GB/server)

Caching
Reverse Proxy
Squid (designed as Forward proxy)
Apache Mod_Proxy (never tried)
Varnish (good response time, only UTF8)

Storage
NAS is painful for scalable engineer
SAN is not for sharing across servers
Solutions
Put the scripts at local disk (rsync)
Put only shared folder at NAS
Benchmark by iozone
Caching Caching & Caching

Clustering Storage [1]
Lustre:
for HPC
Very good performance
No redundant
Gluster
Mountable in userspace
Slow
Painful for operation

Clustering Storage [2]
MogileFS:
good performance
Build on commodity hardware
Acces via API (php, python, perl)
Hadoop
Includes Map/Reduce, HDFS, HBase
May suitable for logging application
Good for batch processing

Security
Hardening at all components
Perform security audit if you can
DNS blacklist monitoring
Try network scan for ports changed
Default: DENY policy

KB
Train the new programmer
Never knew about indexing
Never used Memcache
Never knew reverse proxy
Internal KB (recommend MediaWiki, TWIKI)

Other
Use version control
Find “one click deployment system”
All servers must sync their clock
Centralize log server + Analyzer
Backup & restore
Perform URL monitoring from outside & inside

Try ITIL V3 to improve the quality of IT services

Twitter: @AuntiSpam