1. Scaling PHP/MySQL...Presentation from Flickr

Hardware Layouts for LAMP Installations John Allspaw, Flickr Plumbr Flickr (Yahoo) [email_address] October 18, 2005

Hardware Layouts for LAMP Installations
Hardware requirements for LAMP installs have to do with:
A decent amount about the actual hardware (“in-box” stuff)
A bit more about the hardware architecture
Which should complement the application architecture

What we’ll talk about here:
Database (MySQL) layouts and considerations
Some miscellaneous/esoteric stuff (lessons learned)
Caching content and considerations

Growing Up, “One Box” solution
Basic web application (discussion board, etc.)
Low traffic
Apache/PHP/MySQL on one machine
Bottlenecks will start showing up:
Most likely database before apache/php
Disk I/O (Innodb) or locking wait states (MyISAM)
Context switching between memory work (apache) and CPU work (MySQL)

Hardware Layouts for LAMP Installations ONE BOX

Growing Up, “Two Box” solution
Higher traffic application (more demand)
Apache/PHP on box A, MySQL on box B
Same network = bad (*or is it ?), separate network = good
Bottlenecks with start to be:
Disk I/O on MySQL machine (Innodb)
Locking on MyISAM tables
Network I/O

Hardware Layouts for LAMP Installations TWO BOX

Growing Up, “Many Boxes with Replication” solution
Yet even higher traffic
Writes are separated from reads (master gets IN/UP/DEL, slaves get SELECTs)
Diminishes network bottlenecks, disk I/O, and other “in-box” issues
SELECTs, IN/UP/DEL can be specified within the application,
OR….
Load-balancing can be used

Hardware Layouts for LAMP Installations MANY BOX

Slave Lag
When slaves can’t keep up with replication
They’re too busy:
Reading (production traffic)
Writing (replication)
Manifests as:
Comments/photos/any user-entered data doesn’t show up on the site right away
So users will repeat the action, thinking that it didn’t “take” the first time, makes situation worse

Hardware Layouts for LAMP Installations Insert funny photo here about slave lag* *slave lag isn’t funny

Hardware Load Balancing MySQL

How It’s Usually Done
Standard MySQL master/slave replication
All writes (inserts/updates/deletes) from application go to Master
All reads (selects) from application go to a load-balanced VIP (virtual IP) spreading out load across all slaves

What Is Good About Load Balancing
you can add/remove slaves without affecting application, since queries are atomic (sorta/kinda)
additional monitoring point and some automatic failure handling
you can treat all of your slave pool as one resource, and makes capacity planning a lot easier if you know the ceiling of each slave

How do you know the ceiling (maximum QPS capacity) of each slave ?
First make a guess based on benchmarking (or look up some bench results from Tom’s Hardware or anandtech.com, etc.
Then get more machines than that :)
Scary: in production during a lull in traffic, remove machines from the pool until you detect lag
The QPS you saw right before slave lag set in:
THAT is your ceiling

What Can Be Bad/Tough About Load Balancing:
not all load-balancers are created equal, not all load-balancing companies expect this product use, so support may still be thin
not that many people are doing it in high-volume situations yet, so support from community isn’t large either
Gotchas:
port exhaustion,
health checks,
and balance algorithms

Port Exhaustion
PROBLEM:
LB is basically a traffic cop, nothing more
Side effect of having a lot of connections: only ~64,511 ports per each IP (VIP) to use
64,511 ports/120 sec per port….
~535 max concurrent connections per IP*
* Not really, but close to it: tcp_tw_recycle and tcp_tw_reuse

Port Exhaustion (cont’d)
SOLUTION:
Use a pool of IPs on the database slave/farm side (Netscaler calls these “subnet IPs”, Alteon calls them “PiPs”)
Monitor port/connection usage, know when it’s time to add more

Health checks
LB won’t know anything about how well each MySQL slave is doing, and will pass traffic as long as port 3306 is answering
Load balancers don’t talk SQL, only things like plain old TCP, HTTP/S, maybe FTP

Health checks (cont’d)
Two options:
1. Dirty, but workable:
Have each server monitor itself, and shut off/firewall its own port 3306, even if MySQL is still running

Health checks (cont’d)
2. Cleaner, but a bit more work:
Have each server monitor itself, and run a check via xinetd (for example, a nagios monitor)
So the LB can tickle that port, and expect back an “OK” string. If not, it’ll automatically take that server out of the pool
Good for detecting and counteracting isolated incidents of ‘slave lag’ and automatically handling it

Hardware Layouts for LAMP Installations Health Checks

Balancing Algorithms
Load balancers know HTTP, FTP, basic TCP, but not SQL
Two things to care about:
Should the server still be in the pool ? (health checks)
How should load get balanced ?
“ least connections” or “least bandwidth” or “least anything” = BAD
Because not all SQL queries are created equal
Use “round-robin” or “random”
What happens if you don’t: Evil Favoritism™

Hardware Layouts for LAMP Installations Evil Favoritism

Meanwhile….for “in-the-box considerations”
Interleaving memory *does* make a difference
Always RAID10 (or RAID0 if you’re crazy*) but NEVER RAID5 (for Innodb, anyway)
RAID10 has much more read capacity, and a write penalty , but not as much as RAID5
Always have battery backup for HW RAID write caching
Or, don’t use write caching at all

“ IN-THE-BOX” considerations (cont’d)
Always have proper monitoring (nagios, etc.) for failed/rebuilding drives
SATA or SCSI ? SCSI ! It’s worth it!
10k or 15k RPM SCSI ? 15k! It’s worth it!
(~20% performance increase when you’re disk bound)
For 64bit Linux (AMD64 or EM64T):
Crank up the RAM for Innodb’s buffer pool
Swapping = very very bad either:
Turn it off (slightly scary)
Leave it on and set /proc/sys/vm/swapiness = 0

10k versus 15k drives ?
Does it really matter that much ?
Some in-the-wild proof….

Hardware Layouts for LAMP Installations 10K drives 15K drives Slave Lag in production

Using MySQL with a SAN (Storage Area Network)
Do layout storage same as if they would be local
Do make sure that the HBA (fiber card) driver is well supported by Linux
Don’t share volumes across databases
Don’t forget to correctly tune Queue Depth Size, which should be increasing, from server HBA -> switch -> storage

Caching your static content

Caching Static Content
SQUID = good
Relieve your front-end PHP machines from looking up data that will never (or rarely) change
Generate static pages, and cache them in squid, along with your images

Caching Static Content (cont’d)
Use SQUID to accelerate plain-old origin webservers, also known as “reverse-proxy” HTTP acceleration
Described here and elsewhere:
http://www.squid-cache.org/Doc/FAQ/FAQ-20.html

Hardware Layouts for LAMP Installations Basic SQUID layout
squid accepts requests on 80
passes on cache misses to apache on 81
apache uses as its docroot an NFS mounted dir
should be on local subnet, or dedicated net

Good HW layout for high-volume SQUIDing
Do use SCSI, and many spindles for disk cache dirs
Don’t use RAID
Do use network attached storage, or place the origin servers on separate machines
Do use ext3 with noatime for disk cache dirs
Do monitor squid stats

Hardware Layouts for LAMP Installations Flickr: How We Roll

Yummy SQUID stats:
>2800 images/sec, ~75-80% are cache hits
~10 million photos cached at any time
1.5 million cached in memory

The End

http://www.simpleentrepreneur.com	980
http://poorbuthappy.com	278
http://www.slideshare.net	80
http://thuannvn.blogspot.com	39
http://www.poorbuthappy.com	18
http://kaloyan.info	16
http://www.netvibes.com	14
http://www.schoox.com	13
http://nomadz.wordpress.com	8
http://www.lonerunners.net	6
http://blog.techno.taladnam.com	5
http://webhome.broward.edu	3
http://s3.amazonaws.com	3
http://www.hlibco.com.br	3
http://dominiquelacan.blogspot.com	2
http://localhost	2
http://hellogerard.blogspot.com	2
http://www.filescon.com	2
http://static.slideshare.net	2
http://www.tekevo.net	2
http://saylinux.net	1
http://webcache.googleusercontent.com	1
http://c17-cne-admin-bld1.cnet.com:8081	1
http://91.121.91.10	1
http://209.85.229.132	1
http://jhpatel.blogspot.com	1
http://feeds.feedburner.com	1
https://s3.amazonaws.com	1
https://bluekiwi.groupereflect.net	1
http://www.oplahol.com	1
http://livebuzz.com.br	1

1. Scaling PHP/MySQL...Presentation from Flickr

by akshat on Mar 29, 2007

Accessibility

Categories

Tags

Upload Details

Usage Rights

31 Embeds 1,489

Statistics

1. Scaling PHP/MySQL...Presentation from Flickr — Presentation Transcript