1. Scaling PHP/MySQL...Presentation from Flickr

Slide 1: October 18-21, 2005 San Francisco, CA Hardware Layouts for LAMP Installations John Allspaw, Flickr Plumbr Flickr (Yahoo) jallspaw@yahoo.com October 18, 2005 Hyatt Regency San Francisco Airport Burlingame, CA

Slide 2: Hardware Layouts for LAMP Installations Hardware requirements for LAMP installs have to do with: o A decent amount about the actual hardware (“in-box” stuff) o A bit more about the hardware architecture o Which should complement the application architecture Oct. 18, 2005 #2

Slide 3: Hardware Layouts for LAMP Installations What we’ll talk about here: o Database (MySQL) layouts and considerations o Some miscellaneous/esoteric stuff (lessons learned) o Caching content and considerations Oct. 18, 2005 #3

Slide 4: Hardware Layouts for LAMP Installations • Growing Up, “One Box” solution  Basic web application (discussion board, etc.)  Low traffic  Apache/PHP/MySQL on one machine  Bottlenecks will start showing up: • Most likely database before apache/php • Disk I/O (Innodb) or locking wait states (MyISAM) • Context switching between memory work (apache) and CPU work (MySQL) Oct. 18, 2005 #4

Slide 5: Hardware Layouts for LAMP Installations ONE BOX Oct. 18, 2005 #5

Slide 6: Hardware Layouts for LAMP Installations • Growing Up, “Two Box” solution  Higher traffic application (more demand)  Apache/PHP on box A, MySQL on box B  Same network = bad (*or is it ?), separate network = good  Bottlenecks with start to be: • Disk I/O on MySQL machine (Innodb) • Locking on MyISAM tables • Network I/O Oct. 18, 2005 #6

Slide 7: Hardware Layouts for LAMP Installations TWO BOX Oct. 18, 2005 #7

Slide 8: Hardware Layouts for LAMP Installations • Growing Up, “Many Boxes with Replication” solution • Yet even higher traffic • Writes are separated from reads (master gets IN/UP/DEL, slaves get SELECTs) • Diminishes network bottlenecks, disk I/O, and other “in-box” issues • SELECTs, IN/UP/DEL can be specified within the application, • OR…. • Load-balancing can be used Oct. 18, 2005 #8

Slide 9: Hardware Layouts for LAMP Installations MANY BOX Oct. 18, 2005 #9

Slide 10: Hardware Layouts for LAMP Installations Slave Lag • When slaves can’t keep up with replication • They’re too busy: • Reading (production traffic) • Writing (replication) • Manifests as: • Comments/photos/any user-entered data doesn’t show up on the site right away • So users will repeat the action, thinking that it didn’t “take” the first time, makes situation worse Oct. 18, 2005 # 10

Slide 11: Hardware Layouts for LAMP Installations Insert funny photo here about slave lag* *slave lag isn’t funny Oct. 18, 2005 # 11

Slide 12: Hardware Layouts for LAMP Installations Hardware Load Balancing MySQL Oct. 18, 2005 # 12

Slide 13: Hardware Layouts for LAMP Installations How It’s Usually Done • Standard MySQL master/slave replication • All writes (inserts/updates/deletes) from application go to Master • All reads (selects) from application go to a load- balanced VIP (virtual IP) spreading out load across all slaves Oct. 18, 2005 # 13

Slide 14: Hardware Layouts for LAMP Installations Oct. 18, 2005 # 14

Slide 15: Hardware Layouts for LAMP Installations What Is Good About Load Balancing • you can add/remove slaves without affecting application, since queries are atomic (sorta/kinda) • additional monitoring point and some automatic failure handling • you can treat all of your slave pool as one resource, and makes capacity planning a lot easier if you know the ceiling of each slave Oct. 18, 2005 # 15

Slide 16: Hardware Layouts for LAMP Installations • How do you know the ceiling (maximum QPS capacity) of each slave ? • First make a guess based on benchmarking (or look up some bench results from Tom’s Hardware or anandtech.com, etc. • Then get more machines than that :) • Scary: in production during a lull in traffic, remove machines from the pool until you detect lag • The QPS you saw right before slave lag set in: THAT is your ceiling Oct. 18, 2005 # 16

Slide 18: Hardware Layouts for LAMP Installations What Can Be Bad/Tough About Load Balancing: • not all load-balancers are created equal, not all load- balancing companies expect this product use, so support may still be thin • not that many people are doing it in high-volume situations yet, so support from community isn’t large either • Gotchas: • port exhaustion, • health checks, • and balance algorithms Oct. 18, 2005 # 18

Slide 19: Hardware Layouts for LAMP Installations Port Exhaustion PROBLEM: • LB is basically a traffic cop, nothing more • Side effect of having a lot of connections: only ~64,511 ports per each IP (VIP) to use • 64,511 ports/120 sec per port…. • ~535 max concurrent connections per IP* * Not really, but close to it: tcp_tw_recycle and tcp_tw_reuse Oct. 18, 2005 # 19

Slide 21: Hardware Layouts for LAMP Installations Port Exhaustion (cont’d) SOLUTION: • Use a pool of IPs on the database slave/farm side (Netscaler calls these “subnet IPs”, Alteon calls them “PiPs”) • Monitor port/connection usage, know when it’s time to add more Oct. 18, 2005 # 21

Slide 22: Hardware Layouts for LAMP Installations Health checks • LB won’t know anything about how well each MySQL slave is doing, and will pass traffic as long as port 3306 is answering • Load balancers don’t talk SQL, only things like plain old TCP, HTTP/S, maybe FTP Oct. 18, 2005 # 22

Slide 23: Hardware Layouts for LAMP Installations Health checks (cont’d) • Two options: 1.Dirty, but workable:  Have each server monitor itself, and shut off/firewall its own port 3306, even if MySQL is still running Oct. 18, 2005 # 23

Slide 24: Hardware Layouts for LAMP Installations Health checks (cont’d) 2. Cleaner, but a bit more work: • Have each server monitor itself, and run a check via xinetd (for example, a nagios monitor)  So the LB can tickle that port, and expect back an “OK” string. If not, it’ll automatically take that server out of the pool  Good for detecting and counteracting isolated incidents of ‘slave lag’ and automatically handling it Oct. 18, 2005 # 24

Slide 25: Hardware Layouts for LAMP Installations Health Checks Oct. 18, 2005 # 25

Slide 26: Hardware Layouts for LAMP Installations Balancing Algorithms • Load balancers know HTTP, FTP, basic TCP, but not SQL • Two things to care about: • Should the server still be in the pool ? (health checks) • How should load get balanced ?  “least connections” or “least bandwidth” or “least anything” = BAD  Because not all SQL queries are created equal  Use “round-robin” or “random”  What happens if you don’t: Evil Favoritism™ Oct. 18, 2005 # 26

Slide 27: Hardware Layouts for LAMP Installations Evil Favoritism Oct. 18, 2005 # 27

Slide 29: Hardware Layouts for LAMP Installations • Meanwhile….for “in-the-box considerations”  Interleaving memory *does* make a difference  Always RAID10 (or RAID0 if you’re crazy*) but NEVER RAID5 (for Innodb, anyway)  RAID10 has much more read capacity, and a write penalty, but not as much as RAID5  Always have battery backup for HW RAID write caching  Or, don’t use write caching at all Oct. 18, 2005 # 29

Slide 30: Hardware Layouts for LAMP Installations • “IN-THE-BOX” considerations (cont’d)  Always have proper monitoring (nagios, etc.) for failed/rebuilding drives  SATA or SCSI ? SCSI ! It’s worth it!  10k or 15k RPM SCSI ? 15k! It’s worth it! (~20% performance increase when you’re disk bound)  For 64bit Linux (AMD64 or EM64T): • Crank up the RAM for Innodb’s buffer pool • Swapping = very very bad either: • Turn it off (slightly scary) • Leave it on and set /proc/sys/vm/swapiness = 0 Oct. 18, 2005 # 30

Slide 31: Hardware Layouts for LAMP Installations • 10k versus 15k drives ? • Does it really matter that much ? • Some in-the-wild proof…. Oct. 18, 2005 # 31

Slide 32: Hardware Layouts for LAMP Installations Slave Lag in production 10K drives 15K drives Oct. 18, 2005 # 32

Slide 33: Hardware Layouts for LAMP Installations • Using MySQL with a SAN (Storage Area Network) • Do layout storage same as if they would be local • Do make sure that the HBA (fiber card) driver is well supported by Linux • Don’t share volumes across databases • Don’t forget to correctly tune Queue Depth Size, which should be increasing, from server HBA -> switch -> storage Oct. 18, 2005 # 33

Slide 34: Hardware Layouts for LAMP Installations Caching your static content Oct. 18, 2005 # 34

Slide 35: Hardware Layouts for LAMP Installations • Caching Static Content  SQUID = good  Relieve your front-end PHP machines from looking up data that will never (or rarely) change  Generate static pages, and cache them in squid, along with your images Oct. 18, 2005 # 35

Slide 36: Hardware Layouts for LAMP Installations • Caching Static Content (cont’d)  Use SQUID to accelerate plain-old origin webservers, also known as “reverse-proxy” HTTP acceleration  Described here and elsewhere:  http://www.squid-cache.org/Doc/FAQ/FAQ-20.html Oct. 18, 2005 # 36

Slide 37: Hardware Layouts for LAMP Installations Basic SQUID layout • squid accepts requests on 80 • passes on cache misses to apache on 81 • apache uses as its docroot an NFS mounted dir • should be on local subnet, or dedicated net Oct. 18, 2005 # 37

Slide 38: Hardware Layouts for LAMP Installations • Good HW layout for high-volume SQUIDing  Do use SCSI, and many spindles for disk cache dirs  Don’t use RAID  Do use network attached storage, or place the origin servers on separate machines  Do use ext3 with noatime for disk cache dirs  Do monitor squid stats Oct. 18, 2005 # 38

Slide 39: Hardware Layouts for LAMP Installations Flickr: How We Roll Oct. 18, 2005 # 39

Slide 40: Hardware Layouts for LAMP Installations • Yummy SQUID stats: • >2800 images/sec, ~75-80% are cache hits • ~10 million photos cached at any time • 1.5 million cached in memory Oct. 18, 2005 # 40

Slide 41: Hardware Layouts for LAMP Installations The End Oct. 18, 2005 # 41

Slideshare.net (beta)