SlideShare a Scribd company logo
Hardware Layouts for LAMP Installations John Allspaw, Flickr Plumbr Flickr (Yahoo) [email_address] October 18, 2005
Hardware Layouts for LAMP Installations Hardware requirements for LAMP installs have to do with: A decent amount about the actual hardware (“in-box” stuff) A bit more about the hardware  architecture Which should complement the  application  architecture
Hardware Layouts for LAMP Installations What we’ll talk about here: Database (MySQL) layouts and considerations Some miscellaneous/esoteric stuff (lessons learned) Caching content and considerations
Hardware Layouts for LAMP Installations Growing Up, “One Box” solution Basic web application (discussion board, etc.) Low traffic  Apache/PHP/MySQL on one machine Bottlenecks will start showing up: Most likely database before apache/php Disk I/O (Innodb) or locking wait states (MyISAM) Context switching between memory work (apache) and CPU work (MySQL)
Hardware Layouts for LAMP Installations ONE BOX
Hardware Layouts for LAMP Installations Growing Up, “Two Box” solution Higher traffic application (more demand) Apache/PHP on box A, MySQL on box B Same network = bad (*or is it ?), separate network = good Bottlenecks with start to be: Disk I/O on MySQL machine (Innodb) Locking on MyISAM tables Network I/O
Hardware Layouts for LAMP Installations TWO BOX
Hardware Layouts for LAMP Installations Growing Up, “Many Boxes with Replication” solution Yet even higher traffic  Writes are separated from reads (master gets IN/UP/DEL, slaves get SELECTs) Diminishes network bottlenecks, disk I/O, and other “in-box” issues SELECTs, IN/UP/DEL can be specified  within  the application,  OR…. Load-balancing can be used
Hardware Layouts for LAMP Installations MANY BOX
Hardware Layouts for LAMP Installations Slave Lag When slaves can’t keep up with replication They’re too busy: Reading (production traffic) Writing (replication) Manifests as: Comments/photos/any user-entered data doesn’t show up on the site right away So users will repeat the action, thinking that it didn’t “take” the first time, makes situation worse
Hardware Layouts for LAMP Installations Insert funny photo here about slave lag* *slave lag isn’t funny
Hardware Layouts for LAMP Installations Hardware Load Balancing MySQL
Hardware Layouts for LAMP Installations How It’s Usually Done Standard MySQL master/slave replication All writes (inserts/updates/deletes) from application go to Master All reads (selects) from application go to a load-balanced VIP (virtual IP)  spreading out load across all slaves
Hardware Layouts for LAMP Installations
Hardware Layouts for LAMP Installations What Is Good About Load Balancing you can add/remove slaves without affecting application, since queries are atomic (sorta/kinda) additional monitoring point and some automatic failure handling you can treat all of your slave pool as one resource, and makes capacity planning a lot easier if you know the  ceiling  of each slave
Hardware Layouts for LAMP Installations How do you know the ceiling (maximum QPS capacity) of each slave ? First make a guess based on benchmarking (or look up some bench results from Tom’s Hardware or anandtech.com, etc.  Then get more machines than that :) Scary:  in production during a lull in traffic, remove machines from the pool until you detect lag The QPS you saw right before slave lag set in: THAT  is your ceiling
Hardware Layouts for LAMP Installations
Hardware Layouts for LAMP Installations What Can Be Bad/Tough About Load Balancing:  not all load-balancers are created equal, not all load-balancing companies expect this product use, so support may still be thin not that many people are doing it in high-volume situations yet, so support from community isn’t large either Gotchas:  port exhaustion,  health checks,  and balance algorithms
Hardware Layouts for LAMP Installations Port Exhaustion PROBLEM: LB is basically a traffic cop, nothing more Side effect of having a lot of connections:  only ~64,511 ports per each IP (VIP) to use 64,511 ports/120 sec per port…. ~535 max concurrent connections per IP* * Not really, but close to it:  tcp_tw_recycle and tcp_tw_reuse
Hardware Layouts for LAMP Installations
Hardware Layouts for LAMP Installations Port Exhaustion (cont’d) SOLUTION: Use a pool of IPs on the database slave/farm side (Netscaler calls these “subnet IPs”, Alteon calls them “PiPs”) Monitor port/connection usage, know when it’s time to add more
Hardware Layouts for LAMP Installations Health checks LB won’t know anything about how well each MySQL slave is doing, and will pass traffic as long as port 3306 is answering Load balancers don’t talk SQL, only things like plain old TCP, HTTP/S, maybe FTP
Hardware Layouts for LAMP Installations Health checks (cont’d) Two options: 1. Dirty, but workable:  Have each server monitor itself, and shut off/firewall its own port 3306, even if MySQL is still running
Hardware Layouts for LAMP Installations Health checks (cont’d) 2. Cleaner, but a bit more work: Have each server monitor itself, and run a check via xinetd (for example, a nagios monitor) So the LB can tickle that port, and expect back an “OK” string.  If not, it’ll automatically take that server out of the pool Good for detecting and counteracting isolated incidents of ‘slave lag’ and automatically handling it
Hardware Layouts for LAMP Installations Health Checks
Hardware Layouts for LAMP Installations Balancing Algorithms Load balancers know HTTP, FTP, basic TCP, but not SQL Two things to care about: Should the server still be in the pool ?  (health checks) How should load get balanced ? “ least connections” or “least bandwidth” or “least  anything” =  BAD Because not all SQL queries are created equal Use “round-robin” or “random” What happens if you don’t:  Evil Favoritism™
Hardware Layouts for LAMP Installations Evil Favoritism
Hardware Layouts for LAMP Installations
Hardware Layouts for LAMP Installations Meanwhile….for “in-the-box considerations” Interleaving memory *does* make a difference Always RAID10 (or RAID0 if you’re crazy*) but NEVER RAID5 (for Innodb, anyway) RAID10 has much more read capacity, and a write  penalty , but not as much as RAID5 Always have battery backup for HW RAID write caching Or, don’t use write caching at all
Hardware Layouts for LAMP Installations “ IN-THE-BOX” considerations (cont’d) Always have proper monitoring (nagios, etc.) for failed/rebuilding drives SATA or SCSI ?  SCSI !   It’s worth it! 10k or 15k RPM  SCSI ?  15k!   It’s worth it! (~20% performance increase when you’re disk bound) For 64bit Linux (AMD64 or EM64T): Crank up the RAM for Innodb’s buffer pool Swapping = very very bad either: Turn it off (slightly scary) Leave it on and set /proc/sys/vm/swapiness = 0
Hardware Layouts for LAMP Installations 10k versus 15k drives ? Does it really matter that much ? Some in-the-wild proof….
Hardware Layouts for LAMP Installations 10K drives 15K drives Slave Lag in production
Hardware Layouts for LAMP Installations Using MySQL with a SAN (Storage Area Network) Do  layout storage same as if they would be local Do  make sure that the HBA (fiber card) driver is  well  supported by Linux Don’t  share volumes across databases Don’t  forget to correctly tune Queue Depth Size, which should be increasing, from server HBA -> switch -> storage
Hardware Layouts for LAMP Installations Caching your static content
Hardware Layouts for LAMP Installations Caching Static Content SQUID = good Relieve your front-end PHP machines from looking up data that will never (or rarely) change Generate static pages, and cache them in squid, along with your images
Hardware Layouts for LAMP Installations Caching Static Content (cont’d) Use SQUID to accelerate plain-old origin webservers, also known as  “reverse-proxy”  HTTP acceleration Described here and elsewhere: http://www.squid-cache.org/Doc/FAQ/FAQ-20.html
Hardware Layouts for LAMP Installations Basic SQUID layout squid accepts requests on 80 passes on cache misses to apache on 81 apache uses as its docroot an NFS mounted dir should be on local subnet, or dedicated net
Hardware Layouts for LAMP Installations Good HW layout for high-volume SQUIDing Do  use SCSI, and many spindles for disk cache dirs Don’t  use RAID Do  use network attached storage, or place the origin servers on separate machines Do  use ext3 with noatime for disk cache dirs Do  monitor squid stats
Hardware Layouts for LAMP Installations Flickr: How We Roll
Hardware Layouts for LAMP Installations Yummy SQUID stats: >2800 images/sec, ~75-80% are cache hits ~10 million photos cached at any time 1.5 million cached in memory
Hardware Layouts for LAMP Installations The End

More Related Content

What's hot

Apache Tomcat + Java EE = Apache TomEE
Apache Tomcat + Java EE = Apache TomEEApache Tomcat + Java EE = Apache TomEE
Apache Tomcat + Java EE = Apache TomEE
Jacek Laskowski
 
End-to-end Troubleshooting Checklist for Microsoft SQL Server
End-to-end Troubleshooting Checklist for Microsoft SQL ServerEnd-to-end Troubleshooting Checklist for Microsoft SQL Server
End-to-end Troubleshooting Checklist for Microsoft SQL Server
Kevin Kline
 
CPAN Training
CPAN TrainingCPAN Training
CPAN Training
Pedro Figueiredo
 
All Change
All ChangeAll Change
All Change
Jason Arneil
 
Scale ColdFusion with Terracotta Distributed Caching for Ehchache
Scale ColdFusion with Terracotta Distributed Caching for EhchacheScale ColdFusion with Terracotta Distributed Caching for Ehchache
Scale ColdFusion with Terracotta Distributed Caching for Ehchache
ColdFusionConference
 
Performance all teh things
Performance all teh thingsPerformance all teh things
Performance all teh things
Marcus Deglos
 
JavaOne 2010: Top 10 Causes for Java Issues in Production and What to Do When...
JavaOne 2010: Top 10 Causes for Java Issues in Production and What to Do When...JavaOne 2010: Top 10 Causes for Java Issues in Production and What to Do When...
JavaOne 2010: Top 10 Causes for Java Issues in Production and What to Do When...
srisatish ambati
 
Oracle 12c Parallel Execution New Features
Oracle 12c Parallel Execution New FeaturesOracle 12c Parallel Execution New Features
Oracle 12c Parallel Execution New Features
Randolf Geist
 
AtoM's Command Line Tasks - An Introduction
AtoM's Command Line Tasks - An IntroductionAtoM's Command Line Tasks - An Introduction
AtoM's Command Line Tasks - An Introduction
Artefactual Systems - AtoM
 
Oracle olap-installation
Oracle olap-installationOracle olap-installation
Oracle olap-installation
Amit Sharma
 
Web container and Apache Tomcat
Web container and Apache TomcatWeb container and Apache Tomcat
Web container and Apache Tomcat
Auwal Amshi
 
Swoole Love PHP
Swoole Love PHPSwoole Love PHP
Swoole Love PHP
Yi-Feng Tzeng
 
Drupal, varnish, esi - Toulouse November 2
Drupal, varnish, esi - Toulouse November 2Drupal, varnish, esi - Toulouse November 2
Drupal, varnish, esi - Toulouse November 2
Marcus Deglos
 
Pinto+Stratopan+Love
Pinto+Stratopan+LovePinto+Stratopan+Love
Pinto+Stratopan+Love
Jeffrey Ryan Thalhammer
 
Os Wilhelm
Os WilhelmOs Wilhelm
Os Wilhelm
oscon2007
 
Os Leventhal
Os LeventhalOs Leventhal
Os Leventhal
oscon2007
 
Docker in Continuous Integration
Docker in Continuous IntegrationDocker in Continuous Integration
Docker in Continuous Integration
Alexander Akbashev
 
FITC - Here Be Dragons: Advanced JavaScript Debugging
FITC - Here Be Dragons: Advanced JavaScript DebuggingFITC - Here Be Dragons: Advanced JavaScript Debugging
FITC - Here Be Dragons: Advanced JavaScript Debugging
Rami Sayar
 
Perl in RPM-Land
Perl in RPM-LandPerl in RPM-Land
Perl in RPM-Land
Dave Cross
 
Perl
PerlPerl

What's hot (20)

Apache Tomcat + Java EE = Apache TomEE
Apache Tomcat + Java EE = Apache TomEEApache Tomcat + Java EE = Apache TomEE
Apache Tomcat + Java EE = Apache TomEE
 
End-to-end Troubleshooting Checklist for Microsoft SQL Server
End-to-end Troubleshooting Checklist for Microsoft SQL ServerEnd-to-end Troubleshooting Checklist for Microsoft SQL Server
End-to-end Troubleshooting Checklist for Microsoft SQL Server
 
CPAN Training
CPAN TrainingCPAN Training
CPAN Training
 
All Change
All ChangeAll Change
All Change
 
Scale ColdFusion with Terracotta Distributed Caching for Ehchache
Scale ColdFusion with Terracotta Distributed Caching for EhchacheScale ColdFusion with Terracotta Distributed Caching for Ehchache
Scale ColdFusion with Terracotta Distributed Caching for Ehchache
 
Performance all teh things
Performance all teh thingsPerformance all teh things
Performance all teh things
 
JavaOne 2010: Top 10 Causes for Java Issues in Production and What to Do When...
JavaOne 2010: Top 10 Causes for Java Issues in Production and What to Do When...JavaOne 2010: Top 10 Causes for Java Issues in Production and What to Do When...
JavaOne 2010: Top 10 Causes for Java Issues in Production and What to Do When...
 
Oracle 12c Parallel Execution New Features
Oracle 12c Parallel Execution New FeaturesOracle 12c Parallel Execution New Features
Oracle 12c Parallel Execution New Features
 
AtoM's Command Line Tasks - An Introduction
AtoM's Command Line Tasks - An IntroductionAtoM's Command Line Tasks - An Introduction
AtoM's Command Line Tasks - An Introduction
 
Oracle olap-installation
Oracle olap-installationOracle olap-installation
Oracle olap-installation
 
Web container and Apache Tomcat
Web container and Apache TomcatWeb container and Apache Tomcat
Web container and Apache Tomcat
 
Swoole Love PHP
Swoole Love PHPSwoole Love PHP
Swoole Love PHP
 
Drupal, varnish, esi - Toulouse November 2
Drupal, varnish, esi - Toulouse November 2Drupal, varnish, esi - Toulouse November 2
Drupal, varnish, esi - Toulouse November 2
 
Pinto+Stratopan+Love
Pinto+Stratopan+LovePinto+Stratopan+Love
Pinto+Stratopan+Love
 
Os Wilhelm
Os WilhelmOs Wilhelm
Os Wilhelm
 
Os Leventhal
Os LeventhalOs Leventhal
Os Leventhal
 
Docker in Continuous Integration
Docker in Continuous IntegrationDocker in Continuous Integration
Docker in Continuous Integration
 
FITC - Here Be Dragons: Advanced JavaScript Debugging
FITC - Here Be Dragons: Advanced JavaScript DebuggingFITC - Here Be Dragons: Advanced JavaScript Debugging
FITC - Here Be Dragons: Advanced JavaScript Debugging
 
Perl in RPM-Land
Perl in RPM-LandPerl in RPM-Land
Perl in RPM-Land
 
Perl
PerlPerl
Perl
 

Viewers also liked

Nowa ok prezentacja dla rodzicow 2016ppt [odzyskano]
Nowa ok prezentacja dla rodzicow 2016ppt [odzyskano]Nowa ok prezentacja dla rodzicow 2016ppt [odzyskano]
Nowa ok prezentacja dla rodzicow 2016ppt [odzyskano]
wiosenka
 
Dzień przedszkolaka na Bielanach
Dzień przedszkolaka na BielanachDzień przedszkolaka na Bielanach
Dzień przedszkolaka na Bielanach
wiosenka
 
Projeto Casa de Veraneio
Projeto Casa de VeraneioProjeto Casa de Veraneio
Projeto Casa de Veraneio
Pedro Arthur
 
презентация курса
презентация курсапрезентация курса
презентация курса
DemidovAV
 
john nader CV
john nader CVjohn nader CV
john nader CV
John Nader Michael
 
11 sl k_u_2011
11 sl k_u_201111 sl k_u_2011
11 sl k_u_2011
4book
 
Modelo petição juntada de procuração substabelecimento
Modelo petição juntada de procuração substabelecimentoModelo petição juntada de procuração substabelecimento
Modelo petição juntada de procuração substabelecimento
Bruno Rodrigues De Oliveira
 
Senac assistente de marketing aula 02
Senac assistente de marketing aula 02Senac assistente de marketing aula 02
Senac assistente de marketing aula 02
Marcus Vinícius Liberato
 
10 t k_u
10 t k_u10 t k_u
10 t k_u
4book
 
9 iu ser
9 iu ser9 iu ser
9 iu ser
4book
 
11 iu ser
11 iu ser11 iu ser
11 iu ser
4book
 
6 geog s_ua
6 geog s_ua6 geog s_ua
6 geog s_ua
Agent Plus UK
 
6 m m_2014_ru
6 m m_2014_ru6 m m_2014_ru
6 m m_2014_ru
Agent Plus UK
 
Prezdszkole nr 240
Prezdszkole nr 240 Prezdszkole nr 240
Prezdszkole nr 240
wiosenka
 
My presentation in MST -11 International Workshop
My presentation in MST -11 International WorkshopMy presentation in MST -11 International Workshop
My presentation in MST -11 International Workshop
Arpit Gupta
 

Viewers also liked (15)

Nowa ok prezentacja dla rodzicow 2016ppt [odzyskano]
Nowa ok prezentacja dla rodzicow 2016ppt [odzyskano]Nowa ok prezentacja dla rodzicow 2016ppt [odzyskano]
Nowa ok prezentacja dla rodzicow 2016ppt [odzyskano]
 
Dzień przedszkolaka na Bielanach
Dzień przedszkolaka na BielanachDzień przedszkolaka na Bielanach
Dzień przedszkolaka na Bielanach
 
Projeto Casa de Veraneio
Projeto Casa de VeraneioProjeto Casa de Veraneio
Projeto Casa de Veraneio
 
презентация курса
презентация курсапрезентация курса
презентация курса
 
john nader CV
john nader CVjohn nader CV
john nader CV
 
11 sl k_u_2011
11 sl k_u_201111 sl k_u_2011
11 sl k_u_2011
 
Modelo petição juntada de procuração substabelecimento
Modelo petição juntada de procuração substabelecimentoModelo petição juntada de procuração substabelecimento
Modelo petição juntada de procuração substabelecimento
 
Senac assistente de marketing aula 02
Senac assistente de marketing aula 02Senac assistente de marketing aula 02
Senac assistente de marketing aula 02
 
10 t k_u
10 t k_u10 t k_u
10 t k_u
 
9 iu ser
9 iu ser9 iu ser
9 iu ser
 
11 iu ser
11 iu ser11 iu ser
11 iu ser
 
6 geog s_ua
6 geog s_ua6 geog s_ua
6 geog s_ua
 
6 m m_2014_ru
6 m m_2014_ru6 m m_2014_ru
6 m m_2014_ru
 
Prezdszkole nr 240
Prezdszkole nr 240 Prezdszkole nr 240
Prezdszkole nr 240
 
My presentation in MST -11 International Workshop
My presentation in MST -11 International WorkshopMy presentation in MST -11 International Workshop
My presentation in MST -11 International Workshop
 

Similar to 1. Scaling PHP/MySQL...Presentation from Flickr

MNPHP Scalable Architecture 101 - Feb 3 2011
MNPHP Scalable Architecture 101 - Feb 3 2011MNPHP Scalable Architecture 101 - Feb 3 2011
MNPHP Scalable Architecture 101 - Feb 3 2011
Mike Willbanks
 
Apache Traffic Server
Apache Traffic ServerApache Traffic Server
Apache Traffic Server
supertom
 
Everyone loves PHP
Everyone loves PHPEveryone loves PHP
Everyone loves PHP
Abhijit Das
 
MySQL HA with PaceMaker
MySQL HA with  PaceMakerMySQL HA with  PaceMaker
MySQL HA with PaceMaker
Kris Buytaert
 
Performance Whack-a-Mole Tutorial (pgCon 2009)
Performance Whack-a-Mole Tutorial (pgCon 2009) Performance Whack-a-Mole Tutorial (pgCon 2009)
Performance Whack-a-Mole Tutorial (pgCon 2009)
PostgreSQL Experts, Inc.
 
DrupalCampLA 2011: Drupal backend-performance
DrupalCampLA 2011: Drupal backend-performanceDrupalCampLA 2011: Drupal backend-performance
DrupalCampLA 2011: Drupal backend-performance
Ashok Modi
 
Oracle R12 EBS Performance Tuning
Oracle R12 EBS Performance TuningOracle R12 EBS Performance Tuning
Oracle R12 EBS Performance Tuning
Scott Jenner
 
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYCScalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Cal Henderson
 
Sparklife - Life In The Trenches With Spark
Sparklife - Life In The Trenches With SparkSparklife - Life In The Trenches With Spark
Sparklife - Life In The Trenches With Spark
Ian Pointer
 
Web20expo Scalable Web Arch
Web20expo Scalable Web ArchWeb20expo Scalable Web Arch
Web20expo Scalable Web Arch
guest18a0f1
 
Web20expo Scalable Web Arch
Web20expo Scalable Web ArchWeb20expo Scalable Web Arch
Web20expo Scalable Web Arch
mclee
 
Web20expo Scalable Web Arch
Web20expo Scalable Web ArchWeb20expo Scalable Web Arch
Web20expo Scalable Web Arch
royans
 
Hug Hbase Presentation.
Hug Hbase Presentation.Hug Hbase Presentation.
Hug Hbase Presentation.
Jack Levin
 
WordPress Performance & Scalability
WordPress Performance & ScalabilityWordPress Performance & Scalability
WordPress Performance & Scalability
Joseph Scott
 
WP Sandbox Presentation WordCamp Toronto 2011
WP Sandbox Presentation WordCamp Toronto 2011WP Sandbox Presentation WordCamp Toronto 2011
WP Sandbox Presentation WordCamp Toronto 2011
Alfred Ayache
 
Performance Whack A Mole
Performance Whack A MolePerformance Whack A Mole
Performance Whack A Mole
oscon2007
 
Knowledge share about scalable application architecture
Knowledge share about scalable application architectureKnowledge share about scalable application architecture
Knowledge share about scalable application architecture
AHM Pervej Kabir
 
Automated Deployment using Open Source
Automated Deployment using Open SourceAutomated Deployment using Open Source
Automated Deployment using Open Source
duskglow
 
Planning For High Performance Web Application
Planning For High Performance Web ApplicationPlanning For High Performance Web Application
Planning For High Performance Web Application
Yue Tian
 
Asynchronous programming - .NET Way
Asynchronous programming - .NET WayAsynchronous programming - .NET Way
Asynchronous programming - .NET Way
Bishnu Rawal
 

Similar to 1. Scaling PHP/MySQL...Presentation from Flickr (20)

MNPHP Scalable Architecture 101 - Feb 3 2011
MNPHP Scalable Architecture 101 - Feb 3 2011MNPHP Scalable Architecture 101 - Feb 3 2011
MNPHP Scalable Architecture 101 - Feb 3 2011
 
Apache Traffic Server
Apache Traffic ServerApache Traffic Server
Apache Traffic Server
 
Everyone loves PHP
Everyone loves PHPEveryone loves PHP
Everyone loves PHP
 
MySQL HA with PaceMaker
MySQL HA with  PaceMakerMySQL HA with  PaceMaker
MySQL HA with PaceMaker
 
Performance Whack-a-Mole Tutorial (pgCon 2009)
Performance Whack-a-Mole Tutorial (pgCon 2009) Performance Whack-a-Mole Tutorial (pgCon 2009)
Performance Whack-a-Mole Tutorial (pgCon 2009)
 
DrupalCampLA 2011: Drupal backend-performance
DrupalCampLA 2011: Drupal backend-performanceDrupalCampLA 2011: Drupal backend-performance
DrupalCampLA 2011: Drupal backend-performance
 
Oracle R12 EBS Performance Tuning
Oracle R12 EBS Performance TuningOracle R12 EBS Performance Tuning
Oracle R12 EBS Performance Tuning
 
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYCScalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
 
Sparklife - Life In The Trenches With Spark
Sparklife - Life In The Trenches With SparkSparklife - Life In The Trenches With Spark
Sparklife - Life In The Trenches With Spark
 
Web20expo Scalable Web Arch
Web20expo Scalable Web ArchWeb20expo Scalable Web Arch
Web20expo Scalable Web Arch
 
Web20expo Scalable Web Arch
Web20expo Scalable Web ArchWeb20expo Scalable Web Arch
Web20expo Scalable Web Arch
 
Web20expo Scalable Web Arch
Web20expo Scalable Web ArchWeb20expo Scalable Web Arch
Web20expo Scalable Web Arch
 
Hug Hbase Presentation.
Hug Hbase Presentation.Hug Hbase Presentation.
Hug Hbase Presentation.
 
WordPress Performance & Scalability
WordPress Performance & ScalabilityWordPress Performance & Scalability
WordPress Performance & Scalability
 
WP Sandbox Presentation WordCamp Toronto 2011
WP Sandbox Presentation WordCamp Toronto 2011WP Sandbox Presentation WordCamp Toronto 2011
WP Sandbox Presentation WordCamp Toronto 2011
 
Performance Whack A Mole
Performance Whack A MolePerformance Whack A Mole
Performance Whack A Mole
 
Knowledge share about scalable application architecture
Knowledge share about scalable application architectureKnowledge share about scalable application architecture
Knowledge share about scalable application architecture
 
Automated Deployment using Open Source
Automated Deployment using Open SourceAutomated Deployment using Open Source
Automated Deployment using Open Source
 
Planning For High Performance Web Application
Planning For High Performance Web ApplicationPlanning For High Performance Web Application
Planning For High Performance Web Application
 
Asynchronous programming - .NET Way
Asynchronous programming - .NET WayAsynchronous programming - .NET Way
Asynchronous programming - .NET Way
 

Recently uploaded

leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
alexjohnson7307
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
SynapseIndia
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
ssuser1915fe1
 
Semantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software DevelopmentSemantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software Development
Baishakhi Ray
 
The History of Embeddings & Multimodal Embeddings
The History of Embeddings & Multimodal EmbeddingsThe History of Embeddings & Multimodal Embeddings
The History of Embeddings & Multimodal Embeddings
Zilliz
 
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Zilliz
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
Zilliz
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
DianaGray10
 
Patch Tuesday de julio
Patch Tuesday de julioPatch Tuesday de julio
Patch Tuesday de julio
Ivanti
 
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
bellared2
 
kk vathada _digital transformation frameworks_2024.pdf
kk vathada _digital transformation frameworks_2024.pdfkk vathada _digital transformation frameworks_2024.pdf
kk vathada _digital transformation frameworks_2024.pdf
KIRAN KV
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
bhumivarma35300
 
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdfLeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
SelfMade bd
 
Redefining Cybersecurity with AI Capabilities
Redefining Cybersecurity with AI CapabilitiesRedefining Cybersecurity with AI Capabilities
Redefining Cybersecurity with AI Capabilities
Priyanka Aash
 
The Path to General-Purpose Robots - Coatue
The Path to General-Purpose Robots - CoatueThe Path to General-Purpose Robots - Coatue
The Path to General-Purpose Robots - Coatue
Razin Mustafiz
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
sunilverma7884
 
Mule Experience Hub and Release Channel with Java 17
Mule Experience Hub and Release Channel with Java 17Mule Experience Hub and Release Channel with Java 17
Mule Experience Hub and Release Channel with Java 17
Bhajan Mehta
 
Camunda Chapter NY Meetup July 2024.pptx
Camunda Chapter NY Meetup July 2024.pptxCamunda Chapter NY Meetup July 2024.pptx
Camunda Chapter NY Meetup July 2024.pptx
ZachWylie3
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
Google Developer Group - Harare
 
Sonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdfSonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdf
SubhamMandal40
 

Recently uploaded (20)

leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
 
Semantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software DevelopmentSemantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software Development
 
The History of Embeddings & Multimodal Embeddings
The History of Embeddings & Multimodal EmbeddingsThe History of Embeddings & Multimodal Embeddings
The History of Embeddings & Multimodal Embeddings
 
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
 
Patch Tuesday de julio
Patch Tuesday de julioPatch Tuesday de julio
Patch Tuesday de julio
 
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
 
kk vathada _digital transformation frameworks_2024.pdf
kk vathada _digital transformation frameworks_2024.pdfkk vathada _digital transformation frameworks_2024.pdf
kk vathada _digital transformation frameworks_2024.pdf
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
 
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdfLeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
 
Redefining Cybersecurity with AI Capabilities
Redefining Cybersecurity with AI CapabilitiesRedefining Cybersecurity with AI Capabilities
Redefining Cybersecurity with AI Capabilities
 
The Path to General-Purpose Robots - Coatue
The Path to General-Purpose Robots - CoatueThe Path to General-Purpose Robots - Coatue
The Path to General-Purpose Robots - Coatue
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
 
Mule Experience Hub and Release Channel with Java 17
Mule Experience Hub and Release Channel with Java 17Mule Experience Hub and Release Channel with Java 17
Mule Experience Hub and Release Channel with Java 17
 
Camunda Chapter NY Meetup July 2024.pptx
Camunda Chapter NY Meetup July 2024.pptxCamunda Chapter NY Meetup July 2024.pptx
Camunda Chapter NY Meetup July 2024.pptx
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
 
Sonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdfSonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdf
 

1. Scaling PHP/MySQL...Presentation from Flickr

  • 1. Hardware Layouts for LAMP Installations John Allspaw, Flickr Plumbr Flickr (Yahoo) [email_address] October 18, 2005
  • 2. Hardware Layouts for LAMP Installations Hardware requirements for LAMP installs have to do with: A decent amount about the actual hardware (“in-box” stuff) A bit more about the hardware architecture Which should complement the application architecture
  • 3. Hardware Layouts for LAMP Installations What we’ll talk about here: Database (MySQL) layouts and considerations Some miscellaneous/esoteric stuff (lessons learned) Caching content and considerations
  • 4. Hardware Layouts for LAMP Installations Growing Up, “One Box” solution Basic web application (discussion board, etc.) Low traffic Apache/PHP/MySQL on one machine Bottlenecks will start showing up: Most likely database before apache/php Disk I/O (Innodb) or locking wait states (MyISAM) Context switching between memory work (apache) and CPU work (MySQL)
  • 5. Hardware Layouts for LAMP Installations ONE BOX
  • 6. Hardware Layouts for LAMP Installations Growing Up, “Two Box” solution Higher traffic application (more demand) Apache/PHP on box A, MySQL on box B Same network = bad (*or is it ?), separate network = good Bottlenecks with start to be: Disk I/O on MySQL machine (Innodb) Locking on MyISAM tables Network I/O
  • 7. Hardware Layouts for LAMP Installations TWO BOX
  • 8. Hardware Layouts for LAMP Installations Growing Up, “Many Boxes with Replication” solution Yet even higher traffic Writes are separated from reads (master gets IN/UP/DEL, slaves get SELECTs) Diminishes network bottlenecks, disk I/O, and other “in-box” issues SELECTs, IN/UP/DEL can be specified within the application, OR…. Load-balancing can be used
  • 9. Hardware Layouts for LAMP Installations MANY BOX
  • 10. Hardware Layouts for LAMP Installations Slave Lag When slaves can’t keep up with replication They’re too busy: Reading (production traffic) Writing (replication) Manifests as: Comments/photos/any user-entered data doesn’t show up on the site right away So users will repeat the action, thinking that it didn’t “take” the first time, makes situation worse
  • 11. Hardware Layouts for LAMP Installations Insert funny photo here about slave lag* *slave lag isn’t funny
  • 12. Hardware Layouts for LAMP Installations Hardware Load Balancing MySQL
  • 13. Hardware Layouts for LAMP Installations How It’s Usually Done Standard MySQL master/slave replication All writes (inserts/updates/deletes) from application go to Master All reads (selects) from application go to a load-balanced VIP (virtual IP) spreading out load across all slaves
  • 14. Hardware Layouts for LAMP Installations
  • 15. Hardware Layouts for LAMP Installations What Is Good About Load Balancing you can add/remove slaves without affecting application, since queries are atomic (sorta/kinda) additional monitoring point and some automatic failure handling you can treat all of your slave pool as one resource, and makes capacity planning a lot easier if you know the ceiling of each slave
  • 16. Hardware Layouts for LAMP Installations How do you know the ceiling (maximum QPS capacity) of each slave ? First make a guess based on benchmarking (or look up some bench results from Tom’s Hardware or anandtech.com, etc. Then get more machines than that :) Scary: in production during a lull in traffic, remove machines from the pool until you detect lag The QPS you saw right before slave lag set in: THAT is your ceiling
  • 17. Hardware Layouts for LAMP Installations
  • 18. Hardware Layouts for LAMP Installations What Can Be Bad/Tough About Load Balancing: not all load-balancers are created equal, not all load-balancing companies expect this product use, so support may still be thin not that many people are doing it in high-volume situations yet, so support from community isn’t large either Gotchas: port exhaustion, health checks, and balance algorithms
  • 19. Hardware Layouts for LAMP Installations Port Exhaustion PROBLEM: LB is basically a traffic cop, nothing more Side effect of having a lot of connections: only ~64,511 ports per each IP (VIP) to use 64,511 ports/120 sec per port…. ~535 max concurrent connections per IP* * Not really, but close to it: tcp_tw_recycle and tcp_tw_reuse
  • 20. Hardware Layouts for LAMP Installations
  • 21. Hardware Layouts for LAMP Installations Port Exhaustion (cont’d) SOLUTION: Use a pool of IPs on the database slave/farm side (Netscaler calls these “subnet IPs”, Alteon calls them “PiPs”) Monitor port/connection usage, know when it’s time to add more
  • 22. Hardware Layouts for LAMP Installations Health checks LB won’t know anything about how well each MySQL slave is doing, and will pass traffic as long as port 3306 is answering Load balancers don’t talk SQL, only things like plain old TCP, HTTP/S, maybe FTP
  • 23. Hardware Layouts for LAMP Installations Health checks (cont’d) Two options: 1. Dirty, but workable: Have each server monitor itself, and shut off/firewall its own port 3306, even if MySQL is still running
  • 24. Hardware Layouts for LAMP Installations Health checks (cont’d) 2. Cleaner, but a bit more work: Have each server monitor itself, and run a check via xinetd (for example, a nagios monitor) So the LB can tickle that port, and expect back an “OK” string. If not, it’ll automatically take that server out of the pool Good for detecting and counteracting isolated incidents of ‘slave lag’ and automatically handling it
  • 25. Hardware Layouts for LAMP Installations Health Checks
  • 26. Hardware Layouts for LAMP Installations Balancing Algorithms Load balancers know HTTP, FTP, basic TCP, but not SQL Two things to care about: Should the server still be in the pool ? (health checks) How should load get balanced ? “ least connections” or “least bandwidth” or “least anything” = BAD Because not all SQL queries are created equal Use “round-robin” or “random” What happens if you don’t: Evil Favoritism™
  • 27. Hardware Layouts for LAMP Installations Evil Favoritism
  • 28. Hardware Layouts for LAMP Installations
  • 29. Hardware Layouts for LAMP Installations Meanwhile….for “in-the-box considerations” Interleaving memory *does* make a difference Always RAID10 (or RAID0 if you’re crazy*) but NEVER RAID5 (for Innodb, anyway) RAID10 has much more read capacity, and a write penalty , but not as much as RAID5 Always have battery backup for HW RAID write caching Or, don’t use write caching at all
  • 30. Hardware Layouts for LAMP Installations “ IN-THE-BOX” considerations (cont’d) Always have proper monitoring (nagios, etc.) for failed/rebuilding drives SATA or SCSI ? SCSI ! It’s worth it! 10k or 15k RPM SCSI ? 15k! It’s worth it! (~20% performance increase when you’re disk bound) For 64bit Linux (AMD64 or EM64T): Crank up the RAM for Innodb’s buffer pool Swapping = very very bad either: Turn it off (slightly scary) Leave it on and set /proc/sys/vm/swapiness = 0
  • 31. Hardware Layouts for LAMP Installations 10k versus 15k drives ? Does it really matter that much ? Some in-the-wild proof….
  • 32. Hardware Layouts for LAMP Installations 10K drives 15K drives Slave Lag in production
  • 33. Hardware Layouts for LAMP Installations Using MySQL with a SAN (Storage Area Network) Do layout storage same as if they would be local Do make sure that the HBA (fiber card) driver is well supported by Linux Don’t share volumes across databases Don’t forget to correctly tune Queue Depth Size, which should be increasing, from server HBA -> switch -> storage
  • 34. Hardware Layouts for LAMP Installations Caching your static content
  • 35. Hardware Layouts for LAMP Installations Caching Static Content SQUID = good Relieve your front-end PHP machines from looking up data that will never (or rarely) change Generate static pages, and cache them in squid, along with your images
  • 36. Hardware Layouts for LAMP Installations Caching Static Content (cont’d) Use SQUID to accelerate plain-old origin webservers, also known as “reverse-proxy” HTTP acceleration Described here and elsewhere: http://www.squid-cache.org/Doc/FAQ/FAQ-20.html
  • 37. Hardware Layouts for LAMP Installations Basic SQUID layout squid accepts requests on 80 passes on cache misses to apache on 81 apache uses as its docroot an NFS mounted dir should be on local subnet, or dedicated net
  • 38. Hardware Layouts for LAMP Installations Good HW layout for high-volume SQUIDing Do use SCSI, and many spindles for disk cache dirs Don’t use RAID Do use network attached storage, or place the origin servers on separate machines Do use ext3 with noatime for disk cache dirs Do monitor squid stats
  • 39. Hardware Layouts for LAMP Installations Flickr: How We Roll
  • 40. Hardware Layouts for LAMP Installations Yummy SQUID stats: >2800 images/sec, ~75-80% are cache hits ~10 million photos cached at any time 1.5 million cached in memory
  • 41. Hardware Layouts for LAMP Installations The End