Clustered PHPDC PHP 2009by Marcel Esser – marcel.esser@croscon.comSenior Developer & Systems Analyst, CROSCON

what is clustering?“A computer cluster is a group of linked computers, working together closely so that in many respects they form a single computer.”- Wikipedia

what is clustering?“A computer cluster is a group of linked computers, working together closely so that in many respects they form a single computer.”- Wikipedia…in other words, it’s really hard.

what is clustering not?Clustering is not high availability.… but you often achieve high availability with clustering.

what is clustering not?Clustering is not high availability.… but you often achieve high availability with clustering.Clustering is not high volume.… but you often achieve high volume with clustering.

what is clustering not?Clustering is not high availability.… but you often achieve high availability with clustering.Clustering is not high volume.… but you often achieve high volume with clustering.We want both.

also…clustering is really,really hard

some do it well

some … ehwell, nevermind.

why cluster?service more users

why cluster?service more usersservice users faster

why cluster?service more usersservice users fasterincrease reliability

why cluster?service more usersservice users fasterincrease reliabilityget rich

objectiveslinear capacity increase

1 server - 100 lolcats per second

2 servers - 200 lolcats per second

3 servers - 300 lolcats per second

objectiveslinear capacity increaselinear cost increase

100 lolcats - $100

200 lolcats - $200

300 lolcats - $300

objectiveslinear capacity increaselinear cost increaseexponential reliability increase

common topics in clustering php Load Balancing Database Scaling Replicated Storage Backups Data Caches Distributed Sessions Staging Strategies Debugging Background Services

Load Balancing

Database Scaling

Replicated Storage

Backups

Data Caches

Distributed Sessions

Staging Strategies

Debugging

Background Services

load balancingLoadServersClientBalancer

load balancingYour load balancer may or may not… remove bad nodes from the pool balance by performance balance by weight route by geolocation support sticky sessions have 1 million other features

remove bad nodes from the pool

balance by performance

balance by weight

route by geolocation

support sticky sessions

have 1 million other features

load balancing toolssome among thousands… DNS servers Big IPPerlbal nginx Varnish

DNS servers

Big IP

Perlbal

nginx

Varnish

database scalingcommon things you can do:partitioningreplicationsharding

database partitioning every user is assigned to a database server users don’t share data between each other (between servers) when you need more capacity, add another database server works for some apps, doesn’t work for othersimplementation example: invoice and timesheet management app

every user is assigned to a database server

users don’t share data between each other (between servers)

when you need more capacity, add another database server

works for some apps, doesn’t work for others

database replication (mysql)master - mastermaster - slavemaster - many slave

database replication (mysql)master - masterserver 1 replicates (as master) to server 2 (acting as slave)server 2 replicates (as master) to server 1 (acting as slave) works well to a point complete nightmare when replication gets desynchronized doesn’t actually improve write performance good for basic high availability

works well to a point

complete nightmare when replication gets desynchronized

doesn’t actually improve write performance

good for basic high availability

database replication (mysql)master - slaveserver 1 replicates (as master) to server 2 (acting as slave) good first step makes you re-write your application to consider slave queries doesn’t increase write performance de-synchronization is relatively painless replication lag

good first step

makes you re-write your application to consider slave queries

doesn’t increase write performance

de-synchronization is relatively painless

replication lag

database replication (mysql)master – many slaveserver 1 replicates (as master) to many servers (acting as slaves) thundering read performance makes you re-write your application to consider slave queries doesn’t increase write performance de-synchronization is relatively painless replication lag

thundering read performance

makes you re-write your application to consider slave queries

doesn’t increase write performance

de-synchronization is relatively painless

replication lag

database sharding data is split between multiple database servers logical index is kept of what data is where (for example, a mathematical index or a lookup chart) you have to grab, parse, and correlate data across servers theoretically limitless scalability complicatedimplementation example: digg, facebook, etc

data is split between multiple database servers

logical index is kept of what data is where (for example, a mathematical index or a lookup chart)

you have to grab, parse, and correlate data across servers

theoretically limitless scalability

complicated

replicated storagecommon things you can do:replicated file systemlookup tablesstorage serviceshuge NAS arrays

replicated file system (glusterfs) very affordable various replication modes nothing to keep track of in your app easy to implement can cause massive failures if poorly configured

very affordable

various replication modes

nothing to keep track of in your app

easy to implement

can cause massive failures if poorly configured

lookup tables very affordable limitless mode; entirely up to you entirely dependent on your application logic can cause massive failures if poorly implemented

very affordable

limitless mode; entirely up to you

entirely dependent on your application logic

can cause massive failures if poorly implemented

storage services very expensive theoretically limit-less capacity easy to use data must be pulled back first if used locally costs and bandwidth usage can be mitigated (for example, by putting a proxy in front of it)

very expensive

theoretically limit-less capacity

easy to use

data must be pulled back first if used locally

costs and bandwidth usage can be mitigated (for example, by putting a proxy in front of it)

large NAS arrays insanely expensive insanely expensive insanely expensive bullet-proof fault tolerance… at a price easy to use… for a price

insanely expensive

insanely expensive

insanely expensive

bullet-proof fault tolerance… at a price

easy to use… for a price

backupscommon methods:all-RAID (doesn’t work)snapshotscopying from slaves

backupsall-RAID doesn’t workWhy?RAID won’t keep your application fromdeleting data everywhere

snapshotsuse a mechanism to take a snapshot of the partitioni.e. LVM partitions works really well easy if you do it from the beginning requires some planning should be used with RAID drives

works really well

easy if you do it from the beginning

requires some planning

should be used with RAID drives

copying from slavestake a slave out of rotation and copy from iti.e. MySQL databases works really well easy if you do it from the beginning requires some planning backups can be out of date

works really well

easy if you do it from the beginning

requires some planning

backups can be out of date

data caches (memcached)PHP doesn’t have cross-request persistence, sosomeone added it: memcached in-memory fast scalable proven use itGot configuration data? Small, high-TTL data sets?Use APC.Large, high-TTL data sets? Use files.Mind the race condition.

in-memory

fast

scalable

proven

use it

replicated sessionspick your poison: memcache w. redundancy database shared file system (don’t actually do this)

memcache w. redundancy

database

shared file system (don’t actually do this)

staging strategyIf you value your free time:DevStageLiveTest

staging strategy (dev) do use source control systems (subversion, etc) do profile your to loop for obvious performance issues do use phpdoc tags do make your dev environment as similar to live as practical (i.e., don’t develop on Windows and run live on UNIX) do document all your changes do use TDD (test-driven development)

do use source control systems (subversion, etc)

do profile your to loop for obvious performance issues

do use phpdoc tags

do make your dev environment as similar to live as practical (i.e., don’t develop on Windows and run live on UNIX)

do document all your changes

do use TDD (test-driven development)

staging strategy (test) do make test functionally identical to live, except for data do create data fixtures that are representative of real-life data do create functional tests for the user interface (Selenium) do not push anything to stage that did not pass unit tests

do make test functionally identical to live, except for data

do create data fixtures that are representative of real-life data

do create functional tests for the user interface (Selenium)

do not push anything to stage that did not pass unit tests

staging strategy (stage) do make stage identical to a live node do connect to the live database do have test ‘users’ to perform destructive operations against do have a mechanism to automate pushing stage to live

do make stage identical to a live node

do connect to the live database

do have test ‘users’ to perform destructive operations against

do have a mechanism to automate pushing stage to live

staging strategy (live) do not ever make changes by hand on live do automate pushing updates do take nodes out rotation when you push updates do not allow ssh access to live except when really needed

do not ever make changes by hand on live

do automate pushing updates

do take nodes out rotation when you push updates

do not allow ssh access to live except when really needed

debugging do use xdebug on dev, test, and stage do prepare an automated action that can turn xdebug and profiling on/off on 1 of the live nodes. you can and will run into errors that only exist on live. do write a test case to replicate the the bug first and then fix the bug, whenever possible do first look if bugs are explainable by platform differences between development and production systems (i.e., don’t develop on Windows and deploy on UNIX) do go to my talk at ZendCon in October, “It Works on Dev”

do use xdebug on dev, test, and stage

do prepare an automated action that can turn xdebug and profiling on/off on 1 of the live nodes. you can and will run into errors that only exist on live.

do write a test case to replicate the the bug first and then fix the bug, whenever possible

do first look if bugs are explainable by platform differences between development and production systems (i.e., don’t develop on Windows and deploy on UNIX)

do go to my talk at ZendCon in October, “It Works on Dev”

background services do void launching background processes from the web app PHP doesn’t have a native message queue, so (many) people wrote some. example, gearmand. do use a message queue. do check for memory leaks in background tasks! many php libraries and also many php versions themselves still leak memory. try to write a loop in bash for a background task rather than in php. recycle the process often. do plan your message format carefully do persist important messages

do void launching background processes from the web app

PHP doesn’t have a native message queue, so (many) people wrote some. example, gearmand. do use a message queue.

do check for memory leaks in background tasks! many php libraries and also many php versions themselves still leak memory. try to write a loop in bash for a background task rather than in php. recycle the process often.

do plan your message format carefully

do persist important messages

questions…anybody?

shameless plugCROSCONBespoke Application DevelopmentConsultingSecurityMarcel Essermarcel.esser@croscon.com(202) 730-9728Personal? marcel.esser@gmail.com