Scaling PHP to 40 Million Uniques

Scaling PHP to
40 Million Uniques
Northeast PHP 2013
Jonathan Klein
@jonathanklein
Saturday, August 17, 13

Scaling PHP to
60 Million Uniques
Northeast PHP 2013
Jonathan Klein
@jonathanklein

Who Am I?

Who Am I?
• Performance Engineer at Etsy

Who Am I?
• Boston Web Perf Meetup Organizer

Who Am I?
• Previously at Wayfair

Who Am I?
• Previously at Wayfair
• Led team that converted to PHP

Slides, Links:
jkle.in/nephp

codeascraft.com

Some Etsy Stats

Some Etsy Stats
• 1.5 billion page views/month

Some Etsy Stats
• Almost $1B in sales last year

Some Etsy Stats
• Over 1M lines of PHP

Some Etsy Stats
• Over 1M lines of PHP
• 60M+ unique visitors/month

2012 Trafﬁc

August
2012 Trafﬁc

August
2012 Trafﬁc
50% Higher

Holidays:
~90M Uniques

A few others...

A few others...
• Search (Solr)

A few others...
• Search (Solr)
• Gearman

A few others...
• Search (Solr)
• Gearman
• Redis

A few others...
• Search (Solr)
• Gearman
• Redis
• Postgres (legacy)

Hardware (Supermicro)

2x 8-core Intel E5-2960 CPUs

24GB of RAM

24GB of RAM
160GB SSD

Apache

Apache
• Apache 2.2

Apache
• Apache 2.2
• Prefork MPM

Apache
• Apache 2.2
• Prefork MPM
• mod_php5

Apache
• Apache 2.2
• Prefork MPM
• mod_php5
StartServers 30
MinSpareServers 30
MaxSpareServers 60
ServerLimit 60
MaxClients 60
MaxRequestsPerChild 0

PHP
• PHP 5.4

PHP
• PHP 5.4
• Zend OPCache (3GB Memory Segment)

PHP
• PHP 5.4
• memory_limit: 128M

PHP
• PHP 5.4
• memory_limit: 128M
• max_execution_time: 30

Optimizing PHP
http://www.slideshare.net/jnklein/northeast-php-high-performance-php

Optimizing PHP
• See last year’s talk

Optimizing PHP
• Use an Opcode Cache

Optimizing PHP
• xhprof

Optimizing PHP
• xhprof
• StatsD/Graphite

Optimizing PHP
• xhprof
• StatsD/Graphite
• Find hotspots

Optimizing PHP
• xhprof
• StatsD/Graphite
• Find hotspots
• Upgrade it

Static Arrays

Translations
class language_de {
static $translations = array(
"<hash>" => array("content" => 'Accessoires', "file" => "TAXONOMY"),
"<hash>" => array("content" => 'Keramik', "file" => "CATEGORY"),
"<hash>" => array("content" => 'Dekorieren', "file" => "CATEGORY"),
"<hash>" => array("content" => 'Getaggt %s', "file" => "Foo.php"),
"<hash>" => array("content" => 'Badezusatz', "file" => "CATEGORY"),
"<hash>" => array("content" => 'Datum:{% $date %}', "file" => "bar.tpl")
... snip ...
);
}

Atomic Deploys

Deploying PHP
http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/

Deploying PHP
• mod_realdoc - Apache module

Deploying PHP
• incpath - PHP Extension

Deploying PHP
• A/B symlink swap

Deploying PHP
• A/B symlink swap
• Avoid recompilation to opcodes

Upgrading PHP

PHP 5.4 vs. 5.3 - CPU
PHP 5.3
PHP 5.4

PHP 5.4 vs. 5.3 - Memory
PHP 5.3
PHP 5.4

Understand
Framework
Overhead

http://systemsarchitect.net/performance-benchmark-of-popular-php-frameworks/

Frameworks are
Harder to Scale

Memcached

Memcached
• Roughly same hardware as webs

Memcached
• More memory (48GB)

Memcached
• Shard keys across servers

Memcached
• Shard keys across servers
• mctop - https://github.com/etsy/mctop

MySQL
• MySQL 5.5 (http://jkle.in/nephp)

MySQL
• ~20 master/master pairs

MySQL
• innodb

MySQL
• innodb
• thread_cache=800

MySQL
• innodb
• max_connections=2500

MySQL
• innodb
• max_connections=2500
• [Almost] no joins

Theoretically we can
serve ~9000 req/sec

Static Content

~500 Million Images

Offloading Requests

Offloading Requests
• Multiple CDNs

Offloading Requests
• Multiple CDNs
• Img hit rate > 90%

Offloading Requests
• Multiple CDNs
• CSS/JS hit rate ~100%

Offloading Requests
• Multiple CDNs
• CSS/JS hit rate ~100%
• Edge serves ~7000 reqs/sec

What about misses?

Change Management

Feature Flags

// Owner: Username
$server_config['foo']['bar'] = array(
'users' => array('jsmith', 'jdoe'),
);
// Owner: Username
$server_config['baz'] = [
'enabled' => [
'variant1' => 0,
'variant2' => 0,
'variant3' => 50
],
];
// Owner: Username
$server_config['qux'] = array(
'enabled' => 100,
'data' => array(
'somedata'
)
);

if (Feature::isEnabled('foo.bar')) {
// Code
}

Ramp-up:
1%, 5%, 25%, 100%

Deployinator

Schemanator

Recent Improvements

Sandy Bridge

Baseline Performance (median)

Small code change

Small code change
PHP 5.4

Small code change
PHP 5.4
Disabled
Hyperthreading

Challenges

Options

Options
• Scale Horizontally (more shards)

Options
• Scale Vertically (SSD, bigger drives)

Options
• Scale Vertically (SSD, bigger drives)
• Change architecture

Architecture Changes

• Ongoing conversation

• Separate logical/physical shards

• Complex

• Complex
• Server failure

• Complex
• Server failure
• Migrating data

• Complex
• Server failure
• Migrating data
• Consistency

Monitoring

Graphite

Stacked Search Timers

Ganglia

Nagios

Takeaways

K.I.S.S.

Use Proven
Technologies

Understand Your Stack

Measure Everything

Work at Etsy ;-)

Connect
http://www.meetup.com/Web-Performance-Boston/
www.etsy.com/careers
jonathan@etsy.com
@jonathanklein

Scaling PHP to 40 Million Uniques

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Scaling PHP to 40 Million Uniques

Similar to Scaling PHP to 40 Million Uniques (20)

Recently uploaded

Recently uploaded (20)

Scaling PHP to 40 Million Uniques