Caching and tuning fun for high scalability @ phpBenelux 2011

Caching and tuning fun for high scalability Wim Godden Cu.be Solutions Conference Jan 28, 2011

Who am I ?
Wim Godden
Owner of Cu.be Solutions (http://cu.be)
PHP developer since 1997
Developer of OpenX
Zend Certified Engineer
Zend Framework Certified Engineer
MySQL Certified Developer

Who are you ?
Developers ?
System/network engineers ?
Managers ?
Caching experience ?

Caching and tuning fun for high scalability Wim Godden Cu.be Solutions Conference Jan 28, 2011

Goals of this tutorial
Everything about caching and tuning
A few techniques
-> Increase reliability, performance and scalability
5 visitors/day -> 5 million visitors/day
(Don't expect miracle cure !)

Goals of this tutorial
Concepts & techniques
How to do stuff
How NOT to do stuff

Architecture

Caching

What's caching ?

What is caching ? select * from article join user on article.user_id = user.id order by created desc limit 10

Caching goals - 1 st goal
Reduce # of request
Reduce the load

Caching goals - 2 nd goal

Some figures
Pageviews : 5000 (4000 on 10 pages)
Avg. loading time : 200ms
Cache 10 pages
Avg. loading time : 20ms
-> Total avg. loading time : 56ms
Worth it ?

Caching goals - 3 rd goal
Send less data across the network / Internet
You benefit -> lower bill from upstream provider
Users benefit -> faster page load
Wait a second... that's mostly frontend stuff !

Theory of caching DB

Caching techniques
#1 : Store entire pages
Company Websites
Blogs
Full pages that don't change
Render -> Store in cache -> retrieve from cache

Caching techniques
#1 : Store entire pages

Caching techniques
#2 : Store parts of a page
Most common technique
Usually a small block in a page
Best effect : reused on lots of pages
Can be inserted on dynamic pages

Caching techniques
#2 : Store parts of a page

Caching techniques
#3 : Store SQL queries
↔ SQL query cache
Limited in size

Caching techniques
↔ SQL query cache
Limited in size
Resets on every insert/update/delete
Server and connection overhead
Goal :
not to get rid of DB
free up DB resources for more hits !
Better :
store returned object
store group of objects

Caching techniques

Caching techniques
#4 : Store complex PHP results
Not just calculations
CPU intensive tasks :
Config file parsing
XML file parsing
Loading CSV in an array
Save resources -> more resources available

Caching techniques
#4 : Store complex PHP results

Caching techniques
#xx : Your call Only limited by your imagination ! When you have data, think :
Creating time ?
Modification frequency ?
Retrieval frequency ?

How to find cacheable data
New projects : start from 'cache anything'
Existing projects :
Look at MySQL slow query log
Make a complete query log (don't forget to turn it off !)
Check page loading times

Caching storage - MySQL query cache
Use it
Don't rely on it
Good if you have :
lots of reads
few different queries
Bad if you have :
lots of insert/update/delete
lots of different queries

Caching storage - Database memory tables
Tables stored in memory
In MySQL : memory/heap table
↔ temporary table :
memory tables are persistent
temporary tables are session-specific
Faster than disk-based tables
Can be joined with disk-based tables
But :
default 16MByte limit
master-slave = trouble
if you don't need join -> overhead of DB software
So : don't use it unless you need to join

Caching storage - Opcode caching
DO !

APC
De-facto standard
Will be in PHP core in 5.4 ? 5.5 ? 6.0 ?
PECL or packages
eAccelerator
Zend Accelerator

APC
De-facto standard
Will be in PHP core in 5.4 ? 5.5 ? 6.0 ?
PECL or packages
eAccelerator
Zend Accelerator
X-Cache
WinCacheForPhp
PHP PHP + APC PHP + eAccelerator 42.18 req/sec 206.20 req/sec 211.49 req/sec

Caching storage - Disk
Data with few updates : good
Caching SQL queries : preferably not
DON'T use NFS or other network file systems
especially for sessions
high latency
locking issues !

Caching storage - Memory disk (ramdisk)
Usually faster than physical disk
But : OS file caching makes difference minimal

Caching storage - Disk / ramdisk
Overhead : filesystem access
Limited number of files per directory
-> Subdirectories
Local
5 Webservers -> 5 local caches
-> Hard to scale
How will you keep them synchronized ?
-> Don't say NFS or rsync !

Caching storage - APC variable cache
More than an opcode cache
Store user data in memory
apc_add / apc_store to add/update
apc_fetch to retrieve
apc_delete
Fast -> huge performance impact

Caching storage - APC variable cache
More than an opcode cache
Store user data in memory
apc_add / apc_store to add/update
apc_fetch to retrieve
apc_delete
Fast -> huge performance impact
Session support !
Downside :
local storage -> hard to scale
restart Apache -> cache = empty

Caching storage - Memcache
Facebook, Twitter, Slashdot, … -> need we say more ?
Distributed memory caching system
Key-value storage system
Keys - max. 250bytes
Values - max. 1Mbyte

Caching storage - Memcache
Facebook, Twitter, Slashdot, … -> need we say more ?
Distributed memory caching system
Multiple machines ↔ 1 big memory-based hash-table
Key-value storage system
Keys - max. 250bytes
Values - max. 1Mbyte
Extremely fast... non-blocking, UDP (!)

Memcache - where to install

Memcache - installation & running it
Installation
Distribution package
PECL
Windows : binaries
Running
No config-files
memcached -d -m <mem> -l <ip> -p <port>
ex. : memcached -d -m 2048 -l 127.0.0.1 -p 11211

Caching storage - Memcache - some notes
Not fault-tolerant
Lose session data
Lose shopping cart data
...

Caching storage - Memcache - some notes
Not fault-tolerant
Lose session data
Lose shopping cart data
…
Different libraries
Original : libmemcache
New : libmemcached (consistent hashing, UDP, binary protocol, …)
Firewall your Memcache port !

Caching in PHP <?php $memcache = new Memcache(); $memcache->addServer( '172.16.0.1' , 11211); $memcache->addServer( '172.16.0.2' , 11211); $myData = $memcache->get( 'myKey' ); if ($myData === false ) { $myData = GetMyDataFromDB(); // Put it in Memcache as 'myKey', without compression, with no expiration $memcache->set( 'myKey' , $myData, false , 0); } echo $myData;

Where's the data ?
Memcache client decides (!)
2 hashing algorithms :
Traditional
Server failure -> all data must be rehashed
Consistent
Server failure -> 1/x of data must be rehashed (x = # of servers)
No replication !

Memcache slabs
(or why Memcache says it's full when it's not)
Multiple slabs of different sizes :
Slab 1 : 400 bytes
Slab 2 : 480 bytes (400 * 1.2)
Slab 3 : 576 bytes (480 * 1.2) (and so on...)
Multiplier (1.2 here) can be configured
Each larger slab has room for fewer items (chunks)
-> Store a lot of very large objects
-> Large slabs might be full
-> Rest of slabs might be free
-> Try to store more -> eviction of data !

Memcache - Is it working ?
Connect to it using telnet
"stats" command ->
Use Cacti or other monitoring tools
STAT pid 2941 STAT uptime 10878 STAT time 1296074240 STAT version 1.4.5 STAT pointer_size 64 STAT rusage_user 20.089945 STAT rusage_system 58.499106 STAT curr_connections 16 STAT total_connections 276950 STAT connection_structures 96 STAT cmd_get 276931 STAT cmd_set 584148 STAT cmd_flush 0 STAT get_hits 211106 STAT get_misses 65825 STAT delete_misses 101 STAT delete_hits 276829 STAT incr_misses 0 STAT incr_hits 0 STAT decr_misses 0 STAT decr_hits 0 STAT cas_misses 0 STAT cas_hits 0 STAT cas_badval 0 STAT auth_cmds 0 STAT auth_errors 0 STAT bytes_read 613193860 STAT bytes_written 553991373 STAT limit_maxbytes 268435456 STAT accepting_conns 1 STAT listen_disabled_num 0 STAT threads 4 STAT conn_yields 0 STAT bytes 20418140 STAT curr_items 65826 STAT total_items 553856 STAT evictions 0 STAT reclaimed 0

Memcache - backing up

Memcache - deleting <?php $memcache = new Memcache(); $memcache->delete( 'myKey' ); $myData = $memcache->get( 'myKey' ); // $myData === false

Memcache - caching a page <?php $output = $memcache->get( 'page_' . $page_id); if ($output === false ) { ob_start(); GetMyPageInRegularWay($page_id); $output = ob_get_contents(); ob_end_clean(); $memcache->set( 'page_' . $page_id, $output, false , 600); // Cache 10 mins } echo $output;

Memcache - tip
Page with multiple blocks ? -> use Memcached::getMulti() Warning : what if you get some hits and some misses ?

Naming your keys
Key names must be unique
Prefix / namespace your keys !
Only letters, numbers and underscore
md5() is useful
-> BUT : harder to debug
Use clear names
Document your key names !

Adding/updating data

Adding/updating data $memcache->delete( 'ArticleDetails__Toshiba_32C100U_32_Inch' ); $memcache->delete( 'Homepage_Popular_Product_List' );

Adding/updating data

Adding/updating data - Why it crashed

Cache stampeding

Memcache code ? DB

Creating cache methods static protected function getArticle ($id) { $cache = Zend_Registry:: get ( 'Zend_Cache' ); if (!$article = $cache->load( 'article_' . $id )) { $db = Zend_Registry:: get ( 'Zend_DB' ); $query = ' select article.id as art_id, article.title as art_title, article.body as art_body, date_format(article.created, "%d/%m/%Y %H:%i") as art_created, user.username as use_username from article join user on article.user_id = user.id where article.id = ? ' ; $artice = $db->fetchRow($query, $id); $cache->save($articleList, 'article_' . $id ); } return $articleList; }

Creating cache methods static protected function getArticleUncached ($id) { $db = Zend_Registry:: get ( 'Zend_DB' ); $query = ' select article.id as art_id, article.title as art_title, article.body as art_body, date_format(article.created, "%d/%m/%Y %H:%i") as art_created, user.username as use_username from article join user on article.user_id = user.id where article.id = ? ' ; return $db->fetchRow($query, $id); } static public function getArticle ($id) { if (Zend_Registry:: isRegistered ( 'Zend_Cache' )) { $cache = Zend_Registry:: get ( 'Zend_Cache' ); if (!$articleList = $cache->load( 'article_' . $id )) { $articleList = self :: getArticleUncached ($id); $cache->save($articleList, 'article_' . $id ); } } else { $articleList = self :: getArticleUncached ($id); } return $articleList; }

Cache stampeding - what about locking ?
Seems like a nice idea, but...
Lock in place
-> lots of new connections
-> memory spike
What if the PHP process that created the lock fails ?
Avoids needing caching code in admin part
-> but what about warmup script(s) ?

Cache warmup scripts
Used to fill your cache when it's empty
Run it before starting Webserver !
2 ways :
Visit all URLs
Error-prone
Hard to maintain
Call all cache-updating methods
Means your caching code should be centralized !
Make sure you have a warmup script !

Quick word about expiration
General rule : don't let things expire
Exception to the rule : things that have an end date (calendar items)

So...
DON'T DELETE FROM CACHE (and don't expire unless usefull)

Quick-tip
Start small -> disk or APC
Move to Memcache later
But : is your code ready ?
-> Use a component like Zend_Cache to switch easily !

Time for...
a break (10 min)

LAMP...
-> LAMMP -> LANMMP

Nginx
Web server
Reverse proxy
Lightweight, fast
7.5% of all Websites

Nginx
No threads, event-driven
Uses epoll / kqueue
Low memory footprint
10000 active connections = normal

Nginx - a true alternative to Apache ?
Not all Apache modules
mod_auth_*
mod_dav*
…
Basic modules are available
Some 3 rd party modules (needs recompilation !)

Nginx - Installation
Packages
Win32 binaries
Build from source (./configure; make; make install)

Nginx - Configuration server { listen 80; server_name www.phpbenelux.eu *.phpbenelux.eu; index index.html; root /home/phpbenelux.eu/www; } server { listen 80; server_name conference.phpbenelux.eu; index index.html; root /home/phpbenelux.eu/conference; }

Nginx - Configuration server { listen 80; server_name www.phpbenelux.eu; root /home/phpbenelux.eu/www; location /conference { # We capture the URI and redirect it to the subdomain. rewrite conference(.*) http://conference.phpbenelux.eu$1 permanent; } }

Nginx - phase 1
Move Apache to a different port (8080)
Put Nginx at port 80
Nginx serves all statics (images, css, js, …)
Forward dynamic requests to Apache

Nginx for static files only server { listen 80; server_name www.phpbenelux.eu; location ~* ^.*.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|pdf|ppt|txt|tar|rtf|js)$ { expires 30d; root /home/www.phpbenelux.eu; } location / { proxy_pass http://www.phpbenelux.eu:8080; proxy_pass_header Set-Cookie; proxy_set_header X-Real-IP $remote_addr; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } }

Nginx for PHP ?
Bottleneck = PHP ? Keep it in Apache
Bottleneck = memory ? Go for it !
LANMMP to... LNMPP
(ok, this is getting ridiculous)

Nginx with PHP
In the past : spawn-fcgi (from Lighttpd)
Now : PHP-FPM (in PHP 5.3.3 !)
Runs on port 9000
Nginx connects using fastcgi method
location / { fastcgi_pass 127.0.0.1:9000; fastcgi_index index.php; include fastcgi_params; fastcgi_param SCRIPT_NAME $fastcgi_script_name; fastcgi_param SCRIPT_FILENAME /home/www.phpbenelux.eu/$fastcgi_script_name; fastcgi_param SERVER_NAME $host; fastcgi_intercept_errors on; }

Nginx + PHP-FPM features
Graceful upgrade
Spawn new processes under high load
Chroot
Slow request log !

Nginx + PHP-FPM features
Graceful upgrade
Spawn new processed under high load
Chroot
Slow request log !
fastcgi_finish_request() -> offline processing

Nginx + PHP-FPM - Downside
PHP vs PHP-FPM = no performance difference
But : Apache + mod_php = faster
Why ? fastcgi connection overhead
Difference : 10-15%
Balance : Apache's memory overhead vs. fastcgi performance

Varnish
Not just a load balancer
Reverse proxy cache / http accelerator / …
Caches (parts of) pages in memory
Careful :
uses threads
Nginx might be faster (but doesn't have VCL)

Varnish - Installation & configuration
Installation
Packages
Source : ./configure && make && make install
Configuration
/etc/default/varnish
/etc/varnish/*.vcl

Varnish - backends + load balancing backend server1 { .host = "192.168.0.10"; } backend server2{ .host = "192.168.0.11"; } director example_director round-robin { { .backend = server1; } { .backend = server2; } }

Varnish - backends + load balancing backend server1 { .host = "192.168.0.10"; .probe = { .url = "/"; .interval = 5s; .timeout = 1 s; .window = 5; .threshold = 3; } }

Varnish - VCL
Varnish Configuration Language
DSL (Domain Specific Language)
-> compiled to C
Hooks into each request
Defines :
Backends (web servers)
ACLs
Load balancing strategy
Can be reloaded while running

Varnish - whatever you want
Extremely flexible through VCL
Manipulate headers, request, response, ...
Introduce ACL
Detect flooding (& ban visitors)
Lots of hooks you can plug code onto
Real-time statistics (varnishtop, varnishhist, ...)
ESI

Varnish - ESI
Perfect for caching pages
In your article page output : <esi:include src="/latest-news"/>

Varnish - what can/can't be cached ?
Can :
Static pages
Images, js, css
Pages or parts of pages that don't change often (ESI)
Can't :
POST requests
Requests with Set-Cookie
Very large files (it's not a file server !)
User-specific content

Varnish - want more ?
-> "Varnish, the high performance valhalla ?"
by Jeroen Van Dijck Tomorrow @ 13h40

Apache - tuning tips
Disable unused modules -> fixes 10% of performance issues
Set AllowOverride to None . Enable only where needed !
Disable SymLinksIfOwnerMatch . Enable only where needed !
MinSpareServers, MaxSpareServers, StartServers, MaxClients, MPM selection -> a whole session of its own ;-)
Don't mod_proxy -> use Nginx or Varnish !
High load on an SSL-site ? -> put SSL on a reverse proxy

PHP speed - some tips
Upgrade PHP - every minor release has 5-15% speed gain !
Profile your code
XHProf
Xdebug
But : turn off profilers on acceptance/production platforms !

KCachegrind is your friend

PHP speed - some tips
Most performance issues are in DB queries -> look there first !
Log PHP errors and review those logs !
Shorter code != faster code -> keep your code readable !
Hardware cost < Manpower cost
-> 1 more server < 30 mandays of labor
Keep micro-optimizations in code = last thing on list

DB speed - some tips
Avoid NOW() -> use PHP date("Y-m-d") as a parameter
Why ? Query cache !
Index, index, index ! (where needed only)
Use same types for joins
i.e. don't join decimal with int
RAND() is evil !
Select the right storage engine
Don't use mysql_pconnect()
Other tips -> see slides "MySQL performance tuning"
by Geert Vanderkelen

One more for the backend... HipHop
Developed @ Facebook (runs 95% of servers)
Source code transformer
Converts PHP to C++
Compiles C++ (using g++) into binary code
Has a built-in Webserver
Not for you, unless > 1.000.000 views/day
Want more ?
-> "HipHop for PHP" - Tomorrow @ 10h10 (Scott McVicar)

Caching & Tuning @ frontend http://www.websiteoptimization.com/speed/tweak/average-web-page/

Caching in the browser
HTTP 304 (Not modified)
Expires/Cache-Control header
2 notes :
Don't use POST if you want to cache
Don't cache user-specific pages in browser (security !)

HTTP 304 First request Next requests

HTTP 304 with ETag First request Next requests

Expires/Cache-control header
Cache-Control
HTTP/1.1
Seconds to expiry
Used by browsers
First request Next requests No requests until item expires
Expires
HTTP/1.0
Date to expire on
Used by old proxies
Requires clock to be accurate !

Pragma: no-cache = evil
"Pragma: no cache" doesn't make it uncacheable
Don't want caching on a page ?
HTTP/1.0 : "Expires : Fri, 30 Oct 1998 00:00:00 GMT" (in the past)
HTTP/1.1 : "Cache-Control: no-store"

Frontend tuning
1. You optimize backend 2. Frontend engineers messes up -> havoc on backend 3. Don't forget : frontend sends requests to backend !

Frontend tuning
All frontend code impacts backend !
Care about frontend
Test frontend
Check what requests frontend sends to backend

Tuning frontend
Minimize requests
Combine CSS/JavaScript files
Use inline images in CSS/XHTML (not supported on all browsers yet)

Frontend tuning - inline CSS/XHTML images #navbar span { width: 31px; height: 31px; display: inline; float: left; margin-right: 4px; } .home { background-image: url(data:image/gif;base64,R0lGODlhHwAfAPcAAAAAAIxKAKVjCLW1tb29tcbGvc7OxtZ7ANbWztbW1tbe1t7e1uelMefn1ufn3ufn5+fv3u+MAO/v5+/v7/fGCPf35/f37//nY////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////........MEl0nGVUC6tObNnPceSFBaQVMJAxC4lo3gNOrUaFnTHoAxNm3XVxPfRq139e8BEGAjWD5bgIALw287T8AcAXLly2kjOACdc17higXSIKDO/Lpv7Qq4bw7APgBq8eOzX69InrZ6xe3dbxZffyTGkb8tdx8F+b0Xn2sFsCSBAgTM5lp63RHYnoHUudZgRgkGOGCB+43nGk4OGcQTabKx5dyJKJ7ImoUNCaRRAZYN1ppsrT3Y2gIwyjSQBAtUpABml/0IJGYd6VjQUDH9uBFkGxGm5I8dPQaRUAQUMBdhhBV25ZYUJZBcSAtSJBddWZZ5UAGPOTXlgkNVOSZdBxEwIkYu7VhYnAol5GaadRqF0Uaz0TgXnX2umVFyGakJUUAAADs=); margin-left: 4px; } <img border=0 src="data:image/gif;base64,R0lGODlhHwAfAPcAAAAAAIxKAKVjCLW1tb29tcbGvc7OxtZ7ANbWztbW1tbe1t7e1uelMefn1ufn3ufn5+fv3u+MAO/v5+/v7/fGCPf35/f37//nY/......Uaz0TgXnX2umVFyGakJUUAAADs=">

Tuning frontend
Minimize requests
Use inline images in CSS (not supported on all browsers yet)
Use CSS Sprites

Tuning content - CSS sprites 11 images 11 HTTP requests 24KByte 1 images 1 HTTP requests 14KByte

Tuning content - CSS sprites

Tuning frontend
Minimize requests
Use inline images in CSS (not supported on all browsers yet)
Use CSS Sprites (horizontally if possible)
Put CSS at top
Put JavaScript at bottom
Max. no connections
Especially if JavaScript does Ajax (advertising-scripts, …) !
Avoid iFrames
Again : max no. of connections
JavaScript onLoad() won't be triggered until loaded !
Don't scale images in HTML
Have a favicon.ico and cache it (don't 404 it !)

Tuning frontend
Don't use inline CSS/JavaScript
CSS/JavaScript need to be external files (minified, merged)
Why ? -> Cacheable by browser / reverse proxy
Use GET for Ajax retrieval requests (and cache them !)
Optimize images (average 50-60% !)
Split requests across subdomains
Put statics on a separate subdomain (without cookies !)
www.phpbenelux.eu www.phpbenelux.eu images.phpbenelux.eu

Tuning miscellaneous
Avoid DNS lookups
Frontend : don't use too many subdomains (2 = ideal)
Backend :
Turn off DNS resolution in Apache : HostnameLookups Off
If your app uses external data
Run a local DNS cache (timeout danger !)
Make sure you can trust DNS servers (preferable run your own)
Compress non-binary content (GZIP)
mod_deflate in Apache
HttpGzipModule in Nginx (HttpGzipStaticModule for pre-zipped statics !)
No native support in Varnish

What else can kill your site ?
Redirect loops
Multiple requests
More load on Webserver
More PHP to process
Additional latency for visitor
Try to avoid redirects anyway
-> In ZF : use $this->_forward instead of $this->_redirect
Watch your logs, but equally important...
Watch the logging process ->
Logging = disk I/O -> can kill your site !
Slashdot effect

Above all else... be prepared !
Have a monitoring system
Use a cache abstraction layer (disk -> APC -> Memcache)
Don't install for the worst -> prepare for the worst
Code abstraction
Code separation
Have a plan
Have a test-setup
Be ready to go cloud (experiment !)
Look at business critical level
Have fallbacks
Turn off non-critical functionality
Turn on queueing for non-critical updates

Interested in actual benchmark results ?
Join me for a Webcast with :
Benchmark of an actual site
Step-by-step implementation of caching, Nginx and lots of tuning
See results for yourself
Date : first half of March
Follow me :
Twitter : @wimgtr
Blog : http://techblog.wimgodden.be

Questions ?

Cu.be Solutions
Founded in 2010
Spinoff of FirstLink Networks (founded in 2000)
High-quality open source solutions
Centered around PHP
Contribute to open source projects
(ZF, PHPUnit, PHP_CodeSniffer, OpenX, ...)

Cu.be Solutions - we're hiring !
PHP developers
Go beyond just PHP -> entire PHP ecosystem
Work with latest tools
Exciting projects (Webservices, mobile, high-traffic, ...)
Focus on highest possible quality !
Interested ?
-> http://cu.be/jobs
-> info@cu.be

Contact
Web http://techblog.wimgodden.be
Slides http://www.slideshare.net/wimg
Twitter @wimgtr
E-mail [email_address]

Please...
Rate my talk (evaluation forms)
Rate my talk : http://joind.in/talk/view/2491

Thanks !

http://lanyrd.com	25
http://bayultimate.wordpress.com	19
http://stream.puschkasch.com	11
http://tdpsk.soup.io	7
http://blog.fasoulas.com	4
https://status.ogame1026.de	2

Caching and tuning fun for high scalability @ phpBenelux 2011

by Wim Godden

Accessibility

Categories

Upload Details

Usage Rights

6 Embeds 68

Statistics

Caching and tuning fun for high scalability @ phpBenelux 2011 Presentation Transcript