1.
Caching and tuning fun for high scalability Wim Godden Cu.be Solutions

2.
Who am I ? <ul><li>Wim Godden (@wimgtr)

3.
Owner of Cu.be Solutions (http://cu.be)

4.
PHP developer since 1997

5.
Developer of OpenX

6.
Zend Certified Engineer

7.
Zend Framework Certified Engineer

8.
MySQL Certified Developer </li></ul>

9.
Who are you ? <ul><li>Developers ?

10.
System/network engineers ?

11.
Managers ?

12.
Caching experience ? </li></ul>

13.
Caching and tuning fun for high scalability Wim Godden Cu.be Solutions

14.
Goals of this tutorial <ul><li>Everything about caching and tuning

15.
A few techniques </li><ul><li>How-to

16.
How-NOT-to </li></ul><li>-> Increase reliability, performance and scalability

17.
5 visitors/day -> 5 million visitors/day

18.
(Don't expect miracle cure !) </li></ul>

19.
LAMP

20.
Architecture

21.
Our base benchmark <ul><li>Apachebench = useful enough

22.
Result ? </li></ul>

23.
Caching

24.
What is caching ?

25.
What is caching ? select * from article join user on article.user_id = user.id order by created desc limit 10

26.
Theory of caching if ($data == false) DB

27.
Theory of caching DB

28.
Caching techniques <ul>#1 : Store entire pages #2 : Store part of a page (block) #3 : Store data retrieval (SQL ?) #4 : Store complex processing result #? : Your call ! </ul><ul>When you have data, think : <li>Creating time ?

29.
Modification frequency ?

30.
Retrieval frequency ? </li></ul>

31.
How to find cacheable data <ul><li>New projects : start from 'cache everything'

32.
Existing projects : </li><ul><li>Look at MySQL slow query log

33.
Make a complete query log (don't forget to turn it off !)

34.
Check page loading times </li></ul></ul>

35.
Caching storage - MySQL query cache <ul><li>Use it

36.
Don't rely on it

37.
Bad if you have : </li><ul><li>lots of insert/update/delete

38.
lots of different queries </li></ul></ul>

39.
Caching storage - Disk <ul><li>Data with few updates : good

40.
Caching SQL queries : preferably not

41.
DON'T use NFS or other network file systems </li><ul><li>especially for sessions

42.
locking issues !

43.
high latency </li></ul></ul>

44.
Caching storage - Disk / ramdisk <ul><li>Local </li><ul><li>5 Webservers -> 5 local caches

45.
-> Hard to scale

46.
How will you keep them synchronized ? </li><ul><li>-> Don't say NFS or rsync ! </li></ul></ul></ul>

47.
Caching storage - Memcache <ul><li>Facebook, Twitter, Slashdot, … -> need we say more ?

48.
Distributed memory caching system

49.
Multiple machines ↔ 1 big memory-based hash-table

50.
Key-value storage system </li><ul><li>Keys - max. 250bytes

51.
Values - max. 1Mbyte </li></ul></ul>

52.
Caching storage - Memcache <ul><li>Facebook, Twitter, Slashdot, … -> need we say more ?

53.
Distributed memory caching system

54.
Multiple machines ↔ 1 big memory-based hash-table

55.
Key-value storage system </li><ul><li>Keys - max. 250bytes

56.
Values - max. 1Mbyte </li></ul><li>Extremely fast... non-blocking, UDP (!) </li></ul>

57.
Memcache - where to install

58.
Memcache - where to install

59.
Memcache - installation & running it <ul><li>Installation </li><ul><li>Distribution package

60.
PECL

61.
Windows : binaries </li></ul><li>Running </li><ul><li>No config-files

62.
memcached -d -m <mem> -l <ip> -p <port>

63.
ex. : memcached -d -m 2048 -l 172.16.1.91 -p 11211 </li></ul></ul>

64.
Caching storage - Memcache - some notes <ul><li>Not fault-tolerant </li><ul><li>It's a cache !

65.
Lose session data

66.
Lose shopping cart data

67.
... </li></ul></ul>

68.
Caching storage - Memcache - some notes <ul><li>Not fault-tolerant </li><ul><li>It's a cache !

69.
Lose session data

70.
Lose shopping cart data

71.
… </li></ul><li>Firewall your Memcache port ! </li></ul>

72.
Memcache in code <?php $memcache = new Memcache(); $memcache->addServer( '172.16.0.1' , 11211); $memcache->addServer( '172.16.0.2' , 11211); $myData = $memcache->get( 'myKey' ); if ($myData === false ) { $myData = GetMyDataFromDB(); // Put it in Memcache as 'myKey', without compression, with no expiration $memcache->set( 'myKey' , $myData, false , 0); } echo $myData;

73.
Where's the data ? <ul><li>Memcache client decides (!)

74.
2 hashing algorithms : </li><ul><li>Traditional </li><ul><li>Server failure -> all data must be rehashed </li></ul><li>Consistent </li><ul><li>Server failure -> 1/x of data must be rehashed (x = # of servers) </li></ul></ul></ul>

75.
Benchmark with Memcache

76.
Memcache slabs <ul>(or why Memcache says it's full when it's not) <li>Multiple slabs of different sizes : </li><ul><li>Slab 1 : 400 bytes

77.
Slab 2 : 480 bytes (400 * 1.2)

78.
Slab 3 : 576 bytes (480 * 1.2) (and so on...) </li></ul><li>Multiplier (1.2 here) can be configured

79.
Store a lot of very large objects

80.
-> Large slabs : full

81.
-> Rest : free

82.
-> Eviction of data ! </li></ul>

83.
Memcache - Is it working ? <ul><li>Connect to it using telnet </li><ul><li>&quot;stats&quot; command ->

84.
Use Cacti or other monitoring tools </li></ul></ul>STAT pid 2941 STAT uptime 10878 STAT time 1296074240 STAT version 1.4.5 STAT pointer_size 64 STAT rusage_user 20.089945 STAT rusage_system 58.499106 STAT curr_connections 16 STAT total_connections 276950 STAT connection_structures 96 STAT cmd_get 276931 STAT cmd_set 584148 STAT cmd_flush 0 STAT get_hits 211106 STAT get_misses 65825 STAT delete_misses 101 STAT delete_hits 276829 STAT incr_misses 0 STAT incr_hits 0 STAT decr_misses 0 STAT decr_hits 0 STAT cas_misses 0 STAT cas_hits 0 STAT cas_badval 0 STAT auth_cmds 0 STAT auth_errors 0 STAT bytes_read 613193860 STAT bytes_written 553991373 STAT limit_maxbytes 268435456 STAT accepting_conns 1 STAT listen_disabled_num 0 STAT threads 4 STAT conn_yields 0 STAT bytes 20418140 STAT curr_items 65826 STAT total_items 553856 STAT evictions 0 STAT reclaimed 0

85.
Memcache - backing up

86.
Memcache - tip <ul>Page with multiple blocks ? -> use Memcached::getMulti() Warning : what if you get some hits and some misses ? </ul>

87.
Updating data

88.
Updating data LCD_Popular_Product_List

89.
Adding/updating data $memcache->delete( 'LCD_Popular_Product_List' );

90.
Adding/updating data

91.
Adding/updating data - Why it crashed

92.
Adding/updating data - Why it crashed

93.
Adding/updating data - Why it crashed

94.
Cache stampeding

95.
Cache stampeding

96.
Memcache code ? DB

97.
Cache warmup scripts <ul><li>Used to fill your cache when it's empty

98.
Run it before starting Webserver !

99.
2 ways : </li><ul><li>Visit all URLs </li><ul><li>Error-prone

100.
Hard to maintain </li></ul><li>Call all cache-updating methods </li></ul><li>Make sure you have a warmup script ! </li></ul>

101.
Cache stampeding - what about locking ? <ul>Seems like a nice idea, but... <li>While lock in place

102.
What if the process that created the lock fails ? </li></ul>

103.
LAMP... <ul>-> LAMMP -> LNMMP </ul>

104.
Nginx <ul><li>Web server

105.
Reverse proxy

106.
Lightweight, fast

107.
8.3% of all Websites </li></ul>

108.
Nginx <ul><li>No threads, event-driven

109.
Uses epoll / kqueue

110.
Low memory footprint

111.
10000 active connections = normal </li></ul>

112.
Nginx - a true alternative to Apache ? <ul><li>Not all Apache modules </li><ul><li>mod_auth_*

113.
mod_dav*

114.
… </li></ul><li>Basic modules are available

115.
Some 3 rd party modules (needs recompilation !) </li></ul>

116.
Nginx - Configuration server { listen 80; server_name www.domain.ext *.domain.ext; index index.html; root /home/domain.ext/www; } server { listen 80; server_name photo.domain.ext; index index.html; root /home/domain.ext/photo; }

117.
Nginx + PHP-FPM - performance ?

118.
Reverse proxy time...

119.
Varnish <ul><li>Not just a load balancer

120.
Reverse proxy cache / http accelerator / …

121.
Caches (parts of) pages in memory

122.
Careful : </li><ul><li>uses threads (like Apache)

123.
Nginx usually scales better (but doesn't have VCL) </li></ul></ul>

124.
Varnish - backends + load balancing backend server1 { .host = &quot;192.168.0.10&quot;; } backend server2{ .host = &quot;192.168.0.11&quot;; } director example_director round-robin { { .backend = server1; } { .backend = server2; } }

125.
Varnish - VCL <ul><li>Varnish Configuration Language

126.
DSL (Domain Specific Language) </li><ul><li>-> compiled to C </li></ul><li>Hooks into each request

127.
Defines : </li><ul><li>Backends (web servers)

128.
ACLs

129.
Load balancing strategy </li></ul><li>Can be reloaded while running </li></ul>

130.
Varnish - whatever you want <ul><li>Real-time statistics (varnishtop, varnishhist, ...)

131.
ESI </li></ul>

132.
Varnish - ESI <ul>Perfect for caching pages </ul>In your article page output : <esi:include src=&quot;/news&quot;/> In your Varnish config : sub vcl_fetch { if (req.url == &quot;/news&quot;) { esi; /* Do ESI processing */ set obj.ttl = 2m; } elseif (req.url == &quot;/nav&quot;) { esi; set obj.ttl = 1m; } elseif …. … . }

133.
Varnish with ESI - hold on tight !

134.
Varnish - what can/can't be cached ? <ul><li>Can : </li><ul><li>Static pages

135.
Images, js, css

136.
Pages or parts of pages that don't change often (ESI) </li></ul><li>Can't : </li><ul><li>POST requests

137.
Very large files (it's not a file server !)

138.
Requests with Set-Cookie

139.
User-specific content </li></ul></ul>

140.
ESI -> no caching on user-specific content ? Logged in as : Wim Godden 5 messages TTL = 5min TTL=1h TTL = 0s ?

141.
Under development <ul><li>Release date </li><ul><li>Beta : Dec 2011

142.
Stable : Feb 2012 </li></ul></ul>

143.
Tuning

144.
PHP speed - some tips <ul><li>Upgrade PHP - every minor release has 5-15% speed gain !

145.
Use an opcode cache </li></ul>

146.
Caching storage - Opcode caching

147.
PHP speed - some tips <ul><li>Upgrade PHP - every minor release has 5-15% speed gain !

148.
Use an opcode cache

149.
Profile your code </li><ul><li>XHProf

150.
Xdebug </li></ul></ul>

151.
KCachegrind is your friend

152.
PHP speed - some tips <ul><li>Upgrade PHP - every minor release has 5-15% speed gain !

153.
Use an opcode cache

154.
Profile your code </li><ul><li>XHProf

155.
Xdebug </li></ul><li>But : turn off profilers on production platforms ! </li></ul>

156.
DB speed - some tips <ul><li>Use same types for joins </li><ul><li>i.e. don't join decimal with int </li></ul><li>RAND() is evil !

157.
count(*) is evil in InnoDB without a where clause ! </li><ul><li>(and there are other examples of specific things to avoid) </li></ul><li>Persistent connect is not always good ! </li></ul>

158.
Caching & Tuning @ frontend http://www.websiteoptimization.com/speed/tweak/average-web-page/

159.
Frontend tuning <ul>1. You optimize backend 2. Frontend engineers messes up -> havoc on backend 3. Don't forget : frontend sends requests to backend ! SO... <li>Care about frontend

160.
Test frontend

161.
Check what requests frontend sends to backend </li></ul>

162.
Tuning frontend <ul><li>Minimize requests </li><ul><li>Combine CSS/JavaScript files </li></ul></ul>

163.
Tuning frontend <ul><li>Minimize requests </li><ul><li>Combine CSS/JavaScript files

164.
Use CSS Sprites </li></ul></ul>

165.
CSS Sprites

166.
Tuning content - CSS sprites

167.
Tuning content - CSS sprites 11 images 11 HTTP requests 24KByte 1 image 1 HTTP requests 14KByte

168.
Tuning frontend <ul><li>Minimize requests </li><ul><li>Combine CSS/JavaScript files

169.
Use CSS Sprites (horizontally if possible) </li></ul><li>Put CSS at top

170.
Put JavaScript at bottom </li><ul><li>Max. no connections

171.
Especially if JavaScript does Ajax (advertising-scripts, …) ! </li></ul><li>Avoid iFrames </li><ul><li>Again : max no. of connections </li></ul><li>Don't scale images in HTML

172.
Have a favicon.ico (don't 404 it !) </li><ul><li>-> see my blog </li></ul></ul>

173.
What else can kill your site ? <ul><li>Redirect loops </li><ul><li>Multiple requests </li><ul><li>More load on Webserver

174.
More PHP to process </li></ul><li>Additional latency for visitor

175.
Try to avoid redirects anyway

176.
-> In ZF : use $this->_forward instead of $this->_redirect </li></ul><li>Watch your logs, but equally important...

177.
Watch the logging process ->

178.
Logging = disk I/O -> can kill your server !

179.
Slashdot effect </li></ul>

180.
Above all else... be prepared ! <ul><li>Have a monitoring system

181.
Use a cache abstraction layer (disk -> Memcache)

182.
Don't install for the worst -> prepare for the worst

183.
Have a test-setup

184.
Have fallbacks </li><ul><li>-> Turn off non-critical functionality </li></ul></ul>

185.
<ul>Questions ? </ul>

186.
<ul>Questions ? </ul>

187.
Contact <ul><li>Twitter @wimgtr

188.
Web http://techblog.wimgodden.be

189.
Slides http://www.slideshare.net/wimg

190.
E-mail [email_address]

191.
Please...

192.
Rate my talk : http://joind.in/4361 </li></ul>

193.
<ul>Thanks ! </ul><ul>Please... Rate my talk : http://joind.in/4361 </ul>