1. Caching and tuning fun for high scalability Wim Godden Cu.be Solutions Conference Jan 28, 2011

4. Owner of Cu.be Solutions (http://cu.be)

5. PHP developer since 1997

6. Developer of OpenX

7. Zend Certified Engineer

8. Zend Framework Certified Engineer

9. MySQL Certified Developer

11. System/network engineers ?

12. Managers ?

13. Caching experience ?

14. Caching and tuning fun for high scalability Wim Godden Cu.be Solutions Conference Jan 28, 2011

16. A few techniques

17. -> Increase reliability, performance and scalability

18. 5 visitors/day -> 5 million visitors/day

19. (Don't expect miracle cure !)

21. How to do stuff

22. How NOT to do stuff

23. LAMP

24. LAMP

25. LAMP

26. LAMP

27. Architecture

28. Caching

29. What's caching ?

30. What is caching ? select * from article join user on article.user_id = user.id order by created desc limit 10

32. Reduce the load

33. Caching goals - 2 nd goal

35. Avg. loading time : 200ms

36. Cache 10 pages

37. Avg. loading time : 20ms

38. -> Total avg. loading time : 56ms

39. Worth it ?

41. Users benefit -> faster page load

42. Wait a second... that's mostly frontend stuff !

43. Theory of caching DB

44. Theory of caching DB

46. Blogs

47. Full pages that don't change

48. Render -> Store in cache -> retrieve from cache

51. Usually a small block in a page

52. Best effect : reused on lots of pages

53. Can be inserted on dynamic pages

57. Resets on every insert/update/delete

60. store group of objects

64. XML file parsing

68. Modification frequency ?

69. Retrieval frequency ?

72. Make a complete query log (don't forget to turn it off !)

73. Check page loading times

75. Don't rely on it

78. lots of different queries

80. In MySQL : memory/heap table

83. Can be joined with disk-based tables

85. master-slave = trouble

89. Will be in PHP core in 5.4 ? 5.5 ? 6.0 ?

91. Zend Accelerator

93. Will be in PHP core in 5.4 ? 5.5 ? 6.0 ?

95. Zend Accelerator

96. X-Cache

97. WinCacheForPhp PHP PHP + APC PHP + eAccelerator 42.18 req/sec 206.20 req/sec 211.49 req/sec

99. Caching SQL queries : preferably not

101. high latency

102. locking issues !

104. But : OS file caching makes difference minimal

107. -> Hard to scale

110. Store user data in memory

111. apc_add / apc_store to add/update

112. apc_fetch to retrieve

113. apc_delete

114. Fast -> huge performance impact

116. Store user data in memory

117. apc_add / apc_store to add/update

118. apc_fetch to retrieve

119. apc_delete

120. Fast -> huge performance impact

121. Session support !

123. restart Apache -> cache = empty

125. Distributed memory caching system

127. Values - max. 1Mbyte

129. Distributed memory caching system

130. Multiple machines ↔ 1 big memory-based hash-table

133. Memcache - where to install

134. Memcache - where to install

136. PECL

138. memcached -d -m <mem> -l <ip> -p <port>

139. ex. : memcached -d -m 2048 -l 127.0.0.1 -p 11211

141. Lose shopping cart data

142. ...

144. Lose shopping cart data

147. Caching in PHP <?php $memcache = new Memcache(); $memcache->addServer( '172.16.0.1' , 11211); $memcache->addServer( '172.16.0.2' , 11211); $myData = $memcache->get( 'myKey' ); if ($myData === false ) { $myData = GetMyDataFromDB(); // Put it in Memcache as 'myKey', without compression, with no expiration $memcache->set( 'myKey' , $myData, false , 0); } echo $myData;

151. Slab 2 : 480 bytes (400 * 1.2)

153. Each larger slab has room for fewer items (chunks)

154. -> Store a lot of very large objects

155. -> Large slabs might be full

156. -> Rest of slabs might be free

157. -> Try to store more -> eviction of data !

159. Use Cacti or other monitoring tools STAT pid 2941 STAT uptime 10878 STAT time 1296074240 STAT version 1.4.5 STAT pointer_size 64 STAT rusage_user 20.089945 STAT rusage_system 58.499106 STAT curr_connections 16 STAT total_connections 276950 STAT connection_structures 96 STAT cmd_get 276931 STAT cmd_set 584148 STAT cmd_flush 0 STAT get_hits 211106 STAT get_misses 65825 STAT delete_misses 101 STAT delete_hits 276829 STAT incr_misses 0 STAT incr_hits 0 STAT decr_misses 0 STAT decr_hits 0 STAT cas_misses 0 STAT cas_hits 0 STAT cas_badval 0 STAT auth_cmds 0 STAT auth_errors 0 STAT bytes_read 613193860 STAT bytes_written 553991373 STAT limit_maxbytes 268435456 STAT accepting_conns 1 STAT listen_disabled_num 0 STAT threads 4 STAT conn_yields 0 STAT bytes 20418140 STAT curr_items 65826 STAT total_items 553856 STAT evictions 0 STAT reclaimed 0

160. Memcache - backing up

161. Memcache - deleting <?php $memcache = new Memcache(); $memcache->delete( 'myKey' ); $myData = $memcache->get( 'myKey' ); // $myData === false

162. Memcache - caching a page <?php $output = $memcache->get( 'page_' . $page_id); if ($output === false ) { ob_start(); GetMyPageInRegularWay($page_id); $output = ob_get_contents(); ob_end_clean(); $memcache->set( 'page_' . $page_id, $output, false , 600); // Cache 10 mins } echo $output;

165. Prefix / namespace your keys !

166. Only letters, numbers and underscore

168. Document your key names !

169. Adding/updating data

170. Adding/updating data

171. Adding/updating data $memcache->delete( 'ArticleDetails__Toshiba_32C100U_32_Inch' ); $memcache->delete( 'Homepage_Popular_Product_List' );

172. Adding/updating data

173. Adding/updating data - Why it crashed

174. Adding/updating data - Why it crashed

175. Adding/updating data - Why it crashed

176. Cache stampeding

177. Cache stampeding

178. Memcache code ? DB

179. Memcache code ? DB

180. Memcache code ? DB

181. Creating cache methods static protected function getArticle ($id) { $cache = Zend_Registry:: get ( 'Zend_Cache' ); if (!$article = $cache->load( 'article_' . $id )) { $db = Zend_Registry:: get ( 'Zend_DB' ); $query = ' select article.id as art_id, article.title as art_title, article.body as art_body, date_format(article.created, &quot;%d/%m/%Y %H:%i&quot;) as art_created, user.username as use_username from article join user on article.user_id = user.id where article.id = ? ' ; $artice = $db->fetchRow($query, $id); $cache->save($articleList, 'article_' . $id ); } return $articleList; }

182. Creating cache methods static protected function getArticleUncached ($id) { $db = Zend_Registry:: get ( 'Zend_DB' ); $query = ' select article.id as art_id, article.title as art_title, article.body as art_body, date_format(article.created, &quot;%d/%m/%Y %H:%i&quot;) as art_created, user.username as use_username from article join user on article.user_id = user.id where article.id = ? ' ; return $db->fetchRow($query, $id); } static public function getArticle ($id) { if (Zend_Registry:: isRegistered ( 'Zend_Cache' )) { $cache = Zend_Registry:: get ( 'Zend_Cache' ); if (!$articleList = $cache->load( 'article_' . $id )) { $articleList = self :: getArticleUncached ($id); $cache->save($articleList, 'article_' . $id ); } } else { $articleList = self :: getArticleUncached ($id); } return $articleList; }

184. -> lots of new connections

185. -> memory spike

186. What if the PHP process that created the lock fails ?

187. Avoids needing caching code in admin part

188. -> but what about warmup script(s) ?

190. Run it before starting Webserver !

194. Exception to the rule : things that have an end date (calendar items)

197. Move to Memcache later

198. But : is your code ready ?

199. -> Use a component like Zend_Cache to switch easily !

203. Reverse proxy

204. Lightweight, fast

205. 7.5% of all Websites

207. Uses epoll / kqueue

208. Low memory footprint

209. 10000 active connections = normal

211. mod_dav*

213. Some 3 rd party modules (needs recompilation !)

215. Win32 binaries

216. Build from source (./configure; make; make install)

217. Nginx - Configuration server { listen 80; server_name www.phpbenelux.eu *.phpbenelux.eu; index index.html; root /home/phpbenelux.eu/www; } server { listen 80; server_name conference.phpbenelux.eu; index index.html; root /home/phpbenelux.eu/conference; }

218. Nginx - Configuration server { listen 80; server_name www.phpbenelux.eu; root /home/phpbenelux.eu/www; location /conference { # We capture the URI and redirect it to the subdomain. rewrite conference(.*) http://conference.phpbenelux.eu$1 permanent; } }

220. Put Nginx at port 80

221. Nginx serves all statics (images, css, js, …)

222. Forward dynamic requests to Apache

224. Put Nginx at port 80

225. Nginx serves all statics (images, css, js, …)

226. Forward dynamic requests to Apache

227. Nginx for static files only server { listen 80; server_name www.phpbenelux.eu; location ~* ^.*(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|pdf|ppt|txt|tar|rtf|js)$ { expires 30d; root /home/www.phpbenelux.eu; } location / { proxy_pass http://www.phpbenelux.eu:8080; proxy_pass_header Set-Cookie; proxy_set_header X-Real-IP $remote_addr; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } }

229. Bottleneck = memory ? Go for it !

230. LANMMP to... LNMPP

231. (ok, this is getting ridiculous)

233. Now : PHP-FPM (in PHP 5.3.3 !)

234. Runs on port 9000

235. Nginx connects using fastcgi method location / { fastcgi_pass 127.0.0.1:9000; fastcgi_index index.php; include fastcgi_params; fastcgi_param SCRIPT_NAME $fastcgi_script_name; fastcgi_param SCRIPT_FILENAME /home/www.phpbenelux.eu/$fastcgi_script_name; fastcgi_param SERVER_NAME $host; fastcgi_intercept_errors on; }

237. Now : PHP-FPM (in PHP 5.3.3 !)

238. Runs on port 9000

239. Nginx connects using fastcgi method location / { fastcgi_pass 127.0.0.1:9000; fastcgi_index index.php; include fastcgi_params; fastcgi_param SCRIPT_NAME $fastcgi_script_name; fastcgi_param SCRIPT_FILENAME /home/www.phpbenelux.eu/$fastcgi_script_name; fastcgi_param SERVER_NAME $host; fastcgi_intercept_errors on; }

241. Spawn new processes under high load

242. Chroot

243. Slow request log !

245. Spawn new processed under high load

246. Chroot

247. Slow request log !

248. fastcgi_finish_request() -> offline processing

250. But : Apache + mod_php = faster

251. Why ? fastcgi connection overhead

252. Difference : 10-15%

253. Balance : Apache's memory overhead vs. fastcgi performance

254. LAMP

255. LAMP

257. Reverse proxy cache / http accelerator / …

258. Caches (parts of) pages in memory

260. Nginx might be faster (but doesn't have VCL)

263. /etc/varnish/*.vcl

264. Varnish - backends + load balancing backend server1 { .host = &quot;192.168.0.10&quot;; } backend server2{ .host = &quot;192.168.0.11&quot;; } director example_director round-robin { { .backend = server1; } { .backend = server2; } }

265. Varnish - backends + load balancing backend server1 { .host = &quot;192.168.0.10&quot;; .probe = { .url = &quot;/&quot;; .interval = 5s; .timeout = 1 s; .window = 5; .threshold = 3; } }

269. ACLs

272. Introduce ACL

273. Detect flooding (& ban visitors)

275. ESI

278. Images, js, css

280. Requests with Set-Cookie

281. Very large files (it's not a file server !)

282. User-specific content

285. Set AllowOverride to None . Enable only where needed !

286. Disable SymLinksIfOwnerMatch . Enable only where needed !

287. MinSpareServers, MaxSpareServers, StartServers, MaxClients, MPM selection -> a whole session of its own ;-)

288. Don't mod_proxy -> use Nginx or Varnish !

289. High load on an SSL-site ? -> put SSL on a reverse proxy

293. KCachegrind is your friend

295. Log PHP errors and review those logs !

296. Shorter code != faster code -> keep your code readable !

300. Select the right storage engine

301. Don't use mysql_pconnect()

302. Other tips -> see slides &quot;MySQL performance tuning&quot;

303. by Geert Vanderkelen

305. Source code transformer

306. Converts PHP to C++

307. Compiles C++ (using g++) into binary code

308. Has a built-in Webserver

309. Not for you, unless > 1.000.000 views/day

311. Caching & Tuning @ frontend http://www.websiteoptimization.com/speed/tweak/average-web-page/

313. Expires/Cache-Control header

315. Don't cache user-specific pages in browser (security !)

316. HTTP 304 First request Next requests

317. HTTP 304 with ETag First request Next requests

319. Seconds to expiry

321. Date to expire on

322. Used by old proxies

323. Requires clock to be accurate !

326. HTTP/1.1 : &quot;Cache-Control: no-store&quot;

329. Test frontend

330. Check what requests frontend sends to backend

332. Use inline images in CSS/XHTML (not supported on all browsers yet)

333. Frontend tuning - inline CSS/XHTML images #navbar span { width: 31px; height: 31px; display: inline; float: left; margin-right: 4px; } .home { background-image: url(data:image/gif;base64,R0lGODlhHwAfAPcAAAAAAIxKAKVjCLW1tb29tcbGvc7OxtZ7ANbWztbW1tbe1t7e1uelMefn1ufn3ufn5+fv3u+MAO/v5+/v7/fGCPf35/f37//nY////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////........MEl0nGVUC6tObNnPceSFBaQVMJAxC4lo3gNOrUaFnTHoAxNm3XVxPfRq139e8BEGAjWD5bgIALw287T8AcAXLly2kjOACdc17higXSIKDO/Lpv7Qq4bw7APgBq8eOzX69InrZ6xe3dbxZffyTGkb8tdx8F+b0Xn2sFsCSBAgTM5lp63RHYnoHUudZgRgkGOGCB+43nGk4OGcQTabKx5dyJKJ7ImoUNCaRRAZYN1ppsrT3Y2gIwyjSQBAtUpABml/0IJGYd6VjQUDH9uBFkGxGm5I8dPQaRUAQUMBdhhBV25ZYUJZBcSAtSJBddWZZ5UAGPOTXlgkNVOSZdBxEwIkYu7VhYnAol5GaadRqF0Uaz0TgXnX2umVFyGakJUUAAADs=); margin-left: 4px; } <img border=0 src=&quot;data:image/gif;base64,R0lGODlhHwAfAPcAAAAAAIxKAKVjCLW1tb29tcbGvc7OxtZ7ANbWztbW1tbe1t7e1uelMefn1ufn3ufn5+fv3u+MAO/v5+/v7/fGCPf35/f37//nY/......Uaz0TgXnX2umVFyGakJUUAAADs=&quot;>

335. Use inline images in CSS (not supported on all browsers yet)

336. Use CSS Sprites

337. Tuning content - CSS sprites 11 images 11 HTTP requests 24KByte 1 images 1 HTTP requests 14KByte

338. Tuning content - CSS sprites

340. Use inline images in CSS (not supported on all browsers yet)

345. Have a favicon.ico and cache it (don't 404 it !)

348. Optimize images (average 50-60% !)

349. Split requests across subdomains

350. Put statics on a separate subdomain (without cookies !) www.phpbenelux.eu www.phpbenelux.eu images.phpbenelux.eu

355. HttpGzipModule in Nginx (HttpGzipStaticModule for pre-zipped statics !)

356. No native support in Varnish

359. Try to avoid redirects anyway

361. Watch the logging process ->

362. Logging = disk I/O -> can kill your site !

363. Slashdot effect

365. Use a cache abstraction layer (disk -> APC -> Memcache)

367. Code separation

368. Have a plan

369. Have a test-setup

370. Be ready to go cloud (experiment !)

372. Turn on queueing for non-critical updates

374. Step-by-step implementation of caching, Nginx and lots of tuning

377. Blog : http://techblog.wimgodden.be

380. Spinoff of FirstLink Networks (founded in 2000)

381. High-quality open source solutions

382. Centered around PHP

383. Contribute to open source projects

384. (ZF, PHPUnit, PHP_CodeSniffer, OpenX, ...)

386. Go beyond just PHP -> entire PHP ecosystem

387. Work with latest tools

388. Exciting projects (Webservices, mobile, high-traffic, ...)

389. Focus on highest possible quality !

390. Interested ?

391. -> http://cu.be/jobs

392. -> info@cu.be

394. Slides http://www.slideshare.net/wimg

395. Twitter @wimgtr

396. E-mail [email_address]

398. Rate my talk : http://joind.in/talk/view/2491