Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Caching and tuning fun for high scalability

9,395 views

Published on

Caching has been a 'hot' topic for a few years. But caching takes more than merely taking data and putting it in a cache : the right caching techniques can improve performance and reduce load significantly. But we'll also look at some major pitfalls, showing that caching the wrong way can bring down your site. If you're looking for a clear explanation about various caching techniques and tools like Memcached, Nginx and Varnish, as well as ways to deploy them in an efficient way, this talk is for you.

Published in: Technology
  • Be the first to comment

Caching and tuning fun for high scalability

  1. 1. Caching and tuning fun for high scalability Wim Godden Cu.be Solutions
  2. 2. Who am I ? Wim Godden (@wimgtr) Founder of Cu.be Solutions (http://cu.be) Open Source developer since 1997 Developer of OpenX, PHPCompatibility, Nginx SCL, ... Speaker at PHP and Open Source conferences
  3. 3. Who are you ? Developers ? System/network engineers ? Managers ? Caching experience ?
  4. 4. Goals of this tutorial Everything about caching and tuning A few techniques How-to How-NOT-to → Increase reliability, performance and scalability 5 visitors/day → 5 million visitors/day (Don't expect miracle cure !)
  5. 5. LAMP
  6. 6. Architecture
  7. 7. Test page 3 DB-queries select firstname, lastname, email from user where user_id = 5; select title, createddate, body from article order by createddate desc limit 5; select title, createddate, body from article order by score desc limit 5; Page just outputs result
  8. 8. Our base benchmark Apachebench = useful enough Result ? Single webserver Proxy Static PHP Static PHP Apache + PHP 3900 17.5 6700 17.5 Limit : CPU, network or disk Limit : database
  9. 9. CachingCaching
  10. 10. What is caching ? CACHECACHE
  11. 11. What is caching ? x = 5, y = 2 n = 50 Same result CACHECACHE select * from article join user on article.user_id = user.id order by created desc limit 10 Doesn't change all the time
  12. 12. Theory of caching DB Cache $data = get('key') false GET /page Page select data from table $data = returned result set('key', $data) if ($data == false)
  13. 13. Theory of caching DB Cache HIT
  14. 14. Caching goals - 1st goal Reduce # of concurrent request Reduce the load
  15. 15. Caching goals - 2nd goal
  16. 16. Some figures Pageviews : 5000 (4000 on 10 pages) Avg. loading time : 200ms Cache 10 pages Avg. loading time : 20ms → Total avg. loading time : 56ms Worth it ?
  17. 17. Caching goals - 3rd goal Send less data across the network / Internet You benefit → lower bill from upstream provider Users benefit → faster page load Wait a second... that's a frontend problem ! True, but remember : the backend is transmitting it !
  18. 18. Caching techniques #1 : Store entire pages Company Websites Blogs Full pages that don't change Render → Store in cache → retrieve from cache
  19. 19. Caching techniques #2 : Store parts of a page Most common technique Usually a small block in a page Best effect : reused on lots of pages Can be inserted on dynamic pages
  20. 20. Caching techniques #3 : Store SQL queries ↔ SQL query cache Limited in size
  21. 21. Caching techniques #3 : Store SQL queries ↔ SQL query cache Limited in size Resets on every insert/update/delete Server and connection overhead Goal : not to get rid of DB free up DB resources for more hits ! Better : store processed data instead of raw data store group of objects
  22. 22. Caching techniques #4 : Store complex PHP results Not just calculations CPU intensive tasks : Config file parsing XML file parsing Loading CSV in an array Save resources → more resources available
  23. 23. Caching techniques #xx : Your call Only limited by your imagination ! When you have data, think : Creating time ? Modification frequency ? Retrieval frequency ?
  24. 24. How to find cacheable data New projects : start from 'cache everything' Existing projects : Check page loading times Look at MySQL slow query log Make a complete query log (don't forget to turn it off !) → Use Percona Toolkit (pt-query-digest)
  25. 25. Databases - pt-query-digest # Profile # Rank Query ID Response time Calls R/Call Apdx V/M Item # ==== ================== ================ ===== ======= ==== ===== ========== # 1 0x543FB322AE4330FF 16526.2542 62.0% 1208 13.6806 1.00 0.00 SELECT output_option # 2 0xE78FEA32E3AA3221 0.8312 10.3% 6412 0.0001 1.00 0.00 SELECT poller_output poller_item # 3 0x211901BF2E1C351E 0.6811 8.4% 6416 0.0001 1.00 0.00 SELECT poller_time # 4 0xA766EE8F7AB39063 0.2805 3.5% 149 0.0019 1.00 0.00 SELECT wp_terms wp_term_taxonomy wp_term_relationships # 5 0xA3EEB63EFBA42E9B 0.1999 2.5% 51 0.0039 1.00 0.00 SELECT UNION wp_pp_daily_summary wp_pp_hourly_summary # 6 0x94350EA2AB8AAC34 0.1956 2.4% 89 0.0022 1.00 0.01 UPDATE wp_options # MISC 0xMISC 0.8137 10.0% 3853 0.0002 NS 0.0 <147 ITEMS>
  26. 26. Databases - pt-query-digest # Query 2: 0.26 QPS, 0.00x concurrency, ID 0x92F3B1B361FB0E5B at byte 14081299 # This item is included in the report because it matches --limit. # Scores: Apdex = 1.00 [1.0], V/M = 0.00 # Query_time sparkline: | _^ | # Time range: 2011-12-28 18:42:47 to 19:03:10 # Attribute pct total min max avg 95% stddev median # ============ === ======= ======= ======= ======= ======= ======= ======= # Count 1 312 # Exec time 50 4s 5ms 25ms 13ms 20ms 4ms 12ms # Lock time 3 32ms 43us 163us 103us 131us 19us 98us # Rows sent 59 62.41k 203 231 204.82 202.40 3.99 202.40 # Rows examine 13 73.63k 238 296 241.67 246.02 10.15 234.30 # Rows affecte 0 0 0 0 0 0 0 0 # Rows read 59 62.41k 203 231 204.82 202.40 3.99 202.40 # Bytes sent 53 24.85M 46.52k 84.36k 81.56k 83.83k 7.31k 79.83k # Merge passes 0 0 0 0 0 0 0 0 # Tmp tables 0 0 0 0 0 0 0 0 # Tmp disk tbl 0 0 0 0 0 0 0 0 # Tmp tbl size 0 0 0 0 0 0 0 0 # Query size 0 21.63k 71 71 71 71 0 71 # InnoDB: # IO r bytes 0 0 0 0 0 0 0 0 # IO r ops 0 0 0 0 0 0 0 0 # IO r wait 0 0 0 0 0 0 0 0 # pages distin 40 11.77k 34 44 38.62 38.53 1.87 38.53 # queue wait 0 0 0 0 0 0 0 0 # rec lock wai 0 0 0 0 0 0 0 0 # Boolean: # Full scan 100% yes, 0% no # String: # Databases wp_blog_one (264/84%), wp_blog_tw… (36/11%)... 1 more # Hosts # InnoDB trxID 86B40B (1/0%), 86B430 (1/0%), 86B44A (1/0%)... 309 more # Last errno 0 # Users wp_blog_one (264/84%), wp_blog_two (36/11%)... 1 more # Query_time distribution # 1us # 10us # 100us # 1ms # 10ms ################################################################ # 100ms # 1s # 10s+ # Tables # SHOW TABLE STATUS FROM `wp_blog_one ` LIKE 'wp_options'G # SHOW CREATE TABLE `wp_blog_one `.`wp_options`G # EXPLAIN /*!50100 PARTITIONS*/ SELECT option_name, option_value FROM wp_options WHERE autoload = 'yes'G
  27. 27. Caching storage - MySQL query cache Use it Don't rely on it Good if you have : lots of reads few different queries Bad if you have : lots of insert/update/delete lots of different queries
  28. 28. The problem with SQL query caching select id, name from someTable where x = 5; ← uncached select id, name from someTable where x = 5; ← cached update someTable set name="Jim" where x = 10; select id, name from someTable where x = 5; ← uncached Imagine : 500 select/sec, 10 updates/min → 10 cache purges per min 50 select/sec, 10 update/sec → 10 cache purge per sec
  29. 29. Caching storage - Database memory tables Tables stored in memory In MySQL : memory/heap table ↔ temporary table : memory tables are persistent temporary tables are session-specific Faster than disk-based tables Can be joined with disk-based tables But : default 16MByte limit master-slave = trouble if you don't need join → overhead of DB software So : don't use it unless you need to join
  30. 30. Caching storage - Opcode caching DO !
  31. 31. Caching storage - Opcode caching APC De-facto standard Will be in PHP core in 5.4 ? 5.5 ? 6.0 ? PECL or packages eAccelerator Zend Accelerator X-Cache WinCacheForPhp
  32. 32. Caching storage - Opcode caching APC De-facto standard until 5.4 PECL or packages Zend Optimizer+ Built-in with PHP 5.5 eAccelerator PHP PHP + APC 42.18 req/sec 206.20 req/sec
  33. 33. Caching storage - Disk Data with few updates : good Caching SQL queries : preferably not DON'T use NFS or other network file systems high latency possible problem for sessions : locking issues !
  34. 34. Caching storage - Memory disk (ramdisk) Usually faster than physical disk But : OS file caching makes difference minimal (if you have enough memory)
  35. 35. Caching storage - Disk / ramdisk Overhead : filesystem Limited number of files per directory → Subdirectories Local 5 Webservers → 5 local caches How will you keep them synchronized ? → Don't say NFS or rsync !
  36. 36. Caching storage - APC variable cache More than an opcode cache (PHP 5.5 → use APCu) Store user data in memory apc_add / apc_store to add/update apc_fetch to retrieve apc_delete Fast → huge performance impact Session support ! Downside : local storage → hard to scale restart Apache → cache = empty
  37. 37. Caching storage - Memcache(d) Facebook, Twitter, YouTube, … → need we say more ? Distributed memory caching system Multiple machines ↔ 1 big memory-based hash-table Key-value storage system Keys - max. 250bytes Values - max. 1Mbyte
  38. 38. Caching storage - Memcache(d) Facebook, Twitter, YouTube, … → need we say more ? Distributed memory caching system Multiple machines ↔ 1 big memory-based hash-table Key-value storage system Keys - max. 250bytes Values - max. 1Mbyte Extremely fast... non-blocking, UDP (!)
  39. 39. Memcache - where to install
  40. 40. Memcache - where to install
  41. 41. Memcache - installation & running it Installation Distribution package PECL Windows : binaries Running No config-files memcached -d -m <mem> -l <ip> -p <port> ex. : memcached -d -m 2048 -l 172.16.1.91 -p 11211
  42. 42. Caching storage - Memcache - some notes Not fault-tolerant It's a cache ! Lose session data Lose shopping cart data ...
  43. 43. Caching storage - Memcache - some notes Not fault-tolerant It's a cache ! Lose session data Lose shopping cart data … Different libraries Original : libmemcache New : libmemcached (consistent hashing, UDP, binary protocol, …) Firewall your Memcache port !
  44. 44. Memcache in code <?php $memcache = new Memcache(); $memcache->addServer('172.16.0.1', 11211); $memcache->addServer('172.16.0.2', 11211); $myData = $memcache->get('myKey'); if ($myData === false) { $myData = GetMyDataFromDB(); // Put it in Memcache as 'myKey', without compression, with no expiration $memcache->set('myKey', $myData, false, 0); } echo $myData;
  45. 45. Memcache in code <?php $memcache = new Memcache(); $memcache->addServer('172.16.0.1', 11211); $memcache->addServer('172.16.0.2', 11211); $myData = $memcache->get('myKey'); if ($memcache->getResultCode() == Memcached::RES_NOTSTORED) { $myData = GetMyDataFromDB(); // Put it in Memcache as 'myKey', without compression, with no expiration $memcache->set('myKey', $myData, false, 0); } echo $myData;
  46. 46. Benchmark with Memcache Single webserver Proxy Static PHP Static PHP Apache + PHP 3900 17.5 6700 17.5 Apache + PHP + MC 3900 55 6700 108
  47. 47. Where's the data ? Memcache client decides (!) 2 hashing algorithms : Traditional Server failure → all data must be rehashed Consistent Server failure → 1/x of data must be rehashed (x = # of servers) No replication !
  48. 48. Memcache slabs (or why Memcache says it's full when it's not) Multiple slabs of different sizes : Slab 1 : 40 bytes Slab 2 : 50 bytes (40 * 1.25) Slab 3 : 63 bytes (63 * 1.25) (and so on...) Multiplier (1.25 by default) can be configured Store a lot of objects of different sizes → Certain slabs : full → Other slabs : Mostly empty → Eviction of data !
  49. 49. Memcache - Is it working ? Connect to it using telnet "stats" command → Use Cacti or other monitoring tools STAT pid 2941 STAT uptime 10878 STAT time 1296074240 STAT version 1.4.5 STAT pointer_size 64 STAT rusage_user 20.089945 STAT rusage_system 58.499106 STAT curr_connections 16 STAT total_connections 276950 STAT connection_structures 96 STAT cmd_get 276931 STAT cmd_set 584148 STAT cmd_flush 0 STAT get_hits 211106 STAT get_misses 65825 STAT delete_misses 101 STAT delete_hits 276829 STAT incr_misses 0 STAT incr_hits 0 STAT decr_misses 0 STAT decr_hits 0 STAT cas_misses 0 STAT cas_hits 0 STAT cas_badval 0 STAT auth_cmds 0 STAT auth_errors 0 STAT bytes_read 613193860 STAT bytes_written 553991373 STAT limit_maxbytes 268435456 STAT accepting_conns 1 STAT listen_disabled_num 0 STAT threads 4 STAT conn_yields 0 STAT bytes 20418140 STAT curr_items 65826 STAT total_items 553856 STAT evictions 0 STAT reclaimed 0
  50. 50. Memcache - backing up
  51. 51. Memcache - deleting <?php $memcache = new Memcache(); $memcache->delete('myKey');
  52. 52. Memcache - caching a page <?php $output = $memcache->get('page_' . $page_id); if ($output === false) { ob_start(); GetMyPageInRegularWay($page_id); $output = ob_get_contents(); ob_end_clean(); $memcache->set('page_' . $page_id, $output, false, 600); // Cache 10 mins } echo $output;
  53. 53. Memcache - tip Page with multiple blocks ? → use Memcached::getMulti() But : what if you get some hits and some misses ? getMulti($array) Hashing algorithm
  54. 54. Naming your keys Key names must be unique Prefix / namespace your keys ! Only letters, numbers and underscore Why ? → Change caching layer md5() is useful → BUT : harder to debug Use clear names Document your key names !
  55. 55. Updating data
  56. 56. Updating data LCD_Popular_Product_List
  57. 57. Adding/updating data $memcache->delete('ArticleDetails__Toshiba_32C100U_32_Inch'); $memcache->delete('LCD_Popular_Product_List');
  58. 58. Adding/updating data
  59. 59. Adding/updating data - Why it crashed
  60. 60. Adding/updating data - Why it crashed
  61. 61. Adding/updating data - Why it crashed
  62. 62. Cache stampeding
  63. 63. Cache stampeding
  64. 64. Memcache code ? DB Visitor interface Admin interface Memcache code
  65. 65. Standard caching code public function getArticle($id) { $cache = Zend_Registry::get('Zend_Cache'); if (!$articleList = $cache->load('article_' . $id)) { $select = $this->db->select() ->from('article', array('id', 'title', 'body', 'created')) ->join('user', 'user.id = article.user_id', array('username')) ->where('article.id = ?', $id); $articleList = $db->fetchRow($select, $id); $cache->save($articleList, 'article_' . $id); } return $articleList; }
  66. 66. Standard caching code public function getArticleUncached($id) { $select = $this->db->select() ->from('article', array('id', 'title', 'body', 'created')) ->join('user', 'user.id = article.user_id', array('username')) ->where('article.id = ?', $id); return $db->fetchRow($select, $id); } public function getArticle($id) { $cache = Zend_Registry::get('Zend_Cache'); if (!$articleList = $cache->load('article_' . $id)) { $articleList = $this->getArticleUncached($id); $cache->save($articleList, 'article_' . $id); } return $articleList; } public function updateArticleCache($id) { $cache->save( $this->getArticleUncached($id), 'article_' . $id ); }
  67. 67. Cache warmup scripts Used to fill your cache when it's empty Run it before starting Webserver ! 2 ways : Visit all URLs Error-prone Hard to maintain Call all cache-updating methods Make sure you have a warmup script !
  68. 68. Cache stampeding - what about locking ? Seems like a nice idea, but... While lock in place What if the process that created the lock fails ?
  69. 69. Quick word about expiration General rule : don't let things expire Exception to the rule : things that have an end date (calendar items)
  70. 70. So... DON'T DELETE FROM CACHE & DON'T EXPIRE FROM CACHE (unless you know you'll never store it again)
  71. 71. Quick-tip Start small → disk or APC Move to Memcached/Redis/... later But : is your code ready ? → Use a component like Zend_Cache to switch easily !
  72. 72. Time for... a break (15 min) After the break : Byebye Apache Reverse proxying The importance of frontend ...
  73. 73. LAMP... → LAMMP → LANMMP
  74. 74. Nginx Web server Reverse proxy Lightweight, fast 12.89% of all Websites
  75. 75. Nginx No threads, event-driven Uses epoll / kqueue Low memory footprint 10000 active connections = normal
  76. 76. Nginx - a true alternative to Apache ? Not all Apache modules mod_auth_* mod_dav* … Basic modules are available Some 3rd party modules (needs recompilation !)
  77. 77. Nginx - Installation Packages Win32 binaries → Not for production ! Build from source (./configure; make; make install)
  78. 78. Nginx - Configuration server { listen 80; server_name www.domain.ext *.domain.ext; index index.html; root /home/domain.ext/www; } server { listen 80; server_name photo.domain.ext; index index.html; root /home/domain.ext/photo; }
  79. 79. Nginx - phase 1 Move Apache to a different port (8080) Put Nginx at port 80 Nginx serves all statics (images, css, js, …) Forward dynamic requests to Apache
  80. 80. Nginx for static files only server { listen 80; server_name www.domain.ext; location ~* ^.*.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|pdf|ppt|txt|tar|rtf|js)$ { expires 30d; root /home/www.domain.ext; } location / { proxy_pass http://www.domain.ext:8080; proxy_pass_header Set-Cookie; proxy_set_header X-Real-IP $remote_addr; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } }
  81. 81. Nginx for PHP ? Bottleneck = PHP ? Keep it in Apache Bottleneck = memory ? Go for it ! LANMMP to... LNMPP (ok, this is getting ridiculous)
  82. 82. Nginx with PHP-FPM Since PHP 5.3.3 Runs on port 9000 Nginx connects using fastcgi method location / { fastcgi_pass 127.0.0.1:9000; fastcgi_index index.php; include fastcgi_params; fastcgi_param SCRIPT_NAME $fastcgi_script_name; fastcgi_param SCRIPT_FILENAME /home/www.domain.ext/$fastcgi_script_name; fastcgi_param SERVER_NAME $host; fastcgi_intercept_errors on; }
  83. 83. Nginx + PHP-FPM features Graceful upgrade Spawn new processes under high load Chroot Slow request log !
  84. 84. Nginx + PHP-FPM features Graceful upgrade Spawn new processes under high load Chroot Slow request log ! fastcgi_finish_request() → offline processing
  85. 85. Nginx + PHP-FPM - performance ? Single webserver Proxy Static PHP Static PHP Apache + PHP 3900 17.5 6700 17.5 Apache + PHP + MC 3900 55 6700 108 Nginx + PHP-FPM + MC 11700 57 11200 112 Limit : single-threaded Apachebench
  86. 86. Nginx + PHP-FPM - performance ? Single webserver Proxy Static PHP Static PHP Apache + PHP 3900 17.5 6700 17.5 Apache + PHP + MC 3900 55 6700 108 Nginx + PHP-FPM + MC 11700 57 11200 112 Apache (tuned) + PHP/MC 10600 55 11400 108 Limit : single-threaded Apachebench
  87. 87. Reverse proxy time...
  88. 88. Varnish Not just a load balancer Reverse proxy cache / http accelerator / … Caches (parts of) pages in memory Careful : uses threads (like Apache) Nginx usually scales better (but doesn't have VCL)
  89. 89. Varnish - Installation & configuration Installation Packages Source : ./configure && make && make install Configuration /etc/default/varnish /etc/varnish/*.vcl
  90. 90. Varnish - backends + load balancing backend server1 { .host = "192.168.0.10"; } backend server2 { .host = "192.168.0.11"; } director example_director round-robin { { .backend = server1; } { .backend = server2; } }
  91. 91. Varnish - backends + load balancing backend server1 { .host = "192.168.0.10"; .probe = { .url = "/"; .interval = 5s; .timeout = 1 s; .window = 5; .threshold = 3; } }
  92. 92. Varnish - VCL Varnish Configuration Language DSL (Domain Specific Language) → compiled to C Hooks into each request Defines : Backends (web servers) ACLs Load balancing strategy Can be reloaded while running
  93. 93. Varnish - whatever you want Real-time statistics (varnishtop, varnishhist, ...) ESI
  94. 94. Article content page Article content (TTL : 15 min) /article/732 Varnish - ESI Header (TTL : 60 min) /top Latest news (TTL : 2 min) /news Navigation (TTL : 60 min) /nav
  95. 95. Going to /page/id/732 <esi:include src="/top"/> <esi:include src="/nav"/> <esi:include src="/news"/> <esi:include src="/article/732"/>
  96. 96. Article content page <esi:include src="/article/732"/> Varnish - ESI Perfect for caching pages <esi:include src="/top"/> <esi:include src="/news"/> <esi:include src="/nav"/> In your Varnish config : sub vcl_fetch { if (req.url == "/news") { esi; /* Do ESI processing */ set obj.ttl = 2m; } elseif (req.url == "/nav") { esi; set obj.ttl = 1m; } elseif …. …. }
  97. 97. Varnish with ESI - hold on tight ! Single webserver Proxy Static PHP Static PHP Apache + PHP 3900 17.5 6700 17.5 Apache + PHP + MC 3900 55 6700 108 Nginx + PHP-FPM + MC 11700 57 11200 112 Varnish - - 11200 4200
  98. 98. Varnish - what can/can't be cached ? Can : Static pages Images, js, css Pages or parts of pages that don't change often (ESI) Can't : POST requests Very large files (it's not a file server !) Requests with Set-Cookie User-specific content
  99. 99. ESI → no caching on user-specific content ? Logged in as : Wim Godden 5 messages TTL = 5minTTL=1h TTL = 0s ?
  100. 100. Coming soon... Based on Nginx Reduces load by 50 – 95% Requires code changes ! Well-built project → few changes Effect on webservers and database servers
  101. 101. ESI on Nginx Logged in as : Wim Godden 5 messages NEWSMenu
  102. 102. ESI on Nginx Logged in as : Wim Godden 5 messages NEWSMenu
  103. 103. SCL on Nginx + Memcached <scl:include key="news" src="/news" ttl="5m" /> <scl:include key="menu" src="/menu" ttl="1h" /> <scl:include key="top" src="/top" session="true" ttl="1h" />
  104. 104. Requesting /page (1st time) Nginx Shared memory 1 2 3 4 /page /page
  105. 105. Requesting /page ESI subrequests (1st time) Nginx 1 2 3 /menu /news /top (in SCL session)
  106. 106. Requesting /page (next time) Nginx Shared memory 1 2 /page /menu /news /top (in SCL session) /page
  107. 107. New message is sent... POST /send DB insert into... set(...) top (in SCL session)
  108. 108. Advantages No repeated GET hits to webserver anymore ! At login : POST → warm up the cache ! No repeated hits for user-specific content Not even for non-specific content
  109. 109. First release : ESI Part of the ESI 1.0 spec Only relevant features implemented Extension for dynamic session support But : unavailable for copyright reasons
  110. 110. Rebuilt from scratch : SCL Session-specific Caching Language Ideas for a better name ? Language details : Control structures : if/else, switch/case, foreach Variable handling Strings : concatenation, substring, ...
  111. 111. SCL code samples You are logged in as : <scl:session_var("person_name") /> You are logged in as : <@s("person_name") />
  112. 112. SCL code samples <scl:switch var="session_var('isAdmin')"> <scl:case value="1"> <scl:include key="admin-buttons" src="/admin-buttons.php" /> </scl:case> <scl:default> <div id="just-a-user"> <scl:include key="user-buttons" src="/user-buttons.php" /> </div> </scl:default> </scl:switch>
  113. 113. What's the result ?
  114. 114. What's the result ?
  115. 115. Figures 2nd customer : No. of web servers : 72 → 8 No. of db servers : 15 → 4 Total : 87 → 12 (86% reduction !) Last customer : No. of total servers : +/- 1350 Expected reduction : 1350 → 300 Expected savings : €1.6 Million per year
  116. 116. A real example : vBulletin DB Server Load Web Server Load Max Requests/sec (1 = 282) 0 5 10 15 20 25 30 35 Standard install With Memcached Nginx + SCL + memcached
  117. 117. Why is it so much faster ?
  118. 118. Availability Good news : It will become Open Source It's solid : ESI version stable at 4 customers Bad news : First customer holds copyrights Total rebuild → Open Source release No current projects, so spare time project Beta : Dec 2013 Final : Q1-Q2 2014 (on Github !)
  119. 119. Time to tune...
  120. 120. Apache - tuning tips Disable unused modules → fixes 10% of performance issues Set AllowOverride to None. Enable only where needed ! Disable SymLinksIfOwnerMatch. Enable only where needed ! MinSpareServers, MaxSpareServers, StartServers, MaxClients, MPM selection → a whole session of its own ;-) Don't mod_proxy → use Nginx or Varnish ! High load on an SSL-site ? → put SSL on a reverse proxy
  121. 121. PHP speed - some tips Upgrade PHP - every minor release has 5-15% speed gain ! Use an opcode cache (Zend O+, APC, eAccelerator, XCache) Profile your code XHProf Xdebug But : turn off profilers on acceptance/production platforms !
  122. 122. KCachegrind is your friend
  123. 123. PHP speed - some tips Most performance issues are in DB queries → look there first ! Log PHP errors and review those logs ! Shorter code != faster code → keep your code readable ! Hardware cost < Manpower cost → 1 more server < 30 mandays of labor Keep micro-optimizations in code = last thing on list
  124. 124. DB speed - some tips Avoid dynamic functions Ex. : select id from calendar where startDate > curdate() Better : select id from calendar where startDate > "2013-05-14" Use same types for joins i.e. don't join decimal with int RAND() is evil ! count(*) is evil in InnoDB without a where clause ! Persistent connect is sort-of evil Index, index, index ! → But only on fields that are used in where, order by, group by !
  125. 125. Caching & Tuning @ frontend http://www.websiteoptimization.com/speed/tweak/average-web-page/
  126. 126. Caching in the browser HTTP 304 (Not modified) Expires/Cache-Control header 2 notes : Don't use POST if you want to cache Don't cache user-specific pages in browser (security !)
  127. 127. HTTP 304 Browser Server No header Last Modified: Fri 28 Jan 2011 08:31:01 GMT If-Modified-Since: Fri 28 Jan 2011 08:31:01 GMT 200 OK / 304 Not Modified First request Next requests
  128. 128. HTTP 304 with ETag Browser Server No header Etag: 8a53321-4b-43f0b6dd972c0 If-None-Match: 8a53321-4b-43f0b6dd972c0 200 OK / 304 Not Modified First request Next requests
  129. 129. Expires/Cache-control header Cache-Control HTTP/1.1 Seconds to expiry Used by browsers Browser Server No header Expires: Fri 29 Nov 2011 12:11:08 GMT Cache-Control: max-age=86400 First request Next requests No requests until item expires Expires HTTP/1.0 Date to expire on Used by old proxies Requires clock to be accurate !
  130. 130. Pragma: no-cache = evil "Pragma: no cache" doesn't make it uncacheable Don't want caching on a page ? HTTP/1.0 : "Expires : Fri, 30 Oct 1998 00:00:00 GMT" (in the past) HTTP/1.1 : "Cache-Control: no-store"
  131. 131. Frontend tuning 1. You optimize backend 2. Frontend engineers messes up → havoc on backend 3. Don't forget : frontend sends requests to backend ! SO... Care about frontend Test frontend Check what requests frontend sends to backend
  132. 132. Tuning frontend Minimize requests Combine CSS/JavaScript files Use inline images in CSS/XHTML (not supported on all browsers yet)
  133. 133. Frontend tuning - inline CSS/XHTML images #navbar span { width: 31px; height: 31px; display: inline; float: left; margin-right: 4px; } .home { background-image: url(data:image/gif;base64,R0lGODlhHwAfAPcAAAAAAIxKAKVjCLW1tb29tcbGvc7OxtZ7ANbWztbW1tbe1t7e1uelMefn1ufn3ufn5+fv3u +MAO/v5+/v7/fGCPf35/f37//nY//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// ////////////////////////////////////////////////........MEl0nGVUC6tObNnPceSFBaQVMJAxC4lo3gNOrUaFnTHoAxNm3XVxPfRq139e8BEGAjWD5bgI ALw287T8AcAXLly2kjOACdc17higXSIKDO/Lpv7Qq4bw7APgBq8eOzX69InrZ6xe3dbxZffyTGkb8tdx8F+b0Xn2sFsCSBAgTM5lp63RH YnoHUudZgRgkGOGCB+43nGk4OGcQTabKx5dyJKJ7ImoUNCaRRAZYN1ppsrT3Y2gIwyjSQBAtUpABml/0IJGYd6VjQUDH9uBFkGx Gm5I8dPQaRUAQUMBdhhBV25ZYUJZBcSAtSJBddWZZ5UAGPOTXlgkNVOSZdBxEwIkYu7VhYnAol5GaadRqF0Uaz0TgXnX2umV FyGakJUUAAADs=); margin-left: 4px; } <img border=0 src="data:image/gif;base64,R0lGODlhHwAfAPcAAAAAAIxKAKVjCLW1tb29tcbGvc7OxtZ7ANbWztbW1tbe1t7e1uelMefn1ufn3ufn5+fv 3u+MAO/v5+/v7/fGCPf35/f37//nY/......Uaz0TgXnX2umVFyGakJUUAAADs=">
  134. 134. Tuning frontend Minimize requests Combine CSS/JavaScript files Use inline images in CSS/XHTML (not supported on all browsers yet) Use CSS Sprites
  135. 135. CSS Sprites
  136. 136. Tuning content - CSS sprites
  137. 137. Tuning content - CSS sprites 11 images 11 HTTP requests 24KByte 1 image 1 HTTP requests 14KByte
  138. 138. Tuning frontend Minimize requests Combine CSS/JavaScript files Use inline images in CSS/XHTML (not supported on all browsers yet) Use CSS Sprites (horizontally if possible) Put CSS at top Put JavaScript at bottom Max. no connections Especially if JavaScript does Ajax (advertising-scripts, …) ! Avoid iFrames Again : max no. of connections Don't scale images in HTML Have a favicon.ico (don't 404 it !) → see my blog
  139. 139. Tuning frontend Don't use inline CSS/JavaScript CSS/JavaScript need to be external files (minified, merged) Why ? → Cacheable by browser / reverse proxy Use GET for Ajax retrieval requests (and cache them !) Optimize images (average 50-60% !) Split requests across subdomains Put statics on a separate subdomain (without cookies !) www.phpbenelux.eu Max. 2 requests www.phpbenelux.eu Max. 2 requests Max. 2 requests images.phpbenelux.eu
  140. 140. Tuning miscellaneous Avoid DNS lookups Frontend : don't use too many subdomains (2 = ideal) Backend : Turn off DNS resolution in Apache : HostnameLookups Off If your app uses external data Run a local DNS cache (timeout danger !) Make sure you can trust DNS servers (preferable run your own) Compress non-binary content (GZIP) mod_deflate in Apache HttpGzipModule in Nginx (HttpGzipStaticModule for pre-zipped statics !) No native support in Varnish
  141. 141. What else can kill your site ? Redirect loops Multiple requests More load on Webserver More PHP to process Additional latency for visitor Try to avoid redirects anyway → In ZF : use $this->_forward instead of $this->_redirect Watch your logs, but equally important... Watch the logging process → Logging = disk I/O → can kill your server !
  142. 142. Above all else... be prepared ! Have a monitoring system Use a cache abstraction layer (disk → Memcache) Don't install for the worst → prepare for the worst Have a test-setup Have fallbacks → Turn off non-critical functionality
  143. 143. So... Cache But : never delete, always push ! Have a warmup script Monitor your cache Have an abstraction layer Apache = fine, Nginx = better Static pages ? Use Varnish Tune your frontend → impact on backend !
  144. 144. Questions ?
  145. 145. Questions ?
  146. 146. Contact Twitter @wimgtr Web http://techblog.wimgodden.be Slides http://www.slideshare.net/wimg E-mail wim.godden@cu.be Please... Rate my talk : http://joind.in/9044
  147. 147. Thanks ! Please... Rate my talk : http://joind.in/9044

×