Caching and tuning fun for high scalability @ LOAD2012

Caching and tuning fun for high scalability Wim Godden Cu.be Solutions

Who am I ?Wim Godden (@wimgtr)Owner of Cu.be Solutions (http://cu.be)Open Source developer since 1997Developer of OpenXZend Certified EngineerZend Framework Certified EngineerMySQL Certified Developer

Who are you ?Developers ?System/network engineers ?Managers ?Caching experience ?

Caching and tuning fun for high scalability Wim Godden Cu.be Solutions

Goals of this tutorialEverything about caching and tuningA few techniques How-to How-NOT-to→ Increase reliability, performance and scalability 5 visitors/day → 5 million visitors/day (Dont expect miracle cure !)

Architecture

Our base benchmarkApachebench = useful enoughResult ? Single webserver Proxy Static PHP Static PHPApache + PHP 3900 17.5 6700 17.5 Limit : Limit : CPU, network database or disk

Caching

What is caching ? CACHE

What is caching ? x = 5, y = 2 Same result n = 50 CACHE select * from article join user Doesnt change on article.user_id = user.id all the time order by created desc limit 10

Theory of caching DB le t ab fro m ul t res ata ned e ct d retur sel ta = $da if ($data == false) Page GET /page set( key , $ $da fals data) ta = e get( key ) Cache

Theory of caching DB HIT Cache

Caching techniques #1 : Store entire pages #2 : Store part of a page (block) #3 : Store data retrieval (SQL ?) #4 : Store complex processing result #? : Your call ! When you have data, think : Creating time ? Modification frequency ? Retrieval frequency ?

How to find cacheable dataNew projects : start from cache everythingExisting projects : Look at MySQL slow query log Make a complete query log (dont forget to turn it off !) Check page loading times

Caching storage - DiskData with few updates : goodCaching SQL queries : preferably notDONT use NFS or other network file systems especially for sessions locking issues ! high latency

Caching storage - Disk / ramdiskLocal 5 Webservers → 5 local caches How will you keep them synchronized ? → Dont say NFS or rsync !

Caching storage - Memcache(d)Facebook, Twitter, YouTube, … → need we say more ?Distributed memory caching systemMultiple machines ↔ 1 big memory-based hash-tableKey-value storage system Keys - max. 250bytes Values - max. 1Mbyte

Caching storage - Memcache(d)Facebook, Twitter, YouTube, … → need we say more ?Distributed memory caching systemMultiple machines ↔ 1 big memory-based hash-tableKey-value storage system Keys - max. 250bytes Values - max. 1MbyteExtremely fast... non-blocking, UDP (!)

Memcache - where to install

Memcache - installation & running itInstallation Distribution package PECL Windows : binariesRunning No config-files memcached -d -m <mem> -l <ip> -p <port> ex. : memcached -d -m 2048 -l 172.16.1.91 -p 11211

Caching storage - Memcache - some notesNot fault-tolerant Its a cache ! Lose session data Lose shopping cart data ...

Caching storage - Memcache - some notesNot fault-tolerant Its a cache ! Lose session data Lose shopping cart data …Firewall your Memcache port !

Memcache in code<?php$memcache = new Memcache();$memcache->addServer(172.16.0.1, 11211);$memcache->addServer(172.16.0.2, 11211);$myData = $memcache->get(myKey);if ($myData === false) { $myData = GetMyDataFromDB(); // Put it in Memcache as myKey, without compression, with no expiration $memcache->set(myKey, $myData, false, 0);}echo $myData;

Wheres the data ?Memcache client decides (!)2 hashing algorithms : Traditional Server failure → all data must be rehashed Consistent Server failure → 1/x of data must be rehashed (x = # of servers)

Benchmark with Memcache Single webserver Proxy Static PHP Static PHPApache + PHP 3900 17.5 6700 17.5Apache + PHP + MC 3900 55 6700 108

Memcache slabs (or why Memcache says its full when its not)Multiple slabs of different sizes : Slab 1 : 400 bytes Slab 2 : 480 bytes (400 * 1.2) Slab 3 : 576 bytes (480 * 1.2) (and so on...)Multiplier (1.2 here) can be configuredStore a lot of very large objects→ Large slabs : full→ Rest : free→ Eviction of data !

Memcache - Is it working ?Connect to it using telnet STAT pid 2941 STAT uptime 10878 STAT time 1296074240 "stats" command → STAT version 1.4.5 STAT pointer_size 64 STAT rusage_user 20.089945 Use Cacti or other monitoring tools STAT rusage_system 58.499106 STAT curr_connections 16 STAT total_connections 276950 STAT connection_structures 96 STAT cmd_get 276931 STAT cmd_set 584148 STAT cmd_flush 0 STAT get_hits 211106 STAT get_misses 65825 STAT delete_misses 101 STAT delete_hits 276829 STAT incr_misses 0 STAT incr_hits 0 STAT decr_misses 0 STAT decr_hits 0 STAT cas_misses 0 STAT cas_hits 0 STAT cas_badval 0 STAT auth_cmds 0 STAT auth_errors 0 STAT bytes_read 613193860 STAT bytes_written 553991373 STAT limit_maxbytes 268435456 STAT accepting_conns 1 STAT listen_disabled_num 0 STAT threads 4 STAT conn_yields 0 STAT bytes 20418140 STAT curr_items 65826 STAT total_items 553856 STAT evictions 0 STAT reclaimed 0

Memcache - backing up

Memcache - tipPage with multiple blocks ?→ use Memcached::getMulti() Hashing getMulti($array) algorithmBut : what if you get some hits and some misses ?

Updating data

Updating data LCD_Popular_Product_List

Adding/updating data$memcache->delete(LCD_Popular_Product_List);

Adding/updating data

Adding/updating data - Why it crashed

Cache stampeding

Memcache code ? Memcache code Visitor interface Admin interface DB

Cache warmup scriptsUsed to fill your cache when its emptyRun it before starting Webserver !2 ways : Visit all URLs Error-prone Hard to maintain Call all cache-updating methods Make sure you have a warmup script !

Cache stampeding - what about locking ?Seems like a nice idea, but...While lock in placeWhat if the process that created the lock fails ?

LAMP... → LAMMP → LNMMP

NginxWeb serverReverse proxyLightweight, fast10.2% of all Websites

NginxNo threads, event-drivenUses epoll / kqueueLow memory footprint10000 active connections = normal

Nginx - a true alternative to Apache ?Not all Apache modules mod_auth_* mod_dav* …Basic modules are availableSome 3rd party modules (needs recompilation !)

Nginx - Configuration server { listen 80; server_name www.domain.ext *.domain.ext; index index.html; root /home/domain.ext/www; } server { listen 80; server_name photo.domain.ext; index index.html; root /home/domain.ext/photo; }

Nginx + PHP-FPM - performance ? Single webserver Proxy Static PHP Static PHPApache + PHP 3900 17.5 6700 17.5Apache + PHP + MC 3900 55 6700 108Nginx + PHP-FPM + MC 11700 57 11200 112 Limit : single-threaded Apachebench

Reverse proxy time...

VarnishNot just a load balancerReverse proxy cache / http accelerator / …Caches (parts of) pages in memoryCareful : uses threads (like Apache) Nginx usually scales better (but doesnt have VCL)

Varnish - backends + load balancing backend server1 { .host = "192.168.0.10"; } backend server2{ .host = "192.168.0.11"; } director example_director round-robin { { .backend = server1; } { .backend = server2; } }

Varnish - VCLVarnish Configuration LanguageDSL (Domain Specific Language) → compiled to CHooks into each requestDefines : Backends (web servers) ACLs Load balancing strategyCan be reloaded while running

Varnish - whatever you wantReal-time statistics (varnishtop, varnishhist, ...)ESI

Varnish - ESIPerfect for caching pages Header (TTL : 60 min) In your article page output : /top <esi:include src="/news"/> Latest news (TTL : 2 min) /news In your Varnish config : sub vcl_fetch { if (req.url == "/news") {Navigation Article content page esi; /* Do ESI processing */ (TTL : set obj.ttl = 2m; 60 min) Article content (TTL : 15 min) } elseif (req.url == "/nav") { /nav /article/732 esi; set obj.ttl = 1m; } elseif …. …. }

Varnish with ESI - hold on tight ! Single webserver Proxy Static PHP Static PHPApache + PHP 3900 17.5 6700 17.5Apache + PHP + MC 3900 55 6700 108Nginx + PHP-FPM + MC 11700 57 11200 112Varnish - - 11200 4200

Varnish - what can/cant be cached ?Can : Static pages Images, js, css Pages or parts of pages that dont change often (ESI)Cant : POST requests Very large files (its not a file server !) Requests with Set-Cookie User-specific content

ESI → no caching on user-specific content ? Logged in as : Wim Godden TTL = 0s ? 5 messages TTL=1h TTL = 5min

Coming soon...Based on NginxReduces load by 50 – 95% Requires code changes ! Well-built project → few changes Effect on webservers and database servers

Whats the result ?

FiguresFirst customer : No. of web servers : 18 → 4 No. of db servers : 6 → 2 Total : 24 → 6 (75% reduction !)Second customer (already using Nginx + Memcache) : No. of web servers : 72 → 8 No. of db servers : 15 → 4 Total : 87 → 12 (86% reduction !)

AvailabilityStable at 2 customersStill under heavy developmentBeta : Jul 2012Final : Sep 2012

Tuning

PHP speed - some tipsUpgrade PHP - every minor release has 5-15% speed gain !Use an opcode cache

Caching storage - Opcode caching

DB speed - some tipsUse same types for joins i.e. dont join decimal with intRAND() is evil !count(*) is evil in InnoDB without a where clause !Persistent connect is sort-of evil

Caching & Tuning @ frontend http://www.websiteoptimization.com/speed/tweak/average-web-page/

Frontend tuning1. You optimize backend2. Frontend engineers messes up → havoc on backend3. Dont forget : frontend sends requests to backend ! SO...Care about frontendTest frontendCheck what requests frontend sends to backend

Tuning frontendMinimize requests Combine CSS/JavaScript files

Tuning frontendMinimize requests Combine CSS/JavaScript files Use CSS Sprites

CSS Sprites

Tuning content - CSS sprites

Tuning content - CSS sprites 11 images 1 image 11 HTTP requests 1 HTTP requests 24KByte 14KByte

Tuning frontendMinimize requests Combine CSS/JavaScript files Use CSS Sprites (horizontally if possible)Put CSS at topPut JavaScript at bottom Max. no connections Especially if JavaScript does Ajax (advertising-scripts, …) !Avoid iFrames Again : max no. of connectionsDont scale images in HTMLHave a favicon.ico (dont 404 it !) → see my blog

What else can kill your site ?Redirect loops Multiple requests More load on Webserver More PHP to process Additional latency for visitor Try to avoid redirects anyway → In ZF : use $this->_forward instead of $this->_redirectWatch your logs, but equally important...Watch the logging process →Logging = disk I/O → can kill your server !Slashdot effect

Above all else... be prepared !Have a monitoring systemUse a cache abstraction layer (disk → Memcache)Dont install for the worst → prepare for the worstHave a test-setupHave fallbacks → Turn off non-critical functionality

So...Cache But : never delete, always push ! Have a warmup script Monitor your cache Have an abstraction layerApache = fine, Nginx = betterStatic pages ? Use VarnishTune your frontend → impact on backend !Prepare for the worst

Questions ?

Were hiring !Lots of challengesWork with cutting-edge technologyVaried : PHP development System / network architecture Scalability services Build our own services Work on Open Source → mail us : info@cu.be

ContactTwitter @wimgtrWeb http://techblog.wimgodden.beSlides http://www.slideshare.net/wimgE-mail wim.godden@cu.be Please... Rate my talk : http://speakerrate.com/talks/9501

Thanks ! Please...Rate my talk : http://speakerrate.com/talks/9501

Caching and tuning fun for high scalability @ LOAD2012

by Wim Godden

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 32

Statistics

Caching and tuning fun for high scalability @ LOAD2012 Presentation Transcript

http://speakerrate.com	29
https://si0.twimg.com	2
http://us-w1.rockmelt.com	1