Scaling 101 test

Slide 1: Scaling 101 Chris Finne CTO of Venrock's Quarry

Slide 2: Venrock and the Quarry are looking for • Summer Interns: o Ruby on Rails Engineers o Community Manager o Digital Media Analyst • Digital Media Associate • Full Time Professional Web Engineers o Mobile/Media/Social o MVC / Full stack  OS->DB->App->Code->Web Server->HTML/JS/CSS/Ajax->Flash www.venrock.com

Slide 3: Scaling 101 - Assumptions / Misc • Target Audience o Engineers (but not professional web infrastructure) • Give a "lay of the land" rather than heavy specifics • < 20M database rows • Will distribute the preso • Links to various topics are provided as appendix • Please interrupt with questions, but might table them for later • About 30 slides

Slide 4: Scaling can be rocket science...

Slide 5: Doesn't have to be rocket science... • Scaling 101 isn't hard • Understand the principles • If Scaling 101 isn't enough to handle your traffic... o you've probably have enough traffic to get Series A funding • Sometimes there is a quick-n-easy solution • If not, follow basic problem solving cycle... o analyze o research o trial-n-error (hopefully with testing ;-)

Slide 6: Scaling 101 Principles - Slow to Fast Infrastructure Speed Specific Examples 1. External network accessed • Roundtrip query to 2. Internal network accessed Facebook o DB on another server: • Database table scans on 1.DB Delete large tables (>100 rows) 2.DB Update 3.DB Select (goes to disk) • Dynamically Rendering 4.DB Select (table in high volume generic pages memory) • Too many database calls 5.DB Select cached o Memcached per page load (>5-10) 3. local filesystem 4. local DB 5. local memcached 6. local app server memory

Slide 7: Typical Infrastructure Evolution 1. Single Server - get the app out! 2. Optimize to try to stay on a single server 3. Dedicated Database Server 4. Multiple Web/App Servers - Load Balancing 5. Add More Database Servers 6. More Separation - Web and App servers Specific Guidelines for when to level up...

Slide 8: Specific Guidelines - Nada There aren't any. Why? • heavy static files vs. very dynamic • application code is different • database activity is different o lots of selects vs. inserts vs. updates vs. deletes o lots of complex JOINs vs. simple selects • traffic patterns o 8, 12 or 24 hour day? o Occasional Spikes (Digg, Slashdot) • hardware is different from hosting vendor to hosting vendor

Slide 9: Optimize - Profiling your app Questions to ask: o Which pages are getting hit the most  Google Analytics  Web server logs o Which pages take the longest to render  Firebug - Net tab  Yahoo's YSlow Add-on  Apache Benchmark tool  JMeter  external monitoring site, e.g. Site24x7

Slide 10: Optimize - Profiling your app Next question: o What pieces of those long, popular pages are taking the longest?  Facebook queries  application code blocks  database queries  static file downloads (images,CSS, JS)

Slide 11: Optimize - Profiling your app How to actually find the offenders... o Application Code:  put debug statements to see where the most time is being spent o Page Loading / Static file downloads  Firefox Extensions:  Yahoo YSlow for Firebug: see which pieces take the longest to download / finish rendering  Live HTTP Headers: Are your static files being cached locally? o Database  MySQL slow query log  MySQL Query log  MySQL "EXPLAIN" Queries

Slide 12: Optimize - Profiling your app Now what? o Focus on the bottlenecks

Slide 13: Optimize - Remedies • Facebook queries o Reduce Roundtrips to FB with more complex FQL  (e.g. subqueries) o Cache Results • Use FBML where possible - make FB do the work o fb:user fb:name fb:profile-pic • Fix inefficient application code o any examples?

Slide 14: Optimize - Remedies • Optimize SQL Queries o Database Indexes o Only select what you need to select o Views, Stored Procedures • Confirm static files are being cached locally in browser (Images, JS and CSS) • Apache Config... • Caching...

Slide 15: Optimize - Caching What? - Expensive pieces o Facebook User Information (24hrs) o complex calculations o generic pages (or fragments) o complex, big or long DB queries

Slide 16: Optimize - Caching Where to? o Filesystem  HTML pages served directly from Apache w/ no PHP  expensive HTML fragments loaded via PHP o User's Session  Facebook User Info  Application state o Memcached - (or BerkeleyDB)  HTML  Session  Facebook data (24hrs max)  expensive DB query results o Database Query Cache  repeated queries

Slide 17: Optimize - Apache • Are you using all your resources? • If you have 10 Apache processes (MaxClients) and 15 users hit your app at the same time, 10 will get served, 5 will get an error • Do you need more Apache processes? o No if your box's CPU and/or RAM are maxing out (use top)  need top optimize the app or add more servers o Yes if the requests involve waiting for a long time for Facebook to answer a query (i.e. Apache is just waiting) • Add more processes (MaxClients)

Slide 18: Optimize - Apache # prefork MPM # StartServers: number of server processes to start # MinSpareServers: minimum number of server processes which are kept spare # MaxSpareServers: maximum number of server processes which are kept spare # MaxClients: maximum number of server processes allowed to start # MaxRequestsPerChild: maximum number of requests a server process serves <IfModule mpm_prefork_module> StartServers 5 MinSpareServers 5 MaxSpareServers 10 MaxClients 150 MaxRequestsPerChild 0 </IfModule>

Slide 19: Multi-Server - Dedicated DB • Needs to be sitting next to the web/app server with a fast link • Port 3306 open from Web/App to DB servers • MySQL User account has to allow connections from the web/app servers

Slide 20: Multi-Server - Dedicated DB If standard hardware/slices: o keep your DB server where it is o setup a new slice as a web/app server o get your new slice working  test  load-test o cut-over your DNS address

Slide 21: Multi-Server - Dedicated DB If DB performance is a bottleneck and you are going to a larger server for your DB... o Configure / Test  Configure the new DB  mysqldump of your existing DB  (do you have enough disk? dump over the wire)  new DB: mysql < dump  configure your new DB server as an Apache and PHP server as well for testing  functional test / load test

Slide 22: Multi-Server - Dedicated DB If DB performance is a bottleneck and you are going to a larger server for your DB... o Cut-over  shutdown the production webserver  mysqldump old  import mysql to new  configure your existing web/app server to point to new db

Slide 23: Multiple Web/App Servers • Bring up new app server • Test it by itself • Add to load-balancing...

Slide 24: Load Balancing Goals: o Load and/or Fault Tolerance Technologies o DNS Round Robin  Potential Windows / IE issues due to caching o Open Source Software  Apache reverse Proxy (Apache 2.2)  HAProxy  Pound o Hardware  10x or more performance than software  some hosting vendors provide it as an add-on or part of a package

Slide 25: Multiple DB Servers Getting more involved... • Master/Slave o reads go to slaves o transactional reads or guaranteed updated data reads go to master o writes go to master Getting close to Rock Science... • Master/Master (possibly with slaves) • clusters

Slide 26: Other Easy Tricks Put static files on Amazon's S3 o Images, JS, CSS, Videos Optimize Page Loads (not really scalability, but...) • Put external "stuff" at the bottom of the page outside TABLES o Ad Tags o Google Analytics o Digg buttons • DIV's instead of tables if possible o tables wait for all content to be loaded before rendering o div's typically render piece by piece

Slide 27: Finally - The End • Feedback to Yee or me: cfinne at venrock . com • Follow-on questions or consults: cfinne at venrock . com • Next Talk? o Interaction Design - David Cortright o Venture Capital - Some VC (Brian, Ilya, Dev...) o Code / Web App Design - me (again???)

Slide 28: Links MySQL Slow Query Log: http://dev.mysql.com/doc/refman/5.0/en/slow-query-log.html MySQL General Query Log: http://dev.mysql.com/doc/refman/5.0/en/query-log.html MySQL Query Cache: • http://jayant7k.blogspot.com/2007/07/mysql-query-cache.html • http://dev.mysql.com/doc/refman/5.0/en/query-cache.html MySQL Optimize Queries and DB Indexes • http://www.databasejournal.com/features/mysql/article.php/1382 Memcached: • http://us3.php.net/memcache • http://www.danga.com/memcached/

Slide 29: Links Firebug: • Firebug: https://addons.mozilla.org/en-US/firefox/addon/1843 • YSlow: http://developer.yahoo.com/yslow/ PHP Facebook Sessions: http://wiki.developers.facebook.com/index.php/PHP_Sessions Live HTTP Headers: https://addons.mozilla.org/en-US/firefox/addon/3829 Apache Prefork Config: http://httpd.apache.org/docs/2.0/mod/prefork.html Load balancing: • Apache Reverse Proxy: http://httpd.apache.org/docs/2.2/mod/mod_proxy.html • HAProxy:http://haproxy.1wt.eu/ • Pound: http://www.apsis.ch/pound