Top 10 Scalability Mistakes


Published on

A very successful talk where in I discuss the top 10 failures of organizations I have personally experienced when trying to scale. More than just performance!

Published in: Technology
1 Comment
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Top 10 Scalability Mistakes

  1. Top 10 Scalability Mistakes John Coggeshall
  2. Welcome! <ul><li>Who am I: John Coggeshall </li></ul><ul><ul><li>Sr. Technical Consultant, Zend Technologies </li></ul></ul><ul><ul><li>Author PHP 5 Unleashed </li></ul></ul><ul><ul><li>Zend Educational Advisory Board </li></ul></ul><ul><ul><li>Speaker on PHP-related topics worldwide </li></ul></ul><ul><ul><li>Geek </li></ul></ul>
  3. What is Scalability? <ul><li>Define:Scalability </li></ul><ul><ul><li>The ability and flexibility of an application to meet growth requirements of an organization </li></ul></ul><ul><ul><li>More then making a site go fast(er) </li></ul></ul><ul><ul><ul><li>Scalability in human resources, for example </li></ul></ul></ul><ul><li>The “fastest” approach isn’t always the most scalable </li></ul><ul><ul><li>OO is slower, but more scalable from a code maintence and reuse standpoint </li></ul></ul><ul><ul><li>Failure to consider future needs during architectural stages leading to failure of the application’s API to scale </li></ul></ul>
  4. The secret to scalability is the ability to design, code, and maintain your applications using the same process again and again regardless of size
  5. Mistake #1: Network file systems <ul><li>Problem: We have a server farm of 10 servers and we need to deploy our codebase </li></ul><ul><ul><li>Very common problem </li></ul></ul><ul><ul><li>Many people look to a technology like NFS </li></ul></ul><ul><ul><ul><li>Share one codebase </li></ul></ul></ul><ul><li>At least 90% of the time, this is a bad idea </li></ul><ul><ul><li>NFS/GFS is really slow </li></ul></ul><ul><ul><li>NFS /GFS has tons of locking issues </li></ul></ul>
  6. Mistake #1: Network file systems <ul><li>So how do we deploy our codebase? </li></ul><ul><ul><li>You should always depoly your codebase locally on the machine serving it </li></ul></ul><ul><ul><li>Rsync is your friend </li></ul></ul><ul><li>What about run-time updates? </li></ul><ul><ul><li>Accepting File uploads </li></ul></ul><ul><ul><ul><li>Need to be available to all servers simutaneously </li></ul></ul></ul><ul><ul><li>Solutions vary depending on needs </li></ul></ul><ul><ul><ul><li>NFS may be an option for this small portion of the site </li></ul></ul></ul><ul><ul><ul><li>Database is also an option </li></ul></ul></ul>
  7. Mistake #2: Blocking calls <ul><li>Blocking I/O can always be a problem in an application </li></ul><ul><ul><li>I.e. attempting to open a remote URL from within your PHP scripts </li></ul></ul><ul><li>If the resource is locked / slow / unavailable your script hangs while we wait for a timeout </li></ul><ul><ul><li>Might as well try to scale an application that has a sleep(30) in it </li></ul></ul><ul><ul><li>Very bad </li></ul></ul>
  8. Mistake #2: Blocking calls <ul><li>Solutions </li></ul><ul><ul><li>Don’t use blocking calls in your application </li></ul></ul><ul><ul><li>Don’t use blocking calls in the heavy-load aspects of your application </li></ul></ul><ul><ul><li>Have out-of-process scripts responsible for pulling down data which aren’t connected to the web server </li></ul></ul>
  9. Mistake #3: Poor database design <ul><li>Database design is almost always the most important thing in your application </li></ul><ul><ul><li>PHP can be used completely properly, but if you mess up the databsae you’re hosed anyway </li></ul></ul><ul><li>Take the time to really think about your design </li></ul><ul><ul><li>Read books on designing relational databases </li></ul></ul><ul><ul><li>Understand how Indexes work, and use them </li></ul></ul>
  10. Mistake #3: Poor database design <ul><li>For example.. </li></ul><ul><ul><li>Using MySQL MyISAM tables all the time </li></ul></ul><ul><ul><ul><li>Use InnoDB instead if you can </li></ul></ul></ul><ul><ul><li>Use MyISAM tables only if you plan on doing fulltext searching </li></ul></ul><ul><ul><ul><li>Even then, they shouldn’t be primary tables </li></ul></ul></ul>
  11. Mistake #4: Failure to understand The web server <ul><li>When designing an application, it’s very important that you understand how PHP works in the bigger picture </li></ul><ul><ul><li>Know how PHP interacts and responds to your web server </li></ul></ul><ul><ul><li>For instance – How’s PHP really work with Apache 1.3.x? </li></ul></ul>
  12. Mistake #4: Failure to understand The web server <ul><li>Apache 1.3.x works on a pre-fork model </li></ul><ul><ul><li>One parent process spawns a whole lot of child processes </li></ul></ul><ul><ul><li>Each process handles a single HTTP request at a time </li></ul></ul><ul><ul><ul><li>May handle a finite or infinite number of requests before being destroyed </li></ul></ul></ul><ul><ul><li>PHP exists in the context of an Apache Child process </li></ul></ul><ul><ul><ul><li>This means this like “persistent” resources are only persistent to the individual child process </li></ul></ul></ul><ul><ul><ul><li>Database connections total = Process total </li></ul></ul></ul>
  13. Mistake #5: Hanging up Apache <ul><li>When scaling an application, requests per second is key </li></ul><ul><ul><li>You should have an idea how long a single request will take </li></ul></ul><ul><ul><li>You should know how many of those requests your server farm can handle at once without dying </li></ul></ul><ul><ul><li>You should know you’re requests-per-second figures </li></ul></ul><ul><li>Too often, people let Apache handle things that it really shouldn’t </li></ul><ul><ul><li>I.e. Large file downloads, streamed media, etc. </li></ul></ul>
  14. Mistake #5: Hanging up Apache <ul><li>When Apache is sending a 10 megabyte file, that means that one of your HTTP children is wasting it’s time shuffling data down the pipe </li></ul><ul><ul><li>This is definitely something that can be handled by something else </li></ul></ul><ul><ul><ul><li>A different HTTP server (tHttpd) </li></ul></ul></ul><ul><ul><ul><li>Zend Download Server </li></ul></ul></ul><ul><ul><li>At any given point in time, you should try to design thing so that your primary server function (serving PHP scripts) is the only thing being done by Apache </li></ul></ul>
  15. Mistake 5a: Letting Apache do any static handling <ul><li>On the same note, you can use something like thttpd to serve all static content </li></ul><ul><ul><li>Set up a subdomain </li></ul></ul><ul><ul><li>Put all of your images, flash files, javascript libs, stylesheets, etc. on that server </li></ul></ul>
  16. Tricks of the Trade <ul><li>If you're web application has a lot of semi-static content </li></ul><ul><ul><li>Content that could change so it has to be stored in the DB, but almost never does </li></ul></ul><ul><li>.. And you're running on Apache </li></ul><ul><li>This Design Pattern is killer! </li></ul>
  17. Tricks of the Trade <ul><li>Most people in PHP would implement a page like this: </li></ul><ul><ul><li>http:// =5 </li></ul></ul><ul><li>This would be responsible for generating the semi-static page HTML for the browser </li></ul>
  18. Tricks of the Trade <ul><li>Instead of generating the HTML for the browser, make this script generate another PHP script that contains mostly static content </li></ul><ul><ul><li>Keep things like personalization code, but make the actual article itself static in the file </li></ul></ul><ul><ul><li>Write the file to disk in a public folder under document root </li></ul></ul>
  19. Tricks of the Trade <ul><li>If you put them in this directory </li></ul><ul><ul><li> </li></ul></ul><ul><li>You can create a mod_rewrite rule such that </li></ul><ul><ul><li> maps to </li></ul></ul><ul><ul><li> </li></ul></ul><ul><li>Since show_article.php writes files to articles, once it's been generated no more DB reads! </li></ul>
  20. Tricks of the Trade <ul><li>Simple and Elegant Solution </li></ul><ul><li>Allows you to keep pages “personalized” </li></ul><ul><li>Very easy to Maintain </li></ul>
  21. Mistake #6: Designing without Scalability <ul><li>When designing your application, you should assume it needs to scale </li></ul><ul><ul><li>Quick and dirty prototypes often are exactly what gets to production </li></ul></ul><ul><li>It’s easy to make sure your applications have a decent chance of scaling </li></ul><ul><ul><li>MySQL: Design assuming someday you’ll need master/server replication </li></ul></ul><ul><li>Don’t write an application you’ll need three years from now, write an application you need today </li></ul><ul><ul><li>Just think about what you might need in three years </li></ul></ul>
  22. Mistake #7: Improperly dealing with database connections <ul><li>Improperly using persistent database connections </li></ul><ul><ul><li>Know your database, MySQL has a relatively light handshake process compared to Oracle </li></ul></ul><ul><li>Using PHP to deal with database fail over </li></ul><ul><ul><li>It’s not PHP’s Job, don’t do it. </li></ul></ul><ul><ul><li>Design your PHP applications to work with hostname aliases instead of real addresses </li></ul></ul><ul><ul><ul><li>i.e. mysql-r, mysql-w </li></ul></ul></ul><ul><ul><li>Have external processes responsible for switching the /etc/hosts file in the event something blows up </li></ul></ul>
  23. Tricks of the Trade <ul><li>For those of us using MySQL, here’s a great replication trick from our friends at flickr </li></ul><ul><ul><li>InnoDB is under most circumstances considerably faster then MyISAM </li></ul></ul><ul><ul><li>MyISAM is considerably better suited for full-text searches </li></ul></ul><ul><ul><li>Trick: During a master/slave replication, the slave table type can change </li></ul></ul><ul><ul><ul><li>Set up a separate MyISAM fulltext search farm </li></ul></ul></ul><ul><ul><ul><li>Connect to those servers when performing full-text searches </li></ul></ul></ul>
  25. Mistake #8: Development Infrastructure <ul><li>Every time a client has been in real trouble, they consistently fail to have a development infrastructure </li></ul><ul><ul><li>More then just CVS (although that’s a good start) </li></ul></ul><ul><ul><li>Establishing a development release process early-on is critical to the overall stability of your apps </li></ul></ul><ul><ul><ul><li>Things will go wrong at 3am in production </li></ul></ul></ul><ul><ul><ul><li>You need a process to release code to prevent the very-tempting cowboy-coding </li></ul></ul></ul>
  26. Development Infrastructure <ul><li>Maintaining an existing code base is often the most costly endeavor of any application </li></ul><ul><ul><li>As an application grows, the complexity of it’s release process must scale </li></ul></ul><ul><ul><li>Testing becomes more and more important </li></ul></ul><ul><ul><li>Your release process must be able to scale with your application! </li></ul></ul><ul><ul><ul><li>Staging environments </li></ul></ul></ul><ul><ul><ul><li>Coding Standards </li></ul></ul></ul>“ Scalability marginally impacts procedure, procedure grossly impacts scalability” - Theo Schlossnagle
  27. Mistake #9: Failing to Cache <ul><li>Caching is one of the most important things you can do when writing a scalable application </li></ul><ul><ul><li>A lot of people don’t realize how much they can cache </li></ul></ul><ul><li>You’ve already seen one form of caching in a previous trick of the trade </li></ul><ul><li>What about other techniques? </li></ul>
  28. Mistake #9: Failing to Cache <ul><li>Improving the speed of PHP can be done very easily using an opcode cache </li></ul><ul><li>PHP 6 will have this ability built-in to the engine </li></ul>
  29. Mistake 10: Not Knowing where to optimize <ul><li>Sooner or later, people worry about scalability </li></ul><ul><li>When trying to make scalability decisions, knowledge is the most important thing you can have </li></ul><ul><li>PHP has both closed source and open source profilers which do an excellent job of identifying the bottlenecks in your application </li></ul><ul><ul><li>Optimize where it counts </li></ul></ul>
  30. <ul><li>Instrumentation of your applications is key to determining what matters most when optimizing </li></ul><ul><ul><li>If you’re not logging, you’re shooting in the dark </li></ul></ul><ul><ul><li>White-box monitoring of your applications via tools like Zend Platform are enormously helpful in understanding what is going on </li></ul></ul><ul><ul><li>You can’t make good process (or business) decisions unless you understand how your web site is being used and by whom. </li></ul></ul>Mistake 10: Not Knowing where to optimize
  31. <ul><li>Amdahl’s Law: </li></ul><ul><ul><li>Improving code execution time by 50% when the code executes only 2% of the time will result in a 1% overall improvement </li></ul></ul><ul><ul><li>Improving code execution time by 10% when the code executes 80% of the time will result in a 8% overall improvement </li></ul></ul>Mistake 10: Not Knowing where to optimize
  32. <ul><li>Let’s imagine that each request sent over the wire is like a car driving from point A (the client) to point B (the server) </li></ul><ul><li>Roads are Networks </li></ul>Mistake 11: Because I give 110%
  33. One of the biggest problems with AJAX
  34. One of the biggest problems with AJAX <ul><li>Simple requests seem to work just fine… </li></ul>
  35. One of the biggest problems with AJAX
  36. One of the biggest problems with AJAX
  37. One of the biggest problems with AJAX
  38. One of the biggest problems with AJAX <ul><li>The problem with AJAX has to do with multiple dependent asynchronous requests </li></ul><ul><ul><li>You can’t rely on any order of operations in classical AJAX models </li></ul></ul>
  39. One of the biggest problems with AJAX
  40. One of the biggest problems with AJAX
  41. One of the biggest problems with AJAX
  42. One of the biggest problems with AJAX
  43. Some requests will happen faster <ul><li>When working with AJAX, always know you cannot rely on one request finishing before the next is triggered </li></ul><ul><li>Requests can take different lengths of time based on a huge array of factors </li></ul><ul><ul><li>Server load and Network load come to mind </li></ul></ul><ul><li>Can really mess up your application </li></ul><ul><li>Bad news: None of the current AJAX toolkits account for this latency </li></ul>
  44. Developing with Latency in mind <ul><li>A number of tools exist for developing AJAX applications with latency in mind </li></ul><ul><ul><li>AJAX Proxy is a good example </li></ul></ul><ul><ul><ul><li> </li></ul></ul></ul><ul><ul><ul><li>Allows you to simulate latency in your requests </li></ul></ul></ul><ul><ul><li>You can use it in conjunction with “SwitchProxy” to point your browser at a different proxy server to use it </li></ul></ul><ul><ul><ul><li> </li></ul></ul></ul><ul><li>Not a true solution, but at least let’s you test for the problem. </li></ul>
  45. Final Thoughts Final Thoughts <ul><li>Ultimately the secret of scalability is developing applications and procedures which scale both UP AND DOWN </li></ul><ul><li>You have to be able to afford to make the application to begin with </li></ul><ul><li>You have to be able to afford to make the application ten times bigger then it is </li></ul><ul><li>Without process, you will fail. </li></ul>Questions?