Scaling Drupal: Not IF... HOW

  • 3,973 views
Uploaded on

A brief overview of how to plan for, discover, and treat performance problem areas in Drupal.

A brief overview of how to plan for, discover, and treat performance problem areas in Drupal.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • Drupal cahce and performance and optimization
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
3,973
On Slideshare
0
From Embeds
0
Number of Embeds
6

Actions

Shares
Downloads
114
Comments
1
Likes
10

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • A quick intro of where the session will be goingNote that focus is how to deal with problems on an existing site, rather than building a new site
  • People are always asking, \"Does Drupal scale?\"
  • All these sites attest, it can!
  • Please revise this as needed
  • I really don't know what else should go here. Please add as appropriate.
  • On Mac / MAMP:/Applications/MAMP/Library/my.cnf/etc/my.cnfNote: you can copy one from under the/Applications/MAMP/Library/share/mysql/
  • Discuss:What load testing isWhat jMeter isWhat makes an effective plan
  • I:Tools that I commonly use:JmeterFoxyProxyII:Initialize Test planCreate a thread groupSet HTTP Request DefaultsCreate Test planIII: Options for creating planManually create a planScript one using a proxy server.IV: Scripting Test PlanSet up jmeter proxy serverSet up and enable foxy proxy FF pluginClick through your site on FF
  • jMeter Execution stepsStart slow to make sure the test plan works and makes senseDecide on the number of users you wanna serve in a given period of time. What we expect to get out of thisQueries into the slow query logProblem pages
  • jMeter ResultsVariety of reportings available:GraphsResponse time tablesSummary reports, etc.We’ll get into it more in a bit….
  • Discuss:How to find logsHow to identify \"slow queries\"--what constitutes slow, how to identify them in the logs
  • Discuss:How to find logsHow to identify \"slow queries\"--what constitutes slow, how to identify them in the logs
  • One example (the first query on the previous slide)
  • Potential pitfalls of queriespoor indexinguseless joins and/or group bysreturning too much dataGood practices for writing querieskeep it simpleknow your databe consistent and plan ahead - this will help with generating efficient indexes
  • Interesting information:1) Fastest page response2) Slowest page response3) 90% time4) Errors
  • By enabling the Devel module:1) View the listing of queries executed2) Make note of any:Slow QueriesQueries that are repeated frequentlyAnything that is being executed yet really shouldn’t be
  • Optimize slow queriesLocate if any of the items being executed are the results of blocks or segments that can be cached in some wayWhat ways? - see slide
  • Following slides will outline postives and negatives…Types of caching engines:Caching Engines- Standard Drupal Cache - positive - provides caching out of the box - negative - relies upon the database for storage, which results in a lot of reads/writes which can result in table or row locks, etc.- APC - standard drupal cache + memory? - positive - faster performance than memcached    - negative - not networkable, so not suited to multiple webheads… per server- Memcache -most common - positive -standard drupal cache + memory + multiple servers - negative - limited control over when and how things disappear from cache- Static page cache / Boost - positive - drupal flexibility + performance of static files - negative - generally works only with anonymous users
  • Types of configurable cache:
  • Page cache- positive - allows full pages to be cached, and thus we can serve many more page views on a single server- negative - only works for anonymous users
  • Block cache- positive - configurable to work with anonymous users, as well as logged in users- positive - can be page specific, user specific, both or site wide and set to expire at specified intervals- negative - hard to control the expiration of data outside of the specified intervals(i.e. if a block only shows a certain node type, you can set the cache to regenerate when any node type is created)
  • Types of programmatic cache:Static variable cache- Programmatic data cache- creating caching strategies within your custom modules- you can also do this in the theme layer (if a particular part on a user's homepage takes a lot of queries or effort to run, you could consider caching it for even 5 minutes)
  • only retrieve a single variable once per session- positive - eliminates often duplicate rendered objects- positive - core already uses this in some places so that a second node_load() in a single page request doesn't hit the database again.
  • creating caching strategies within your custom modules- you can also do this in the theme layer (if a particular part on a user's homepage takes a lot of queries or effort to run, you could consider caching it for even 5 minutes)
  • Discuss:Why ongoing monitoring is importantWhat Cacti isHow Cacti should be used
  • - Free!- Monitors various aspect of your site. such as: - cpu utilization - server load - memory usage - database activity- Historical trends graphically represented- There are (unsupported) community add-ons for things like SOLR
  • Discuss:Why ongoing monitoring is importantWhat Cacti isHow Cacti should be used
  • - Database Scaling - Addition of read-only slave servers - InnoDB / MyISM / Memory optimization? - You must currently patch modules or add slave queries to your own modules to take advantage of this. - Drupal 7's new database layer should let us do this automatically- Hardware changes - Load balancers - caching - redundancy
  • Not sure how you want to go with this part, if you want to cover each item individually like the others, or talk very briefly about each.Bullets randomly grabbed from notes...please change as you like.
  • Front-end performance matters!If a page takes 0.5s to generate and 7s to render, it will still seem slowUse YSlow! to get a score on how fast your website's front-end isJS Aggregation / CompressionSimilar to CSS AggregationLoad JS on the bottomJS minification is one step further down this road, but it is not safe to do without using a Java-based minifier like YUI Compressor or Dojo Shrinksafe
  • YSlow! in actionbefore optimizing:The site is 212.7Kfor an authenticateduser and will take almost 6 seconds torender.- YSlow! gives you aset of bullet pointsthat you can work on.- Each one has ascore associatedwith it, and we barelyhad a D to start.
  • - With Drupal's JSaggregation turned on,the site already scoresmuch better.- The page has notgotten much smallerin page size (only .6kis saved because coredoesn't compress JS)- The page is almost33% faster to load andour score is muchbetter.
  • - CSS sprites let you reuse a single image as a background on multiple objects, again dropping the number of http requests
  • Not sure how you want to go with this part, if you want to cover each item individually like the others, or talk very briefly about each.Bullets randomly grabbed from notes...please change as you like.
  • the site with all the images, css and js on SimpleCDN.- SimpleCDN lets youserve up your static content with gzipcompression and expiry headers far in the future.- With the content from the CDN gzipped, the page size is 40% of its original and the load time is down to 45%.
  • Not sure how you want to go with this part, if you want to cover each item individually like the others, or talk very briefly about each.Bullets randomly grabbed from notes...please change as you like.
  • - Here's the final resultwith our Apache serversending far-futureexpiry headers andgzipping html content.- To boost from a scoreof 90 - 99, I also broughtthe ShareThis module tothe 1.4 version, whichstopped it from beingincluded on every page.- Overall savings:212.7k - 99.96k page size5.7s - 1.7s page load timea
  • These are my inferences. Please add or change as appropriate.

Transcript

  • 1. TreeHouseAgency.com! Who?
  • 2. Thomas Wysocki! Lead Architect! Thomas@treehouseagency.com! Steven Merrill! Sr. Developer! Steven@treehouseagency.com!
  • 3. What we're doing today Identify scaling issues!   Introduce an optimization   process! Look at useful testing and   monitoring tools!
  • 4. Does Drupal Scale?
  • 5. When is scaling a concern? When there are a lot of:!   Pageviews!   Authenticated users!  
  • 6. What if a site doesn't scale? Crash!!
  • 7. Enable slow query logging! Test Plan! Ongoing Monitoring! Analyze Optimize slow queries! queries! Load Site Live?! Front-end Optimization! Testing! Analyze Additional! slow Caching! pages! Additional Techniques!
  • 8. Enable Slow Query Log •  Ensure you have a my.cnf active! •  Enable the logging of slow queries inside my.cnf  !
  • 9. Enable Slow Query Log  !  
  • 10. Load testing - why?   Simulate traffic before your site hits the real world!   Populate the site with data automatically!
  • 11. Load testing - how?   Utilize load testing software or services!   HP Load Runner!   OpenLoad!   jMeter !
  • 12. Load testing - differences? Cost!   Complexity!   Hardware requirements!   Types of testing available!  
  • 13. Load Testing - jMeter Note: I have a swf video for now but   not sure how to embed that here.   See attachment via email.!
  • 14. Load Testing - jMeter - Execute  !  
  • 15.  !  
  • 16. Reviewing slow query logs Check if any files were output to you 1.  slow log file.! Generate a summary of the log file 2.  using mysqldumpslow!
  • 17. Reviewing slow query logs  !  
  • 18. Slow Query Logs •  Typical Problems:! Poor Indexing resulting in Table   scans! Locking!   large result sets !   Slow, complicated queries!  
  • 19. Slow Query Log For example:!  
  • 20. Query Optimization Things to keep in mind!   keep it simple!   know your data!   be consistent and plan ahead !  
  • 21. Analyze Page execution time screenshot!  
  • 22. Test with the devel module features, screenshot, screencast?!  
  • 23. Implement caching •  Types of cache! Caching Engines!   Drupal Configurable cache!   Programmatic caching!  
  • 24. Implement caching •  Types of caching engines:! Standard Drupal Cache!   APC - standard drupal cache +   memory?! Memcache!   Static page cache / Boost !  
  • 25. Standard Drupal Cache •  positive ! provides caching out of the   box! •  negative ! relies upon the database for   storage, which results in a lot of reads/writes which can result in table or row locks, etc.!
  • 26. APC •  positive ! much faster performance than   memcached! •  negative ! not networkable, so not suited   to multiple webheads!
  • 27. Memcache •  positive ! standard drupal cache +   memory + multiple servers! •  negative ! limited control over when and   how things disappear from cache!
  • 28. Static Page Cache/Boost •  positive! drupal flexibility +   performance of static files! •  negative ! generally works only with   anonymous users!
  • 29. Implement caching •  Types of configurable cache:! Page cache!   Block Cache!  
  • 30. Page Cache •  positive ! allows full pages to be   cached, and thus we can serve many more page views on a single server! •  negative ! only works for anonymous   users!
  • 31. Block Cache •  positive ! configurable to work with   anonymous users, as well as logged in users! can be page specific, user specific,   both or site wide and set to expire at specified intervals! •  negative ! hard to control the expiration of   data outside of the specified intervals!
  • 32. Implement caching •  Types of programmatic cache:! Static variable cache!   Programmatic data cache!  
  • 33. Static Variable Cache • positive !  eliminates often duplicate rendered objects!  core already uses this in some places so that a second node_load() in a page ! request doesn't hit the database • negative !  Only applicable in the course of a single web request!
  • 34. Programmatic Data Cache •  positive ! Expensive items can be cached   for a period of time! Easily added into custom   modules or custom themes! •  negative ! Difficult to know where to do   it, and invalidating content becomes complicated!
  • 35. Ongoing monitoring •  Tools available! Cacti!   MySql Enterprise Manager!   Gomez!  
  • 36. Ongoing monitoring - Cacti Free!!   Monitors various aspect of your site.  !   cpu utilization!   server load!   memory usage!   database activity!   Trend visualization!   Community add-ons !  
  • 37.  !  
  • 38. Additional measures Database Scaling!   Addition of read-only slave servers!   InnoDB / MyISM / Memory optimization!   You must currently patch modules or add   slave queries to your own modules! Drupal 7 should inherently let us do this !   Hardware changes!   Load balancers!   caching!   redundancy!  
  • 39. Additional measures Front-end techniques!   Apache tweaks!   CDN!  
  • 40. Front end Techniques Front-end performance matters!!   If a page takes 0.5s to generate and   7s to render, it will still seem slow! Use YSlow! to get a score on how   fast your website's front-end is! JS Aggregation / Compression!   Similar to CSS Aggregation !   Load JS on the bottom!  
  • 41. Front end Techniques YSlow! in action!   before optimizing:!   212.7K (auth user)!   Almost 6 seconds!   Barely earns a ‘D’!  
  • 42. Front end Techniques JS Aggregation!   Same size!   33% faster!   Better score!  
  • 43. Front end Techniques JS Aggregation!   Core in D6, Available as contrib in D5!   JS Minification!   Unsafe in PHP!   CSS Aggregation!   Available out of the box, extremely useful   in reducing the number of http requests needed for each page.   !
  • 44. CDN Load static assets from several   points around the country or world! Eliminates unnecessary load on your   web servers! Relatively cheap!   Easy to implement   !   As cloud storage evolves, CDN   integration is becoming commoditized, like SimpleCDN!
  • 45. Front end Techniques All images, css   and js on SimpleCDN! Gzip   compression! Far-future   expiry headers! Page size 40% of   original! Load time down   45%!
  • 46. Apache Techniques GZip compression!   Move .htaccess   configurations inside your http.config!
  • 47. Results Far-Future   Expiry Headers! Gzipping!   Updated   ShareThis! 212.7k to 99.96k!   5.7s to 1.7s!   D to A!!  
  • 48. Key takeaways Follow a logical, deliberate   process! Leverage the tools available   to you! Use a multi-faceted approach  !   Test, test, test!!