1Optimizing Your Cloud Applicationsin RightScaleRafael H. SaavedraVP of Engineering, RightScaleWatch the video of this pre...
2#Agenda  •   Introduction  •   3-tier application architecture  •   Vertical & horizontal scaling  •   RightScale monitor...
3#Cloud computing characteristics  •   Multi-tenancy  •   Shared resource pooling  •   Geo-distribution and ubiquitous net...
4#Cloud computing advantages  •   No upfront investment  •   Lowering operating costs  •   Highly scalable  •   Easy acces...
5#3-tier application architecture  • Load balancers  • An array of application servers  • Master-slaveReal Cloud Experienc...
6Optimizing Your Cloud Applications in RightScaleVertical & Horizontal Scaling
7#Cloud performance optimization  • Instance size (vertical scaling)  • Instance autoscaling (horizontal scaling)       • ...
8#Scaling up – spectrum of instance sizes  • Compute units vs memory                    128.0                             ...
9#Server arrays provide horizontal scalingReal Cloud Experience. Shared.
10#Server arrays provide horizontal scaling  • The array scales up or down based on performance votes       •   Tags allow...
11 Optimizing Your Cloud Applications in RightScaleMonitoring & Cluster Graphs     with RightScale
12#Server monitoring graphsReal Cloud Experience. Shared.
13#Cluster monitoring  • Individual graphs       • Good for a dozen servers       • Displays all standard graphs with full...
14#Cluster monitoring architecture • Architecture      • Monitoring front-end servers        pull data from storage server...
15#Cluster monitoring  • Current cluster monitoring: one graph per serverReal Cloud Experience. Shared.
16#Stacked graphs  • Each color band shows contribution of one server  • Servers are stacked on top of one anotherReal Clo...
17#Heat maps  • Each horizontal strip shows one server  • The color shows how “hot” the server is runningReal Cloud Experi...
18#Heat map with 100 serversReal Cloud Experience. Shared.
19#Stacked graph of the same 100 serversReal Cloud Experience. Shared.
20   Optimizing Your Cloud Applications in RightScaleApplication Performance Analytics          with New Relic
21#New Relic RPM  • Real-Time App Performance Analytics  • Supports Ruby, PHP, Java & .Net  • SQL & NoSQL performance  • W...
22#New Relic RPM  • Direct access from RightScale dashboardReal Cloud Experience. Shared.
23#New Relic RPM  • Historical statistics over a period of timeReal Cloud Experience. Shared.
24#New Relic RPM  • Distribution of the most time consuming requestsReal Cloud Experience. Shared.
25#New Relic RPM  • Statistics about response times from different countriesReal Cloud Experience. Shared.
26#New Relic RPM  • Detailed response times by browserReal Cloud Experience. Shared.
27#New Relic RPM – 2 Examples  • An expensive query  • The N+1 query problemReal Cloud Experience. Shared.
28   Optimizing Your Cloud Applications in RightScaleOptimizing Database Performance
29#Optimizing DB performance• RightScale MySQL ServerTemplates     • Configuration files tailored to instance size        ...
30#Schema considerations  • Lookups need to be indexed  • Sorting requires an index  • Joins need to be done on indices   ...
31#Monitoring DB performance  • Standard collectd statistics       • User vs wait time (disk operations)       • Performan...
32#MySQL Collectd Plugin  • Uses MySQL SHOW STATUS command to collect statistics  • A large set of counters that are divid...
33#MySQL Collectd Plugin  • Uses MySQL SHOW STATUS command to collect statisticsReal Cloud Experience. Shared.
34#Mysqlslow.log & explain commandReal Cloud Experience. Shared.
35#MySQL performance depends on locality  • Wait time should be minimum when working set fits in memory  • Performance deg...
36#MySQL reads graphs  • Read-random-next represents a table scan  • Read-next represents an index scanReal Cloud Experien...
37Optimizing Your Cloud Applications in RightScale            Load Testing
38#Load testing using httperf  • RightScale provides ServerTemplates in the marketplace       •   https://my.rightscale.co...
Upcoming SlideShare
Loading in …5
×

Optimizing Your Cloud Applications in RightScale

1,225 views
1,142 views

Published on

RightScale Conference Santa Clara 2011 - How do you tune application performance in the cloud? What if you want elasticity? We’ll talk through our experiences scaling and performance tuning the RightScale Cloud Management Platform in the cloud. Plus we’ll share tips for sizing, auto- scaling, monitoring, and troubleshooting large-scale cloud deployments.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,225
On SlideShare
0
From Embeds
0
Number of Embeds
46
Actions
Shares
0
Downloads
56
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • The cluster monitoring is very powerful in that it provides different types of views into the operation of large clusters of servers
  • The cluster monitoring is very powerful in that it provides different types of views into the operation of large clusters of servers
  • The architecture behind the cluster monitoring is rather extensiveCustomer (i.e. your) servers send monitoring data every 20 seconds to our serversThe data points are cached in-memory on those servers and flushed to disk periodicallyCluster monitoring graphs are produced on separate front-end servers, which pull the data from over 100 monitoring storage serversThe graphs are produced using rrdtool and auto-refresh
  • Walk through ofhow it works: in any deployment, go to the monitoring tab select servers select metric to plot familiar controls to switch time period and graph size displays one graph per server, here core1.rightscale.com through core8.rightscale.com in this example the graphs show cpu utilization for the past week, where blue is busy time and green is idle
  • Individual graphs only work for so many servers, they also don’t show what is happening as an aggregateStacked graphs stack the contribution of each server on top of one anotherWalk through what the graph shows
  • Stacked graphs are great to see the aggregate, but it is often difficult to see abnormal server behaviorHeat maps show many servers on one graph by plotting one horizontal bar per serverThe time axis is the same for all servers and it is shown at the bottom of the graphThe color of the bar shows the value of the metric for the serverWalk through the graphIt’s easy to see that there are 6 servers sharing the load, and two servers that are different
  • At scale this is how all this looks and comes togetherThis example is real, it shows an incident we had with our monitoring cluster a few months agoThis heat map shows 100 servers out of one of our monitoring clusters (we want to be vague here…)When there are more than 100 servers, the heat map shows a sampling of 100Describe the sampling: most recently launched, longest running, some of each server template, rest randomStory:This heat map plots I/O wait for our monitoring servers on a day where we suddenly received a number of alerts for a few serversThe heap map shows these servers clearly as red bands starting between 7am and 8amSo we could clearly see that something was going on with a small number of servers and that it started more or less at the same time on all themTo see what happened in aggregate, we can switch graph type…
  • This shows the same incident as on the previous slide, but with a timescale of a weekIt shows the number of servers handled by each monitoring server, i.e. each color bar shows one serverIt is easy to see that some customer launched a large number of servers right at the time the overload beganFurther investigation showed that due to a bug these servers were allocated unevenly across the cluster causing the overload’
  • Optimizing Your Cloud Applications in RightScale

    1. 1. 1Optimizing Your Cloud Applicationsin RightScaleRafael H. SaavedraVP of Engineering, RightScaleWatch the video of this presentation
    2. 2. 2#Agenda • Introduction • 3-tier application architecture • Vertical & horizontal scaling • RightScale monitoring and cluster graphs • New Relic RPM • Support for optimizing DB performance • Load testingReal Cloud Experience. Shared.
    3. 3. 3#Cloud computing characteristics • Multi-tenancy • Shared resource pooling • Geo-distribution and ubiquitous network access • Service oriented • Dynamic resource provisioning • Self-organizing • Utility based pricingReal Cloud Experience. Shared.
    4. 4. 4#Cloud computing advantages • No upfront investment • Lowering operating costs • Highly scalable • Easy access • Reduces business risk and maintenance costs • Enables process automationReal Cloud Experience. Shared.
    5. 5. 5#3-tier application architecture • Load balancers • An array of application servers • Master-slaveReal Cloud Experience. Shared.
    6. 6. 6Optimizing Your Cloud Applications in RightScaleVertical & Horizontal Scaling
    7. 7. 7#Cloud performance optimization • Instance size (vertical scaling) • Instance autoscaling (horizontal scaling) • Server arrays • RightScale support for performance optimization • ServerTemplates are configured to capture performance data • Collectd RightScripts • Hardware & OS monitoring data • Specialized plugins – MySQL, HAProxy, Apache, NgInx, IIS, etc • Monitoring graphs: individual, cluster, stacked, heat maps • Alerts & escalations • New Relic RPMReal Cloud Experience. Shared.
    8. 8. 8#Scaling up – spectrum of instance sizes • Compute units vs memory 128.0 m2.4xlarge 64.0 m2.2xlarge 32.0 cc1/cg1.4xlarge m2.xlarge c1.xlarge Memory (GB) 16.0 m1.xlarge Scalable m1.large Applications 8.0 High Performance 4.0 Computing m1.small 2.0 c1.medium 1.0 Test & Dev t1.micro 0.5 0.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0 40.0 Compute UnitsReal Cloud Experience. Shared.
    9. 9. 9#Server arrays provide horizontal scalingReal Cloud Experience. Shared.
    10. 10. 10#Server arrays provide horizontal scaling • The array scales up or down based on performance votes • Tags allow scaling on an arbitrary decision set • Decision threshold controls reaction time • Sleep time allows new resources to have an impact • Scaling can be time dependent • Detailed setup instructions: http://bit.ly/c1oLr2 • Fast response to changes in load conditions using alerts • Allocation of servers to availability zones based on weights • Deployment-based so configuration is consistent • Arrays can be pre-scaled to support anticipated demandReal Cloud Experience. Shared.
    11. 11. 11 Optimizing Your Cloud Applications in RightScaleMonitoring & Cluster Graphs with RightScale
    12. 12. 12#Server monitoring graphsReal Cloud Experience. Shared.
    13. 13. 13#Cluster monitoring • Individual graphs • Good for a dozen servers • Displays all standard graphs with full detail • Stacked graphs • Displays the contribution of many servers to a total • Great to see the sum and variability of activity in a cluster • Difficult to make out individual servers • Examples: requests/sec, cpu busy cycles, I/O bytes/sec • Heat maps • Displays a bar for each server • Great to see uneven distribution across servers • Great to quickly spot performance problems across many servers • Difficult to read absolute values or see the total cluster activityReal Cloud Experience. Shared.
    14. 14. 14#Cluster monitoring architecture • Architecture • Monitoring front-end servers pull data from storage servers • Up to 100 servers on one graph (to be increased) monitoring monitoring storage front-end servers servers your serversReal Cloud Experience. Shared.
    15. 15. 15#Cluster monitoring • Current cluster monitoring: one graph per serverReal Cloud Experience. Shared.
    16. 16. 16#Stacked graphs • Each color band shows contribution of one server • Servers are stacked on top of one anotherReal Cloud Experience. Shared.
    17. 17. 17#Heat maps • Each horizontal strip shows one server • The color shows how “hot” the server is runningReal Cloud Experience. Shared.
    18. 18. 18#Heat map with 100 serversReal Cloud Experience. Shared.
    19. 19. 19#Stacked graph of the same 100 serversReal Cloud Experience. Shared.
    20. 20. 20 Optimizing Your Cloud Applications in RightScaleApplication Performance Analytics with New Relic
    21. 21. 21#New Relic RPM • Real-Time App Performance Analytics • Supports Ruby, PHP, Java & .Net • SQL & NoSQL performance • Web transaction tracing • Performance notifications • Availability monitoring • Scalability analysisReal Cloud Experience. Shared.
    22. 22. 22#New Relic RPM • Direct access from RightScale dashboardReal Cloud Experience. Shared.
    23. 23. 23#New Relic RPM • Historical statistics over a period of timeReal Cloud Experience. Shared.
    24. 24. 24#New Relic RPM • Distribution of the most time consuming requestsReal Cloud Experience. Shared.
    25. 25. 25#New Relic RPM • Statistics about response times from different countriesReal Cloud Experience. Shared.
    26. 26. 26#New Relic RPM • Detailed response times by browserReal Cloud Experience. Shared.
    27. 27. 27#New Relic RPM – 2 Examples • An expensive query • The N+1 query problemReal Cloud Experience. Shared.
    28. 28. 28 Optimizing Your Cloud Applications in RightScaleOptimizing Database Performance
    29. 29. 29#Optimizing DB performance• RightScale MySQL ServerTemplates • Configuration files tailored to instance size • innodb_buffer_pool_size • key_buffer_size • thread_size • sort_buffer_size• The never ending task of identifying current bottlenecks • Disk seeks • Performance of disk operations • Scale up when working set cannot fit in memory – avoid active swapping • Constant monitoring of performance graphs, logs and query• Schema considerationsReal Cloud Experience. Shared.
    30. 30. 30#Schema considerations • Lookups need to be indexed • Sorting requires an index • Joins need to be done on indices • Become slower as tables grow • Compounded indices should be used consistently • Do not abuse indices • Each index requires a disk write • Compact tables if they become fragmented • Deleted rows do not remove the corresponding index entriesReal Cloud Experience. Shared.
    31. 31. 31#Monitoring DB performance • Standard collectd statistics • User vs wait time (disk operations) • Performance of disk operations • Scale up when working set cannot fit in memory • MySQL collectd plugin • Monitor INSERT, SELECT, UPDATE operations • The breakdown of read operations can indicate missing indices • Monitoring /var/log/mysqlslow.log file • Identify slow queries • Use MySQL EXPLAIN command to identify query planReal Cloud Experience. Shared.
    32. 32. 32#MySQL Collectd Plugin • Uses MySQL SHOW STATUS command to collect statistics • A large set of counters that are divided into 10 categories • Connections • IO Requests • Select Rates • Read Rates • Key Rates • Commands Rates • Query Cache • Tables • Memory • Misc.Real Cloud Experience. Shared.
    33. 33. 33#MySQL Collectd Plugin • Uses MySQL SHOW STATUS command to collect statisticsReal Cloud Experience. Shared.
    34. 34. 34#Mysqlslow.log & explain commandReal Cloud Experience. Shared.
    35. 35. 35#MySQL performance depends on locality • Wait time should be minimum when working set fits in memory • Performance degrades once wait time is significant wait time insignificant user time dominatesReal Cloud Experience. Shared.
    36. 36. 36#MySQL reads graphs • Read-random-next represents a table scan • Read-next represents an index scanReal Cloud Experience. Shared.
    37. 37. 37Optimizing Your Cloud Applications in RightScale Load Testing
    38. 38. 38#Load testing using httperf • RightScale provides ServerTemplates in the marketplace • https://my.rightscale.com/library/server_templates/Httperf-Load-Tester/24714 • Tutorial on httperf setup and configuration • http://support.rightscale.com/03-Tutorials/02-AWS/E2E_Examples/E2E_Gaming_Deployment/Adding_Httperf_Load_TesterReal Cloud Experience. Shared.

    ×