Capacity Planning with Free Tools

15,686 views

Published on

Half day workshop slides that have been presented at Computer Measurement Group for the last few years, and at Usenix 08 and LISA 08. This version is what will be presented at Usenix 09, San Diego, June 16th, along with the Solaris/Linux Performance slide deck.

Published in: Technology
1 Comment
33 Likes
Statistics
Notes
No Downloads
Views
Total views
15,686
On SlideShare
0
From Embeds
0
Number of Embeds
134
Actions
Shares
0
Downloads
0
Comments
1
Likes
33
Embeds 0
No embeds

No notes for slide

Capacity Planning with Free Tools

  1. 1. Performance Management with Free and Bundled Tools Adrian Cockcroft Netflix Inc. acockcroft@netflix.com (Co-authored with Mario Jauvin MFJ Associates mario@mfjassociates.net) 2 May, 2009 The Performance People
  2. 2. Agenda Overview of Capacity Planning Requirements  and Data Sources Performance Data Collection  Free Network Monitoring Tools  Free System Monitoring Tools  Free Load Generation and Modelling Tools  Licences and References  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  3. 3. What are we talking about? QA Load generation with Grinder or SLAMD, modelling with PDQ and R Network monitoring with Application Tier WireShark, MRTG, monitoring with Orca, Hobbit/Xymon, Cacti, Hobbit/Xymon, BigSister, Cacti, BigSister, Ganglia, Nagios, OpenNMS, XEtoolkit Zenoss, Openxtra, ntop Database Tier monitoring With SEtoolkit, Orca, XEtoolkit May 2, 2009 Adrian Cockcroft and Mario Jauvin
  4. 4. Capacity Planning Requirements and Data Sources May 2, 2009 Adrian Cockcroft and Mario Jauvin
  5. 5. Definitions Capacity  – Resource utilization and headroom Planning  – Predicting future needs by analyzing historical data and modeling future scenarios Performance Monitoring  – Collecting and reporting on performance data Free Tools  – Bundled with the OS or available for no $$$ May 2, 2009 Adrian Cockcroft and Mario Jauvin
  6. 6. Capacity Planning Requirements We care about CPU, Memory, Network and Disk  resources, and Application response times We need to know how much of each resource we  are using now, and will use in the future We need to know how much headroom we have to  handle higher loads We want to understand how headroom varies, and  how it relates to application response times and throughput May 2, 2009 Adrian Cockcroft and Mario Jauvin
  7. 7. Capacity Planning Challenges Constantly changing infrastructure  Limited attention span from staff  Horizontally scaled commodity systems  Per node software licencing costs too much  Too many tools, too many agents per node  Too much data, not enough analysis  Non-linear and non-intuitive scalability  Lack of tools and metrics for virtualized resources  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  8. 8. Observability Four different viewpoints  Management – Engineering – QA Testing – Operations – Each needs very different information  Ideal would be different views of the same  performance database Reality is a mess of disjoint tools  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  9. 9. Management Viewpoint Daily summary of status and problems  Business oriented metrics  Future scenario planning  Marketing and management input  Concise report with dashboard style status  indicators Free tools: R, Spreadsheet and Web  based displays, no good summarization tools May 2, 2009 Adrian Cockcroft and Mario Jauvin
  10. 10. Engineering Viewpoint Large volumes of detailed data at several different  time scales Input to tuning, reconfiguring and future product  development Low level problem diagnosis  Detailed reports with drill down and correlation  analysis Free tools: XE/SE Toolkit, Orca, Ganglia, Cacti, R  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  11. 11. QA Test Viewpoint Workload specification tools  Load generation frameworks  Testing for functionality and performance  Regression tools to compare releases  Modelling difference between test configuration  and production configuration Free Tools: The Grinder, SLAMD, R, PDQ  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  12. 12. Operations Viewpoint Immediate timeframe  Real time display, updated in seconds  Alert based monitoring  High level problem diagnosis  Simple high level graphs and views  Free tools: Hobbit/Xymon, BigSister,  Nagios, OpenNMS, MRTG, Cacti, Ganglia, WireShark, ntop May 2, 2009 Adrian Cockcroft and Mario Jauvin
  13. 13. Measurement Data Interfaces Several generic raw access methods  Read the kernel directly (not a good idea) – Structured system data (Solaris kstat, Linux /proc) – Process data – Network data – Accounting data – Application data – Command based data interfaces  Scrape data from vmstat, iostat, netstat, sar, ps – Higher overhead, lower resolution, missing metrics – Data available is platform specific either way  Much more detail on this topic in the Solaris/Linux Performance  Measurement and Tuning slide deck May 2, 2009 Adrian Cockcroft and Mario Jauvin
  14. 14. Free Network Monitoring Tools May 2, 2009 Adrian Cockcroft and Mario Jauvin
  15. 15. SNMP Simple network management protocol  UDP protocol based on port 161  Client/server like  Client is called management application entity – Server is called an agent entity – Agent entity is designed to be implemented  on network hardware, router, switches, etc May 2, 2009 Adrian Cockcroft and Mario Jauvin
  16. 16. SNMP – MIBs Management information base  Defines the structure and the semantic of the  information that can be reported on Most commonly used is MIB-II which defines a set of  standard networking attributes Interface tables – System level information – Routing tables – Specified using ASN.1 (abstract syntax notation 1)  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  17. 17. SNMP – commands Called PDU (protocol data units)  GET  GETNEXT  GETBULK  SET  Encoded using BER (basic encoding rules)  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  18. 18. Versions Version 1, original version done in May 1991  Version 2, around 1993. Failed because the  IETF credo of “rough consensus and running code” could not be met on securing SNMP Turned into V2c for community string security  (like V1) Version 3, added security and complexity in  1998 May 2, 2009 Adrian Cockcroft and Mario Jauvin
  19. 19. SNMP tools Too numerous to name all but…  OpenNMS  Nagios  Cacti  MRTG  Net-snmp  See www.snmplink.org – May 2, 2009 Adrian Cockcroft and Mario Jauvin
  20. 20. SNMP tools Snmpwalk – will report all data in a specified  MIB getIf – will report data about interfaces and  includes built-in MIB browser Snmptable – will report tabular data from MIB  tables May 2, 2009 Adrian Cockcroft and Mario Jauvin
  21. 21. OpenNMS Well…. it’s not that portable  95% java is not 100% java – Requires about 20-30 different platform specific – packages (PostgreSQL, Perl, RRD tool, Tomcat 4 etc…) Difficult to install – Easy auto discovery – Web-based interface – May 2, 2009 Adrian Cockcroft and Mario Jauvin
  22. 22. OpenNMS Main screen shot  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  23. 23. OpenNMS Node screen shot  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  24. 24. Nagios Easy to build/compile (on Solaris 10)  Easy to install  Quick response from CGI  Configuration is manual and a pain  13 configuration files with all kinds of interrelated – entries Tedious and error prone – Requires plugins to do anything  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  25. 25. Nagios Main screen shot  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  26. 26. Nagios Host detail screen shot  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  27. 27. May 2, 2009 Adrian Cockcroft and Mario Jauvin
  28. 28. ntop Similar to familiar UNIX top tool for  processes but used for network Provide huge selection of real-time data  Can be found at http://www.openxtra.co.uk/  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  29. 29. ntop – Active Sessions May 2, 2009 Adrian Cockcroft and Mario Jauvin
  30. 30. ntop Hosts May 2, 2009 Adrian Cockcroft and Mario Jauvin
  31. 31. ntop Network Load May 2, 2009 Adrian Cockcroft and Mario Jauvin
  32. 32. ntop_Network_Thruput May 2, 2009 Adrian Cockcroft and Mario Jauvin
  33. 33. ntop Port Dist May 2, 2009 Adrian Cockcroft and Mario Jauvin
  34. 34. ntop_Protocol_Dist May 2, 2009 Adrian Cockcroft and Mario Jauvin
  35. 35. ntop Protocols May 2, 2009 Adrian Cockcroft and Mario Jauvin
  36. 36. Zenoss Open source monitoring and management of  IT infrastructure Zenoss core is free  Other editions are for a fee  Get it from http://www.zenoss.com/download/  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  37. 37. zenoss Architecture May 2, 2009 Adrian Cockcroft and Mario Jauvin
  38. 38. zenoss Dash Config May 2, 2009 Adrian Cockcroft and Mario Jauvin
  39. 39. zenoss Google May 2, 2009 Adrian Cockcroft and Mario Jauvin
  40. 40. zenoss Google Alerts May 2, 2009 Adrian Cockcroft and Mario Jauvin
  41. 41. Zenoss Graphs May 2, 2009 Adrian Cockcroft and Mario Jauvin
  42. 42. zenoss Topology May 2, 2009 Adrian Cockcroft and Mario Jauvin
  43. 43. Zeroshell 5/2/2009 Adrian Cockcroft and Mario Jauvin
  44. 44. Zeroshell (cont) Small Linux distribution for servers and  embedded devices Provides network services:  Load balancing – RADIUS – QoS – Captive portal, proxy, wireless access point – 5/2/2009 Adrian Cockcroft and Mario Jauvin
  45. 45. IPT-Netflow Netflow Linux kernel module for collecting  per connection traffic Designed for Linux router with heavy network  load 5/2/2009 Adrian Cockcroft and Mario Jauvin
  46. 46. NFDUMP and NfSen NFDUMP is a set of tools to capture/record,  dump, filter and replay Netflow data Filters based on user-defines profiles – NfSen is a graphical Web-based front-end for  NFDUMP 5/2/2009 Adrian Cockcroft and Mario Jauvin
  47. 47. NetTraMet and others Meter for network traffic flows  First implementation of real time flow  measurement (RTFM) or RFC 1272 Flow-tools  Software package for collecting and processing – NetFlow data See HUGE list of flow management tools at  http://www.switch.ch/network/projects/completed/ – TF-NGN/floma/software.html 5/2/2009 Adrian Cockcroft and Mario Jauvin
  48. 48. Internet2 Performance Node: Live CD perfSONAR Network Performance Toolkit  2.0 Customized version of a Knoppix (bootable  Linux distribution) Ping/traceroute  OWANP (one-way ping)  BWCTL (bandwidth Test Controller)  Iperf (measure maximum TCP bandwidth) – Trulay – 5/2/2009 Adrian Cockcroft and Mario Jauvin
  49. 49. Internet2 NetFlow reports Interactive weekly reports  Dates back to 2002  Bulk data (more than 10MB) and full data  Throughputs, size, durations, percentiles  Application types  5/2/2009 Adrian Cockcroft and Mario Jauvin
  50. 50. Internet2 NetFlow reports 5/2/2009 Adrian Cockcroft and Mario Jauvin
  51. 51. MRTG Really simple to install and configure  Require manual config file creation  Only for MIB-II interface plotting out of the  box Graphing not flexible, axis, time etc  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  52. 52. MRTG Interface screen shot  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  53. 53. MRTG Other CPU screen shot  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  54. 54. RRD tool Software to store, retrieve and graph  numerical time series data Use a round robin algorithm  Data files are a fixed size  Don’t grow – Don’t require maintenance – May 2, 2009 Adrian Cockcroft and Mario Jauvin
  55. 55. RRD tool Compiles on most platforms  Used by many SNMP based tools  OpenNMS – Cacti – BigSister – WeatherMap4RRD – MailGraph – May 2, 2009 Adrian Cockcroft and Mario Jauvin
  56. 56. RRD tool 14all CGI script that plots data similar to  MRTG Configurable to collect data at different  interval (unlike MRTG) Flexible and variable in what data can be  collected May 2, 2009 Adrian Cockcroft and Mario Jauvin
  57. 57. RRD tool Sample screen shot  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  58. 58. RRD tool Screen shot  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  59. 59. RRD tool Create a RRD database  rrdtool create test.rrd --start 920804400 DS:speed:COUNTER:600:U:U RRA:AVERAGE:0.5:1:24 RRA:AVERAGE:0.5:6:10 May 2, 2009 Adrian Cockcroft and Mario Jauvin
  60. 60. RRD tool Create a graph  rrdtool graph speed.png --start 920804400 --end 920808000 DEF:myspeed=test.rrd:speed:AVERAGE LINE2:myspeed#FF0000 May 2, 2009 Adrian Cockcroft and Mario Jauvin
  61. 61. Free Performance Data Collection and Rules Toolkits May 2, 2009 Adrian Cockcroft and Mario Jauvin
  62. 62. SE toolkit Example Tools A free performance toolkit for rapidly creating custom data sources  Makes all the very extensive Solaris metrics easily available  Very system specific and not enough metrics exist to port to Linux  Written by Rich Pettit with contributions from Adrian Cockcroft  Get SE3.4 from http://sourceforge.net/projects/setoolkit/  Open source with support for SPARC & x86 Solaris 8, 9, 10  Function Example SE Programs Rule Monitors cpg.se monlog.se mon_cm.se live_test.se percollator.se zoom.se virtual_adrian.se virtual_adrian_lite.se Disk Monitors siostat.se xio.se xiostat.se iomonitor.se iost.se xit.se disks.se CPU Monitors cpu_meter.se vmmonitor.se mpvmstat.se Process Monitors msacct.se pea.se ps-ax.se ps-p.se pwatch.se pw.se Network Monitors net.se tcp_monitor.se netmonitor.se netstatx.se nfsmonitor.se nx.se Clones iostat.se uname.se vmstat.se nfsstat-m.se perfmeter.se xload.se Data browsers aw.se infotool.se multi_meter.se Contributed Code anasa dfstats kview systune watch orcollator.se Test Programs syslog.se cpus.se pure_test.se collisions.se uptime.se dumpkstats.se net_example nproc.se kvmname.se May 2, 2009 Adrian Cockcroft and Mario Jauvin
  63. 63. SE language features SE is a 64bit interpreted dialect of C  – Not a new language to learn from scratch! – Standard C /usr/ccs/bin/cpp used at runtime to preprocess SE scripts – Main omissions - pointer types and goto – Main additions - classes and “string” type – powerful ways to handle dynamically allocated data – built-in fast balanced tree routines for storing key indexed data Dynamic linking to all existing C libraries  – Built-in classes access kernel data – Supplied class code hides details, provides the data you want Example scripts improve on basic utilities e.g. siostat.se, nx.se, pea.se  Example rule based monitors e.g. virtual_adrian.se, orcallator.se  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  64. 64. Creating Rules Based on real experiences of all the things that go  wrong Capture an approximation to intuition  Test and calibrate rules on as many systems as  possible Easy??  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  65. 65. Configuring Rules Thresholds should be configured  Very application dependent  Capture the operating envelope  Measure the underlying values – Measure peaks in normal operation – Note values during problems – Set thresholds to capture the difference – This applies to any tool  SE Toolkit, Cacti, Ganglia, Nagios, OpenNMS etc. – May 2, 2009 Adrian Cockcroft and Mario Jauvin
  66. 66. Rules as Objects Define only the input and output information  Hide implementation details  Make high level rule objects trivial to use and  reuse SE Toolkit does it in three lines of code:  #include <rules file> – Declare rule object as a typed variable – Read and use or print object status – May 2, 2009 Adrian Cockcroft and Mario Jauvin
  67. 67. quot;virtual adrianquot; rules summary Disk Rule for all disks at once  – Looks for slow disks and unbalanced usage Network Rule for all networks at once  – Looks for slow nets and unbalanced usage Swap Rule - Looks for lack of available swap space  RAM Rule - Looks for short page residence times  CPU Power Rule  – Scales on MP systems – Looks for long run queue delays Mutex Rule - Looks for kernel lock contention and high sys CPU time  TCP Rule  – Looks for listen queue problems – Reports on connection attempt failures May 2, 2009 Adrian Cockcroft and Mario Jauvin
  68. 68. XE Toolkit - www.xetoolkit.com Complete re-write of SE Toolkit by Rich Pettit  Extensible Java collector, customize with jar files – Release 1.2 available April 2008 – Multi-platform support Solaris, Linux/x86, Windows, BSD, – OSX, HP-UX, AIX, Linux/s390, Linux/Power Licencing  Free GPL version for standard use and shared derivations – Open source, hosted at http://sourceforge.net/projects/xe-toolkit/ – Commercial support available if needed – Commercial product license for custom in-house derivations – Addresses all the issues people had with SE toolkit !  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  69. 69. Captive Metrics / XE Toolkit Architecture May 2, 2009 Adrian Cockcroft and Mario Jauvin
  70. 70. Free System Monitoring Tools May 2, 2009 Adrian Cockcroft and Mario Jauvin
  71. 71. Collated Performance Data - Orca Problems with time sync when collecting data from multiple tools  – No timestamp at all for vmstat, netstat, df... – No timestamp by default for iostat and ps... – No way to collect realtime stats from an http logfile Use SE Toolkit to generate one timestamped row containing all the data  – First version of percollator.se written by Adrian Cockcroft in 1996 – Extended orcallator.se written by Blair Zajac a few years later – Graphs generated by orca batch job feeding rrdtool based web pages – Active community developing tool at http://www.orcaware.com – Extended to collect much more data, including process workloads – Basic data collection ported to Linux, HP-UX and Windows Orca is basically MRTG for System metrics rather than Network  See http://www.orcaware.com/orca/docs/Orca_Understanding_Performance_Data.ppt  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  72. 72. Orca data collections Collected using “procollator” reading info from /proc on Linux  [Uptime] [Average # Processes in Run Queue (Load Average)] [CPU Usage] [New Process Spawn Rate] [Number of System & Running Processes] [Context Switches & Interrupts Rate] [Interface Input Bits Per Second] [Interface Output Bits Per Second] [Interface Input Packets Per Second] [Interface Output Packets Per Second] [Interface Input Errors Per Second] [Interface Output Errors Per Second] [Interface Input Dropped Per Second] [Interface Output Dropped Per Second] [Interface Output Collisions] [Interface Output Carrier Losses] [TCP Current Connections] [IP Statistics] [TCP Statistics] [ICMP Statistics] [UDP Statistics] [Disk System Wide Reads/Writes Per Second] [Disk System Wide Transfer Rate] [Disk Reads/Writes Per Second] [Disk Transfer Rate] [Disk Space Percent Usage] [Physical Memory Usage] [Swap Usage] [Page Ins & Outs Rate] [Swap Ins & Outs Rate] Orca on Solaris collects many more metrics than shown above  Strength of Orca is lots of detailed metrics with low overhead for collection  Easily customized to add more system metrics or application metrics  Orca can already track HTTP traffic and parse log files  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  73. 73. All metrics are stored in “round robin database” format using RRDtool to generate displays over different time spans Web page is simple collection of plots with drill down by metric or by time Suitable for monitoring a relatively small number of systems in great detail, e.g. backend database servers May 2, 2009 Adrian Cockcroft and Mario Jauvin
  74. 74. Cacti – www.cacti.net Web based user interface based on RRDtool  More sophisticated GUI than Orca or MRTG  Less sophisticated system metric collection,  but more coverage of networking Better management of groups of systems  and devices than Orca, useful for tens to hundreds of nodes Access control and personalization for users  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  75. 75. May 2, 2009 Adrian Cockcroft and Mario Jauvin
  76. 76. May 2, 2009 Adrian Cockcroft and Mario Jauvin
  77. 77. Ganglia – www.ganglia.info Web based RRDtool GUI somewhat similar to Cacti  Better management of clusters of systems and  devices than Cacti, useful for hundreds to thousands of nodes in a hierarchy of clusters Provides many summary statistic plots at cluster  level and collects detailed configuration data XML based data representation  Uses low overhead network protocol  In common use at hundreds of large HPC Grid sites,  less visibly in use at some large commercial sites May 2, 2009 Adrian Cockcroft and Mario Jauvin
  78. 78. May 2, 2009 Adrian Cockcroft and Mario Jauvin
  79. 79. May 2, 2009 Adrian Cockcroft and Mario Jauvin
  80. 80. May 2, 2009 Adrian Cockcroft and Mario Jauvin
  81. 81. BigBrother, BigSister, Hobbit/Xymon Network and system dashboard alert monitor  Widely used at internet sites  Bigbrother is at http://www.bb4.com  Taken over by Quest Inc. – Bigsister is at http://bigsister.graeff.com  Doesn’t seem to be widely used – Hobbit changed its name to Xymon  Xymon is a scalable and efficient open source re-write – of BigBrother, compatible with BB collector scripts Distribution at http://www.xymon.com documentation at – http://en.wikibooks.org/wiki/System_Monitoring_with_Xymon – May 2, 2009 Adrian Cockcroft and Mario Jauvin
  82. 82. May 2, 2009 Adrian Cockcroft and Mario Jauvin
  83. 83. May 2, 2009 Adrian Cockcroft and Mario Jauvin
  84. 84. May 2, 2009 Adrian Cockcroft and Mario Jauvin
  85. 85. Free QA Test and Modelling Tools May 2, 2009 Adrian Cockcroft and Mario Jauvin
  86. 86. QA Test Requirements Generate test workload  SLAMD, Grinder – Collect performance metrics  Any of the tools already mentioned – Report regression against baseline  Predict capacity needed for production system  Use spreadsheets for simple linear prediction – Use modelling tools such as PDQ for queuing models – May 2, 2009 Adrian Cockcroft and Mario Jauvin
  87. 87. Grinder 3 - Powerful New Features 100% Pure Java - works on any hardware platform and any  operating system that supports J2SE 1.3 and above. Java and Jython based load testing framework  Web Browsers: simulate web browsers using HTTP, and HTTPS. – Web Services: test interfaces using SOAP and XML-RPC. – Database: test databases using JDBC. – Middleware: RPC and MOM based systems using IIOP, RMI/IIOP, – RMI/JRMP, and JMS. Other Internet protocols: POP3, SMTP, FTP, and LDAP. – See http://grinder.sourceforge.net/g3/features.html  J2EE Performance Testing with BEA WebLogic Server by  Peter Zadrozny, Philip Aston and Ted Osborne, originally published by Expert Press and now by APress uses Grinder 2 throughout. May 2, 2009 Adrian Cockcroft and Mario Jauvin
  88. 88. SLAMD Load generation framework, written in Java  Originally built to test LDAP servers by Sun  Extended to be very generic and published  as open source. Actively being developed. Sophisticated functions and user interface  See http://www.slamd.com  Latest Release 2.0 has better usability focus  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  89. 89. May 2, 2009 Adrian Cockcroft and Mario Jauvin
  90. 90. May 2, 2009 Adrian Cockcroft and Mario Jauvin
  91. 91. May 2, 2009 Adrian Cockcroft and Mario Jauvin
  92. 92. PDQ Modelling Tool Dr Neil Gunther’s toolkit at  http://www.perfdynamics.com Library used from C or Perl provides MVA queueing  models Use to calibrate in QA and predict in production  PDQ modelling tool details:  The Practical Performance Analyst Dr. Neil Gunther - – McGraw-Hill, 1998 ISBN 0-07-912946-3 Analyzing Computer System Performance with Perl:PDQ – 2004, ISBN 3-54-020865-8 May 2, 2009 Adrian Cockcroft and Mario Jauvin
  93. 93. R – A Free Statistics Package http://www.r-project.org   Runs on any platform  Compatible with S-PLUS  Very powerful/nice scripting language May 2, 2009 Adrian Cockcroft and Mario Jauvin
  94. 94. CHP - “Cockcroft Headroom Plot” R Based plotting tool (demo)   Examples importing iostat, vxstat  Series of articles in CMG Measure-IT  Blogged at http://perfcap.blogspot.com May 2, 2009 Adrian Cockcroft and Mario Jauvin
  95. 95. Web Site Behavior
  96. 96. Web Site Headroom
  97. 97. Java App Server Headroom Plot Mixed Page Weights  Thread-limited appserver  CPU utilization is low  Measurements are of a  single SOA service pool Response is in  milliseconds Throughput is executions/s  Exec Resp Min. : 1.00 Min. : 0.0 1st Qu.: 2.00 1st Qu.: 150.0 Median : 8.00 Median : 361.0 Mean : 64.68 Mean : 533.5 3rd Qu.: 45.00 3rd Qu.: 771.9 Max. :10795.00 Max. :19205.0
  98. 98. Storage Headroom - Reads
  99. 99. Storage Headroom - Reads
  100. 100. References and Conclusion May 2, 2009 Adrian Cockcroft and Mario Jauvin
  101. 101. Licences for Free Tools Open Source Initiative  “OSI Approved licences” – http://opensource.org/licenses/category – Comparisons of Common Licences  http://zooko.com/license_quick_ref.html – May 2, 2009 Adrian Cockcroft and Mario Jauvin
  102. 102. Web Pages and Books Adrian’s Performance and other topics blog  http://perfcap.blogspot.com – MFJ Associates performance tools link page  http://www.mfjassociates.net/perf_links.html – More free tools compiled by John Sellens  http://www.generalconcepts.com/resources/monitoring/ – More tools compiled by Openxtra  http://www.openxtra.co.uk/resource-center/open_source_network_monitor_tools.php – SE toolkit info: Sun Performance and Tuning - Java and the Internet - Adrian  Cockcroft and Richard Pettit - Sun Press/Prentice Hall, 2nd Edition, 1998 ISBN 0-13- 095249-4 Solaris 8 and Linux: System Performance Tuning 2nd Edition – Gian-Paolo Musumeci,  O’Reilly 2002 ISBN: 0-596-00284-X Solaris Internals http://www.solarisinternals.com  Richard McDougall and James Mauro - new 2nd edition and new performance book by – Richard McDougall and Brendan Gregg May 2, 2009 Adrian Cockcroft and Mario Jauvin
  103. 103. Concluding Remarks Many large installations depend on free tools  A full suite of functionality is available  Several tools are needed to cover the bases  Tradeoff between function and ease of use  Support may be available, but typically  Google is the best support tool Functionality is increasing….  May 2, 2009 Adrian Cockcroft and Mario Jauvin
  104. 104. Questions? acockcroft@netflix.com mario@mfjassociates.net Updated presentations http://www.slideshare.net/adrianco May 2, 2009 Adrian Cockcroft and Mario Jauvin

×