• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Leveraging Open Source to Manage SAN Performance
 

Leveraging Open Source to Manage SAN Performance

on

  • 2,624 views

Scope - The primary focus of this presentation is how to leverage open source software to help in managing Shared Storage performance. The storage server will be the focus with particular emphasis ...

Scope - The primary focus of this presentation is how to leverage open source software to help in managing Shared Storage performance. The storage server will be the focus with particular emphasis on ESS. This solution is a small one-off solution.

Statistics

Views

Total Views
2,624
Views on SlideShare
2,613
Embed Views
11

Actions

Likes
1
Downloads
65
Comments
0

5 Embeds 11

http://www.slideshare.net 7
http://www.linkedin.com 1
http://www.slashdocs.com 1
http://www.docshut.com 1
https://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Scope - The primary focus of this presentation is how to leverage open source software to help in managing Shared Storage performance. The storage server will be the focus with particular emphasis on ESS. This solution is a small one-off solution.
  • “ Shared storage” typically refers to the storage shared on a SAN. This includes the s torage Area Network Switches and other Fabric components (ISL’s, routers, etc) We can measure many of the components in the SAN including but not limited to: Server HBA’s, Switch ports, and Storage Server I/O components. Link information includes Tput, packets/sec, errors
  • From the point of view of the server storage is allocated to the physical disks. These disks are accessed via Host Bus Adapters (HBA). Throughput statistics are available on most systems at the HBA level. If a multi-pathing software is implemented, the virtual path typically corresponds to the storage allocation unit on the storage server (LUN). Most servers provide throughput information to the virtual paths. When multi-pathing software is implemented, more than 1 host physical volume will point to the same virtual path. In addition to throughput information, most servers provide end-to-end response time for the physical volumes. This provides the ability to identify if 1 path is performing better than another path (ie – Fabric congestion). All of this information can be measured on most servers using native utilities. There is typically very little visibility into the storage server or network pathing from the server’s view point. In some cases vendor specific server based utilities can provide configuration information that can be used to summarize server based performance data for things like I/Os to a certain storage server component. In any case, these views do not provide visibility outside the server.
  • We can measure many of the components in the SAN including but not limited to: Server HBA’s, Switch ports, and Storage Server I/O components. The switch ports can provide information such as Kbytes/sec Throughput, packets/sec, and errors/sec
  • Kbytes/sec and response time for the Ports on the HA side are only available via the API, and the CLI. They are not available in the TSE, MDM, or TPC-DiskT The TSE has 2 performance tables: VPCRK (Array/DG) and VPCCH (LUN). Higher level components do not have measurement, however, you can roll up the data to higher levels such as ESS, Cluster, Adapter, and Loop. In addition to the raw data, several important fields are included in the array level data: Array Avg RT ms, Avg Disk Utilization, % Sequential, % Read The NVS and Cache hit stats are stored at a LUN level.
  • At this point I have not found a single OSS or Vendor Tool that provides all the necessary infromation to manage the performance of the SAN. There are a number of vendors that seem to have tools that will do this including IBM/Tivoli but either the tools are not compatible with our environment, or they are not ready able to provide all the features we require at this point.
  • Attached server configuration data is important and the commands used are dependent on the OS type and the Storage Server type. There are a number of options for monitoring ESS including TSE, MDM, TPC for Disk, CLI, API -> CIM agent/CIMOM and several 3 rd party products, and they all have their advantages and disadvantages. The TSE/MDM/TPC-Disk all offer similar data stored in DB2 tables. The CLI provides the information in a formatted report that would require significant formatting prior to analysis. The API is a potential option but I simply did not have the time to create my own collector. As a result, I am currently using a number of different tools to manage the performance of the environment. Collectively these tools provide the essentials for managing the environment. Open source is used in some cases but not all as seen in the slide. Requirements Monitor SAN components Utilize existing data sources: TotalStorage Expert for ESS EFCM log files for Fabric – SNMP in future with MRTG Native distributed server utilities for attached servers Monitor frequently enough to be useful Store data for historical purposes and trending Correlate disparate components
  • To collect mrtg you can run it as a daemon or run it from cron (Most of the standard linux distributions have mrtg) What type of Data: Port level – Octets in/out, uptime SNMP can be used to pull other information, wwpn, error counts, firmware level, switch rebooted. PERL::SNMP module OR NET::SNMP binary distribution contains SNMP library and PERL MIB module that ties into binary. The problem with PERL::SNMP it does not load the MIB modules CRICKET is another way to do this; http://cricket.sourceforge.net/
  • mrtgrrd is a CGI script that will query RRD files – takes about a seconds. Included in the set of contributed files. Easy to set up. Set a couple vars and point to the mrtg files to read RRDTool is OS specific so when moving between OS’s you must exported as XML Standard import script for XML formatted data to RRDTool is called: rrdtool dump, and rrdrestore
  • Not generating graphs automatically is a Good Thing – Processing time big time!
  • This slide provides a high level description of the key components required for collecting the data. As a side note, the queries used against the TSE db2 database are very similar to what would be ran against a TPC for Disk or MDM DB2 database. It is assumed that a directory structure is already configured. The script to execute the SQL query does 2 things: Takes a query template and replaces key parameters such as start and end date and time with the correct values, It then creates a small shell script that executes the SQL query The queries provided in the appendices listed above are used to gather array level configuration and performance data
  • Array level data is used because it provides physical measurements for the arrays. Using the configuration information gathered it can also provide a summary of ESS performance at various higher level components. These components include Cluster and Disk Adapter. This report should inlclude a calculated metric for scoring the health of the arrays. The VPCRK table where the array level data is extracted from provides some cluster level information: cache, nvs. These should be save in the Exception table as
  • Exception charts inlcude the array level and cluster level exceptions that were created and imported in the previous step from the array level data. The healthcheck reports should provide a high level summary of the health of ESS at both a server level, a component level (Cluster, Adapter, Array) as well as a summary of all ESS’s for a given customer. For instances of shared ESS’s, the customer does not reflect a single customer’s perspective. In this step it is necessary to define the data necessary to complete the report. The easiest way to do this is to design the report, and then map the data from the report to the data in the DB. During this step or before the next step you should define your SQL queries required for each report and any business logic required
  • Forms should provide a means for the user (you) to select the report Type: Healtcheck (Server Level), Rank Report, as well as the required parameters for the SQL statement that will pull the data required for the specific report: Parms include: Start Date, End Date, Start Time, End Time, ESS, Report Type
  • Forms should provide a means for the user (you) to select the report Type: Healtcheck (Server Level), Rank Report, as well as the required parameters for the SQL statement that will pull the data required for the specific report: Parms include: Start Date, End Date, Start Time, End Time, ESS, Report Type
  • Forms should provide a means for the user (you) to select the report Type: Healtcheck (Server Level), Rank Report, as well as the required parameters for the SQL statement that will pull the data required for the specific report: Parms include: Start Date, End Date, Start Time, End Time, ESS, Report Type
  • Forms should provide a means for the user (you) to select the report Type: Healtcheck (Server Level), Rank Report, as well as the required parameters for the SQL statement that will pull the data required for the specific report: Parms include: Start Date, End Date, Start Time, End Time, ESS, Report Type
  • Generally speaking the I/O response time is the amount of time it takes from the point where the I/O request hits the device driver until the I/O is returned from the device driver
  • For IBM’ers I have a sample script that I can make available. For external customers I would advise you to contact your local IBM AIX field reps to see if they have anything or roll your own script.
  • For IBM’ers I have a sample script that I can make available. For external customers I would advise you to contact your local IBM AIX field reps to see if they have anything or roll your own script.
  • For IBM’ers I have a sample script that I can make available. For external customers I would advise you to contact your local IBM AIX field reps to see if they have anything or roll your own script.

Leveraging Open Source to Manage SAN Performance Leveraging Open Source to Manage SAN Performance Presentation Transcript

  • IBM GLOBAL SERVICES New Orleans, LA P12 Brett Allison Leveraging Open Source to Manage SAN Performance July 25-29, 2005 © IBM Corporation 2005
  • Trademarks & Disclaimer
    • The following terms are trademarks of the IBM Corporation:
    • Enterprise Storage Server® - Abbreviated: ESS
    • TotalStorage® Expert TSE
    • FAStT/DS4000/DS8000
    • AIX®
    • Other trademarks appearing in this report may be considered trademarks of their respective companies.
    • Perl is governed by the GNU General Public License
    • Apache is governed by the GNU General Public License
    • MYSQL is governed by the GNU General Public License
    • PEAR is governed by the GNU General Public License
    • PHP is governed by the GNU General Public License
    • MRTQ is governed by the GNU General Public License
    • RRDTool is governed by the GNU General Public License
    • SANavigator,EFCM McDATA
    • UNIX is a registered trademark in the United States and other countries, licensed exclusively through X/Open Company Limited.
    • Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. EMC is a registered trademark of EMC Inc.
    • HP-UX is a registered trademark of HP Inc.
    • Solaris is a registered trademark of SUN Microsystems, Inc
    • Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.
    • UNIX is a registered trademark of The Open Group in the United States and other countries.
    • Disclaimer
    • The views in this presentation are those of the author and are not necessarily those of IBM
  • Table of Contents
    • Scope of presentation and solution
    • What is Open Source Software (OSS)?
    • What is the problem we wanted to address?
    • What is a SAN?
    • How can we measure and manage a SAN?
    • How can we use OSS to manage SAN performance?
  • What Is Open Source Software?
    • Open Source Software (OSS) is software that is developed, tested and maintained through community participation
    • OSS refers to open standards, shared source code and public collaborative development
    • OSS communities usually enabled via the internet
    • OSS does not mean free-of-charge software, but that the source code is freely available
    • OSS allows redistribution of copies
    • OSS allows modifications and distribution
  • What is a SAN? ISL’s Core Switch - A Core Switch - B Fabric Edge Switch - A Edge Switch - B Links Links Storage Switch - A Storage Switch - B Servers Storage Servers
  • What Can We Measure on the Attached Server? Server View SAN Storage Fabric LUN PATH A PATH B Read Kbytes/sec, Write Kbytes/sec, I/Os per second, Reads/sec, Writes/sec, End-to-End Response Time Physical Volume Read Kbytes/sec, Write Kbytes/sec, I/Os per second, Reads/sec, Writes/sec Virtual Path/LUN Read Kbytes/sec, Write Kbytes/sec, I/Os per second Adapter Metrics Component HBA HBA
  • What SAN Fabric Components Can We Measure? Fabric Edge Switch - A Edge Switch - B ISL’s Core Switch - A Core Switch - B Links Links Storage Switch - A Storage Switch - B What can be measured?
  • What Can We Measure on the Storage Server? KB/sec, RT Physical NVS Delays Cache Hits Logical Volume: Reads, Writes, Sequential I/Os, KB/sec, I/O Time Physical
  • Problem Definition and Constraints
    • Lack of centralized end-to-end view of SAN
    • Insufficient tools and processes to determine ESS and SAN performance issues
    • Inadequate visibility into fabric performance
    • Lack of historical reporting
    • Cost prohibitive enterprise tools
  • What is the Solution? See Appendix H for Requirements sar, iostat, filemon MRTG SNMP TSE, DB2, PERL Collect Post Process PERL, PHP PERL, PHP Extract/Show MYSQL Store RRDTOOL MYSQL Mixed OSS Legend OSS+Glue Server ESS Switch Apache, PHP Browser
  • Server Collection/Post-processing/Store
    • Use UNIX shell scripts to gather data
    • Use PERL or PHP to process
    • Create DB tables: server_performance, server_configuration
    • Use PERL or PHP to import processed data
    • Use Apache/PHP/Browser to view server data
    Appendix A,B Collect Post Process Appendix C Appendix D Store Server Extract/Show Apache, PHP Browser
  • SAN Fabric – Collection with MRTG/SNMP
    • Use MRTG to gather data over SNMP
    • Custom perl scripts could be used, but MRTG is designed to do this
    • Easy to configure
    • Switches are queried every 5 minutes
    • Separate mrtg.cfg’s per fabric recommended
    Switch MRTG SNMP Collect
  • SAN Fabric Data Storage with MRTG/RRDTool
    • Fast!
    • Easy integration with mrtg (set “LogFormat: rrdtool” in config file)
    • Fixed-space – rrd size will never grow past its size at creation
    • Portable
      • Easy to move rrd’s between boxes
    • processing done by rrdtool itself during data insert
    • Various language extensions available
      • php-rrd extension
      • RRDs.pm perl module
    Switch MRTG SNMP Collect RRDtool Store
  • SAN Fabric View with RRDTOOL/PERL/Apache
    • Data presented as graphs
    • With MRTG using rrdtool, graphs are not automatically generated
    • Use a cgi script to generate images “on Demand”
      • mrtg-rrd, 14all.cgi, etc, or a custom CGI
      • Many included with the mrtg package
    • Many of the CGI’s include an image caching mechanism
    Switch MRTG SNMP Collect RRDTOOL Apache, PHP Browser Store Extract/Show
  • Sample Port Throughput Chart
  • Storage Server – Collection
    • Create DB tables: Array_Summary, Exception
    • Configure TSE to collect performance and capacity data:
      • http://www.redbooks.ibm.com/abstracts/sg246102.html?Open
    • Create script to execute query (See Appendix E)
    • Gather ESS data with DB2 queries (See Appendix F & G)
  • Storage Server – Post Process and Import
    • Create array level summary report for each shift of importance, i.e. Prime Shift, 24-hour (See Appendix I).
    Array Configuration
    • Create exception report
    • Import the summary report, and exception above into MYSQL (See Appendix D) for an example (The SQL statement will need to be modified to fit the summary report file)
    Array Performance 2 B 3 1 72.8 10000 1024 12288 800 023-12345 23.59 0 20050121 DISK GROUP LOOP ADAPTER CLUSTER SIZE GB RPM NVS PER CLUSTER CACHE PER CLUSTER MODEL ESS SN END TIME START TIME START DATE 7 0.23 308 14919.43 43.05 94.69 25.37 1076.54 21.59 98 6.21 1046 34.41 rank10 MAX NVS FULL AVG NVS FULL CACHE HOLD MIN CACHE HOLD AVG SEQ PCT READ PCT AVG WRITEKBYTESRATE AVG READKBYTESRATE AVG ARRAY RT MS DISK UTIL MAX DISK UTIL AVG MAX IO Rate AVG IO RATE ARRAY
  • Storage Server – View – Define Reports/Charts MYSQL Capacity Health Check Server Component Customer Exceptions Business Logic, SQL Queries
  • Storage Server – View – Define Forms 1) Select ESS Reports, then “continue” 2) Click to select the ESS, or hold the ctrl key to select multiples
  • Storage Server – View ESS Array Summary Report
  • Storage Server – Chart Array Exceptions Based on the exception table in the previous slide we can drill down by clicking on the exception and chart the exceptions
  • Storage Server – ESS Health Check Customer View
    • The pie chart shows the distribution of ranks across the ESS base by score category as a percentage of total ranks.
    • The tables shows the count of each ESS's ranks per rank score as a percentage of total ranks.
  • Appendix A - Measure End-to-End Host Disk I/O Response Time The iostat package for Linux is only valid with a 2.4 & 2.6 kernel See Appendix B for links to more information Avg. Disk sec/Read Physical Disk perfmon NT/Wintel svctm (ms) iostat –d 2 5 *iostat Linux iostat –xcn 2 5 sar –d filemon -o /tmp/filemon.log -O all Command/Object iostat sar filemon Native Tool svc_t (ms) Solaris avserv (ms) HP-UX read time (ms) write time (ms) AIX Metric(s) OS
  • Appendix B: Getting LUN Serial Numbers for ESS Devices Note : ESS Utilities for AIX/HP-UX/Solaris are available at: http://www-1.ibm.com/servers/storage/support/disk/2105/downloading.html Host config. - http://www.redbooks.ibm.com/abstracts/tips0553.html Device Name LUN SN lsvpcfg SDD Linux SDD ESS Util Tool Device Name Serial Datapath query device Wintel VG, hostname, Connection, hdisk LUN SN lsvp –a AIX, HP-UX, Solaris Other Metrics Key Command OS
  • Appendix C: Format ‘ lsvp –a ’ and ‘ filemon ’ (Logic)
    • Process ‘lsvp –a’ file
      • Build hdisk hash with key = hdisk and value = LUN SN
      • Build ess hash with key = hdisk and value = last 5 chars of LUN SN
    • Process ‘filemon ’ file
      • Create hashes for each of the following values with hdisk as the key: Date, Start time, Physical Volume, Reads, Avg Read Time, Avg Read Size, Writes, Avg Write Time, Avg Write Size
    • Print data to file with headers and commas to separate fields
      • Iterate through hdisk hash and use the common hdisk key to index into the other hashes and printing out those that have values
  • Appendix D: Import Data into MYSQL - Logic
    • Consolidate files into import directory (FTP/SFTP/RCOPY)
    • Create array of files to import
    • Loop through each file
    • If file is readable open file for reading ($filehandle = fopen($file,’rb’))
    • Loop through each $line in $file and create SQL statement (Example):
      • $sql = "INSERT INTO server ( DATE, TIME, SERVER_NAME, LUN, ESS_SN, HDISK, READS, READ_TIME, READ_SIZE, WRITES, WRITE_TIME, WRITE_SIZE”
    • Run the insert:
      • $db->query($sql,$line);
  • Appendix E: DB2 Query Wrapper - Logic
    • Build static configuration file containing remote DB2 aliases (TSEs)
    • Build query template with keyword substitution. For example
      • %ESSID, %STARTTIME,%ENDTIME,%STARTDATE, etc.
    • Execute query wrapper with query template and configuration file as parameters (default values for %STARTTIME, %STARTDATE, etc)
    • Loop through static configuration file and do the following for each DB:
      • Replace keywords with current variables (This is in the $runfile)
      • Create a shell script to execute the query file (This is the $kshfile)
    ### PERL Code snippet #### This is not a fully functional script, it is just an example open (KSHFILE, "> $kshfile") || &msg("die","Could not write $kshfile! $!"); # Open shell script print KSHFILE "db2 connect to $remote user $db2user using $db2pass "; # Print db2 connect info to shell print KSHFILE "db2 -tf $runfile "; # Print command to run query file print KSHFILE "db2 connect reset "; # Reset connection close(KSHFILE); # Close the shell script system("chmod +x $kshfile"); # Modify shell script to have execute perms exec("$kshfile"); # Execute shell script
  • Appendix F: DB2 Query for Array Performance Data Note : This information is relevant only if you have the TotalStorage Expert installed and access to the DB2 command line on the TSE server. SELECT DISTINCT A.*, B.M_CARD_NUM, B.M_LOOP_ID, B.M_GRP_NUM FROM DB2ADMIN.VPCRK A, DB2ADMIN.VPCFG B WHERE ( ( A.PC_DATE_B >= '%STARTDATE' AND A.PC_DATE_E <= '%ENDDATE' AND A.PC_TIME_B >= '%STARTTIME' AND A.PC_TIME_E <= '%ENDTIME' AND A.M_MACH_SN = '%ESSID' AND A.M_MACH_SN = B.M_MACH_SN AND A.M_ARRAY_ID = B.M_ARRAY_ID AND A.P_TASK = B.P_TASK ) ) ORDER BY A.M_ARRAY_ID, A.PC_DATE_B, A.PC_DATE_E with ur;
  • Appendix G: DB2 Query for Array Configuration Data Note : This information is relevant only if you have the TotalStorage Expert installed and access to the DB2 command line on the TSE server. SELECT DISTINCT A.M_MACH_SN, A.M_MODEL_N, A.M_CLUSTER_N, A.M_RAM, A.M_NVS, C.I_DDM_RPM, C.I_DDM_GB_CAPACITY FROM DB2ADMIN.VPVPD A, DB2ADMIN.VMPDX B, DB2ADMIN.VcMDDM C WHERE ( ( A.M_MACH_SN = B.I_VSM_SN AND B.I_VSM_IDX = C.I_VSM_IDX ) ) ORDER BY A.M_MACH_SN, A.M_CLUSTER_N;
  • Appendix H: Requirements - URLs
    • Software Requirements
      • DB2 SDK 7.2.6 (Default TSE DB2 is 7.2 and is not fully compatible with newer versions!)
      • PERL v 5.6 or later http://www.perl.com/download.csp
      • Apache v1.3.33: http://httpd.apache.org/
      • PHP 4.0 or later (5.0 has full OO support): http://www.php.net/downloads.php
      • MYSQL 4.1.9 or later: http://dev.mysql.com/downloads/
      • PEAR 1.3.4: http://www.go-pear.org /
        • Installed packages:
        • ===================
        • Package Version State
        • Archive_Tar 1.1 stable
        • Console_Getopt 1.2 stable
        • DB 1.6.8 stable
        • HTML_Common 1.2.1 stable
        • HTML_QuickForm 3.2.4pl1 stable
        • HTML_Table 1.5 stable
        • PEAR 1.3.4 stable
        • XML_RPC 1.1.0 stable
      • JpGraph 2.0alpha2(22 Jan 2005): http://www.aditus.nu/jpgraph/
      • MRTG/RRDTool 2.12..1 http://people.ee.ethz.ch/~oetiker/webtools/mrtg/
    • Hardware Requirements – Server with connectivity to all devices
  • Appendix I: Array Summary Report Logic
    • Read in configuration data (Appendix G)
      • Create hashes with ESS as key and values for Model, Cache, RPM, DDM Size, NVS
    • Sort array performance data file (Appendix F) by ESS, Array
    • Read in Array level metrics into arrays with ESS, Array, Time Stamp as keys filtering out any records with times that are outside of the start and end time specified by user
    • Create exception report based on pre-defined thresholds
    • Loop through each array metric and calculate minimums, maximums, averages, etc.
    • Print out file to CSV (Use ESS as key to configuration data)
  • Appendix J: Useful Links Slide
    • AIX documentation: http://www-1.ibm.com/servers/aix/library/index.html
    • Linux – iostat: http://www.linuxinsight.com/
    • HP-UX documentation: http:// docs.hp.com /
    • Solaris documentation: http:// docs.sun.com /app/docs
    • SNIA standards
    • The Fibre Channel HBA API Project
    • Perl-SNMP
    • RRDtool
    • ESS documentation
      • ESS Model 800 Performance
      • IBM TotalStorage Expert Reporting: How to Produce Built-In and Customized Reports
      • IBM TotalStorage Expert Hands-On Usage Guide
  • Biography Brett Allison has been doing distributed systems performance related work since 1997 including J2EE application analysis, UNIX/NT, and Storage technologies. His current role is Performance and Capacity Management team lead ITDS. He has developed tools, processes, and service offerings to support storage performance and capacity. He has spoken at a number of conferences and is the author of several White Papers on performance