Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Storage Consumption and Chargeback

2,946 views

Published on

The information in this paper is based on my experience ‘fixing’ the storage consumption process for a large customer ( 3 PB SAN environment) in a managed services (outsourced) environment. The names have been changed to protect the innocent but the processes, tools, and issues are very real.
Regardless of whether you are the customer (end-consumer) of storage or the manager/provider of storage, the accuracy of storage consumption reports is essential to maintaining IT’s reputation and customer trust.
In this paper we discuss the challenges of creating a sustainable, reliable, and robust charge back processes and tools for SAN storage consumption reporting.

  • Be the first to comment

Storage Consumption and Chargeback

  1. 1. Developing a SAN Storage Consumption Solution CMG 2009
  2. 2. Trademarks & Disclaimer <ul><li>The following terms are trademarks of the IBM Corporation: </li></ul><ul><li>Enterprise Storage Server® - Abbreviated: ESS </li></ul><ul><li>TotalStorage® Expert TSE </li></ul><ul><li>FAStT/DS4000/DS8000 </li></ul><ul><li>AIX® </li></ul><ul><li>IBM SAN Volume Controller </li></ul><ul><li>IBM Flash Copy is a registered trademark of IBM </li></ul><ul><li>TPC is a registered trademark of IBM </li></ul><ul><li>Other trademarks appearing in this report may be considered trademarks of their respective companies. </li></ul><ul><li>UNIX is a registered trademark in the United States and other countries, licensed exclusively through X/Open Company Limited. </li></ul><ul><li>Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. </li></ul><ul><li>EMC is a registered trademark of EMC Inc. </li></ul><ul><li>BCV is a registered trademark of EMC Inc. </li></ul><ul><li>StorageScope is a registered trademark of EMC Inc. </li></ul><ul><li>CLARiiON is a registered trademark of EMC. </li></ul><ul><li>HP-UX is a registered trademark of HP Inc. </li></ul><ul><li>Solaris is a registered trademark of SUN Microsystems, Inc </li></ul><ul><li>Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. </li></ul><ul><li>UNIX is a registered trademark of The Open Group in the United States and other countries. </li></ul><ul><li>NetApp is a registered trademark of Network Appliance Inc. </li></ul><ul><li>NetApp Operations Manager is a registered trademark of Network Appliance Inc. </li></ul><ul><li>Disclaimer </li></ul><ul><li>The views in this presentation are those of the author and are not necessarily those of IBM </li></ul>
  3. 3. Abstract <ul><li>The information in this paper is based on my experience ‘fixing’ the storage consumption process for a large a managed infrastructure services environment. The names have been changed to protect the innocent but the processes, tools, and data issues are very real. The accuracy of storage consumption reporting used for chargeback is essential in maintaining IT’s reputation and customer trust. </li></ul><ul><li>This presentation discusses the challenges, processes and tools involved when establishing a robust SAN storage consumption process. </li></ul>
  4. 4. Agenda <ul><li>Storage consumption basics </li></ul><ul><li>Requirements and objectives </li></ul><ul><li>The environment </li></ul><ul><li>The processes </li></ul><ul><li>The tools </li></ul><ul><li>Lessons learned </li></ul>
  5. 5. Storage Consumption Basics: What are we measuring?
  6. 6. Solution Objectives <ul><li>Process: </li></ul><ul><ul><li>Fix orphan creating processes </li></ul></ul><ul><ul><li>Develop process for identifying and remediating data issues </li></ul></ul><ul><ul><li>Develop operational processes for automating problem identification </li></ul></ul><ul><li>Tools </li></ul><ul><ul><li>Automate process of storage consumption reporting </li></ul></ul><ul><ul><li>Develop re-usable tools for data collection and post-processing </li></ul></ul><ul><ul><li>Provide minimal manual processing of data from sources </li></ul></ul><ul><ul><li>Improve coverage of storage platforms through use of new tools and data sources </li></ul></ul><ul><li>Data </li></ul><ul><ul><li>Report 99% of storage allocated </li></ul></ul><ul><ul><li>Identify secondary sources of data for reconciliation </li></ul></ul><ul><ul><li>Remediate data issues </li></ul></ul><ul><li>Business </li></ul><ul><ul><li>Provide justification of variances based on change tickets </li></ul></ul>
  7. 7. Storage Consumption Challenges
  8. 8. Environment
  9. 9. Environment Size – Data Complexity 54 Number of Array Migrations 15 NAS Filers 17 Number of Input data Sources 6 Number of SAN Fabric Locations 3 ECC/StorageScope Servers 900 Server Count Other Logistics      19 IBM 4 HP 36 EMC Array Count
  10. 10. Environment - Data Inputs Manually created Array management tools Flash copy relationships Automatically generated report Problem and Change Management Tools Change Tickets Manually created Application table Application Correlation Manually created Backup reporting team Backup Media Servers Manually created Capacity report Array Location Automatically generated report Asset Management Database Exclusion Host Names Automatically generated report Asset Management Database Host Status, LOB Manually created HP-Compaq CLI HP-Compaq Host-LUN Report template - Manually ran EMC StorageScope 6.0 Host_Consumption_Query Automatically generated via queries Configuration database Symdev list - fibre - nofibre Automatically gathered by shell scripts EMC CLI Symbcv output Automatically generated via query Configuration database DS8K Configuration Automatically generated via query Configuration database Clariion Configuration Automatically generated via query Configuration database EMC Configuration Automatically generated via query Billing database Host Allocation Report template - Manually ran EMC StorageScope 6.0 EMC NAS Report template – Manually ran Ops Mgr NAS Base Composite Description Input Source Input Description
  11. 11. Process Section
  12. 12. Storage Consumption/Chargeback High Level Process Flow
  13. 13. Processes that Impact Storage Consumption
  14. 14. Migration Process Validate SAN removal
  15. 15. Server Decommission Process
  16. 16. Host Rename/Host Repurpose
  17. 17. BCV Removal
  18. 18. Server/HBA Re-use
  19. 19. Storage Change Process
  20. 20. SRM Operational Processes
  21. 21. Exception Reporting and Remediation Process
  22. 22. Sample Exception Report no no yes yes yes 770.00 Production Pluto server19 yes no yes yes yes 385.00 Decommissioned Mars server18 yes no yes yes yes 385.00 Decommissioned Pluto server17 no yes yes yes yes 1,120.00 Decommissioned Mars server16 no yes yes yes yes 864.00 Decommissioned Pluto server15 no no yes yes yes 851.35 Production Mars server14 no no yes yes yes 851.35 Production Pluto server13 no no yes yes yes 1,121.00 Production Mars server12 no no yes yes no 1,121.00   Pluto server11 no no yes yes no 1,280.00   Mars server10 IBM & EMC? In TPC? In Billing DB? IBM Supported In Asset DB Usable GB Status Location Server Name
  23. 23. Tools
  24. 24. Tool/Reporting Requirements Tool Reporting Legend
  25. 25. Change Management Required Fields <ul><li>Server Name </li></ul><ul><li>Purpose of Request </li></ul><ul><li>Location or datacenter </li></ul><ul><li>Physical Server name to SAN </li></ul><ul><li>Tier </li></ul><ul><li>Amount of usable storage being requested </li></ul><ul><li>New number of SAN ports </li></ul>
  26. 26. Historical Consumption Analyst Process Logical Flow
  27. 27. Improved Consumption Analyst Process Logical Flow
  28. 28. Sample Storage Consumption Summary Report 59521 256.00 292.57 EMC T2 Uranus Server15 19652 288.00 336.00 EMC T2 Uranus Server14 27053 385.00 443.05 EMC T2 Uranus Server13 48131 4,097.00 4,712.76 EMC T2 Uranus Server12 19851 256.00 295.62 IBM T2 Uranus Server11 16917 6,154.00 7,100.43 IBM T2 Uranus Server10 34741 1,025.00 1,189.71 IBM T1 Uranus Server9 31390 1,120.00 1,303.62 EMC T2 Pluto Server8 74167 832.00 955.43 EMC T1 Pluto Server7 27178 864.00 992.00 EMC T1 Pluto Server6 15571 106.00 121.14 EMC T1 Pluto Server5 39607 851.35 1,442.60 EMC T1 Pluto Server4 80509 1,121.00 1,281.17 IBM T1 Pluto Server3 44874 514.00 588.19 IBM T2 Venus Server2 25892 944.00 1,083.62 IBM T2 Mars Server1 57955 376.00 431.81 IBM T1 Mars Server1 Change Ticket Usable Total (GB) Raw Total (GB) Type Tier Location Server/Cluster Name
  29. 29. Sample Detailed Consumption Report 33.72 38.53 BCV 3007 EMC   T2       33.72 38.53 BCV 3006 EMC   T2       33.72 38.53 BCV 3005 EMC   T2       33.72 38.53 BCV 3004 EMC   T2       33.72 38.53 BCV 3003 EMC   T2       33.72 38.53 BCV 3002 EMC   T2       33.72 38.53 BCV 3001 EMC   T2       33.72 67.43 STD 1006 EMC T1       33.72 67.43 STD 1005 EMC T1       33.72 67.43 STD 1004 EMC T1       33.72 67.43 STD 1003 EMC T1       33.72 67.43 STD 1002 EMC T1       33.72 67.43 STD 1001 EMC T1       33.72 67.43 STD 1000 EMC T1 1234567 Pluto Server3_Server4 Sum of Usable (GB) Sum of Raw (GB) LUN Type LUN Type Tier Array SN Array Location Server/Cluster Name
  30. 30. Sample Invoice Continued - Gigabyte Disk Storage GBs WebSphere Barad-dur 5/14/2011 88305 Rohan Riders Gigabyte Disk Storage GBs Oracle Barad-dur 5/14/2011 88305 Rohan Riders Gigabyte Disk Storage GBs WebSphere Barad-dur 5/14/2011 88305 Rohan Riders Gigabyte Disk Storage GBs Oracle Orthanc 5/14/2011 88305 Rohan Riders Gigabyte Disk Storage GBs WebSphere Orthanc 5/14/2011 88305 Rohan Riders Gigabyte Disk Storage GBs Oracle Orthanc 5/14/2011 88305 Rohan Riders Unit of Measure Rate Element Application ID Tower Invoice Date Invoice Number Supplier Name $1,938.61 $1.94 Shelob 1,000.00 Mordor EMC server52 $1,938.61 $1.94 Shelob 1,000.00 Mordor EMC server51 $1,938.61 $1.94 Shelob 1,000.00 Mordor EMC server50 $969.31 $1.94 Palantir 500.00 Isengard IBM server49 $969.31 $1.94 Urak-hai 500.00 Isengard IBM server46 $969.31 $1.94 Urak-hai 500.00 Isengard IBM server45 Amount Rate Per GB LOB Useable Quantity Location Equipment Server Name
  31. 31. Lessons Learned Get Sponsored True-Up Candidate Project Management Create Baseline Avoid Surprises Persevere
  32. 33. Definitions <ul><li>Storage consumption – How much space is associated with each server </li></ul><ul><li>SAN – Storage area network </li></ul><ul><li>Tier – Classification of storage based on hardware attributes such as disk size, speed, capacity, controller type or RAID configuration. </li></ul><ul><li>HBA – Host Bus Adapter provides SAN connectivity from a host to the backend storage device. </li></ul><ul><li>WWPN – World Wide Port Name is a unique identifier tied to the HBA hardware. It is used to manage the connectivity access from the host to the backend storage targets. </li></ul><ul><li>LUN – SAN volumes assigned to hosts </li></ul><ul><li>BCV – Business Continuance Volumes are an EMC specific technology used for creating copies of production volumes. </li></ul>
  33. 34. Appendix A – What is a SAN? Fabric Edge Switch - A Edge Switch - B Core Switch - A Core Switch - B Links Links Storage Switch - A Storage Switch - B Servers Storage Servers

×