Elements of SAN Capacity Planning

1,236 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,236
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
54
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • 04/12/10
  • Elements of SAN Capacity Planning

    1. 1. Elements of SAN capacity planning Mark Friedman VP, Storage Technology [email_address] (941) 261-8945
    2. 2. DataCore Software Corporation <ul><li>Founded 1998 - Storage networking Software </li></ul><ul><li>170+ employees, private - Over $45M raised </li></ul><ul><ul><li>Top Venture firms - NEA, OneLiberty </li></ul></ul><ul><ul><li>Funds – VanWagoner, Bank of America, etc </li></ul></ul><ul><ul><li>Intel Business and Technical collaboration agreement </li></ul></ul><ul><li>Exec. Team </li></ul><ul><ul><li>Proven Storage expertise </li></ul></ul><ul><ul><li>Proven Software company experience </li></ul></ul><ul><ul><li>Operating systems, high-availability, Caching, networking </li></ul></ul><ul><ul><li>Enterprise level support and training </li></ul></ul><ul><li>Worldwide: Ft. Lauderdale HQ, Silicon Valley, Canada France, Germany, U.K., Japan </li></ul>
    3. 3. Overview <ul><li>How do we take what we know about storage processor performance and apply it to emerging SAN technology? </li></ul><ul><li>What is a SAN ? </li></ul><ul><li>Planning for SANs: </li></ul><ul><ul><li>SAN performance characteristics </li></ul></ul><ul><ul><li>Backup and replication performance </li></ul></ul>
    4. 4. Evolution Of Disk Storage Subsystems See: Dr. Alexandre Brandwajn, “ A study of cached RAID 5 I/O” CMG Proceedings , 1994. Write-thru Cached subsystems Cached Disk Strings & Farms Spindles Storage Processors
    5. 5. What Is A SAN? <ul><li>Storage Area Networks are designed to exploit Fibre Channel plumbing </li></ul><ul><li>Approaches to simplified networked storage: </li></ul><ul><ul><li>SAN appliances </li></ul></ul><ul><ul><li>SAN Metadata Controllers (“out of band”) </li></ul></ul><ul><ul><li>SAN storage managers (“in band”) </li></ul></ul>
    6. 6. The Difference Between NAS and SAN <ul><li>Storage Area Network (SAN) designed to exploit Fibre Channel plumbing require a new infrastructure. </li></ul><ul><li>Network Attached Storage (NAS) devices plug into the existing networking infrastructure. </li></ul><ul><ul><li>Networked file access protocols (NFS, SMB, CIFS) </li></ul></ul><ul><ul><li>TCP/IP stack </li></ul></ul>Media Access: Ethernet, FDDI Application: HTTP, RPC Host-to-Host: TCP, UDP Internet Protocol: IP Packet Packet Packet Packet
    7. 7. The Difference Between NAS and SAN <ul><li>NAS devices plug into existing TCP/IP networking support. </li></ul><ul><li>Performance considerations: </li></ul><ul><ul><li>1500 byte Ethernet MTU </li></ul></ul><ul><ul><li>TCP requires acknowledgement of each packet, limiting performance. </li></ul></ul>Application Interfaces RPC DCOM Winsock NetBIOS Named Pipes NetBT Redirector Server <ul><ul><li>User Mode </li></ul></ul><ul><ul><li>Kernel </li></ul></ul>TCP UDP IP ARP ICMP IGMP IP Filtering IP Forwarding Packet Scheduler NDIS Wrapper NDIS Miniport NIC Device Driver TDI NDIS
    8. 8. The Difference Between NAS and SAN <ul><li>Performance considerations: </li></ul><ul><ul><li>e.g., </li></ul></ul><ul><ul><li>1.5 KB Ethernet MTU </li></ul></ul><ul><ul><ul><li>Requires processing 80,000 Host interrupts/sec @ 1 Gb/sec </li></ul></ul></ul><ul><ul><ul><li>or Jumbo frames, which also requires installing a new infrastructure </li></ul></ul></ul><ul><ul><li>Which is why Fibre Channel was designed the way it is! </li></ul></ul>Source: Alteon Computers, 1999.
    9. 9. <ul><li>Universal data sharing is developing ad hoc on top of de facto industry standards designed for network access. </li></ul><ul><ul><li>Sun NFS </li></ul></ul><ul><ul><li>HTTP, FTP </li></ul></ul><ul><ul><li>Microsoft CIFS (and DFS) </li></ul></ul><ul><ul><ul><li>also known as SMB </li></ul></ul></ul><ul><ul><ul><li>CIFS-compatible is the the largest and fastest growing category of data </li></ul></ul></ul>Competing Network File System Protocols
    10. 10. CIFS Data Flow <ul><li>Session-oriented: e.g., call backs </li></ul>SMB Request SMB Request Server Network Interface System Cache File Server Redirector MS Word System Cache Network Interface Client
    11. 11. What About Performance? NFSD Daemon TCP/IP Driver Remote Procedure Call (RPC) Response Data TCP/IP Network NFS Client NFS Server Media Access: Ethernet, FDDI Application: HTTP, RPC Host-to-Host: TCP, UDP Internet Protocol: IP User Process Client Process TCP/IP Driver
    12. 12. What About Performance? <ul><li>Network-attached yields fraction of the performance of direct-attached drives when block size does not match frame size. </li></ul><ul><li>See ftp://ftp.research.microsoft.com/pub/tr/tr-2000-55.pdf </li></ul>SMB Request SMB Request Server Network Interface System Cache File Server Redirector MS Word System Cache Network Interface Client Media Access: Ethernet, FDDI Application: HTTP, RPC Host-to-Host: TCP, UDP Internet Protocol: IP
    13. 13. <ul><li>Add a network delay component to interconnect two Central Server models and iterate. </li></ul>What about modeling?
    14. 14. The Holy Grail! <ul><li>Storage Area Networks </li></ul><ul><li>Uses low latency, high performance Fibre Channel switching technology (plumbing) </li></ul><ul><li>100 MB/sec Full duplex serial protocol over copper or fiber </li></ul><ul><li>Extended distance using fiber </li></ul><ul><li>Three topologies: </li></ul><ul><ul><li>Point-to-Point </li></ul></ul><ul><ul><li>Arbitrated Loop : 127 addresses, but can be bridged </li></ul></ul><ul><ul><li>Fabric : 16 MB addresses </li></ul></ul>
    15. 15. The Holy Grail! <ul><li>Storage Area Networks </li></ul><ul><li>FC delivers SCSI commands, but Fibre Channel exploitation requires new infrastructure and driver support </li></ul><ul><li>Objectives: </li></ul><ul><ul><li>Extended addressing of shared storage pools </li></ul></ul><ul><ul><li>Dynamic, hot-plugable interfaces </li></ul></ul><ul><ul><li>Redundancy, replication & failover </li></ul></ul><ul><ul><li>Security administration </li></ul></ul><ul><ul><li>Storage resource virtualization </li></ul></ul>
    16. 16. Distributed Storage & Centralized Administration <ul><li>Traditional tethered vs untethered SAN storage </li></ul><ul><li>Untethered storage can (hopefully) be pooled for centralized administration </li></ul><ul><li>Disk space pooling (virtualization) </li></ul><ul><ul><li>Currently, using LUN virtualization </li></ul></ul><ul><ul><li>In the future, implementing dynamic virtual:real address mapping (e.g., the IBM Storage Tank) </li></ul></ul><ul><li>Centralized back-up </li></ul><ul><ul><li>SAN LAN-free backup </li></ul></ul>
    17. 17. <ul><li>FC is packet-oriented (designed for routing). </li></ul><ul><li>FC pushes many networking functions into the hardware layer. </li></ul><ul><ul><li>e.g., </li></ul></ul><ul><ul><li>Packet fragmentation </li></ul></ul><ul><ul><li>Routing </li></ul></ul>Storage Area Networks Upper Level Protocol SCSI IPI-3 HIPPI IP Fc4 Framing Protocol/Flow Control Fc2 8B/10B Encode/Decode Fc1 100MB/s Physical Layer Fc0 Common Services Fc3
    18. 18. <ul><li>FC is designed to work with optical fiber and lasers consistent with Gigabit Ethernet hardware </li></ul><ul><ul><li>100 MB/sec interfaces </li></ul></ul><ul><ul><li>200 MB/sec interfaces </li></ul></ul><ul><li>This creates a new class of hardware that you must budget for: FC hubs and switches . </li></ul>Storage Area Networks
    19. 19. <ul><li>Performance characteristics of FC switches: </li></ul><ul><ul><li>Extremely low latency (  1  sec), except when cascaded switches require frame routing </li></ul></ul><ul><ul><li>Deliver dedicated 100 MB/sec point-to-point virtual circuit bandwidth </li></ul></ul><ul><ul><li>Measured 80 MB/sec effective data transfer rates per 100 MB/sec Port </li></ul></ul>Storage Area Networks
    20. 20. <ul><li>When will IP and SCSI co-exist on the same network fabric? </li></ul><ul><ul><li>iSCSI </li></ul></ul><ul><ul><li>Nishan </li></ul></ul><ul><ul><li>Others? </li></ul></ul>Storage Area Networks Upper Level Protocol SCSI IPI-3 HIPPI IP Fc4 Framing Protocol/Flow Control Fc2 8B/10B Encode/Decode Fc1 100MB/s Physical Layer Fc0 Common Services Fc3
    21. 21. Storage Area Networks <ul><li>FC zoning is used to control access to resources (security) </li></ul><ul><li>Two approaches to SAN management: </li></ul><ul><ul><li>Management functions must migrate to the switch, storage processor, or…. </li></ul></ul><ul><ul><li>OS must be extended to support FC topologies. </li></ul></ul>
    22. 22. Approaches to building SANs <ul><li>Fibre Channel-based Storage Area Networks (SANs) </li></ul><ul><ul><li>SAN appliances </li></ul></ul><ul><ul><li>SAN Metadata Controllers </li></ul></ul><ul><ul><li>SAN Storage Managers </li></ul></ul><ul><li>Architecture (and performance) considerations </li></ul>
    23. 23. Approaches to building SANs <ul><li>Where does the logical device:physical device mapping run? </li></ul><ul><ul><li>Out-of-band : on the client </li></ul></ul><ul><ul><li>In-band : inside the SAN appliance, transparent to the client </li></ul></ul><ul><li>Many industry analysts have focused on this relatively unimportant distinction. </li></ul>
    24. 24. SAN appliances <ul><li>Conventional storage processors with </li></ul><ul><li>Fibre Channel interfaces </li></ul><ul><li>Fibre Channel support </li></ul><ul><ul><li>FC Fabric </li></ul></ul><ul><ul><li>Zoning </li></ul></ul><ul><ul><li>LUN virtualization </li></ul></ul>
    25. 25. SAN Appliance Performance <ul><li>Same as before, except faster Fibre Channel interfaces </li></ul><ul><ul><li>Commodity processors, internal buses, disks, front-end and back-end interfaces </li></ul></ul><ul><ul><li>Proprietary storage processor architecture considerations </li></ul></ul>Internal Bus Multiple Processors Cache Memory FC Interfaces FC Disks Host Interfaces
    26. 26. SAN appliances <ul><li>SAN and NAS convergence? </li></ul><ul><ul><li>Adding Fibre Channel interfaces and Fibre Channel support to a NAS box </li></ul></ul><ul><ul><li>SAN-NAS hybrids when SAN appliances are connected via TCP/IP. </li></ul></ul><ul><li>Current Issues: </li></ul><ul><ul><li>Managing multiple boxes </li></ul></ul><ul><ul><li>Proprietary management platforms </li></ul></ul>
    27. 27. SAN Metadata Controller <ul><li>SAN clients acquire an access token from the Metadata Controller (out-of-band) </li></ul><ul><li>SAN clients then access disks directly using proprietary distributed file system </li></ul>SAN Clients Pooled Storage Resources SAN Metadata Controller 1 3 Token Fibre Channel 2
    28. 28. SAN Metadata Controller <ul><li>Performance considerations: </li></ul><ul><ul><li>MDC latency (low access rate assumed) </li></ul></ul><ul><ul><li>Additional latency to map client file system request to the distributed file system </li></ul></ul><ul><li>Other administrative considerations: </li></ul><ul><ul><li>Requirement for client-side software is a burden! </li></ul></ul>
    29. 29. SAN Storage Manager <ul><li>Requires all access to pooled disks through the SAN Storage Manager </li></ul><ul><li>(in-band)! </li></ul>SAN Clients Pooled Storage Resources Fibre Channel Storage Domain Servers
    30. 30. SAN Storage Manager <ul><li>SAN Storage Manager adds latency to every I/O request </li></ul><ul><li>How much latency is involved? </li></ul><ul><li>Can this latency be reduced using traditional disk caching strategies? </li></ul>SAN Clients Pooled Storage Resources Fibre Channel Storage Domain Servers
    31. 31. Architecture of a Storage Domain Server <ul><li>Runs on an ordinary Win2K Intel server </li></ul><ul><li>The SDS intercepts SAN I/O requests, impersonating a SCSI disk </li></ul><ul><li>Leverages: </li></ul><ul><ul><li>Native Device drivers </li></ul></ul><ul><ul><li>Disk management </li></ul></ul><ul><ul><li>Security </li></ul></ul><ul><ul><li>Native CIFS support </li></ul></ul>Fibre Channel HBA Driver SCSI miniport Driver Disk Driver Diskperf (measurement) Fault Tolerance (Optional) Data Cache Fault Tolerance Initiator/Target Emulation FC Adaptor Polling Threads Security Natives W2K I/O Manager SANsymphony Storage Domain Server Client I/O
    32. 32. Sizing the SAN Storage Manager server <ul><li>In-band latency is a function of Intel server front-end bandwidth: </li></ul><ul><ul><li>Processor speed </li></ul></ul><ul><ul><li>Number of processors </li></ul></ul><ul><ul><li>PCI bus bandwidth </li></ul></ul><ul><ul><li>Number of HBAs </li></ul></ul><ul><li>and performance of the back-end Disk configuration </li></ul>
    33. 33. SAN Storage Manager <ul><li>Can SAN Storage Manager in-band latency be reduced using traditional disk caching strategies? </li></ul><ul><ul><li>Read hits </li></ul></ul><ul><ul><li>Read misses </li></ul></ul><ul><ul><ul><li>Disk I/O + (2 * data transfer) </li></ul></ul></ul><ul><ul><li>Fast Writes to cache (with mirrored caches) </li></ul></ul><ul><ul><ul><li>2 * data transfer </li></ul></ul></ul><ul><ul><ul><li>Write performance ultimately determined by the disk configuration </li></ul></ul></ul>
    34. 34. SAN Storage Manager <ul><li>Read hits (16 KB block): </li></ul><ul><li>Timings from an FC hardware monitor </li></ul><ul><li>1Gbit/s Interfaces </li></ul><ul><li>No bus arbitration delays! </li></ul>140  sec 27  sec Status Frame SCSI Read Command Length = 4000 16x1024 Byte Data Frames
    35. 35. Read vs. Write hits (16 KB block) Fibre Channel Latency (16KB Blocks) SCSI Command Write Setup Data Frames SCSI Status
    36. 36. <ul><li>How is time being spent inside the server? </li></ul><ul><li>PCI bus? </li></ul><ul><li>Host Bus adaptor? </li></ul><ul><li>Device polling? </li></ul><ul><li>Software stack? </li></ul>Decomposing SAN in-band Latency SCSI Command Write Setup Data Frames SCSI Status
    37. 37. <ul><li>4-way 550 MHz PC </li></ul><ul><ul><li>Maximum of three </li></ul></ul><ul><ul><li>FC interface polling threads </li></ul></ul><ul><li>3 PCI buses </li></ul><ul><li>(528MB/s Total) </li></ul><ul><li>1, 4, or 8 </li></ul><ul><li>QLogic 2200 HBAs </li></ul>Benchmark Configuration Memory Bus 64bit/33MHz PCI 32bit/33MHz PCI 32bit/33MHz PCI 4x550MHz XEON Processors
    38. 38. <ul><li>How is time being spent inside the SDS? </li></ul><ul><li>PCI bus? </li></ul><ul><li>Host Bus adaptor? </li></ul><ul><li>Device polling: </li></ul><ul><ul><li>1 CPU is capable of 375,000 unproductive polls/sec </li></ul></ul><ul><ul><li> 2.66  secs per poll </li></ul></ul><ul><li>Software stack: </li></ul><ul><ul><li>3 CPUs are capable of fielding 40,000 Read I/Os per second from cache </li></ul></ul><ul><ul><li> 73  secs per 512-byte I/O </li></ul></ul>Decomposing SAN in-band Latency
    39. 39. Decomposing SAN in-band Latency SANsymphony in-band Latency (16KB Blocks) SDS FC Interface Data Transfer
    40. 40. Impact Of New Technologies <ul><li>Front-end bandwidth: </li></ul><ul><ul><li>Different speed Processors </li></ul></ul><ul><ul><li>Different number of processors </li></ul></ul><ul><ul><li>Faster PCI Bus </li></ul></ul><ul><ul><li>Faster HBAs </li></ul></ul><ul><li>e.g. Next Generation Server </li></ul><ul><ul><li>2GHz GHz Processors (4x Benchmark System) </li></ul></ul><ul><ul><li>200MB/sec FC interfaces (2x Benchmark System) </li></ul></ul><ul><ul><li>4x800MB/s PCI bus (6x Benchmark System) </li></ul></ul><ul><li>... </li></ul>
    41. 41. Impact Of New Technologies 2GHz CPU & New HBAs 2GHz CPU, New HBAs, 2Gbit Switching Today
    42. 42. Sizing the SAN Storage Manager <ul><li>Scalability </li></ul><ul><ul><li>Processor speed </li></ul></ul><ul><ul><li>Number of processors </li></ul></ul><ul><ul><li>PCI bus bandwidth </li></ul></ul><ul><ul><ul><li>32bit/33MHz 132MB/sec </li></ul></ul></ul><ul><ul><ul><li>64bit/33MHz 267MB/sec </li></ul></ul></ul><ul><ul><ul><li>64bit/66MHz 528MB/sec </li></ul></ul></ul><ul><ul><ul><li>64bit/100MHz 800MB/s (PCI-X) </li></ul></ul></ul><ul><ul><li>Infiniband technology??? </li></ul></ul><ul><ul><li>Number of HBAs </li></ul></ul><ul><ul><ul><li>200 MB/sec FC interfaces feature faster internal processors </li></ul></ul></ul>
    43. 43. Sizing the SAN Storage Manager <ul><li>Entry level system: </li></ul><ul><ul><li>Dual Processor, single PCI bus, 1 GB RAM </li></ul></ul><ul><li>Mid-level departmental system: </li></ul><ul><ul><li>Dual Processor, dual PCI bus, 2 GB RAM </li></ul></ul><ul><li>Enterprise-class system: </li></ul><ul><ul><li>Quad Processor, triple PCI bus, 4 GB RAM </li></ul></ul>
    44. 44. SAN Storage Manager PC scalability
    45. 45. SAN Storage Manager PC scalability Departmental SAN Enterprise class Entry level
    46. 46. SANsymphony Performance <ul><li>Conclusions </li></ul><ul><ul><li>FC switches provide virtually unlimited bandwidth with exceptionally low latency so long as you do not cascade switches </li></ul></ul><ul><ul><li>General purpose Intel PCs are a great source of inexpensive MIPS. </li></ul></ul><ul><ul><li>In-band SAN management is not a CPU-bound process. </li></ul></ul><ul><ul><li>PCI bandwidth is the most significant bottleneck in the Intel architecture. </li></ul></ul><ul><ul><li>FC Interface cards speeds and feeds are also very significant </li></ul></ul>
    47. 47. SAN Storage Manager – Next Steps <ul><li>Cacheability of Unix and NT workloads </li></ul><ul><ul><li>Domino, MS Exchange </li></ul></ul><ul><ul><li>Oracle, SQL Server, Apache, IIS </li></ul></ul><ul><li>Given mirrored writes, what is the effect of different physical disk configurations? </li></ul><ul><ul><li>JBOD </li></ul></ul><ul><ul><li>RAID 0 disk striping </li></ul></ul><ul><ul><li>RAID 5 write penalty </li></ul></ul><ul><li>Asynchronous disk mirroring over long distances </li></ul><ul><li>Backup and Replication (snapshot) </li></ul>
    48. 48. <ul><li>? </li></ul>Questions
    49. 49. www.datacore.com

    ×