Storage for Virtual Environments 2011 R2

Uploaded on

The Storage for Virtual Environments seminar focuses on the challenges of backup and recovery in a virtual infrastructure, the various solutions that users are now using to solve those challenges, and …

The Storage for Virtual Environments seminar focuses on the challenges of backup and recovery in a virtual infrastructure, the various solutions that users are now using to solve those challenges, and a roadmap for making the most of all an organization’s virtualization initiatives.

This slide deck was used by Stephen Foskett for his

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide
  • Mirror Mode paper:
  • Up to 256 FC or iSCSI LUNsESX multipathingLoad balancingFailoverFailover between FC and iSCSI*Beware of block sizes greater than 256 KB!If you want virtual disks greater than 256 GB, you must use a VMFS block size larger than 1 MBAlign your virtual disk starting offset to your array (by booting the VM and using diskpart, Windows PE, or UNIX fdisk)*
  • Link Aggregate Control Protocol (LACP) for trunking/EtherChannel - Use “fixed” path policy, not LRUUp to 8 (or 32) NFS mount pointsTurn off access time updatesThin provisioning? Turn on AutoSize and watch out
  • FC storage the recommended latency threshold is  20 – 30 MSFor SAS storage the recommended latency threshold is  20 – 30 MSFor SATA storage the recommended latency threshold is 30 – 50 MSFor SSD storage the recommended latency threshold is 15 – 20 MS
  • FC storage the recommended latency threshold is  20 – 30 MSFor SAS storage the recommended latency threshold is  20 – 30 MSFor SATA storage the recommended latency threshold is 30 – 50 MSFor SSD storage the recommended latency threshold is 15 – 20 MS


  • 1. Storage for Virtual Environments
    Stephen Foskett
    Foskett Services and Gestalt IT
    Live Footnotes:
    • @Sfoskett
    • 2. #VirtualStorage
  • This is Not a Rah-Rah Session
  • 3. Agenda
  • 4. Introducing the Virtual Data Center
  • 5. This Hour’s Focus:What Virtualization Does
    Introducing storage and server virtualization
    The future of virtualization
    The virtual datacenter
    Virtualization confounds storage
    Three pillars of performance
    Other issues
    Storage features for virtualization
    What’s new in VMware
  • 6. Virtualization of Storage, Serverand Network
    Storage has been stuck in the Stone Age since the Stone Age!
    Fake disks, fake file systems, fixed allocation
    Little integration and no communication
    Virtualization is a bridge to the future
    Maintains functionality for existing apps
    Improves flexibility and efficiency
  • 7. A Look at the Future
  • 8. Server Virtualization is On the Rise
    Data: InformationWeek Analytics 2010 Virtualization Management Survey of 316 business technology professionals, August 2010
  • 9. Server Virtualization is a Pile of Lies!
    What the OS thinks it’s running on…
    What the OS is actually running on…
    Physical Hardware
    Binary Translation, Paravirtualization, Hardware Assist
    Guest OS
    Guest OS
    Scheduler and Memory Allocator
    NIC Driver
    I/O Driver
  • 10. And It Gets Worse Outside the Server!
  • 11. The Virtual Data Center of Tomorrow
    The Cloud™
  • 12. The Real Future of IT Infrastructure
    Orchestration Software
  • 13. Three Pillars of VM Performance
  • 14. Confounding Storage Presentation
    Storage virtualization is nothing new…
    RAID and NAS virtualized disks
    Caching arrays and SANs masked volumes
    New tricks: Thin provisioning, automated tiering, array virtualization
    But, we wrongly assume this is where it ends
    Volume managers and file systems
    Now we have hypervisors virtualizing storage
    VMFS/VMDK = storage array?
    Virtual storage appliances (VSAs)
  • 15. Begging for Converged I/O
    4G FC Storage
    1 GbE Network
    1 GbE Cluster
    How many I/O ports and cables does a server need?
    Typical server has 4 ports, 2 used
    Application servers have 4-8 ports used!
    Do FC and InfiniBand make sense with 10/40/100 GbE?
    When does commoditization hit I/O?
    Ethernet momentum is unbeatable
    Blades and hypervisors demand greater I/O integration and flexibility
    Other side of the coin – need to virtualize I/O
  • 16. Driving Storage Virtualization
    Server virtualization demands storage features
    Data protection with snapshots and replication
    Allocation efficiency with thin provisioning+
    Performance and cost tweaking with automated sub-LUN tiering
    Improved locking and resource sharing
    Flexibility is the big one
    Must be able to create, use, modify and destroy storage on demand
    Must move storage logically and physically
    Must allow OS to move too
  • 17. “The I/O Blender” Demands New Architectures
    Shared storage is challenging to implement
    Storage arrays “guess” what’s coming next based on allocation (LUN) taking advantage of sequential performance
    Server virtualization throws I/O into a blender – All I/O is now random I/O!
  • 18. Server Virtualization Requires SAN and NAS
    Server virtualization has transformed the data center and storage requirements
    VMware is the #1 driver of SAN adoption today!
    60% of virtual server storage is on SAN or NAS
    86% have implemented some server virtualization
    Server virtualization has enabled and demanded centralization and sharing of storage on arrays like never before!
    Source: ESG, 2008
  • 19. Keys to the Future For Storage Folks
    Ye Olde Seminar Content!
  • 20. Primary Production Virtualization Platform
    Data: InformationWeek Analytics 2010 Virtualization Management Survey of 316 business technology professionals, August 2010
  • 21. Storage Features for Virtualization
  • 22. Which Features Are People Using?
    Source: 2010 virtualization survey of 125 readers
  • 23. What’s New in vSphere 4 and 4.1
    VMware vSphere 4 (AKA ESX/ESXi 4) is a major upgrade for storage
    Lots of new features like thin provisioning, PSA, any-to-any Storage VMotion, PVSCSI
    Massive performance upgrade (400k IOPS!)
    vSphere 4.1 is equally huge for storage
    Boot from SAN
    vStorage APIs for Array Integration (VAAI)
    Storage I/O control (SIOC)
  • 24. What’s New in vSphere 5
    VMFS-5 – Scalability and efficiency improvements
    Storage DRS – Datastore clusters and improved load balancing
    Storage I/O Control – Cluster-wide and NFS support
    Profile-Driven Storage – Provisioning, compliance and monitoring
    FCoE Software Initiator
    iSCSI Initiator GUI
    Storage APIs – Storage Awareness (VASA)
    Storage APIs – Array Integration (VAAI 2) – Thin Stun, NFS, T10
    Storage vMotion - Enhanced with mirror mode
    vSphere Storage Appliance (VSA)
    vSphere Replication – New in SRM
  • 25. And Then, There’s VDI…
    Virtual desktop infrastructure (VDI) takes everything we just worried about and amplifies it:
    Massive I/O crunches
    Huge duplication of data
    More wasted capacity
    More user visibility
    More backup trouble
  • 26. What’s next
    Vendor Showcase and Networking Break
  • 27. Technical Considerations - Configuring Storage for VMs
    The mechanics of presenting and using storage in virtualized environments
  • 28. This Hour’s Focus:Hypervisor Storage Features
    Storage vMotion
    Storage presentation: Shared, raw, NFS, etc.
    Thin provisioning
    Multipathing (VMware Pluggable Storage Architecture)
    VAAI and VASA
    Storage I/O control and storage DRS
  • 29. Storage vMotion
    Introduced in ESX 3 as “Upgrade vMotion”
    ESX 3.5 used a snapshot while the datastore was in motion
    vSphere 4 used changed-block tracking (CBT) and recursive passes
    vSphere 5 Mirror Mode mirrors writes to in-progress vMotions and also supports migration of vSphere snapshots and Linked Clones
    Can be offloaded for VAAI-Block (but not NFS)
  • 30. vSphere 5: What’s New in VMFS 5
    Max VMDK size is still 2 TB – 512 bytes
    Virtual (non-passthru) RDM still limited to 2 TB
    Max LUNs per host is still 256
  • 31. Hypervisor Storage Options:Shared Storage
    The common/ workstation approach
    VMware: VMDK image in VMFS datastore
    Hyper-V: VHD image in CSV datastore
    Block storage (direct or FC/iSCSI SAN)
    Traditional, familiar, common (~90%)
    Prime features (Storage VMotion, etc)
    Multipathing, load balancing, failover*
    Overhead of two storage stacks (5-8%)
    Harder to leverage storage features
    Often shares storage LUN and queue
    Difficult storage management
    DAS or SAN
  • 32. Hypervisor Storage Options:Shared Storage on NAS
    Skip VMFS and use NAS
    NFS or SMB is the datastore
    Simple – no SAN
    Multiple queues
    Flexible (on-the-fly changes)
    Simple snap and replicate*
    Enables full Vmotion
    Link aggregation (trunking) is possible
    Less familiar (ESX 3.0+)
    CPU load questions
    Limited to 8 NFS datastores (ESX default)
    Snapshot consistency for multiple VMDK
  • 33. Hypervisor Storage Options:Guest iSCSI
    Skip VMFS and use iSCSI directly
    Access a LUN just like any physical server
    VMware ESX can even boot from iSCSI!
    Storage folks love it!
    Can be faster than ESX iSCSI
    Very flexible (on-the-fly changes)
    Guest can move and still access storage
    Less common to VM folks
    CPU load questions
    No Storage VMotion (but doesn’t need it)
  • 34. Hypervisor Storage Options:Raw Device Mapping (RDM)
    Guest VM’s access storage directly over iSCSI or FC
    VM’s can even boot from raw devices
    Hyper-V pass-through LUN is similar
    Per-server queues for performance
    Easier measurement
    The only method for clustering
    Supports LUNs larger than 2 TB (60 TB passthru in vSphere 5!)
    Tricky VMotion and dynamic resource scheduling (DRS)
    No storage VMotion
    More management overhead
    Limited to 256 LUNs per data center
    Mapping File
    SAN Storage
  • 35. Hypervisor Storage Options:Direct I/O
    VMware ESX VMDirectPath - Guest VM’s access I/O hardware directly
    Leverages AMD IOMMU or Intel VT-d
    Potential for native performance
    Just like RDM but better!
    No VMotion or Storage VMotion
    No ESX fault tolerance (FT)
    No ESX snapshots or VM suspend
    No device hot-add
    No performance benefit in the real world!
    Mapping File
    SAN Storage
  • 36. Which VMware Storage Method Performs Best?
    Mixed random I/O
    CPU cost per I/O
    RDM (p), or RDM (v)
    Source: “Performance Characterization of VMFS and RDM Using a SAN”, VMware Inc.,ESX 3.5, 2008
  • 37. vSphere 5: Policy or Profile-Driven Storage
    Allows storage tiers to be defined in vCenter based on SLA, performance, etc.
    Used during provisioning, cloning, Storage vMotion, Storage DRS
    Leverages VASA for metrics and characterization
    All HCL arrays and types (NFS, iSCSI, FC)
    Custom descriptions and tagging for tiers
    Compliance status is a simple binary report
  • 38. Native VMware Thin Provisioning
    VMware ESX 4 allocates storage in 1 MB chunks as capacity is used
    Similar support enabled for virtual disks on NFS in VI 3
    Thin provisioning existed for block, could be enabled on the command line in VI 3
    Present in VMware desktop products
    vSphere 4 fully supports and integrates thin provisioning
    Every version/license includes thin provisioning
    Allows thick-to-thin conversion during Storage VMotion
    In-array thin provisioning also supported (we’ll get to that…)
  • 39. Four Types of VMware ESX Volumes
    Note: FT is not supported
    What will your array do? VAAI helps…
    Friendly to on-array thin provisioning
  • 40. Storage Allocation and Thin Provisioning
    VMware tests show no performance impact from thin provisioning after zeroing
  • 41. Pluggable Storage Architecture:Native Multipathing
    VMware ESX includes multipathing built in
    Basic native multipathing (NMP) is round-robin fail-over only – it will not load balance I/O across multiple paths or make more intelligent decisions about which paths to use
    Pluggable Storage Architecture (PSA)
    VMware NMP
    Third-Party MPP
    VMware SATP
    Third-Party SATP
    VMware PSP
    Third-Party PSP
  • 42. Pluggable Storage Architecture: PSP and SATP
    vSphere 4 Pluggable Storage Architecture allows third-party developers to replace ESX’s storage I/O stack
    ESX Enterprise+ Only
    There are two classes of third-party plug-ins:
    Path-selection plug-ins (PSPs) optimize the choice of which path to use, ideal for active/passive type arrays
    Storage array type plug-ins (SATPs) allow load balancing across multiple paths in addition to path selection for active/active arrays
    EMC PowerPath/VE for vSphere does everything
  • 43. Storage Array Type Plug-ins (SATP)
    ESX native approaches
    Pseudo Active
    Storage Array Type Plug-Ins
    VMW_SATP_LOCAL – Generic local direct-attached storage
    VMW_SATP_DEFAULT_AA – Generic for active/active arrays
    VMW_SATP_DEFAULT_AP – Generic for active/passive arrays
    VMW_SATP_LSI – LSI/NetApp arrays from Dell, HDS, IBM, Oracle, SGI
    VMW_SATP_SVC – IBM SVC-based systems (SVC, V7000, Actifio)
    VMW_SATP_ALUA – Asymmetric Logical Unit Access-compliant arrays
    VMW_SATP_CX – EMC/Dell CLARiiON and Celerra (also VMW_SATP_ALUA_CX)
    VMW_SATP_SYMM – EMC Symmetrix DMX-3/DMX-4/VMAX, Invista
    VMW_SATP_INV – EMC Invista and VPLEX
    VMW_SATP_EQL – Dell EqualLogic systems
    Also, EMC PowerPath and HDS HDLM and vendor-unique plugins not detailed in the HCL
  • 44. Path Selection Plug-ins (PSP)
    VMW_PSP_MRU – Most-Recently Used (MRU) – Supports hundreds of storage arrays
    VMW_PSP_FIXED – Fixed - Supports hundreds of storage arrays
    VMW_PSP_RR – Round-Robin - Supports dozens of storage arrays
    DELL_PSP_EQL_ROUTED – Dell EqualLogic iSCSI arrays
    Also, EMC PowerPath and other vendor unique
  • 45. vStorage APIs for Array Integration (VAAI)
    VAAI integrates advanced storage features with VMware
    Basic requirements:
    A capable storage array
    ESX 4.1+
    A software plug-in for ESX
    Not every implementation is equal
    Block zeroing can be very demanding for some arrays
    Zeroing might conflict with full copy
  • 46. VAAI Support Matrix
  • 47. vSphere 5: VAAI 2
    T10 compliance is improved - No plug-in needed for many arrays
    NAS plugins come from vendors, not VMware
  • 48. vSphere 5: vSphereStorage APIs – Storage Awareness (VASA)
    VASA is communication mechanism for vCenter to detect array capabilities
    RAID level, thin provisioning state, replication state, etc.
    Two locations in vCenter Server:
    “System-Defined Capabilities” – per-datastore descriptors
    Storage views and SMS API’s
  • 49. Storage I/O Control (SIOC)
    Storage I/O Control (SIOC) is all about fairness:
    Prioritization and QoS for VMFS
    Re-distributes unused I/O resources
    Minimizes “noisy neighbor” issues
    ESX can provide quality of service for storage access to virtual machines
    Enabled per-datastore
    When a pre-defined latency level is exceeded on a VM it begins to throttle I/O (default 30 ms)
    Monitors queues on storage arrays and per-VM I/O latency
    vSphere 4.1 with Enterprise Plus
    Disabled by default but highly recommended!
    Block storage only (FC or ISCSI)
    Whole-LUN only (no extents)
    No RDM
  • 50. Storage I/O Control in Action
  • 51. Virtual Machine Mobility
    Moving virtual machines is the next big challenge
    Physical servers are difficult to move around and between data centers
    Pent-up desire to move virtual machines from host to host and even to different physical locations
    VMware DRS would move live VMs around the data center
    The “Holy Grail” for server managers
    Requires networked storage (SAN/NAS)
  • 52. vSphere 5: Storage DRS
    Datastore clusters aggregate multiple datastores
    VMs and VMDKs placement metrics:
    Space - Capacity utilization and availability (80% default)
    Performance – I/O latency (15 ms default)
    When thresholds are crossed, vSphere will rebalance all VMs and VMDKs according to Affinity Rules
    Storage DRS works with either VMFS/block or NFS datastores
    Maintenance Mode evacuates a datastore
  • 53. What’s next
  • 54. Expanding the Conversation
    Converged I/O, storage virtualization and new storage architectures
  • 55. This Hour’s Focus:Non-Hypervisor Storage Features
    Converged networking
    Storage protocols (FC, iSCSI, NFS)
    Enhanced Ethernet (DCB, CAN, FCoE)
    I/O virtualization
    Storage for virtual storage
    Tiered storage and SSD/flash
    Specialized arrays
    Virtual storage appliances (VSA)
  • 56. Introduction: Converging on Convergence
    Data centers rely more on standard ingredients
    What will connect these systems together?
    IP and Ethernet are logical choices
  • 57. Drivers of Convergence
  • 58. Which Storage Protocol to Use?
    Server admins don’t know/care about storage protocols and will want whatever they are familiar with
    Storage admins have preconceived notions about the merits of various options:
    FC is fast, low-latency, low-CPU, expensive
    NFS is slow, high-latency, high-CPU, cheap
    iSCSI is medium, medium, medium, medium
  • 59. vSphere Protocol Performance
  • 60. vSphere CPU Utilization
  • 61. vSphere Latency
  • 62. Microsoft Hyper-V Performance
  • 63. Which Storage Protocols Do People Use?
    Source: 2010 virtualization survey of 125 readers
  • 64. The Upshot: It Doesn’t Matter
    Use what you have and are familiar with!
    FC, iSCSI, NFS all work well
    Most enterprise production VM data is on FC, many smaller shops using iSCSI or NFS
    Either/or? - 50% use a combination
    For IP storage
    Network hardware and config matter more than protocol (NFS, iSCSI, FC)
    Use a separate network or VLAN
    Use a fast switch and consider jumbo frames
    For FC storage
    8 Gb FC/FCoE is awesome for VMs
    Look into NPIV
    Look for VAAI
  • 65. The Storage Network Roadmap
  • 66. Serious Performance
    10 GbE is faster than most storage interconnects
    iSCSI and FCoE both can perform at wire-rate
  • 67. Latency is Critical Too
    Latency is even more critical in shared storage
    FCoE with 10 GbE can achieve well over 500,000 4K IOPS (if the array and client can handle it!)
  • 68. Benefits Beyond Speed
    10 GbE takes performance off the table (for now…)
    But performance is only half the story:
    Simplified connectivity
    New network architecture
    Virtual machine mobility
    1 GbE Cluster
    4G FC Storage
    1 GbE Network
    10 GbE
    (Plus 6 Gbps extra capacity)
  • 69. Enhanced 10 Gb Ethernet
    • Ethernet and SCSI were not made for each other
    • 70. SCSI expects a lossless transport with guaranteed delivery
    • 71. Ethernet expects higher-level protocols to take care of issues
    • 72. “Data Center Bridging” is a project to create lossless Ethernet
    • 73. AKA Data Center Ethernet (DCE), Converged Enhanced Ethernet (CEE)
    • 74. iSCSI and NFS are happy with or without DCB
    • 75. DCB is a work in progress
    • 76. FCoE requires PFC (Qbb or PAUSE), DCBX (Qaz)
    • 77. QCN (Qau) is still not ready
    Priority Flow Control (PFC)
    Congestion Management (QCN)
    Bandwidth Management (ETS)
    Data Center Bridging Exchange Protocol (DCBX)
    Traffic Classes 802.1p/Q
  • 78. FCoE CNAs for VMware ESX
    No Intel (OpenFCoE) or Broadcom support in vSphere 4…
  • 79. vSphere 5: FCoE Software Initiator
    Dramatically expands the FCoE footprint from just a few CNAs
    Based on Intel OpenFCoE? – Shows as “Intel Corporation FCoE Adapter”
  • 80. I/O Virtualization: Virtual I/O
    Extends I/O capabilities beyond physical connections (PCIe slots, etc)
    Increases flexibility and mobility of VMs and blades
    Reduces hardware, cabling, and cost for high-I/O machines
    Increases density of blades and VMs
  • 81. I/O Virtualization: IOMMU (Intel VT-d)
    IOMMU gives devices direct access to system memory
    AMD IOMMU or Intel VT-d
    Similar to AGP GART
    VMware VMDirectPath leverages IOMMU
    Allows VMs to access devices directly
    May not improve real-world performance
    System Memory
    I/O Device
  • 82. Does SSD Change the Equation?
    RAM and flash promise high performance…
    But you have to use it right
  • 83. Flash is Not A Disk
    Flash must be carefully engineered and integrated
    Cache and intelligence to offset write penalty
    Automatic block-level data placement to maximize ROI
    IF a system can do this, everything else improves
    Overall system performance
    Utilization of disk capacity
    Space and power efficiency
    Even system cost can improve!
  • 84. The Tiered Storage Cliché
    Cost and Performance
    Optimized for Savings!
  • 85. Tiered Storage Evolves
  • 86. Three Approaches to SSD For VM
    EMC Project Lightning promises to deliver all three!
  • 87. Storage for Virtual Servers (Only!)
    New breed of storage solutions just for virtual servers
    Highly integrated (vCenter, VMkernel drivers, etc.)
    High-performance (SSD cache)
    Mostly from startups (for now)
    Tintri– NFS-based caching array
    Virsto+EvoStor – Hyper-V software, moving to VMware
  • 88. Virtual Storage Appliances (VSA)
    What if the SAN was pulled inside the hypervisor?
    VSA = A virtual storage array as a guest VM
    Great for lab or PoC
    Some are not for production
    Can build a whole data center in a hypervisor, including LAN, SAN, clusters, etc
    Physical Server Resources
    VM Guest
    VM Guest
    Virtual Storage Appliance
    Virtual SAN
    Virtual LAN
  • 89. vSphere 5: vSphere Storage Appliance (VSA)
    Aimed at SMB market
    Two deployment options:
    2x replicates storage 4:2
    3x replicates round-robin 6:3
    Uses local (DAS) storage
    Enables HA and vMotion with no SAN or NAS
    Uses NFS for storage access
    Also manages IP addresses for HA
  • 90. Virtual Storage Appliance Options
  • 91. Whew! Let’s Sum Up
    Server virtualization changes everything
    Throw your old assumptions about storage workloads and presentation out the window
    We (storage folks) have some work to do
    New ways of presenting storage to the server
    Converged I/O (Ethernet!)
    New demand for storage virtualization features
    New architectural assumptions
  • 92. Thank You!
    Stephen Foskett