Your SlideShare is downloading. ×
Next-Generation Best Practices for VMware and Storage
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Next-Generation Best Practices for VMware and Storage

7,979
views

Published on

This is the opening keynote presentation, focusing on VMware and storage best practices, from the Midwest Regional VMUG in Kansas City on December 6, 2010.

This is the opening keynote presentation, focusing on VMware and storage best practices, from the Midwest Regional VMUG in Kansas City on December 6, 2010.

Published in: Technology

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
7,979
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
335
Comments
0
Likes
3
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Midwest Regional VMUG
    Next Generation Best Practices for Storage and VMware
    Scott Lowe, VCDX #39
    vSpecialist, EMC Corporation
    Author, Mastering VMware vSphere 4
    http://blog.scottlowe.org
    http://twitter.com/scott_lowe
  • 2. The “Great” Protocol Debate
    Every protocol can Be Highly Available, and generally, every protocol can meet a broad performance band
    Each protocol has different configuration considerations
    Each Protocol has a VMware “super-power”, and also a “kryponite”
    In vSphere, there is core feature equality across protocols
    Conclusion: there is no debate – pick what works for you!
    The best flexibility comes from a combination of VMFS and NFS
  • 3. First - Key Things To Know – “A” thru “F”
    Key Best Practices circa 2010/2011
  • 4. Leverage Key Docs
    Key Best Practices circa 2010/2011
    A
  • 5. Key Docs, and Storage Array Taxonomy
    Key VMware Docs:
    Fibre Channel SAN Configuration Guide
    iSCSI SAN Configuration Guide
    Storage/SAN Compatibility Guide
    …Understand VMware Storage Taxonomy:
    Active/Active (LUN ownership)
    Active/Passive (LUN ownership)
    Virtual Port (iSCSI only)
  • 6. Key Docs, and Storage Array Taxonomy
    Key Storage Partner Docs:
    Each Array is very different. Storage varies far more vendor to vendor than servers do
    Find, read, and stay current on your array’s Best Practices Doc – most are excellent.
    Even if you’re NOT the storage team, read them – it will help you.
    http://www.emc.com/collateral/hardware/solution-overview/h2529-vmware-esx-svr-w-symmetrix-wp-ldv.pdf
    http://www.emc.com/collateral/hardware/technical-documentation/h5536-vmware-esx-srvr-using-celerra-stor-sys-wp.pdf
    http://www.emc.com/collateral/software/solution-overview/h2197-vmware-esx-clariion-stor-syst-ldv.pdf
  • 7. Setup Multipathing Right
    Key Best Practices circa 2010/2011
    B
  • 8. Understanding the vSphere Pluggable Storage Architecture
  • 9. What’s “out of the box” in vSphere 4.1?
    [root@esxi ~]# vmware -vVMware ESX 4.1.0 build-260247 [root@esxi ~]# esxcli nmp satp listName                 Default PSP       DescriptionVMW_SATP_SYMM        VMW_PSP_FIXED     Placeholder (plugin not loaded)VMW_SATP_SVC         VMW_PSP_FIXED     Placeholder (plugin not loaded)VMW_SATP_MSA         VMW_PSP_MRU       Placeholder (plugin not loaded)VMW_SATP_LSI         VMW_PSP_MRU       Placeholder (plugin not loaded)VMW_SATP_INV         VMW_PSP_FIXED     Placeholder (plugin not loaded)VMW_SATP_EVA         VMW_PSP_FIXED     Placeholder (plugin not loaded)VMW_SATP_EQL         VMW_PSP_FIXED     Placeholder (plugin not loaded)VMW_SATP_DEFAULT_AP  VMW_PSP_MRU       Placeholder (plugin not loaded)VMW_SATP_ALUA_CX     VMW_PSP_FIXED_AP  Placeholder (plugin not loaded)VMW_SATP_CX          VMW_PSP_MRU       Supports EMC CX that do not use the ALUA protocolVMW_SATP_ALUA        VMW_PSP_RR        Supports non-specific arrays that use the ALUA protocolVMW_SATP_DEFAULT_AA  VMW_PSP_FIXED     Supports non-specific active/active arraysVMW_SATP_LOCAL       VMW_PSP_FIXED     Supports direct attached devices
  • 10. What’s “out of the box” in vSphere?
    PSPs:
    • Fixed (Default for Active-Active LUN ownership models)
    • 11. All IO goes down preferred path, reverts to preferred path after original path restore
    • 12. MRU (Default for Active-Passive LUN ownership models)
    • 13. All IO goes down active path, stays after original path restore
    • 14. Round Robin
    • 15. n IO operations goes down active path then rotate (default is 1000)
    HOWTO – setting PSP for a specific device (can override default selected by SATP detected ARRAYID):
    esxcli nmp device setpolicy --device <device UID> --psp VMW_PSP_RR
    (check with your vendor first!)
  • 16. Changing Round Robin IOOperationLimit
    esxcli nmp roundrobin setconfig --device <device UID> –iops
    check with your storage vendor first! This setting can cause problems on arrays. Has been validated ok, but not necessary in most cases
  • 17. Effect of different RR IOOperationLimit settings
    NOTE:
    This is with a SINGLE LUN.
    This is the case where the larger IOOperationLimit default is the worst
    In a real-world environment – lots of LUNs and VMs results in decent overall loadbalancing
    Recommendation – if you can, stick with the default
  • 18. What is Asymmetric Logical Unit (ALUA)?
    Many storage arrays have Active/Passive LUN ownership
    All paths show in the vSphere Client as:
    Active (can be used for I/O)
    I/O is accepted on all ports
    All I/O for a LUN is serviced on its owning storage processor
    • In reality some paths are preferred over others
    • 19. Enter ALUA to solve this issue
    Supported introduced in vSphere 4.0
    SP A
    SP B
    LUN
  • 20. What is Asymmetric Logical Unit (ALUA)?
    • ALUA Allows for paths to be profiled
    Active (can be used for I/O)
    Active (non-optimized – not normally used for I/O)
    Standby
    Dead
    Ensures optimal path selection/usage by vSphere PSP and 3rd Party MPPs
    Supports Fixed, MRU, & RR PSP
    Supports EMC PowerPath/VE
    ALUA is not supported in ESX 3.5
    SP A
    SP B
    LUN
  • 21. Understanding MPIO
    MPIO is based on “initiator-target” sessions – not “links”
  • 22. MPIO Exceptions – Windows Clusters
    Among a long list of “not supported” things:
    NO Clustering on NFS datastores
    No Clustering on iSCSI, FCoE (unless using PP/VE)
    No round-robin with native multipathing (unless using PP/VE)
    NO Mixed environments, such as configurations where one cluster node is running a different version of ESX/ESXi than another cluster node.
    NO Use of MSCS in conjunction with VMware Fault Tolerance.
    NO Migration with vMotion of clustered virtual machines.
    NO N-Port ID Virtualization (NPIV)
    You must use hardware version 7 with ESX/ESXi 4.1
  • 23. APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    APP
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    OS
    Shared
    Storage
    STORAGE
    PowerPath – a Multipathing Plugin (MPP)
    Simple Storage manageability
    Simple Provisioning = “Pool of Connectivity”
    Predictable and consistent
    Optimize server, storage, and data-path utilization
    Performance and Scale
    Tune infrastructure performance, LUN/Path Prioritization
    Predictive Array Specific Load Balancing Algorithms
    Automatic HBA, Path, and storage processor fault recovery
    Other 3rd party MPPs:
    Dell/Equalogic PSP
    Uses a “least deep queue” algorithm rather than basic round robin
    Can redirect IO to different peer storage nodes
    See this at the Dell/EqualLogic booth
    PowerPath
    PowerPath
    PowerPath
    PowerPath
  • 24. NFS
    Considerations
  • 25. General NFS Best Practices
    • Start with Vendor Best Practices:
    EMC Celerra H5536 & NetApp TR-3749
    While these are constantly being updated, at any given time, they are authoritative
    • Use the EMC & NetApp vCenter plug-ins, automates best practices
    • 26. Use Multiple NFS datastores & 10GbE
    • 27. 1GbE requires more complexity to address I/O scaling due to one data session per connection with NFSv3
  • General NFS Best Practices - Timeouts
    • Configure the following on each ESX server (automated by vCenter plugins):
    • 28.  NFS.HeartbeatFrequency = 12
    • 29. NFS.HeartbeatTimeout = 5
    • 30. NFS.HeartbeatMaxFailures = 10
    • 31. Increase Guest OS time-out values to match
    Back up your Windows registry.
    Select Start>Run, regedit
    In the left‐panel hierarchy view, double‐click HKEY_LOCAL_MACHINE> System> CurrentControlSet> Services> Disk.
    Select the TimeOutValue and set the data value to 125 (decimal).
    Note: this is not reset when VMtools are updated
    • Increase Net.TcpIpHeapSize (follow vendor recommendation)
  • General NFS Best Practices – Traditional Ethernet switches
    Mostly seen with older 1GbE switching platforms
    Each switch operates independently
    More complex network design
    Depends on routing, requires two (or more) IP subnets for datastore traffic
    Multiple Ethernet options based on Etherchannel capabilities and preferences
    Some links may be passive standby links
  • 32. General NFS Best Practices – Multi-Switch Link Aggregation
    Allows two physical switches to operate as a single logical fabric
    Much simpler network design
    Single IP subnet
    Provides multiple active connections to each storage controller
    Easily scales to more connections by adding NICs and aliases
    Storage controller connection load balancing is automatically managed by the EtherChannel IP load-balancing policy
  • 33. General NFS Best Practices – HA and Scaling
    10GbE?
    Yes
    No
    Support
    multi-switch
    Link
    aggr?
    One VMKernel port
    & IP subnet
    Yes
    Use multiple links with
    IP hash load balancing on
    the NFS client (ESX)
    Use multiple VMKernel
    Ports & IP subnets
    Use multiple links with
    IP hash load balancing on
    The NFS server (array)
    Use ESX routing table
    Storage needs multiple
    sequential IP addresses
    Storage needs multiple
    sequential IP addresses
  • 34. iSCSI & NFS – Ethernet Jumbo Frames
    What is an Ethernet Jumbo Frame?
    Ethernet frames with more than 1500 bytes of payload (9000 is common)
    Commonly ‘thought of’ as having better performance due to greater payload per packet / reduction of packets
    • Should I use Jumbo Frames?
    Supported by all major storage vendors & VMware
    Adds complexity & performance gains are marginal with common block sizes
    FCoE uses MTU of 2240 which is auto-configured via switch and CAN handshake
    All IP traffic transfers at default MTU size
    Stick with the defaults when you can
  • 35. iSCSI & NFS caveat when used together
    Remember – iSCSI and NFS network HA models = DIFFERENT
    iSCSI uses vmknics with no Ethernet failover – using MPIO instead
    NFS client relies on vmknics using link aggregation/Ethernet failover
    NFS relies on host routing table
    • NFS traffic will use iSCSI vmknic and results in links without redundancy
    Use of multiple session iSCSI with NFS is not supported by NetApp
    EMC supports, but best practice is to have separate subnets, virtual interfaces
  • 36. Summary of “Setup Multipathing Right”
    VMFS/RDMs
    Round Robin policy for NMP is default BP on most storage platforms
    PowerPath/VE further simplifies/automates multipathing on all EMC (and many non-EMC) platforms.
    Notably supports MSCS/WSFC including vMotion and VM HA
    NFS
    For load balancing, distribute VMs across multiple datastores on multiple I/O paths. Follow the resiliency procedure in the TechBook to ensure VM resiliency to storage failover and reboot over NFS
  • 37. Alignment = good hygiene
    Key Best Practices circa 2010/2011
    C
  • 38. “Alignment = good hygiene”
    Misalignment of filesystems results in additional work on storage controller to satisfy IO request
    Affects every protocol, and every storage array
    VMFS on iSCSI, FC, & FCoE LUNs
    NFS
    VMDKs & RDMs with NTFS, EXT3, etc
    Filesystems exist in the datastore and VMDK
    Datastore Alignment
    VMFS 1MB-8MB
    VMFS 1MB-8MB
    Block
    Array 4KB-64KB
    Chunk
    Chunk
    Chunk
    Array 4KB-64KB
  • 39. “Alignment = good hygiene”
    Misalignment of filesystems results in additional work on storage controller to satisfy IO request
    Affects every protocol, and every storage array
    VMFS on iSCSI, FC, & FCoE LUNs
    NFS
    VMDKs & RDMs with NTFS, EXT3, etc
    Filesystems exist in the datastore and VMDK
    Datastore Alignment
    VMFS 1MB-8MB
    VMFS 1MB-8MB
    Block
    Array 4KB-64KB
    Array 4KB-64KB
    Chunk
    Chunk
    Chunk
  • 40. “Alignment = good hygiene”
    Misalignment of filesystems results in additional work on storage controller to satisfy IO request
    Affects every protocol, and every storage array
    VMFS on iSCSI, FC, & FCoE LUNs
    NFS
    VMDKs & RDMs with NTFS, EXT3, etc
    Filesystems exist in the datastore and VMDK
    Cluster
    Cluster
    Cluster
    FS 4KB-1MB
    Guest
    Alignment
    VMFS 1MB-8MB
    Block
    VMFS 1MB-8MB
    Array 4KB-64KB
    Chunk
    Chunk
    Chunk
    Array 4KB-64KB
  • 41. “Alignment = good hygiene”
    Misalignment of filesystems results in additional work on storage controller to satisfy IO request
    Affects every protocol, and every storage array
    VMFS on iSCSI, FC, & FCoE LUNs
    NFS
    VMDKs & RDMs with NTFS, EXT3, etc
    Filesystems exist in the datastore and VMDK
    Cluster
    Cluster
    Cluster
    FS 4KB-1MB
    Guest
    Alignment
    Block
    VMFS 1MB-8MB
    Chunk
    Chunk
    Chunk
    Array 4KB-64KB
  • 42. Alignment – Best Solution: “Align VMs”
    VMware, Microsoft, Citrix, EMC all agree, align partitions
    Plug-n-Play Guest Operating Systems
    Windows 2008, Vista, & Win7
    • They just work as their partitions start at 1MB
    Guest Operating Systems requiring manual alignment
    Windows NT, 2000, 2003, & XP (use diskpart to set to 1MB)
    Linux (use fdisk expert mode and align on 2048 = 1MB)
  • 43. Alignment – “Fixing after the fact”
    VMFS is misaligned
    Occurs If you created the VMFS via CLI and not via vSphere client and didn’t specify an offset.
    Resolution:
    Step 1: Take an array snapshot/backup
    Step 2: Create new datastore & migrate VMs using SVMotion
    Filesystem in the VMDK is misaligned
    Occurs If you are are using older OSes and didn’t align when you created the guest filesystem
    Resolution:
    Step 1: Take an array snapshot/backup
    Step 2: Use tools to realign (all VM to be shutdown)
    • GParted (free, but some assembly required)
    • 44. Quest vOptimizer (good mass scheduling and reporting)
  • Leverage Free Plugins/VAAI
    Key Best Practices circa 2010/2011
    D
  • 45. “Leverage Free Plugins and VAAI”
    Use Vendor plug-ins for VMware vSphere
    All provide better visibility
    Some provide integrated provisioning
    Some integrate array features like VM snapshots, dedupe, compression and more
    Some automate multipathing setup
    Some automate best practices and remediation
    Most are FREE
    VAAI – it is just “on”
    With vSphere 4.1, VAAI increases VM scalability and reduces the amount of I/O traffic sent between the host and storage system and makes “never put more than ___ VMs per datastore” a thing of the past.
    Some individual operations can be faster also (2-10x!)
  • 46. KISS on Layout
    Key Best Practices circa 2010/2011
    E
  • 47. “KISS on Layout”
    Use VMFS and NFS together – no reason not to
    Strongly consider 10GbE, particularly for new deployments
    Avoid RDMs, use “Pools” (VMFS or NFS)
    Make the datastores big
    VMFS – make them ~1.9TB in size (2TB – 512 bytes is the max for a single volume), 64TB for a single filesystem
    NFS – make them what you want (16TB is the max)
    With vSphere 4.0 and later, you can have many VMs per VMFS datastore – and VAAI increases this to a non-issue.
    On the array, default to Storage Pools, not traditional RAID Groups / Hypers
    Default to single extent VMFS datastores
    Default to Thin Provisioning models at the array level, optionally at the VMware level.
    Make sure you enable vCenter managed datastore alerts
    Make sure you enable Unisphere/SMC thin provisioning alerts and auto-expansion
    Use “broad” data services – i.e. FAST, FAST Cache (things that are “set in one place”)
  • 48. Use SIOC if you can
    Key Best Practices circa 2010/2011
    F
  • 49. “Use SIOC if you can”
    This is a huge vSphere 4.1 feature
    “If you can” equals:
    vSphere 4.1, Enterprise Plus
    VMFS (NFS targeted for future vSphere releases – not purely a qual)
    Enable it (not on by default), even if you don’t use shares – will ensure no VM swamps the others
    Bonus is you will get guest-level latency alerting!
    Default threshold is 30ms
    Leave it at 30ms for 10K/15K, increase to 50ms for 7.2K, decrease to 10ms for SSD
    Fully supported with array auto-tiering - leave it at 30ms for FAST pools
    Hard IO limits are handy for View use cases
    Some good recommended reading:
    http://www.vmware.com/files/pdf/techpaper/VMW-vSphere41-SIOC.pdf
    http://virtualgeek.typepad.com/virtual_geek/2010/07/vsphere-41-sioc-and-array-auto-tiering.html
    http://virtualgeek.typepad.com/virtual_geek/2010/08/drs-for-storage.html
    http://www.yellow-bricks.com/2010/09/29/storage-io-fairness/
  • 50. Second - What to do when you’re in trouble...
    Getting yourself out of a jam
  • 51. “My VM is not performing as expected”
    How do I know: application not meeting a pre-defined SLA, or SIOC GOS thresholds being exceeded
    What do I do:
    Step 1, pinpoint (thank you Scott Drummonds!)
    Use ESXTop first: http://communities.vmware.com/docs/DOC-5490
    ..then vSCSIstats: http://communities.vmware.com/docs/DOC-10095
    Step 2, if the backend:
    Use Unisphere Analyzer, SPA (start with backend and CPU)
    Check VM alignment (will show excessive stripe crossings)
    Cache enabled, FAST/FAST Cache settings on the storage pool
    ensure FAST and SIOC settings are consistent
    if your VM is compressed with EMC Data deduplication/compression, consider uncompressing it using the plug-in
  • 52. “I see all these device events in vSphere”
    How do I know: VM is not performing well and LUN trespasses warning messages in event log
    What do I do: ensure the right failover mode and policy are used. Ensure you have redundant paths from host to storage system
  • 53. “Datastore capacity utilization is low/high”
    How do I know:
    Managed Datastore Reports in vCenter 4.x
    Array tools - e.g. Unisphere (vCenter Integration) Report
    What do I do:
    Migrate the VM to a datastore that is configured over a virtually provisioned storage. For VMFS datastore, ESX thin provisioning/compress/dedupe can also be utilized
    For VM on NFS, Data Deduplication can be used via the plug-in to compress the VM when some performance impact is acceptable
  • 54. “My storage team gives me tiny devices”
    How do I know:
    Often I hear “they tell us we can’t get more than 240GB”
    What do I do:
    This means you have an “oldey timey” storage team 
    Symmetrix uses hyper devices, and hypers are assembled into meta devices (which then are presented to hosts)
    Hyper devices have a maximum of 240GB
    Configuring meta devices is EASY.
    Engage your array vendor to move your storage team into the 21st century 
  • 55. “What? VAAI isn’t working….”
    How do I know: Testing Storage VMotion/Cloning with no-offload versus Offload
    What do I do:
    Ensure the block storage initiators for the ESX host is configured ALUA on, also ensure the ESX server recognizes the change in the SATP – look at IO bandwidth in vSphere client and storage array.
    Benefit tends to be higher when svmotion across SPs
    Biggest benefit isn’t any single operation being faster, but rather overall system (vSphere, network, storage) load lightened
  • 56. “My NFS based VM is impacted following a storage reboot or failover”
    How do I know: VM freezes or, even worse, crashes
    What do I do:
    Check your ESX NFS timeout settings compare to TechBook recommendations (only needed if the datastore wasn’t created using the plug-in)
    Review your VM and guest OS settings for resiliency. See TechBook for detailed procedure on VM resiliency
  • 57. Third – knowing when to break the rules…
    Top 5 Exceptions for said best practices
  • 58. 5 Exceptions to the rules
    Create “planned datastore designs” (rather than big pools and correct after the fact) for larger IO use cases (View, SAP, Oracle, Exchange)
    Use the VMware + Array Vendor reference architectures.
    Generally the cases where > 32 HBA queue & consider > 1 vSCSI adapters
    Over time, SIOC may prove to be a good approach
    Some relatively rare cases where large spanned VMFS datastores make sense
    When NOT to use “datastore pools”, but pRDMs (narrow use cases!)
    MSCS/WSFC
    Oracle – pRDMs and NFS can do rapid VtoP with array snapshots
    When NOT to use NMP Round Robin
    Arrays that are not active/active AND use ALUA using only SCSI-2
    When NOT to use array thin-provisioned devices
    Datastores with extremely high amount of small block random IO
    In FLARE 30, always use storage pools, LUN migrate to thick devices if needed
    When NOT to use the vCenter plugins? Trick question – always “yes”
  • 59. Fourth – a peak into the future…
    Amazing things we’re working on….
  • 60. 5 Amazing things we’re working on….
    Storage Policy
    How should storage inform vSphere of capabilities and state (and vice versa)
    SIOC and Auto-Tiering complement today, how can we integrate?
    How can we embed VM-level Encryption?
    “Bolt-on” vs. “Built for Purpose” using Virtual Appliance constructs
    EMC has 3 shipping virtual storage appliances (Atmos/VE, Avamar/VE, Networker/VE)
    Every EMC array is really a cluster of commodity servers with disks
    What more could we do to make “bolt on value” easier this way?
    “follow the breadcrumb trail”: http://stevetodd.typepad.com/my_weblog/2010/09/csx-technology.html
    Maturing scale-out NAS/pNFS models
    Desired, not demanded in enterprise, demanded, not desired for scale-out public cloud NAS (EMC has GA’ed pNFS, but vSphere client is still NFSv3)
    Large-scale, long distance geo-dispersion/federation of transactional workloads
    VM Teleportation – around the world, at many sites
    Geo-location to meet FISMA and other standards
    Making Object storage act transactional – for real
    Would blend best of all worlds & enable VM-level policy and enforcement.

×