This is the opening keynote presentation, focusing on VMware and storage best practices, from the Midwest Regional VMUG in Kansas City on December 6, 2010.
Vector Search -An Introduction in Oracle Database 23ai.pptx
Â
Next-Generation Best Practices for VMware and Storage
1. Midwest Regional VMUG Next Generation Best Practices for Storage and VMware Scott Lowe, VCDX #39 vSpecialist, EMC Corporation Author, Mastering VMware vSphere 4 http://blog.scottlowe.org http://twitter.com/scott_lowe
2. The âGreatâ Protocol Debate Every protocol can Be Highly Available, and generally, every protocol can meet a broad performance band Each protocol has different configuration considerations Each Protocol has a VMware âsuper-powerâ, and also a âkryponiteâ In vSphere, there is core feature equality across protocols Conclusion: there is no debate â pick what works for you! The best flexibility comes from a combination of VMFS and NFS
3. First - Key Things To Know â âAâ thru âFâ Key Best Practices circa 2010/2011
5. Key Docs, and Storage Array Taxonomy Key VMware Docs: Fibre Channel SAN Configuration Guide iSCSI SAN Configuration Guide Storage/SAN Compatibility Guide âŠUnderstand VMware Storage Taxonomy: Active/Active (LUN ownership) Active/Passive (LUN ownership) Virtual Port (iSCSI only)
6. Key Docs, and Storage Array Taxonomy Key Storage Partner Docs: Each Array is very different. Storage varies far more vendor to vendor than servers do Find, read, and stay current on your arrayâs Best Practices Doc â most are excellent. Even if youâre NOT the storage team, read them â it will help you. http://www.emc.com/collateral/hardware/solution-overview/h2529-vmware-esx-svr-w-symmetrix-wp-ldv.pdf http://www.emc.com/collateral/hardware/technical-documentation/h5536-vmware-esx-srvr-using-celerra-stor-sys-wp.pdf http://www.emc.com/collateral/software/solution-overview/h2197-vmware-esx-clariion-stor-syst-ldv.pdf
9. Whatâs âout of the boxâ in vSphere 4.1? [root@esxi ~]# vmware -vVMware ESX 4.1.0 build-260247 [root@esxi ~]# esxcli nmp satp listName                 Default PSP       DescriptionVMW_SATP_SYMM        VMW_PSP_FIXED     Placeholder (plugin not loaded)VMW_SATP_SVC         VMW_PSP_FIXED     Placeholder (plugin not loaded)VMW_SATP_MSA         VMW_PSP_MRU       Placeholder (plugin not loaded)VMW_SATP_LSI         VMW_PSP_MRU       Placeholder (plugin not loaded)VMW_SATP_INV         VMW_PSP_FIXED     Placeholder (plugin not loaded)VMW_SATP_EVA         VMW_PSP_FIXED     Placeholder (plugin not loaded)VMW_SATP_EQL         VMW_PSP_FIXED     Placeholder (plugin not loaded)VMW_SATP_DEFAULT_AP  VMW_PSP_MRU       Placeholder (plugin not loaded)VMW_SATP_ALUA_CX     VMW_PSP_FIXED_AP  Placeholder (plugin not loaded)VMW_SATP_CX          VMW_PSP_MRU       Supports EMC CX that do not use the ALUA protocolVMW_SATP_ALUA        VMW_PSP_RR        Supports non-specific arrays that use the ALUA protocolVMW_SATP_DEFAULT_AA  VMW_PSP_FIXED     Supports non-specific active/active arraysVMW_SATP_LOCAL       VMW_PSP_FIXED     Supports direct attached devices
10.
11. All IO goes down preferred path, reverts to preferred path after original path restore
15. n IO operations goes down active path then rotate (default is 1000)HOWTO â setting PSP for a specific device (can override default selected by SATP detected ARRAYID): esxcli nmp device setpolicy --device <device UID> --psp VMW_PSP_RR (check with your vendor first!)
16. Changing Round Robin IOOperationLimit esxcli nmp roundrobin setconfig --device <device UID> âiops check with your storage vendor first! This setting can cause problems on arrays. Has been validated ok, but not necessary in most cases
17. Effect of different RR IOOperationLimit settings NOTE: This is with a SINGLE LUN. This is the case where the larger IOOperationLimit default is the worst In a real-world environment â lots of LUNs and VMs results in decent overall loadbalancing Recommendation â if you can, stick with the default
18.
19. Enter ALUA to solve this issueSupported introduced in vSphere 4.0 SP A SP B LUN
22. MPIO Exceptions â Windows Clusters Among a long list of ânot supportedâ things: NO Clustering on NFS datastores No Clustering on iSCSI, FCoE (unless using PP/VE) No round-robin with native multipathing (unless using PP/VE) NO Mixed environments, such as configurations where one cluster node is running a different version of ESX/ESXi than another cluster node. NO Use of MSCS in conjunction with VMware Fault Tolerance. NO Migration with vMotion of clustered virtual machines. NO N-Port ID Virtualization (NPIV) You must use hardware version 7 with ESX/ESXi 4.1
23. APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS OS Shared Storage STORAGE PowerPath â a Multipathing Plugin (MPP) Simple Storage manageability Simple Provisioning = âPool of Connectivityâ Predictable and consistent Optimize server, storage, and data-path utilization Performance and Scale Tune infrastructure performance, LUN/Path Prioritization Predictive Array Specific Load Balancing Algorithms Automatic HBA, Path, and storage processor fault recovery Other 3rd party MPPs: Dell/Equalogic PSP Uses a âleast deep queueâ algorithm rather than basic round robin Can redirect IO to different peer storage nodes See this at the Dell/EqualLogic booth PowerPath PowerPath PowerPath PowerPath
32. General NFS Best Practices â Multi-Switch Link Aggregation Allows two physical switches to operate as a single logical fabric Much simpler network design Single IP subnet Provides multiple active connections to each storage controller Easily scales to more connections by adding NICs and aliases Storage controller connection load balancing is automatically managed by the EtherChannel IP load-balancing policy
33. General NFS Best Practices â HA and Scaling 10GbE? Yes No Support multi-switch Link aggr? One VMKernel port & IP subnet Yes Use multiple links with IP hash load balancing on the NFS client (ESX) Use multiple VMKernel Ports & IP subnets Use multiple links with IP hash load balancing on The NFS server (array) Use ESX routing table Storage needs multiple sequential IP addresses Storage needs multiple sequential IP addresses
34.
35.
36. Summary of âSetup Multipathing Rightâ VMFS/RDMs Round Robin policy for NMP is default BP on most storage platforms PowerPath/VE further simplifies/automates multipathing on all EMC (and many non-EMC) platforms. Notably supports MSCS/WSFC including vMotion and VM HA NFS For load balancing, distribute VMs across multiple datastores on multiple I/O paths. Follow the resiliency procedure in the TechBook to ensure VM resiliency to storage failover and reboot over NFS
38. âAlignment = good hygieneâ Misalignment of filesystems results in additional work on storage controller to satisfy IO request Affects every protocol, and every storage array VMFS on iSCSI, FC, & FCoE LUNs NFS VMDKs & RDMs with NTFS, EXT3, etc Filesystems exist in the datastore and VMDK Datastore Alignment VMFS 1MB-8MB VMFS 1MB-8MB Block Array 4KB-64KB Chunk Chunk Chunk Array 4KB-64KB
39. âAlignment = good hygieneâ Misalignment of filesystems results in additional work on storage controller to satisfy IO request Affects every protocol, and every storage array VMFS on iSCSI, FC, & FCoE LUNs NFS VMDKs & RDMs with NTFS, EXT3, etc Filesystems exist in the datastore and VMDK Datastore Alignment VMFS 1MB-8MB VMFS 1MB-8MB Block Array 4KB-64KB Array 4KB-64KB Chunk Chunk Chunk
40. âAlignment = good hygieneâ Misalignment of filesystems results in additional work on storage controller to satisfy IO request Affects every protocol, and every storage array VMFS on iSCSI, FC, & FCoE LUNs NFS VMDKs & RDMs with NTFS, EXT3, etc Filesystems exist in the datastore and VMDK Cluster Cluster Cluster FS 4KB-1MB Guest Alignment VMFS 1MB-8MB Block VMFS 1MB-8MB Array 4KB-64KB Chunk Chunk Chunk Array 4KB-64KB
41. âAlignment = good hygieneâ Misalignment of filesystems results in additional work on storage controller to satisfy IO request Affects every protocol, and every storage array VMFS on iSCSI, FC, & FCoE LUNs NFS VMDKs & RDMs with NTFS, EXT3, etc Filesystems exist in the datastore and VMDK Cluster Cluster Cluster FS 4KB-1MB Guest Alignment Block VMFS 1MB-8MB Chunk Chunk Chunk Array 4KB-64KB
42.
43.
44.
45. âLeverage Free Plugins and VAAIâ Use Vendor plug-ins for VMware vSphere All provide better visibility Some provide integrated provisioning Some integrate array features like VM snapshots, dedupe, compression and more Some automate multipathing setup Some automate best practices and remediation Most are FREE VAAI â it is just âonâ With vSphere 4.1, VAAI increases VM scalability and reduces the amount of I/O traffic sent between the host and storage system and makes ânever put more than ___ VMs per datastoreâ a thing of the past. Some individual operations can be faster also (2-10x!)
47. âKISS on Layoutâ Use VMFS and NFS together â no reason not to Strongly consider 10GbE, particularly for new deployments Avoid RDMs, use âPoolsâ (VMFS or NFS) Make the datastores big VMFS â make them ~1.9TB in size (2TB â 512 bytes is the max for a single volume), 64TB for a single filesystem NFS â make them what you want (16TB is the max) With vSphere 4.0 and later, you can have many VMs per VMFS datastore â and VAAI increases this to a non-issue. On the array, default to Storage Pools, not traditional RAID Groups / Hypers Default to single extent VMFS datastores Default to Thin Provisioning models at the array level, optionally at the VMware level. Make sure you enable vCenter managed datastore alerts Make sure you enable Unisphere/SMC thin provisioning alerts and auto-expansion Use âbroadâ data services â i.e. FAST, FAST Cache (things that are âset in one placeâ)
48. Use SIOC if you can Key Best Practices circa 2010/2011 F
49. âUse SIOC if you canâ This is a huge vSphere 4.1 feature âIf you canâ equals: vSphere 4.1, Enterprise Plus VMFS (NFS targeted for future vSphere releases â not purely a qual) Enable it (not on by default), even if you donât use shares â will ensure no VM swamps the others Bonus is you will get guest-level latency alerting! Default threshold is 30ms Leave it at 30ms for 10K/15K, increase to 50ms for 7.2K, decrease to 10ms for SSD Fully supported with array auto-tiering - leave it at 30ms for FAST pools Hard IO limits are handy for View use cases Some good recommended reading: http://www.vmware.com/files/pdf/techpaper/VMW-vSphere41-SIOC.pdf http://virtualgeek.typepad.com/virtual_geek/2010/07/vsphere-41-sioc-and-array-auto-tiering.html http://virtualgeek.typepad.com/virtual_geek/2010/08/drs-for-storage.html http://www.yellow-bricks.com/2010/09/29/storage-io-fairness/
50. Second - What to do when youâre in trouble... Getting yourself out of a jam
51. âMy VM is not performing as expectedâ How do I know: application not meeting a pre-defined SLA, or SIOC GOS thresholds being exceeded What do I do: Step 1, pinpoint (thank you Scott Drummonds!) Use ESXTop first: http://communities.vmware.com/docs/DOC-5490 ..then vSCSIstats: http://communities.vmware.com/docs/DOC-10095 Step 2, if the backend: Use Unisphere Analyzer, SPA (start with backend and CPU) Check VM alignment (will show excessive stripe crossings) Cache enabled, FAST/FAST Cache settings on the storage pool ensure FAST and SIOC settings are consistent if your VM is compressed with EMC Data deduplication/compression, consider uncompressing it using the plug-in
52. âI see all these device events in vSphereâ How do I know: VM is not performing well and LUN trespasses warning messages in event log What do I do: ensure the right failover mode and policy are used. Ensure you have redundant paths from host to storage system
53. âDatastore capacity utilization is low/highâ How do I know: Managed Datastore Reports in vCenter 4.x Array tools - e.g. Unisphere (vCenter Integration) Report What do I do: Migrate the VM to a datastore that is configured over a virtually provisioned storage. For VMFS datastore, ESX thin provisioning/compress/dedupe can also be utilized For VM on NFS, Data Deduplication can be used via the plug-in to compress the VM when some performance impact is acceptable
54. âMy storage team gives me tiny devicesâ How do I know: Often I hear âthey tell us we canât get more than 240GBâ What do I do: This means you have an âoldey timeyâ storage team ï Symmetrix uses hyper devices, and hypers are assembled into meta devices (which then are presented to hosts) Hyper devices have a maximum of 240GB Configuring meta devices is EASY. Engage your array vendor to move your storage team into the 21st century ï
55. âWhat? VAAI isnât workingâŠ.â How do I know: Testing Storage VMotion/Cloning with no-offload versus Offload What do I do: Ensure the block storage initiators for the ESX host is configured ALUA on, also ensure the ESX server recognizes the change in the SATP â look at IO bandwidth in vSphere client and storage array. Benefit tends to be higher when svmotion across SPs Biggest benefit isnât any single operation being faster, but rather overall system (vSphere, network, storage) load lightened
56. âMy NFS based VM is impacted following a storage reboot or failoverâ How do I know: VM freezes or, even worse, crashes What do I do: Check your ESX NFS timeout settings compare to TechBook recommendations (only needed if the datastore wasnât created using the plug-in) Review your VM and guest OS settings for resiliency. See TechBook for detailed procedure on VM resiliency
57. Third â knowing when to break the rules⊠Top 5 Exceptions for said best practices
58. 5 Exceptions to the rules Create âplanned datastore designsâ (rather than big pools and correct after the fact) for larger IO use cases (View, SAP, Oracle, Exchange) Use the VMware + Array Vendor reference architectures. Generally the cases where > 32 HBA queue & consider > 1 vSCSI adapters Over time, SIOC may prove to be a good approach Some relatively rare cases where large spanned VMFS datastores make sense When NOT to use âdatastore poolsâ, but pRDMs (narrow use cases!) MSCS/WSFC Oracle â pRDMs and NFS can do rapid VtoP with array snapshots When NOT to use NMP Round Robin Arrays that are not active/active AND use ALUA using only SCSI-2 When NOT to use array thin-provisioned devices Datastores with extremely high amount of small block random IO In FLARE 30, always use storage pools, LUN migrate to thick devices if needed When NOT to use the vCenter plugins? Trick question â always âyesâ
59. Fourth â a peak into the future⊠Amazing things weâre working onâŠ.
60. 5 Amazing things weâre working onâŠ. Storage Policy How should storage inform vSphere of capabilities and state (and vice versa) SIOC and Auto-Tiering complement today, how can we integrate? How can we embed VM-level Encryption? âBolt-onâ vs. âBuilt for Purposeâ using Virtual Appliance constructs EMC has 3 shipping virtual storage appliances (Atmos/VE, Avamar/VE, Networker/VE) Every EMC array is really a cluster of commodity servers with disks What more could we do to make âbolt on valueâ easier this way? âfollow the breadcrumb trailâ: http://stevetodd.typepad.com/my_weblog/2010/09/csx-technology.html Maturing scale-out NAS/pNFS models Desired, not demanded in enterprise, demanded, not desired for scale-out public cloud NAS (EMC has GAâed pNFS, but vSphere client is still NFSv3) Large-scale, long distance geo-dispersion/federation of transactional workloads VM Teleportation â around the world, at many sites Geo-location to meet FISMA and other standards Making Object storage act transactional â for real Would blend best of all worlds & enable VM-level policy and enforcement.