Successfully reported this slideshow.
Your SlideShare is downloading. ×

Vmug V Sphere Storage (Rev E)

Upcoming SlideShare
Les 08 tune_rman
Les 08 tune_rman
Loading in …3

Check these out next

1 of 78 Ad

More Related Content

Similar to Vmug V Sphere Storage (Rev E) (20)


Recently uploaded (20)

Vmug V Sphere Storage (Rev E)

  1. 1. vSphere 4.0 Storage: Features and Enhancements Nathan Small Staff Engineer Rev E Last updated 3 rd August 2009 VMware Inc
  2. 2. Introduction <ul><li>This presentation is a technical overview and deep dive of some of the features and enhancements to the storage stack and related storage components of vSphere 4.0 </li></ul>
  3. 3. New Acronyms in vSphere 4 <ul><ul><li>MPX = VMware Generic Multipath Device (No Unique Identifier) </li></ul></ul><ul><ul><li>NAA = Network Addressing Authority </li></ul></ul><ul><ul><li>PSA = Pluggable Storage Architecture </li></ul></ul><ul><ul><li>MPP = Multipathing Plugin </li></ul></ul><ul><ul><li>NMP = Native Multipathing </li></ul></ul><ul><ul><li>SATP = Storage Array Type Plugin </li></ul></ul><ul><ul><li>PSP = Path Selection Policy </li></ul></ul>
  4. 4. vSphere Storage <ul><ul><li>Section 1 - Naming Convention Change </li></ul></ul><ul><ul><li>Section 2 - Pluggable Storage Architecture </li></ul></ul><ul><ul><li>Section 3 - iSCSI Enhancements </li></ul></ul><ul><ul><li>Section 4 - Storage Administration (VC) </li></ul></ul><ul><ul><li>Section 5 - Snapshot Volumes & Resignaturing </li></ul></ul><ul><ul><li>Section 6 - Storage VMotion </li></ul></ul><ul><ul><li>Section 7 - Volume Grow / Hot VMDK Extend </li></ul></ul><ul><ul><li>Section 8 - Storage CLI Enhancements </li></ul></ul><ul><ul><li>Section 9 - Other Storage Features/Enhancements </li></ul></ul>
  5. 5. Naming Convention Change in vSphere 4 <ul><ul><li>Although the vmhbaN:C:T:L:P naming convention is visible, it is now known as the run-time name and is no longer guaranteed to be persistent through reboots. </li></ul></ul><ul><ul><li>ESX 4 now uses the unique LUN identifiers, typically the NAA (Network Addressing Authority) ID. This is true for the CLI as well as the GUI and is also the naming convention used during the install. </li></ul></ul><ul><ul><ul><li>The IQN (iSCSI Qualified Name) is still used for iSCSI targets. </li></ul></ul></ul><ul><ul><ul><li>The WWN (World Wide Name) is still used for Fiber Channel targets. </li></ul></ul></ul><ul><ul><li>For those devices which do not have a unique ID, you will observe an MPX reference (which is basically stands for VMware Multipath X device). </li></ul></ul>
  6. 6. vSphere Storage <ul><ul><li>Section 1 - Naming Convention Change </li></ul></ul><ul><ul><li>Section 2 - Pluggable Storage Architecture </li></ul></ul><ul><ul><li>Section 3 - iSCSI Enhancements </li></ul></ul><ul><ul><li>Section 4 - Storage Administration (VC) </li></ul></ul><ul><ul><li>Section 5 - Snapshot Volumes & Resignaturing </li></ul></ul><ul><ul><li>Section 6 - Storage VMotion </li></ul></ul><ul><ul><li>Section 7 - Volume Grow / Hot VMDK Extend </li></ul></ul><ul><ul><li>Section 8 - Storage CLI Enhancements </li></ul></ul><ul><ul><li>Section 9 - Other Storage Features/Enhancements </li></ul></ul>
  7. 7. Pluggable Storage Architecture <ul><ul><li>PSA , the Pluggable Storage Architecture , is a collection of VMkernel APIs that allow third party hardware vendors to insert code directly into the ESX storage I/O path. </li></ul></ul><ul><ul><li>This allows 3 rd party software developers to design their own load balancing techniques and failover mechanisms for particular storage array types. </li></ul></ul><ul><ul><ul><li>This also means that 3 rd party vendors can now add support for new arrays into ESX without having to provide internal information or intellectual property about the array to VMware. </li></ul></ul></ul><ul><ul><li>VMware, by default, provide a generic Multipathing Plugin ( MPP ) called NMP ( Native Multipathing Plugin ). </li></ul></ul><ul><ul><li>PSA co-ordinates the operation of the NMP and any additional 3 rd party MPP. </li></ul></ul>
  8. 8. PSA Tasks <ul><ul><li>Loads and unloads multipathing plugins (MPPs). </li></ul></ul><ul><ul><li>Handles physical path discovery and removal (via scanning). </li></ul></ul><ul><ul><li>Routes I/O requests for a specific logical device to an appropriate MPP . </li></ul></ul><ul><ul><li>Handles I/O queuing to the physical storage HBAs & to the logical devices. </li></ul></ul><ul><ul><li>Implements logical device bandwidth sharing between Virtual Machines. </li></ul></ul><ul><ul><li>Provides logical device and physical path I/O statistics . </li></ul></ul>
  9. 9. MPP Tasks <ul><ul><li>The PSA discovers available storage paths and based on a set of predefined rules, the PSA will determine which MPP should be given ownership of the path. </li></ul></ul><ul><ul><li>The MPP then associates a set of physical paths with a specific logical device. </li></ul></ul><ul><ul><li>The specific details of handling path failover for a given storage array are delegated to a sub-plugin called a Storage Array Type Plugin (SATP). </li></ul></ul><ul><ul><ul><li>SATP is associated with paths. </li></ul></ul></ul><ul><ul><li>The specific details for determining which physical path is used to issue an I/O request ( load balancing ) to a storage device are handled by a sub-plugin called Path Selection Plugin (PSP). </li></ul></ul><ul><ul><ul><li>PSP is associated with logical devices. </li></ul></ul></ul>
  10. 10. NMP Specific Tasks <ul><ul><li>Manage physical path claiming and unclaiming . </li></ul></ul><ul><ul><li>Register and de-register logical devices. </li></ul></ul><ul><ul><li>Associate physical paths with logical devices . </li></ul></ul><ul><ul><li>Process I/O requests to logical devices: </li></ul></ul><ul><ul><ul><li>Select an optimal physical path for the request (load balance) </li></ul></ul></ul><ul><ul><ul><li>Perform actions necessary to handle failures and request retries . </li></ul></ul></ul><ul><ul><li>Support management tasks such as abort or reset of logical devices. </li></ul></ul>
  11. 11. Storage Array Type Plugin - SATP <ul><ul><li>An Storage Array Type Plugin (SATP) handles path failover operations . </li></ul></ul><ul><ul><li>VMware provides a default SATP for each supported array as well as a generic SATP (an active/active version and an active/passive version) for non-specified storage arrays. </li></ul></ul><ul><ul><li>If you want to take advantage of certain storage specific characteristics of your array, you can install a 3rd party SATP provided by the vendor of the storage array, or by a software company specializing in optimizing the use of your storage array. </li></ul></ul><ul><ul><li>Each SATP implements the support for a specific type of storage array, e.g. VMW_SATP_SVC for IBM SVC . </li></ul></ul>
  12. 12. SATP (ctd) <ul><ul><li>The primary functions of an SATP are: </li></ul></ul><ul><ul><ul><li>Implements the switching of physical paths to the array when a path has failed. </li></ul></ul></ul><ul><ul><ul><li>Determines when a hardware component of a physical path has failed . </li></ul></ul></ul><ul><ul><ul><li>Monitors the hardware state of the physical paths to the storage array. </li></ul></ul></ul><ul><ul><li>There are many storage array type plug-ins. To see the complete list, you can use the following commands: </li></ul></ul><ul><ul><ul><li># esxcli nmp satp list </li></ul></ul></ul><ul><ul><ul><li># esxcli nmp satp listrules </li></ul></ul></ul><ul><ul><ul><li># esxcli nmp satp listrules –s < specific SATP> </li></ul></ul></ul>
  13. 13. Path Selection Plugin (PSP) <ul><ul><li>If you want to take advantage of more complex I/O load balancing algorithms, you could install a 3rd party Path Selection Plugin (PSP). </li></ul></ul><ul><ul><li>A PSP handles load balancing operations and is responsible for choosing a physical path to issue an I/O request to a logical device. </li></ul></ul><ul><ul><li>VMware provide three PSP: Fixed , MRU or Round Robin . </li></ul></ul><ul><ul><ul><li># esxcli nmp psp list </li></ul></ul></ul><ul><ul><ul><ul><li>Name Description </li></ul></ul></ul></ul><ul><ul><ul><ul><li>VMW_PSP_MRU Most Recently Used Path Selection </li></ul></ul></ul></ul><ul><ul><ul><ul><li>VMW_PSP_RR Round Robin Path Selection </li></ul></ul></ul></ul><ul><ul><ul><ul><li>VMW_PSP_FIXED Fixed Path Selection </li></ul></ul></ul></ul>
  14. 14. NMP Supported PSPs <ul><ul><li>Most Recently Used (MRU) — Selects the first working path discovered at system boot time. If this path becomes unavailable, the ESX host switches to an alternative path and continues to use the new path while it is available. </li></ul></ul><ul><ul><li>Fixed — Uses the designated preferred path , if it has been configured. Otherwise, it uses the first working path discovered at system boot time. If the ESX host cannot use the preferred path, it selects a random alternative available path. The ESX host automatically reverts back to the preferred path as soon as the path becomes available. </li></ul></ul><ul><ul><li>Round Robin (RR) – Uses an automatic path selection rotating through all available paths and enabling load balancing across the paths. </li></ul></ul>
  15. 15. Enabling Additional Logging on vSphere 4.0 <ul><ul><li>For additional SCSI Log Messages, set: </li></ul></ul><ul><ul><ul><li>Scsi.LogCmdErrors = &quot;1“ </li></ul></ul></ul><ul><ul><ul><li>Scsi.LogMPCmdErrors = &quot;1“ </li></ul></ul></ul><ul><ul><ul><ul><li>At GA, the default setting for Scsi.LogMPCmdErrors is &quot;1“ </li></ul></ul></ul></ul><ul><ul><li>These can be found in the Advanced Settings . </li></ul></ul>
  16. 16. Viewing Plugin Information <ul><ul><li>The following command lists all multipathing modules loaded on the system. At a minimum, this command returns the default VMware Native Multipath (NMP) plugin & the MASK_PATH plugin. Third-party MPPs will also be listed if installed: </li></ul></ul><ul><ul><ul><li># esxcfg-mpath -G </li></ul></ul></ul><ul><ul><ul><ul><li>MASK_PATH </li></ul></ul></ul></ul><ul><ul><ul><ul><li>NMP </li></ul></ul></ul></ul><ul><ul><li>For ESXi , the following VI CLI 4.0 command can be used: </li></ul></ul><ul><ul><ul><li># vicfg-mpath –G –-server <IP> --username <X> --password <Y> </li></ul></ul></ul><ul><ul><ul><ul><li>MASK_PATH </li></ul></ul></ul></ul><ul><ul><ul><ul><li>NMP </li></ul></ul></ul></ul><ul><ul><li>LUN path masking is done via the MASK_PATH Plug-in. </li></ul></ul>
  17. 17. Viewing Device Information <ul><ul><li>The command esxcli nmp device list lists all devices managed by the NMP plug-in and the configuration of that device, e.g.: </li></ul></ul><ul><ul><ul><li># esxcli nmp device list </li></ul></ul></ul><ul><ul><ul><ul><li>naa.600601601d311f001ee294d9e7e2dd11 </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Device Display Name: DGC iSCSI Disk (naa.600601601d311f001ee294d9e7e2dd11) </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Storage Array Type: VMW_SATP_CX </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Storage Array Type Device Config: {navireg ipfilter} </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Path Selection Policy: VMW_PSP_MRU </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Path Selection Policy Device Config: Current Path=vmhba33:C0:T0:L1 </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Working Paths: vmhba33:C0:T0:L1 </li></ul></ul></ul></ul><ul><ul><ul><ul><li>mpx.vmhba1:C0:T0:L0 </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Device Display Name: Local VMware Disk (mpx.vmhba1:C0:T0:L0) </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Storage Array Type: VMW_SATP_LOCAL </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Storage Array Type Device Config: </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Path Selection Policy: VMW_PSP_FIXED </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Path Selection Policy Device Config: {preferred=vmhba1:C0:T0:L0;current=vmhba1:C0:T0:L0} </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Working Paths: vmhba1:C0:T0:L0 </li></ul></ul></ul></ul>Specific configuration for EMC Clariion & Invista products mpx is used as an identifier for devices that do not have their own unique ids NAA is the Network Addressing Authority (NAA) identifier guaranteed to be unique
  18. 18. Viewing Device Information (ctd) <ul><ul><li>Get current path information for a specified storage device managed by the NMP. </li></ul></ul><ul><ul><ul><li># esxcli nmp device list -d naa.600601604320170080d407794f10dd11 </li></ul></ul></ul><ul><ul><ul><li>naa.600601604320170080d407794f10dd11 </li></ul></ul></ul><ul><ul><ul><li>Device Display Name: DGC Fibre Channel Disk (naa.600601604320170080d407794f10dd11) </li></ul></ul></ul><ul><ul><ul><li>Storage Array Type: VMW_SATP_CX </li></ul></ul></ul><ul><ul><ul><li>Storage Array Type Device Config: {navireg ipfilter} </li></ul></ul></ul><ul><ul><ul><li>Path Selection Policy: VMW_PSP_MRU </li></ul></ul></ul><ul><ul><ul><li>Path Selection Policy Device Config: Current Path=vmhba2:C0:T0:L0 </li></ul></ul></ul><ul><ul><ul><li>Working Paths: vmhba2:C0:T0:L0 </li></ul></ul></ul>
  19. 19. Viewing Device Information (ctd) <ul><ul><li>Lists all paths available for a specified storage device on ESX: </li></ul></ul><ul><li># esxcfg-mpath -b -d naa.600601601d311f001ee294d9e7e2dd11 </li></ul><ul><li>naa.600601601d311f001ee294d9e7e2dd11 : DGC iSCSI Disk (naa.600601601d311f001ee294d9e7e2dd11) </li></ul><ul><ul><li>vmhba33:C0:T0:L1 LUN:1 state:active iscsi Adapter: Target: Alias= Session=00023d000001 PortalTag=1 </li></ul></ul><ul><ul><li>vmhba33:C0:T1:L1 LUN:1 state:standby iscsi Adapter: Target: Alias= Session=00023d000001 PortalTag=2 </li></ul></ul><ul><ul><li>ESXi has an equivalent vicfg-mpath command. </li></ul></ul>
  20. 20. Viewing Device Information (ctd) <ul><li># esxcfg-mpath -l -d naa.6006016043201700d67a179ab32fdc11 </li></ul><ul><li>,,t,2-naa.600601601d311f001ee294d9e7e2dd11 </li></ul><ul><li>Runtime Name: vmhba33:C0:T1:L1 </li></ul><ul><li>Device: naa.600601601d311f001ee294d9e7e2dd11 </li></ul><ul><li>Device Display Name: DGC iSCSI Disk (naa.600601601d311f001ee294d9e7e2dd11) </li></ul><ul><li>Adapter: vmhba33 Channel: 0 Target: 1 LUN: 1 </li></ul><ul><li>Adapter Identifier: </li></ul><ul><li>Target Identifier: 00023d000001,,t,2 </li></ul><ul><li>Plugin: NMP </li></ul><ul><li>State: standby </li></ul><ul><li>Transport: iscsi </li></ul><ul><li>Adapter Transport Details: </li></ul><ul><li>Target Transport Details: Alias= Session=00023d000001 PortalTag=2 </li></ul><ul><li>,,t,1-naa.600601601d311f001ee294d9e7e2dd11 </li></ul><ul><li>Runtime Name: vmhba33:C0:T0:L1 </li></ul><ul><li>Device: naa.600601601d311f001ee294d9e7e2dd11 </li></ul><ul><li>Device Display Name: DGC iSCSI Disk (naa.600601601d311f001ee294d9e7e2dd11) </li></ul><ul><li>Adapter: vmhba33 Channel: 0 Target: 0 LUN: 1 </li></ul><ul><li>Adapter Identifier: </li></ul><ul><li>Target Identifier: 00023d000001,,t,1 </li></ul><ul><li>Plugin: NMP </li></ul><ul><li>State: active </li></ul><ul><li>Transport: iscsi </li></ul><ul><li>Adapter Transport Details: </li></ul><ul><li>Target Transport Details: Alias= Session=00023d000001 PortalTag=1 </li></ul>Storage array (target) iSCSI Qualified Names (IQNs)
  21. 21. Third-Party Multipathing Plug-ins (MPPs) <ul><ul><li>You can install the third-party multipathing plug-ins (MPPs) when you need to change specific load balancing and failover characteristics of ESX/ESXi. </li></ul></ul><ul><ul><li>The third-party MPPs replace the behaviour of the NMP and entirely take control over the path failover and the load balancing operations for certain specified storage devices. </li></ul></ul>
  22. 22. Third-Party SATP & PSP <ul><ul><li>Third-party SATP </li></ul></ul><ul><ul><ul><li>Generally developed by third-party hardware manufacturers who have ‘expert’ knowledge of the behaviour of their storage devices. </li></ul></ul></ul><ul><ul><ul><li>Accommodates specific characteristics of storage arrays and facilitates support for new arrays. </li></ul></ul></ul><ul><ul><li>Third-party PSP </li></ul></ul><ul><ul><ul><li>Generally developed by third-party software companies. </li></ul></ul></ul><ul><ul><ul><li>More complex I/O load balancing algorithms. </li></ul></ul></ul><ul><ul><li>NMP coordination </li></ul></ul><ul><ul><ul><li>Third-party SATPs and PSPs are coordinated by the NMP, and can be simultaneously used with the VMware SATPs and PSPs. </li></ul></ul></ul>
  23. 23. vSphere Storage <ul><ul><li>Section 1 - Naming Convention Change </li></ul></ul><ul><ul><li>Section 2 - Pluggable Storage Architecture </li></ul></ul><ul><ul><li>Section 3 - iSCSI Enhancements </li></ul></ul><ul><ul><li>Section 4 - Storage Administration (VC) </li></ul></ul><ul><ul><li>Section 5 - Snapshot Volumes & Resignaturing </li></ul></ul><ul><ul><li>Section 6 - Storage VMotion </li></ul></ul><ul><ul><li>Section 7 - Volume Grow / Hot VMDK Extend </li></ul></ul><ul><ul><li>Section 8 - Storage CLI Enhancements </li></ul></ul><ul><ul><li>Section 9 - Other Storage Features/Enhancements </li></ul></ul>
  24. 24. iSCSI Enhancements <ul><ul><li>ESX 4 includes an updated iSCSI stack which offers improvements to both software iSCSI (initiator that runs at the ESX layer) and hardware iSCSI (a hardware-optimized iSCSI HBA). </li></ul></ul><ul><ul><li>For both software and hardware iSCSI, functionality (e.g. CHAP support, digest acceleration, etc.) and performance are improved. </li></ul></ul><ul><ul><li>Software iSCSI can now be configured to use host based multipathing if you have more than one physical network adapter. </li></ul></ul><ul><ul><li>In the new ESX 4.0 Software iSCSI stack, there is no longer any requirement to have a Service Console connection to communicate to an iSCSI target. </li></ul></ul>
  25. 25. Software iSCSI Enhancements <ul><ul><li>iSCSI Advanced Settings </li></ul></ul><ul><ul><ul><li>In particular, data integrity checks in the form of digests . </li></ul></ul></ul><ul><ul><li>CHAP Parameters Settings </li></ul></ul><ul><ul><ul><li>A user will be able to specify CHAP parameters as per-target CHAP and mutual per-target CHAP. </li></ul></ul></ul><ul><ul><li>Inheritance model of parameters. </li></ul></ul><ul><ul><ul><li>A global set of configuration parameters can be set on the initiator and propagated down to all targets. </li></ul></ul></ul><ul><ul><li>Per target/discovery level configuration. </li></ul></ul><ul><ul><ul><li>Configuration settings can now be set on a per target basis which means that a customer can uniquely configure parameters for each array discovered by the initiator. </li></ul></ul></ul>
  26. 26. Software iSCSI Multipathing – Port Binding <ul><ul><li>You can now create a port binding between a physical NIC and a iSCSI VMkernel port in ESX 4.0. </li></ul></ul><ul><ul><li>Using the “port binding&quot; feature, users can map the multiple iSCSI VMkernel ports to different physical NICs. This will enable the software iSCSI initiator to use multiple physical NICs for I/O transfer. </li></ul></ul><ul><ul><li>Connecting the software iSCSI initiator to the VMkernel ports can only be done from the CLI using the esxcli swiscsi commands. </li></ul></ul><ul><ul><li>Host based multipathing can then manage the paths to the LUN. </li></ul></ul><ul><ul><li>In addition, Round Robin path policy can be configured to simultaneously use more than one physical NIC for the iSCSI traffic to the iSCSI. </li></ul></ul>
  27. 27. Hardware iSCSI Limitations <ul><ul><li>Mutual Chap is disabled. </li></ul></ul><ul><ul><li>Discovery is supported by IP address only (storage array name discovery not supported). </li></ul></ul><ul><ul><li>Running with the Hardware and Software iSCSI initiator enabled on the same host at the same time is not supported . </li></ul></ul>
  28. 28. vSphere Storage <ul><ul><li>Section 1 - Naming Convention Change </li></ul></ul><ul><ul><li>Section 2 - Pluggable Storage Architecture </li></ul></ul><ul><ul><li>Section 3 - iSCSI Enhancements </li></ul></ul><ul><ul><li>Section 4 - Storage Administration (VC) </li></ul></ul><ul><ul><li>Section 5 - Snapshot Volumes & Resignaturing </li></ul></ul><ul><ul><li>Section 6 - Storage VMotion </li></ul></ul><ul><ul><li>Section 7 - Volume Grow / Hot VMDK Extend </li></ul></ul><ul><ul><li>Section 8 - Storage CLI Enhancements </li></ul></ul><ul><ul><li>Section 9 - Other Storage Features/Enhancements </li></ul></ul>
  29. 29. GUI Changes - Display Device Info Note that there are no further references to vmhbaC:T:L. Unique device identifiers such as the NAA id are now used.
  30. 30. GUI Changes - Display HBA Configuration Info Again, notice the use of NAA ids rather than vmhbaC:T:L.
  31. 31. GUI Changes - Display Path Info Note the reference to the PSP & SATP Note the (I/O) status designating the active path
  32. 32. GUI Changes - Data Center Rescan
  33. 33. Degraded Status <ul><ul><li>If we detect less than 2 HBAs or 2 Targets in the paths of the datastore, we mark the datastore multipathing status as “ Partial/No Redundancy “ in the Storage Views . </li></ul></ul>
  34. 34. Storage Administration <ul><ul><li>VI4 also provides new monitoring , reporting and alarm features for storage management. </li></ul></ul><ul><ul><li>This now gives an administrator of a vSphere the ability to: </li></ul></ul><ul><ul><ul><li>Manage access/permissions of datastores/folders </li></ul></ul></ul><ul><ul><ul><li>Have visibility of a Virtual Machine’s connectivity to the storage infrastructure </li></ul></ul></ul><ul><ul><ul><li>Account for disk space utilization </li></ul></ul></ul><ul><ul><ul><li>Provide notification in case of specific usage conditions </li></ul></ul></ul>
  35. 35. Datastore Monitoring & Alarms <ul><ul><li>vSphere introduces new datastore and VM-specific alarms/alerts on storage events: </li></ul></ul><ul><ul><ul><li>New datastore alarms: </li></ul></ul></ul><ul><ul><ul><ul><li>Datastore disk usage % </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Datastore Disk Over allocation % </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Datastore connection state to all hosts </li></ul></ul></ul></ul><ul><ul><ul><li>New VM alarms: </li></ul></ul></ul><ul><ul><ul><ul><li>VM Total size on disk (GB) </li></ul></ul></ul></ul><ul><ul><ul><ul><li>VM Snapshot size (GB) </li></ul></ul></ul></ul>Customer’s can now track snapshot usage
  36. 36. New Storage Alarms New Datastore specific Alarms New VM specific Alarms This alarms allow the tracking of Thin Provisioned disks This alarms will trigger if a datastore becomes unavailble to the host This alarms will trigger if a snapshot delta file becomes too large
  37. 37. vSphere 4 Storage <ul><ul><li>Section 1 - Naming Convention Change </li></ul></ul><ul><ul><li>Section 2 - Pluggable Storage Architecture </li></ul></ul><ul><ul><li>Section 3 - iSCSI Enhancements </li></ul></ul><ul><ul><li>Section 4 - Storage Administration (VC) </li></ul></ul><ul><ul><li>Section 5 - Snapshot Volumes & Resignaturing </li></ul></ul><ul><ul><li>Section 6 - Storage VMotion </li></ul></ul><ul><ul><li>Section 7 - Volume Grow / Hot VMDK Extend </li></ul></ul><ul><ul><li>Section 8 - Storage CLI Enhancements </li></ul></ul><ul><ul><li>Section 9 - Other Storage Features/Enhancements </li></ul></ul>
  38. 38. Traditional Snapshot Detection <ul><ul><li>When an ESX 3.x server finds a VMFS-3 LUN, it compares the SCSI_DiskID information returned from the storage array with the SCSI_DiskID information stored in the LVM Header. </li></ul></ul><ul><ul><ul><li>If the two IDs don’t match, then by default, the VMFS-3 volume will not be mounted and thus be inaccessible. </li></ul></ul></ul><ul><ul><li>A VMFS volume on ESX 3.x could be detected as a snapshot for a number of reasons: </li></ul></ul><ul><ul><ul><li>LUN ID changed </li></ul></ul></ul><ul><ul><ul><li>SCSI version supported by array changed (firmware upgrade) </li></ul></ul></ul><ul><ul><ul><li>Identifier type changed – Unit Serial Number vs NAA ID </li></ul></ul></ul>
  39. 39. New Snapshot Detection Mechanism <ul><ul><li>When trying to determine if a device is a snapshot in ESX 4.0, the ESX uses a globally unique identifier to identify each LUN, typically the NAA (Network Addressing Authority) ID. </li></ul></ul><ul><ul><li>NAA IDs are unique and are persistent across reboots. </li></ul></ul><ul><ul><li>There are many different globally unique identifiers (EUI, SNS, T10, etc). If the LUN does not support any of these globally unique identifiers, ESX will fall back to the serial number + LUN ID used in ESX 3.0. </li></ul></ul>
  40. 40. SCSI_DiskId Structure <ul><ul><li>The internal VMkernel structure SCSI_DiskId is populated with information about a LUN. </li></ul></ul><ul><ul><li>This is stored in the metadata header of a VMFS volume. </li></ul></ul><ul><ul><li>if the LUN does have a globally unique (NAA) ID, the field in the SCSI_DiskId structure will hold it. </li></ul></ul><ul><ul><li>If the NAA ID in the stored in the metadata does not match the NAA ID returned by the LUN, the ESX knows the LUN is a snapshot. </li></ul></ul><ul><ul><li>For older arrays that do not support NAA IDs, the earlier algorithm is used where we compare other fields in the SCSI_DISKID structure to detect whether a LUN is a snapshot or not. </li></ul></ul>
  41. 41. <ul><li>8:00:45:51.975 cpu4:81258)ScsiPath: 3685: Plugin 'NMP' claimed path 'vmhba33:C0:T1:L2' </li></ul><ul><li>8:00:45:51.975 cpu4:81258)ScsiPath: 3685: Plugin 'NMP' claimed path 'vmhba33:C0:T0:L2' </li></ul><ul><li>8:00:45:51.977 cpu2:81258)VMWARE SCSI Id: Id for vmhba33:C0:T0:L2 </li></ul><ul><li>0x60 0x06 0x01 0x60 0x1d 0x31 0x1f 0x00 0xfc 0xa3 0xea 0x50 0x1b 0xed 0xdd 0x11 0x52 0x41 0x49 0x44 0x20 0x35 </li></ul><ul><li>8:00:45:51.978 cpu2:81258)VMWARE SCSI Id: Id for vmhba33:C0:T1:L2 </li></ul><ul><li>0x60 0x06 0x01 0x60 0x1d 0x31 0x1f 0x00 0xfc 0xa3 0xea 0x50 0x1b 0xed 0xdd 0x11 0x52 0x41 0x49 0x44 0x20 0x35 </li></ul><ul><li>8:00:45:52.002 cpu2:81258)LVM: 7125: Device naa.600601601d311f00fca3ea501beddd11:1 detected to be a snapshot: </li></ul><ul><li>8:00:45:52.002 cpu2:81258)LVM: 7132: queried disk ID: <type 2, len 22, lun 2, devType 0, scsi 0, h(id) 3817547080305476947> </li></ul><ul><li>8:00:45:52.002 cpu2:81258)LVM: 7139: on-disk disk ID: <type 2, len 22, lun 1, devType 0, scsi 0, h(id) 6335084141271340065> </li></ul><ul><li>8:00:45:52.006 cpu2:81258)ScsiDevice: 1756: Successfully registered device &quot;naa.600601601d311f00fca3ea501beddd11&quot; from plugin &quot; </li></ul>Snapshot Log Messages
  42. 42. Resignature & Force-Mount <ul><ul><li>We have a new naming convention in ESX 4. </li></ul></ul><ul><ul><ul><li>“ Resignature ” is equivalent to EnableResignature = 1 in ESX 3.x. </li></ul></ul></ul><ul><ul><ul><li>“ Force-Mount ” is equivalent to DisallowSnapshotLUN = 0 in ESX 3.x. </li></ul></ul></ul><ul><ul><li>The advanced configuration options EnableResignature and DisallowSnapshotLUN have been replaced in ESX 4 with a new CLI utility called esxcfg-volume ( vicfg-volume for ESXi) . </li></ul></ul><ul><ul><li>Historically, the EnableResignature and DisallowSnapshotLUN were applied server wide and applied to all volumes on an ESX. The new Resignature and Force-mount are volume specific so offer much greater granularity in the handling of snapshots. </li></ul></ul>
  43. 43. Persistent Or Non-Persistent Mounts <ul><ul><li>If you use the GUI to force-mount a VMFS volume, it will make it a persistent mount which will remain in place through reboots of the ESX host. VC will not allow this volume to be resignatured. </li></ul></ul><ul><ul><li>If you use the CLI to force-mount a VMFS volume, you can choose whether it persists or not through reboots. </li></ul></ul><ul><ul><li>Through the GUI , the Add Storage Wizard now displays the VMFS label. Therefore if a device is not mounted, but it has a label associated with it, you can make the assumption that it is a snapshot, or to use ESX 4 terminology, a Volume Copy . </li></ul></ul>
  44. 44. Mounting A Snapshot Original Volume is still presented to the ESX Snapshot – notice that the volume label is the same as the original volume.
  45. 45. Snapshot Mount Options <ul><ul><li>Keep Existing Signature – this is a force-mount operation: similar to disabling DisallowSnapshots in ESX 3.x. New datastore has original UUID saved in the file system header. </li></ul></ul><ul><ul><ul><li>If the original volume is already online, this option will not succeed and will print a ‘ Cannot change the host configuration ’ message when resolving the VMFS volumes.. </li></ul></ul></ul><ul><ul><li>Assign a new Signature – this is a resignature operation: similar to enabling EnableResignature in ESX 3.x. New datastore has a new UUID saved in the file system header. </li></ul></ul><ul><ul><li>Format the disk – destroys the data on the disk and creates a new VMFS volume on it. </li></ul></ul>
  46. 46. New CLI Command: esxcfg-volume <ul><ul><li>There is a new CLI command in ESX 4 for resignaturing VMFS snapshots. Note the difference between ‘-m’ and ‘-M’: </li></ul></ul><ul><ul><ul><ul><li># esxcfg-volume </li></ul></ul></ul></ul><ul><ul><ul><ul><li>esxcfg-volume <options> </li></ul></ul></ul></ul><ul><ul><ul><ul><li>-l|--list List all volumes which have been </li></ul></ul></ul></ul><ul><ul><ul><ul><li>detected as snapshots/replicas. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>-m|--mount <VMFS UUID|label> Mount a snapshot/replica volume, </li></ul></ul></ul></ul><ul><ul><ul><ul><li>if its original copy is not </li></ul></ul></ul></ul><ul><ul><ul><ul><li>online. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>-u|--umount <VMFS UUID|label> Umount a snapshot/replica volume. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>-r|--resignature <VMFS UUID|label> Resignature a snapshot/replica </li></ul></ul></ul></ul><ul><ul><ul><ul><li>volume. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>-M|--persistent-mount <VMFS UUID|label> Mount a snapshot/replica volume </li></ul></ul></ul></ul><ul><ul><ul><ul><li>persistently, if its original </li></ul></ul></ul></ul><ul><ul><ul><ul><li>copy is not online. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>-h|--help Show this message. </li></ul></ul></ul></ul>
  47. 47. esxcfg-volume (ctd) <ul><ul><li>The difference between a mount and a persistent mount is that the persistent mounts will be maintained through reboots. </li></ul></ul><ul><ul><li>ESX manages this by adding entries for force mounts into the /etc/vmware/esx.conf . </li></ul></ul><ul><ul><li>A typical set of entries for a force mount look like: </li></ul></ul><ul><ul><ul><li>/fs/vmfs[48d247dd-7971f45b-5ee4-0019993032e1]/forceMountedLvm/ forceMount = &quot; true &quot; </li></ul></ul></ul><ul><ul><ul><li>/fs/vmfs[48d247dd-7971f45b-5ee4-0019993032e1]/forceMountedLvm/ lvmName = &quot;48d247da-b18fd17c-1da1-0019993032e1&quot; </li></ul></ul></ul><ul><ul><ul><li>/fs/vmfs[48d247dd-7971f45b-5ee4-0019993032e1]/forceMountedLvm/ readOnly = &quot; false &quot; </li></ul></ul></ul>
  48. 48. Mount With the Original Volume Still Online <ul><li>/var/log # esxcfg-volume -l </li></ul><ul><li>VMFS3 UUID/label: 496f202f-3ff43d2e-7efe-001f29595f9d/Shared_VMFS_For_FT_VMs </li></ul><ul><li>Can mount: No (the original volume is still online) </li></ul><ul><li>Can resignature: Yes </li></ul><ul><li>Extent name: naa.600601601d311f00fca3ea501beddd11:1 range: 0 - 20223 (MB) </li></ul><ul><li>/var/log # esxcfg-volume -m 496f202f-3ff43d2e-7efe-001f29595f9d </li></ul><ul><li>Mounting volume 496f202f-3ff43d2e-7efe-001f29595f9d </li></ul><ul><li>Error: Unable to mount this VMFS3 volume due to the original volume is still online </li></ul>
  49. 49. esxcfg-volume (ctd) <ul><ul><li>In this next example, a clone LUN of a VMFS LUN is presented back to the same ESX server. Therefore we cannot use either the mount or the persistent mount options since the original LUN is already presented to the host so we will have to resignature: </li></ul></ul><ul><ul><ul><li># esxcfg-volume -l </li></ul></ul></ul><ul><ul><ul><li>VMFS3 UUID/label: 48d247dd-7971f45b-5ee4-0019993032e1/cormac_grow_vol </li></ul></ul></ul><ul><ul><ul><li>Can mount: No </li></ul></ul></ul><ul><ul><ul><li>Can resignature: Yes </li></ul></ul></ul><ul><ul><ul><li>Extent name: naa.6006016043201700f30570ed09f6da11:1 range: 0 - 15103 (MB) </li></ul></ul></ul>
  50. 50. esxcfg-volume (ctd) <ul><li># esxcfg-volume -r 48d247dd-7971f45b-5ee4-0019993032e1 </li></ul><ul><li>Resignaturing volume 48d247dd-7971f45b-5ee4-0019993032e1 </li></ul><ul><li># vdf </li></ul><ul><li>Filesystem 1K-blocks Used Available Use% Mounted on </li></ul><ul><li>/dev/sdg2 5044188 1595804 3192148 34% / </li></ul><ul><li>/dev/sdd1 248895 50780 185265 22% /boot </li></ul><ul><li>. </li></ul><ul><li>. </li></ul><ul><li>/vmfs/volumes/48d247dd-7971f45b-5ee4-0019993032e1 </li></ul><ul><li>15466496 5183488 10283008 33% /vmfs/volumes/cormac_grow_vol </li></ul><ul><li>/vmfs/volumes/48d39951-19a5b934-67c3-0019993032e1 </li></ul><ul><li>15466496 5183488 10283008 33% /vmfs/volumes/snap-397419fa-cormac_grow_vol </li></ul>Warning – there is no vdf command in ESXi. However the df command reports on VMFS filesystems in ESXi.
  51. 51. Ineffective Workarounds Under ESX 4 <ul><ul><li>Some of our customers have been running with DisallowSnapshotLUN set to ‘0’ as a workaround solution due their choice to use inconsistent LUN presentation numbers across their hosts since the early ESX 3.0.x days. </li></ul></ul><ul><ul><li>Other customers have been running with this setting after enabling the SPC-2 and SC-3 bits on their FA ports for their EMC DMX/Symmetrix array and found that their LUNs were now seen as snapshots. This was due to the LUN ID’s for their LUNs changing since we reference the LUN by a unique ID (in page 0x83) rather than the serial number for the LUN or the NAA (if running ESX 3.5) once these director bits are enabled. This behavior was also observed in other arrays when upgrading firmware/OS. </li></ul></ul><ul><ul><li>As there is no longer a global DisallowsnapshotLUN setting available in ESX 4, if your environment is running entirely on “snapshots” then you can use the following script to streamline the force-mount operation on each ESX host: </li></ul></ul><ul><li>for i in `/usr/sbin/esxcfg-volume -l|grep VMFS3|awk {print $3}`;do /usr/sbin/esxcfg-volume -M $i;done </li></ul>
  52. 52. vSphere Storage <ul><ul><li>Section 1 - Naming Convention Change </li></ul></ul><ul><ul><li>Section 2 - Pluggable Storage Architecture </li></ul></ul><ul><ul><li>Section 3 - iSCSI Enhancements </li></ul></ul><ul><ul><li>Section 4 - Storage Administration (VC) </li></ul></ul><ul><ul><li>Section 5 - Snapshot Volumes & Resignaturing </li></ul></ul><ul><ul><li>Section 6 - Storage VMotion </li></ul></ul><ul><ul><li>Section 7 - Volume Grow / Hot VMDK Extend </li></ul></ul><ul><ul><li>Section 8 - Storage CLI Enhancements </li></ul></ul><ul><ul><li>Section 9 - Other Storage Features/Enhancements </li></ul></ul>
  53. 53. Storage VMotion Enhancements In VI4 <ul><ul><li>The following enhancements have been made to the VI4 version of Storage VMotion: </li></ul></ul><ul><ul><ul><li>GUI Support. </li></ul></ul></ul><ul><ul><ul><li>Leverages new features of VI4, including fast suspend/resume and Change Block Tracking. </li></ul></ul></ul><ul><ul><ul><li>Supports moving VMDKs from Thick to Thin formats & vice versa </li></ul></ul></ul><ul><ul><ul><li>Ability to migrate RDMs to VMDKs. </li></ul></ul></ul><ul><ul><ul><li>Ability to migrate RDMs to RDMs. </li></ul></ul></ul><ul><ul><ul><li>Support for FC, iSCSI & NAS. </li></ul></ul></ul><ul><ul><ul><li>Storage VMotion no longer requires 2 x memory. </li></ul></ul></ul><ul><ul><ul><li>No requirement to create a VMotion interface for Storage VMotion. </li></ul></ul></ul><ul><ul><ul><li>Ability to move an individual disk without moving the VM’s home. </li></ul></ul></ul>
  54. 54. New Features <ul><ul><li>Fast Suspend/Resume of VMs </li></ul></ul><ul><ul><ul><li>This provides the ability to quickly transition between the source VM to the destination VM reliably and in a fast switching time . </li></ul></ul></ul><ul><ul><ul><ul><li>This is only necessary when migrating the .vmx file </li></ul></ul></ul></ul><ul><ul><li>Changed Block Tracking </li></ul></ul><ul><ul><ul><li>Very much like how we handle memory with standard VMotion in that a bitmap of changed disk blocks is used rather than a bitmap of changed memory pages. </li></ul></ul></ul><ul><ul><ul><li>This means Storage VMotion no longer needs to snapshot the original VM and commit it to the destination VM so the Storage VMotion operation performs much faster. </li></ul></ul></ul><ul><ul><ul><li>Multiple iterations of the disk copy goes through, but each time the number of disk blocks that changed reduces, until eventually all disk blocks have been copied and we have a complete copy of the disk at the destination. </li></ul></ul></ul>
  55. 55. Storage VMotion – GUI Support <ul><li>Storage VMotion is still supported via the VI CLI 4.0 as well as the API , so customers wishing to use this method can continue to do so. </li></ul><ul><li>The Change both host and datastore option is only available to powered off VMs. </li></ul>For a non-passthru RDM, you can select to convert it to either Thin Provisioned or Thick when converting it to a VMDK, or you can leave it as a non-passthru RDM.
  56. 56. Storage VMotion – CLI (ctd) <ul><li># svmotion --interactive </li></ul><ul><li>Entering interactive mode. All other options and environment variables will be ignored. </li></ul><ul><li>Enter the VirtualCenter service url you wish to connect to (e.g., or just </li></ul><ul><li>Enter your username: Administrator </li></ul><ul><li>Enter your password: ******** </li></ul><ul><li>Attempting to connect to </li></ul><ul><li>Connected to server. </li></ul><ul><li>Enter the name of the datacenter: Embedded-ESX40 </li></ul><ul><li>Enter the datastore path of the virtual machine (e.g. [datastore1] myvm/myvm.vmx): [CLAR_L52] W2K3SP2/W2K3SP2.vmx </li></ul><ul><li>Enter the name of the destination datastore: CLAR_L53 </li></ul><ul><li>You can also move disks independently of the virtual machine. If you want the disks to stay with the virtual machine, then skip this step.. </li></ul><ul><li>Would you like to individually place the disks (yes/no)? no </li></ul><ul><li>Performing Storage VMotion. </li></ul><ul><li>0% |----------------------------------------------------------------------------------------------------| 100% </li></ul><ul><li>########## </li></ul>
  57. 57. Limitations <ul><ul><li>The migration of Virtual Machines which have snapshots will not be supported at GA. </li></ul></ul><ul><ul><ul><li>Currently the plan is to have this in a future release </li></ul></ul></ul><ul><ul><li>The migration of Virtual Machines to a different host and a different datastore simultaneously is not yet supported. </li></ul></ul><ul><ul><ul><li>No firm date for support of this feature yet. </li></ul></ul></ul>
  58. 58. Storage VMotion Timeouts <ul><ul><li>There are also a number of tunable timeout values: </li></ul></ul><ul><ul><ul><li>Downtime timeout </li></ul></ul></ul><ul><ul><ul><ul><li>Failure: Source detected that destination failed to resume. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Update fsr.maxSwitchoverSeconds (default 100 seconds) in the VM’s .vmx file. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>May be observed on Virtual Machines that have lots of virtual disks. </li></ul></ul></ul></ul><ul><ul><ul><li>Data timeout </li></ul></ul></ul><ul><ul><ul><ul><li>Failure: Timed out waiting for migration data. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Update migration.dataTimeout (default 60 seconds ) in the VM’s .vmx file. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>May be observed when migrating from NFS to NFS on slow networks. </li></ul></ul></ul></ul>
  59. 59. vSphere Storage <ul><ul><li>Section 1 - Naming Convention Change </li></ul></ul><ul><ul><li>Section 2 - Pluggable Storage Architecture </li></ul></ul><ul><ul><li>Section 3 - iSCSI Enhancements </li></ul></ul><ul><ul><li>Section 4 - Storage Administration (VC) </li></ul></ul><ul><ul><li>Section 5 - Snapshot Volumes & Resignaturing </li></ul></ul><ul><ul><li>Section 6 - Storage VMotion </li></ul></ul><ul><ul><li>Section 7 - Volume Grow / Hot VMDK Extend </li></ul></ul><ul><ul><li>Section 8 - Storage CLI Enhancements </li></ul></ul><ul><ul><li>Section 9 - Other Storage Features/Enhancements </li></ul></ul>
  60. 60. Supported Disk Growth/Shrink Operations <ul><ul><li>VI4 introduces the following growth/shrink operations: </li></ul></ul><ul><ul><ul><li>Grow VMFS volumes: yes </li></ul></ul></ul><ul><ul><ul><li>Grow RDM volumes: yes </li></ul></ul></ul><ul><ul><ul><li>Grow *. vmdk : yes </li></ul></ul></ul><ul><ul><ul><li>Shrink VMFS volumes : no </li></ul></ul></ul><ul><ul><ul><li>Shrink RDM volumes : yes </li></ul></ul></ul><ul><ul><ul><li>Shrink *. vmdk : no </li></ul></ul></ul>
  61. 61. Volume Grow & Hot VMDK Extend <ul><ul><li>Volume Grow </li></ul></ul><ul><ul><ul><li>VI4 allows dynamic expansion of a volume partition by adding capacity to a VMFS without disrupting running Virtual Machines. </li></ul></ul></ul><ul><ul><ul><li>Once the LUN backing the datastore has been grown (typically through an array management utility), Volume Grow expands the VMFS partition on the expanded LUN. </li></ul></ul></ul><ul><ul><ul><li>Historically, the only way to grow a VMFS volume was to use the extent-based approach. Volume Grow offers a different method of capacity growth . </li></ul></ul></ul><ul><ul><ul><li>The newly available space appears as a larger VMFS volume along with an associated grow event in vCenter. </li></ul></ul></ul><ul><ul><li>Hot VMDK Extend </li></ul></ul><ul><ul><ul><li>Hot extend is supported for VMFS flat virtual disks in persistent mode and without any Virtual Machine snapshots. </li></ul></ul></ul><ul><ul><ul><li>Used in conjunction with the new Volume Grow capability, the user has maximum flexibility in managing growing capacity in VI4. </li></ul></ul></ul>
  62. 62. Comparison: Volume Grow & Add Extent Volume Grow Add Extent Must power-off VMs No No Can be done on newly-provisioned LUN No Yes Can be done on existing array-expanded LUN Yes Yes (but not allowed through GUI) Limits An extent can be grown any number of times, up to 2TB. A datastore can have up to 32 extents, each up to 2TB. Results in creation of new partition No Yes VM availability impact None, if datastore has only one extent. Introduces dependency on first extent.
  63. 63. Volume Grow GUI Enhancements Here I am choosing the same device on which the VMFS is installed – there is currently 4GB free. This option selects to expand the VMFS using free space on the current device Notice that the current extent capacity is 1GB.
  64. 64. VMFS Grow - Expansion Options LUN Provisioned at Array VMFS Volume/Datastore Provisioned for ESX Virtual Disk Provisioned for VM VMFS Volume Grow Dynamic LUN Expansion VMDK Hot Extend
  65. 65. Hot VMDK Extend
  66. 66. vSphere Storage <ul><ul><li>Section 1 - Naming Convention Change </li></ul></ul><ul><ul><li>Section 2 - Pluggable Storage Architecture </li></ul></ul><ul><ul><li>Section 3 - iSCSI Enhancements </li></ul></ul><ul><ul><li>Section 4 - Storage Administration (VC) </li></ul></ul><ul><ul><li>Section 5 - Snapshot Volumes & Resignaturing </li></ul></ul><ul><ul><li>Section 6 - Storage VMotion </li></ul></ul><ul><ul><li>Section 7 - Volume Grow / Hot VMDK Extend </li></ul></ul><ul><ul><li>Section 8 - Storage CLI Enhancements </li></ul></ul><ul><ul><li>Section 9 - Other Storage Features/Enhancements </li></ul></ul>
  67. 67. ESX 4.0 CLI <ul><ul><li>There have been a number of new storage commands introduced with ESX 4.0 as well as enhancements to the more traditional commands. </li></ul></ul><ul><ul><li>Here is a list of commands that have changed or have been added: </li></ul></ul><ul><ul><ul><li>esxcli </li></ul></ul></ul><ul><ul><ul><li>esxcfg-mpath / vicfg-mpath </li></ul></ul></ul><ul><ul><ul><li>esxcfg-volume / vicfg-volume </li></ul></ul></ul><ul><ul><ul><li>esxcfg-scsidevs / vicfg-scsidevs </li></ul></ul></ul><ul><ul><ul><li>esxcfg-rescan / vicfg-rescan </li></ul></ul></ul><ul><ul><ul><li>esxcfg-module / vicfg-module </li></ul></ul></ul><ul><ul><ul><li>vmkfstools </li></ul></ul></ul><ul><ul><li>This slide deck will only cover a few of the above mentioned. </li></ul></ul>
  68. 68. New/Updated CLI Commands(1) : esxcfg-scsidevs <ul><ul><li>The esxcfg-vmhbadevs command has been replaced by the esxcfg-scsidevs command. </li></ul></ul><ul><ul><li>To display the old VMware Legacy identifiers (vml), use: </li></ul></ul><ul><ul><ul><ul><li># esxcfg-scsidevs –u </li></ul></ul></ul></ul><ul><ul><li>To display Service Console devices: </li></ul></ul><ul><ul><ul><ul><li># esxcfg-scsidevs –c </li></ul></ul></ul></ul><ul><ul><li>To display all logical devices on this host: </li></ul></ul><ul><ul><ul><ul><li># esxcfg-scsidevs –l </li></ul></ul></ul></ul><ul><ul><li>To show the relationship between COS native devices (/dev/sd) and vmhba devices: </li></ul></ul><ul><ul><ul><li># esxcfg-scsidevs -m </li></ul></ul></ul><ul><ul><li>The VI CLI 4.0 has an equivalent vicfg-scsidevs for ESXi. </li></ul></ul>
  69. 69. esxcfg-scsidevs (ctd) <ul><ul><li>Sample output of esxcfg-scsidevs –l : </li></ul></ul><ul><ul><ul><li>naa.600601604320170080d407794f10dd11 </li></ul></ul></ul><ul><ul><ul><li>Device Type: Direct-Access </li></ul></ul></ul><ul><ul><ul><li>Size: 8192 MB </li></ul></ul></ul><ul><ul><ul><li>Display Name: DGC Fibre Channel Disk (naa.600601604320170080d407794f10dd11) </li></ul></ul></ul><ul><ul><ul><li>Plugin: NMP </li></ul></ul></ul><ul><ul><ul><li>Console Device: /dev/sdb </li></ul></ul></ul><ul><ul><ul><li>Devfs Path: /vmfs/devices/disks/naa.600601604320170080d407794f10dd11 </li></ul></ul></ul><ul><ul><ul><li>Vendor: DGC Model: RAID 5 Revis: 0224 </li></ul></ul></ul><ul><ul><ul><li>SCSI Level: 4 Is Pseudo: false Status: on </li></ul></ul></ul><ul><ul><ul><li>Is RDM Capable: true Is Removable: false </li></ul></ul></ul><ul><ul><ul><li>Is Local: false </li></ul></ul></ul><ul><ul><ul><li>Other Names: </li></ul></ul></ul><ul><ul><ul><li>vml.0200000000600601604320170080d407794f10dd11524149442035 </li></ul></ul></ul>Note that this is one of the few CLI commands which will report the LUN size
  70. 70. New/Updated CLI Commands(2): esxcfg-rescan <ul><ul><li>You now have the ability to rescan based on whether devices were added or device were removed. </li></ul></ul><ul><ul><li>You can also rescan the current paths and not try to discover new ones. </li></ul></ul><ul><ul><ul><ul><li># esxcfg-rescan -h </li></ul></ul></ul></ul><ul><ul><ul><ul><li>esxcfg-rescan <options> [adapter] </li></ul></ul></ul></ul><ul><ul><ul><ul><li>-a|--add Scan for only newly added devices. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>-d|--delete Scan for only deleted devices. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>-u|--update Scan existing paths only and update their state. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>-h|--help Display this message. </li></ul></ul></ul></ul><ul><ul><li>The VI CLI 4.0 has an equivalent vicfg-rescan command for ESXi . </li></ul></ul>
  71. 71. New/Updated CLI Commands(3): vmkfstools <ul><ul><li>The vmkfstools commands exists in the Service Console and VI CLI 4.0 </li></ul></ul><ul><ul><li>Grow a VMFS: </li></ul></ul><ul><ul><ul><li>vmkfstools –G </li></ul></ul></ul><ul><ul><li>Inflate a VMDK from thin to thick: </li></ul></ul><ul><ul><ul><li>vmkfstools –j </li></ul></ul></ul><ul><ul><li>Import a thick VMDK to thin: </li></ul></ul><ul><ul><ul><li>vmkfstools –i <src> -d thin </li></ul></ul></ul><ul><ul><li>Import a thin VMDK to thick: </li></ul></ul><ul><ul><ul><li>vmkfstools –i <src thin disk> -d zeroedthick </li></ul></ul></ul>
  72. 72. vSphere Storage <ul><ul><li>Section 1 - Naming Convention Change </li></ul></ul><ul><ul><li>Section 2 - Pluggable Storage Architecture </li></ul></ul><ul><ul><li>Section 3 - iSCSI Enhancements </li></ul></ul><ul><ul><li>Section 4 - Storage Administration (VC) </li></ul></ul><ul><ul><li>Section 5 - Snapshot Volumes & Resignaturing </li></ul></ul><ul><ul><li>Section 6 - Storage VMotion </li></ul></ul><ul><ul><li>Section 7 - Volume Grow / Hot VMDK Extend </li></ul></ul><ul><ul><li>Section 8 - Storage CLI Enhancements </li></ul></ul><ul><ul><li>Section 9 - Other Storage Features/Enhancements </li></ul></ul>
  73. 73. Other Storage Features/Enhancements <ul><ul><li>Storage General </li></ul></ul><ul><ul><ul><li>The number of LUNs that can be presented to the ESX 4.0 server is still 256 . </li></ul></ul></ul><ul><ul><li>VMFS </li></ul></ul><ul><ul><ul><li>The maximum extent volume size in VI4 is still 2TB . </li></ul></ul></ul><ul><ul><ul><li>Maximum number of extents is still 32, so maximum volume size is still 64TB . </li></ul></ul></ul><ul><ul><ul><li>We are still using VMFS3, not VMFS 4 (although the version has increased to 3.33). </li></ul></ul></ul><ul><ul><li>iSCSI Enhancements </li></ul></ul><ul><ul><ul><li>10 GbE iSCSI Initiator – iSCSI over a 10GbE interface is supported. </li></ul></ul></ul><ul><ul><ul><ul><li>First introduced in ESX/ESXi 3.5 u2 & extended back to ESX/ESXi 3.5 u1. </li></ul></ul></ul></ul>
  74. 74. Other Storage Features/Enhancements (ctd) <ul><ul><li>NFS Enhancements </li></ul></ul><ul><ul><ul><li>IPv6 support (experimental) </li></ul></ul></ul><ul><ul><ul><li>Support for up to 64 NFS volumes (the old limit was 32) </li></ul></ul></ul><ul><ul><ul><li>10 GbE NFS Support – NFS over a 10GbE interface is supported. </li></ul></ul></ul><ul><ul><ul><ul><li>First introduced in ESX/ESXi 3.5 u2 </li></ul></ul></ul></ul><ul><ul><li>FC Enhancements </li></ul></ul><ul><ul><ul><li>Support for 8Gb Fibre Channel </li></ul></ul></ul><ul><ul><ul><ul><li>First introduced in ESX/ESXi 3.5 u2 </li></ul></ul></ul></ul><ul><ul><ul><li>Support for FC over Ethernet (FCoE) </li></ul></ul></ul>
  75. 75. Other Storage Features/Enhancements (ctd) <ul><ul><li>Paravirtualized SCSI Driver (Guest OS) </li></ul></ul><ul><ul><ul><li>Eliminates the need to trap privileged instructions as it uses hypercalls to request that the underlying hypervisor execute those privileged instructions . </li></ul></ul></ul><ul><ul><ul><ul><li>Handling unexpected or unallowable conditions via trapping can be time-consuming and can impact performance. </li></ul></ul></ul></ul>
  76. 76. Other Storage Features/Enhancements (ctd) ESX 3.x boot time LUN selection – which sd device represents an iSCSI disk and which represents an FC disk?
  77. 77. Other Storage Features/Enhancements (ctd) ESX 4.0 boot time LUN selection. Hopefully this will address incorrect LUN selections during install/upgrade.
  78. 78. Questions? Nathan Small Staff Engineer [email_address]

Editor's Notes

  • New storage acronyms for vSphere 4.0
  • The C in vmhbaN:C:T:L:P is for channel. We only see a channel 0 in FC &amp; iSCSI typically. Other identifiers might include a T10 name, as observed with the OpenFiler appliance. We’re moving away from the Controller Target Lun Partition and using more NAA id. MPX Multipath X device = unknown (cdroms, some local storage types too)
  • The SDK for third parties to write their own code for the PSA is called the VMKDH. This eliminates the recertification issue with the storage vendors. EMC will be first for GA (Powerpath). Remember that the PSA and the Multi Pathing Plugin (MPP) are just replacing the tasks that were carried out by the SCSI Mid Layer in previous ESX versions.
  • The multipath plug-ins are VMkernel modules, e.g. NMP Advanced functions like the quantumSched – are now apart of the PSA. Such as how much time VM’s can run on the run-queue. Note that this is still SCSI mid layer stuff, except we now split the operations between the PSA &amp; the MPP.
  • Multi Path Plugin (MPP) Point 2: PSA discovers IBM on 2 paths and EMC on 4 Paths and determins which MPP to give to the plugin. Collaps the 4 paths for one array and notes it as one array
  • NMP Native Multipathing Plugin
  • 3 rd point: At this point in time, there are no known partners working on SATPs.
  • The esxcli command also appears in the RCLI for VI4.
  • The new Pluggable Storage Architecture (PSA) also allows for third party plug-ins to take control of the entire path failover and load balancing operations, replacing VMware’s NMP. Also in ESX 4, there are two ways to set a PSP for a device: 1. By setting the default PSP for a given SATP in a claim rule. 2. By setting the managing PSP for a given device, like so: # esxcli nmp device setpolicy --device naa.50060160ffe0000150060160ffe00001 --psp PSP_RR There is currently no way to set PSP&apos;s by vendor/model strings. As for config options, there is currently no way to provide PSP-wide config options in a claimrule the way there is for a SATP. However, the per-path and per-device configuration you set using esxcli, such as: # esxcli nmp psp setconfig --device naa.50060160ffe0000150060160ffe00001 --config iops=213 are persistent across reboots, so you only need to set it up once per device. Actually, once all your devices are claimed, you can just make shell script to set the per-device configuration for all your devices.
  • Round Robin is the only Path Selection Policy which load balances. It has been around in experimental form since ESX 3.0 but is finally supported in ESX 4.0. It only uses active paths so is of most use on Active/Active arrays. However in Active/Passive arrays, it will load balance between on ports to the same Storage Processor.
  • &amp;quot;mpx&amp;quot; is a VMware specific namespace. It is not an acronym - it could roughly stand for &amp;quot;Mult Pathing x&amp;quot;. The mpx name space is used when no other valid namespace can be obtained from the LUN, such as NAA. It is not globally unique and is not persistent across reboots. Typically only local devices will not have NAA, IQN, etc namespaces and so have names starting with &amp;quot;mpx.&amp;quot;. The Storage Array Type Device Config: {navireg ipfilter} config settings are specific to the SATP_CX, SATP_INV, and SATP_ALUA_CX. The accepted values are: navireg_on, navireg_off, ipfilter_on, ipfilter_off            navireg_on starts automatic registration of the device with Navisphere           navireg_off stops the automatic registration of the device           ipfilter_on stops the sending of the host name for Navisphere registration,  used if host is known as &amp;quot;localhost&amp;quot;           ipfilter_off  enables the sending of the host name during Navisphere registration # esxcli nmp satp getconfig -d naa.60060160432017003461b060f9f6da11 {navireg ipfilter}
  • This following slides are collected in the vm-support dumps
  • The vicfg-mpath command also takes --username &amp; --password as arguments.
  • For FC arrays, this will show World Wide Names (WWNs)
  • To my knowledge, there is only one MPP under development at present (Sep 2008), and that is EMC Powerpath version 5.4.
  • No one is currently develponing SATP’s or PSP’s but this is the framework for new arrays so they don’t have to be qualified by VMware. Third party PSP and SATP’s are to be treated like third party software and can be unloaded in order to troubleshoot. and a reboot is required after unloading the module.
  • Port binding is a mechanism of identifying certain VMkernel ports for use by the iSCSI storage stack . Port binding is necessary to enable storage multipathing policies, such as the VMware round-robin load-balancing, or MRU, or fixed-path, to apply to iSCSI NIC ports and paths. Port binding does not work in combination with IPV6. When users configure port binding they expect to see additional paths for each bound VMkernel NIC. However, when they configure the array under a IPV6 global scope address the additional paths will not be established. Users only see paths established on the IPV6 routable VMkernel NIC. For instance if users have two target portals and two VMkernel NICs, they see four paths when using IPv4. They see only two paths when using IPv6. Because there are no paths for failover, path policy setup does not make sense. Workaround: Use IPV4 and port binding, or configure the storage array and the ESX/ESXi host with LOCAL SCOPE IPv6 addresses on the same subnet (switch segment). You cannot currently use global scope IPv6 with port binding.
  • Taken from the SAN Compatibility Guide
  • Animation: Click 1 – Highlight the Storage Link in Hardware Configuration Click 2 – Highlight the datastores available on this host Click 3 – Highlight the details for a particular datastore Click 4 – NAA id banner launches - NAA – Network Address Authority
  • Animation: Click 1 – Highlight the Storage Adapter Link in Hardware Configuration Click 2 – Highlight one particular adapter available on this host, in this case vmhba33. Note the NMP on the bottom right
  • Animation: Click 1: Select a datastore on your ESX, click on the Properties link Click 2: This launches the VMFS properties window Click 3: Highlight the ‘Manage Paths’ button which, when clicked, launches the Manage Path window Click 4: The Manage Path window opens which displays the PSP, SATP and other pathing info used by these paths and LUN.
  • Animation for Datacenter wide rescan: Click 1: Select your DataCenter, right click and select Rescan for Datastores Click 2: Acknowledge that the rescan may take a long time Click 3: Scan for either New Storage devices, New VMFS volumes or both Click 4: Notice the Rescan tasks for each ESX appearing in the Recent Tasks window in vCenter
  • Identifiers that are persistent and globally unique:     t10     eui     naa     rtp1     tpg     lug     md5     sns This started in 3.5 where we started using the NAA id to identify each LUN.
  • This output was achieved by creating a real snapshot of an EMC Clariion LUN which has a VMFS volume and presented it back to the same ESX. I tried changing the original LUN’s ID but the ESX handled this without a problem since we reference the LUN by the NAA.
  • We have got rid of DisallowSnapshotLUN but we retain the EnableResignature purely for supporting SRM. If you do use the GUI, you need to be aware that if you use the CLI utility, or you use a VI Client to directly attach to a host, these do not get reflected automatically in VC. This causes an issue with customers whose environment is running with DisallowSnapshotLUN = 0. There is a solution for this covered in the coming slides.
  • The equivalent RCLI for ESXi is vicfg-volume
  • Note that after you have unmounted the persistently mounted LUN, these entries stay in the esx.conf except that the forceMount option is set to false . /fs/vmfs[48d247dd-7971f45b-5ee4-0019993032e1]/forceMountedLvm/forceMount = &amp;quot;false&amp;quot; /fs/vmfs[48d247dd-7971f45b-5ee4-0019993032e1]/forceMountedLvm/lvmName = &amp;quot;48d247da-b18fd17c-1da1-0019993032e1&amp;quot; /fs/vmfs[48d247dd-7971f45b-5ee4-0019993032e1]/forceMountedLvm/readOnly = &amp;quot;false&amp;quot; Readonly is for NFS volumes, but it will not be displayed in the UI.
  • It will let you know if it can be resignatured, and if it resides or is visibile to the same ESX host. It will not allow you to mount the snapshot if you try.
  • Attempting to mount a snapshot volume on the same ESX as the original results in the following message in /var/log/vmkernel: Oct  1 18:11:00 cs-tse-f116 vmkernel: 4:05:33:23.794 cpu5:4111)LVM: MountSnapshotVolume:9536: Volume 48d247da-b18fd17c-1da1-0019993032e1 is already mounted This appears if attempting the operation via the GUI or CLI. The GUI doesn’t show any error or event if this operation is attempted – one simply sees the volume not becoming available. From the CLI, the following is observed: root@cs-tse-f116 ~]# esxcfg-volume -l VMFS3 UUID/label: 48d247dd-7971f45b-5ee4-0019993032e1/cormac_grow_vol Can mount: Yes Can resignature: Yes Extent name: naa.6006016043201700f30570ed09f6da11:1     range: 0 - 15103 (MB) [root@cs-tse-f116 ~]# esxcfg-volume -m 48d247dd-7971f45b-5ee4-0019993032e1 Mounting volume 48d247dd-7971f45b-5ee4-0019993032e1 Error: SysinfoException: Node (5015) ; Status(bad0007)= Bad parameter; Message= Module: lvmdriver Instance(0): Input(3) 48d247da-b18fd17c-1da1-0019993032e1 rw naa.6006016043201700f30570ed09f6da11:1 [root@cs-tse-f116 ~]# esxcfg-volume -M 48d247dd-7971f45b-5ee4-0019993032e1 Persistently mounting volume 48d247dd-7971f45b-5ee4-0019993032e1 Error: SysinfoException: Node (5015) ; Status(bad0007)= Bad parameter; Message= Module: lvmdriver Instance(0): Input(3) 48d247da-b18fd17c-1da1-0019993032e1 rw naa.6006016043201700f30570ed09f6da11:1 [root@cs-tse-f116 ~]# esxcfg-volume -r 48d247dd-7971f45b-5ee4-0019993032e1 Resignaturing volume 48d247dd-7971f45b-5ee4-0019993032e1
  • I am working with our engineering team on a permanent fix
  • The first point regarding GUI support is important since previously, SVMotion operations could only be initiated via the Remote CLI in ESX 3.5. The final point, the ability to move an individual disk without moving the VM’s home was not available in previous versions of SVMotion in ESX 3.5. SVMotion of power on VM with RDM preserves RDM Cold migration of powered off VM with RDM still converts RDM to flat VMDK file
  • This works with non-pass thru RDMs only. Cold migration of an RDM when the VM is powered off converts it to a VMDK. If the VM is powered on, it preserved the RDM. You can also use the GUI to do Storage VMotions on older ESX 3.x hosts, but this continues to use the old snapshot mechanism.
  • There is still no plan to include an svmotion executable on the Service Console of the ESX.
  • Previously with RDMs, we had to remove the mapping file and remake it – now this is updated ‘on-the-fly’
  • VM availability impact refers to what happens to availability if we use the feature. With Volume Grow, availability is no different. With extents, there is a dependency on the first extent, so that if the first extent goes, they all go, i.e. we loose access to the whole volume.
  • This option first became available in ESX/ESXi 3.5 U2. There is currently no shrinking capability for a VMDK.
  • There should be esxcli and esxcfg-volumes RCLI commands in RC &amp; GA….
  • esxcfg-mpath used to show the size of the LUN in earlier ESX versions. It does not in ESX 4.0. This is why the output above is useful if you wish to identify a LUN based on size.
  • vmkfstools –G takes the same partition as both the source and destination – i.e. you are growing this partition with the same partition.
  • There is no VMFS upgrade to worry about. Older versions will stay as they are. Newly formatted volumes from ESX4.x boxes will get the new version. Newly formatted volumes from ESX3.x.x boxes will get the old version. All VMFS-3.xx versions are supported by all ESX3.x.x hosts. The 2 enhancements that lead us from 3.31 to 3.33 are both applicable to the yet unreleased VMFS-4, so nothing to report here. The following is true though (as has always been, since we throw in optimizations with every ESX release): ESX4.x.x boxes will work more efficiently on any given VMFS-3.xx volume than ESX3.x.x boxes working on the same volume.
  • The new in-guest virtualization-optimized SCSI driver has been developed to combat performance competition with hyper-V. Paravirtual SCSI adapters are supported on the following guest operating systems: Windows 2008 Windows 2003 Red Hat Linux (RHEL) 5 The following features are not supported with Paravirtual SCSI adapters: Boot disks Record/Replay Fault Tolerance MSCS Clustering