Lxp storage iSCSI Best PracticePresentation Transcript
Storage LunchCédric MégrozArchitect22.11.2012
Agenda Why IP Storage IP storage NFS iSCSI SMB3 iSCSI concept iSCSI architecture iSCSI name hba or software iscsi iSCSI dependant independant hardware iSCSI Configuration Binding or not binding that is the question Stacked or not stacked that is the question Masking on the array with iqn Multipathing, Policy & Pluggable Storage Architecture VMFS 5 Miscellaneous ALUA Jumbo Frame Delay Ack Equallogic Equallogic configurations Host Software for VMware Host Software for Windows Questions ?
Why IP Storage• Low cost infrastructure with 1 GB Ethernet• No switch and special connectivity to implement it• For small infrastructure with less 50 VMs• BUT not so easy to implement (a lot of best practice to find perf and throughput)
IP Storage NFS iSCSI SMB3• Two storage family• Block (SCSI, iSCSI, FC, FCoE, AoE)• NAS Network Attached storage ( NFS, SMB3 => Hyper-V)
Storage with VMwareVMwarevSphere®ESXi™ hostsDatastore VMware vSphere® VMFS NFStypes file systemStorage direct Fibretechnology FCoE iSCSI NAS attached Channel
iSCSI – What is it? • iSCSI specifies a way to “encapsulate” SCSI commands in a TCP/IP network connection: IP TCP iSCSIGigabit Header Header SCSI commands and data HeaderEthernet Explains how to extract SCSI commands and data Provides information necessary toProvides Link guarantee deliveryAddressing, largeframe sizes, and Contains “routing” informationflow control so the message can find its way through the network
iSCSI concept• Internet Small Computer Systems Interface (iSCSI)• Physical network isolation (dedicated sw) Logical network isolation (VLAN)• Default Port Number: 3260• An optional CHAP Secret (e.g., "secretsarefun") do not use for masking• LUN (Logical unit number)
ISCSI nameiSCSI target name:iqn.1992-08.com.mycompany:stor1-47cf3c25oreui.fedcba9876543210iSCSI alias: stor1IP address: 192.168.36.101 iSCSI initiator name: iqn.1998-01.com.vmware:train1-64ad4c29 or eui.1234567890abcdef iSCSI alias: train1 IP address: 192.168.36.88
HBA or software iSCSI initiator
Network software iSCSI Configuration• A VMkernel port must be created for ESXi to access software iSCSI. – The same port can be used to access NAS/NFS storage.• To optimize your vSphere networking setup: Separate iSCSI networks from NAS/NFS networks. • Physical separation is preferred. • If physical separation is not possible, use VLANs.
iSCSI Target-Discovery Methods• Two discovery methods are supported: • Static • Dynamic (also known as SendTargets) By default Dynamic This functionality is used by Equallogic to redirect the host to 192.168.36.101:3260 the right target
iSCSI Security: CHAP• iSCSI initiators use CHAP for Software iSCSI properties > General tab authentication purposes. • By default, CHAP is not configured.• ESXi supports two types of CHAP authentication: • Unidirectional • Bidirectional: – Software iSCSI only Target authenticates host.• CHAP is not masking !!!! Host authenticates target.
Multipathing with iSCSI• Hardware iSCSI: • Use two or more hardware iSCSI adapters.• Software or dependent hardware iSCSI: • Use multiple network interface cards (NICs). • Connect each NIC to a separate VMkernel port. • Associate VMkernel ports with iSCSI initiator.
Configuration iSCSI 1/4• Create a dedicated vSwitch with 2 VMKernel and 2 VMNIC connected on the IP storage network• If same subnet dedicate a VMNIC per VMKernel• If not do nothing
Configuration iSCSI 2/4• vmk_iscsi1 with VMNIC5• vmk_iscsi2 with VMNIC4• Idem for DVS
Configuration iSCSI 3/4
Configuration iSCSI 4/4
Binding or not binding VMKernel on iSCSI sw
Binding or not binding VMKernel on iSCSI sw• Rescan times take longer than usual Incorrect• number of paths are seen per device• Unable to see any storage from the storage device• CISCO NEXUS with LACP impose this config. with AA array without ALUA
Stacked or not stacked that is the questionThe architecture of the array can have twodevelopementMulti target: for each Ethernet port you havea iSCSI target => do not stack the switch likethe FC word. (HDS or VNX)Uni target: only one target is generated forall the Ethernet port => stack the switch tohave a failover in case of lost connectivity(NetApp, Equallogic)
Dedicated network or not …For small network yes it’s betterThink IOS updateWith 10GB and Nexus… no choiceGeneraly network infrastructure isstacked and it’s not a good thinks in caseof lost of the master (old switch 15 secwithout network)
What is Asymmetric Logical Unit(ALUA)?ALUA Allows for paths to beprofiledActive (optimized)Active (non-optimized)
Le VMFS 5• VMFS-5: • Allows concurrent access to shared storage • Can be dynamically expanded • Uses a 1MB block size, good for storing large virtual disk files • Uses subblock addressing, good for storing small files: – The subblock size is 8KB. • Provides on-disk, block-level locking host host VMFS datastore
VAAI• vStorage APIs for Array Integration is a feature introduced ESX/ESXi 4.1 that provides hardware acceleration functionality.• It enables your host to offload specific virtual machine and storage management operations to compliant storage hardware.• Used for lock file, clone & copie, svMotion• NFS is supported with a ESXi agent• Need license Ent & Ent +
Datastore size• Updating the metadata of VMFS requires a SCSI reserve and release• 1. Starting and stopping a VM• 2. vMotioning a VM• 3. Using VMware snapshots• 4. Using vAPI Dataprotection (use VMware snapshots)• . => 15 VMs / block datastore• VMware Best Practice 300 – 700 GB• But actually with VAAI => no limit• Use Large capacity datastores Avoid RDMs• NFS: 16TB• VMFS: vSphere 5 = 64TB• vSphere 4 = 2TB• Avoid extents
Miscellaneous• Never route network IP storage !!! Warning with the loop !!!!• Ask for compliant subnets with the Network team• Use always the vCenter Plug-In to apply the best practice on the ESXi host• Use always the vCenter Plug-In to create and mount the datastore
Jumbo Frame• What is an Ethernet Jumbo Frame? Ethernet frames with more than 1500 bytes of payload (9000 is common – FCoE is 2240)• Commonly ‘thought of’ as having better performance• Should I use Jumbo Frames?• Adds complexity & performance gains (while existent) are relatively marginal with common block sizes• Stick with the defaults when you can
Delayed Ack mandatory (EMC et HDS hyper-V)TCP delayed acknowledgment is a technique used by some implementations ofthe Transmission Control Protocol in an effort to improve network performance. Inessence, several ACK responses may be combined together into a singleresponse, reducing protocol overhead. However, in some circumstances, thetechnique can reduce application performance.http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002598http://support.microsoft.com/kb/981482http://www.axiomdynamics.com/it-tweaks-and-tricks/iSCSI-best-practices/html/iSCSI-Best-Practices-VMware-EMC-Microsoft.html
Advanced network configurations• Enable Rapid spanning-tree (portfast)• Enable flow control on all ports in the switch and array• Disable port storm control (broadcast- unicast)• Enable jumbo frames if possible
Alignment is Optimal I/OMisalignment of filesystems results in additional work on storage controller tosatisfy IO requestAffects every protocol, and every storage array VMFS & NFS DatastoresVMDKs & RDMs with NTFS, EXT3, LUN, etcFilesystems exist in the datastore and VMDK Create VMFS with GUI not vCLI Windows NT, 2000, 2003, & XP (use diskpart to set to 1MB) Linux (use fdisk expert mode and align on 2048 = 1MB)
Fixing MisalignmentGuest OS filesystem is misaligned• Step 1: Take an array snapshot/backup• Step 2: Use offline tools to realign EMC UBERAlign (open, works with all, scheduler, and in- guest reclaim) vSphere Migrator Alternate: Use online tool to align NetApp Migrate & Optimize (VSC feature)
SIOC NIOC SDRS• QoS on storage and network dVS• NIOC offert QoS compatible 802.1p for Virtual machine traffic, Management traffic, iSCSI traffic, NFS traffic, Fault- tolerant traffic, VMware vMotion traffic, User-defined traffic, vSphere replication traffic• SIOC use queue depth to manage contention. SIOC monitors the latency of I/Os to datastores at each ESX host sharing that device. When the average normalized datastore latency exceeds a set threshold (30ms by default), the datastore is considered to be congested, and SIOC kicks in to distribute the available storage resources to virtual machines in proportion to their shares.• Need license Ent+ …
iSCSI Storage arrayiSCSI Storage Switch Multi target Jumbo Frame 10GB ALUA array stackedEMC VNX no yes yes yes yes depend Need differentNetApp subnet yes yes FC onlyHDS HUS no yes 1GB ? yes AAHDS AMS no yes 1GB no Only AMS AA 2500Equallogic yes no yes yes -
Equallogic• Can be combined with other EqualLogic PS Series arrays in the same SAN group, online. Up to 16 total members per group• Automatic RAID 5, RAID 6, RAID 10 and RAID 50• 1 mgmt target per SAN Group but each stack has these own tragets.• The mgmt target redirect to the target with the LUN• Need to use de MPIO of Dell Equallogic for Windows
Equallogic configurations• There are three load balancers that operate within a pool:• The NLB (Network Load Balancer) manages the assignment of individual iSCSI connections to Ethernet ports on the pool members• The CLB (Capacity Load Balancer) manages the utilization of the disk capacity in the pool• The APLB (Automatic Performance Load Balancer) manages the distribution of high I/O data within the pool.
Host Software for VMware• Host Integration Tools for VMware®• Auto-Snapshot Manager/VMware Edition (ASM/VE): Integrates with VMware VirtualCenter and PS Series snapshots to allow administrators to enable Smart Copy protection of VirtualCenter folders, datastores, and virtual machines.• EqualLogic Datastore Manager: Allows administrators to create and manage datastores on EqualLogic PS Series arrays from within vCenter.• Virtual Desktop Deployment Utility: Automates the deployment of virtual desktops in a VMware View™ environment.• Storage Adapter for Site Recovery Manager (SRM): Allows SRM to understand and recognize PS Series replication for full SRM integration.• Multipathing Extension Module for VMware® vSphere™: Provides connection awareness enhancements to the existing VMware multipathing functionality that understands PS Series network load balancing and facilitates host connections to PS Series volumes.
Host Software for Windows• Host Integration Tools for Microsoft®• Remote Setup Wizard (RSW): Initializes new PS Series arrays, configures host connections to PS Series SANs, and configures and manages multipathing.• Multipath I/O Device Specific Module (MPIO DSM): Includes a connection awareness-module that understands PS Series network load balancing and facilitates host connections to PS Series volumes.• VSS and VDS Provider Services: Allows 3rd party backup software vendors to perform off-host backups.• Auto-Snapshot Manager/Microsoft Edition (ASM/ME): Provides point-in- time SAN protection of critical application data using PS Series snapshots, clones, and replicas of supported applications such as SQL Server®, Exchange Server®, Hyper-V™, and NTFS file shares.• PowerShell Tools: Provides a comprehensive set of PowerShell cmdlets for managing one or many PS Series groups.
linkshttp://www.vmware.com/files/pdf/techpaper/vmware-multipathing-configuration-software-iSCSI-port-binding.pdfhttp://blogs.vmware.com/vsphere/2011/08/vsphere-50-storage-features-part-12-iscsi-multipathing-enhancements.htmlbinding or nothttp://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2038869PSAhttp://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1011375Equallogichttp://docs.danielkassner.com/dell/ISCSI_optimization_EQL.pdfiSCSI best practicehttp://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002598http://support.microsoft.com/kb/981482http://www.axiomdynamics.com/it-tweaks-and-tricks/iSCSI-best-practices/html/iSCSI-Best-Practices-VMware-EMC-Microsoft.htmlNetApp & EMC VMworld session Best Practicehttp://download3.vmware.com/vmworld/2012/top10/sto2980.pdf