IBM XIV Gen3 Storage System

IBM XIV Gen3 Storage System
Storage built for VMware vSphere infrastructures

Peter Kisich
IBM Systems and Technology Group ISV Enablement
October 2011

© Copyright IBM Corporation, 2011

Table of contents
Abstract........................................................................................................................................1
Introduction .................................................................................................................................1
New in the IBM XIV Gen3 Storage System................................................................................1
Key integration with vSphere.....................................................................................................2
VAAI..............................................................................................................................................3
Full copy................................................................................................................................................... 3
How effective is the offload process? ...................................................................................................... 4
Hardware-assisted locking (atomic test and set)..................................................................................... 4
Block zeroing (write same) ...................................................................................................................... 6
vStorage APIs for Array Awareness..........................................................................................6
IBM System Storage Plug-in for vCenter ..................................................................................7
Storage Replication Adapter (SRA) ...........................................................................................8
Snapshot management in vSphere .........................................................................................10
QoS options...............................................................................................................................11
Summary....................................................................................................................................12
Appendix A: Ordering options for the XIV Gen3 Storage System........................................13
Appendix B: SAN ordering considerations ............................................................................16
Resources..................................................................................................................................18
About the author .......................................................................................................................18
Trademarks and special notices..............................................................................................19


Abstract
This paper provides information about integration, feature description, and ordering considerations for
the IBM XIV Gen3 Storage System for VMware vSphere 4 and 5 infrastructures.

Introduction
The IBM® XIV® Gen3 Storage System employs an advanced auto-balancing architecture built for today’s
dynamic data centers with an eye towards tomorrow. With industry-leading storage software and a high-
speed InfiniBand® grid, the XIV Gen3 system delivers storage features and required performance
demanded in VMware infrastructures including:
 VMware vStorage APIs for Array Integration (VAAI)
 vStorage APIs for Storage Awareness (VASA) integration
 Storage Replication Adapter for Site Recovery Manager
 Centralized management in vCenter
 Quality of service (QoS) features for mixed workload priorities
 Automation and simplicity for virtual infrastructures
 Co-development for VMware vSphere 5, and beyond

The global partnership with IBM and VMware, coupled with the forward-thinking architecture of the XIV
Gen3, provides a solid foundation for virtual infrastructures today and in the future.

New in the IBM XIV Gen3 Storage System
All data modules in the XIV Gen3 Storage System are interconnected with a high speed, low latency
InfiniBand fabric setting a new standard of performance and response time. The XIV Gen3 system then
automates the storage configuration of the high performance architecture to ensure that every disk
allocated to VMware will have optimal performance.

Infiniband Switching

Figure 1: Data stores optimally distributed across all spindles, processors, and cache

1

The XIV Gen3 Storage System from IBM provides an advanced grid architecture that is built for dynamic
virtual infrastructures. All volumes allocated to VMware are balanced across all processors, spindles and
cache in the XIV system. Automated data distribution by the XIV Gen3 system provides a perfectly
balanced configuration that avoids hot spots and ensures maximum performance.
Also, new in the XIV Gen3, is the introduction of solid-state drive (SSD) slots that can enable an optional
field upgrade of 7.5 TB of secondary cache in 2012. This secondary cache can provide an automated
staging area for frequently accessed data improving performance for even the highest transactional and
database workloads.

Key integration with vSphere
Along with the great synergy provided by the enhanced XIV Gen3 architecture, IBM provides integrated
software for vSphere for every XIV system such as:
 VAAI
 vStorage APIs for Storage Awareness
 IBM Storage Management Console for VMware vCenter
 Storage Replication Adapter (SRA) for Site Recovery Manager
 Snapshot management in vCenter
 QoS for mixed workload priorities


2

VAAI
Virtualization administrators look for ways to improve scalability, performance, and efficiency of their
vSphere infrastructure. One way is by utilizing storage integration with VMware VAAI. VAAI is a set of
APIs or primitives that allow vSphere infrastructures to offload processing of data-related tasks, which can
burden a VMware ESX server. Utilizing a storage platform with VAAI enabled, similar to the IBM XIV
Storage System, can provide significant improvements in vSphere performance, scalability, and
availability.
Adding the VAAI driver for XIV enables the following primitives:
 Full copy (also known as hardware copy offload):
− Benefit: Considerable boost in system performance and fast completion of copy
operations; minimizes host processing and network traffic
 Hardware-assisted locking (also known as atomic test and set): Replacement of the
SCSI-2 lock/reservation in Virtual Machine File System (VMFS)
− Benefit: Significantly improves scalability and performance
 Block zeroing (also known as write same)
− Benefit: Reduces the amount of processor effort, and input/output operations per
second (IOPS) required to write zeroes across an entire EagerZeroedThick (EZT)
Virtual Machine Disk (VMDK)

XIV system now provides full support for VAAI. The following sections describe each of these primitives.

Full copy
Tasks such as Virtual Machine (VM) provisioning and VM migration are part of everyday activities of most
VMware administrators. As the virtual environment continues to scale, it is important to monitor the overall
impact that these activities have on the VMware infrastructure.
Toggle hardware assisted copy by changing the DataMover.HardwareAcceleratedMove parameter in the
Advanced Settings tab in vSphere Virtual Center to 1.
When the value for hardware acceleration is 1, the data path changes for tasks such as Storage vMotion,
as illustrated in Figure 2.


3

Figure 2: Full copy primitive

Here, the ESX server is removed from the data path of the data copy when hardware copy is enabled.
Removing copy transactions from the server workload greatly increases the speed of these copy functions
while reducing the impact to the ESX server.

How effective is the offload process?
During IBM lab testing, data retrieved from the VMware monitoring tool, esxtop showed that commands
per second on the ESX host were reduced by a factor of 10. Copy time reduction varies depending on the
VM but is usually significant (over 50% for most profiles).
A few examples of this performance boost at customer data centers are shown in Table 1.

Time reduction
Customer Test Before VAAI After VAAI
(in percentage)
Major Financial 2 VMs 433 Sec 180 sec 59%

Electric Company 2 VMs 944 sec 517 sec 45%

Petroleum 40 VMs 1 hour 20 min 67%
Company

Table 1: Field results for full copy

Full copy effect: Thousands of commands and IOPs on the ESX server are freed up for other tasks and
promote greater scalability.

Hardware-assisted locking (atomic test and set)
Just as important as the demonstrated effect of hardware-assisted copy, the hardware-assisted locking
primitive also greatly enhances VMware cluster scalability and disk operations by preventing disk
corruptions of the clustered file system (VMFS) with tighter granularity and efficiency.
It is important to understand why locking occurs in the first place. For block storage environments,
VMware data stores are formatted with VMFS. VMFS is a clustered file system that uses Small Computer


4

System Interface (SCSI) reservations to handle distributive lock management. When there is a change to
the metadata of the file system by an ESX server, the SCSI reservation process ensures that shared
resources do not overlap with other connected ESX hosts by obtaining exclusive access to the logical unit
number (LUN).
A SCSI reservation is created on VMFS when (not a complete list):
 VMDK is first created
 VMDK is deleted
 VMDK is migrated
 VMDK is created via a template
 A template is created from a VMDK
 Creating or deleting VM snapshots
 VM is switched on or off

Although normal I/O operations do not require this mechanism, these boundary conditions have become
more common as features such as vMotion with Distributed Resource Scheduler (DRS) are used more
frequently. This SCSI reservation design leads to early storage area network (SAN) best practices for
vSphere to dictate a limit in cluster size for block storage (about 8 to 10 ESX hosts).
With hardware-assisted locking, the LUN locking processing is transferred to the storage system. This
reduces the number of commands required to access a lock, provides locks to be more granular, and
leads to better scalability of the virtual infrastructure.

V V V V V V
M M M M M M
D D D D D D
K K K K K K

Figure 3: Atomic test and set primitive

Hardware-assisted locking effect: Hardware-assisted locking will increase VMs per data store, ESX
servers per data store, and overall performance. This functionality coupled with 60 cores and 360 GB of
memory for the XIV Gen3 Storage System yields best in industry scalability and performance.


5

Block zeroing (write same)
Block zeroing is designed to reduce the amount of processor and storage I/O utilization required to write
zeroes across an entire EZT VMDK when it is created. With the block zeroing primitive, zeroing operation
for EZT VMDK files are offloaded to the XIV Storage System without the host having to issue several
commands.

Figure 4: the write same primitive

Block zeroing effect: Block zeroing reduced overhead and provided better performance for creating EZT
virtual disks. With XIV, EZT volumes are available immediately through fast write caching and de-staging.
VAAI support on XIV storage systems liberates value compute resources in the virtual infrastructure
Offloading processor and disk intensive activities from the ESX server to the storage system provides
significant improvements in vSphere performance, scalability and availability.
Note: Before installing the VAAI driver for the XIV storage system, ensure 10.2.4a or higher is the installed
microcode. For vSphere 5.x and higher, the VAAI driver is no longer required for IBM Storage.

vStorage APIs for Array Awareness
The IBM Storage provider for VMware VASA provides even more real-time information about the XIV
Storage System. The IBM XIV Storage System VASA provider for VMware vCenter adds:
 Real-time disk status
 Real-time alerts and events from the XIV Storage System to vCenter
 Support for multiple vCenter consoles and multiple XIV Storage Systems
 Continuous monitoring through storage monitoring service for vSphere
 Support for policy-driven storage profiles


6

Figure 5: VASA block diagram

Adding VASA support that is available in vSphere 5, allows VMware and Cloud administrators insights to
help provide improved availability, performance, and management of the storage infrastructure.
In addition to VASA, the XIV Storage System also provides a vCenter Plug-in for vSphere 4 and
vSphere 5, which extends management of the storage to provisioning, mapping, and monitoring of
replication, snapshots, and capacity.

IBM System Storage Plug-in for vCenter
The IBM System Storage Plug-in for vCenter couples the XIV architecture advantages with powerful
storage management tools. Features include:
 Volume creation
 Mapping and automated discovery
 Configuration optimization
 Capacity reports and monitoring
 Snapshot details
 Replication status (RPO)

With these plug-in tools, management processes and operations can be streamlined in vSphere
infrastructures.


7

The XIV GUI provides simplicity and functionality unmatched in the industry today. This leading and
forward design has carried over to the vCenter plug-in. Combining the plug-in with the automation applied
to every LUN created on the XIV storage system ensures that storage provisioned from vCenter will
always be optimally placed in the storage configuration.

Along with strong functionality, the IBM XIV vCenter Storage Plug-in maintains strong secure use of
storage administration tasks with multiple levels of access.

Storage Replication Adapter (SRA)
Site Recovery Manager is a VMware product for managing, testing and automating disaster recovery
workflows.
The SRA is a critical part of Site Recovery Manager deployment, but it is supplied by the storage vendor
rather than VMware.
The XIV SRA not only provides the underlying disk-based replication with VMware awareness but also
greatly simplifies disk replication without additional complexity and hardware components required by
other vendor solutions.


8

Through advanced XIV virtualization, storage replication configurations can be completed with drag and
drop and right-click automation.

Figure 6: Drag-and-drop replication configuration

Site Recovery Manager provides powerful testing capabilities. Disaster recovery scripts can be tested
using XIV high performance and incremental snapshots at the remote site. These tests not only provide a
deep understanding of the potential point in time recovery, which is recovery point objective (RPO), but
also the time it takes to recover, which is recovery time objective (RTO).

The software for Site Recovery Manager integration is included with the XIV Storage System, and you can
download it directly from the IBM website at: ibm.com/systems/storage/disk/xiv/


9

Snapshot management in vSphere
IBM Tivoli® Storage FlashCopy® Manager allows administrators to manage, schedule, and recover
snapshots on the IBM XIV Storage Systems directly from the vCenter GUI.

Figure 7: vCenter console

Along with management capabilities, snapshots taken in VMware integrate with the VMware snapshot
process to ensure that all VMs are recoverable when restored.
When taking a snapshot in vCenter:
 FlashCopy Manager initiates a VMware software snapshot of the VM through the vSphere
API
 FlashCopy Manager determines the LUN(s) that are associated with the virtual machine
 FlashCopy Manager invokes a hardware snapshot, creating a persistent copy of the virtual
disk(s) and associated VMware snapshots
 The hardware snapshot is kept for restore until specified for deletion through user-created
policies

Figure 8: Snapshot process

IBM XIV Storage System performs snapshots with virtually no overhead with re-direct on write technology.
Snapshots are made without the overhead associated with copy on write snapshot processing used by
other vendor solutions. Adding to this efficiency, XIV incremental snapshots are optimally configured (just


10

as primary volumes) ensuring enterprise-class read and write performance while eliminating the need for
processing and space overhead.

QoS options
As data center consolidation rates accelerate, there is a larger demand to apply different priorities to
VMware workloads and better balance workloads on storage devices. These features include:
 Storage I/O Control (SIOC)
 Storage DRS

In addition to full support for these features, the XIV Storage System provides an always autobalanced
storage configuration. If throttling particular storage is required, the XIV also provides advanced and
tunable QoS features on the storage system.

The combination of VMware features and storage features for QoS provides the maximum flexibility and
granularity when applying restricted or limited storage resource allocation to vSphere.

SIOC
With SIOC, VMware administrators can apply restricted storage performance access to a virtual disk on a
data store in vSphere. This function fits well with the always balanced disk performance provisioned for the
XIV Gen3 Storage System. That is, with the XIV, storage is physically optimized by default and vSphere
administrators need to restrict only virtual disks and machines that might conflict with higher priority virtual
systems.

Storage DRS
Storage DRS establishes data store clusters. The data store clusters allow the dynamic transfer of virtual
disks to different data stores according to the policies set in DRS. The XIV Storage System supports
storage DRS but also provides a more global approach to load balancing.

All storage created for vSphere on the XIV system is balanced across all processors, ports, spindles, and
cache modules in the XIV Storage System. The XIV Gen3 storage configuration in optimal balance
prevents hitting thresholds and triggers that set off redistribution of disk resources.

Storage QoS on the XIV Storage System
By default, the XIV Storage System provides optimal performance for all the disks provisioned to vSphere.
There might be requirements for limiting host performance to the disk. With the XIV system, you can
restrict the IOPS or bandwidth provided to a host or group of hosts.
The XIV system gives you the choice of utilizing VMware QoS features or off-loading performance
management to the hardware. Scalable QoS in XIV Gen3 gives cloud administrators maximum flexibility
for managing storage resources across the diverse workloads in consolidated vSphere infrastructures.


11

Summary
The IBM XIV Gen3 Storage System provides a solid foundation for vSphere. The automation provided by
the XIV system reduces the steps required to realize outstanding storage performance and functionality for
VMware environments. This coupled with a strong global partnership between VMware and IBM ensures
that the IBM XIV Gen3 Storage System continues to lead for vSphere infrastructures today and in the
future.
You can find more information about the IBM XIV Gen3 at: ibm.com/systems/storage/disk/xiv/


12

Appendix A: Ordering options for the XIV Gen3 Storage System

The XIV Gen3 Storage System is ordered according to the capacity required. The minimum physical
installation is six data modules yielding 55 TB of usable storage (there is also a Capacity on Demand
(CoD) plan allowing a three-module 28 TB minimum entry point). Although 28 TB is the minimum
configuration for XIV using CoD, it is appropriate only for storage requirements that will grow to at least
55TB in a one- to three-year period.

Figure 9: XIV Gen3 Internal Components


13

Initial configuration options for the XIV Gen3 are shown in Table 2.

Number of
6 9 10 11 12 13 14 15
data modules
Number of disks 72 108 120 132 144 156 168 180
Usable capacity TBs
55 88 102 111 125 134 149 161
(2-TB drives)
Usable capacity TBs
84 132 154 168 190 203 225 243
(3-TB drives)
Fibre Channel (FC)
8 16 16 20 20 24 24 24
ports (8 GB)
Processor (Quad
6 9 10 11 12 13 14 15
core)
Memory (GB) 144 216 240 264 288 312 336 360

Table 2: Configuration options for XIV Gen3 Storage System

If ordered initially with less than 15 data modules, the XIV Gen3 Storage System can be upgraded using
the increments in the above table. Additional capacity, processing, connectivity and cache can all be
added on-the-fly without disruption or performance impact.

Capacity on demand (CoD)
With the CoD ordering option for the XIV Gen3 Storage System, customers benefit from the full
processing, caching, and drive spindle performance of the physically deployed system; yet only an initial
usable capacity is purchased.
The minimum usable capacity purchased in a CoD configuration is 28 TB (other initial increments are
available). This minimum entry point consists of three CoD data modules and three CoD I/O data modules.
As shown in Figure 10, this configuration provides 28 TB of purchased capacity and 27 TB of in-place
expansion.
Although there is a slight premium per usable TB for CoD options, the performance of CoD purchased
capacity benefits from the additional processor, memory, FC ports, and spindles in a base 55 TB
configuration.


14

60
50
40
Un-act.
30
Active
20
10
0
TBu
Figure 10: Available in-place upgrade with CoD

For more information about the CoD program as well as terms and conditions, contact your IBM
representative or authorized partner.


15

Appendix B: SAN ordering considerations

The SAN ordering considerations include queue depth, multipathing, and zoning and cabling.

Queue Depth
In general, the queue depth setting in the host bus adapter (HBA) has to be extended as consolidation
scales higher and I/O requirements are high. Refer to IBM Redbooks® mentioned in the “Resources”
section.

Multipathing
Round robin policy should be implemented for each LUN from the XIV. This will achieve the best
performance results.
VMware 4.1 update 1 and higher versions will automatically detect XIV and set the round robin policy.
VMware 5.x and higher versions can automatically detect XIV and set the round robin policy.

Zoning and cabling
As the XIV Storage System is an active grid, ensure that the workload from VMware is balanced across all
the six I/O controllers. A typical cabling diagram is shown in Figure 11.

All the paths are
active and use native
VMware round robin
multipathing.

Figure 11: A typical cabling diagram


16

In this example, a zone should be created for each initiator. Depending on the SAN vendor
recommendations, all the six paths can then be added to that zone or separate zones can be created for
each initiator / target path. Consult your SAN fabric vendor for best practices.
The end results provide a path from each host initiator to all the six targets cabled in the diagram.


17

Resources
The following websites provide useful references to supplement the information contained in this paper:
 IBM Systems on PartnerWorld
ibm.com/partnerworld/systems

 IBM Redbooks
ibm.com/redbooks

 XIV Storage System: Host Attachment and Interoperability
ibm.com/redbooks/abstracts/sg247904.html

 IBM Publications Center
www.elink.ibmlink.ibm.com/public/applications/publications/cgibin/pbi.cgi?CTY=US

About the author
At IBM, Peter Kisich works closely with storage development organizations and field solutions architects to
ensure that IBM technologies can continue to innovate and build solutions for enterprise cloud
environments. An accomplished speaker, Mr. Kisich presents the value of IBM storage for virtual
infrastructures to the global community of IBM customers, Business Partners and employees.
Before IBM, Pete Kisich served as a Technical Alliance Manager for VMware, supporting storage partners
NetApp, Hitachi, and Oracle.


18

Trademarks and special notices
© Copyright IBM Corporation 2011. All rights Reserved.
References in this document to IBM products or services do not imply that IBM intends to make them
available in every country.

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business
Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked
terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these
symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information
was published. Such trademarks may also be registered or common law trademarks in other countries. A
current list of IBM trademarks is available on the Web at "Copyright and trademark information" at
www.ibm.com/legal/copytrade.shtml.
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or
its affiliates.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the
United States, other countries, or both.
Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States,
other countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

SET and the SET Logo are trademarks owned by SET Secure Electronic Transaction LLC.
Other company, product, or service names may be trademarks or service marks of others.
Information is provided "AS IS" without warranty of any kind.

All customer examples described are presented as illustrations of how those customers have used IBM
products and the results they may have achieved. Actual environmental costs and performance
characteristics may vary by customer.

Information concerning non-IBM products was obtained from a supplier of these products, published
announcement material, or other publicly available sources and does not constitute an endorsement of
such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly
available information, including vendor announcements and vendor worldwide homepages. IBM has not
tested these products and cannot confirm the accuracy of performance, capability, or any other claims
related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the
supplier of those products.
All statements regarding IBM future direction and intent are subject to change or withdrawal without notice,
and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller for the
full text of the specific Statement of Direction.
Some information addresses anticipated future capabilities. Such information is not intended as a definitive
statement of a commitment to specific levels of performance, function or delivery schedules with respect to
any future products. Such commitments are only made in IBM product announcements. The information is


19

presented here to communicate IBM's current investment and development activities as a good faith effort
to help with our customers' future planning.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled
environment. The actual throughput or performance that any user will experience will vary depending upon
considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the
storage configuration, and the workload processed. Therefore, no assurance can be given that an
individual user will achieve throughput or performance improvements equivalent to the ratios stated here.
Photographs shown are of engineering prototypes. Changes may be incorporated in production models.
Any references in this information to non-IBM websites are provided for convenience only and do not in
any manner serve as an endorsement of those websites. The materials at those websites are not part of
the materials for this IBM product and use of those websites is at your own risk.


20

IBM XIV Gen3 Storage System

More Related Content

What's hot

Viewers also liked

Similar to IBM XIV Gen3 Storage System

More from IBM India Smarter Computing

Recently uploaded

IBM XIV Gen3 Storage System