VMworld 2013: Capacity Jail Break: vSphere 5 Space Reclamation Nuts and Bolts

Capacity Jail Break: vSphere 5 Space Reclamation
Nuts and Bolts
Aboubacar Diare, Hewlett-Packard
Abid Saeed, VMware
STO4907
#STO4907

© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.2
Disclaimer
All information related to non release ESX versions discussed in this
presentation is a shared as a technology preview and is subject to
change without notice

Agenda
1. Problem statement
2. What is UNMAP
− VAAI UNMAP recap
− UNMAP description
3. UNMAP in ESXi5.x
4. Test configuration
5. Results and considerations
6. Summary
7. Call to action

Problem Statement
 VMFS datastore freed capacity from
VM deletions is not release
automatically at the storage and when
a thin provisioned LUN is used
 If this capacity is never re-used by the
VMFS file system. It amounts to
stranded capacity allocated to a
volume no longer using this capacity
 For improved capacity utilization
efficiency, it is ideal to jailbreak this
capacity so it may be consumed
where needed
VMware ESX
VMFS
Storage TP
SPACE

VAAI Space Reclamation Support
Primitive Equivalent
Names
ESX & vCenter Name SCSI Command
ESXi5Primitives
UNMAP
• Space
Reclamation
• /VMFS3/EnableBlockDel
ete
UNMAP
(SCSI OP Code:42h)
OUT OF
SPACE
CONDITION
• TP STUN
• VM Pause
• Temporary
lack of LBA
• Permanent
lack of LBA
NA
Return codes:
• Not Ready Allocation in progress
• Chk Condition Allocation failed write
protect
QUOTA
EXCEEDED
BEHAVIOR
• TP Soft
threshold
reached
NA
Return codes:
• Unit Attention – Thin provisioning
Soft threshold reached
TP LUN
REPORTING
• Report thin
provisioning
capability
NA
Read Capacity TPE Bit
(SCSI OP Code: 16h)

• When a LUN is thin provisioned, the
storage target assigns physical blocks
for each LBA on a LUN on the first write to
that LBA
• When a VMFS datastore is created on a
LUN, the file system allocates blocks as per
user requests and tracks the free blocks
• As files are created, written to and deleted
on the datastore, a thin provisioned LUN
effectively becomes “thick provisioned” –
i.e. all the LBA on the LUN are written to
and thus have physical storage assigned
• UNMAP is a SCSI primitive that allows
VMFS to communicate to the storage
target that certain LBA are no longer
used on a LUN backing the datastore
With VAAI
VMware ESX
VMFS
Storage TP
SPACE
What Is UNMAP

Used Capacity (vCenter & Array Views)
After UNMAP
Before UNMAP
After VM deletion

Array Level vs. ESX Level Thin Provisioning
20GB
Thick or Thin SAN
Volume
VM
20GB
Thin SAN Volume
VM
Thin VMDK
20GB
Thick VMDK
VMFSVMFS
VMware ESXiVMware ESXi
Thin VMDK

Does Your Device Support UNMAP
ESXi5.0 / ESXi5.1 / ESXi5.5
esxcli storage core device vaai status get –d
esxcli system settings advanced list –option
/VMFS3/EnableBlockDelete
(no affect on system for ESXi5.0 patch2, ESXi5.0Ux, ESXi5.1Ux and ESXi5.5)

VMWare vCG/HCL VAAI UNMAP Support

UNMAP in ESXi5.x Introduction
• In vSphere 5.0 GA, as part of a file delete operation, VMFS would
issue UNMAP commands to notify the storage target of the LBA
that were being marked as “free” in the FS metadata
‒ File delete operations were bound to the time taken by storage
targets to complete the UNMAP commands. vSphere operations
that involved a file delete failed when the storage target delayed
completing UNMAP commands
• To handle the above problem, in vSphere 5.0U1 and vSphere 5.1,
VMFS stopped issuing UNMAP commands as part of the file
delete operation. These releases introduced a command line
triggered “asynchronous” way of issuing UNMAP commands on
ALL the unused blocks on a VMFS volume (vmkfstools –y)
• In ESXi5.5 the vmkfstools –y was replaced with an esxcli
command and the reclaim operation was improved

UNMAP in ESXi5.x

Space Reclamation in ESXi5.0
Synchronous ‒ IDEAL
HP 3PAR
Disk Array
1
2
4
3
User deletes a VM/VMDK
VMFS frees the blocks in VMFS metadata
Storage frees physical blocks
ESX sends UNMAP commands to the storage
VMFS free capacity Storage free capacity
VMFS
unmap
STEP 1
VM
VM
VM
VM

Space Reclamation in ESXi5.0patch2
HP 3PAR
Disk Array
1
2
4
3
Storage DO NOT free physical blocks
NO UNMAP commands to the storage
VMFS
STEP 1
VM
VM
VM
VM

Space Reclamation in ESXi5.0Ux, 5.1Ux and 5.5
Asynchronous
1
2
4
3
Storage frees physical blocks
ESX sends UNMAP commands to storage to
reclaim temp file allocated capacity.
HP 3PAR
Disk Array
VMFS
1 User runs vmkfstools / esxcli command
STEP 1
STEP 2
unmap
2 VMFS inflates temp file
?
VM
VM
VM
VM

What Is VMFS Free Capacity?
VMFS only view
VM
[50GB]
VM
[180GB]
VM
[160GB]
VM
[80GB]
FREED
[100GB]
FREED
[10GB]
NEVER USED
[160GB]
NEVER USED
[100GB]
FREED
[20GB]
NEVER
USED
[10GB]
FREE CAPACITY= Freed + Never used
FREE CAPACITY= 400GB
FREED
[200GB]
VM
[200GB]
FREE CAPACITY= 600GB

Space Reclamation in ESXi5.0Ux and ESXi5.1Ux
Stack view
VM
[100GB]
VMDK
[200GB]
VM
VMD
KVM
VMD
K
Thin
VMFS
Volume
VMDK
Thin
Thick
Physical
Storage
0
0
5TB
5TB
VMFS Free
Capacity
550GB
Storage Free
Capacity
550GB
vmkfstools –y 50
RECLAIM: 50% of 850GB = ANY 425GB of 850GB
UNMAP READWRITE WRITE
425GB
[200GB]
[50GB]
[50GB]
[50GB]
[200GB][300GB]
VMFS Free
Capacity
850GB
UNMAP
FREE CAPACITY =
Freed + Never used
This command will reclaim 50% of the free
capacity all at once. Default is 60% of the free
capacity.

Space Reclamation in ESXi5.0Ux and ESXi5.1Ux
• vmkfstools –y X (X percentage of free capacity)
• X defaults to 60%
• Must be run via CLI from the ESX host
• Must navigate to root of VMFS filesystem to UNMAP
• Utilizes a temp balloon file: .vmfsBalloonsXZYAB
• Max size of balloon file is 2TB
- Bug in ESXi5.1 will truncate larger than 2TB reclaims down to 2TB
- Fix in upcoming patch of ESXi5.1 will create multiple 2TB files
• Returned capacity is random

Space Reclamation in ESXi5.5 (Tech Preview)
Stack view
VM
[100GB]
VMDK
[200GB]
VM
VMD
KVM
VMD
K
Thin
VMFS
Volume
VMDK
Thin
Thick
Physical
Storage
0
0
5TB
5TB
FREE CAPACITY =
Freed + Never used
VMFS Free
Capacity
550GB
Storage Free
Capacity
550GB
esxcli storage vmfs unmap –u uuid –n reclaim_unit
RECLAIM: 100% of 850GB in 200MB chunks
[50GB]
[50GB]
[50GB]
[200GB][300GB]
VMFS Free
Capacity
850GB
UNMAP READWRITE WRITE UNMAP
[200GB][250GB]
Storage Free
Capacity
850GB
Temp VMDK
This command will reclaim 100% of the free
capacity in chunks. The chunk size is specified
with the reclaim_unit value. Default 200 blocks.

Space Reclamation in ESXi5.5 (Tech Preview)
• esxcli storage vmfs unmap –u uuid –n reclaim_Unit
• vmkfstools –y <datastore_path> --reclaimBlockUnit X
(deprecated)
• Reclaim Block Unit defaults at 200 blocks
- 200MB on VMFS5, VMFS3 depends of FS block size (ex: 1600MB for 8MB blk size VMFS) .
• Can be run through vMA, vCLI (esxcli)
• Utilizes a temp balloon file: .asyncUnmapFile
• Max size of balloon file is ~64TB
• Max reclaimBlockUnit is ~max file size
• 100% capacity returned

So What’s in it for Me?
In depth understanding of how the ESXi
UNMAP process works
Avoid downtime and reclaim
capacity online

Test Configuration

Test Configuration/Methodology
- Storage VMotion VMs from DS1 to DS2.
- Reclaim freed capacity on DS1 using
command line tools

Results and Considerations

Considerations
Free Capacity reservation
Reclaim efficiency
Performance impact

ESXi5.0Ux ‒ ESXi5.1Ux
Free Capacity Reservation
ESXi5.5
[600GB] [600GB]
[600GB]
[600GB]
VM VM
VM VM
VM
VM
[600GB]VM VM
[600GB]VM
VM
vmkfstools –y 30
vmkfstools –y 60
vmkfstools –y 99
esxcli storage unmap --reclaimBlockUnits 200
esxcli storage unmap --reclaimBlockUnits 300GB
VMesxcli storage unmap --reclaimBlockUnits 600GB
VM

Considerations
Reclaim efficiency
Performance impact

Reclaim Efficiency
vmkfstools / esxcli before and after capacity utilization – STORAGE CAPACITY VIEW
ESXi5.5ESXi5.1
0
5
10
15
20
25
30
0
100000
200000
300000
400000
500000
600000
30% 60% 99% 200MB 400MB 800MB 1600MB 3200MB 6400MB 12800MB
(mins)(MB)
ESXi5.1 and ESXi5.5 Reclaim efficiency and performance for ~150GB reclaim
Used Capacity before reclaim Used Capacity after reclaim Used capacity goal Reclaim Duration

ESXi5.1 30% Iterative Reclaim Efficiency
0
0.5
1
1.5
2
2.5
3
3.5
0
50000
100000
150000
200000
250000
300000
350000
400000
450000
Iteration 1 Iteration 2 Iteration 3 Iteration 4
(Durationmins)
(UsedCapacityMB)
30% reclaim efficiency – USED STORAGE CAPACITY
Used Capacity before reclaim Used Capacity after Reclaim Used Capacity goal Reclaim Duration

Considerations
Reclaim efficiency
Performance impact

ESXi 5.1 Performance Impact
30% reclaim of ~150GB Heading Heading
Duration 1 min 3 sec
Capacity
reclaimed
3793MB
~1%
Peak latency
observed
~9 ms.

Capacity
reclaimed
10041 MB
~7%
Peak latency
observed
9 ms.

Capacity
reclaimed
99%
Peak latency
observed
~10 ms.

Heading Heading
Capacity
reclaimed
100%
Peak latency
observed
~11 ms.
ESXi 5.5 Performance Impact (Tech Preview)
12800MB per iteration reclaim ~150GB
ESXi5.1
Capacity
reclaimed
99%
Peak latency
observed
10 ms.

Performance Impact for
Larger Capacity Reclaim

ESXi5.1 Performance Impact
99% reclaim of 720GB
Heading Heading
Capacity
reclaimed
99%
Peak latency
observed
~10 ms.

Heading Heading
Capacity
reclaimed
100%
Peak latency
observed
~11 ms.
ESXi5.5 Performance Impact (Tech Preview)
12800MB per iteration reclaim of ~720Gb

UNMAP in ESXi5.1 vs. ESXi5.5 (Tech Preview)
Comparison summary
ESX5.1/ESXi5.0U1 ESXi5.5
Capacity reservation High Low
Reclaim efficiency Random 100%
Maintenance window recommended to
reclaim 100%
YES
NO
(array & capacity
dependent)
Uses esxcli cmd NO YES
Uses vmkfstools cmd YES YES
(deprecated)
Performs better at higher reclaim
increments
NO
YES
(Up to a threshold &
dependent on various
factors)
Support for multiple descriptors in
UNMAP command
NO YES
Efficiently reclaim small capacity chunks
online as they are freed
NO YES

In Sum

Sync UNMAP in vSphere 5.0 GA
Pros:
• Freed VMFS capacity is immediately reclaimed in storage capacity pool
• VMFS issues UNMAP only on those LBA that were actually written to
(FREE CAPACITY RESERVATION)
• 100% reclaim efficiency. (RECLAIM EFFICIENCY)
Cons:
• Possible failures of SVMotion, Snapshot consolidation and other
hypervisor operations
• Impact on VM workloads (PERFORMANCE IMPACT)
• Array ability to quickly turn around an UNMAPed block if immediately allocated
FREE CAPACITY
RESERVATION
RECLAIM EFFICIENCY
PERFORMANCE IMPACT(*)
HP 3PAR StoreServ UNMAP
(*) Performance impact is array
and workload dependent
Reclaim not supported

Async UNMAP in vSphere 5.0Ux and 5.1Ux
Pros:
• The file delete operations are decoupled from VM operations
Cons:
• Freed blocks aren’t tracked. UNMAP sent percentage of free blocks
• FS block allocation may fail when the UNMAP operation is running
• Nondeterministic reclaim inefficiency (RECLAIM EFFICIENCY)
• Higher impact on VM workloads as reclaim % increases
(PERFORMANCE IMPACT)
FREE CAPACITY
RESERVATION
RECLAIM EFFICIENCY
Offline reclaim suitable

Async UNMAP in vSphere 5.5 (Tech Preview)
Pros:
• The file delete operations are decoupled from VM operations
• FS block allocation will not fail when the UNMAP operation is running
• 100% reclaim efficiency (RECLAIM EFFICIENCY)
• Allows online capacity reclaim with less overall impact on workload
(PERFORMANCE IMPACT)
• Able to reduce the number of UNMAP commands required with multiple
descriptor per command support
• The esxcli implementation allows users to script the operation across hosts over
the network
Cons:
• Freed blocks aren’t tracked. UNMAP sent to all free blocks all the time.
FREE CAPACITY
RESERVATION
RECLAIM EFFICIENCY
Online & offline reclaim suitable

HP Arrays UNMAP Support
Array UNMAP Support
HP 3PAR StoreServ YES
HP P9500 YES
HP StoreVirtual Planned
HP P6000 Planned
HP P2000 NO

Q & A

Call to Action
Attend or download these sessions:
STO4907: Capacity Jail Break: vSphere 5 space reclamation nuts and bolts
Monday, 8/26: 2:30pm – 3:30pm
VSVC6656: OpenStack for the Enterprise
Tuesday, 8/27: 11:30am – 12:30pm
STO5545: Top 10 things you must know about storage for vSphere
Tuesday, 8/27: 5:00pm – 6:00pm
EUC5672: Implementing a Scalable and Highly Available Desktop and Application
Architecture with a VMware AlwaysOn Solution
Wednesday, 8/28: 11:30am – 12:30pm
STO5787: Storage - The Next Frontier of Virtualization –
How VMware technologies can enable and accelerate Software Defined Storage
Wednesday, 8/28: 2:30pm – 3:30pm
Talk to HP experts
Booth 1405
• More then 20 demos including Cloud, Virtualization, and VMware integration
• On-going theater sessions
Booth 2235
• Software Defined Zone

VMworld 2013: Capacity Jail Break: vSphere 5 Space Reclamation Nuts and Bolts

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to VMworld 2013: Capacity Jail Break: vSphere 5 Space Reclamation Nuts and Bolts

Similar to VMworld 2013: Capacity Jail Break: vSphere 5 Space Reclamation Nuts and Bolts (20)

More from VMworld

More from VMworld (20)

Recently uploaded

Recently uploaded (20)

VMworld 2013: Capacity Jail Break: vSphere 5 Space Reclamation Nuts and Bolts