This document summarizes Nexenta Systems' involvement in providing storage for the Hands-on-Labs (HOL) at VMworld 2011 in Las Vegas and Copenhagen. It describes the architecture developed using Nexenta storage and VMware technologies on a commodity "SuperRack". Key statistics are provided on the number of VMs created, IOPS performance achieved, and how Nexenta storage handled the workload with high availability. Custom analytics dashboards were also developed to visualize storage metrics for attendees.

1. VMworld 2011:
Running the Hands On Labs (HOL)
and
How We Got There
Andy Bennett, Craig Morgan
Nexenta Systems

2. VMware Trial Cloud - SuperRack
– Background
• Approached February 2011
• Co-design for a commodity white box solution
• To Be Comparable to Vpod and Vblock
– Architecture Team
• Pulled together with VMware Integration Engineering
Team (IE)
• Responsible for bleeding edge deployments and PoCs
• Brains behind the HoL as a testbed for early
deployments of VMware and partner technologies

3. VMware Trial Cloud - SuperRack

4. VMware Trial Cloud - SuperRack
• Requirements
– run High Density Generic Cloud Workloads
– Cost per TB primary design consideration but good
IOP’s performance important
– commodity x86 hardware via technology partners
– Nexenta HA design and scale out for future with
NFS referrals
– Classic POD Cloud Design

5. SuperRack

6. Nexenta Storage Module
Part Description Total Qty
System 2u 6 HDD Chassis 4
Motherboard X8DAH+F 4
Processors X5670 2.93G 8
Memory 8GB NETLIST 72
HDD Seagate 2TB SAS 7.2k 360
JBODs DataOn 60x bay 6
Controllers LSI SAS 8
10GbE NIC STD Dual-port w. SFP+ 4
SATA DOM Innodisk 64GB 4
License 720TB raw 1
SSD 2x STEC ZeusRAM 8GB 8
SSD STEC ZeusIOPS 200GB 8

7. Nexenta Storage Module
• Nexenta Layout
– multiple pools (active/active cluster)
– 2x data pools
• Per Pool
– vdev layout is 6x RAID Z2
– 30x vdevs per pool
– 2x SLOG devices (STEC ZeusRAM mirrored)
• Totals (Rack)
– 360 drives, 720TB raw, 480TB usable

8. Other Storage Service Providers
2x FAS 3270 Appliances
144 x 15K Hard Disks
4 PAM Cards per Appliance
2x EMC VNX 7500 platforms
3x File Blades per VNX
1 VNX was all SSD 96 SSD’s in total
1 VNX 140+ 15K Disks FAST Cache, FAST VP SSD Tier

9. Hands-on-Labs Workload
– Some highlights
• each lab provision
– 26x distinct labs
– 4x to 25x VMs per lab
– max VM size was 26GB
– create, deploy, destroy per login
– lab choice random (student choice), no pre-pop possible!
• nested VMs workload, highly latency sensitive
– Takeaways
• 148,103 VM’s created during Vegas show
• 1x VM created every 1.215 seconds

10. HoL Statistics
Element Las Vegas Copenhagen Total
Lab Time 50 38 100 (12@Sydney)
Labs Taken 13415 4797 18912 (700@Sydney)
Lab Hours 12388 ?
Lab Seats available 487 280 767
VMs per Lab 4-25x 4-22x
Unique Attendees 5042 1045 6207 (120@Sydney)
VMs Created 148103 53358 201461

11. Nexenta Statistics
• Ran 4 of 8 VMware verticals in Vegas
• 10.3 billion NFS IOPs served
– 7.9 billion in Vegas, 2.4 billion in Copenhagen
• 3 billion NFS IOPs from one head in Vegas
• Peak controller load
– 154,000x 4K NFS ops/sec at sub 1 ms latency
– 38,590x 16K NFS ops/sec on a single controller

12. Nexenta Statistics (cont.)
– Highest Bandwidth (single head, 16K average I/O)
• 1,305 MB/sec total
• 928MB/sec read
• 376MB/sec write
• ... less than 2ms latency throughout above!

13. Nexenta Operational Issues
– DRAM failure
• DRAM failure in one head triggered HA failover
• partner head ran the workload of “both” for 6hrs until
evening maintenance window
• NS called out failed DIMM serial number
• DIMM replaced head back in service 12mins
– High Availability RSF-1 HA plugin worked flawlessly
• VMware saw no loss of service
• monitoring informed NOC before they attributed issue
• head over provisioning in design meant solution didn’t
glitch with extra workload

14. Nexenta VMworld Custom Analytics
– Tech demo (give us feedback!)
• Sketch concept to delivered in 1 1/2 days
• Custom written with distance visualisation in mind
• 30ft screen (jumbotron) at 100ft distance
• Convey as much useful info as densely as possible
• High level, useful statistics
• NFS IOPs (not raw I/O or bytes)
• VM & VMDK latency and OPS (appropriate objects)
• Response to VMworld NOC team input
• Deliver
• bandwidth, flow, read, write, top 10x, server, latency,
alignment

15. Nexenta VMworld Custom Analytics

16. Nexenta VMworld Custom Analytics
VMworld Vegas 2011 Prototype
Video Clip available at: http://bit.ly/aura-vegas

17. Nexenta VMworld Custom Analytics
VMworld Copenhagen 2011 Prototype
Video Clip available at: http://bit.ly/aura-copenhagen

18. Thanks!
Question and Answer
Andy Bennett (andy@nexenta.com)
Craig Morgan (craig@nexenta.com)
Richard Elling (richard@nexenta.com)