The document describes a stretched metro cluster configuration using NexentaStor software and Coraid SRX storage arrays. Key features include high availability across geographically separate sites, non-disruptive failover for block and file storage, and synchronous replication providing a disaster recovery solution. The configuration was implemented for a Belgian managed services provider, delivering over 10,000 IOPS and 200MB/s of throughput between sites 15km apart with failover times under a minute. A video demo and performance benchmarks are presented.

1. How to Design your Metro Cluster with
NexentStor
Roel De Frene, CEO S3S
Andy Bennett, Nexenta

2. Stretched Metro Cluster – Key Features
• Provides High-Availability Clustering across Geographically disparate sites
• Non Disruptive Failover for Block and File Storage Services
• Manual or Automatic Failover between sites
• ZFS Filesystem Block Level Mirroring delivers optimal performance
• End to End checksum on every transaction
• Volume Service Locking prevents traditional exposure to Site Split Brain
• World leading Triple Site Mirroring
• Local Read weighting and SSD cache accelerates performance

3. About Me
Roel De Frene
CTO of Server Storage Solutions Belgium
12-years in Server & Storage business
My company offers Built to Order Server & Storage solutions, on customer demand and we
offer the best-fit solution available. We started as Nexenta partner in 2009, and after 2 years
we can profit off all our hard work to get “certified” Nexenta dealership.
At OpenStorage Summit 2011, we will explain how to built a Stretched Metro cluster with
Nexenta. This case study is in production since last Summit in Amsterdam. In Amsterdam we
also showed a live failover between 15km distance, running VMware hypervisors...

4. About the end-user BENESOL
Benesol is one of the key players in managed services, cloud computing, and server colocation in
Belgium. They own one of the greenest datacenters in Benelux and have multiple datacenter
rooms connected with a dark fibre ring across Brussel - Antwerp & Oostkamp with a total range
of 120km max distance between sites.

5. Technical requests of BENESOL
- High Available iSCSI services between the headnodes
- Distribute data synchronous between 2 physical sites (+15km and more)
- Failover time to data center 2 must be less then a minute, and no crashes to VM’s
- Vmware storage motion and Vmotion between two sites
- High IOPS > 10.000 IOPS (4K random)
- High Performance > 200MB/s
- Datacenter shutdown proof!
- Start with 11TB netto capacity and able to grow to 500TB
- budget: <100.000 $

6. The ingredients to answer the requests
- Nexenta HA cluster running on off the shelve hardware (Supermicro)
+
- Coraid SRX2800 AoE storage enclosures
+
- 10Gbit Fibre-optic switches & cabling with a latency less then 1ms between the sites
+
- Customer agreed for a first time POC on Nexenta Alpha solution
+
- Lot’s of technical support from Nexenta & Coraid working together on this !
- Once fully tested customer goes to Beta with small production Trial

7. Stretched Metro Cluster layout

8. Stretched Metro Cluster – Performance
Volume 'test1' is the only available volume, starting benchmark...
test1: running optimal mode benchmark
test1: generating 49134MB files, using 32768 blocks
test1: started 2 benchmark processes
test1: this may take a few minutes, please wait or press Ctrl-C to interrupt ............
test1: benchmark in progress, elapsed 60 sec............
test1: benchmark in progress, elapsed 120 sec............
test1: benchmark in progress, elapsed 180 sec............
test1: benchmark in progress, elapsed 240 sec............
test1: benchmark in progress, elapsed 300 sec............
test1: benchmark in progress, elapsed 360 sec............
test1: benchmark in progress, elapsed 420 sec.........
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
356MB/s 26% 220MB/s 22% 535MB/s 24% 678/sec
356MB/s 27% 221MB/s 22% 538MB/s 24% 675/sec
--------- ---- --------- ---- --------- ---- ---------
713MB/s 26% 441MB/s 22% 1073MB/s 24% 677/sec

9. Disk Only Performance

10. Actual Performance Benchmarks - EMEA Demo
• Local Site performance Brussels
• 25,625 IOPS at 4K
• 32K Benchmarked at over 400 MB/sec
• Failover to Antwerp Site
• Failover event takes less than 25 Seconds
• I/O paused on Client machines for approx 10 secs
• I/O Continues without Client disconnect
• Performance picks backup to steady state
• 15,000 IOPS at 4K
• 32K Benchmarked at over 300 MB/sec
• End to End checksum on every transaction

11. Video Demo

12. Stretched Metro Cluster Storage Options
SRX 3200 AoE
24 Disk High Performance Ethernet SAN Array
10 GbE Connectivity up to 72TB per shelf
NXTSMC-3600F
36 Disk High Performance Array
Fibre channel Connectivity up to 108 TB per shelf
NXTSMC-3600FE
36 Disk High Performance Ethernet SAN Array
FCoE Connectivity up to 108 TB per shelf
Other Storage Options In Certification

13. Triple Site Redundancy – Extreme Metro Cluster