1. The document proposes a novel disaster recovery solution combining interconnect-transparent VM migration (Ninja migration) and incremental block migration to enable quick recovery from catastrophic events like earthquakes. 2. An experiment was conducted migrating VMs between an Infiniband cluster and Ethernet cluster on an emulated wide area network, demonstrating the feasibility of the approach. 3. The results estimate that 56-480 VMs could be evacuated within an hour depending on the distance between data centers.

1. Preliminary*Experiment*of*Disaster*Recovery**
based*on*Interconnect9transparent*Migra<on
Ryousei(Takano,(Hidemoto(Nakada,(Takahiro(Hirofuchi,(Yoshio(Tanaka,(and(Tomohiro(Kudoh(
Na9onal(Ins9tute(of(Advanced(Industrial(Science(and(Technology((AIST),(Japan
Summary* Incremental*block*migra<on*
! Quick$recovery$is$desirable$from$catastrophic$events,$e.g.,$ ! An$incremental$block$migraDon$is$a$storage$migraDon$
the$great$eastRJapan$earthquake$on$March$11,$2011.$ mechanism$that$enables$us$to$migrate$a$VM$without$shared$
! We$propose$a$novel$disaster$recovery$soluDon$based$on$ storage.$$A$base$(master)$image$is$shared$between$source$
virtual$machine$(VM)$migraDon$technologies,$combining$ and$desDnaDon,$and$only$the$delta$image$is$transferred$to$
interconnectRtransparent$migraDon$(Ninja$migraDon)$and$ the$desDnaDon.$
incremental$block$migraDon.$ ! We$adapt$it$for$disaster$recovery$on$WAN$environments.$$
! To$demonstrate$the$feasibility,$we$have$conducted$an$ The$base$image$is$stored$in$shared$storage$of$each$data$
experiment$of$VM$migraDon$between$an$Infiniband$cluster$ center,$and$it$is$synchronized$between$data$centers$in$
and$an$Ethernet$cluster$on$an$emulated$WAN$environment. advance.$
! The$cost$of$storage$migraDon$can$be$significantly$reduced$
Survey*of*IT*infrastructure*damages* since$compute$nodes$of$a$cluster$is$homogeneous.$
Iwate
Prefectural Figure 1. Location of the
Data$center$1 Synchroniza9on( Data$center$2
University institutions that provided IT in(advance
infrastructure damage reports: cmp00 cmp01 cmp04 cmp05
Iwate Prefectural University,
Tohoku University, High mem. mem.
Energy Acceleration Research
Organization (KEK), University VM00 VM01
of Tsukuba, and AIST. The red
Tohoku Univ. dot represents the epicenter. base$ Wide$area$ base$
The map is a courtesy of image network image
KEK
wikipedia. Red regions mem. mem.
Tsukuba Univ. experienced JMA seismic str00 str01
intensity of 6 or more, and 4 to VM02 VM03
AIST
6 in yellow and orange
regions. cmp02 cmp03 cmp06 cmp07
! Data$collected$about$IT$infrastructure$damages$on$the$great$ Ninja(migra9on(+(incremental(block(migra9on
east$Japan$earth$quake$is$summarized$in$the$table$below$[1].$ Figure 3. A disaster recovery scheme using incremental block migration
! IT$equipment$suffered$minimal$damages.$
! 30$to$60$minutes$available$for$VM$migraDon$acDviDes. Evalua<on*
Distance from Seismic IT equipment.
Electrical Power Network Connectivity
the Epicenter Intensity damages
Iwate Prefectural
University
220 km 6- none
Power uninterrupted
(generators)
Redundant links kept
connectivity alive
! We$demonstrate$migraDon$of$4$VMs$from$an$Infiniband$
Tohoku
150 km 6- to 6+ none
UPS supplied tens of Lost after 28 minutes, cluster$to$an$Ethernet$cluster$by$using$the$proposed$soluDon.$
University minutes due to SINET shutdown
! We$used$two$5$nodeRclusters$connected$with$Gigabit$
UPS supplied tens of
KEK 310 km 6- none
minutes
Data not available
Ethernet$via$a$hardware$network$testbed$GtrcNETR1,$which$
Univ. of Tsukuba 310 km 6- none
UPS supplied tens of
minutes
Lost immediately emulates$the$roundRtrip$latency$varied$from$0$to$200$ms$and$
AIST 310 km 6- minimal
UPS supplied 15 to
60 minutes
Available for 60 minutes the$bandwidth$varied$from$100$Mbps$to$1$Gbps.$
! The$overhead$of$Ninja$migraDon$is$about$3$seconds.$
! We$esDmate$that$the$number$of$VMs$which$can$evacuate$in$
Ninja*migra<on* one$hour$are$from$56$(JP$to$US)$to$480$(JP$to$JP).
! Ninja$migraDon$[2]$is$an$interconnectRtransparent$migraDon$ 300$ Host*machine*environment
mechanism$to$migrate$a$cluster$of$VMs$between$data$centers$ 250$ 229$
246$ 251$ 253$ OS Debian$Linux$7.0
equipped$with$heterogeneous$interconnect$devices.$ QEMU 1.4
Elapsed*<me*(second)
200$
! Ninja$migraDon$has$no$performance$overhead$during$normal$ VM*environment
150$ VCPU 8
operaDons,$and$MPI$processes$can$migrate$between$data$
Memory 512$MB
centers$without$restarDng$the$processes.$ 100$
Disk 5$GB
! Ninja$migraDon$expands$the$opportunity$and$the$availability$ 50$ 30$ 30$ 33$ 35$
OS ScienDfic$Linux$6.3
for$VM$migraDon$between$data$centers.$ 0$ MPI$lib. Open$MPI$1.6.4
1$Gbps$ 100$Mbps$
OFED 3.5
0$ms$ 10$ms$ 100$ms$ 200$ms$
VM VM VM
MPI app. MPI app. MPI app. Figure 4. Experimental setting and result
Infiniband node
MPI lib. MPI lib. MPI lib.
IB TCP
..
.
IB TCP
..
.
IB TCP
..
. Future*work*
IB$HCA virDo_net$ detach reDaEach
(Ethernet)
VMM VMM VMM VMM
! An$MPI$free$implementaDon$is$a$future$work.$
Migration to Migration to
! We$also$plan$to$evaluate$the$feasibility$of$the$proposed$
Ethernet node Infiniband node disaster$recovery$soluDon$on$a$real$distributed$environment$
VM
MPI app. like$PRAGMA$Cloud.
Ethernet node
MPI(library(switches( MPI lib.
the(transport(protocol( [1]$M.$Tsugawa,$R.$Figueiredo,$J.$Fortes,$T.$Hirofuchi,$H.$Nakada,$and$R.$Takano,$“On$the$Use$of$
..
in(coopera9on(with(VM( IB TCP
migra9on.(
. VirtualizaDon$Technologies$to$Support$Uniterrupted$IT$Services,”$RIDO$2012.$
[2]$R.$Takano,$H.$Nakada,$T.$Hirofuchi,$Y.$Tanaka,$and$T.$Kudoh,$“Ninja$MigraDon:$An$InterconnectR$
transparent$MigraDon$for$Heterogeneous$Data$Centers,”$HPGC$2013$(to$be$published).$
VMM VMM VMM VMM $
Acknowledgement:$This$work$was$partly$supported$by$JSPS$KAKENHI$Grant$Number$24700040$and$
Time
1) 2) 3) 4) JRRAPID$of$Japan$Science$and$Technology$Agency.
Figure 2. An overview of Ninja migration. It works with cooperation between a VMM
and an MPI runtime system on the guest OSs.