This project implements a Disaster Recovery Manager for a data center that monitors virtual machines and recovers them if they fail. It uses the VMware vSphere API to connect to ESXi hypervisors and vCenter. The manager includes components to ping VMs, take periodic snapshots, and recover failed VMs either by reverting snapshots or migrating VMs to a new host. It detects failures by checking for missed heartbeat pings and excludes VMs intentionally powered off by a user from recovery. The manager was implemented in Java using multithreading and allows for conversion between image formats to support multiple hypervisors.

1. San José State University
College of Engineering
Department of Computer Engineering
CMPE 283-01, Virtualization Technology
Fall 2014
Individual Project – 1: Disaster Recovery Manager for DataCenter
Submitted to
Prof. Simon Shim
Submitted By
Gaurav Bhardwaj
gaurav.bhardwaj@outlook.com
SJSU ID 009297431
March 30th
2014

2. 1. Introduction:
Goal: The aim of the project is to gain hands-on experience with ESXi Hypervisor and vCentre management server,
explore the capabilities and their API’s provided by VMware. It also aims towards learning how the server
virtualization can automate nearly all traditional server administration activities by automated tools.
Objective: The objective of the project is to perform disaster recovery of the Virtual Machines while managing a
DataCenter. Disaster Recovery of any Virtual Machine is catered by building an Availability Manager that gathers
statistics of each VM in the data-center and displays them in text format. It also implements recovery measures for
failed VMs by snapshot recovery and cold-migration. It takes periodic snapshots to maintain highest possible
consistency. Also, it creates an alarm for each VM which signals that a VM is being manually shutdown and
prevents recovery of that VM. The manager pings each VM periodically to determine if it is alive or failed.
Need: Disaster recovery management systems are primarily used when a machine dies unexpectedly or stops
responding to client requests, now a days you can not afford a downtime of a single minute. Disaster Recovery
provides assurance of virtual machine availability with minimum hassle when a disaster strikes. This project intends
towards building a prototype of an availability manager which monitors the Virtual Machines in a DataCenter.
2. Background
Disaster Recovery helps organizations to prevent from sudden data loss or machine failures.
Disaster recovery in general terms is the process that is taken in order to recover or continue the normal flow of any
technology infrastructure that is vital to the organization despite an occurrence of disaster. It is the process which
increases availability of the components involved in the infrastructure.
3. Requirement:
Functional Requirements:
• Minimum infrastructure required to initiate this program is one vHost connected to one vCentre with
having at least one Virtual Machine which is supposed to fail in future.
• Virtual machine is called alive if it responds to ping requests.
• System should be able to keep a periodic backup cache to revert to the machine in case it fails.
• System should be able to detect machine failures which is different from machine being powered off by
user.
• System should be able to dynamically add one vHost to the vCentre in case all current vHost are found
failed.
• Migrate the same Machine to another vHost.
• User powered off Machine should not be recovered by system.
• System should be able to write all current statistics.
• Image format conversion is to be done to clone this Virtual Machine to other HyperVisors.
Non-Functional requirements:
• System should be able to run with minimal number of threads.
• System should be able to add newly created virtual Machines in HeartBeat.
• System should not hit a web service everytime it needs a ServiceInstance. Connections should be cached to
be used across the recovery manager.

3. 4. Design and Architecture
Architecture Diagram for Disaster Recovery System
Our DRS sits on client side and uses WS api to connect to remote managed entities offered by Vmware. Each Virtual
Machine is hosted on ESX system which is being managed by vCentre server.
Components of Disaster Recovery System:
1. HeartBeat Manager: responsible for creating optimal number of threads responsible for pinging all the
virtual machines inside infrastructure. Sleeps for some configurable time , wakes up and starts pinging the
infrastructure. Invokes Recovery Manager as soon as it sees any VM or vHost dead.
2. Backup- Manager: Starts up, creates optimal number of threads to create clone and snapshot of Virtual
Machines. Thread sleeps and wakes up every 10 minutes. This component is individual in its operation and
acts as a helper.
3. Recovery-Manager: Implementation of complete algorithm and flow of process to recover.

4. Component Design
5. Implementation:
In order to implement the effective Disaster Recovery the following tools have been used:
• j2se 1.7
• Eclipse IDE
• vSphere VI API
• VMware vSphere server and client.
• VMware tools installed on each VM.
Implementation Approach:
• DRS initiates itself by reading a config. file which contains IP of vCentre, userID and password. DRS then
populates the inventory and starts the Ping thread which pings all Virtual Machine in the DataCentre. Ping
thread keeps a count of missed heartbeat, if number of missed HeartBeat goes beyond 5, ping thread
declares machine as dead and starts the Recovery Manager.
• Snapshot thread provides periodic snapshot creation for both Virtual-Machine and vHost on which it is
being hosted. This thread takes a sleep time of nearly 10 minutes which is configurable. Snapshot are being
created with memory state true, so as to revert to the most current state of system
• Along with virtual machine the liveliness of Host is also being monitored by Ping thread, since we can
safely assume that if vHost is dead all its hosted Virtual-machine will be dead.

5. • Alarms are created for each virtual machine so as to check if a user has turned off a virtual machine or it
got dead. Every time alarm is triggered, DRS checks triggered parameter to validate user powered off
event.
• Virtual Machine OVF format is being used to manage compatibilty across different Hypervisors, hence our
system uses OVF export and import to another vHost while migrating VM.
Screenshots:
1.) Pinging VM
2). System starts Recovery Manager when Machine miss 5 continuous pings.
3.) Not initiating the Recovery process when user has turned it off.

6. 4.) Creating Snapshot of Virtual Machine and vHost.
5.) DRS reverting to most recent snapshot when VM is dead and vHost is alive.

7. 6.) when vHost is dead and VM is alive, reverting to host's most recent snapshot.
7.) Adding a vHost to vCentre.

8. 6. Discussions:
• The host add/remove mechanism
Host has been added to vCentre only in the situation when current host is dead even after being reverted to
most recent snapshot. System takes the name of vHost available and tries to add that vHost to current
vCentre.
• The approach used to configure the failure detection for each VM
Failure detection has been automated by keeping a count of missed heartbeat, currently 5 missed heartbeat
initiates the recovery manager. Each thread is responsible for pinging VM inside inventory. Each pinging
thread has the right to notify Recovery Manager about VM failure.
• How host failures were detected
Host failures uses same mechanism for failures. Host is pinged only once the VM doesn't respond to check
if Host has failed.
• The mechanism used to convert between the image formats used by the hypervisors.
OVF format are being used for exporting VM images out of vHost. Since OVF provides interoperability
between all hypervisors, its pretty good idea to export VM.
7. Conclusion :
We can communicate with one ESX server with web service as well as many ESX servers with vCentre
server, both provides capabilities as managed entities, required to automate infrastructure provisioning and
management. This project gave me pretty good exposure of VMware api and Hypervisor capabilities.
Apart from all other learnings, Java multithreading and its features was a great learning. For future
projects I would try to create and use CacheInstance, as it seems to be very effective when application has
to scale and is suppoed to support a major infrastructure.