This presentation will investigate the advantages and disadvantage of running an n-node Hadoop cluster in each of the following environments: • Bare metal • Private cloud using a traditional virtual machine environment • Private cloud using a containerized virtual environment Tom will follow this with a discussion of real Hadoop Use Cases and how each would, or would not, be suitable for running in each environment.

1. A Discussion of Hadoop Use Cases &
Runtime Environments
Tom Phelan, Chief Architect of BlueData
Los Angeles Hadoop Users Group
Sept 25, 2014

2. … or when should I virtualize my Hadoop cluster?

3. First - Some Definitions

4. Physical Hadoop Cluster
 AKA “bare metal” installation
 The Hadoop distribution is installed as an application on top of the operating
system.
 A set of physical servers run the various Hadoop services, forming the Hadoop
cluster.
• File System (HDFS, NameNode etc)
• Processing Framework (JobTracker etc)
 Original design goal was reduced Cost and not necessarily improved
performance.

5. Physical Hadoop Cluster
NameNode JobTracker
Server
Disk Disk Disk
DataNode TaskTracker
Server
Disk Disk Disk
DataNode TaskTracker
Server
Disk Disk Disk
Controller
Worker
Worker

6. Virtual Hadoop Cluster
 The Hadoop distribution is installed as an application running within the
context of a collection of virtual machines .
 A virtual machine is software that presents an abstraction that is
identical to the underlying hardware. In general, the software running
within the VM cannot tell the difference from a physical server.
 If the collection of virtual machines is spread across more than one
physical server, it is typically referred to as a cloud,. The cloud can be
either public or private.
 The type of virtualization technology used can be one of :
• Type I Hypervisor , VMW ESX
• Type II Hypervisor , KVM
• Linux Containers, LXC

7. Virtual Hadoop Cluster – Public Cloud
 IaaS – infrastructure as a service
 The type of virtualization is unknown
 Typically the physical hosts are not located within the enterprise data center
 Data security can be an issue
 Can be expensive
Examples: AWS, Azure

8. Virtual Hadoop Cluster – Private Cloud
 IAAS – infrastructure as a service
 The type of virtualization is known but not specified
 Typically the physical hosts are located within the enterprise data center
 Data security enforced by the enterprise
 Can be expensive
Examples: VMware vSphere, OpenStack. CloudStack

9. Virtual Hadoop Cluster – Private Cloud -
Hypervisor
 IAAS – infrastructure as a service
 The type of virtualization is Type I or II hypervisor. Generically referred to
as “hypervisor” or “virtual machine”
 Typically the physical hosts are located within the enterprise data center
 Data security enforced by the enterprise
 Strong fault isolation - a fault in the VM cannot cause the physical cluster
to crash
 Strong resource partitioning
 Moderate amount of “overhead” to implement the virtualization.
 Can be expensive
Examples: VMW Vsphere, OpenStack

10. Virtual Hadoop Cluster – Private Cloud -
Containers
 IAAS – infrastructure as a service
 The type of virtualization is Linux Containers, LXC
 Typically the physical hosts are located within the enterprise data center
 Data security enforced by the enterprise
 Currently weak fault isolation - a fault in the VM can the physical cluster
to crash
 Moderate resource partitioning
 Low amount of “overhead” to implement the virtualization.
 Can be expensive
Examples:Docker, Mesos, CoreOS, LXC

11. Virtual Hadoop Cluster - Hypervisor
Controller VM
NameNode JobTracker
Hadoop
Server
vDisk vDisk vDisk
Worker VM
Cloud
Server
Disk Disk Disk
Cloud
Server
Worker VM
Disk Disk Disk Cloud
Server
Disk Disk Disk
Cloud
Server
Disk Disk Disk
DataNode TaskTracker
Hadoop
Server
vDisk vDisk vDisk
DataNode TaskTracker
Hadoop
Server
vDisk vDisk vDisk

12. Virtual Hadoop Cluster - Containers
JobTracker
Controller
Container
Cloud
Server
Disk Disk Disk
Cloud
Server
Disk Disk Disk Cloud
Server
Disk Disk Disk
Cloud
Server
Disk Disk Disk
TaskTracker
Worker
Container
TaskTracker
Worker
Container
NameNode
DataNode
DataNode
DataNode

13. Virtual Hadoop Cluster- Hypervisors
Cloud
Server
Disk Disk Disk
Cloud
Server
Disk Disk Disk Cloud
Server
Disk Disk Disk
Cloud
Server
Disk Disk Disk

14. Virtual Hadoop Cluster – Private Cloud – Data
Para Virtualization
 Paravirtualization means that the abstraction the virtualization software provides is
similar, but not identical, to the underlying hardware.
 The differences are designed to reduce the virtualization “overhead” by taking
advantage of some knowledge about the tasks running in the virtual machine.
Examples: BlueData

15. Virtual Hadoop Cluster – Paravirtualization
Controller VM
NameNode JobTracker
Hadoop
Server
vDisk vDisk vDisk
Worker VM
Cloud
Server
Disk Disk Disk
Cloud
Server
Worker VM
Disk Disk Disk Cloud
Server
Disk Disk Disk
Cloud
Server
Disk Disk Disk
DataNode TaskTracker
Hadoop
Server
vDisk vDisk vDisk
DataNode TaskTracker
Hadoop
Server
vDisk vDisk vDisk
Data
Connection
Data
Connection
Data
Connection
NFS HDFS
GlusterFS

16. In which situations should an enterprise run
their Hadoop jobs in a virtual or physical
environment?

17. Evaluation based on:
Faster …
– Deployment
– Runtime
Easier …
– Deployment
– Management
Cheaper …
– Hardware costs
– Management costs

18. Questions not to ask
•How fast does the job need to run?
•How much does the cluster cost?
•How easy is it to use?
Any application can be run with the needed speed in either a
virtual or physical environment if enough money is spent.
Any tool is easy to use once you are familiar with it.
Other attributes indicate if the best solution is with physical or
virtual clusters.

19. Answers
There are multiple clusters and each is lightly used.
– Virtual cluster
There is one cluster, it runs a single Hadoop query job. It
runs 7 x 24 and demands instant response.
– Physical cluster

20. Answers
Test & Dev environment where Hadoop clusters need to be
built quickly and have short lifespan. Each developer gets
their own cluster. No security concerns.
– Virtual cluster - LXC
An environment with multiple Hadoop applications constantly
running and requiring access to a common data set. No
expected change in applications or load.
– Physical cluster

21. Answers
IAAS environment with multiple external customers each with
different QoS agreements, Hadoop distros, and data
security needs.
– Virtual cluster - Hypervisor

22. Those scenarios are too easy!

23. What the obvious answers tell us:
 Situations that require many distinct Hadoop clusters, or clusters that require
frequent provisioning, or clusters that have a relatively short lifespan are well
suited for virtualized Hadoop.
 Flexibility and speed of cluster creation are critical.
 Situations that require few distinct Hadoop clusters, have long lifespans, and
static configurations are well suited for Bare Metal Hadoop.
 No reason to pay virtualization “tax” in exchange for flexibility.

24. Questions to ask
 How many clusters will be needed?
• Over what time span?
 What is the life span of the clusters?
 Will the clusters have idle time?
 What are the fault isolation needs?
 What is the source of the big data?

25. Other questions to ask*
 Are multiple levels of priority job priority required?
 Are multiple levels of data security required?
 Is resource usage tracking/billing required?
* The implementation of these may be different between different distributions of Hadoop
and so the level of effort to implement in virtual and physical environments may be different.

26. Use Case I
Large manufacturing company
– Internal Customers
Started out with one Hadoop cluster
– Success!
Soon everyone wanted one.
– Many lightly used
– Different configurations.
– “Cluster Sprawl”
Virtual Clusters
Either hypervisor or LXC could be used

27. Use Case II
Development group within a large tech company
– Internal Customers
Built out a physical Hadoop cluster.
– No data security requirement
– No expectation of growth in foreseeable future.
Single Physical Cluster

28. Use Case III
Large service company.
– Internal and External Customers
No use of Hadoop.
Fault containment and data security required
Small IT department tasked with all Hadoop support for the company.
– No clear way to predict growth.
Virtual Clusters - hypervisor

29. Use Case IV
Large tech company
– Internal Customers
Constant stream of low priority jobs
Bursty stream of very high priority, low latency, jobs
No future demand growth.
Very Hadoop savvy IT organization.
Two Physical Clusters
OR virtual clusters

30. Use Case V
Startup online service company
– Selling information gathered using unique Hadoop analytics
External Customers
Multiple data sources
Customer data security requirements
Rapid growth of customer base
No in-house Hadoop expertise
Virtual Clusters
Could benefit from paravirtualization

31. Q & A

32. Contact
Tom Phelan
tap@bluedata.com

33. Virtual Hadoop Cluster
NameNode Controller VM
JobTracker
Hadoop
Server
vDisk vDisk vDisk
Worker VM
Cloud
Server
Disk Disk Disk
Cloud
Server
Worker VM
Disk Disk Disk Cloud
Server
Disk Disk Disk
Cloud
Server
Disk Disk Disk
DataNode TaskTracker
Hadoop
Server
vDisk vDisk vDisk
DataNode TaskTracker
Hadoop
Server
vDisk vDisk vDisk