Operate your hadoop cluster like a high eff goldmine

Operate Your Hadoop Cluster
Like a High-Efficiency
Data Goldmine
Greg Bruno
VP Engineering & Co-founder / StackIQ
Thursday 14 June 2012

© 2012 StackIQ. All rights reserved. 1

The California Gold Rush


Some will succeed wildly…

…while others will fail miserably.


It pays to
start out with
a plan…


You Need a Repeatable Process to
Successfully Deploy Hadoop Clusters

1 2 3
Select
Operate
Provision


1
Select Commodity Hardware
Three Server Types
Networking Considerations


•  Standard high volume equipment = “best
bang for the buck”
•  Flexible configuration
Commodity You pick the CPUs, memory and disks that make
Hardware sense for your deployment

•  Easy to “tune” your configuration
Need more disk caching? Add memory!
CPU pegged? Upgrade CPU!
Need more disk I/Os per second? Add spindles
or faster spinning drives!


Choose Servers Suited to
their Purpose in the Cluster

Name Nodes
Maintain metadata

Data Nodes
Lots of big disks
to store all the data

Cluster Manager
Provisions and monitors
name nodes and data nodes
Needs much less disk
storage than data nodes


Networking Factors to Consider
•  Hadoop is designed to access
data locally from data nodes so
a high-performance network is
often not needed
•  Standard Gigabit Ethernet
generally works well


Single Rack Configuration
648 TB of raw storage à 216 TB usable!

Gigabit Ethernet switch
Dell PowerConnect 5524

Cluster Manager node
Dell PowerEdge R410
16 GB memory
2 x 1 TB disks

18 Data Nodes
Dell PowerEdge R720xd
96 GB memory
12 x 3 TB 3.5” disks

Name Nodes
Dell PowerEdge R720xd
96 GB memory
12 x 3 TB 3.5” disks

Multi-Rack Configuration


Expansion Rack

•  Gigabit Ethernet Switch

•  20 Data Nodes
•  720 TB of raw storage à 240 TB usable


Interconnection Rack

•  10 Gigabit Ethernet switch
Dell PowerConnect 8024F

•  Gigabit Ethernet switch

•  20 Data Nodes

•  720 TB of raw storage à 240 TB usable


5 Racks = 1.1 Petabyte of Usable Storage!


2
Starting with Bare Metal
Hadoop Dependencies
Provision
System Test


Starting with Bare Metal
•  Assume nothing
Our Hadoop provisioning tool installs all
the bits (e.g., OS, libraries, Hadoop
software) and configures all the services
(e.g., network, firewall, disks, Hadoop
services).
Other Hadoop provisioning tools assume
all nodes in the cluster have a base OS
installed and they are up on the network

•  Step 1: Install Cluster Manager
node
•  Step 2: Install backend nodes
Cluster Manager node is used to install all
backend nodes


Install Cluster Manager
•  Boot the Cluster Manager node
with the StackIQ Enterprise Data
DVD
•  Input basic information about the
node (e.g., FQDN, public IP
address, timezone, etc.)
•  Installer copies content of DVD to
the local disk and node installs
itself from that repository
•  After the node boots it is ready to
install the backend nodes


Install Backend Nodes
•  Set boot order to:
1.  PXE
2.  Local hard disk

•  Put Cluster Manager node into
“discover” mode
•  Boot backend nodes
Nodes are discovered by the Cluster
Manager, auto-assigned a name and
an IP address, then sent a kickstart
file
Discovery and installation are fully
automated


Hadoop Dependencies
•  After backend nodes install and reboot, they are up on the
network, with all the Hadoop software installed
And all the required software that the Hadoop services need are installed too

•  The Cluster Manager has information about every backend node
Name / IP
Disk parameters
Number of CPUs

•  Node info is used to auto-tune the backend nodes, specifically,
the Hadoop services


Now It’s Time
for a Basic
System Test
•  Do all nodes
respond to “ssh”?
•  Login to each node via
ssh and execute
“uptime”:


3
Service Configuration
Operate Service Monitoring
Service Management


Configuring Hadoop Services
•  The hardware has been racked and cabled

•  All the software is in place
•  Time to define and start Hadoop Services
HDFS
MapReduce
ZooKeeper
HBase


clustermgr

compute-0-0 compute-1-0

compute-0-1 compute-1-1

Configuring compute-0-2

HDFS rack 0 rack 1


Configuring HDFS


Configuring MapReduce


Configuring ZooKeeper


Configuring Hbase


Monitoring Hadoop


Ganglia Status


HDFS Status


HDFS Data Movement Visualization

Node Color Rack Membership

Vector Color Operation (read or write)

Line Width Data Volume


Dealing with
the Inevitable
•  No matter how well planned,
every cluster will need to
undergo changes
Expansion (add)
Failures (replace)
Changes (reconfigure)

•  We manage all the above with
one operation – install!
Expansion (discover and install!)
Failure (replace node and install!)
Changes (add packages to the
distribution and reinstall!)


We Just Went
From Zero to
Hadoop Cluster!
•  You now have a blueprint to:
Procure hardware
Provision an entire software stack
OS, libraries, cluster
management services, Hadoop
software, etc.
Configure Hadoop services

•  Now go build your own Hadoop
cluster and start mining the gold
in your data!


Operate your hadoop cluster like a high eff goldmine

Operate your hadoop cluster like a high eff goldmine

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Operate your hadoop cluster like a high eff goldmine

Similar to Operate your hadoop cluster like a high eff goldmine (20)

More from DataWorks Summit

More from DataWorks Summit (20)

Recently uploaded

Recently uploaded (20)

Operate your hadoop cluster like a high eff goldmine