Hadoop Cluster Management

Hadoop Cluster Management Dheeraj Kapur Principal Engineer, Yahoo! dheerajk@yahoo-inc.com

What it is…§  Workflow based system for cluster management.§  Completely modular & distributed design.§  Has its own JMX based library(can be used to monitor other services on cluster).§  Fully controllable from WebUI.§  Has command line utility for adhoc administration. 2

What it does…§  Manage clusters.§  Break fixing.§  Upgrades OS seamlessly.§  Consistency/efficiency of clusters.§  Proactive self-healing Model.§  User Management. 3

Manage Clusters§  Its has well defined workflow to manage clusters.§  No/Minimal human intervention required.§  Keep up efficiency of cluster.§  Keep track of Missing/Bad blocks on system.§  Well defined WebUI and Command line utility 4

System Overview 5

Workflow 6

Contd.. 7

Command Line Utility 8

Web Interface 9

Web Interface contd… 10

fixing bad/mal-performing nodesThese errors can lead to SLA miss or Job failures§  Takes care of Blacklisted JT nodes.§  Errors like high load average, wrong network speed.§  Parse system logs at X frequency (thru workflows) and look for patterns.§  Visit each node multiple times in a day and check health of node. 11

Upgrade OS§  Upgrade & rollback OS seamlessly.§  Upgrading on production, heavily used clusters. 12

Consistency & efficiency of clusters§  Keep track of cluster MR capacity§  Proactive Fixing of sick nodes, which can cause potential issues. 13

Introducing Proactive self-healing systemLet me set the ground for it.§  Wounded hosts Called Set A - Hosts having issues, but still in service (with degraded services), Which can cause potential SLA misses and job execution issues.(which we have seen in past)§  Fractured Hosts Called Set B - Hosts already in Break fix cycle and getting fixed§  All grid hosts Called Set X - all grid hosts healthy + fine§  Set A & B are sub-set of set X§  to find wounded hosts we have to scan entire infrastructure once a day.§  Calculate Symmetric difference b/w Set A & B, we will get actual wounded hosts needs service. 14

Proactive self-healing contd…. All Grid Hosts - X Set A Set B 15

Proactive self-healing contd…. 16

User Management§  We have one of the most complex and secure environment.§  User access and management is a complex task, due to the number of users, security constraints and complexity involved in provisioning access.§  Single request provisioning requires change at multiple places.§  Well defined workflow based system, where 100% automation is achieved.§  Great help during system audit and compliance. 17

Q&A 18

Thank You 19

Sessions will resume at 4:30pm Page 20

Hadoop Cluster Management

by Hadoop_Summit on Jun 20, 2012

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 8

Statistics

Hadoop Cluster Management Presentation Transcript