The document discusses challenges in managing Hadoop clusters and how tools can help address them. It outlines issues like manual operations, lack of centralized management and monitoring. Automating cluster setup and provisioning processes can help minimize time and costs. Hadoop cluster management tools provide integrated support for setup, management and monitoring of clusters across various environments while ensuring security and compatibility with different Hadoop versions. The webinar presents Ankush, Impetus' tool for centralized management of multiple Hadoop clusters on-premises and on cloud.

1. Impetus Technologies Inc.
1 © 2014 Impetus Technologies
Effective Hadoop Cluster
Management
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

2. Outline
• HadoopTM - Quick overview
• HadoopTM clusters
• Addressing the HadoopTM cluster challenges
– Operational
– Cluster Setup
– Cluster Management
– Cluster Sharing
• Quick provisioning of Cloud-based HadoopTM clusters
• Available solutions for cluster setup and management
2 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

3. Hadoop Overview
 Hadoop
 Nodes, roles, services
 Configurations & modes
 Hadoop sub-components
Feature /
Cluster Mode
3 © 2014 Impetus Technologies
Production Development & Testing usage
Distributed Pseudo Distributed Standalone
How Services Run Truly Distributed Single Machine - Separate Java processes Single Java Process
HDFS Yes Yes No (Uses local File System)
Replication Yes No No
XML Conf Init required Yes Yes No
Services Start/Stop req. Yes Yes No
Node addition Yes No No
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

4. Hadoop Cluster- The Pain Areas
• Operations
• Cluster setup
• Cluster management
• HadoopTM compatibility
• Cluster sharing
4 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

5. Hadoop Cluster- What is Missing?
• Automation
• Setup support
• User friendly interface
• Cluster management capabilities
– Centralized management for multiple clusters
• Monitoring & notifications support
• Provision for cluster sharing
5 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

6. Hadoop Cluster- Key Challenges
Operations
Challenges
– Manual operations
– Console based interface
Addressing Challenges
– Automated activities
– User friendly GUI as an alternate interface
6 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

7. Hadoop Cluster- Key Challenges
Cluster Setup
Challenges
 Manual execution
 Interactive setup
 Setup procedure varies as per the situation
 Set-up environment - Cloud & local clusters
Addressing the challenges
 Context aware, non-interactive automation
 HadoopTM components initialization
 Click based cluster provisioning
7 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

8. Quick Provisioning of Cloud Clusters
Understanding the difference
Action Cloud Clusters Local Clusters
Instances Launch Required --- N.A. ---
Required additional Info
8 © 2014 Impetus Technologies
Machine image, Instance type,
Security groups, Key
Cluster machine types Homogeneous/ Heterogeneous Homogeneous /Heterogeneous
OS Normally same Could be different
Java Same on all Could be different
Password-less SSH Setup Already Exists Setup Required
Hadoop Components Setup of components nearly remains same in both case
Instances Termination Required, if requested --- N.A. ---
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

9. Quick Provisioning of Cloud Clusters
Challenges
– Manual operation
– Efforts increases with cluster
size
– Interface switching
Addressing the Challenges
• Automated provisioning
– Scripts
– Cloud APIs
9 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

10. Hadoop Cluster- Key Challenges
Cluster Management
Challenges
 Lack of appropriate management support
Addressing the challenges
 Support for managing
 Configuration
 Nodes & services
 Jobs
 Keys & users
 Monitoring & notifications
10 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

11. Hadoop Cluster- Key Challenges
HadoopTM Compatibility & Issues
Challenges
 Version compatibility
 Cluster modes
 Version, vendor, bundle format differences
 Vendor specific features
Addressing the challenges
 Workaround solution for compatibility issues
 Automation for version, vendor, bundle format
 Context aware management support
 Vendor specific tools
11 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

12. Hadoop Cluster- Key Challenges
Cluster Sharing
Challenges
 Ways of sharing
 No sharing / Sharing with limited security
Addressing these challenges
 Sharing through alternate interface
 Purpose specific role based sharing
12 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

13. How HCM Tools can Help?
• Quick turnaround
• Helps to minimize time, cost & efforts
• Integrated support (setup, management & monitoring)
• Easy execution
• Proven usefulness in production & testing
• Adhering to best practices
13 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

14. How HCM Tools can Help?
Setup
Automation for
setup of Cluster
Components local &
cloud environments
14 © 2014 Impetus Technologies
Management
Nodes, Services,
Configuration, Keys,
Users, Jobs, Job
Templates,
Performance
Tracking
Upgrade, Auditing,
Monitoring,
Import/Export,
Event Mail/Alerts/
Notifications, Active
working
Sharing
Alternate access
interface
Role based access
Security preserved
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

15. Some Available Options
• Amazon Elastic Map-Reduce
• Whirr
• Cloudera SCM
• Impetus Ankush
15 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

16. Ankush- Impetus HCM Tool
16 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

17. Ankush: Key Features
• Centralized multiple cluster management
• Common interface for cloud and local clusters
• Integrated support for setup, management & cloud
provisioning
• Anytime-anywhere access to cluster functionalities
• Cloud independent
• Reduces cluster setup time by 60 - 80%
• Bundle setup optimization
17 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

18. Summary
• Automation facilitates smooth HadoopTMworking
– Eases cloud cluster provisioning
• Alternate UI enables secure cluster sharing
• HadoopTM still in evolving stage
• Cluster management tools for pertinent problem areas
18 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

19. 19 © 2014 Impetus Technologies
About Impetus

20. Our Expertise
• Strategic partners for software product engineering and
R&D
• Thought leaders in cutting-edge technologies
• Mature processes and practices that are methodical, yet
flexible
• Diverse domain expertise
20 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

21. 21 © 2014 Impetus Technologies
Q & A

22. 22 © 2014 Impetus Technologies
Thank You
Write to us at inquiry@impetus.com
Follow us on Twitter @impetustech
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51