Your SlideShare is downloading. ×
Effective Hadoop Cluster Management - Impetus Webinar
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Effective Hadoop Cluster Management - Impetus Webinar

786
views

Published on

Effective Hadoop Cluster Management - Impetus Webinar …

Effective Hadoop Cluster Management - Impetus Webinar

Register at http://bit.ly/vXPfpd

Published in: Technology, Health & Medicine

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
786
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Impetus Technologies Inc. 1 © 2014 Impetus Technologies Effective Hadoop Cluster Management Recorded version available at http://www.impetus.com/webinar_registration?event=archived&eid=51
  • 2. Outline • HadoopTM - Quick overview • HadoopTM clusters • Addressing the HadoopTM cluster challenges – Operational – Cluster Setup – Cluster Management – Cluster Sharing • Quick provisioning of Cloud-based HadoopTM clusters • Available solutions for cluster setup and management 2 © 2014 Impetus Technologies Recorded version available at http://www.impetus.com/webinar_registration?event=archived&eid=51
  • 3. Hadoop Overview  Hadoop  Nodes, roles, services  Configurations & modes  Hadoop sub-components Feature / Cluster Mode 3 © 2014 Impetus Technologies Production Development & Testing usage Distributed Pseudo Distributed Standalone How Services Run Truly Distributed Single Machine - Separate Java processes Single Java Process HDFS Yes Yes No (Uses local File System) Replication Yes No No XML Conf Init required Yes Yes No Services Start/Stop req. Yes Yes No Node addition Yes No No Recorded version available at http://www.impetus.com/webinar_registration?event=archived&eid=51
  • 4. Hadoop Cluster- The Pain Areas • Operations • Cluster setup • Cluster management • HadoopTM compatibility • Cluster sharing 4 © 2014 Impetus Technologies Recorded version available at http://www.impetus.com/webinar_registration?event=archived&eid=51
  • 5. Hadoop Cluster- What is Missing? • Automation • Setup support • User friendly interface • Cluster management capabilities – Centralized management for multiple clusters • Monitoring & notifications support • Provision for cluster sharing 5 © 2014 Impetus Technologies Recorded version available at http://www.impetus.com/webinar_registration?event=archived&eid=51
  • 6. Hadoop Cluster- Key Challenges Operations Challenges – Manual operations – Console based interface Addressing Challenges – Automated activities – User friendly GUI as an alternate interface 6 © 2014 Impetus Technologies Recorded version available at http://www.impetus.com/webinar_registration?event=archived&eid=51
  • 7. Hadoop Cluster- Key Challenges Cluster Setup Challenges  Manual execution  Interactive setup  Setup procedure varies as per the situation  Set-up environment - Cloud & local clusters Addressing the challenges  Context aware, non-interactive automation  HadoopTM components initialization  Click based cluster provisioning 7 © 2014 Impetus Technologies Recorded version available at http://www.impetus.com/webinar_registration?event=archived&eid=51
  • 8. Quick Provisioning of Cloud Clusters Understanding the difference Action Cloud Clusters Local Clusters Instances Launch Required --- N.A. --- Required additional Info 8 © 2014 Impetus Technologies Machine image, Instance type, Security groups, Key Cluster machine types Homogeneous/ Heterogeneous Homogeneous /Heterogeneous OS Normally same Could be different Java Same on all Could be different Password-less SSH Setup Already Exists Setup Required Hadoop Components Setup of components nearly remains same in both case Instances Termination Required, if requested --- N.A. --- Recorded version available at http://www.impetus.com/webinar_registration?event=archived&eid=51
  • 9. Quick Provisioning of Cloud Clusters Challenges – Manual operation – Efforts increases with cluster size – Interface switching Addressing the Challenges • Automated provisioning – Scripts – Cloud APIs 9 © 2014 Impetus Technologies Recorded version available at http://www.impetus.com/webinar_registration?event=archived&eid=51
  • 10. Hadoop Cluster- Key Challenges Cluster Management Challenges  Lack of appropriate management support Addressing the challenges  Support for managing  Configuration  Nodes & services  Jobs  Keys & users  Monitoring & notifications 10 © 2014 Impetus Technologies Recorded version available at http://www.impetus.com/webinar_registration?event=archived&eid=51
  • 11. Hadoop Cluster- Key Challenges HadoopTM Compatibility & Issues Challenges  Version compatibility  Cluster modes  Version, vendor, bundle format differences  Vendor specific features Addressing the challenges  Workaround solution for compatibility issues  Automation for version, vendor, bundle format  Context aware management support  Vendor specific tools 11 © 2014 Impetus Technologies Recorded version available at http://www.impetus.com/webinar_registration?event=archived&eid=51
  • 12. Hadoop Cluster- Key Challenges Cluster Sharing Challenges  Ways of sharing  No sharing / Sharing with limited security Addressing these challenges  Sharing through alternate interface  Purpose specific role based sharing 12 © 2014 Impetus Technologies Recorded version available at http://www.impetus.com/webinar_registration?event=archived&eid=51
  • 13. How HCM Tools can Help? • Quick turnaround • Helps to minimize time, cost & efforts • Integrated support (setup, management & monitoring) • Easy execution • Proven usefulness in production & testing • Adhering to best practices 13 © 2014 Impetus Technologies Recorded version available at http://www.impetus.com/webinar_registration?event=archived&eid=51
  • 14. How HCM Tools can Help? Setup Automation for setup of Cluster Components local & cloud environments 14 © 2014 Impetus Technologies Management Nodes, Services, Configuration, Keys, Users, Jobs, Job Templates, Performance Tracking Upgrade, Auditing, Monitoring, Import/Export, Event Mail/Alerts/ Notifications, Active working Sharing Alternate access interface Role based access Security preserved Recorded version available at http://www.impetus.com/webinar_registration?event=archived&eid=51
  • 15. Some Available Options • Amazon Elastic Map-Reduce • Whirr • Cloudera SCM • Impetus Ankush 15 © 2014 Impetus Technologies Recorded version available at http://www.impetus.com/webinar_registration?event=archived&eid=51
  • 16. Ankush- Impetus HCM Tool 16 © 2014 Impetus Technologies Recorded version available at http://www.impetus.com/webinar_registration?event=archived&eid=51
  • 17. Ankush: Key Features • Centralized multiple cluster management • Common interface for cloud and local clusters • Integrated support for setup, management & cloud provisioning • Anytime-anywhere access to cluster functionalities • Cloud independent • Reduces cluster setup time by 60 - 80% • Bundle setup optimization 17 © 2014 Impetus Technologies Recorded version available at http://www.impetus.com/webinar_registration?event=archived&eid=51
  • 18. Summary • Automation facilitates smooth HadoopTMworking – Eases cloud cluster provisioning • Alternate UI enables secure cluster sharing • HadoopTM still in evolving stage • Cluster management tools for pertinent problem areas 18 © 2014 Impetus Technologies Recorded version available at http://www.impetus.com/webinar_registration?event=archived&eid=51
  • 19. 19 © 2014 Impetus Technologies About Impetus
  • 20. Our Expertise • Strategic partners for software product engineering and R&D • Thought leaders in cutting-edge technologies • Mature processes and practices that are methodical, yet flexible • Diverse domain expertise 20 © 2014 Impetus Technologies Recorded version available at http://www.impetus.com/webinar_registration?event=archived&eid=51
  • 21. 21 © 2014 Impetus Technologies Q & A
  • 22. 22 © 2014 Impetus Technologies Thank You Write to us at inquiry@impetus.com Follow us on Twitter @impetustech Recorded version available at http://www.impetus.com/webinar_registration?event=archived&eid=51