The document discusses cluster management tools for big data applications, illustrated through examples such as Cloudera Manager and Hortonworks Ambari. It emphasizes the importance of managing deployment across environments, monitoring systems, and facilitating the big data lifecycle through various tools like HDFS, Hive, Spark, and more. Additionally, it compares features and pros and cons of different management tools for Hadoop environments.

1. by Ran Silberman
DevOps for Big Data
Cluster management tools
20.4.2015
Hosted by:
FullStack Developers Israel

2. Ran Silberman,
Big Data Architect
...and amateur birder

3. ● Explain Cluster Management
tools by example
● Demo Cloudera Management
● Pros and Cons
Agenda

4. Birds of Brazil Wiki application

6. ● Input photos and locations
● Batch: Display statistics on bird,
location & photographer.
● Real-time: Count how many birds
were seen in the last minute from
each species
Application
requirements

7. ● Volume growth
● Velocity of Streaming and Batch
● Same env from DEV to PROD
● Data from PROD to test on DEV
● Manage Deployment of many
applications on many nodes
Big Data lifecycle
considerations

8. ● HDFS for storing the data
● Hive for batch processing
● Solr/elasticsearch for search
● Spark for streaming
● ...Home-grown applications
Choosing the
Infrastructures

9. Many Infrastructures

10. How can we manage all those
infrastructures?

11. ● Hortonworks Ambari
or
● Cloudera Manager
Choosing the
Management
tool

12. ● All platforms & infrastructures are
installed by the tool
● Monitoring, Audits & logs are
built-in
● Easy installation and upgrade
● Save scripting work
What are the
news for DevOps
pipeline?

13. ● Manage cluster with GUI or API
● Hadoop installation and setup
● System monitoring & alerts
● Built-in systems: Zookeeper,
Spark, Hive Impala and more
● Ability to add parcels
CM features

14. ● Monolithic packages
● Relocatable
● sudo-less installs
● Rolling upgrade
Parcels

15. Custom Service Descriptors
● CSD is a descriptor for a service
used by CM
● Defines how to install start/stop
a service and the logic used by
CM
CSD

16. Demo

17. ● Archive data in Hadoop
● Growing data affects DWH
performance & capabilities
● Creating realistic testing data
● Dev and Prod env. may differ in
cluster size (dev may be 1 node)
More DevOps
considerations

18. Tools Comparison
CM Ambari
Licence Paid Ent edition Free Apache Open Source
Technology Cloudera puppet, ganglia, nagios
Dependency CDH HDP
Manage cluster Parcels Yum
REST API + +
Extra Features Rolling Upgrade, 3rd-
parties Mngt,
Extendable by REST API

19. CM features
Express Enterprise
Subscription Free Annual
Deployment &
Configuration
+ +
Management + +
Monitoring + +
Diagnostic + +
Extra Features Reports, Rollbacks, Rolling
Upgrade, AD Kerberos, Kerberos
wizard, Backup & DR

20. ● Fast Deploy
● Easy management by GUI
● Built in monitoring and alerts
● Simple upgrades
● Same management and deploy
in Dev and Prod
Pros. of Hadoop
Management
tools

21. ● Tied to specific vendor
proprietary system
● Tied to system version by
Parcels
● Less flexibility to low-level
management
Cons. of Hadoop
Management
tools

22. THANK YOU
Ran Silberman
Email: ran@tikalk.com