Downloaded 52 times
![The
Code
#!/bin/bash
name
:
“spark”,
roles
:
[{
name
:
"master",
CMD=$1
MASTER_PORT=<read
in
from
./params.proper;es>
startRunner
:
{
program
:
"scripts/control.sh",
args
:
[
"start_master",
case
$CMD
in
(start_master)
exec
$SPARK_HOME/scripts/spark-‐start.sh
master"
"./params.proper;es"]
},
;;
parameters
:
[{
(*)
echo
"$;mestamp
Don't
understand
[$CMD]"
name
:
"master_port",
type
:
"port",
;;
esac
default
:
7077
}],
configWriter
:
{
generators
:
[{
filename
:
"params.proper;es"
}]
}]
©2013Cloudera, Inc. All Rights Reserved.](https://image.slidesharecdn.com/pa-clouderamanagerapisextensibilityv2-131212115623-phpapp01/85/Extending-and-Automating-Cloudera-Manager-via-API-18-320.jpg)
More Related Content
![[@IndeedEng] Large scale interactive analytics with Imhotep](https://cdn.slidesharecdn.com/ss_thumbnails/largescaleinteractiveanalyticswithimhotep-140507102130-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Extending and Automating Cloudera Manager via API
Extending and Automating Cloudera Manager via API
- 1. Cloudera Manager – API’s & Extensibility Patrick Angeles, Director Field Technical Services December 2013 1 CONFIDENTIAL -‐ RESTRICTED
- 2. Cloudera Manager End-‐to-‐End AdministraHon for CDH 1 Monitor 2 Diagnose 3 Integrate 4 Manage Easily deploy, configure & opHmize clusters Maintain a central view of all acHvity Easily idenHfy and resolve issues Use Cloudera Manager with exisHng tools 2 ©2013 Cloudera, Inc. All Rights Reserved.
- 3. IntegraHng with your IT Mgmt tools Datacenter Opera*ons Various op*ons of integra*ng Cloudera Manager into your exis*ng Installa;on, Datacenter Opera*ons/Tools Monitoring Deployment Aler;ng Tools tools Tools e.g. Orion, • Cloudera Manager API e.g. Chef, e.g Nagios, Tivoli, BMC Puppet etc. SNMP etc. • Introduced in CM4 (June 2etc. 012) • Installa*on & deployment • Monitoring • SNMP Alerts • Introduced in CM4.5 (Feb 2013) • Hadoop Opera*ons And more… Cloudera • Monitoring ‘tsquery’ (Feb 2013) Manager • User-‐defined triggers/alarms (new for C5!) • Service extensibility (new for C5!) 3 ©2013 Cloudera, Inc. All Rights Reserved.
- 4. Cloudera Manager (CM) API • • 4 API access was a new feature introduced in Cloudera Manager 4.0, providing programmaHc access to cluster operaHons (such as configuraHon and restart) and monitoring informaHon (such as health and metrics). The CM API is an HTTP REST API, using JSON serializaHon. The API is served on the same host and port as the CM web UI, and does not require an extra process or extra configuraHon. API users have the same privileges as they do in the web UI world. • Docs & Examples hZp://cloudera.github.io/cm_api/ hZps://github.com/cloudera/cm_api • Java/Python clients hZp://blog.cloudera.com/blog/2013/05/how-‐to-‐ automate-‐your-‐hadoop-‐cluster-‐from-‐java/ ©2013Cloudera, Inc. All Rights Reserved.
- 5. Examples of integraHon with CM API • Installa;on & Deployment • • • Chef Puppet Dell Crowbar • • hZp://blog.cloudera.com/blog/2013/08/how-‐to-‐deploy-‐hadoop-‐clusters-‐automaHcally-‐with-‐dell-‐crowbar-‐and-‐cloudera-‐manager/ StackIQ • hZp://web.stackiq.com/blog/bid/312064/StackIQ-‐Cluster-‐Manager-‐now-‐integrated-‐with-‐Cloudera • • • WANdisco – non-‐stop NN setup Several other customers/partners leveraging the API’s as part of their install & deployment process Monitoring & Aler;ng • • Oracle Enterprise Manager (via Big Data Appliance) Nagios • • • hZps://github.com/cloudera/cm_api/tree/master/nagios hZps://github.com/harisekhon/nagios-‐plugins/blob/master/ check_hadoop_cloudera_manager_metrics.pl SNMP alerts integraHon with IBM Netcool Develop & Contribute your plug-‐in’s using Cloudera Manager API 5 ©2013 Cloudera, Inc. All Rights Reserved.
- 6. Cloudera Manager – Monitoring via ‘tsquery’ • Introduced as part of CM4.5 release (Feb 2013) • Great way to add interesHng charts (above & beyond what is provided by default) and monitor metrics that are relevant to your clusters • • The tsquery language s used to Manager Hme-‐series diata store specify statements for retrieving Hme-‐series data from the Cloudera Example: How do I compare all disk IO for all the DataNodes that belong to a specific HDFS service? select bytes_read, bytes_wriZen where roleType=DATANODE and serviceName=hdfs1 • Retrieved Hme-‐series data can be ploZed via various opHons – line, bar, scaZer, heat maps, table list etc. • Extending this concept to create user-‐defined triggers/alarms (new for C5!). More details • hZp://www.cloudera.com/content/cloudera-‐content/cloudera-‐docs/CM5/latest/Cloudera-‐ Manager-‐DiagnosHcs-‐Guide/cm5dg_chart_Hme_series_data.html • 6 ©2013 Cloudera, Inc. All Rights Reserved.
- 7. Examples of Cloudera Manager ‘tsquery’ Example1: How do I track the aggregate Cluster Disk IO? select dt0(read_bytes_disk_sum), dt0(write_bytes_disk_sum) where category = CLUSTER and clusterId = $CLUSTERID Example2: How do I compare CPU usage across hosts? select dt0(total_cpu_user) / getHostFact(numCores, 1) * 100, dt0(total_cpu_system) / getHostFact(numCores, 1) * 100, dt0(total_cpu_nice) / getHostFact(numCores, 1) * 100, dt0(total_cpu_iowait) / getHostFact(numCores, 1) * 100, dt0(total_cpu_irq) / getHostFact(numCores, 1) * 100, dt0(total_cpu_so`_irq) / getHostFact(numCores, 1) * 100 Create & Contribute your ‘tsqueries’! hZps://github.com/cloudera/cm_charHng_scrapbook 7 ©2013 Cloudera, Inc. All Rights Reserved.
- 8. Cloudera Manager – Service Extensibility • Introduced in C5 • SHll in Beta! • Some aspects (espcially Parcel mgmt) available in CM4.x • Example: CollaboraHon with Syncsort to deploy DMX-‐h libraries • Single management console for CDH, non-‐CDH services and ISV applicaHons • Similar look and feel as exisHng services • Easy to write (Java-‐free!) • Flexible • Independent release cycle ©2013Cloudera, Inc. All Rights Reserved.
- 9. Analogy from OperaHng Systems (OS) world ISV’s view of OS Systems Management Package Mgmt Process/ Resource Mgmt Security Mgmt Core OS kernel 9 ©2013Cloudera, Inc. All Rights Reserved. Data Access Mgmt
- 10. Bringing ISV Apps to CDH ISV’s view of Hadoop Cloudera Manager Parcels Resource Mgmt Security Mgmt CDK API’s Core Hadoop/CDH kernel 10 ©2013Cloudera, Inc. All Rights Reserved.
- 11. IntegraHng into the Cloudera Product Porpolio Features Examples -‐ Ability to easily package and distribute binaries/ jars via “Parcels” -‐ InformaHca -‐ Syncsort Resource Mgmt -‐ Ability to deploy applicaHons as stand-‐alone processes or via YARN* on the Hadoop grid -‐ Resource isolaHon of cluster resources -‐ SAS -‐ 0xData -‐ Accumulo Security Mgmt Cloudera Manager Descrip;on Package Mgmt ISV’s -‐ Support for Kerberos Mgmt -‐ Role bases access control for Tables/Views in Hive/Impala via Sentry Data Access Mgmt -‐ HDFS and HBase API abstracHon and simplificaHon Systems Mgmt Manage Monitor Non-‐CDH Apps… Accumulo, Spark, Giraph etc. -‐ Deploy and upgrade (rolling) services and pkgs -‐ Manage configuraHons -‐ ProacHve health checks -‐ Track resource uHlizaHon -‐ Custom metrics charts Diagnose -‐ Distributed log collecHon and searching -‐ Tag and track key events Integrate -‐ Access operaHonal tools via API -‐ Surface overall cluster metrics to ISV dashboard * Support for YARN planned as part of CM5.x in FY14 11 ©2013Cloudera, Inc. All Rights Reserved.
- 12. So.. How does it work? • A JSON file that describes of your service • Set of control scripts • Packaged as a JAR file • As promised, Java-‐free ©2013Cloudera, Inc. All Rights Reserved.
- 13. Example: Cloudera Manager Extensions -‐ Spark ©2013Cloudera, Inc. All Rights Reserved.
- 14. Cloudera Manager Extensions ©2013Cloudera, Inc. All Rights Reserved.
- 15. Cloudera Manager Extensions: Spark ©2013Cloudera, Inc. All Rights Reserved.
- 16. Cloudera Manager Extensions: Spark ©2013Cloudera, Inc. All Rights Reserved.
- 17. Cloudera Manager Extensions: Spark ©2013Cloudera, Inc. All Rights Reserved.
- 18. The Code #!/bin/bash name : “spark”, roles : [{ name : "master", CMD=$1 MASTER_PORT=<read in from ./params.proper;es> startRunner : { program : "scripts/control.sh", args : [ "start_master", case $CMD in (start_master) exec $SPARK_HOME/scripts/spark-‐start.sh master" "./params.proper;es"] }, ;; parameters : [{ (*) echo "$;mestamp Don't understand [$CMD]" name : "master_port", type : "port", ;; esac default : 7077 }], configWriter : { generators : [{ filename : "params.proper;es" }] }] ©2013Cloudera, Inc. All Rights Reserved.
- 19. Next Steps • DocumentaHon & SDK as part of C5 Beta2 or later (definitely before GA!) • Working with select ISV’s (SAS, Syncsort, 0xData etc.) as part of Beta to further fine-‐ tune this feature Develop & Contribute your Cloudera Manager service extensibility plug-‐in’s ! ©2013Cloudera, Inc. All Rights Reserved.
- 20. Horizontal Extension Security ISV’s 0xData SAS Syncsort Informatica Revolution API Ops Apps Capacity Mgr ServiceExtensibility Vertical Extension Vision of CM Extensibility SLA Mgr Cost Optimizer CDH CM SNMP API Oracle OEM 20 Nagios Dell Chef/ Puppet ©2012Cloudera, Inc. All Rights Reserved. Accumulo Spark Giraph
- 21. Q&A ©2013Cloudera, Inc.All Rights Reserved.