Cloudera	
  Manager	
  –	
  API’s	
  &	
  
Extensibility	
  	
  
Patrick	
  Angeles,	
  Director	
  Field	
  Technical	
  ...
Cloudera	
  Manager	
  
End-­‐to-­‐End	
  AdministraHon	
  for	
  CDH	
  

1
Monitor	
  
2
Diagnose	
  
3
Integrate	
  
4
...
IntegraHng	
  with	
  your	
  IT	
  Mgmt	
  tools	
  
Datacenter	
  Opera*ons	
  

Various	
  op*ons	
  of	
  integra*ng	
...
Cloudera	
  Manager	
  (CM)	
  API	
  
	
  
• 

• 

	
  
	
  

4

API	
  access	
  was	
  a	
  new	
  feature	
  introduce...
Examples	
  of	
  integraHon	
  with	
  CM	
  API	
  
• 

Installa;on	
  &	
  Deployment	
  
• 
• 
• 

Chef	
  
Puppet	
  ...
Cloudera	
  Manager	
  –	
  Monitoring	
  via	
  ‘tsquery’	
  
• 

Introduced	
  as	
  part	
  of	
  CM4.5	
  	
  release	...
Examples	
  of	
  Cloudera	
  Manager	
  ‘tsquery’	
  
Example1:	
  How	
  do	
  I	
  track	
  the	
  
	
  
aggregate	
  C...
Cloudera	
  Manager	
  –	
  Service	
  Extensibility	
  
• 

Introduced	
  in	
  C5	
  
• 

SHll	
  in	
  Beta!	
  

• 

S...
Analogy	
  from	
  OperaHng	
  Systems	
  (OS)	
  world	
  
	
  
	
  
ISV’s	
  view	
  of	
  OS	
  	
  

Systems	
  Manage...
Bringing	
  ISV	
  Apps	
  to	
  CDH	
  
	
  
	
  

ISV’s	
  view	
  of	
  Hadoop	
  	
  

Cloudera	
  Manager	
  
Parcels...
IntegraHng	
  into	
  the	
  Cloudera	
  Product	
  Porpolio	
  
Features	
  

Examples	
  

-­‐	
  Ability	
  to	
  easil...
So..	
  How	
  does	
  it	
  work?	
  
	
  
•  A	
  JSON	
  file	
  that	
  describes	
  of	
  your	
  service	
  
•  Set	
...
Example:	
  Cloudera	
  Manager	
  Extensions	
  -­‐	
  Spark	
  

	
  

	
  

©2013Cloudera, Inc. All Rights Reserved.
 

Cloudera	
  Manager	
  Extensions	
  
	
  

©2013Cloudera, Inc. All Rights Reserved.
Cloudera	
  Manager	
  Extensions:	
  Spark	
  

©2013Cloudera, Inc. All Rights Reserved.
Cloudera	
  Manager	
  Extensions:	
  Spark	
  

©2013Cloudera, Inc. All Rights Reserved.
Cloudera	
  Manager	
  Extensions:	
  Spark	
  

©2013Cloudera, Inc. All Rights Reserved.
The	
  Code	
  
	
  

	
  
#!/bin/bash	
  

name	
  :	
  “spark”,	
  
roles	
  :	
  [{	
  
	
  	
  	
  	
  	
  name	
  :	
...
Next	
  Steps	
  
•  DocumentaHon	
  &	
  SDK	
  as	
  part	
  of	
  C5	
  Beta2	
  
or	
  later	
  (definitely	
  before	
...
Horizontal Extension

Security
ISV’s

0xData

SAS

Syncsort

Informatica

Revolution

API

Ops Apps
Capacity
Mgr

Service ...
Q&A	
  

©2013Cloudera, Inc. All Rights Reserved.
Upcoming SlideShare
Loading in...5
×

Extending and Automating Cloudera Manager via API

877

Published on

As delivered by Patrick Angeles, Director of Field Technical Services

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
877
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
17
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Extending and Automating Cloudera Manager via API

  1. 1. Cloudera  Manager  –  API’s  &   Extensibility     Patrick  Angeles,  Director  Field  Technical  Services   December  2013     1 CONFIDENTIAL  -­‐  RESTRICTED  
  2. 2. Cloudera  Manager   End-­‐to-­‐End  AdministraHon  for  CDH   1 Monitor   2 Diagnose   3 Integrate   4 Manage   Easily  deploy,  configure  &  opHmize  clusters   Maintain  a  central  view  of  all  acHvity   Easily  idenHfy  and  resolve  issues   Use  Cloudera  Manager  with  exisHng  tools   2 ©2013  Cloudera,  Inc.  All  Rights  Reserved.  
  3. 3. IntegraHng  with  your  IT  Mgmt  tools   Datacenter  Opera*ons   Various  op*ons  of  integra*ng  Cloudera  Manager  into  your  exis*ng   Installa;on,   Datacenter  Opera*ons/Tools  Monitoring   Deployment   Aler;ng   Tools   tools   Tools   e.g.  Orion,     •  Cloudera  Manager  API   e.g.  Chef,   e.g  Nagios,   Tivoli,  BMC   Puppet  etc.   SNMP  etc.   •  Introduced  in  CM4  (June  2etc.   012)     •  Installa*on  &  deployment   •  Monitoring   •  SNMP  Alerts   •  Introduced  in  CM4.5  (Feb  2013)   •  Hadoop  Opera*ons   And  more…   Cloudera   •  Monitoring  ‘tsquery’  (Feb  2013)   Manager   •  User-­‐defined  triggers/alarms  (new  for  C5!)   •  Service  extensibility  (new  for  C5!)   3 ©2013  Cloudera,  Inc.  All  Rights  Reserved.  
  4. 4. Cloudera  Manager  (CM)  API     •  •      4 API  access  was  a  new  feature  introduced  in  Cloudera  Manager  4.0,  providing  programmaHc  access  to   cluster  operaHons  (such  as  configuraHon  and  restart)  and  monitoring  informaHon  (such  as  health  and   metrics).     The  CM  API  is  an  HTTP  REST  API,  using  JSON  serializaHon.  The  API  is  served  on  the  same  host  and  port  as   the  CM  web  UI,  and  does  not  require  an  extra  process  or  extra  configuraHon.  API  users  have  the  same   privileges  as  they  do  in  the  web  UI  world.   •  Docs  &  Examples   hZp://cloudera.github.io/cm_api/   hZps://github.com/cloudera/cm_api   •  Java/Python  clients   hZp://blog.cloudera.com/blog/2013/05/how-­‐to-­‐ automate-­‐your-­‐hadoop-­‐cluster-­‐from-­‐java/       ©2013Cloudera,  Inc.  All  Rights  Reserved.  
  5. 5. Examples  of  integraHon  with  CM  API   •  Installa;on  &  Deployment   •  •  •  Chef   Puppet   Dell  Crowbar   •  •  hZp://blog.cloudera.com/blog/2013/08/how-­‐to-­‐deploy-­‐hadoop-­‐clusters-­‐automaHcally-­‐with-­‐dell-­‐crowbar-­‐and-­‐cloudera-­‐manager/ StackIQ   •    hZp://web.stackiq.com/blog/bid/312064/StackIQ-­‐Cluster-­‐Manager-­‐now-­‐integrated-­‐with-­‐Cloudera   •  •  •  WANdisco  –  non-­‐stop  NN  setup   Several  other  customers/partners  leveraging  the  API’s  as  part  of    their  install  &  deployment     process   Monitoring  &  Aler;ng   •  •  Oracle  Enterprise  Manager  (via  Big  Data  Appliance)   Nagios   •  •  •  hZps://github.com/cloudera/cm_api/tree/master/nagios   hZps://github.com/harisekhon/nagios-­‐plugins/blob/master/ check_hadoop_cloudera_manager_metrics.pl   SNMP  alerts  integraHon  with  IBM  Netcool   Develop  &  Contribute  your  plug-­‐in’s  using  Cloudera   Manager  API     5 ©2013  Cloudera,  Inc.  All  Rights  Reserved.  
  6. 6. Cloudera  Manager  –  Monitoring  via  ‘tsquery’   •  Introduced  as  part  of  CM4.5    release  (Feb  2013)   •  Great  way  to  add  interesHng    charts  (above  &  beyond  what  is  provided  by  default)    and  monitor   metrics  that  are  relevant  to  your  clusters   •  •  The  tsquery  language   s  used  to    Manager  Hme-­‐series  diata  store   specify  statements  for  retrieving  Hme-­‐series  data  from  the  Cloudera     Example:  How  do  I  compare  all  disk  IO  for  all  the  DataNodes  that  belong  to  a  specific  HDFS  service?   select  bytes_read,  bytes_wriZen  where  roleType=DATANODE  and  serviceName=hdfs1   •  Retrieved  Hme-­‐series  data  can  be  ploZed  via  various  opHons  –  line,  bar,  scaZer,    heat  maps,  table  list   etc.   •  Extending  this  concept  to  create  user-­‐defined  triggers/alarms  (new  for  C5!).     More  details   •  hZp://www.cloudera.com/content/cloudera-­‐content/cloudera-­‐docs/CM5/latest/Cloudera-­‐ Manager-­‐DiagnosHcs-­‐Guide/cm5dg_chart_Hme_series_data.html   •  6 ©2013  Cloudera,  Inc.  All  Rights  Reserved.  
  7. 7. Examples  of  Cloudera  Manager  ‘tsquery’   Example1:  How  do  I  track  the     aggregate  Cluster  Disk  IO?     select  dt0(read_bytes_disk_sum),   dt0(write_bytes_disk_sum)  where   category  =  CLUSTER  and  clusterId  =   $CLUSTERID   Example2:  How  do  I  compare  CPU   usage  across  hosts?   select  dt0(total_cpu_user)  /  getHostFact(numCores,  1)  *  100,   dt0(total_cpu_system)  /  getHostFact(numCores,  1)  *  100,   dt0(total_cpu_nice)  /  getHostFact(numCores,  1)  *  100,   dt0(total_cpu_iowait)  /  getHostFact(numCores,  1)  *  100,   dt0(total_cpu_irq)  /  getHostFact(numCores,  1)  *  100,   dt0(total_cpu_so`_irq)  /  getHostFact(numCores,  1)  *  100   Create  &  Contribute  your  ‘tsqueries’!   hZps://github.com/cloudera/cm_charHng_scrapbook   7 ©2013  Cloudera,  Inc.  All  Rights  Reserved.  
  8. 8. Cloudera  Manager  –  Service  Extensibility   •  Introduced  in  C5   •  SHll  in  Beta!   •  Some  aspects  (espcially  Parcel  mgmt)  available  in  CM4.x   •  Example:  CollaboraHon  with  Syncsort  to  deploy  DMX-­‐h  libraries   •  Single  management  console  for  CDH,  non-­‐CDH  services  and  ISV   applicaHons   •  Similar  look  and  feel  as  exisHng  services   •  Easy  to  write  (Java-­‐free!)   •  Flexible   •  Independent  release  cycle   ©2013Cloudera, Inc. All Rights Reserved.
  9. 9. Analogy  from  OperaHng  Systems  (OS)  world       ISV’s  view  of  OS     Systems  Management   Package   Mgmt   Process/   Resource   Mgmt   Security   Mgmt   Core  OS  kernel   9 ©2013Cloudera,  Inc.  All  Rights  Reserved.   Data   Access   Mgmt  
  10. 10. Bringing  ISV  Apps  to  CDH       ISV’s  view  of  Hadoop     Cloudera  Manager   Parcels   Resource     Mgmt   Security   Mgmt   CDK  API’s   Core  Hadoop/CDH  kernel   10 ©2013Cloudera,  Inc.  All  Rights  Reserved.  
  11. 11. IntegraHng  into  the  Cloudera  Product  Porpolio   Features   Examples   -­‐  Ability  to  easily  package  and  distribute  binaries/ jars  via  “Parcels”   -­‐ InformaHca   -­‐ Syncsort   Resource   Mgmt   -­‐  Ability  to  deploy  applicaHons  as  stand-­‐alone   processes    or  via  YARN*  on  the  Hadoop  grid   -­‐  Resource  isolaHon  of  cluster  resources       -­‐ SAS   -­‐ 0xData   -­‐ Accumulo   Security   Mgmt   Cloudera  Manager   Descrip;on   Package   Mgmt   ISV’s   -­‐  Support  for  Kerberos  Mgmt   -­‐  Role  bases  access  control  for  Tables/Views  in   Hive/Impala  via  Sentry   Data  Access   Mgmt   -­‐  HDFS  and  HBase  API  abstracHon  and   simplificaHon   Systems  Mgmt   Manage   Monitor   Non-­‐CDH  Apps…   Accumulo,   Spark,  Giraph   etc.   -­‐ Deploy  and  upgrade  (rolling)  services  and  pkgs   -­‐ Manage  configuraHons   -­‐ ProacHve  health  checks   -­‐ Track  resource  uHlizaHon     -­‐ Custom  metrics  charts   Diagnose   -­‐ Distributed  log  collecHon  and  searching   -­‐ Tag  and  track  key  events   Integrate   -­‐ Access  operaHonal  tools  via  API   -­‐ Surface  overall  cluster  metrics  to  ISV  dashboard     *  Support  for  YARN  planned  as  part  of  CM5.x  in  FY14   11 ©2013Cloudera,  Inc.  All  Rights  Reserved.  
  12. 12. So..  How  does  it  work?     •  A  JSON  file  that  describes  of  your  service   •  Set  of  control  scripts   •  Packaged  as  a  JAR  file   •  As  promised,  Java-­‐free   ©2013Cloudera, Inc. All Rights Reserved.
  13. 13. Example:  Cloudera  Manager  Extensions  -­‐  Spark       ©2013Cloudera, Inc. All Rights Reserved.
  14. 14.   Cloudera  Manager  Extensions     ©2013Cloudera, Inc. All Rights Reserved.
  15. 15. Cloudera  Manager  Extensions:  Spark   ©2013Cloudera, Inc. All Rights Reserved.
  16. 16. Cloudera  Manager  Extensions:  Spark   ©2013Cloudera, Inc. All Rights Reserved.
  17. 17. Cloudera  Manager  Extensions:  Spark   ©2013Cloudera, Inc. All Rights Reserved.
  18. 18. The  Code       #!/bin/bash   name  :  “spark”,   roles  :  [{            name  :  "master",   CMD=$1   MASTER_PORT=<read  in  from  ./params.proper;es>            startRunner  :  {                    program  :  "scripts/control.sh",                    args  :  [    "start_master",     case  $CMD  in      (start_master)      exec  $SPARK_HOME/scripts/spark-­‐start.sh  master"                                                  "./params.proper;es"]                },          ;;                parameters  :  [{        (*)          echo  "$;mestamp  Don't  understand  [$CMD]"                      name  :  "master_port",                      type  :  "port",          ;;   esac                      default  :  7077                  }],                configWriter  :  {                      generators  :  [{                            filename  :  "params.proper;es"                      }]   }]   ©2013Cloudera, Inc. All Rights Reserved.
  19. 19. Next  Steps   •  DocumentaHon  &  SDK  as  part  of  C5  Beta2   or  later  (definitely  before  GA!)   •  Working  with  select  ISV’s  (SAS,  Syncsort,   0xData  etc.)  as  part  of  Beta  to  further  fine-­‐ tune  this  feature     Develop  &  Contribute  your    Cloudera  Manager  service   extensibility  plug-­‐in’s  !   ©2013Cloudera, Inc. All Rights Reserved.
  20. 20. Horizontal Extension Security ISV’s 0xData SAS Syncsort Informatica Revolution API Ops Apps Capacity Mgr Service Extensibility Vertical Extension Vision  of  CM  Extensibility   SLA Mgr Cost Optimizer CDH CM SNMP API Oracle OEM 20 Nagios Dell Chef/ Puppet ©2012Cloudera,  Inc.  All  Rights  Reserved.   Accumulo Spark Giraph
  21. 21. Q&A   ©2013Cloudera, Inc. All Rights Reserved.
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×