Your SlideShare is downloading. ×
Extending and Automating Cloudera Manager via API
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Extending and Automating Cloudera Manager via API

663
views

Published on

As delivered by Patrick Angeles, Director of Field Technical Services

As delivered by Patrick Angeles, Director of Field Technical Services

Published in: Technology, Education

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
663
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
13
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Cloudera  Manager  –  API’s  &   Extensibility     Patrick  Angeles,  Director  Field  Technical  Services   December  2013     1 CONFIDENTIAL  -­‐  RESTRICTED  
  • 2. Cloudera  Manager   End-­‐to-­‐End  AdministraHon  for  CDH   1 Monitor   2 Diagnose   3 Integrate   4 Manage   Easily  deploy,  configure  &  opHmize  clusters   Maintain  a  central  view  of  all  acHvity   Easily  idenHfy  and  resolve  issues   Use  Cloudera  Manager  with  exisHng  tools   2 ©2013  Cloudera,  Inc.  All  Rights  Reserved.  
  • 3. IntegraHng  with  your  IT  Mgmt  tools   Datacenter  Opera*ons   Various  op*ons  of  integra*ng  Cloudera  Manager  into  your  exis*ng   Installa;on,   Datacenter  Opera*ons/Tools  Monitoring   Deployment   Aler;ng   Tools   tools   Tools   e.g.  Orion,     •  Cloudera  Manager  API   e.g.  Chef,   e.g  Nagios,   Tivoli,  BMC   Puppet  etc.   SNMP  etc.   •  Introduced  in  CM4  (June  2etc.   012)     •  Installa*on  &  deployment   •  Monitoring   •  SNMP  Alerts   •  Introduced  in  CM4.5  (Feb  2013)   •  Hadoop  Opera*ons   And  more…   Cloudera   •  Monitoring  ‘tsquery’  (Feb  2013)   Manager   •  User-­‐defined  triggers/alarms  (new  for  C5!)   •  Service  extensibility  (new  for  C5!)   3 ©2013  Cloudera,  Inc.  All  Rights  Reserved.  
  • 4. Cloudera  Manager  (CM)  API     •  •      4 API  access  was  a  new  feature  introduced  in  Cloudera  Manager  4.0,  providing  programmaHc  access  to   cluster  operaHons  (such  as  configuraHon  and  restart)  and  monitoring  informaHon  (such  as  health  and   metrics).     The  CM  API  is  an  HTTP  REST  API,  using  JSON  serializaHon.  The  API  is  served  on  the  same  host  and  port  as   the  CM  web  UI,  and  does  not  require  an  extra  process  or  extra  configuraHon.  API  users  have  the  same   privileges  as  they  do  in  the  web  UI  world.   •  Docs  &  Examples   hZp://cloudera.github.io/cm_api/   hZps://github.com/cloudera/cm_api   •  Java/Python  clients   hZp://blog.cloudera.com/blog/2013/05/how-­‐to-­‐ automate-­‐your-­‐hadoop-­‐cluster-­‐from-­‐java/       ©2013Cloudera,  Inc.  All  Rights  Reserved.  
  • 5. Examples  of  integraHon  with  CM  API   •  Installa;on  &  Deployment   •  •  •  Chef   Puppet   Dell  Crowbar   •  •  hZp://blog.cloudera.com/blog/2013/08/how-­‐to-­‐deploy-­‐hadoop-­‐clusters-­‐automaHcally-­‐with-­‐dell-­‐crowbar-­‐and-­‐cloudera-­‐manager/ StackIQ   •    hZp://web.stackiq.com/blog/bid/312064/StackIQ-­‐Cluster-­‐Manager-­‐now-­‐integrated-­‐with-­‐Cloudera   •  •  •  WANdisco  –  non-­‐stop  NN  setup   Several  other  customers/partners  leveraging  the  API’s  as  part  of    their  install  &  deployment     process   Monitoring  &  Aler;ng   •  •  Oracle  Enterprise  Manager  (via  Big  Data  Appliance)   Nagios   •  •  •  hZps://github.com/cloudera/cm_api/tree/master/nagios   hZps://github.com/harisekhon/nagios-­‐plugins/blob/master/ check_hadoop_cloudera_manager_metrics.pl   SNMP  alerts  integraHon  with  IBM  Netcool   Develop  &  Contribute  your  plug-­‐in’s  using  Cloudera   Manager  API     5 ©2013  Cloudera,  Inc.  All  Rights  Reserved.  
  • 6. Cloudera  Manager  –  Monitoring  via  ‘tsquery’   •  Introduced  as  part  of  CM4.5    release  (Feb  2013)   •  Great  way  to  add  interesHng    charts  (above  &  beyond  what  is  provided  by  default)    and  monitor   metrics  that  are  relevant  to  your  clusters   •  •  The  tsquery  language   s  used  to    Manager  Hme-­‐series  diata  store   specify  statements  for  retrieving  Hme-­‐series  data  from  the  Cloudera     Example:  How  do  I  compare  all  disk  IO  for  all  the  DataNodes  that  belong  to  a  specific  HDFS  service?   select  bytes_read,  bytes_wriZen  where  roleType=DATANODE  and  serviceName=hdfs1   •  Retrieved  Hme-­‐series  data  can  be  ploZed  via  various  opHons  –  line,  bar,  scaZer,    heat  maps,  table  list   etc.   •  Extending  this  concept  to  create  user-­‐defined  triggers/alarms  (new  for  C5!).     More  details   •  hZp://www.cloudera.com/content/cloudera-­‐content/cloudera-­‐docs/CM5/latest/Cloudera-­‐ Manager-­‐DiagnosHcs-­‐Guide/cm5dg_chart_Hme_series_data.html   •  6 ©2013  Cloudera,  Inc.  All  Rights  Reserved.  
  • 7. Examples  of  Cloudera  Manager  ‘tsquery’   Example1:  How  do  I  track  the     aggregate  Cluster  Disk  IO?     select  dt0(read_bytes_disk_sum),   dt0(write_bytes_disk_sum)  where   category  =  CLUSTER  and  clusterId  =   $CLUSTERID   Example2:  How  do  I  compare  CPU   usage  across  hosts?   select  dt0(total_cpu_user)  /  getHostFact(numCores,  1)  *  100,   dt0(total_cpu_system)  /  getHostFact(numCores,  1)  *  100,   dt0(total_cpu_nice)  /  getHostFact(numCores,  1)  *  100,   dt0(total_cpu_iowait)  /  getHostFact(numCores,  1)  *  100,   dt0(total_cpu_irq)  /  getHostFact(numCores,  1)  *  100,   dt0(total_cpu_so`_irq)  /  getHostFact(numCores,  1)  *  100   Create  &  Contribute  your  ‘tsqueries’!   hZps://github.com/cloudera/cm_charHng_scrapbook   7 ©2013  Cloudera,  Inc.  All  Rights  Reserved.  
  • 8. Cloudera  Manager  –  Service  Extensibility   •  Introduced  in  C5   •  SHll  in  Beta!   •  Some  aspects  (espcially  Parcel  mgmt)  available  in  CM4.x   •  Example:  CollaboraHon  with  Syncsort  to  deploy  DMX-­‐h  libraries   •  Single  management  console  for  CDH,  non-­‐CDH  services  and  ISV   applicaHons   •  Similar  look  and  feel  as  exisHng  services   •  Easy  to  write  (Java-­‐free!)   •  Flexible   •  Independent  release  cycle   ©2013Cloudera, Inc. All Rights Reserved.
  • 9. Analogy  from  OperaHng  Systems  (OS)  world       ISV’s  view  of  OS     Systems  Management   Package   Mgmt   Process/   Resource   Mgmt   Security   Mgmt   Core  OS  kernel   9 ©2013Cloudera,  Inc.  All  Rights  Reserved.   Data   Access   Mgmt  
  • 10. Bringing  ISV  Apps  to  CDH       ISV’s  view  of  Hadoop     Cloudera  Manager   Parcels   Resource     Mgmt   Security   Mgmt   CDK  API’s   Core  Hadoop/CDH  kernel   10 ©2013Cloudera,  Inc.  All  Rights  Reserved.  
  • 11. IntegraHng  into  the  Cloudera  Product  Porpolio   Features   Examples   -­‐  Ability  to  easily  package  and  distribute  binaries/ jars  via  “Parcels”   -­‐ InformaHca   -­‐ Syncsort   Resource   Mgmt   -­‐  Ability  to  deploy  applicaHons  as  stand-­‐alone   processes    or  via  YARN*  on  the  Hadoop  grid   -­‐  Resource  isolaHon  of  cluster  resources       -­‐ SAS   -­‐ 0xData   -­‐ Accumulo   Security   Mgmt   Cloudera  Manager   Descrip;on   Package   Mgmt   ISV’s   -­‐  Support  for  Kerberos  Mgmt   -­‐  Role  bases  access  control  for  Tables/Views  in   Hive/Impala  via  Sentry   Data  Access   Mgmt   -­‐  HDFS  and  HBase  API  abstracHon  and   simplificaHon   Systems  Mgmt   Manage   Monitor   Non-­‐CDH  Apps…   Accumulo,   Spark,  Giraph   etc.   -­‐ Deploy  and  upgrade  (rolling)  services  and  pkgs   -­‐ Manage  configuraHons   -­‐ ProacHve  health  checks   -­‐ Track  resource  uHlizaHon     -­‐ Custom  metrics  charts   Diagnose   -­‐ Distributed  log  collecHon  and  searching   -­‐ Tag  and  track  key  events   Integrate   -­‐ Access  operaHonal  tools  via  API   -­‐ Surface  overall  cluster  metrics  to  ISV  dashboard     *  Support  for  YARN  planned  as  part  of  CM5.x  in  FY14   11 ©2013Cloudera,  Inc.  All  Rights  Reserved.  
  • 12. So..  How  does  it  work?     •  A  JSON  file  that  describes  of  your  service   •  Set  of  control  scripts   •  Packaged  as  a  JAR  file   •  As  promised,  Java-­‐free   ©2013Cloudera, Inc. All Rights Reserved.
  • 13. Example:  Cloudera  Manager  Extensions  -­‐  Spark       ©2013Cloudera, Inc. All Rights Reserved.
  • 14.   Cloudera  Manager  Extensions     ©2013Cloudera, Inc. All Rights Reserved.
  • 15. Cloudera  Manager  Extensions:  Spark   ©2013Cloudera, Inc. All Rights Reserved.
  • 16. Cloudera  Manager  Extensions:  Spark   ©2013Cloudera, Inc. All Rights Reserved.
  • 17. Cloudera  Manager  Extensions:  Spark   ©2013Cloudera, Inc. All Rights Reserved.
  • 18. The  Code       #!/bin/bash   name  :  “spark”,   roles  :  [{            name  :  "master",   CMD=$1   MASTER_PORT=<read  in  from  ./params.proper;es>            startRunner  :  {                    program  :  "scripts/control.sh",                    args  :  [    "start_master",     case  $CMD  in      (start_master)      exec  $SPARK_HOME/scripts/spark-­‐start.sh  master"                                                  "./params.proper;es"]                },          ;;                parameters  :  [{        (*)          echo  "$;mestamp  Don't  understand  [$CMD]"                      name  :  "master_port",                      type  :  "port",          ;;   esac                      default  :  7077                  }],                configWriter  :  {                      generators  :  [{                            filename  :  "params.proper;es"                      }]   }]   ©2013Cloudera, Inc. All Rights Reserved.
  • 19. Next  Steps   •  DocumentaHon  &  SDK  as  part  of  C5  Beta2   or  later  (definitely  before  GA!)   •  Working  with  select  ISV’s  (SAS,  Syncsort,   0xData  etc.)  as  part  of  Beta  to  further  fine-­‐ tune  this  feature     Develop  &  Contribute  your    Cloudera  Manager  service   extensibility  plug-­‐in’s  !   ©2013Cloudera, Inc. All Rights Reserved.
  • 20. Horizontal Extension Security ISV’s 0xData SAS Syncsort Informatica Revolution API Ops Apps Capacity Mgr Service Extensibility Vertical Extension Vision  of  CM  Extensibility   SLA Mgr Cost Optimizer CDH CM SNMP API Oracle OEM 20 Nagios Dell Chef/ Puppet ©2012Cloudera,  Inc.  All  Rights  Reserved.   Accumulo Spark Giraph
  • 21. Q&A   ©2013Cloudera, Inc. All Rights Reserved.