Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Simplified Cluster Operation & Troubleshooting

568 views

Published on

Simplified Cluster Operation & Troubleshooting

Published in: Technology
  • Be the first to comment

Simplified Cluster Operation & Troubleshooting

  1. 1. Simplified Cluster Operation & Troubleshooting Alejandro Fernandez + Jayush Luniya
  2. 2. Speakers Alejandro Fernandez Sr. Software Engineer @ Hortonworks Apache Ambari PMC alejandro@apache.org Jayush Luniya Staff Engineer @ Hortonworks Apache Ambari PMC jluniya@apache.org
  3. 3. What is Apache Ambari? Apache Ambari is the open-source platform to provision, manage and monitor Hadoop clusters
  4. 4. New Enterprise Features Ambari 2.4 • New Services: Log Search, Zeppelin, Hive LLAP • Role Based Access Control • Management Packs • Grafana UI for Ambari Metrics System • New Views: Zeppelin, Storm
  5. 5. Apache Ambari Jiras April 2015 1690 1864 277 379 797 206 488 July - Sept 2015 Dec 2015 – Feb 2016 Today v2.0 v2.1 v2.2 v2.4 1542 and growing
  6. 6. Deploy Secure/L DAP Smart Configs Monitor Upgrade Scale, Extend, Analyze Simply Operations - Lifecycle Ease-of-Use Deploy
  7. 7. Deploy On Premise Ambari UI wizard handles all of these combinations and makes recommendations based on host specs.
  8. 8. Deploy On The Cloud Certified environments Sysprepped VMs Hundreds of similar clusters
  9. 9. Deploy with Blueprints • Systematic way of defining a cluster • Export existing cluster into blueprint /api/v1/clusters/:clusterName?format=blueprint Configs Topology Hosts Cluster
  10. 10. Create a cluster with Blueprints { "configurations" : [ { "hdfs-site" : { "dfs.datanode.data.dir" : "/hadoop/1, /hadoop/2,/hadoop/3" } } ], "host_groups" : [ { "name" : "master-host", "components" : [ { "name" : "NAMENODE” }, { "name" : "RESOURCEMANAGER” }, … ], "cardinality" : "1" }, { "name" : "worker-host", "components" : [ { "name" : "DATANODE" }, { "name" : "NODEMANAGER” }, … ], "cardinality" : "1+" }, ], "Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.5" } } { "blueprint" : "my-blueprint", "host_groups" :[ { "name" : "master-host", "hosts" : [ { "fqdn" : "master001.ambari.apache.org" } ] }, { "name" : "worker-host", "hosts" : [ { "fqdn" : "worker001.ambari.apache.org" }, { "fqdn" : "worker002.ambari.apache.org" }, … { "fqdn" : "worker099.ambari.apache.org" } ] } ] } 1. POST /api/v1/blueprints/my-blueprint 2. POST /api/v1/clusters/my-cluster
  11. 11. Create a cluster with Blueprints { "configurations" : [ { "hdfs-site" : { "dfs.datanode.data.dir" : "/hadoop/1, /hadoop/2,/hadoop/3" } } ], "host_groups" : [ { "name" : "master-host", "components" : [ { "name" : "NAMENODE” }, { "name" : "RESOURCEMANAGER” }, … ], "cardinality" : "1" }, { "name" : "worker-host", "components" : [ { "name" : "DATANODE" }, { "name" : "NODEMANAGER” }, … ], "cardinality" : "1+" }, ], "Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.5" } } { "blueprint" : "my-blueprint", "host_groups" :[ { "name" : "master-host", "hosts" : [ { "fqdn" : "master001.ambari.apache.org" } ] }, { "name" : "worker-host", "hosts" : [ { "fqdn" : "worker001.ambari.apache.org" }, { "fqdn" : "worker002.ambari.apache.org" }, … { "fqdn" : "worker099.ambari.apache.org" } ] } ] } 1. POST /api/v1/blueprints/my-blueprint 2. POST /api/v1/clusters/my-cluster
  12. 12. Create a cluster with Blueprints { "configurations" : [ { "hdfs-site" : { "dfs.datanode.data.dir" : "/hadoop/1, /hadoop/2,/hadoop/3" } } ], "host_groups" : [ { "name" : "master-host", "components" : [ { "name" : "NAMENODE” }, { "name" : "RESOURCEMANAGER” }, … ], "cardinality" : "1" }, { "name" : "worker-host", "components" : [ { "name" : "DATANODE" }, { "name" : "NODEMANAGER” }, … ], "cardinality" : "1+" }, ], "Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.5" } } { "blueprint" : "my-blueprint", "host_groups" :[ { "name" : "master-host", "hosts" : [ { "fqdn" : "master001.ambari.apache.org" } ] }, { "name" : "worker-host", "hosts" : [ { "fqdn" : "worker001.ambari.apache.org" }, { "fqdn" : "worker002.ambari.apache.org" }, … { "fqdn" : "worker099.ambari.apache.org" } ] } ] } 1. POST /api/v1/blueprints/my-blueprint 2. POST /api/v1/clusters/my-cluster
  13. 13. Create a cluster with Blueprints { "configurations" : [ { "hdfs-site" : { "dfs.datanode.data.dir" : "/hadoop/1, /hadoop/2,/hadoop/3" } } ], "host_groups" : [ { "name" : "master-host", "components" : [ { "name" : "NAMENODE” }, { "name" : "RESOURCEMANAGER” }, … ], "cardinality" : "1" }, { "name" : "worker-host", "components" : [ { "name" : "DATANODE" }, { "name" : "NODEMANAGER” }, … ], "cardinality" : "1+" }, ], "Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.5" } } { "blueprint" : "my-blueprint", "host_groups" :[ { "name" : "master-host", "hosts" : [ { "fqdn" : "master001.ambari.apache.org" } ] }, { "name" : "worker-host", "hosts" : [ { "fqdn" : "worker001.ambari.apache.org" }, { "fqdn" : "worker002.ambari.apache.org" }, … { "fqdn" : "worker099.ambari.apache.org" } ] } ] } 1. POST /api/v1/blueprints/my-blueprint 2. POST /api/v1/clusters/my-cluster
  14. 14. Blueprints for Large Scale • Kerberos, secure out-of-the-box • High Availability is setup initially for NameNode, YARN, Hive, Oozie, etc • Host Discovery allows Ambari to automatically install services for a Host when it comes online • Stack Advisor recommendations
  15. 15. POST /api/v1/clusters/MyCluster/hosts [ { "blueprint" : "single-node-hdfs-test2", "host_groups" :[ { "host_group" : "slave", "host_count" : 3, "host_predicate" : "Hosts/cpu_count>1” }, { "host_group" : "super-slave", "host_count" : 5, "host_predicate" : "Hosts/cpu_count>2& Hosts/total_mem>3000000" } ] } ] Blueprint Host Discovery
  16. 16. Kerberos Available since Ambari 2.0 • Ambari manages Kerberos principals and keytabs • Works with existing MIT KDC or Active Directory • Once Kerberized, handles • Adding hosts • Adding components to existing hosts • Adding services • Moving components to different hosts
  17. 17. Management Packs - Motivation • Release Management o Ambari core and stacks released together o Stack changes require Ambari release o Decouple stack and Ambari core releases • Add-on Services o Release vehicle for 3rd party services o Self contained release artifacts
  18. 18. Management Packs – Release Trains
  19. 19. Management Packs • Generalized release artifact for stacks, add-on services, views, etc • Decouples stack releases from Ambari core release • Tarballs with metadata for applicability and content • Stack is an overlay of multiple management packs
  20. 20. Overlay of Management Packs
  21. 21. Management Pack++ Short Term Goals (Ambari 2.4) • Retrofit in Stack Processing Framework • Enable 3rd party to ship add-on services • Command line support Long Term Goals (Future) • Management Pack Framework • Deliver Views • Rest API support
  22. 22. Role Based Access Control (RBAC) As Ambari & organizations grow, so do security needs Ambari integrates with external authentication systems & LDAP
  23. 23. RBAC Terms • Roles have permissions, e.g., add services to cluster • Roles are applied to Resources e.g., Ambari, particular Cluster, particular View • Users belong to groups • A group has a role • Users can also have additional roles
  24. 24. New RBAC Roles allAmbari Admin Cluster Admin except manage permissions Cluster Op except add services, Kerberos, manage Alerts, & upgrades Service Admin except alter cluster topology or install components Service Op except change configs Read-Only only view
  25. 25. Background: Upgrade Terminology Manual Upgrade The user follows instructions to upgrade the stack Incurs downtime
  26. 26. Background: Upgrade Terminology Manual Upgrade The user follows instructions to upgrade the stack Incurs downtime Rolling Upgrade Automated Upgrades one component per host at a time Preserves cluster operation and minimizes service impact
  27. 27. Background: Upgrade Terminology Express Upgrade Automated Runs in parallel across hosts Incurs downtime Manual Upgrade The user follows instructions to upgrade the stack Incurs downtime Rolling Upgrade Automated Upgrades one component per host at a time Preserves cluster operation and minimizes service impact
  28. 28. Automated Upgrade: Rolling or Express Check Prerequisites Review the prereqs to confirm your cluster configs are ready Prepare Take backups of critical cluster metadata Perform Upgrade Perform the HDP upgrade. The steps depend on upgrade method: Rolling or Express Register + Install Register the HDP repository and install the target HDP version on the cluster Finalize Finalize the upgrade, making the target version the current version
  29. 29. Process: Rolling Upgrade ZooKeeper Ranger Core Masters Core Slaves Hive Oozie Falcon Clients Kafka Knox Storm Slider Flume Finalize or Downgrade HDFS, YARN, MR, Tez, HBase, Pig. Hive, etc. HDFS YARN HBas e
  30. 30. Grafana for Ambari Metrics • Grafana as a “Native UI” for Ambari Metrics • Pre-built Dashboards Host-level, Service-level • Supports HTTPS • System Home, Servers • HDFS Home, NameNodes, DataNodes • YARN Home, Applications, Job History Server • HBase Home, Performance, Misc FEATURES DASHBOARDS
  31. 31. Grafana includes pre-built dashboards for visualizing the most important cluster metrics.
  32. 32. The HDFS NameNode dashboard highlights file system activity.
  33. 33. Future of Ambari • Cloud features • Multiple instances of same service at different versions, e.g., Spark 1.6 and Spark 2.0 • YARN assemblies • Component & Patch Upgrades: upgrade individual components in the same stack version, e.g., just DN and RM in HDP 2.4.*.* with zero downtime

×