Ambari talk at Hadoop Summit, Tokyo 2016 Abstract Apache Ambari has become an indispensable tool for operating Hadoop clusters from as small as 10s of nodes to 1000s of nodes. Ambari’s deep knowledge of the Hadoop stack allows it to deploy a cluster within minutes and manage the entire lifecycle: scaling, security, upgrades, and more. This talk will cover the central features important to cluster operators and the latest innovations from the community. We will discuss automatically deploying clusters with Blueprints, adding custom services, scaling the number of hosts as the data needs grow, adding High Availability for critical services, securing with MIT kerberos, and upgrading the Hadoop stack with features like Rolling & Express Upgrade. More advanced users will also be interested in using Ambari’s powerful REST API to automate workflows. For users and data scientists, Ambari provides LDAP sync, Role-Based Access Control to handle user permissions, and a framework to host Ambari Views such as as the newly added Views for Hive, Oozie, Capacity Scheduler, Tez, Storm, and Zeppelin. Lastly, we will cover how to monitor the health of the cluster via Alerts and troubleshoot problems by using new features like LogSearch and Ambari Metrics Systems integrated with Grafana UI.

1. Streamline
Hadoop DevOps
with Apache
Ambari
Jayush Luniya
Hadoop Summit, Tokyo

2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Speaker
Jayush Luniya
Staff Software Engineer @ Hortonworks
Apache Ambari PMC
jluniya@apache.org

3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Open-source platform to provision, manage and monitor
Hadoop clusters

4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Why Ambari?

5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Why Ambari?

6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
1690
1864
797
2251
277
206
34379
488
0
500
1000
1500
2000
2500
3000
Ambari 2.0 Ambari 2.1 Ambari 2.2 Ambari 2.4
April '15 Jul - Sept'15 Dec'15 - Feb'16 Aug'16 - Sept '16
#.#.2
#.#.1
GA
No.ofJIRAs
Ambari Releases

8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari Architecture

9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Exciting Enterprise Features in
Ambari 2.4
 New Services: Log Search, Zeppelin, Hive LLAP
 Role Based Access Control
 Management Packs
 Grafana UI for Ambari Metrics System
 New Views: Zeppelin, Storm

10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Deploy
Secure
Config
Upgrade
Monitor
Extend
Operations - Lifecycle
Ease-of-Use

11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Deploy

12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Deploy On Premise
Ambari handles all of these combinations and
makes recommendations based on host specs.

13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Deploy In The Cloud
Certified environments
Sysprepped VMs
Hundreds of similar clusters

14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Deploy with Blueprints
 Systematic way of defining a cluster
 Export existing cluster into blueprint
/api/v1/clusters/:clusterName?format=blueprint
Configs Topology Hosts Cluster

15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Create Cluster with Blueprints
{
"configurations" : [
{
"hdfs-site" : {
"dfs.datanode.data.dir" : "/hadoop/1,
/hadoop/2,/hadoop/3"
}
}
],
"host_groups" : [
{
"name" : "master-host",
"components" : [
{ "name" : "NAMENODE” },
{ "name" : "RESOURCEMANAGER” },
…
],
"cardinality" : "1"
},
{
"name" : "worker-host",
"components" : [
{ "name" : "DATANODE" },
{ "name" : "NODEMANAGER” },
…
],
"cardinality" : "1+"
},
],
"Blueprints" : {
"stack_name" : "HDP",
"stack_version" : "2.5"
}
}
{
"blueprint" : "my-blueprint",
"host_groups" :[
{
"name" : "master-host",
"hosts" : [
{
"fqdn" : "master001.ambari.apache.org"
}
]
},
{
"name" : "worker-host",
"hosts" : [
{
"fqdn" : "worker001.ambari.apache.org"
},
{
"fqdn" : "worker002.ambari.apache.org"
},
…
{
"fqdn" : "worker099.ambari.apache.org"
}
]
}
]
}
1. POST /api/v1/blueprints/my-blueprint 2. POST /api/v1/clusters/my-cluster

16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Create Cluster with Blueprints
{
"configurations" : [
{
"hdfs-site" : {
"dfs.datanode.data.dir" : "/hadoop/1,
/hadoop/2,/hadoop/3"
}
}
],
"host_groups" : [
{
"name" : "master-host",
"components" : [
{ "name" : "NAMENODE” },
{ "name" : "RESOURCEMANAGER” },
…
],
"cardinality" : "1"
},
{
"name" : "worker-host",
"components" : [
{ "name" : "DATANODE" },
{ "name" : "NODEMANAGER” },
…
],
"cardinality" : "1+"
},
],
"Blueprints" : {
"stack_name" : "HDP",
"stack_version" : "2.5"
}
}
{
"blueprint" : "my-blueprint",
"host_groups" :[
{
"name" : "master-host",
"hosts" : [
{
"fqdn" : "master001.ambari.apache.org"
}
]
},
{
"name" : "worker-host",
"hosts" : [
{
"fqdn" : "worker001.ambari.apache.org"
},
{
"fqdn" : "worker002.ambari.apache.org"
},
…
{
"fqdn" : "worker099.ambari.apache.org"
}
]
}
]
}
1. POST /api/v1/blueprints/my-blueprint 2. POST /api/v1/clusters/my-cluster

17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Create Cluster with Blueprints
{
"configurations" : [
{
"hdfs-site" : {
"dfs.datanode.data.dir" : "/hadoop/1,
/hadoop/2,/hadoop/3"
}
}
],
"host_groups" : [
{
"name" : "master-host",
"components" : [
{ "name" : "NAMENODE” },
{ "name" : "RESOURCEMANAGER” },
…
],
"cardinality" : "1"
},
{
"name" : "worker-host",
"components" : [
{ "name" : "DATANODE" },
{ "name" : "NODEMANAGER” },
…
],
"cardinality" : "1+"
},
],
"Blueprints" : {
"stack_name" : "HDP",
"stack_version" : "2.5"
}
}
{
"blueprint" : "my-blueprint",
"host_groups" :[
{
"name" : "master-host",
"hosts" : [
{
"fqdn" : "master001.ambari.apache.org"
}
]
},
{
"name" : "worker-host",
"hosts" : [
{
"fqdn" : "worker001.ambari.apache.org"
},
{
"fqdn" : "worker002.ambari.apache.org"
},
…
{
"fqdn" : "worker099.ambari.apache.org"
}
]
}
]
}
1. POST /api/v1/blueprints/my-blueprint 2. POST /api/v1/clusters/my-cluster

18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Create Cluster with Blueprints
{
"configurations" : [
{
"hdfs-site" : {
"dfs.datanode.data.dir" : "/hadoop/1,
/hadoop/2,/hadoop/3"
}
}
],
"host_groups" : [
{
"name" : "master-host",
"components" : [
{ "name" : "NAMENODE” },
{ "name" : "RESOURCEMANAGER” },
…
],
"cardinality" : "1"
},
{
"name" : "worker-host",
"components" : [
{ "name" : "DATANODE" },
{ "name" : "NODEMANAGER” },
…
],
"cardinality" : "1+"
},
],
"Blueprints" : {
"stack_name" : "HDP",
"stack_version" : "2.5"
}
}
{
"blueprint" : "my-blueprint",
"host_groups" :[
{
"name" : "master-host",
"hosts" : [
{
"fqdn" : "master001.ambari.apache.org"
}
]
},
{
"name" : "worker-host",
"hosts" : [
{
"fqdn" : "worker001.ambari.apache.org"
},
{
"fqdn" : "worker002.ambari.apache.org"
},
…
{
"fqdn" : "worker099.ambari.apache.org"
}
]
}
]
}
1. POST /api/v1/blueprints/my-blueprint 2. POST /api/v1/clusters/my-cluster

19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Blueprints for Large Scale
 Kerberos, secure out-of-the-box
 High Availability is setup initially for
NameNode, YARN, Hive, Oozie, etc
 Host Discovery allows Ambari to
automatically install services for a Host when
it comes online
 Stack Advisor recommendations

20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
POST /api/v1/clusters/MyCluster/hosts
[
{
"blueprint" : "single-node-hdfs-test2",
"host_groups" :[
{
"host_group" : "slave",
"host_count" : 3,
"host_predicate" : "Hosts/cpu_count>1”
}, {
"host_group" : "super-slave",
"host_count" : 5,
"host_predicate" : "Hosts/cpu_count>2&
Hosts/total_mem>3000000"
}
]
}
]
Blueprint Host Discovery

21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Secure

22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Comprehensive Security
LDAP/AD
• User Auth.
• Sync
Kerberos
• MIT KDC
• Keytab
Management
Atlas
• Governance
• Compliance
• Data Classify
• Lineage & History
Ranger
• Security policies
• Audit
• Authorization
Knox
• Perimeter Sec.
• LDAP/AD
• Sec. REST/HTTP
• SSL

23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Kerberos
Ambari manages Kerberos principals and keytabs
Works with existing MIT KDC or Active Directory
Once Kerberized, handles
 Adding Services
 Adding Hosts
 Adding Host Components
 Moving Host Components

24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Role Based Access Control (RBAC)
As Ambari & organizations grow,
so do security needs
Ambari integrates with external
authentication systems & LDAP

25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
RBAC Terms
Users belong to groups
A group has a role
Users can also have additional roles
Roles are applied to Resources. E.g.,
Ambari, particular Cluster, particular View
Roles have permissions
e.g., add services to cluster

26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
New RBAC Roles
only view
↑, except change configs
↑, except alter cluster topology
or install components
Ambari Admin
Cluster Admin
Cluster Op
Service Admin
Service Op
Read-Only
↑, except add services, Kerberos,
manage alerts & upgrades
↑, except manage permissions
all

27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Config

28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Config Management
 Config Groups
 Different config settings for individual host components
 Config Versioning
 Revert back to old configs
 Smart Configs
 Highlight most important configs
 Stack Advisor
 Recommend configurations

29. 29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Smart Configs
Widgets
- Sliders
- Combos
- Toggles
- Spinners
- Lists

30. 30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Stack Advisor
Kerberos
HTTPS
Zookeeper Servers
Memory Settings
…
High Availability
atlas.rest.address =
http(s)://host:port
# Atlas Servers
atlas.enabletTLS = true|false
atlas.server.http.port = 21000
atlas.server.https.port = 21443
Example
Configurations

31. 31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Upgrade

32. 32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Background: Upgrade Terminology
Manual
Upgrade
The user follows instructions to upgrade
the stack
Incurs downtime

33. 33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Background: Upgrade Terminology
Manual
Upgrade
The user follows instructions to upgrade
the stack
Incurs downtime
Rolling
Upgrade
Automated
Upgrades one component
per host at a time
Preserves cluster operation
and minimizes service impact

34. 34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Background: Upgrade Terminology
Express
Upgrade
Automated
Runs in parallel across hosts
Incurs downtime
Manual
Upgrade
The user follows instructions to upgrade
the stack
Incurs downtime
Rolling
Upgrade
Automated
Upgrades one component
per host at a time
Preserves cluster operation
and minimizes service impact

35. 35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Check
Prerequisites
Review the
prereqs to
confirm
your cluster
configs are
ready
Prepare
Take
backups of
critical
cluster
metadata
Register +
Install
Register the
HDP
repository
and install
the target
HDP version
on the
cluster
Automated Upgrade: Rolling/Express
Perform
Upgrade
Perform the
HDP
upgrade.
The steps
depend on
upgrade
method:
Rolling or
Express
Finalize
Finalize the
upgrade,
making the
target
version the
current
version

36. 36 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Process: Rolling Upgrade
ZooKeeper
Ranger
Hive
Oozie
Falcon
Kafka
Knox
Storm
Slider
Flume
Finalize or
Downgrade
Clients HDFS, YARN, MR, Tez,
HBase, Pig. Hive, etc.
Core
Masters
Core Slaves
HDFS
YARN
HBase

37. 37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Monitor

38. 38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Alerting Framework
Alert Type Description Thresholds (units)
WEB Connects to a Web URL. Alert status is based
on the HTTP response code
Response Code (n/a)
Connection Timeout (seconds)
PORT Connects to a port. Alert status is based on
response time
Response (seconds)
METRIC Checks the value of a service metric. Units
vary, based on the metric being checked
Metric Value (units vary)
Connection Timeout (seconds)
AGGREGATE Aggregates the status for another alert % Affected (percentage)
SCRIPT Executes a script to handle the alert check Varies
SERVER Executes a server-side runnable class to
handle the alert check
Varies

39. 39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Alert UI

40. 40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Grafana for Ambari Metrics
 Grafana as a “Native UI” for
Ambari Metrics
 Pre-built Dashboards
Host-level, Service-level
 Supports HTTPS
 System Home, Servers
 HDFS Home, NameNodes,
DataNodes
 YARN Home, Applications,
Job History Server
 HBase Home, Performance
FEATURES DASHBOARDS

41. 41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Grafana includes pre-built
dashboards for visualizing the most
important cluster metrics.

42. 42 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
The HDFS NameNode
dashboard highlights
file system activity.

43. 43 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Storm Monitoring View

44. 44 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Log Search
Search and index Hadoop logs!
Capabilities
• Rapid Search of all Hadoop component logs
• Search across time ranges, log levels, and for
keywords
Solr
Logsearch
Ambari

45. 45 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Log Search
WO R K E R
N O D E
L O G
F E E D E R
Solr
LO G
S EA RC H
U I
Solr
Solr
A M BA R I
Java Process
Multi-output Support
Grok filters
Solr Cloud
Local Disk Storage

46. 46 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Extend

47. 47 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Management Packs
 Improved Release Management:
Decouple Ambari core from stacks releases
 Support Add-ons:
–Release vehicle for 3rd party services, views
–Self-contained release artifacts
–Stack is an overlay of multiple management
packs

48. 48 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Overlay of Management Packs
inherits from 2.3
inherits from 2.4
inherits from 2.5

49. 49 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Management Pack++
Short Term Goals (Ambari 2.4)
 Retrofit in Stack Processing Framework
 Enable 3rd party to ship add-on services
Future Goals
 Management Pack Framework
 Include Views

50. 50 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Service Level Extensions
 Service Role Command Order
 Service Advisor
 Service Repos
 Service Upgrade Packs

51. 51 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Future

52. 52 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Future of Ambari
 Cloud Focus
 Multiple Service Instance (Two ZK quorums)
 Multiple Service Versions (Spark 1.6 & Spark 2.0)
 YARN Assemblies
 Granular Upgrades: Patch, Component, Service
 Ambari High Availability

53. 53 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Thank You