Apache Ambari BOF Meet Up @ Hadoop Summit 2013 Overview Slides http://www.meetup.com/Apache-Ambari-User-Group/events/119184782/

1. © Hortonworks Inc. 2013
Welcome to Ambari BoF Meetup!
Hadoop Summit
June 2013

2. © Hortonworks Inc. 2013
Agenda: June 25, 2013
1. Ambari Overview, Current State, Recent
Developments
Yusaku Sako @ Hortonworks
Siddharth Wagle @ Hortonworks
2. APIs and SPIs – How to Integrate with Ambari
Tom Beerbower @ Hortonworks
Nate Cole @ Hortonworks
3. Ambari Blueprint
Brian Swan @ Microsoft
Sumit Mohanty @ Hortonworks
4. Ambari and OpenStack
Himanshu Bari @ Hortonworks
Page 2

3. © Hortonworks Inc. 2013
“Make Hadoop Management Easy for Everyone”
Page 3
Provision, Manage,
Monitor
Hadoop Clusters
Hadoop Operators
Analyze Jobs
(MR, Hive, Pig, etc)
Hadoop Users
Integrate Hadoop
management/monitoring
capabilities to software
Software Vendors
Ambari’s Mission

4. © Hortonworks Inc. 2013
Apache Ambari – 100% Open Source!
• Active and growing community
• 50+ Contributors / 20+ Committers
• 140 Ambari User Group Members
• Steady progress / short release cycle
Page 4
Release
Version
Release
Date
JIRAs
Resolved
0.9.0 Sep 2012 402
1.2.0 Feb 2013 441
1.2.1 Mar 2013 134
1.2.2 Apr 2013 106
1.2.3 Jun 2013 515
1.2.4 Jul 2013 109+
1.2.5 Jul 2013 131+
 Current Release
 Today’s Demo

5. © Hortonworks Inc. 2013
Ambari System Architecture
5
Ambari Server
Host
Agent
gmond
Host
Agent
gmond
Ganglia
Server
Agent
Host
Agent
gmondgmetad
gmond
Ambari
Web
DB
REST
/clusters
Nagios
Server
Agent

6. © Hortonworks Inc. 2013
Ambari Today
• Live Demo of a cluster installed with Ambari 1.2.5
Page 6

7. © Hortonworks Inc. 2013
Ambari Architecture
Page 7
DB
Orchestrator SPI
REST API
Request Dispatcher
Ambari
Web
Ambari
Server
Metrics
AuthProvider
/clusters
/services
/hosts
/workflows/jobs
/users, …
User
Store
java
RDBMS
javascript
RDBM
S
AD/
LDAP
REST API for
integration
Auth
Provider
Cluster
Configurations
Web Client
100% REST
Ambari
Agents ganglia nagios
Alerts
Pluggable
Service
Providersfalcon
Data Mgmt
jmx
python
puppet

8. © Hortonworks Inc. 2013
Hadoop Security - Background
Page 8
• Non secure mode = No auth
• Kerberos tickets allow NN/JT to mutually
authenticate (principals/keytabs)
• Guidelines for Kerberos based security
– Daemon principals and keytabs
– Mapping of secure user to FS user
– Re-configuring individual daemons
– Re-starting services and testing for secure setup
KDC
HM
JT
NN
Kerberosclient
Client
TT
DN
Kerberosclient

9. © Hortonworks Inc. 2013
Hadoop Security with Ambari
Page 9
• Ambari 1.2.5 introduces Security Wizard
– Re-configuring services with security enabled
– Mapping of secure users to FS user
– Re-starting services in secure mode and run
smoke tests
– Note: User needs to generate principals and
keytabs
• Future:
– Automatically create principals and keytabs
– Secure distributions of keytabs

10. © Hortonworks Inc. 2013
Password Encryption and Web Access
Page 10
• Allow trusted signed certs to be imported
into Ambari
– Keystore: Open SSL with PKCS 12 keystore
– User provided cert is used for server-agent
communication
• Database and LDAP passwords no longer in
clear-text
– JCEKS keystore to store credentials, locked with
a user-provided master key

11. © Hortonworks Inc. 2013
REST API – Centralized & Consistent
Page 11
Ambari REST API
Alerts
Job
HistoryMetricsConfigurations
Config
DB
Nagios
Server
Ganglia
Server
…
HTTP GET, POST, PUT, DELETE
:8080
HTTP Status Code / JSON
core-
site.xml
core-
site.xml
Config
files
Config
files
Config
files
JMX
Realtime Historical*-site.xml…
Job History
DB
Hosts / ServicesCluster

12. © Hortonworks Inc. 2013
Ambari + Teradata Viewpoint Integration
Page 12
• Ambari = Key enabler for
integrating Hadoop monitoring
capabilities to Teradata
Viewpoint
• Get Hadoop metrics from a non-
Ambari deployed cluster
– Ambari REST API + Custom Service
Providers

13. © Hortonworks Inc. 2013
Stack Definitions
• Design Goals
–Ambari should be able to support choice of Hadoop stacks
–Ambari should enable adding new components to an existing
stack
• Define which Services are available (services)
• Define where to get the packages (repos)
13
S S S SStack B
repos
services
S S S SStack A
repos
services
S S S S
Stack C
extends
Stack B
repos
services
S
S+

14. © Hortonworks Inc. 2013
Ambari + Redhat GlusterFS Integration
• Using Ambari to deploy / manage cluster with
distributed file system other than HDFS
–HCFS: GlusterFS as first implementation
–Pluggability with other HCFS’s
–See AMBARI-1817
Page 14
MapReduce
Hive
Distributed File System
HDFS
GlusterFS
HBasePig
Other HCFS …

15. © Hortonworks Inc. 2013
Ambari + Accumulo Integration
• Using Ambari to deploy / manage cluster with
Accumulo
–Google Summer of Code project
–See AMBARI-1930
MapReduce
Hive
Distributed File System
HBasePig

16. © Hortonworks Inc. 2013
Splunk + Ambari Integration
• Head over to Splunk’s booth tomorrow to see Ambari
integrated into Splunk’s Management UI for Hadoop
monitoring and management
Page 16
+

17. © Hortonworks Inc. 2013
Thanks!
• Next Up: APIs and SPIs – How to Integrate with
Ambari
Page 17