Page1 © Hortonworks Inc. 2014
Discover HDP 2.1
Using Apache Ambari to Manage Hadoop Clusters
Hortonworks. We do Hadoop.
Page2 © Hortonworks Inc. 2014
Speakers
Justin Sears
Hortonworks Product Marketing Manager
Jeff Sposetti
Hortonworks Senior Director of Product Management and
Committer for Apache Ambari
Mahadev Konar
Hortonworks Co-Founder, Committer and PMC Member for
Apache Hadoop, Apache Ambari & Apache ZooKeeper
Page3 © Hortonworks Inc. 2014
Agenda
• Overview of Apache Ambari
• New Ambari Features
• Demo
• Q & A
We’ll move quickly:
• Attendee phone lines are muted
• Text any questions to Mahadev Konar using Webex chat
• Questions will be answered at the end of the presentation
• Unanswered questions and answers in upcoming FAQ/blog post
Page4 © Hortonworks Inc. 2014
OPERATIONS TOOLS
Provision,
Manage &
Monitor
DEV & DATA TOOLS
Build & Test
A Modern Data Architecture
APPLICATIONSDATASYSTEM
REPOSITORIES
RDBMS EDW MPP
Business
Analytics
Custom Applications
Packaged
Applications
Governance
&Integration
ENTERPRISE HADOOP
Security
Operations
Data Access
Data Management
SOURCES
OLTP, ERP,
CRM Systems
Documents,
Emails
Web Logs,
Click Streams
Social Networks Machine
Generated
Sensor
Data
Geolocation Data
Page5 © Hortonworks Inc. 2014
HDP 2.1: Enterprise Hadoop
HDP 2.1
Hortonworks Data Platform
Provision,
Manage &
Monitor
Ambari
Zookeeper
Scheduling
Oozie
Data Workflow,
Lifecycle &
Governance
Falcon
Sqoop
Flume
NFS
WebHDFS
YARN : Data Operating System
DATA MANAGEMENT
DATA ACCESS
GOVERNANCE &
INTEGRATION
OPERATIONS
Script
Pig
Search
Solr
SQL
Hive/Tez,
HCatalog
NoSQL
HBase
Accumulo
Stream
Storm
Others
In-Memory
Analytics,
ISV engines
1 ° ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° ° °
°
°
N
HDFS
(Hadoop Distributed File System)
Batch
Map
Reduce
SECURITY
Authentication
Authorization
Accounting
Data Protection
Storage: HDFS
Resources: YARN
Access: Hive, …
Pipeline: Falcon
Cluster: Knox
Page6 © Hortonworks Inc. 2014
HDP 2.1: Enterprise Hadoop
HDP 2.1
Hortonworks Data Platform
Scheduling
Oozie
Data Workflow,
Lifecycle &
Governance
Falcon
Sqoop
Flume
NFS
WebHDFS
YARN : Data Operating System
DATA MANAGEMENT
DATA ACCESS
GOVERNANCE &
INTEGRATION
Script
Pig
Search
Solr
SQL
Hive/Tez,
HCatalog
NoSQL
HBase
Accumulo
Stream
Storm
Others
In-Memory
Analytics,
ISV engines
1 ° ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° ° °
°
°
N
HDFS
(Hadoop Distributed File System)
Batch
Map
Reduce
SECURITY
Authentication
Authorization
Accounting
Data Protection
Storage: HDFS
Resources: YARN
Access: Hive, …
Pipeline: Falcon
Cluster: Knox
Provision,
Manage &
Monitor
Ambari
Zookeeper
OPERATIONS
Page7 © Hortonworks Inc. 2014
Agenda
Ambari
Overview
New
Ambari
Features
Demo Q & A
Page8 © Hortonworks Inc. 2014
Driving Themes for Apache Ambari
Operate Hadoop
at Scale
Integrate with
the Enterprise
Extend for the
Ecosystem
Page9 © Hortonworks Inc. 2014
Apache Ambari
Apache Ambari is a 100% open source
framework for provisioning, managing and
monitoring Apache Hadoop clusters
AMBARI WEB
Others
compute
&
storage
. . .
. . .
. .
compute
&
storage
.
.EXTEND
AMBARI REST API
AMBARI SERVER
PROVISION | MANAGE | MONITOR
Integration With Existing Operations Tools
OPERATE
AMBARI STACKS
Page10 © Hortonworks Inc. 2014
100% Apache Open Source
2014
April
Apache Ambari 1.5 Released
Adds support for Hortonworks Data Platform 2.1
Apache Ambari Graduates to Top Level Project
2013
Dec
2014
June
Apache Ambari 1.6 Released
Adds new operational and extensibility capabilities
Page11 © Hortonworks Inc. 2014
Agenda
Ambari
Overview
New
Ambari
Features
Demo Q & A
Page12 © Hortonworks Inc. 2014
New Ambari Features
Operating at Scale
• Maintenance Mode
• Rolling Restarts
• Ambari Blueprints
Extensibility
• Ambari Stacks
Page13 © Hortonworks Inc. 2014
Maintenance Mode
• Silence alerts for services and hosts when performing maintenance
• Ability to put Service or Host “Out of Service”
• Retain full operational control during maintenance period
Page14 © Hortonworks Inc. 2014
Rolling Restarts
• Minimize cluster downtime + service impact when making changes
• Ability to initiate a "rolling restart" of components across many hosts
• Optionally include only hosts with configurations changes
Page15 © Hortonworks Inc. 2014
Ambari Blueprints
• API-driven method for consistent and rapid creation of clusters
• Enables automation for dev, test and short-lived cluster use cases
• Encapsulates “Best Practices” for cluster layout and configuration
STACK
DEFINITION
LAYOUT
& CONFIGS
BLUEPRINT INSTANTIATE CLUSTER
Page16 © Hortonworks Inc. 2014
Ambari Stacks
• Ambari operational control is dynamically driven by “Stacks”
• Defines a consistent lifecycle management interface
• Dynamically extend a “Stack”, bring complementary Services to Ambari
AMBARI
SERVER
Stacks
Command
Scripts
Service
Definitions
AMBARI
AGENT/S
AMBARI
AGENT/S
AMBARI
AGENT/S
pythonxml
Repos
Page17 © Hortonworks Inc. 2014
Agenda
Ambari
Overview
New
Ambari
Features
Demo Q & A
Page18 © Hortonworks Inc. 2014
Agenda
Ambari
Overview
New
Ambari
Features
Demo Q & A
Page19 © Hortonworks Inc. 2014
Learn More About Hadoop Cluster Operations
Hortonworks.com/labs/operations/
Learn About It
ambari.apache.org OR
cwiki.apache.org/confluence/display/AMBARI/Ambari
Get It
hortonworks.com/hdp/downloads/
Try It with Hortonworks Sandbox
hortonworks.com/products/hortonworks-sandbox/
Page20 © Hortonworks Inc. 2014
Thank you!

Discover HDP 2.1: Using Apache Ambari to Manage Hadoop Clusters

  • 1.
    Page1 © HortonworksInc. 2014 Discover HDP 2.1 Using Apache Ambari to Manage Hadoop Clusters Hortonworks. We do Hadoop.
  • 2.
    Page2 © HortonworksInc. 2014 Speakers Justin Sears Hortonworks Product Marketing Manager Jeff Sposetti Hortonworks Senior Director of Product Management and Committer for Apache Ambari Mahadev Konar Hortonworks Co-Founder, Committer and PMC Member for Apache Hadoop, Apache Ambari & Apache ZooKeeper
  • 3.
    Page3 © HortonworksInc. 2014 Agenda • Overview of Apache Ambari • New Ambari Features • Demo • Q & A We’ll move quickly: • Attendee phone lines are muted • Text any questions to Mahadev Konar using Webex chat • Questions will be answered at the end of the presentation • Unanswered questions and answers in upcoming FAQ/blog post
  • 4.
    Page4 © HortonworksInc. 2014 OPERATIONS TOOLS Provision, Manage & Monitor DEV & DATA TOOLS Build & Test A Modern Data Architecture APPLICATIONSDATASYSTEM REPOSITORIES RDBMS EDW MPP Business Analytics Custom Applications Packaged Applications Governance &Integration ENTERPRISE HADOOP Security Operations Data Access Data Management SOURCES OLTP, ERP, CRM Systems Documents, Emails Web Logs, Click Streams Social Networks Machine Generated Sensor Data Geolocation Data
  • 5.
    Page5 © HortonworksInc. 2014 HDP 2.1: Enterprise Hadoop HDP 2.1 Hortonworks Data Platform Provision, Manage & Monitor Ambari Zookeeper Scheduling Oozie Data Workflow, Lifecycle & Governance Falcon Sqoop Flume NFS WebHDFS YARN : Data Operating System DATA MANAGEMENT DATA ACCESS GOVERNANCE & INTEGRATION OPERATIONS Script Pig Search Solr SQL Hive/Tez, HCatalog NoSQL HBase Accumulo Stream Storm Others In-Memory Analytics, ISV engines 1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° N HDFS (Hadoop Distributed File System) Batch Map Reduce SECURITY Authentication Authorization Accounting Data Protection Storage: HDFS Resources: YARN Access: Hive, … Pipeline: Falcon Cluster: Knox
  • 6.
    Page6 © HortonworksInc. 2014 HDP 2.1: Enterprise Hadoop HDP 2.1 Hortonworks Data Platform Scheduling Oozie Data Workflow, Lifecycle & Governance Falcon Sqoop Flume NFS WebHDFS YARN : Data Operating System DATA MANAGEMENT DATA ACCESS GOVERNANCE & INTEGRATION Script Pig Search Solr SQL Hive/Tez, HCatalog NoSQL HBase Accumulo Stream Storm Others In-Memory Analytics, ISV engines 1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° N HDFS (Hadoop Distributed File System) Batch Map Reduce SECURITY Authentication Authorization Accounting Data Protection Storage: HDFS Resources: YARN Access: Hive, … Pipeline: Falcon Cluster: Knox Provision, Manage & Monitor Ambari Zookeeper OPERATIONS
  • 7.
    Page7 © HortonworksInc. 2014 Agenda Ambari Overview New Ambari Features Demo Q & A
  • 8.
    Page8 © HortonworksInc. 2014 Driving Themes for Apache Ambari Operate Hadoop at Scale Integrate with the Enterprise Extend for the Ecosystem
  • 9.
    Page9 © HortonworksInc. 2014 Apache Ambari Apache Ambari is a 100% open source framework for provisioning, managing and monitoring Apache Hadoop clusters AMBARI WEB Others compute & storage . . . . . . . . compute & storage . .EXTEND AMBARI REST API AMBARI SERVER PROVISION | MANAGE | MONITOR Integration With Existing Operations Tools OPERATE AMBARI STACKS
  • 10.
    Page10 © HortonworksInc. 2014 100% Apache Open Source 2014 April Apache Ambari 1.5 Released Adds support for Hortonworks Data Platform 2.1 Apache Ambari Graduates to Top Level Project 2013 Dec 2014 June Apache Ambari 1.6 Released Adds new operational and extensibility capabilities
  • 11.
    Page11 © HortonworksInc. 2014 Agenda Ambari Overview New Ambari Features Demo Q & A
  • 12.
    Page12 © HortonworksInc. 2014 New Ambari Features Operating at Scale • Maintenance Mode • Rolling Restarts • Ambari Blueprints Extensibility • Ambari Stacks
  • 13.
    Page13 © HortonworksInc. 2014 Maintenance Mode • Silence alerts for services and hosts when performing maintenance • Ability to put Service or Host “Out of Service” • Retain full operational control during maintenance period
  • 14.
    Page14 © HortonworksInc. 2014 Rolling Restarts • Minimize cluster downtime + service impact when making changes • Ability to initiate a "rolling restart" of components across many hosts • Optionally include only hosts with configurations changes
  • 15.
    Page15 © HortonworksInc. 2014 Ambari Blueprints • API-driven method for consistent and rapid creation of clusters • Enables automation for dev, test and short-lived cluster use cases • Encapsulates “Best Practices” for cluster layout and configuration STACK DEFINITION LAYOUT & CONFIGS BLUEPRINT INSTANTIATE CLUSTER
  • 16.
    Page16 © HortonworksInc. 2014 Ambari Stacks • Ambari operational control is dynamically driven by “Stacks” • Defines a consistent lifecycle management interface • Dynamically extend a “Stack”, bring complementary Services to Ambari AMBARI SERVER Stacks Command Scripts Service Definitions AMBARI AGENT/S AMBARI AGENT/S AMBARI AGENT/S pythonxml Repos
  • 17.
    Page17 © HortonworksInc. 2014 Agenda Ambari Overview New Ambari Features Demo Q & A
  • 18.
    Page18 © HortonworksInc. 2014 Agenda Ambari Overview New Ambari Features Demo Q & A
  • 19.
    Page19 © HortonworksInc. 2014 Learn More About Hadoop Cluster Operations Hortonworks.com/labs/operations/ Learn About It ambari.apache.org OR cwiki.apache.org/confluence/display/AMBARI/Ambari Get It hortonworks.com/hdp/downloads/ Try It with Hortonworks Sandbox hortonworks.com/products/hortonworks-sandbox/
  • 20.
    Page20 © HortonworksInc. 2014 Thank you!