Discover.hdp2.2.ambari.final[1]

© Hortonworks Inc. 2014
Discover HDP 2.2:
Using Apache Ambari to Manage Hadoop Clusters
Hortonworks. We do Hadoop.

Speakers
Justin Sears
Hortonworks Product Marketing Manager
Jeff Sposetti
Hortonworks Senior Director of Product Management and
Committer for Apache Ambari
Mahadev Konar
Hortonworks Co-Founder, Committer and PMC Member for
Apache Hadoop, Apache Ambari & Apache ZooKeeper

Agenda
•  Introduction to Apache Ambari
•  New Ambari Innovation in HDP 2.2
–  Configuration Enhancements, including Versioning & History
–  Ambari Administration, including Views Framework
–  Ambari Stacks “Stack Advisor”
•  Demo
•  Q & A
We’ll move quickly:
•  Attendee phone lines are muted
•  Text any questions to Mahadev Konar using Webex chat
•  Questions answered at the end
•  Unanswered questions and answers in upcoming blog post

Big Data, Hadoop & Data Center Re-platforming
Business Drivers
•  From reactive analytics
to proactive interactions
•  Insights that drive
competitive advantage
& optimal returns
Financial Drivers
•  Cost of data systems, as
% of IT spend,
continues to grow
•  Cost advantages of
commodity hardware
& open source software
$
Technical Drivers
•  Data is growing
exponentially & existing
systems overwhelmed
•  Predominantly driven by
NEW types of data that
can inform analytics
There is an inequitable balance between vendor and customer in the market

Clickstream
Capture and analyze
website visitors’ data
trails and optimize
your website
Sensors
Discover patterns in
data streaming
automatically from
remote sensors and
machines
Server Logs
Research logs to
diagnose process
failures and prevent
security breaches
New Types of DataHadoop Value:
Sentiment
Understand how
your customers feel
about your brand
and products –
right now
Geographic
Analyze location-
based data to
manage operations
where they occur
Unstructured
Understand patterns
in files across millions
of web pages, emails,
and documents

A Shift from Reactive to Proactive Interactions
HDP and Hadoop allow
organizations to use
data to shift interactions
from…
Reactive
Post Transaction
Proactive
Pre Decision
…to Real-time PersonalizationFrom static branding
…to repair before breakFrom break then fix
…to Designer MedicineFrom mass treatment
…to Automated AlgorithmsFrom Educated Investing
…to 1x1 TargetingFrom mass branding
A shift in Advertising
A shift in Financial Services
A shift in Healthcare
A shift in Retail
A shift in Telco

Enterprise Goals for the Modern Data Architecture
•  Consolidate siloed data sets structured
and unstructured
•  Central data set on a single cluster
•  Multiple workloads across batch
interactive and real time
•  Central services for security, governance
and operation
•  Preserve existing investment in current
tools and platforms
•  Single view of the customer, product,
supply chain
APPLICATIONSDATASYSTEM
Business
Analytics
Custom
Applications
Packaged
Applications
RDBMS
EDW
MPP
YARN: Data Operating System
1 ° ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° ° N
Interactive Real-TimeBatch
CRM
ERP
Other
1 ° ° °
° ° ° °
HDFS
(Hadoop Distributed File System)
SOURCES
EXISTING

Systems

Clickstream
Web

&Social

Geoloca9on
Sensor

&
Machine

Server

Logs

Unstructured

YARN Transformed Hadoop & Opened a New Era
YARN
The Architectural
Center of Hadoop
•  Common data platform, many applications
•  Support multi-tenant access & processing
•  Batch, interactive & real-time use cases
(Cluster Resource Management)
1 ° ° ° ° ° ° °
° ° ° ° ° ° ° °
Script
Pig
SQL
Hive
Tez
Tez
Java
Scala
Cascading
Tez
° °
° °
° ° ° ° °
° ° ° ° °
Others
ISV
Engines
HDFS
Stream
Storm
Search
Solr
NoSQL
HBase
Accumulo
Slider
Slider
BATCH, INTERACTIVE & REAL-TIME DATA ACCESS
In-Memory
Spark

YARN Extends Hadoop to Other Data Center Leaders
YARN
The Architectural
Center of Hadoop
•  Common data platform, many applications
•  Support multi-tenant access & processing
•  Batch, interactive & real-time use cases
•  Supports 3rd-party ISV tools
(ex. SAS, Syncsort, Actian, etc.)
YARN Ready Applications
Facilitates ongoing innovation and enterprise adoption via
ecosystem of new and existing “YARN Ready” solutions
1 ° ° ° ° ° ° °
° ° ° ° ° ° ° °
Script
Pig
SQL
Hive
Tez
Tez
Java
Scala
Cascading
Tez
° °
° °
° ° ° ° °
° ° ° ° °
Others
ISV
Engines
HDFS
Stream
Storm
Search
Solr
NoSQL
HBase
Accumulo
Slider
Slider
In-Memory
Spark

Enterprise Hadoop: Central Set of Services
1 ° ° ° ° ° ° °
° ° ° ° ° ° ° °
° °
° °
° ° ° ° °
° ° ° ° °
Enables Apache Hadoop to be
an Enterprise Data Platform
with centralized services for:
•  Governance
•  Operations
•  Security
Everything that plugs into
Hadoop inherits these services
Provision,
Manage &
Monitor
Ambari
Zookeeper
Scheduling
Oozie
Load data and
manage
according
to policy
Deploy and
effectively
manage the
platform
Provide layered
approach to
security through
Authentication,
Authorization,
Accounting, and
Data Protection
SECURITYGOVERNANCE OPERATIONS
Script
Pig
SQL
Hive
Java
Scala
Cascading
Stream
Storm
Search
Solr
NoSQL
HBase
Accumulo
In-Memory
Spark
Others
ISV
Engines
HDFS
Tez
Slider
Slider
Tez
Tez

Hortonworks Data Platform 2.2
HDP Delivers Enterprise Hadoop
1 ° ° ° ° ° ° °
° ° ° ° ° ° ° °
Script
Pig
SQL
Hive
Tez
Tez
Java
Scala
Cascading
Tez
° °
° °
° ° ° ° °
° ° ° ° °
HDFS
Stream
Storm
Search
Solr
NoSQL
HBase
Accumulo
Slider
Slider
SECURITYGOVERNANCE OPERATIONSBATCH, INTERACTIVE & REAL-TIME DATA ACCESS
In-Memory
Spark
Provision,
Manage &
Monitor
Ambari
Zookeeper
Scheduling
Oozie
Data Workflow,
Lifecycle &
Governance
Falcon
Sqoop
Flume
Kafka
NFS
WebHDFS
Authentication
Authorization
Audit
Data Protection
Storage: HDFS
Resources: YARN
Access: Hive
Pipeline: Falcon
Cluster: Ranger
Cluster: Knox
Deployment ChoiceLinux Windows Cloud
YARN is the architectural
center of HDP
•  Common data set across all
applications
•  Batch, interactive & real-time
workloads
•  Multi-tenant access & processing
Provides comprehensive
enterprise capabilities
•  Governance
•  Security
•  Operations
Enables broad
ecosystem adoption
•  ISVs can plug directly into Hadoop
The widest range of deployment options
•  Linux & Windows
•  On premises & cloud
Others
ISV
Engines
On-Premises

Hortonworks Data Platform 2.2
HDP Delivers Enterprise Hadoop
1 ° ° ° ° ° ° °
° ° ° ° ° ° ° °
Script
Pig
SQL
Hive
Tez
Tez
Java
Scala
Cascading
Tez
° °
° °
° ° ° ° °
° ° ° ° °
HDFS
Stream
Storm
Search
Solr
NoSQL
HBase
Accumulo
Slider
Slider
SECURITYGOVERNANCE BATCH, INTERACTIVE & REAL-TIME DATA ACCESS
In-Memory
Spark
Scheduling
Oozie
Data Workflow,
Lifecycle &
Governance
Falcon
Sqoop
Flume
Kafka
NFS
WebHDFS
Authentication
Authorization
Audit
Data Protection
Storage: HDFS
Resources: YARN
Access: Hive
Pipeline: Falcon
Cluster: Ranger
Cluster: Knox
Deployment ChoiceLinux Windows Cloud
YARN is the architectural
center of HDP
•  Common data set across all
applications
•  Batch, interactive & real-time
workloads
•  Multi-tenant access & processing
Provides comprehensive
enterprise capabilities
•  Governance
•  Security
•  Operations
Enables broad
ecosystem adoption
•  ISVs can plug directly into Hadoop
The widest range of deployment options
•  Linux & Windows
•  On premises & cloud
Others
ISV
Engines
On-Premises
OPERATIONS
Provision,
Manage &
Monitor
Ambari
Zookeeper

Introduction to Apache Ambari

How do you Operate a Hadoop Cluster?
Apache Ambari is a
framework to provision,
manage and monitor
Hadoop clusters

© Hortonworks Inc. 2011 – 2014. All Rights Reserved
Apache Ambari Themes
Operate
Hadoop
at

Scale

Deliver
the
core
opera-onal
capabili-es
to
provision,
manage

and
monitor
Hadoop
clusters
at
scale.

Integrate
with
the

Enterprise

Robust
API
for
integra-on
with
exis9ng
enterprise
systems,

such
as
Teradata
Viewpoint
and
MicrosoL
SCOM.

Extend
for
the

Ecosystem

Provide
an
extensible
plaNorm
for
Enterprises,
Partners
and

the
Community,
via
Stacks
and
Views.

© Hortonworks Inc. 2011 – 2014. All Rights Reserved
What’s New in Ambari 1.7.0
Core Services
•  ResourceManager HA
•  Capacity Scheduler Refresh Queues
•  HDFS Rebalance
•  Service Config Versioning + History
•  Manage -env.sh Files
•  Set <final> Config Properties
•  Download Client Configs
Ambari Platform
•  Ambari Administration
•  Ambari Views Framework
•  Ambari Blueprints Export Configs
•  Ubuntu 12 Platform Support
Stacks
•  Support for HDP 2.2
•  Stack Advisor
For a complete list of enhancements…
http://www.slideshare.net/hortonworks/apache-ambari-whats-new-in-170

New in HDP 2.2: Configuration Enhancements

Configuration Versioning and History
•  Service Config Versions (saved per service)
•  List of Config History
•  Compare Versions
•  Filter by “Changed Properties”
•  Revert Changes (i.e. “Make Current”)
•  Audit Log of Changes

Configuration History History of Changes
Filter,
Sort,
Search

Service Configuration Controls
Most Recent Versions (view, compare, revert)
Compare
Versions
Revert
Version
Filter by
“Changed”

New in HDP 2.2: Views Framework

Ambari Extension Points
Ambari
Server
Ambari
AgentAmbari
AgentAmbari
Agent
Ambari
Web
Stacks
Stacks
Stacks
java!js! python!
Ambari Views Ambari Stacks

Ambari Views Framework
Goal: enable the delivery of custom UI experiences in Ambari Web
Developers can extend the Ambari Web interface
•  Views expose custom UI features for Hadoop Services
Ambari Admins can entitle Views to Ambari Web users
•  Entitlements framework for controlling access to Views

Example Views
“Queue Manager” View “Jobs” View

View Components
•  Serve client-side assets (such as HTML + JavaScript)
•  Expose server-side resources (such as REST endpoints)
VIEW

Client-‐side

assets

(.js,
html)

AMBARI
WEB

VIEW

Server-‐side

resources

(java)

AMBARI
SERVER

{rest}!
Hadoop
and other
systems

Versions and Instances
•  Deploy multiple versions and create multiple instances of a view
•  Manage accessibility and usage

Choice of Deployment Model
•  For Hadoop Operators:
Deploy Views in an Ambari Server that is managing a Hadoop cluster
•  For Data Workers:
Run Views in a “standalone” Ambari Server
Ambari
Server
HADOOP

Store
&
Process

Ambari
Server
Operators
manage the
cluster, may
have Views
deployed
Data
Workers use
the cluster
and use a
“standalone”
Ambari
Server for
Views

Learn More About Views Framework
https://github.com/apache/ambari/blob/trunk/ambari-views/docs/index.md
https://github.com/apache/ambari/tree/trunk/ambari-views/examples
https://cwiki.apache.org/confluence/display/AMBARI/Views
https://github.com/apache/ambari/tree/trunk/contrib/views

New in HDP 2.2: Stack Advisor

Ambari Stacks
•  Defines a consistent Stack lifecycle interface that can be extended
•  Encapsulates Stack Versions, Services, Components, Dependencies,
Cardinality, Configurations, Commands
•  Dynamically add Stack + Service definitions
AMBARI

{rest}!
<ambari-web>!
Stacks
HDFS
YARN
MR2

Hive

Pig

Oozie
HBase

Storm
Falcon

Stacks In Action
http://hortonworks.com/partners/certified/ops-ready/

Stack Advisor
•  Extends Ambari Stacks to include a “Stack Advisor”
•  Provides recommendations for and performs validation on component
layout & configuration
•  Improves Stack pluggability
•  Exposes new REST endpoints:
/recommendations!
!/validations!
•  REST endpoints used during Cluster Install Wizard and Configs UI

DEMO

Q & A

Thank you!
Learn more at:
hortonworks.com/hadoop/ambari/

Discover.hdp2.2.ambari.final[1]

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Discover.hdp2.2.ambari.final[1]

Similar to Discover.hdp2.2.ambari.final[1] (20)

More from Hortonworks

More from Hortonworks (20)

Recently uploaded

Recently uploaded (20)

Discover.hdp2.2.ambari.final[1]