Streamline Apache Hadoop Operations with Apache Ambari and SmartSense

1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Streamline Apache Hadoop
Operations with Apache Ambari
and SmartSense

2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Streamlining Apache Hadoop Operations with Ambari & SmartSense
Roni Fontaine
Director Product
Marketing
Paul Codding
Director Product
Management

Agenda
 Newest features and highlights in Ambari 2.5
 How to double Hadoop performance using SmartSense 1.4
 Flexible enterprise support model options

What’s New in Ambari 2.5
 Service Auto Start (AMBARI-2330)
 DB Inconsistency Self-Healing (AMBARI-18990)
 Simplified Log Rotation Configuration (AMBARI-16880)
 HDFS TopN User & Operation Visualization (AMBARI-19320)
 Download All Client Configurations (AMBARI-19275)
 Configuration Change Communication (AMBARI-19572)
 Add/Remove JournalNodes (AMBARI-7748)
 Ignore Host Pre-Check when adding Hosts (AMBARI-18817)
 Grafana dashboard for Ambari (AMBARI-17589)
 AMS Collector High Availability TP (AMBARI-15901)
 Password Credential Store Management (AMBARI-18650)
 Post-user-creation script hook (AMBARI-18722)
 ZK ACL and SASL Configuration (AMBARI-17324)
 Ambari SPNEGO support (AMBARI-18365)
Core Features
Security Features
 HDP “Fenton” Support
 Port Preserving HS2 Rolling Upgrade (AMBARI-18591)
 Log Search TP Update (AMBARI-18821)
 Built-In SNMP MIB (AMBARI-19257)
 SmartSense Mandatory Install (AMBARI-18346)
 RegionServer GC Configuration Optimization (AMBARI-19573)
Core Features Continued

Ambari 2.5 + HDP Support Matrix
 Added support for HDP 2.6
 Deprecated support for HDP 2.4, and removed support for HDP 2.2
HDP 2.6 HDP 2.5 HDP 2.4 HDP 2.3 HDP 2.2
Ambari 2.5
Ambari 2.4
Ambari 2.2.1
Ambari 2.2
deprecated
deprecated deprecated
deprecated
deprecated

Ambari 2.5 + OS Support Matrix
 Added support for Ubuntu 16 (for HDP 2.6 Only)
 Dropped support for Ubuntu 12
RHEL 6 RHEL 7 Debian 7 SLES 11 SLES 12 Ubuntu 12 Ubuntu 14 Ubuntu 16
Ambari 2.5
HDP “Fenton” Only
Ambari 2.4
HDP “Erie” Only
Ambari 2.2

Agenda
What’s New in Ambari 2.5.0
Feature Highlights: Service Auto Start

How it works
Ambari Agent
Host
Ambari Agent
Host
Ambari Agent
Host
Ambari
Server
Ambari
DB
LDAP
AuthN
Host Restart
• Agent checks in with Server
• Server asks agent to check status of
deployed components
• Agent reports back state
• Server asks agent to start components
not in the desired state
HDP Component
Component Stops

Agenda
Feature Highlights: Add/Remove JournalNodes

Agenda
Feature Highlights: Simplified Log Rotation
Configuration

Agenda
Feature Highlights: Log Search Improvements

Log Search Demo

Agenda
Feature Highlights: Configuration Change
Communication

Stack Advisor Behavioral Changes
 Add Service
 Delete Service
 Add Host Component
 Move Master
 HA Wizards
 Delete Host
 Add/Remove ZooKeeper Server
Goal: Ensure users are aware of configuration
changes related to the activity they are
performing.
What we found: Identified multiple
locations in which configurations were
being changed without notifying the user
explicitly
What Changed: New pop-ups and more
communication when changes are
required

Agenda
Feature Highlights: Download All Client
Configurations

Client Configuration

Agenda
Feature Highlights: HDFS TopN User & Operation
Visualization

HDFS TopN User & Operation Visualization*
Operations
Users
Users / Operations
*Only works with HDP 2.6

Agenda
Feature Highlights: AMS Collector High Availability
TP

AMS Current state
AMBARI
Collector API
GRAFANA
HBASE
PHOENIX
SYSTEM
MONITORS
HDP
SERVICES
SINKS
METRICS COLLECTOR
API consumers
 Sinks – HDFS, YARN, HBASE, STORM, KAFKA, FLUME, ACCUMULO, LogSearch, Hive, Nifi
 Monitors – System metrics
 Grafana
 Ambari

Agenda
Feature Highlights: Post User Creation Hook

Post User Creation Hook
 Disabled by default
 Enabled with two properties
– ambari.post.user.creation.hook.enabled=true
– ambari.post.user.creation.hook=/var/lib/ambari-
server/resources/scripts/post-user-creation-hook.sh
 Works with manual user creation as well as LDAP sync
 Can be run as a one-off command as well

Hortonworks Connected Data Platforms and Solutions
Data Services
Hortonworks Solutions
Enterprise Data
Warehouse Optimization
Cyber Security and
Threat Management
Internet of Things
and Streaming Analytics
Data Center
Hortonworks Data Suite
HDFHDP
Hortonworks
Connection
Cloud
Hortonworks Data Cloud
AWS HDInsight
Hortonworks Connection
Enablement Subscription
SmartSense™
Premier Operational Support
Educational Services
Professional Services
Community Connection

Accelerate Case Resolution
Prevent Issues
Understand Your Cluster

SmartSense Today – Accelerate Case Resolution
SmartSense provides Hadoop Operators with an Ambari Integrated tool to quickly capture
diagnostic information for specific services and hosts into a single “Bundle” that’s automatically
uploaded to Hortonworks Support.
Significantly reduces the back-and-forth nature of troubleshooting issues.
A M B A R I
O P S
H O R TO N WO R K S
S U P P O R T
S U P P O R T
C A S E
S m a r t S e n s e
S E R V E R
B U N D L E
G AT E WAY

Prevent Issues

SmartSense Today – Prevent Issues
SmartSense analyzes Bundles for configuration issues – recommendations are produced and
made available for each cluster in the Hortonworks Support Portal
Recommendations prevent operational issues, and improve performance and overall cluster
throughput.

After Applying only 3 SmartSense Recommendations
They can now run 1200 concurrent jobs
...with only 350 waiting jobs at peak hours
Issue: YARN @ capacity, struggling to add
more use cases
Before SmartSense
 Could only run 500 jobs concurrently
 1100 jobs would be pending waiting for
resources at peak hours
With SmartSense = 2X Throughput Improvement

Prevent Issues

“Who’s creating all of these small files in HDFS!?”
“What are my top 10 most active users, and longest running
jobs?”
“How much should I charge users for their cluster resource
use?”
SmartSense Today – Understand Your Cluster

Chargeback Reporting

HDFS Dashboards

HDFS Dashboards
YARN Dashboards

Impact of Hortonworks SmartSense
0
200
400
600
800
1000
1200
1400
Without
SmartSense
With
SmartSense
Concurrent Jobs
B U N D L E
2X Throughput
Improvement
Address 30% of
Issues
Configuration Issues
Avoid 10% of
Sev1 Issues
Production Down
Single-Bundle
Case Resolution
25% of the Time
SmartSense
Troubleshooting Bundle

Hortonworks Connection Ensures Success of Your Big Data Journey

New Hortonworks Flex Support Subscription
Universal, Usage-based
Support Subscription
Cloud & On-Prem
HDCloud
IaaS
On-Prem
Single, flexible, portable support
for transition to cloud
New support offering for Spark
Data Science, ETL, EDW-Analytics
in the Cloud
Performance optimization with
Hortonworks SmartSense

Hortonworks Data Cloud for AWS
 Pre-tuned for use with AWS
 Powered by HDP
 Focused on business agility
 Prescriptive, ephemeral use cases
• Data Science (Apache Spark & Zeppelin)
• Analytics (Apache HIVE)
• ETL (Apache HIVE & Spark)
• Business Intelligence (OLAP/Druid)

EASE OF USE: Choose from a set of pre-
tuned and pre-configured templates.

Choose from a Set of Prescriptive, Ephemeral Workload Clusters
Data Science
Spark 1.6, 2.1
Business
intelligence
(OLAP) Druid TP
Analytics &
Reporting with
HIVE 2.2 & LLAP
ETL with Spark

Try it! Hortonworks Data Cloud for AWS Marketplace
How –to Video: https://hortonworks.com/video/hd-cloud-aws/
Try it Now: https://aws.amazon.com/marketplace/pp/B01LXOQBOU

Flex Support Subscription Components

Access to Expertise Plus Freedom, Agility and Flexibility
Expertise for Data Science, ETL and Analytics in HDCloud
Match support costs to usage patterns
Migrate infrastructure on your own terms
Support for Dev, QA & ephemeral production
Built-In Performance Optimization

Questions?
References
https://hortonworks.com/apache/ambari/
hortonworks.com/services/support/smartsense/
hortonworks.com/services/support/flex/
hortonworks.com/products/cloud/aws/

Streamline Apache Hadoop Operations with Apache Ambari and SmartSense

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (8)

Similar to Streamline Apache Hadoop Operations with Apache Ambari and SmartSense

Similar to Streamline Apache Hadoop Operations with Apache Ambari and SmartSense (20)

More from Hortonworks

More from Hortonworks (20)

Recently uploaded

Recently uploaded (20)

Streamline Apache Hadoop Operations with Apache Ambari and SmartSense

Editor's Notes