The Future of Apache Ambari

Future of Apache Ambari
DataWorks Summit 2017
San Jose, USA
06/14/2017

2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Speakers
Apache Ambari PMC
Staff Engineer @ Hortonworks
jaimin@apache.org
Apache Ambari PMC
Principal Engineer @ Hortonworks
jluniya@apache.org
Jaimin Jetly Jayush Luniya

Agenda

Agenda
 Ambari Overview
 v.Next Architecture
 Management Packs
 Multi Everything
 Upgrades
 UI Enhancements
 Other Improvements

Ambari Overview

Apache Ambari
Ambari Server
Provision | Manage | Monitor
Ambari REST API
Ambari Stacks
Apache Ambari is a 100% open source
platform for provisioning, managing and
monitoring Apache Hadoop clusters
Other Operations Tools
AmbariAgents

Lifecycle Management

Ambari Architecture

Limitations
One-One Relationships
Monolithic
 Single monolithic stack
 Entire cluster upgrade
 Ambari upgrades for refreshing stack definitions
Ambari Cluster
Cluster Stack
Stack
Service
Version
Cluster
Service
Instance
Host
Component
Instance
Ambari
Hosting
Platform

v.Next Architecture

Vision
 Refactor at service level
 Replace monolithic stacks
 Decouple Ambari core release
 Independent release cycles
 Modular upgrades
 3rd party onboarding platform
 Multiple Clusters
 Multiple Stacks
 Multiple Services
 Multiple Service Versions
 Multiple Host Components
 Multiple Hosting Platforms
Modular Multi Everything

New Concepts
 Deployment artifact
 Individual ownership of services
 Independent release cycles
 Examples:
o HDFS-3.0.0-packlet
o SPARK-2.0.0-packlet
 Distribution artifact
 Composed of multiple packlets
 Modular release & upgrades
 Examples:
o HDP-3.0.0-mpack
o HDF-3.1.0-mpack
Packlets Management Packs (Mpacks)
Mpack Repository
 Catalog of management packs
 Compatibility between mpacks
 Examples:
o Hortonworks Mpack Repository

New Concepts
 Runtime concept
 Logical grouping of services
 Examples:
o Core Service Group
o Stream Service Group
 Defined in management packs
 Prescribes how to deploy cluster
 Examples:
o Shared v/s Dedicated Zookeeper
Service Groups Deployment Models

Cluster2
Cluster1
Ambari Hosting Platform
v.Next Architecture
ZK1 HDFS YARN HBASE
Core SG
NIFIZK2 KAFKA
Stream SG
Mpack Repository
HDP
3.0.0
HDF
3.1.0
HDF
3.2.0
Ambari
BROKER-1
BROKER-2
BROKER-3
HBASE2
Yapp SG
YarnApp
1.0.0
Yarn Hosting Platform
HMaster
RServer-1
RServer-2

Management Packs

Management Packs
 Decouple Ambari Core release
 Replacement for monolithic stacks
 Composition of verified combination of packlets
 Support dynamic installation
 Mpacks released and upgraded independently
Release Management

Management Packs
 Support multiple mpack repositories
 Search and discover management packs
 Compatibility between management packs
 Recommendations for mpack bundles
Add-on Mpacks
 First class support for 3rd party services
 Mpacks provide a third party onboarding platform
 Independently upgradable services
Mpack Repositories

Multi Everything

Multi Everything
 Logical grouping of services
Service Groups
Cluster v/s Service Group
 Cluster – Uniquely identifies collection of physical hosts
 Service Group – Logical grouping of correlated services
Mpack v/s Service Group
 Mpack – Delivery concept
 Service Group – Runtime concept
 Inheritance to recommend service dependencies
Service Group Inheritance

Multi Everything
 Defined in management packs
 Enriched user deployment experience
 Prescriptions for deploying a cluster
o Service groups to create
o Service dependencies
 Examples:
Deployment Models
Core SG
Stream SG
ZK HDFS
KAFKA
Core SG
Stream SG
ZK1 HDFS
ZK2 KAFKASAM SAM
Shared Zookeeper Dedicated Zookeeper

Multi Everything
Talk: Apache Hadoop Yarn: Present and Future - Vinod Vavilapalli
 Long Lived Yarn Services - YARN-896
 First class support for Yarn Services - YARN-4692 (Yarn Assemblies)
 Multiple Hosting Platforms
o Ambari Hosting Platform
o Yarn Hosting Platform
 Ambari should enable first class support for Yarn services
 Mpacks provide delivery platform for Yarn Services & Yarn Assemblies
Yarn Hosted Services

Upgrades

Upgrades
 Modular upgrades
 Ability to upgrade subset of services in a cluster
 Patch services without affecting other parts of the cluster
 Performance improvements
Goals
Mpack Upgrades
 Upgrade packs defined at mpack level
 Upgrade subset of services in a cluster that belong to a mpack
 Download new version of a mpack and kickoff an upgrade

Upgrades
 Upgrade mechanism for hotfix releases
 Selectively patch components without restarting other services
 Purely Version Definition File (VDF) driven; no configuration changes
 Does not require a new management pack; uses same upgrade pack
 Applied patches can be reverted; no finalization step
Patch Upgrades
Service Upgrades
 VDF driven if no configurations changes needed
 Mpack driven for more complicated service upgrades
 No revert capability (similar to current cluster upgrades)

UI Enhancements

UI Enhancements
 Cosmetic changes
o New style for all UI elements
o New design for all pages
 Consistent experience
o New bootstrap skin
 Utilize available screen space
o Expandable Navigation bar
New Look & Feel

UI Enhancements – Install Wizard

UI Enhancements – Dashboard

UI Enhancements – Alerts

UI Enhancements
Ambari REST API explorer
 Swagger Integration
 Interactive console to execute REST APIs
 REST API documentation

UI Enhancements – REST API Explorer
Demo

Other Improvements

Other Improvements
 Topic: Tuning Apache Ambari Performance for Big Data at Scale
 Speakers: Alejandro, Aravindan
 Performance: Ambari Scale Testing
 Upgrade: Host Ordered Upgrades
 AMS: Anomaly Detection
 AMS: Extensible - Support other storage choices
 AMS: Scalability
Talk

Summary

mpacksmodular packlets
service groups multi
yarn hostedadd-ons
performance

Apache JIRAs
 AMBARI-14714 – Multi Everything Architecture
Umbrella Apache EPIC
 AMBARI-19621 – Mpack Based Operations Model
 AMBARI-20463 – Multi Service Instances
 AMBARI-20465 – Multi Host Component Instances
 AMBARI-17353 – Yarn Hosted Services
 AMBARI-12556 – Patch Upgrades
 AMBARI-20466 – Multi Cluster
 AMBARI-20435 - Swagger Integration
Other Apache EPICs
Release Timelines
 Features will be incrementally delivered in Apache Ambari 3.x releases.

Thank You

The Future of Apache Ambari

More Related Content

What's hot

Similar to The Future of Apache Ambari

More from DataWorks Summit

Recently uploaded

The Future of Apache Ambari