Apache Ambari is an extensible framework that simplifies provisioning, managing and monitoring Hadoop clusters. Apache Ambari was built on a standardized stack-based operations model. Stacks wrap services of all shapes and sizes with a consistent definition and lifecycle-control layer; thereby providing a consistent approach for managing and monitoring the services. This also provided a natural extension point for operators and the community to bring in their own add-on services and “plug-in” the new services into the stack. However, one of the fundamental limitations of the current Apache Ambari architecture has been that there is a strong one-on-one coupling between entities. For instance, a cluster is tied to a single stack and a Hadoop operator can only deploy services defined in that stack, a cluster can have only a single instance of a service and a host can have only a single instance of a component. Taking into consideration various use case scenarios that cannot be enabled due to these limitations there is a growing need to revamp the Ambari architecture. In this talk, we propose a revamped Apache Ambari architecture that will open up the floodgates for a wide range of scenarios that wouldn’t have been possible thus far. We will focus the discussion on a new mpack-based operations model that will replace the stack-based operations model. A management package is a self-contained deployment artifact that includes all the details for deploying, managing and upgrading a set of services bundled in the package. A third-party provider can also build their own management package containing their custom services. This eliminates the need to plug-in their services into a stack and also can define their own upgrade story for these custom services. A Hadoop operator will be able to deploy a Hadoop cluster with a mix of services across multiple packages instead of being limited to a single stack. For example, it would be possible to deploy a cluster with HDFS from HDP and NIFI from HDF. Further, we will also discuss about the architectural changes needed to enable a multi instance architecture in future Ambari releases to support deploying multiple instances of a service in a cluster, deploying multiple instances of a component on a host as well as future proofing the Ambari architecture to leverage some of the advancements happening in the Hadoop community like YARN services (YARN-4692). We will wrap up the conversation with a brief overview of other improvements planned for future releases of Ambari.

1. Future of Apache Ambari
DataWorks Summit 2017
San Jose, USA
06/14/2017

2. 2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Speakers
Apache Ambari PMC
Staff Engineer @ Hortonworks
jaimin@apache.org
Apache Ambari PMC
Engineering Manager @ Hortonworks
jluniya@apache.org
Jaimin Jetly Jayush Luniya

3. 3 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Agenda

4. 4 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Agenda
 Ambari Overview
 v.Next Architecture
 Management Packs
 Multi Everything
 Upgrades
 UI Enhancements
 Other Improvements

5. 5 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Ambari Overview

6. 6 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

7. 7 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Apache Ambari
Ambari Server
Provision | Manage | Monitor
Ambari REST API
Ambari Stacks
Apache Ambari is a 100% open source
platform for provisioning, managing and
monitoring Apache Hadoop clusters
Other Operations Tools
AmbariAgents

8. 8 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Lifecycle Management

9. 9 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Ambari Architecture

10. 10 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Limitations
One-One Relationships
Monolithic
 Single monolithic stack
 Entire cluster upgrade
 Ambari upgrades for refreshing stack definitions
Ambari Cluster
Cluster Stack
Stack
Service
Version
Cluster
Service
Instance
Host
Component
Instance
Ambari
Hosting
Platform

11. 11 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
v.Next Architecture

12. 12 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Vision
 Refactor at service level
 Replace monolithic stacks
 Decouple Ambari core release
 Independent release cycles
 Modular upgrades
 3rd party onboarding platform
 Multiple Clusters
 Multiple Stacks
 Multiple Services
 Multiple Service Versions
 Multiple Host Components
 Multiple Hosting Platforms
Modular Multi Everything

13. 13 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
New Concepts
 Deployment artifact
 Individual ownership of services
 Independent release cycles
 Examples:
o HDFS-3.0.0-packlet
o SPARK-2.0.0-packlet
 Distribution artifact
 Composed of multiple packlets
 Modular release & upgrades
 Examples:
o HDP-3.0.0-mpack
o HDF-3.1.0-mpack
Packlets Management Packs (Mpacks)
Mpack Repository
 Catalog of management packs
 Compatibility between mpacks
 Examples:
o Hortonworks Mpack Repository

14. 14 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
New Concepts
 Runtime concept
 Logical grouping of services
 Examples:
o Core Service Group
o Stream Service Group
 Defined in management packs
 Prescribes how to deploy cluster
 Examples:
o Shared v/s Dedicated Zookeeper
Service Groups Deployment Models

15. 15 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Cluster2
Cluster1
Ambari Hosting Platform
v.Next Architecture
ZK1 HDFS YARN HBASE
Core SG
NIFIZK2 KAFKA
Stream SG
Mpack Repository
HDP
3.0.0
HDF
3.1.0
HDF
3.2.0
Ambari
BROKER-1
BROKER-2
BROKER-3
HBASE2
Yapp SG
YarnApp
1.0.0
Yarn Hosting Platform
HMaster
RServer-1
RServer-2

16. 16 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Cluster2
Cluster1
Ambari Hosting Platform
v.Next Architecture
ZK1 HDFS YARN HBASE
Core SG
NIFIZK2 KAFKA
Stream SG
Mpack Repository
HDP
3.0.0
HDF
3.1.0
HDF
3.2.0
Ambari
BROKER-1
BROKER-2
BROKER-3
HBASE2
Yapp SG
YarnApp
1.0.0
Yarn Hosting Platform
HMaster
RServer-1
RServer-2

17. 17 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Management Packs

18. 18 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Management Packs
 Decouple Ambari Core release
 Replacement for monolithic stacks
 Composition of verified combination of packlets
 Support dynamic installation
 Mpacks released and upgraded independently
Release Management

19. 19 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Management Packs
 Support multiple mpack repositories
 Search and discover management packs
 Compatibility between management packs
 Recommendations for mpack bundles
Add-on Mpacks
 First class support for 3rd party services
 Mpacks provide a third party onboarding platform
 Independently upgradable services
Mpack Repositories

20. 20 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Multi Everything

21. 21 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Multi Everything
 Logical grouping of services
Service Groups
Cluster v/s Service Group
 Cluster – Uniquely identifies collection of physical hosts
 Service Group – Logical grouping of correlated services
Mpack v/s Service Group
 Mpack – Delivery concept
 Service Group – Runtime concept
 Inheritance to recommend service dependencies
Service Group Inheritance

22. 22 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Multi Everything
 Defined in management packs
 Enriched user deployment experience
 Prescriptions for deploying a cluster
o Service groups to create
o Service dependencies
 Examples:
Deployment Models
Core SG
Stream SG
ZK HDFS
KAFKA
Core SG
Stream SG
ZK1 HDFS
ZK2 KAFKASAM SAM
Shared Zookeeper Dedicated Zookeeper

23. 23 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Multi Everything
Talk: Apache Hadoop Yarn: Present and Future - Vinod Vavilapalli
 Long Lived Yarn Services - YARN-896
 First class support for Yarn Services - YARN-4692 (Yarn Assemblies)
 Multiple Hosting Platforms
o Ambari Hosting Platform
o Yarn Hosting Platform
 Ambari should enable first class support for Yarn services
 Mpacks provide delivery platform for Yarn Services & Yarn Assemblies
Yarn Hosted Services

24. 24 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Upgrades

25. 25 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Upgrades
 Modular upgrades
 Ability to upgrade subset of services in a cluster
 Patch services without affecting other parts of the cluster
 Performance improvements
Goals
Mpack Upgrades
 Upgrade packs defined at mpack level
 Upgrade subset of services in a cluster that belong to a mpack
 Download new version of a mpack and kickoff an upgrade

26. 26 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Upgrades
 Upgrade mechanism for hotfix releases
 Selectively patch components without restarting other services
 Purely Version Definition File (VDF) driven; no configuration changes
 Does not require a new management pack; uses same upgrade pack
 Applied patches can be reverted; no finalization step
Patch Upgrades
Service Upgrades
 VDF driven if no configurations changes needed
 Mpack driven for more complicated service upgrades
 No revert capability (similar to current cluster upgrades)

27. 27 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
UI Enhancements

28. 28 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
UI Enhancements
 Cosmetic changes
o New style for all UI elements
o New design for all pages
 Consistent experience
o New bootstrap skin
 Utilize available screen space
o Expandable Navigation bar
New Look & Feel

29. 29 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
UI Enhancements – Install Wizard

30. 30 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
UI Enhancements – Dashboard

31. 31 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
UI Enhancements – Dashboard

32. 32 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
UI Enhancements – Alerts

33. 33 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
UI Enhancements
Ambari REST API explorer
 Swagger Integration
 Interactive console to execute REST APIs
 REST API documentation

34. 34 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
UI Enhancements – REST API Explorer
Demo

35. 35 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Other Improvements

36. 36 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Other Improvements
 Topic: Tuning Apache Ambari Performance for Big Data at Scale
 Speakers: Alejandro, Aravindan
 Performance: Ambari Scale Testing
 Upgrade: Host Ordered Upgrades
 AMS: Anomaly Detection
 AMS: Extensible - Support other storage choices
 AMS: Scalability
Talk

37. 37 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Summary

38. 38 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
mpacksmodular packlets
service groups multi
yarn hostedadd-ons
performance

39. 39 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Apache JIRAs
 AMBARI-14714 – Multi Everything Architecture
Umbrella Apache EPIC
 AMBARI-19621 – Mpack Based Operations Model
 AMBARI-20463 – Multi Service Instances
 AMBARI-20465 – Multi Host Component Instances
 AMBARI-17353 – Yarn Hosted Services
 AMBARI-12556 – Patch Upgrades
 AMBARI-20466 – Multi Cluster
 AMBARI-20435 - Swagger Integration
Other Apache EPICs
Release Timelines
 Features will be incrementally delivered in Apache Ambari 3.x releases.

40. 40 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Thank You