This document discusses moving from a stack-based operations model in Apache Ambari to a management pack (Mpack) based operations model. Some key points: - The current model uses monolithic stacks that are embedded in Ambari and don't allow mixing services from different vendors. Mpacks will allow for a more modular, independent approach. - Mpacks will allow for individual services and third-party add-ons to have independent release cycles rather than being tied to a larger stack release. - The new model aims to support modular upgrades where only a subset of services in a cluster would be upgraded at a time if they belong to an upgraded mpack. This provides more flexibility compared to requiring a full cluster

1. Jayush Luniya
Apache Ambari PMC
Principal Software Engineer @ Hortonworks
MPACK BASED OPERATIONS MODEL

2. Agenda
Agenda

3. Agenda
 Stack Based Operations Model
 Management Packs – V1
 Limitations
 Mpacks Based Operations Model
 Summary
 Q&A

4. Stack Based Operations Model
Agenda

5. Stack Based Operations Model

6. Stack Based Operations Model
 Ambari built on a standardized stack based operations model
 Pluggable architecture; no vendor lock-in
 Stacks wrap services of all shapes and sizes
 Consistent definition & lifecycle control layer
 Consistent approach for managing & monitoring services
 Stacks embedded in Ambari

7. Management Packs – V1
Agenda

8. Management Packs – V1
 Introduced in 2.4
 V1 release based on existing stack based operations model
 First take on decoupling stack releases
 Bundles stacks, add-on services
 Final stack definition is overlay of multiple mpacks
 Stacks don’t need to be embedded in Ambari

9. Management Packs – V1

10. Service Level Extension Points
Extension Apache JIRA Apache Wiki
Stack Advisor Extensions AMBARI-15226 Ambari Wiki
Role Command Order Extensions AMBARI-9363 Ambari Wiki
Quick Links Extensions AMBARI-11268 Ambari Wiki
Service Repos Extensions AMBARI-15538 Ambari Wiki
Upgrade Pack Extensions AMBARI- 15388 Ambari Wiki

11. Limitations
Agenda

12. Limitations
 Cluster attached to single stack
 Cannot deploy HDFS from HDP, NiFi from HDF in same cluster
 Entire cluster upgrade
 Stacks are monolithic
 Ambari upgrade for refreshing stacks
 No clear upgrade story for add-on services
 Ownership of service deployment scripts

13. Mpack Based Operations Model
Agenda

14. Goals
 Replacement for monolithic stacks
 Cluster to multiple mpacks
 Scenario: Deploy HDP +HDF cluster
 Decouple Ambari core release
 Independent release cycles
 First class support for add-on services
 Support dynamic installation
 Modular upgrades

15. New Concepts
 Deployment artifact
 Individual service ownership
 Independent release cycles
 Examples
 HDFS-3.0.0-packlet
 SPARK-2.0.0-packlet
 Distribution artifact
 Composition of packlets
 Modular release & upgrades
 Examples
 ODPi-2.0.0-mpack
 HDP-3.0.0-mpack
 HDF-3.1.0-mpack
Packlets Management Packs (Mpacks V2)

16. New Concepts
 Catalog of management packs
 Compatibility between mpacks
 Search and discover mpacks
 Support multiple mpack repos
 Examples
 ODPi Mpack Repository
 Hortonworks Mpack Repository
Mpack Repository

17. Mpack Based Operations Model

18. Mpack Processing Engine
Mpack Repository
DistroX
3.0.0
DistroY
3.2.0

19. Release Management
 Completely decouple Ambari releases
 Service ownership with domain experts
 Independent release cadence

20. Mpack Based Upgrades
 Modular upgrades
 Upgrade subset of services in a cluster that belong to mpack
 Upgrade packs defined at mpack level
 Download new version of mpack and kickoff upgrade
 Upgrade Pack
 “upgrade-from” instead of “upgrade-to”

21. Add-on Mpacks
 First class support for 3rd party services
 Mpacks provide a 3rd party onboarding platform
 Add-on services not overlaid into stack
 Independent release management
 Independent and modular upgrades

22. Summary
Agenda

23. Summary
 Modular build & release
 Independent release mgmt
 Modular upgrades
 3rd party onboarding platform
 Multi-everything vision
 Service level refactoring
 Resource management library
 No network connectivity
 Ownership
 Testing
Key Takeaways Challenges

24. Summary
 Future
 YARN Services
 Umbrella Apache Epic
 AMBARI-14714 – Multi Everything Architecture
 Apache Epic
 AMBARI-19621 – Mpack Based Operations Model
 Dataworks Summit Talk
 Future of Apache Ambari – Jaimin, Jayush

25. Q&A
Agenda

26. Backup Slides
Agenda