Apache Ambari Meetup - Apache Ambari Stack Extensibility

1. APACHE AMBARI
STACK EXTENSIBILITY
Juanjo Marron @ IBM
Apache Ambari Committer
Jayush Luniya @ Hortonworks
Apache Ambari PMC

2. Agenda
• Ambari Stack Framework
• Current State of Affairs
• New Extensibility Features
• Stack Featurization
• Service Level Extension Points
• Stack Extensions
• Ambari Management Packs
• Future Goals
• Q&A

3. AMBARI STACK
FRAMEWORK

4. Anatomy of Ambari Extension Points

5. Stack Terminologies
Term Definition Examples
STACK Defines a set of Services, where to obtain
the software packages and how to manage
the lifecycle.
HDP-2.3, HDP-2.2
SERVICE Defines the Components that make-up the
service.
HDFS, YARN
COMPONENT The building-blocks of a Service, that adhere
to a certain lifecycle.
NAMENODE, DATANODE,
OOZIE_SERVER
CATEGORY The category of Component. MASTER, SLAVE, CLIENT
REPO Repository metadata where the artifacts
reside
http://public-repo-
1.hortonworks.com/HDP/centos6/2
.x/GA/2.3.0.0

6. Ambari Stacks
• Stacks define Services + Repo
• What is a stack composed of and where to get the bits
• Each service has a definition
• What components are part of the Service
• Each service has defined lifecycle commands
• start, stop, status, install, configure
• Lifecycle is controlled via command scripts
Ambari Server
Stack
Service
Definitions
Command
Scripts
xml python
Ambari Agents
Repos

7. Ambari Stack Framework
• Ambari’s extensible stack framework allows different stack vendors to
create their own custom stacks.

8. CURRENT STATE OF
AFFAIRS

9. Current State
• Extensible Framework
• Provides ability for onboarding new Hadoop distributions
• Key role in avoiding vendor lock-in
• Stack driven
• Service lifecycle management
• Custom service commands
• UI layouts and themes
• Upgrade packs
• Stack advisors
• Metrics
• Kerberos
• Stack Inheritance
• Code reuse between stack versions
• Common Service Definitions
• Code reuse between multiple stacks
• ODPi Operations Spec

10. Limitations – Common Services
• AMBARI-7201 focused on adding notion of common services
• Service definitions scripts were not refactored
• Common service scripts still tightly coupled with HDP distribution
• Hardcoded references for stack name (HDP), stack version checks
(HDP-2.2+), stack tools (hdp-select, conf-select), installation paths
(/usr/hdp), method names (format_hdp_stack_version) etc.

11. Limitations – Custom Services
• Rudimentary approach to add third party custom services Ambari Wiki
• No release vehicle for building, releasing and deploying custom
services
• No service level extension points for stack features
• Role command order
• Stack advisor
• Upgrade packs
• Repositories
• Custom service definitions not self-contained and require hacks inside
stack definitions

12. Limitations – Release Management
• Ambari core and stack definitions released together
• Bug fix in stack definition requires Ambari release
• New stack version requires Ambari release
• Stack releases and Ambari releases have to be coordinated
• Need to decouple stack definition releases from Ambari core release

13. NEW EXTENSIBILITY
FEATURES

14. Stack Featurization
• Provides a basic framework for refactoring stack-specific hardcoded logic in
common service definitions.
• Features defined at stack-level based on which execution logic in service
definition is triggered.
• All common service definitions have been stack featurized and HDP-specific
hardcodings have been removed.
• Stacks similar to HDP can now use common service definitions.
• EPIC: AMBARI-13363
Apache JIRA Description Target Release
AMBARI-15420 Generalize resource management library 2.4.0.0
AMBARI-13364 Parameterize stack information used by common services 2.4.0.0
AMBARI-15329 Remove HDP hardcoding 2.4.0.0

15. Stack Featurization
New Config Properties
ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml
<property>
<name>stack_tools</name>
<value></value>
<description>Stack specific tools</description>
<property-type>VALUE_FROM_PROPERTY_FILE</property-type>
<value-attributes>
<property-file-name>stack_tools.json</property-file-name>
<property-file-type>json</property-file-type>
</value-attributes>
</property>
<property>
<name>stack_root</name>
<value>/usr/hdp</value>
<description>Stack root folder</description>
</property>
<property>
<name>stack_features</name>
<value></value>
<description>Stack specific tools</description>
<property-type>VALUE_FROM_PROPERTY_FILE</property-type>
<value-attributes>
<property-file-name>stack_features.json</property-file-name>
<property-file-type>json</property-file-type>
</value-attributes>
</property>

16. Stack Features
Without Stack Featurization:
if(Script.is_stack_greater_or_equal("2.2”) && Script. is_stack_less_than("2.5")) {
audit_jdbc_url = format('jdbc:mysql://{db_host}/{ranger_auditdb_name}'
}
With Stack Featurization:
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
{
"stack_features": [
{
"name": "ranger_audit_db_support",
"description": "Ranger Audit to DB support",
"min_version": "2.2.0.0",
"max_version": "2.5.0.0"
}
]
}
if(check_stack_feature(StackFeature.RANGER_AUDIT_DB_SUPPORT, stack_version_formatted)) {
audit_jdbc_url = format('jdbc:mysql://{db_host}/{ranger_auditdb_name}'
}

17. Stack Tools
Without Stack Featurization:
packages.append('hdp-select')
command = format('{sudo} /usr/bin/hdp-select set all `ambari-python-wrap /usr/bin/hdp-select versions |
grep ^{version_to_select} | tail -1`’) only_if_command = format('ls -d /usr/hdp/{version_to_select}*')
With Stack Featurization:
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json
{
"stack_selector": ["hdp-select", "/usr/bin/hdp-select", "hdp-select"],
"conf_selector": ["conf-select", "/usr/bin/conf-select", "conf-select"]
}
stack_selector_package = stack_tools.get_stack_tool_package(stack_tools.STACK_SELECTOR_NAME)
packages.append(stack_selector_package)
stack_root = Script.get_stack_root()
(stack_selector_name, stack_selector_path, stack_selector_package) =
stack_tools.get_stack_tool(stack_tools.STACK_SELECTOR_NAME)
command = format('{sudo} {stack_selector_path} set all `ambari-python-wrap {stack_selector_path}
versions | grep ^{version_to_select} | tail -1`’) only_if_command = format('ls -d
{stack_root}/{version_to_select}*’)

18. Service Level Extension Points
Apache JIRA Description Ambari Wiki Target Release
AMBARI- 15388 Upgrade Pack Extensions Ambari Wiki 2.4.0.0
AMBARI-15226 Stack Advisor Extensions Ambari Wiki 2.4.0.0
AMBARI-9363 Role command order Extensions Ambari Wiki 2.4.0.0
AMBARI-11268 Quick Links for custom services Ambari Wiki 2.4.0.0
AMBARI-15538 Service-specific repo definitions 2.X.X.X
• Facilitate integration of third-party custom services to a stack definition
• Self-contained service definitions without requiring changes to stack definition
• Upgrade Pack Extensions
• Service-level upgrade packs that specify how to upgrade the custom service
• Defines how to integrate into the stack upgrade pack
• Stack Advisor Extensions
• Service Advisors (service_advisor.py) define recommendations and validations for the service
• Stack framework extends stack advisors with service advisors
• Role Command Order
• Define role command order at service level
• Stack framework merges all role command orders to create dependencies
• Quick Links
• Define quick links (quicklinks.json) in the service definition that is used by Ambari Web UI
• No hardcoded quick links in the Ambari Web UI
• Repo Definitions
• Service specific repositories so that third party artifacts can reside on their own repositories
• Under development
EPIC: AMBARI-15537

19. Upgrade Pack Service Extension
Without Upgrade Pack Extensions:
- Upgrade logic defined at stack level
- Custom services need to modify the stack's upgrade-packs in order to integrate themselves into the cluster
With Upgrade Pack Extensions:
- Each service can define upgrade-packs XML files placed in the service's upgrades/ folder
ambari-server/src/test/resources/stacks/HDP/2.0.5/services/HDFS/upgrades/HDP/2.2.0/upgrade_test_15388.xml
<target>2.4.*</target>
<target-stack>HDP-2.4.0</target-stack>
<type>ROLLING</type>
<prerequisite-checks>
<check>org.apache.ambari.server.checks.FooCheck</check>
</prerequisite-checks>
<order>
<group xsi:type="cluster" name="PRE_CLUSTER" title="Pre {{direction.text.proper}}">
<add-after-group-entry>HDFS</add-after-group-entry>
<execute-stage service="FOO" component="BAR" title="Backup FOO">
<task xsi:type="manual">
<message>Back FOO up.</message>
</task>
</execute-stage>
</group>
….

20. Service Advisor
Without Service Advisor:
- Stack advisor defined at stack level
- Custom services need to modify stack advisor files in order to recommend/validate dynamically service
configurations and the layout of the service on cluster
With Service Advisor:
- Each service can choose to define its own service advisor and explicitly extend parent's service-advisor script
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py
def getServiceComponentLayoutValidations(self, services, hosts):
componentsListList = [service["components"] for service in services["services"]]
componentsList = [item["StackServiceComponents"] for sublist in componentsListList for item in sublist]
hawqMasterHosts = self.getHosts(componentsList, "HAWQMASTER")
hawqStandbyHosts = self.getHosts(componentsList, "HAWQSTANDBY")
hawqSegmentHosts = self.getHosts(componentsList, "HAWQSEGMENT")
datanodeHosts = self.getHosts(componentsList, "DATANODE")
# Generate WARNING if any HAWQSEGMENT is not colocated with a DATANODE
mismatchHosts = sorted(set(hawqSegmentHosts).symmetric_difference(set(datanodeHosts)))
if len(mismatchHosts) > 0:
hostsString = ', '.join(mismatchHosts)
message = "HAWQ Segment must be installed on all DataNodes. "

21. Quick Links
Without Quick Links Extensions:
- Hardcoded quick links in the Ambari Web UI
With Quick Links Extensions :
- A service can add a list of quick links to the Ambari web UI by adding predefined JSON format file to metainfo.xml
ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/metainfo.xml
ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/quicklinks/quicklinks.json
<quickLinksConfigurations>
<quickLinksConfiguration>
<fileName>quicklinks.json</fileName>
<default>true</default>
</quickLinksConfiguration>
</quickLinksConfigurations>
mapQuickLinks: function (finalJson, item){
if(!(item && item.ServiceInfo)) return;
var quickLinks = {
OOZIE: [19],
GANGLIA: [20],
STORM: [31],
FALCON: [32],
RANGER: [33],
SPARK: [34],
MY_CUSTOM_SERVICE: [35]
};
{
"name": "default",
"description": "default quick links configuration",
"configuration": {
"protocol":
{
"type":"http"
},
"links": [ {
"name": "hbase_master_ui",
"label": "HBase Master UI",
….. }
}, {
"name": "hbase_logs",
"label": "HBase Logs",
…

22. Stack Extensions
• A stack extension is a collection of custom services which are packaged
together.
• Provides a REST API for installing extensions.
• Extensions are staged at /var/lib/ambari-server/resources/extensions
• After installing extensions requires explicit linking of extensions with the
supported stack versions.
• Custom services contained in the extension may be added to the cluster like
any other service in the stack.
• EPIC: AMBARI-12885

23. Stack Extensions
Structure
|_ extensions
|_ <extension_name>
|_ <extension_version>
metainfo.xml
|_ services
|_ <service_name>
metainfo.xml
metrics.json
|_ configuration
{configuration files}
|_ package
{files, scripts, templates}
Stack Applicability
<metainfo>
<prerequisites>
<min-stack-versions>
<stack>
<name>HDP</name>
<version>2.4</version>
</stack>
<stack>
<name>OTHER</name>
<version>1.0</version>
</stack>
</min-stack-versions>
</prerequisites>
</metainfo>

24. Ambari Management Packs
Motivation

25. Ambari Management Packs
• Release artifact to bundle stack definitions, service definitions, add-on service
definitions, views etc.
• Decouple stack definition releases from Ambari core release.
• Also provide a release vehicle for add-on services.
• Released as tarballs but contains metadata that describes applicability and
contents of the management pack
• Staging Location: /var/lib/ambari-server/resources/mpacks
• Final stack definition can be an overlay of multiple management packs.
• EPIC: AMBARI-14854
• Ambari Wiki
• Target Release: 2.4.0.0

26. Overlay of Management Packs

27. Stack Management Pack
{
"type" : "full-release",
"name" : ”stack1-ambari-mpack",
"version": “1.0.0.0",
"description" : ”Stack1 Ambari Management Pack",
"prerequisites": {
"min-ambari-version" : ”2.4.0.0",
"max-ambari-version" : “"
},
"artifacts": [
{
"name" : ”stack1-service-definitions",
"type" : "service-definitions",
"source_dir": "common-services"
},
{
"name" : ”stack1-stack-definitions",
"type" : "stack-definitions",
"source_dir": "stacks"
}
]
}
hdp-ambari-mpack-1.0.0.0
├── mpack.json
├── common-services
│ └── NEWSERVICE
│ └── 1.0.0
│ ├── configuration
└── stacks
└── STACK1
└── 2.0
├── metainfo.xml
├── repos
│ └── repoinfo.xml
├── role_command_order.json
└── services
│ ├── KAFKA
│ │ ├── configuration
│ │ │ ├── ranger-kafka-audit.xml
│ │ │ └── ranger-kafka-policymgr-ssl.xml
│ │ └── metainfo.xml
│ ├── NEWSERVICE
│ │ └── metainfo.xml
│ └── ZOOKEEPER
│ └── metainfo.xml
└── widgets.json

28. Addon Service Management Pack
{
"type": "full-release",
"name": "myservice-ambari-mpack",
"version": "1.0.0.0",
"description": "MyService Ambari Mpack" ,
"prerequisites": {
"min-ambari-version": "2.4.0.0",
"min-stack-versions": [
{
"stack_name": "STACK1",
"stack_version": ”2.1"
}
]
},
"artifacts": [
{
"name": "myservice-common-services",
"type" : "service-definitions",
"source_dir" : "common-services"
},
{
"name" : "myservice-addon-services",
"type" : "stack-addon-service-definitions",
"source_dir": "stack-addon-services",
"service_versions_map": [
{
"service_name" : "MyService",
"service_version" : ”0.8.0.0",
"applicable_stacks" : [
{
"stack_name" : "STACK1",
"stack_version" : "2.1"
}
]
}
]
}
]
}

29. Future Goals
• Service level repos
• Management Pack++
• Short Term Goals (Ambari 2.4.0.0)
o Release vehicle for stacks
o HDP management pack, IOP management pack
o Release vehicle for add-on services (custom services)
o Microsoft-R management pack
o Retrofit in existing stack processing infrastructure
o Command line to update stack and service definitions
• Long Term Goals (Ambari 2.4+)
o Release HDP stacks as mpacks
o Build management pack processing infrastructure
o Dynamic creation of stack definitions by processing mpacks
o Rest API for adding/removing mpacks
• UI wizards stack driven

30. Q&A