ApacheCon 2014 Apache Slider (in incubation)

1. Slider: Applications on YARN
Steve Loughran– Hortonworks
stevel at hortonworks.com
@steveloughran
ApacheCon EU, November 2014

2. Modern applications
log
front end
front end
web
phones
devices
feeds log
stream
processing
front end
log
database
analytics

3. Fun operational problems
Placement: where to run?
Installation
Configuration & Binding
Client configuration
Lifecycle
Failure handing and recovery
Logging
Upgrading
Metrics & Monitoring
– Start/Stop
– Reconfigure
– Scale up/down
– Rolling-restart
– Decommission/Recommission

4. Getting your code to work across a
set of fairly reliable machines

5. Apache Slider
Deploying and managing
applications on Apache Hadoop YARN
http://slider.incubator.apache.org/

6. Background: YARN
YARN Node Manager
HDFS
YARN Resource Manager
“The RM”
HDFS
YARN Node Manager
HDFS
YARN Node Manager
HDFS
• Servers run YARN Node Managers (NM)
• NM's heartbeat to Resource Manager (RM)
• RM schedules work over cluster
• RM allocates containers to apps
• NMs start containers
• NMs report container health

7. Client creates App Master
YARN Node Manager
HDFS
YARN Resource Manager
“The RM”
HDFS
YARN Node Manager
HDFS
YARN Node Manager
HDFS
Client
Slider Application Master

8. Slider AM requests containers
YARN Node Manager
HDFS
YARN Resource Manager
HDFS
YARN Node Manager
HDFS
YARN Node Manager
Slider Application Master
HDFS
HBase Master
HBase Region Server
HBase Region Server

9. Slider: architecture
1. AppMaster
2. AgentProvider
3. Agent
4. AppPackage
5. CLI
Slider
App
Package
YARN Resource Manager
“The RM”
HDFS
YARN Node Manager
Agent Component
HDFS
Slider CLI
YARN Node Manager
Slider Application Master
Agent Component
HDFS

10. Desired State, agents & scripts
Slider
App
Package
CLI
YARN Resource Manager
“The RM”
HDFS
YARN Node Manager
Agent Component
HDFS
YARN Node Manager
Agent Component
HDFS
1. CLI starts an instance of the AM
2. AM requests containers
3. Containers activate with an Agent
4. Agent gets application definition
5. Agent registers with Slider AM
6. AM issues commands
7. Agent reports back, status,
configuration, etc.
8. AM publishes endpoints,
configurations
Slider App Master
Registry

11. App Package: zip XML .py
© Hortonworks Inc. 2014:

12. App package Metainfo XML
Publish an URI
Page 12
<service>
<name>HBASE</name>
<version>0.96.0.2.1.1</version>
<exportGroups><exportGroup>
<name>QuickLinks</name>
<exports><export>
<name>org.apache.slider.jmx</name>
<value>http://${HBASE_MASTER_HOST}:${site.hbase-site.hbase.master.info.port}/jmx</value>
</export></exports></exportGroup></exportGroups>
<commandOrders><commandOrder>
<command>HBASE_REGIONSERVER-START</command>
<requires>HBASE_MASTER-STARTED</requires>
</commandOrder></commandOrders>
<components><component>
<name>HBASE_MASTER</name>
<minInstanceCount>1</minInstanceCount>
<commandScript>
<script>scripts/hbase_master.py</script>
</commandScript></component></components>
</service>
Application Info
Dependencies
Contains
components
scripts

13. deploy: resource.json
{
"schema": "http://example.org/specification/v2.0.0",
"global": {
"yarn.memory": "512"
},
"components": {
"HBASE_MASTER": {
"yarn.role.priority": "1",
"yarn.component.instances": "1",
"yarn.memory": "1024",
"yarn.vcores": "1",
},
"HBASE_REGIONSERVER": {
"yarn.role.priority": "2",
"yarn.component.instances": "1"
}
} }

14. configure: app_config.json
{
"application.def": "/slider/hbase_v096.zip",
"site.global.app_log_dir": "${AGENT_LOG_ROOT}/app/log",
"site.global.app_pid_dir": "${AGENT_WORK_ROOT}/app/run",
"site.global.hbase_master_heapsize": "1024m",
"site.global.ganglia_server_host": "${NN_HOST}",
"site.global.ganglia_server_port": "8667",
"site.global.ganglia_server_id": "Application1",
Variables for the
application scripts
"site.hbase-site.hbase.tmp.dir": "${AGENT_WORK_ROOT}/work/app/tmp",
"site.hbase-site.hbase.master.info.port": "${HBASE_MASTER.ALLOCATED_PORT}",
"site.hbase-site.hbase.regionserver.port": "0",
"site.hbase-site.hbase.zookeeper.quorum": "${ZK_HOST}",
"site.core-site.fs.defaultFS": "${NN_URI}",
}
Configurations needed by
Slider
Named variables
Site variables for
application
Allocate and
advertise
Named variables
for cluster details

15. create, start, stop, destroy
$ slider create hbase1 --resources resources.json --template appConfig.json
$ slider list
$ slider status hbase1
$ slider stop hbase1
$ slider start hbase1
$ slider stop hbase1--force
$ slider destroy hbase1

16. Dynamic application resize
slider flex hbase1
--component HBASE_REGIONSERVER 2

17. Registry lookup
$ slider resolve --path ~/services/org-apache-slider/storm1
{ "type" : "JSONServiceRecord",
"external" : [ {
"api" : "http://",
"addressType" : "uri",
"protocolType" : "webui",
"addresses" : [ {
"uri" : "http://nn.example.com:46132"
} ]
}, {
"api" : "classpath:org.apache.slider.publisher.configurations",
"addressType" : "uri",
"protocolType" : "REST",
"addresses" : [ {
"uri" : "http://nn.example.com:46132/ws/v1/slider/publisher/slider"
} ]
} } ] }

20. Failures
YARN Node Manager
HDFS
YARN Resource Manager
HDFS
YARN Node Manager
HDFS
YARN Node Manager
Slider App Master
HDFS
HBase Master
HBase Region Server
HBase Region Server

21. YARN notifies AM
YARN Node Manager
HDFS
YARN Resource Manager
HDFS
YARN Node Manager
HDFS
YARN Node Manager
Slider App Master
HDFS
HBase Master
HBase Region Server

22. request on old node (lax)
YARN Node Manager
HDFS
YARN Resource Manager
HDFS
YARN Node Manager
HDFS
YARN Node Manager
Slider App Master
HDFS
HBase Master
HBase Region Server
HBase Region Server

23. Summary
• XML + python defined app deployment
• Declarative YARN resource specification
• Declarative application configuration
• Slider AM keeps app in desired state
• Lifecycle commands
• Binding and configuration via registry

24. Slider View for Apache Ambari
http://ambari.apache.org/

25. Managed Slider
Slider
App
Package
Ambari
YARN Resource Manager
“The RM”
HDFS
YARN Node Manager
Agent Component
HDFS
Registry
YARN Node Manager
Slider App Master
Agent Component
HDFS
• Ambari imports app packages
• Starts the AM
• Interacts with AM to start
containers
• Agents register with Ambari
• Ambari sends commands/
receives results
• YARN manages containers
• Ambari interacts with Registry Registry

28. Project Status
• In ASF incubation
• 0.60-incubating release this week(!)
• Need more developers!
• Need more application packages!

29. If you want to run existing apps
in a YARN cluster:
Get involved!
http://slider.incubator.apache.org/