Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Deploying Docker applications on YARN via Slider

4,457 views

Published on

Presentation slides from the Apache Slider Meetup: Deploying Long-running services onto YARN Using Apache Slider

Published in: Software
  • Be the first to comment

Deploying Docker applications on YARN via Slider

  1. 1. © Hortonworks Inc. 2015 Deploying Docker applications on YARN via Slider HW Slider team Presenter: Thomas Liu Page 1
  2. 2. © Hortonworks Inc. 2015 Slider is an incubation project with one goal: Make it easy to deploy and manage applications in a Hadoop cluster with YARN Page 2
  3. 3. © Hortonworks Inc. Project Status • In ASF incubation • 0.70-incubating release this month • Growing set of application packages • Working in simplifying packaging • HDP 2.2 added Slider component for easy on-boarding of long-lived service on YARN http://slider.incubator.apache.org/
  4. 4. © Hortonworks Inc. An application consists of “Application Package” JSON configuration files: YARN resources + config Data persisted in HDFS (May) Placement history in HDFS Page 4
  5. 5. © Hortonworks Inc. 2015 Slider: architecture 1. AppMaster 2. AgentProvider 3. Agent 4. AppPackage 5. CLI Slider App config files HDFS YARN Resource Manager “The RM” HDFS YARN Node Manager Agent Component HDFS YARN Node Manager Agent Component Slider Application Master Slider CLI
  6. 6. © Hortonworks Inc. 2015 Slider App Master runs the application HDFS YARN Node Manager HDFS YARN Node Manager HDFS YARN Resource Manager HDFS YARN Node Manager Slider Application Master Redis Server Nodejs Server
  7. 7. © Hortonworks Inc. On-boarding Dockerized Application on YARN via Slider • Application definition – metainfo.json (contains docker image name) • Application instance configuration (if any) – app_config.json • Application resource definition – resources.json No Application bundle and lifecycle scripts Page 7
  8. 8. © Hortonworks Inc. 2015 Demo Dockerized node.js app using dockerized redis for persistence Page 8
  9. 9. © Hortonworks Inc. metainfo.json – structure of the application { "schemaVersion": "2.1", "application": { "name": "NODEJS-REDIS", "components": [ { "name": "NODEJS", "type": "docker", "dockerContainers": [ { "name": "nodejs”, "image": "rsahahw/centos-node-redis”, "ports": [{"containerPort" : "8000” }] } ] }, { "name": "REDIS", "type": "docker", "dockerContainers": [ { "name": "redis", "image": "tutum/redis", "ports": [{ "containerPort" : "6379", "hostPort": "6379” ...
  10. 10. © Hortonworks Inc resources.json – resource requirement of the application { "schema" : "http://example.org/specification/v2.0.0", "metadata" : { }, "global" : { }, "components": { "NODEJS": { "yarn.role.priority": "1", "yarn.component.instances": "1", "yarn.memory": "512" }, "REDIS": { "yarn.role.priority": "2", "yarn.component.instances": "1", "yarn.memory": "512" } } }
  11. 11. © Hortonworks Inc. app_config.json – instance specific configuration of the application { "schema": "http://example.org/specification/v2.0.0", "metadata": { }, "global": { }, "components": { "NODEJS": { "nodejs.options":"-d -e REDIS_HOST=${REDIS_HOST}", "nodejs.statusCommand":"/usr/bin/docker ps" }, "REDIS": { "redis.options":"-d -e REDIS_PASS=**None**", "redis.statusCommand":"/usr/bin/docker ps" } } }
  12. 12. © Hortonworks Inc Commands: create, start, stop, destroy $ slider create dockerapp1 --resources resources.json -–template appConfig.json -–metainfo metainfo.json $ slider list $ slider stop dockerapp1 $ slider destroy dockerapp1
  13. 13. © Hortonworks Inc. How to get started • Use –Pre-requisite – Hadoop 2.6 or above [ or HDP 2.2 ] –Build Slider [ link ] –Author slider configuration files for your dockerized app or use examples • Contribute –Explore Slider @ http://slider.incubator.apache.org/ –Few ideas – Aligning Docker security model with YARN’s – Docker images with multiple initialization steps – Active resource monitoring docker containers by YARN
  14. 14. © Hortonworks Inc. 2015 Q&A

×