• Save
YARN: Hadoop Beyond MapReduce - Andreas Neumann
Upcoming SlideShare
Loading in...5

Like this? Share it with your network


YARN: Hadoop Beyond MapReduce - Andreas Neumann



When Apache Hadoop was first introduced to the Open Source world, it was focused on implementing Google's Map/Reduce, a framework for batch processing of very large files in a distributed system. ...

When Apache Hadoop was first introduced to the Open Source world, it was focused on implementing Google's Map/Reduce, a framework for batch processing of very large files in a distributed system. Hadoop implemented this framework along with a built-in cluster resource manager specialized in the execution of Map/Reduce programs. The popularity of Hadoop has since led to the (re-)emergence of various other "Big Data" processing paradigms, such as real-time stream processing (Storm, S4), message-passing (MPI), and graph processing (Giraph). However, because Hadoop was specialized in Map/Reduce, it has been hard to leverage existing Hadoop infrastructure for these other paradigms. This changes with the release of Apache Hadoop 2.0, the next-generation Map/Reduce engine: Its new resource manager, YARN, not only improves availability, scalability, security and multi-tenancy, it also decouples the cluster management from the Map/Reduce engine. YARN manages the cluster's compute resources as "compute slots". Through its application master API, each application can obtain slots from YARN and is free to use them for any type of computation. YARN manages and schedules available slots in the cluster, monitors running slots, and notifies the application if one of its slots has died. The application master decides how many slots to allocate and what tasks run in those slots. With this architecture, YARN supports running arbitrary compute paradigms that can all share the resources of one cluster. This allows for innovation, agility and better hardware utilization. This talk will give a detailed introduction of YARN's concepts, architecture and interfaces. We will illustrate the use of YARN's APIs and show examples of YARN applications. We will share the experience and lessons learned from building a real-time streaming engine on top of YARN and running it at scale in production, and we will present best practices and design patterns around YARN.



Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds


Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

YARN: Hadoop Beyond MapReduce - Andreas Neumann Presentation Transcript

  • 1. YARN: HADOOPBEYOND MAPREDUCEAndreas Neumann @anew68TerenceYim @chtyim
  • 2. Andreas Neumann, Terence YimIN THIS TALK, YOU WILL• Learn about Hadoop• Learn about other distributed compute paradigms• Learn about YARN and how it works• Learn how to get more value out of your Hadoop cluster• Learn about Weave
  • 3. Andreas Neumann, Terence YimA WEB-APPDatabasePresentation(stateless)Business Logic(stateless)
  • 4. Andreas Neumann, Terence YimA SCALABLE WEB-APPDatabasePresentation(stateless)Business Logic(stateless)
  • 5. Andreas Neumann, Terence YimA LOGGING WEB-APPPresentation(stateless)Business Logic(stateless) log log logDatabase
  • 6. Andreas Neumann, Terence YimWHAT TO DO WITH MY LOGS?Analyzing web logs can create real business value• Optimize for observed user behavior• Personalize each user’s experience• ...Many web logs are simply being discarded• Too expensive?• Too Hard to analyze?• Too much heavy lifting?
  • 7. Andreas Neumann, Terence YimWHAT TO DO WITH MY LOGS?
  • 8. Andreas Neumann, Terence YimAPACHE• Open source and free• Runs on commodity hardware - cheap to store• Programming Paradigm: Map/Reduce (Google 2004)• Batch processing of very large files in a distributed file system• Simple yet powerful• Turn your web app into a data-driven app
  • 9. Andreas Neumann, Terence YimA MAP/REDUCE APPReducersMappers(shuffle)split split splitpart part part
  • 10. Andreas Neumann, Terence YimMAP/REDUCE CLUSTERsplit split splitpart part partsplit split splitpart part partsplit splitpart partsplit splitpart part
  • 11. Andreas Neumann, Terence YimI HAVE HADOOP - WHAT NEXT?What if:• Map/Reduce is not the only thing I want to run?• I would like to do real-time analysis?• Or run a message-passing algorithm over my data?• My cluster has more compute capacity than my big data needs?Can I utilize its idle resources for other, compute-intensive tasks?
  • 12. Andreas Neumann, Terence YimA MESSAGE PASSING (MPI) APPdatadata datadata datadata
  • 13. Andreas Neumann, Terence YimA STREAMING APPDatabaseevents
  • 14. Andreas Neumann, Terence YimA DISTRIBUTED LOAD TESTtest testtesttest test testtest test testtesttesttestWebService
  • 15. Andreas Neumann, Terence YimHADOOP AT CONTINUUITYContinuuity AppFabric• Developer centric big data app platform• Runs many different types of jobs in Hadoop cluster• Real-time stream processing• Ad-hoc OLAP queries• Map/Reduce
  • 16. Andreas Neumann, Terence YimA MULTI-PURPOSE CLUSTERlog log logDatabasesplit spli splipar par pardatadata datadata datadatatest testtesttest test testtest test testtesttesttestWebServiceDatabaseevents
  • 17. Andreas Neumann, Terence YimTHE ANSWER: YARN• The Resource Manager of Hadoop 2.0• Separates resource management from programming paradigm• Allows (almost) all kinds of distributed application in Hadoop cluster• Multi-tenant, secure• Scales to many thousands of nodes• Pluggable scheduling algorithms
  • 18. Andreas Neumann, Terence YimYARN APPLICATION MASTERlog log logDatabasesplit spli splipar par pardatadata datadata datadatatest testtesttest test testtest test testtesttesttestWebServiceDatabaseeventsAM AM AMAMAMYARNResourceManager
  • 19. Andreas Neumann, Terence YimYARN - HOW IT WORKS• Every node (machine) in the cluster has several slots.• A node manager runs on each node to manages the node’s slots.• The resource manager assigns the slots of the cluster toapplications.• Applications must have a master.• The master requests slots from the resource manager and startstasks.
  • 20. Andreas Neumann, Terence YimYARN - HOW IT WORKSProtocols1) Client-RM: Submit the app master2) RM-NM: Start the app master3) AM-RM: Request + release containers4) RM-NM: Start tasks in containersYARNResourceManagerNode MgrAMNode MgrNode MgrNode MgrTask TaskTaskTaskTaskTaskYARNClient1)2)3)4)
  • 21. Andreas Neumann, Terence YimBACKGROUND: HADOOP+HDFSLocalfile systemLocalfile system. . .NameNodeHDFS distributed file systemDataNode• Every node contributes part of itslocal file system to HDFS.• Tasks can only depend on thelocal file system(JVM class path does notunderstand HDFS protocol)DataNode
  • 22. Andreas Neumann, Terence YimBACKGROUND: HADOOP+HDFSYARNResourceManager AMYARNClientLocalfile systemLocalfile system. . .NameNodeHDFS distributed file systemLocalfile systemAM.jarNodeMgrNodeMgrDataNodeDataNodeX
  • 23. Andreas Neumann, Terence YimBACKGROUND: HADOOP+HDFSYARNResourceManager AMYARNClientLocalfile system. . .NameNodeHDFS distributed file systemLocalfile systemAM.jar1) copyto HDFSAM.jar2) copyto localAM.jar3)NodeMgrNodeMgrDataNodeDataNode
  • 24. Andreas Neumann, Terence YimWRITING THE YARN CLIENT1) Connect to the Resource Manager.2) Request a new application ID.3) Create a submission context and a container launch context.4) Define the local resources for the AM.5) Define the environment for the AM.6) Define the command to run for the AM.7) Define the resource limits for the AM.8) Submit the request to start the app master.
  • 25. Andreas Neumann, Terence YimWRITING THE YARN CLIENTConnect to the Resource Manager:YarnConfiguration yarnConf = new YarnConfiguration(conf);InetSocketAddress rmAddress =NetUtils.createSocketAddr(yarnConf.get(YarnConfiguration.RM_ADDRESS,YarnConfiguration.DEFAULT_RM_ADDRESS));LOG.info("Connecting to ResourceManager at " + rmAddress);configuration rmServerConf = new Configuration(conf);rmServerConf.setClass(YarnConfiguration.YARN_SECURITY_INFO,ClientRMSecurityInfo.class, SecurityInfo.class);ClientRMProtocol resourceManager = ((ClientRMProtocol) rpc.getProxy(ClientRMProtocol.class, rmAddress, appsManagerServerConf));
  • 26. Andreas Neumann, Terence YimWRITING THE YARN CLIENTRequest an application ID, create a submission context and a launchcontext:GetNewApplicationRequest request =Records.newRecord(GetNewApplicationRequest.class);GetNewApplicationResponse response =resourceManager.getNewApplication(request);LOG.info("Got new ApplicationId=" + response.getApplicationId());ApplicationSubmissionContext appContext =Records.newRecord(ApplicationSubmissionContext.class);appContext.setApplicationId(appId);appContext.setApplicationName(appName);ContainerLaunchContext amContainer =
  • 27. Andreas Neumann, Terence YimWRITING THE YARN CLIENTDefine the local resources:Map<String, LocalResource> localResources = Maps.newHashMap();// assume the AM jar is here:Path jarPath; // <- known path to jar file// Create a resource with location, time stamp and file lengthLocalResource amJarRsrc = Records.newRecord(LocalResource.class);amJarRsrc.setType(LocalResourceType.FILE);amJarRsrc.setResource(ConverterUtils.getYarnUrlFromPath(jarPath));FileStatus jarStatus = fs.getFileStatus(jarPath);amJarRsrc.setTimestamp(jarStatus.getModificationTime());amJarRsrc.setSize(jarStatus.getLen());localResources.put("AppMaster.jar", amJarRsrc);amContainer.setLocalResources(localResources);
  • 28. Andreas Neumann, Terence YimWRITING THE YARN CLIENTDefine the environment:// Set up the environment needed for the launch contextMap<String, String> env = new HashMap<String, String>();// Setup the classpath needed.// Assuming our classes are available as local resources in the// working directory, we need to append "." to the path.String classPathEnv = "$CLASSPATH:./*:";env.put("CLASSPATH", classPathEnv);// setup more environmentenv.put(...);amContainer.setEnvironment(env);
  • 29. Andreas Neumann, Terence YimWRITING THE YARN CLIENTDefine the command to run for the AM:// Construct the command to be executed on the launched containerString command ="${JAVA_HOME}" + /bin/java" +" MyAppMaster" +" arg1 arg2 arg3" +" 1>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stdout" +" 2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stderr";List<String> commands = new ArrayList<String>();commands.add(command);// Set the commands into the container specamContainer.setCommands(commands);
  • 30. Andreas Neumann, Terence YimWRITING THE YARN CLIENTDefine the resource limits for the AM:// Define the resource requirements for the container.// For now, YARN only supports memory constraints.// If the process takes more memory, it is killed by the framework.Resource capability = Records.newRecord(Resource.class);capability.setMemory(amMemory);amContainer.setResource(capability);// Set the container launch content into the submission contextappContext.setAMContainerSpec(amContainer);
  • 31. Andreas Neumann, Terence YimWRITING THE YARN CLIENTSubmit the request to start the app master:// Create the request to send to the Resource ManagerSubmitApplicationRequest appRequest =Records.newRecord(SubmitApplicationRequest.class);appRequest.setApplicationSubmissionContext(appContext);// Submit the application to the ApplicationsManagerresourceManager.submitApplication(appRequest);
  • 32. Andreas Neumann, Terence YimYARN: APPLICATION MASTERYARNResourceManager AMYARNClientLocalfile systemLocalfile system. . .NameNodeHDFS distributed file systemNodeMgrNodeMgrDataNodeDataNode
  • 33. Andreas Neumann, Terence YimYARN - APPLICATION MASTER• Now we have an Application Master running in a container.• Next, the master starts the application’s tasks in containers• For each task, similar procedure as starting the master
  • 34. Andreas Neumann, Terence YimYARN: APPLICATION MASTERYARNResourceManager AMYARNClientLocalfile systemLocalfile system. . .NameNodeHDFS distributed file systemNodeMgrNodeMgrDataNodeDataNodeTaskTaskTask
  • 35. Andreas Neumann, Terence YimYARN - APPLICATION MASTER1) Connect with Resource Manager and register itself.2) Loop:• Send request to Resource Manager, to:a) Send heartbeatb) Request containersc) Release containers• Receive Response from Resource Managera) Receive containersb) Receive notification of containers that terminated• For each container received, send request to Node Manager to start task.
  • 36. Andreas Neumann, Terence YimYARN - APPLICATION MASTER• Master should terminate after all containers have terminated.• If master crashes (or fails to send heart beat), all containers are killed.• In the future, YARN will support restart of the master.• Starting a task in a container• Very similar to YARN client starting the master.• Node manager starts and monitors task.• If task crashes, Node Manager informs Resource Manager• Resource Manager informs master in next response• Master must still release the container.
  • 37. Andreas Neumann, Terence YimYARN - APPLICATION MASTER• Resource Manager assigns containers asynchronously:• Requested containers are returned at the earliest in the nextcall.• Master must send empty requests until all it receives allcontainers.• Subsequent requests are incremental and can ask for additionalcontainers.• Master must keep track of what it has requested and received.
  • 38. Andreas Neumann, Terence YimYARN - APPLICATION LOGSYARNResourceManager AMTaskTaskTaskYARNClientLocalfile system. . .NameNodeHDFS distributed file system2) copyto HDFSlogs4) log access through Name Nodelogs1) log4j,stdout,stderr3)queryRM forlogslocationNodeMgrNodeMgrDataNodeDataNode
  • 39. Andreas Neumann, Terence YimEXAMPLE - DISTRIBUTED SHELL• Example application in the YARN/Hadoop distribution• Runs a shell command in a number of containers• hadoop-2.0.0-alpha: 1681 lines of code• some simplification in 2.0.3-alpa: 1543 lines• Conclusion: YARN is complex!
  • 40. Andreas Neumann, Terence YimYARN IS NOT SIMPLE!
  • 41. Andreas Neumann, Terence YimYARN - SHORTCOMINGS• Complexity• Protocols are at very low level and very verbose.• Client must prepare all dependent jars on HDFS.• Client must set up environment, class path etc.• Logs are only collected after application terminates• What about long-running apps?• Applications don’t survive master crash: SPOF!• No built-in communication between containers and master.• Hard to debug
  • 42. Andreas Neumann, Terence YimTYPICAL YARN APP• Many distributed applications are simple.• For example: Distributed load test for a queuing system:• “Run n producers and m consumers for k seconds”• Neither producer nor consumer know that they run in YARN• This would be easy if we run them as threads in a single JVM• Can we make YARN as simple as Java Threads?
  • 43. Andreas Neumann, Terence YimTYPICAL NEEDS ...... of a distributed application:• Monitor and restart containers• Collect logs and metrics• Elastic scale (dynamic adding/removing of instances)• Long-running jobs
  • 44. Andreas Neumann, Terence YimWEAVE IT!
  • 45. Andreas Neumann, Terence YimHERE COMES WEAVE• Open-Source, Apache License• Does not replace YARN, but complements it• Generic App Master+ a DSL for specifying applications+ an API for writing container tasks+ built-in service discovery, logging, metrics, monitoring
  • 46. Andreas Neumann, Terence YimSIMPLICITY140. public class ApplicationMaster {141..150. private AMRMClientAsync resourceManager;.435. public boolean run() throws YarnRemoteException {436. LOG.info(“Starting ApplicationMaster”);.498. }.500. public void finish() {504. for(Thread launchThread: launchThreads) {. ...511. }517. FinalApplicationStatus appStatus;518. String appMessage = null;538. }.790. public ContainerRequest setupContainerAskForRM(int numContainers) {.800. Resource capability = Records.newRecord(Resource.class);801. capability.setMemory(containerMemory);.803. ContainerRequest request = new ContainerRequest(capability, ..);807. }808. }..1102. public class Client extends YarnClientImpl {.1336. public boolean run() throws IOException {.1354. GetNewApplicationResponse newApp = super.getNewApplication();1355. ApplicationId appId = newApp.getApplicationId();.1583. return monitorApplication(appId);1585. }.1594. private boolean monitorApplication(ApplicationId appId) throws ... {.1596. while(true) {.1606. ApplicationReport report = super.getApplicationReport(appId);.1647. }1648. }.1657. private void forceKillApplication(ApplicationId appId) throws ... {.1664. super.killApplication(appId);.1665. }.1667. }001. public class DistributedShell extends AbstractWeaveRunnable {002. static Logger LOG = LoggerFactory.getLogger(DistributedShell.class);003. private String command;004.005. public DistributedShell(String command) {006. this.command = command;007. }008.009. @Override010. public WeaveRunnableSpecification configure() {011. return WeaveRunnableSpecification.builder().with()012. .setName("Executor " + command)013. .withArguments(ImmutableMap.of("cmd", this.command))014. .build();015. }016.017. @Override018. public void initialize(WeaveContext context) {019. this.command = context.getSpecification().getArguments("cmd");020. }021.022. @Override023. public void run() {024. try {025. Process process = Runtime.getRuntime().exec(this.command);026. InputStream stdin = proc.getInputStream();027. BufferedReader br = new BufferedReader(new InputStreamReader(stdin));028.029. String line;030. while( (line = br.readline()) != null) {031. LOG.info(line);032. }033. } catch (Throwable t) {034. LOG.error(t.getMessage());035. }036. }037. }038.039. public static main(String[] args) {040. String zkStr = args[0];041. String command = args[1];042. yarnConfig = ...043. WeaveRunnerService weaveService = new YarnWeaveRunnerService(yarnConfig, zkStr);044. weaveService.startAndWait();045. WeaveController controller = weaveService.prepare(new DistributedShell(command))046. .addLogHandler(new PrinterLogHandler(new PrintWriter(System.out)))047. .start();048. }Distributed Shell - Vanilla YARN Distributed Shell - WEAVE~ 500 lines ~ 50 lines
  • 47. Andreas Neumann, Terence YimWEAVEYARNResourceManagerNode Mgr Node MgrNode MgrNode MgrTaskWeaveClient...AM+KafkaTaskTaskTaskTask TaskWeaveRunnableWeaveRunnerWeaveRunnablelogZooKeeperZooKeeperZooKeeperstate +messageslog streamThis is the onlyprogramminginterface you need
  • 48. Andreas Neumann, Terence YimRUNNABLEDefine a task:public class EchoServer implements WeaveRunnable{static Logger LOG = ...private WeaveContext context;//....@Overridepublic void run() {ServerSocket serverSocket = new ServerSocket(0);context.announce("echo", serverSocket.getLocalPort());while (isRunning()) {Socket socket = serverSocket.accept();//...}}}WeaveRunnable = single taskSLF4J can be used for logging- collected via Kafka- available to clientWeaveRunnable can be run in- Thread- YARN containerService endpoint announcement
  • 49. Andreas Neumann, Terence YimRUNNERStart a task:WeaveRunner runner = ...;WeaveController controller =runner.prepare(new EchoServer()).addLogHandler(new PrinterLogHandler(new PrintWriter(System.out))).start();//...controller.sendCommand(Commands.create("stat", "incoming"));//...controller.stop();WeaveRunner- packages and its dependencies- plus specified resources- attaches log collector to all containersWeaveController is used to managethe lifecycle of a runnable or applicationController also provides ability tosend commands to running containers
  • 50. Andreas Neumann, Terence YimAPPLICATIONDefine an application:public class WebServer implements WeaveApplication {@Overridepublic WeaveSpecification configure() {return WeaveSpecification.Builder.with().setName(“Jetty WebServer”).withRunnables().add(new JettyWebServer()).withLocalFiles().add(“www-html”, new File(“/var/html”)).apply().add(new LogsCollector()).anyOrder().build();}}WeaveController controller =runner.prepare(new WebServer()).start();WeaveApplication = collection of tasksApplication is given bya WeaveSpecificationApplication can specify additional localdirectories to be made available.Weave starts containers in no order.WeaveRunner can also startan application
  • 51. Andreas Neumann, Terence YimAPPLICATIONDefine an application with order:public class DataApplication implements WeaveApplication{@Overridepublic WeaveSpecification configure() {return WeaveSpecification.Builder.with().setName(“Cool Data Application”).withRunnables().add(“reader”, new InputReader()).add(“processor”, new Processor()).add(“writer”, new OutputWriter()).order().first(“reader”).next(“processor”).next(“writer”).build();}}starts containers in given order.
  • 52. Andreas Neumann, Terence YimSUMMARY• Web apps generate data.• Data driven apps with Hadoop.• More value out of Hadoop with YARN.• Easier with Weave.• Now every Java developer can write distributed apps.
  • 53. Andreas Neumann, Terence YimTHANK YOUhttps://github.com/continuuity/weavehttp://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.htmlYes, we are hiring:careers@continuuity.com