Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Apache REEF - stdlib for big data

Slides from my talk at 2016 Apache: Big Data conference.

Resource managers like Apache YARN and Mesos have emerged as a critical layer in the cloud computing system stack, but the developer abstractions for leasing cluster resources and instantiating application logic are very low-level. We present Apache REEF, a powerful yet simple framework that helps developers of big data systems to retain fine-grained control over the cloud resources and address common problems of fault-tolerance, task scheduling and coordination, caching, interprocess communication, and bulk-data transfers. We will guide the developers through a simple REEF application and discuss current state of Apache REEF project and its place in the Hadoop ecosystem.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

Apache REEF - stdlib for big data

  1. 1. a library to simplify and unify the lower layers of big data systems on modern resource managers.
  2. 2. • One cluster used by all workloads (interactive, batch, streaming, …) • Resources are handed out as containers Container is slice of a machine Fixed RAM, CPU, I/O, … • Examples: Apache Hadoop YARN Apache Mesos Google Borg Resource Managers
  3. 3. Resource Managers Enable true multi-tenancy… Many workloads: Streaming, Batch, Interactive… Many users: Production, Ad-Hoc, Experiments… …but, only for sophisticated apps
  4. 4. Resource Managers Enable true multi-tenancy… Many workloads: Streaming, Batch, Interactive… Many users: Production, Ad-Hoc, Experiments… …but, only for sophisticated apps Fault tolerance
  5. 5. Resource Managers Enable true multi-tenancy… Many workloads: Streaming, Batch, Interactive… Many users: Production, Ad-Hoc, Experiments… …but, only for sophisticated apps Fault tolerance
  6. 6. Resource Managers Enable true multi-tenancy… Many workloads: Streaming, Batch, Interactive… Many users: Production, Ad-Hoc, Experiments… …but, only for sophisticated apps Fault tolerance Preemption
  7. 7. Resource Managers Enable true multi-tenancy… Many workloads: Streaming, Batch, Interactive… Many users: Production, Ad-Hoc, Experiments… …but, only for sophisticated apps Fault tolerance Preemption Elasticity
  8. 8. Example 1: SQL / MapReduce Elasticity Fault tolerance σ π ⋈
  9. 9. Example 1: SQL / MapReduce Elasticity Fault tolerance σ π ⋈ ⋈⋈ ⋈⋈
  10. 10. Example 1: SQL / MapReduce Elasticity Fault tolerance π ⋈ ⋈⋈ ⋈⋈
  11. 11. Example 2: Machine learning Fault tolerance Elasticity Iterative computations
  12. 12. Example 3: Graph processing Fault tolerance Elasticity Iterative computations Low latency communication
  13. 13. Silos Silos are hard to build Each duplicates the same mechanisms under the hood In practice, silos form pipelines • In each step: Read from and write to HDFS • Synchronize on complete data between steps  Slow! σ π ⋈
  14. 14. Fun new problems, boring old ones Control flow setup Master / Slave Membership Via heartbeats Task Submission Inter-Task Messaging … σ π ⋈
  15. 15. REEF Breadth Mechanism over Policy Avoid silos Recognize the need for different models But: allow them to be composed RM portability Make app independent from low- level Resource Manager APIs. Bridge the JVM/CLR/… divide Different parts of the computation can be in either of them. Resource Manager and DFS := Cluster OS REEF := stdlib
  16. 16. REEF Control Flow Yarn ( ) handles resource management (security, quotas, priorities) Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc… + 3
  17. 17. REEF Control Flow Yarn ( ) handles resource management (security, quotas, priorities) Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc. + 3
  18. 18. REEF Control Flow Yarn ( ) handles resource management (security, quotas, priorities) Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc… REEF Evaluators ( ) hold hardware resources, allowing multiple Tasks ( , , , , , , etc…) to use the same cached state. πσ + 3
  19. 19. REEF Control Flow Yarn ( ) handles resource management (security, quotas, priorities) Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc… REEF Evaluators ( ) hold hardware resources, allowing multiple Tasks ( , , , , , , etc…) to use the same cached state. πσ
  20. 20. REEF Control Flow Yarn ( ) handles resource management (security, quotas, priorities) Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc… REEF Evaluators ( ) hold hardware resources, allowing multiple Tasks ( , , , , , , etc…) to use the same cached state. πσ
  21. 21. REEF Control Flow Yarn ( ) handles resource management (security, quotas, priorities) Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc… REEF Evaluators ( ) hold hardware resources, allowing multiple Tasks ( , , , , , , etc…) to use the same cached state. πσ
  22. 22. REEF Control Flow Yarn ( ) handles resource management (security, quotas, priorities) Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc… REEF Evaluators ( ) hold hardware resources, allowing multiple Tasks ( , , , , , , etc…) to use the same cached state. πσ σ σ σ
  23. 23. REEF Control Flow Yarn ( ) handles resource management (security, quotas, priorities) Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc… REEF Evaluators ( ) hold hardware resources, allowing multiple Tasks ( , , , , , , etc…) to use the same cached state. πσ
  24. 24. Hello World 1. Client submits the Driver allocation request to the Resource Manager ( ) $…
  25. 25. Hello World 1. Client submits the Driver allocation request to the Resource Manager ( ) 2. Once started, Driver ( ) requests one Evaluator container from YARN $…
  26. 26. Hello World 1. Client submits the Driver allocation request to the Resource Manager ( ) 2. Once started, Driver ( ) requests one Evaluator container from YARN 3. Driver submits HelloWorld Task to the newly allocated Evaluator $…
  27. 27. Hello World 1. Client submits the Driver allocation request to the Resource Manager ( ) 2. Once started, Driver ( ) requests one Evaluator container from YARN 3. Driver submits a HelloWorld Task to the newly allocated Evaluator 4. HelloWorld Task prints a greeting to the log and quits $… LOG.log(Level.INFO, "Hello REEF!");
  28. 28. Hello World 1. Client submits the Driver allocation request to the Resource Manager ( ) 2. Once started, Driver ( ) requests one Evaluator container from YARN 3. Driver submits a HelloWorld Task to the newly allocated Evaluator 4. HelloWorld Task prints a greeting to the log and quits 5. Driver receives CompletedTask notification, releases the Evaluator and quits $…
  29. 29. public final class HelloTask implements Task { @Inject private HelloTask() {} @Override public byte[] call(final byte[] memento) { LOG.log(Level.INFO, "Hello REEF!"); return null; } }
  30. 30. public final class StartHandler implements EventHandler<StartTime> { @Override public void onNext(final StartTime startTime) { requestor.submit(EvaluatorRequest.newBuilder() .setNumber(1).setMemory(64).setNumberOfCores(1).build()); } } public final class EvaluatorAllocatedHandler implements EventHandler<AllocatedEvaluator> { @Override public void onNext(final AllocatedEvaluator allocatedEvaluator) { allocatedEvaluator.submitTask(TaskConfiguration.CONF .set(TaskConfiguration.IDENTIFIER, "HelloREEFTask") .set(TaskConfiguration.TASK, HelloTask.class) .build(); } }
  31. 31. final Configuration runtimeConfig = YarnRuntimeConfiguration.CONF.build(); final Configuration driverConfig = DriverConfiguration.CONF .set(DriverConfiguration.DRIVER_IDENTIFIER, "HelloREEF") .set(DriverConfiguration.GLOBAL_LIBRARIES, EnvironmentUtils.getClassLocation(HelloDriver.class)) .set(DriverConfiguration.ON_DRIVER_STARTED, HelloDriver.StartHandler.class) .set(DriverConfiguration.ON_EVALUATOR_ALLOCATED, HelloDriver.EvaluatorAllocatedHandler.class) .build(); DriverLauncher .getLauncher(runtimeConfig) .run(driverConfig, JOB_TIMEOUT);
  32. 32. Tang Command = ‘ls’ Error: container-487236457-02.stderr: NullPointerException at: java…eval():1234 ShellTask.helper():546 ShellTask.onNext():789 YarnEvaluator.onNext():12 Configuration is hard - Errors often show up at runtime only - State of receiving process is unknown to the configuring process
  33. 33. Tang Command = ‘ls’ Error: Unknown parameter "Command" Missing required parameter "cmd" Configuration is hard - Errors often show up at runtime only - State of receiving process is unknown to the configuring process Our approach: - Use Dependency Injection - Configuration is pure data  Early static and dynamic checks
  34. 34. Tang cmd = ‘ls’ ShellTask Evaluator Error: Required instanceof Evaluator Got ShellTask Configuration is hard - Errors often show up at runtime only - State of receiving process is unknown to the configuring process Our approach: - Use Dependency Injection - Configuration is pure data  Early static and dynamic checks
  35. 35. Tang cmd = ‘ls’ ShellTask Task YarnEvaluator Evaluator Configuration is hard - Errors often show up at runtime only - State of receiving process is unknown to the configuring process Our approach: - Use Dependency Injection - Configuration is pure data  Early static and dynamic checks
  36. 36. Wake: Events + I/O Event-based programming and remoting API: A static subset of Rx - Static checking of event flows - Aggressive JVM event inlining Implementation: “SEDA++” - Global thread pool - Thread sharing where possible
  37. 37. Distributed Shell 1. Client submits Driver configuration to YARN runtime. Configuration has cmd shell command, and n number of Evaluators to run it on $…
  38. 38. Distributed Shell 1. Client submits Driver configuration to YARN runtime. Configuration has cmd shell command, and n number of Evaluators to run it on 2. Once started, Driver requests n Evaluators from YARN $…
  39. 39. Distributed Shell 1. Client submits Driver configuration to YARN runtime. Configuration has cmd shell command, and n number of Evaluators to run it on 2. Once started, Driver requests n Evaluators from YARN 3. Driver submits ShellTask with cmd to each Evaluator #! $… #! #!
  40. 40. Distributed Shell 1. Client submits Driver configuration to YARN runtime. Configuration has cmd shell command, and n number of Evaluators to run it on 2. Once started, Driver requests n Evaluators from YARN 3. Driver submits ShellTask with cmd to each Evaluator 4. Each ShellTask runs the command, logs its stdout, and quits #! $… #! #!
  41. 41. Distributed Shell 1. Client submits Driver configuration to YARN runtime. Configuration has cmd shell command, and n number of Evaluators to run it on 2. Once started, Driver requests n Evaluators from YARN 3. Driver submits ShellTask with cmd to each Evaluator 4. Each ShellTask runs the command, logs its stdout, and quits 5. Driver receives CompletedTask notifications, releases Evaluators and quits when all Evaluators are gone $…
  42. 42. @Inject private ShellTask(@Parameter(Command.class) final String command) { this.command = command; } @Override public byte[] call(final byte[] memento) { final String result = CommandUtils.runCommand(this.command); LOG.log(Level.INFO, result); return CODEC.encode(result); }
  43. 43. @NamedParameter(doc="Number of evaluators", short_name="n", default_value="1") public final class NumEvaluators implements Name<Integer> {} @NamedParameter(doc="The shell command", short_name="cmd") public final class Command implements Name<String> {}
  44. 44. public final class EvaluatorAllocatedHandler implements EventHandler<AllocatedEvaluator> { @Override public void onNext(final AllocatedEvaluator allocatedEvaluator) { final JavaConfigurationBuilder taskConfigBuilder = tang.newConfigurationBuilder(TaskConfiguration.CONF .set(TaskConfiguration.IDENTIFIER, "ShellTask") .set(TaskConfiguration.TASK, ShellTask.class) .build()); taskConfigBuilder.bindNamedParameter(Command.class, command); allocatedEvaluator.submitTask(taskConfigBuilder.build()); } }
  45. 45. final JavaConfigurationBuilder driverConfig = tang.newConfigurationBuilder(DriverConfiguration.CONF .set(DriverConfiguration.DRIVER_IDENTIFIER, "DistributedShell") .set(DriverConfiguration.GLOBAL_LIBRARIES, EnvironmentUtils.getClassLocation(ShellDriver.class)) .set(DriverConfiguration.ON_DRIVER_STARTED, ShellDriver.StartHandler.class) .set(DriverConfiguration.ON_EVALUATOR_ALLOCATED, ShellDriver.EvaluatorAllocatedHandler.class) .build()); new CommandLine(driverConfigBuilder) .registerShortNameOfClass(Command.class) .registerShortNameOfClass(NumEvaluators.class) .processCommandLine(args); DriverLauncher.getLauncher(runtimeConfig).run(driverConfig.build(), JOB_TIMEOUT);
  46. 46. YARN Example: 1333 lines of code
  47. 47. YARN Example: 1333 lines of code Simple REEF Application: 83 lines 94%
  48. 48. YARN Example: 1333 lines of code Simple REEF Application: 83 lines Interactive Fault-Tolerant Web Application on REEF: 361 lines 27%
  49. 49. • • • • •
  50. 50. Group Communication Also: Collective Communication Communications with many participants Contrast: Peer-to-peer communication Most commonly used interface: MPI
  51. 51. Broadcast Mechanism The sender (S) sends a value All receivers (R) receive the identical value Use case Distribute a model Distribute a descent direction for line search Optimizations Trees to distribute the data Be mindful of the topology of machines Do peer-to-peer sends … S
  52. 52. Reduce Mechanism The senders (S) each send a value Those values are aggregated The receiver (R) receives the aggregate total Use case Aggregate gradients, losses Optimizations Aggregation trees Pipelining
  53. 53. // We can send and receive any Java serializable data, e.g. jBLAS matrices private final Broadcast.Sender<DoubleMatrix> modelSender; private final Broadcast.Receiver<DoubleMatrix[]> resultReceiver; // Broadcast the model, collect the results, repeat. do { this.modelSender.send(modelSlice); // ... final DoubleMatrix[] result = this.resultReceiver.reduce(); } while (notConverged(modelSlice, prevModelSlice));
  54. 54. http://reef.apache.org
  55. 55. http://github.com/cmssnu/dolphin http://github.com/Microsoft-CISL/TensorFactorization-LDA
  56. 56. Contact dev@reef.apache.org Website http://reef.apache.org
  57. 57. Start with a random 𝑤0 Until convergence: Compute the gradient 𝜕 𝑤 = ෍ 𝑥,𝑦 𝜖 𝑋 𝑤, 𝑥 − 𝑦 Apply gradient and regularizer to the model 𝑤𝑡+1 = 𝑤𝑡 − 𝜂 𝜕 𝑤 + 𝜆 Data parallel in X  Reduce Needed by Partitions  Broadcast
  58. 58. On REEF Driver requests Evaluators
  59. 59. On REEF Driver requests Evaluators Driver sends Tasks to load & parse data
  60. 60. On REEF Driver requests Evaluators Driver sends Tasks to load & parse data Driver sends ComputeGradient and master Tasks
  61. 61. On REEF Driver requests Evaluators Driver sends Tasks to load & parse data Driver sends ComputeGradient and master Tasks Computation commences in sequence of Broadcast and Reduce
  62. 62.
  63. 63. Start with a random 𝑤0 Until convergence: Compute the gradient 𝜕 𝑤 = ෍ 𝑥,𝑦 𝜖 𝑋 2 𝑤, 𝑥 − 𝑦 Apply gradient to the model 𝑤𝑡+1 = 𝑤𝑡 − 𝜕 𝑤 Not having some machines means training on a (random) subset of X
  64. 64. On REEF First iteration
  65. 65. On REEF Second Iteration
  66. 66. On REEF End state

×