Adaptive MapReduce using Situation-Aware                     Mappers                    Rares Vernica1 (HP Labs),    Andre...
Outline1   Motivation2   Problem Statement3   Situation-Aware Mappers       Adaptive Mappers       Adaptive Combiners     ...
MapReduce Reviewmap    (k,v)       → list(k,v);reduce (k,list(v)) → list(k,v).                 Input:             Output: ...
MapReduce Reviewmap    (k,v)       → list(k,v);reduce (k,list(v)) → list(k,v).                 Input:             Output: ...
Motivation: MapReduce IssuesMapReduce     Parallel data-processing framework     Open-source implementation (Hadoop)     S...
Solutions to MapReduce IssuesMapReduce-inspired alternatives     Dryad (Microsoft)     Spark (UC Berkeley)     Hyracks (UC...
Our Solution: Adaptive MapReduceMake MapReduce (Hadoop) more flexible    Leverage existing investment in:           Framewo...
Problem Statement: Adaptive MapReduceGoalsImprove MapReduce (Hadoop) performance by:    New runtime options    Adaptive to...
Outline1   Motivation2   Problem Statement3   Situation-Aware Mappers       Adaptive Mappers       Adaptive Combiners     ...
Situation-Aware MappersMain idea    Make MapReduce more dynamic  Rares Vernica (HP Labs)   Adaptive MapReduce   EDBT 2012 ...
Situation-Aware MappersMain idea    Make MapReduce more dynamic    Mappers:           Aware of the global state of the job...
Situation-Aware MappersMain idea    Make MapReduce more dynamic    Mappers:           Aware of the global state of the job...
Situation-Aware MappersMain idea    Make MapReduce more dynamic    Mappers:           Aware of the global state of the job...
Situation-Aware MappersMain idea    Make MapReduce more dynamic    Mappers:           Aware of the global state of the job...
Adaptive MapReduceDFS                                                         DFS              MAP              MAP       ...
Adaptive MapReduce                             Distributed Meta-Data Store                                    Distributed ...
Adaptive MapReduce                     DMDSDFS                                                                   DFS      ...
Adaptive Mappers Motivation    Input data is divided into splits    One-to-one correspondence of mappers and splits    AM ...
Adaptive Mappers Motivation    Input data is divided into splits    One-to-one correspondence of mappers and splits    AM ...
Adaptive Mappers Algorithm        MapReduce Client ZooKeeper         1Root  JobID    locations      Host1       [Split1,  ...
Adaptive Mappers Algorithm        MapReduce Client                                  Host1 ZooKeeper                  2    ...
Adaptive Mappers Algorithm        MapReduce Client                                      Host1 ZooKeeper                   ...
Adaptive Mappers Algorithm        MapReduce Client                                  Host1 ZooKeeper                  2    ...
Adaptive Mappers Algorithm        MapReduce Client                                   Host1 ZooKeeper                  2   ...
Adaptive Mappers AlgorithmAdditional Features    Process local splits first, then remote splits    Fault tolerance         ...
Experimental SettingHardware    40-node IBM Systemx iDataPlex dx340    Two quad-core Intel Xeon E5540 64-bit 2.83GHz    32...
Start-up Cost vs. ZooKeeper Overhead                  300          Regular Mappers                  280          Adaptive ...
Adaptive Mappers Workloads 1     Set-Similarity Join [Vernica et al., 2010]              Publication datasets             ...
Adaptive Mappers Experiments - Set-Similarity Join                  1000                              Regular Mappers     ...
Adaptive Mappers Experiments - JOIN                                           Regular Mappers                             ...
Adaptive MapReduce                     DMDSDFS                                                           DFS              ...
Adaptive CombinersMain idea    Replace sort with hashing    Reduce serialization, sort, and IO                            ...
Adaptive CombinersMain idea    Replace sort with hashing    Reduce serialization, sort, and IO                            ...
Adaptive CombinersMain idea    Replace sort with hashing    Reduce serialization, sort, and IO                            ...
Adaptive CombinersMain idea    Replace sort with hashing    Reduce serialization, sort, and IO                            ...
Adaptive CombinersMain idea    Replace sort with hashing    Reduce serialization, sort, and IO                            ...
Adaptive CombinersMain idea    Replace sort with hashing    Reduce serialization, sort, and IO                            ...
Adaptive Combiners Details    “Best-effort” aggregation    Never spill to disk    Hash-table replacement policies:        ...
Adaptive Combiners ExperimentsGROUP-BY    Synthetic dataset with 3 dimensions (A1, A2, and A3) and 1 fact    Group records...
Adaptive MapReduce                     DMDSDFS                                                           DFS              ...
Adaptive Sampling and Partitioning                            MAP                                                 REDUCE  ...
Adaptive Sampling and Partitioning                                                   DMDSStep 1 Compute and publish       ...
Adaptive Sampling and Partitioning                                                   DMDSStep 1 Compute and publish       ...
Adaptive Sampling and Partitioning                                                     DMDSStep 1 Compute and publish     ...
Summary    Adaptive runtime techniques for MapReduce    Situation-Aware Mappers    Make MapReduce more dynamic    Up to ×3...
Vernica, R., Carey, M., and Li, C. (2010). Efficient parallel set-similarity joins using MapReduce. In SIGMOD Conference.Ra...
Upcoming SlideShare
Loading in …5
×

Adaptive MapReduce using Situation-Aware Mappers

1,552 views

Published on

We propose new adaptive runtime techniques for MapReduce that improve performance and simplify job tuning. We implement these techniques by breaking a key assumption of MapReduce that mappers run in isolation. Instead, our mappers communicate through a distributed meta-data store and are aware of the global state of the job. However, we still preserve the fault-tolerance, scalability, and programming API of MapReduce. We utilize these situation-aware mappers to develop a set of techniques that make MapReduce more dynamic: (a) Adaptive Mappers dynamically take multiple data partitions (splits) to amortize mapper start-up costs; (b) Adaptive Combiners improve local aggregation by maintaining a cache of partial aggregates for the frequent keys; (c) Adaptive Sampling and Partitioning sample the mapper outputs and use the obtained statistics to produce balanced partitions for the reducers. Our experimental evaluation shows that adaptive techniques provide up to 3x performance improvement, in some cases, and dramatically improve performance stability across the board.

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,552
On SlideShare
0
From Embeds
0
Number of Embeds
634
Actions
Shares
0
Downloads
28
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Adaptive MapReduce using Situation-Aware Mappers

  1. 1. Adaptive MapReduce using Situation-Aware Mappers Rares Vernica1 (HP Labs), Andrey Balmin, Kevin S. Beyer, Vuk Ercegovac (IBM Research) 1 Work done at IBM Research.15th International Conference on Extending Database Technology, March 26-30 2012 Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 1 / 25
  2. 2. Outline1 Motivation2 Problem Statement3 Situation-Aware Mappers Adaptive Mappers Adaptive Combiners Adaptive Sampling and Partitioning4 Summary Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 2 / 25
  3. 3. MapReduce Reviewmap (k,v) → list(k,v);reduce (k,list(v)) → list(k,v). Input: Output: (k,v) list(k,v) Input: Output: DFS MAP (k, list(v)) list(k,v) DFS INPUT 1/3 REDUCE OUTPUT 1/2 INPUT 2/3 MAP OUTPUT 2/2 INPUT 3/3 REDUCE MAP MERGE SHUFFLEcombine (k,list(v)) → list(k,v). Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 3 / 25
  4. 4. MapReduce Reviewmap (k,v) → list(k,v);reduce (k,list(v)) → list(k,v). Input: Output: (k,v) list(k,v) Input: Output: DFS MAP (k, list(v)) list(k,v) DFS INPUT 1/3 REDUCE OUTPUT 1/2 INPUT 2/3 MAP OUTPUT 2/2 INPUT 3/3 REDUCE MAP MERGE SHUFFLEcombine (k,list(v)) → list(k,v). Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 3 / 25
  5. 5. Motivation: MapReduce IssuesMapReduce Parallel data-processing framework Open-source implementation (Hadoop) Simple programming environmentMapReduce: “simplicity over performance”Limited choice of execution strategies: Mappers checkpoint after every split Map outputs are sorted and written to file Reducer read statically predetermined partitions Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 4 / 25
  6. 6. Solutions to MapReduce IssuesMapReduce-inspired alternatives Dryad (Microsoft) Spark (UC Berkeley) Hyracks (UC Irvine) Nephele (TU Berlin)Have more choices in runtime execution Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 5 / 25
  7. 7. Our Solution: Adaptive MapReduceMake MapReduce (Hadoop) more flexible Leverage existing investment in: Framework (Hadoop) Query processing systems (Jaql, Pig, Hive) Techniques for: Dynamic checkpoint intervals (Map) Best-effort hash-based aggregation (Combine) Dynamic, sample-based, partitioning (Reduce) Performance tuning: Cardinality and cost estimation (due to UDFs) Adaptive to runtime environment Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 6 / 25
  8. 8. Problem Statement: Adaptive MapReduceGoalsImprove MapReduce (Hadoop) performance by: New runtime options Adaptive to runtime environmentPreserve Hadoop’s Fault-tolerance Scalability Programability Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 7 / 25
  9. 9. Outline1 Motivation2 Problem Statement3 Situation-Aware Mappers Adaptive Mappers Adaptive Combiners Adaptive Sampling and Partitioning4 Summary Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 8 / 25
  10. 10. Situation-Aware MappersMain idea Make MapReduce more dynamic Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 9 / 25
  11. 11. Situation-Aware MappersMain idea Make MapReduce more dynamic Mappers: Aware of the global state of the job Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 9 / 25
  12. 12. Situation-Aware MappersMain idea Make MapReduce more dynamic Mappers: Aware of the global state of the job Communicate through a distributed meta-data store Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 9 / 25
  13. 13. Situation-Aware MappersMain idea Make MapReduce more dynamic Mappers: Aware of the global state of the job Communicate through a distributed meta-data store Break assumption: isolation Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 9 / 25
  14. 14. Situation-Aware MappersMain idea Make MapReduce more dynamic Mappers: Aware of the global state of the job Communicate through a distributed meta-data store Break assumption: isolation Situation-Aware Mappers Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 9 / 25
  15. 15. Adaptive MapReduceDFS DFS MAP MAP REDUCE MAP REDUCE Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 10 / 25
  16. 16. Adaptive MapReduce Distributed Meta-Data Store Distributed read/write Transactional DMDS e.g., ZooKeeperDFS DFS MAP MAP REDUCE MAP REDUCE Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 10 / 25
  17. 17. Adaptive MapReduce DMDSDFS DFS MAP AM AC AP AS MAP REDUCE REDUCE MAPAdaptive Techniques AM: Adaptive Mappers AC: Adaptive Combiners AS: Adaptive Sampling AP: Adaptive Partitioning Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 10 / 25
  18. 18. Adaptive Mappers Motivation Input data is divided into splits One-to-one correspondence of mappers and splits AM decouple # splits from # mappers Large splits Small startup cost Inbalanced workload Small splits Large startup cost Balanced workload : Startup cost, e.g., scheduling, loading ref. data , : Split processing cost Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 11 / 25
  19. 19. Adaptive Mappers Motivation Input data is divided into splits One-to-one correspondence of mappers and splits AM decouple # splits from # mappers Large splits Small startup cost Inbalanced workload Small splits Large startup cost Balanced workload Adaptive Mappers Small startup cost Balanced workload : Startup cost, e.g., scheduling, loading ref. data , : Split processing cost Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 11 / 25
  20. 20. Adaptive Mappers Algorithm MapReduce Client ZooKeeper 1Root JobID locations Host1 [Split1, Split2, ... ] Host2 ... Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 12 / 25
  21. 21. Adaptive Mappers Algorithm MapReduce Client Host1 ZooKeeper 2 Map1 1Root Init JobID locations Host1 Map2 [Split1, Init Split2, ... ] Host2 ... ... Host2 ... ... Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 12 / 25
  22. 22. Adaptive Mappers Algorithm MapReduce Client Host1 ZooKeeper 2 Map1 1Root Init JobID locations Host1 Map2 [Split1, Init Split2, 3 ... ] Host2 ... ... Host2 ... ... Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 12 / 25
  23. 23. Adaptive Mappers Algorithm MapReduce Client Host1 ZooKeeper 2 Map1 1Root Init JobID locations Host1 Map2 [Split1, Init 3 Split1 Split2, ... ] Host2 ... ... Host2 assigned 4 ... Split1{Map2} ... Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 12 / 25
  24. 24. Adaptive Mappers Algorithm MapReduce Client Host1 ZooKeeper 2 Map1 1Root Init JobID locations Store meta-data in Host1 Map2 ZooKeeper [Split1, Init Implemented as a new 3 Split1 Split2, 5 InputFormat ... ] Host2 ... ... Host2 assigned 4 ... Split1{Map2} OK/Fail ... Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 12 / 25
  25. 25. Adaptive Mappers AlgorithmAdditional Features Process local splits first, then remote splits Fault tolerance Restated task unlocks splits Split reprocessing is shared Scheduler aware (FIFO, FAIR, and FLEX) Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 13 / 25
  26. 26. Experimental SettingHardware 40-node IBM Systemx iDataPlex dx340 Two quad-core Intel Xeon E5540 64-bit 2.83GHz 32GB RAM Four SATA disks 160 map and 160 reduce slotsSoftware Ubuntu Linux, kernel 2.6.32-24 64-bit server edition Java 1.6 64-bit server edition Hadoop 0.20.2 ZooKeeper 3.3.1 Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 14 / 25
  27. 27. Start-up Cost vs. ZooKeeper Overhead 300 Regular Mappers 280 Adaptive Mappers 2000 1-byte recordsTime (seconds) Sleep 1s/record 140 5 nodes, 20 map slots 120 20-2000 Reg. Mappers 100 20 Adaptive Mappers 80 60 Small ZooKeeper 40 overhead 20 0 Large Map startup 20 200 2000 cost ∼2s/map Number of Splits Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 15 / 25
  28. 28. Adaptive Mappers Workloads 1 Set-Similarity Join [Vernica et al., 2010] Publication datasets DBLP: 1.2M records, 310MB CITESEERX: 1.3M records, 1,750MB Increased to ×10 and ×100 2 JOIN Single dataset (“fact” table), Sort Benchmark data generator Fan-out coefficient (“dimension” table) average join fan-out 1 : 30 TERASORT: 1B records, 93GB Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 16 / 25
  29. 29. Adaptive Mappers Experiments - Set-Similarity Join 1000 Regular Mappers Stage 3: Adaptive Mappers 800 One-Phase Record Join Time (seconds) Broadcast join equivalent 600 DBLP and CITESEERX ×10 400 Single wave of AM 200 ×3 speedup over default 0 Hadoop split size (64MB) Optimal with no tuning 20 10 51 25 12 64 32 AM 48 24 2 6 8 Split Size (MB) Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 17 / 25
  30. 30. Adaptive Mappers Experiments - JOIN Regular Mappers Map-only job 1200 Adaptive Mappers 1B TERASORT recordsTime (seconds) 900 Models a skewed join Single wave of AM 600 Regular Mappers: 300 Large split: data skew Small split: scheduling 0 and start-up overhead Optimal with no tuning 10 51 25 12 64 32 16 8 AM 24 2 6 8 Split Size (MB) Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 18 / 25
  31. 31. Adaptive MapReduce DMDSDFS DFS MAP AM AC AP AS MAP REDUCE AM AC AP AS MAP REDUCE AM AC AP ASAdaptive Techniques AM: Adaptive Mappers AC: Adaptive Combiners AS: Adaptive Sampling AP: Adaptive Partitioning Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 19 / 25
  32. 32. Adaptive CombinersMain idea Replace sort with hashing Reduce serialization, sort, and IO Regular Combiners Sort Buffer Map : User code : Data Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 20 / 25
  33. 33. Adaptive CombinersMain idea Replace sort with hashing Reduce serialization, sort, and IO Regular Combiners Sort Buffer Map Sort Combine : User code : Data Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 20 / 25
  34. 34. Adaptive CombinersMain idea Replace sort with hashing Reduce serialization, sort, and IO Regular Combiners Sort Buffer Map Sort Combine : User code : Data Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 20 / 25
  35. 35. Adaptive CombinersMain idea Replace sort with hashing Reduce serialization, sort, and IO Regular Combiners Sort Buffer Map Sort Combine Merge : User code : Data Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 20 / 25
  36. 36. Adaptive CombinersMain idea Replace sort with hashing Reduce serialization, sort, and IO Regular Combiners Sort Buffer Map Sort Combine Merge : User code Adaptive Combiners : Data Hash-group and Combine Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 20 / 25
  37. 37. Adaptive CombinersMain idea Replace sort with hashing Reduce serialization, sort, and IO Regular Combiners Sort Buffer Map Sort Combine Merge : User code Adaptive Combiners : Data Hash-group and Combine Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 20 / 25
  38. 38. Adaptive Combiners Details “Best-effort” aggregation Never spill to disk Hash-table replacement policies: No-Replacement (NR) Least-Recently-Used (LRU) Implemented as: Library for Hadoop Optimization choice for Jaql Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 21 / 25
  39. 39. Adaptive Combiners ExperimentsGROUP-BY Synthetic dataset with 3 dimensions (A1, A2, and A3) and 1 fact Group records and apply aggregation function TWL: 10B records, 120GB 180 350 1.00 300 150 Time (seconds) 0.75 Miss Ratio (%) Time (seconds) 250 120 200 0.50 90 150 100 0.25 60 50 30 0 0.00 Re AM 1 25 10 0 0 g. Re AM AC AM Cache Size (K) g. ,A C Regular Combiners Adaptive Combiners NR Regular Combiners Adaptive Combiners LRU Adaptive Combiners NR Miss Ratio NR Adaptive Combiners LRU Miss Ratio LRU GROUP-BY on A1 GROUP-BY on A1 and A2 ×2.5 speedup ×3 speedup Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 22 / 25
  40. 40. Adaptive MapReduce DMDSDFS DFS MAP AM AC AP AS MAP REDUCE AM AC AP AS MAP REDUCE AM AC AP ASAdaptive Techniques AM: Adaptive Mappers AC: Adaptive Combiners AS: Adaptive Sampling AP: Adaptive Partitioning Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 23 / 25
  41. 41. Adaptive Sampling and Partitioning MAP REDUCE MAP REDUCE MAP Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 24 / 25
  42. 42. Adaptive Sampling and Partitioning DMDSStep 1 Compute and publish local histogram MAP REDUCE MAP REDUCE MAP Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 24 / 25
  43. 43. Adaptive Sampling and Partitioning DMDSStep 1 Compute and publish local histogram MAPStep 2 Collect local histograms and REDUCE compute partitioning function MAP REDUCE MAP Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 24 / 25
  44. 44. Adaptive Sampling and Partitioning DMDSStep 1 Compute and publish local histogram MAPStep 2 Collect local histograms and REDUCE compute partitioning function MAPStep 3 Broadcast partitioning function REDUCE MAP Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 24 / 25
  45. 45. Summary Adaptive runtime techniques for MapReduce Situation-Aware Mappers Make MapReduce more dynamic Up to ×3 speedup for well-tuned jobs Orders of magnitude speedup for badly tuned jobs Never hurt performance Configure themselves Part of IBM InfoSphere BigInsights Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 25 / 25
  46. 46. Vernica, R., Carey, M., and Li, C. (2010). Efficient parallel set-similarity joins using MapReduce. In SIGMOD Conference.Rares Vernica (HP Labs) Adaptive MapReduce EDBT 2012 25 / 25

×