Author: Shubin Zhang, et al.
Institute of Computing Technology, Beijing, China
Reported by: Tzu-Li Tai
National Cheng Kung...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
A. Background and Motivation
...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
Background and Motivation
Pre...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
Background and Motivation
• S...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
Goals and Design Decisions
• ...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
Goals and Design Decisions
• ...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
Goals and Design Decisions
• ...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
System Overview
• Use Memcach...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
System Overview
• Map side
! ...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
System Overview
[Extra] from ...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
System Overview
• Reduce side
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
System Overview
[Extra] from ...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
System Details
1. Memory Cach...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
System Details
1. Memory Cach...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
System Details
2. Network Tra...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
System Details
2. Network Tra...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
System Overview
• Map side
! ...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
System Details
3. Fault Toler...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
System Details
3. Fault Toler...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
System Details
3. Fault Toler...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
Experimental Results and Anal...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
Experimental Results and Anal...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
Experimental Results and Anal...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
Experimental Results and Anal...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
Experimental Results and Anal...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
Experimental Results and Anal...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
Experimental Results and Anal...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
Experimental Results and Anal...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
Conclusion and Future Works
•...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
Future Studies for Topic
• Da...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
Discussion: Our Chances
1.Nec...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
Discussion: Our Chances
1.Nec...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
Discussion: Our Chances
2. Mo...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
Discussion: Our Chances
2. Mo...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
Discussion: Our Chances
2. Mo...
HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU
Discussion: Our Chances
3. It...
Upcoming SlideShare
Loading in …5
×

Accelerating MapReduce with Distributed Memory Cache

2,348 views

Published on

[Paper Study Report]
Presentation Date: 2013/08/11
Original IEEE Xplore link:
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=&arnumber=5395321&queryText%3DAccelerating+MapReduce+with+Distributed

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,348
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
54
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Accelerating MapReduce with Distributed Memory Cache

  1. 1. Author: Shubin Zhang, et al. Institute of Computing Technology, Beijing, China Reported by: Tzu-Li Tai National Cheng Kung University, Taiwan High Performance Parallel and Distributed Systems Lab 2009 IEEE 15th International Conference on Parallel and Distributed Systems
  2. 2. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU A. Background and Motivation B. Goals and Design Decisions C. System Overview D. System Details E. Experimental Results and Analysis F. Conclusion and Future Works G. Future Studies for Topic E. Discussion: Our Chances
  3. 3. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU Background and Motivation Pre-Notes: - Published in 2009 (1st paper on topic) - Outdated hardware/software and data size - Focus on methodology and reasoning of using distributed cache in Hadoop - Learn possible tackle points and what to avoid
  4. 4. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU Background and Motivation • Shuffle time becomes the bottleneck M Inter. Data M Inter. Data M Inter. Data R HDFS pipeline replication HDFS HDFS HDFS 1 local write 2 remote read GOAL
  5. 5. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU Goals and Design Decisions • Target clusters are small-scale - Bandwidth is not scarce - Node failures are uncommon - Commodity machines - Heterogeneous - GB Ethernet
  6. 6. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU Goals and Design Decisions • Stay close to the original • Retain fault-tolerance (!) • Local decision-making
  7. 7. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU Goals and Design Decisions • Low-latency, high-throughput access to map outputs: global storage system - No central coordinator - Uniform global namespace - Low-latency, high-throughput data access - Concurrent access - Large capacity - Scalable ⇒ 𝑫𝒊𝒔𝒕𝒓𝒊𝒃𝒖𝒕𝒆𝒅 𝑴𝒆𝒎𝒐𝒓𝒚 𝑪𝒂𝒄𝒉𝒆
  8. 8. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU System Overview • Use Memcached: http://memcached.org/ - Open-source distributed memory caching system - Daemon processes on servers - Global K-V store
  9. 9. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU System Overview • Map side ! Buffer details
  10. 10. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU System Overview [Extra] from O’Reilly
  11. 11. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU System Overview • Reduce side
  12. 12. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU System Overview [Extra] from O’Reilly
  13. 13. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU System Details 1. Memory Cache Capacity M M M M M M M M M M M R R
  14. 14. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU System Details 1. Memory Cache Capacity 𝑆𝑖𝑧𝑒 𝑚𝑒𝑚𝑐𝑎𝑐ℎ𝑒𝑑 = 𝑚 𝑐 × 𝑠 × (𝑟 − 𝑟𝑎) 𝒎 𝒄: completed map tasks 𝒔: avg. map output size 𝒓: total no. of reduce tasks 𝒓 𝒂: no. of early scheduled reduce tasks 𝑆𝑖𝑧𝑒 𝑚𝑒𝑚𝑐𝑎𝑐ℎ𝑒𝑑𝑀𝑖𝑛 = 𝑚 × 𝑠 × (𝑟 − 𝑟𝑎) 𝒎: total no. of map tasks
  15. 15. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU System Details 2. Network Traffic Demand 𝑆ℎ𝑢𝑓𝑓𝑙𝑒𝑑 𝐷𝑎𝑡𝑎 = 2 ∗ 𝑆𝑖𝑧𝑒 𝑚𝑒𝑚𝑐𝑎𝑐ℎ𝑒𝑑 M R
  16. 16. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU System Details 2. Network Traffic Demand M R - Double amount of data shuffled (!) - A compression algorithm is used on map outputs to lessen network traffic
  17. 17. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU System Overview • Map side ! Hashing function?
  18. 18. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU System Details 3. Fault Tolerance • Map task failure - rerun outputs not yet in memcache/disk • Reduce task failure (!) - For inputs that are not yet deleted from memcache, copy and execute - For inputs that are already deleted from memcache, rerun the map task • Memcached Server failure (!) - Reinitialize all related map tasks • Tasktracker failure - All currently running map tasks and reduce tasks needs to be reinitialized - Memcache data is still valid, so reduce tasks can still access them
  19. 19. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU System Details 3. Fault Tolerance Reduce task failure M R
  20. 20. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU System Details 3. Fault Tolerance Memcached Server Failure M M M
  21. 21. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU Experimental Results and Analysis Environment • Hardware - Intel Pentium 4, 2.8GHz processor - 2GB RAM - 80GB 7200RPM SATA disk • Software - RedHat AS4.4, kernel 2.6.9 OS - Hadoop 0.19.1 - Memcached 1.2.8 - Memcached client for Java 2.5.1
  22. 22. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU Experimental Results and Analysis Hadoop+Memcached Setup 1 node NameNode + JobTracker + Memcached Server (1GB RAM) 1~6 nodes DataNode + TaskTracker • 2 map slots + 2 reduce slots per TaskTracker • 4MB HDFS file block • 5 shuffle threads in reduce tasks
  23. 23. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU Experimental Results and Analysis Benchmark Applications • Wordcount - 491.4 MB English text • Spatial Join Algorithm - 2 data sets from TIGER/Line files
  24. 24. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU Experimental Results and Analysis 1. Impact of different node numbers • No. of reduce tasks: 2*n • Wordcount improvement: 43.1% • Spatial Join improvement: 32.9%
  25. 25. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU Experimental Results and Analysis 2. Impact on job progress *Note: Hadoop job progress calculation - For Map tasks: % of input processed - For Reduce tasks: 1/3 (copy) + 1/3 (sort) + 1/3 (actual processing)
  26. 26. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU Experimental Results and Analysis 2. Impact on job progress - WordCount !
  27. 27. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU Experimental Results and Analysis 2. Impact on job progress – Spatial Join !
  28. 28. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU Experimental Results and Analysis 2. Impact on job progress - Extra 33% 66% sort copy reduce
  29. 29. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU Conclusion and Future Works • Enhanced Hadoop to accelerate data shuffling by using distributed memory cache (memcached) • Prototype performs much better than original Hadoop under moderate load. • Will modify task scheduling algorithm (earlier reduce tasks)
  30. 30. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU Future Studies for Topic • Dache: A Data Aware Caching for Big Data Applications Using The MapReduce Framework, 2013 IEEE INFOCOM • A Distributed Cache for Hadoop Distributed File System in Real-Time Cloud Services, 2012 ACM/IEEE GRID
  31. 31. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU Discussion: Our Chances 1.Necessity of using Memcached?
  32. 32. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU Discussion: Our Chances 1.Necessity of using Memcached? Properties for map task buffer: io.sort.mb: buffer size io.sort.spill.percent: spill-to-disk threshold Hypothesis: • Achieve map intermediate output local cache • Modify reduce shuffle threads + TaskTracker • RDD for Fault Tolerance?
  33. 33. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU Discussion: Our Chances 2. Moving the idea to YARN
  34. 34. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU Discussion: Our Chances 2. Moving the idea to YARN NodeManager NodeManager NodeManager MR Application Manager MAP Task REDUCE Task “Shuffle and Sort” NodeManager Auxiliary Service
  35. 35. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU Discussion: Our Chances 2. Moving the idea to YARN yarn-site.xml • Entire shuffle and sort phase is implemented as a pluggable aux. service in YARN
  36. 36. HPDS Lab, Institute of Computer and Communication Engineering, Electrical Engineering - NCKU Discussion: Our Chances 3. Iterative applications NodeManager MR Application Manager NodeManager MR Application Manager “Result caching + reuse” NodeManager Auxiliary Service NM M NM R NM M NM R

×