Download as PDF, PPTX
MSR 2009
- 1. MapReduce as a General Framework to Support Research in Mining So8ware Repositories(MSR) Weiyi Shang, Zhen Ming Jiang, Bram Adams, Ahmed Hassan So8ware Analysis and Intelligence Lab(SAIL) School of CompuCng, Queen’s University
- 2. As an MSR researcher, have you ever been in such a situa>on? • Analyzing gigabytes of data? • WaiCng hours for experimental results? • Experiments fail with “out of memory” excepCons?
- 3. To overcome these problems, you could … … buy more powerful machines … spend weeks to make your tools more efficient
- 4. However! • The data will keep on growing • Spend Cme on research not on speeding up experiments Debian doubles in size approximately every two years
- 5. • Idle compuCng power is available in every lab • We can bundle these computers together • A distributed framework can help us do so
- 6. General requirements for a distributed framework: 1. Efficiency speed up the process significantly 2. Scalability scale with data size and compuCng power 3. Adaptability require only minimal programming effort 4. Flexibility run in various environments
- 7. Google’s MapReduce is an idea of distributed computa8on
- 8. Google’s MapReduce is an idea of distributed computa8on • Open-‐source MapReduce implementaCon • Well documented and many examples available • Well supported by large user base and news groups • Straight forward API
- 9. Example: coun>ng the frequency of word lengths dog cat fish good hello night happy school # WordsLength 23 24 35 16
- 10. dog cat fish hello good night happy school Example: coun>ng the frequency of word length 1. Deploy data into a distributed file system data network compuCng environment
- 11. Example: coun>ng the frequency of word length 2. Read data as records Data dog cat fish hello good night happy school
- 12. Example: coun>ng the frequency of word length 3. Generate keys of each record by Mappers Data dog cat fish hello good night happy school ValueKey dog3 cat3 fish4 hello5 good4 night5 happy5 school6 Mapper Mapper Mapper Mapperdog3 cat3 fish4 hello5 good4 night5 happy5 school6
- 13. Example: coun>ng the frequency of word length 4. Group and sort records by keys ValueKey dog3 cat3 fish4 hello5 good4 night5 happy5 school6 ValueKey dog3 cat3 fish4 good4 hello5 night5 happy5 school6
- 14. Example: coun>ng the frequency of word length 5. Send records with the same key to one reducer ValueKey dog3 cat3 fish4 good4 hello5 night5 happy5 school6 Reducer Reducer Reducer dog3 cat3 Reducer fish4 good4 hello5 night5 happy5 school6
- 15. Example: coun>ng the frequency of word length 6. Generate outputs by Reducers ValueKey dog3 cat3 fish4 good4 hello5 night5 happy5 school6 Reducer Reducer Reducer dog3 cat3 Reducer fish4 good4 hello5 night5 happy5 school6 ValueKey 23 24 35 16
- 16. A typical MSR analysis Extract all versions of all files Analyze each version Compare versions to each other We implement MapReduce on a typical MSR tool Repository
- 17. Applying MapReduce to typical MSR tools Repository Data a0.java a1.java b0.java a2.java b1.java
- 18. Applying MapReduce to typical MSR tools Mapper Mapper Mapper Data a0.java a1.java b0.java a2.java b1.java ValueKey a.java a.java b.java a.java b.java a0.java a1.java b0.java a2.java b1.java a.java a.java a0.java a1.java b.java a.java b0.java a2.java b.java b1.java
- 19. Applying MapReduce to typical MSR tools ValueKey a.java a.java b.java a.java b.java a0.java a1.java b0.java a2.java b1.java ValueKey a.java a.java a.java b.java b.java a0.java a1.java a2.java b0.java b1.java
- 20. Applying MapReduce to typical MSR tools Reducer Reducer ValueKey a.java a.java a.java b.java b.java a0.java a1.java a2.java b0.java b1.java a.java a.java a.java a0.java a1.java a2.java b.java b.java b0.java b1.java
- 21. Applying MapReduce to typical MSR tools Reducer Reducer ValueKey a.java a.java a.java b.java b.java a0.java a1.java a2.java b0.java b1.java a.java a.java a.java a0.java a1.java a2.java b.java b.java b0.java b1.java ValueKey a.outputa.java b.outputb.java
- 22. Case study： J-‐REX Extract snapshots from CVS repository Use Eclipse JDT to parse source code to XML files Compare each XML file to generate evoluCon informaCon XML output n … JDT EvoluCon Analyzer EvoluConary Change Data … Snapshot n XML output 1 Snapshot extractor CVS ExtracCon phase Parsing phase Analysis phase Snapshot 1
- 23. Case study: data Repository Size #Source Code Files Length of History #Revisions Datatools 394MB 10,552 2 years 2,398 BIRT 810MB 13,002 4 years 19,583 Eclipse 4.2GB 56,851 8 years 82,682
- 24. Case study: experimental setup CPU type #CPU Memory size Disk type Desktop Intel Quad Core Q6600 @ 2.40 GHz 4 2GB SATA Server Intel Quad Core Q6600 @ 2.40 GHz 4 8GB RAID5 Server Intel Core i7 920 @ 2.67 GHz 8 6GB SSD
- 25. Efficiency: significant reduc>on of running >me by using MapReduce Desktop Server(SSD) With MapReduce 70% less 64% less Running >me (hour) faste r 59% less
- 26. Scalability: dras>c reduc>on of run >me by adding machines • When adding machines – Time to deploy data increases – Time to process decreases 2nodes 3nodes 4nodes faster
- 27. Adaptability: liale effort to apply MapReduce to MSR tool • J-‐REX logic unchanged • Only 300-‐400 LOC to implement Map and Reduce • Typical MapReduce examples available • Less than one hour for deployment
- 28. Flexibility: run on various environments
- 29. Conclusions • Distributed frameworks are needed to – deal with growing data – make best use of available compuCng resources • A MapReduce soluCon of a typical MSR analysis is: – straight forward – scalable – efficient