SlideShare a Scribd company logo
1 of 10
The MSR Cookbook
Mining a Decade of Research
Hadi Hemmati, Sarah Nadi, Olga Baysal, Oleksii Kononenko,
Wei Wang, Reid Holmes, Michael W. Godfrey
David R. Cheriton School of Computer Science
University of Waterloo, Canada
MSR-2013, May 19, 2013
Why Do We Need a Cookbook?
MSR
Cookbook
2
Idea Came from MSR Vision 2020
3
Methodology
Review Open coding
MSR 2004–2012 (9 yrs)
270 papers
117 full papers
Comments:
Generalizable observations
or suggestions supported
by evidence in the paper
268 comments
4 themes with
16 recommendations
Recommendation:
Comments supported
by evidence from at least
5 papers
4
Themes
5
Top 5 Recommendations
Validate your assumptions and
heuristics; repos are noisy
[22 c, 17 p]
Watch out for collinearities and
skewness when synthesizing
models from data
[22 c, 17 p]
[20 c,16 p]
Manually verify outputs
[12 c, 10 p]
Sometimes other measures
work better than precision/
recall
Sometimes practicality
trumps statistical rigour
[16 c, 15 p]
6
Theme/Recommendation Trends
• MSR research is maturing as a field
• Apparent shift from data extraction toward deeper
analysis of the results and their practical use
7
Online Forum of MSR Cookbook
!"#$%%&'()*+'(,-./00*1(%2030404-4%5&.%6
8
Take-away Message
9
• Newcomers to MSR have questions
• Recommendations of best practices for conducting
MSR research
• Online forum to learn, discuss and contribute
• Check the paper for the full list of guidelines!
10

More Related Content

Viewers also liked

Deficient Documentation Detection: A Methodology to Locate Deficient Project D...
Deficient Documentation Detection: A Methodology to Locate Deficient Project D...Deficient Documentation Detection: A Methodology to Locate Deficient Project D...
Deficient Documentation Detection: A Methodology to Locate Deficient Project D...Chenlei Zhang
 
intensive metrics software evolution
intensive metrics software evolutionintensive metrics software evolution
intensive metrics software evolutionIsrael Herraiz
 
MSR2014 opening
MSR2014 openingMSR2014 opening
MSR2014 openingSung Kim
 
Linux Variability Anomalies: What Causes Them and How do They get Fixed?
Linux Variability Anomalies: What Causes Them and How do They get Fixed?Linux Variability Anomalies: What Causes Them and How do They get Fixed?
Linux Variability Anomalies: What Causes Them and How do They get Fixed?sarah_nadi
 

Viewers also liked (6)

Deficient Documentation Detection: A Methodology to Locate Deficient Project D...
Deficient Documentation Detection: A Methodology to Locate Deficient Project D...Deficient Documentation Detection: A Methodology to Locate Deficient Project D...
Deficient Documentation Detection: A Methodology to Locate Deficient Project D...
 
intensive metrics software evolution
intensive metrics software evolutionintensive metrics software evolution
intensive metrics software evolution
 
MSR2014 opening
MSR2014 openingMSR2014 opening
MSR2014 opening
 
Linux Variability Anomalies: What Causes Them and How do They get Fixed?
Linux Variability Anomalies: What Causes Them and How do They get Fixed?Linux Variability Anomalies: What Causes Them and How do They get Fixed?
Linux Variability Anomalies: What Causes Them and How do They get Fixed?
 
MSR 2015 Announcement
MSR 2015 AnnouncementMSR 2015 Announcement
MSR 2015 Announcement
 
MSR 2013 Preview
MSR 2013 PreviewMSR 2013 Preview
MSR 2013 Preview
 

Similar to MSR Cookbook

SuccessfulWriting aThesis orDissertation.docx
SuccessfulWriting  aThesis orDissertation.docxSuccessfulWriting  aThesis orDissertation.docx
SuccessfulWriting aThesis orDissertation.docxAbhinav816839
 
KV713 Session 3
KV713 Session 3KV713 Session 3
KV713 Session 3kturvey
 
KV713 Session 3
KV713 Session 3 KV713 Session 3
KV713 Session 3 kturvey
 
A Collaborative Document Ranking Model for a Multi-Faceted Search
A Collaborative Document Ranking Model for a Multi-Faceted SearchA Collaborative Document Ranking Model for a Multi-Faceted Search
A Collaborative Document Ranking Model for a Multi-Faceted SearchUPMC - Sorbonne Universities
 
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...PROF. PAUL ALLIEU KAMARA
 
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...PROF. PAUL ALLIEU KAMARA
 
Publishing Qualitative Research
Publishing Qualitative ResearchPublishing Qualitative Research
Publishing Qualitative ResearchJoel West
 
Questionnaires hampshire teaching schools_final
Questionnaires hampshire teaching schools_finalQuestionnaires hampshire teaching schools_final
Questionnaires hampshire teaching schools_finalRuth Deakin Crick
 
Greener S. - Business Research Methods (2008).pdf
Greener S. - Business Research Methods (2008).pdfGreener S. - Business Research Methods (2008).pdf
Greener S. - Business Research Methods (2008).pdfarpadsipos
 
How do we begin research
How do we begin researchHow do we begin research
How do we begin researchAngela Tironi
 
KV713 Session 3
KV713 Session 3KV713 Session 3
KV713 Session 3kturvey
 
Trends in Project Management 1966-2015
Trends in Project Management 1966-2015Trends in Project Management 1966-2015
Trends in Project Management 1966-2015dbiggins
 
Eight Steps Approach to Design Research Protocol
Eight Steps Approach to Design Research ProtocolEight Steps Approach to Design Research Protocol
Eight Steps Approach to Design Research Protocolnetraangadi2
 
Introduction research(1).pptx
Introduction research(1).pptxIntroduction research(1).pptx
Introduction research(1).pptxAlmaAtakeluargaAA
 
Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...OCLC
 
Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...Lynn Connaway
 
Research methodology in finance l2(2020)
Research methodology in finance l2(2020)Research methodology in finance l2(2020)
Research methodology in finance l2(2020)datkumari
 

Similar to MSR Cookbook (20)

LEAD 901 Chapter 8
LEAD 901 Chapter 8LEAD 901 Chapter 8
LEAD 901 Chapter 8
 
SuccessfulWriting aThesis orDissertation.docx
SuccessfulWriting  aThesis orDissertation.docxSuccessfulWriting  aThesis orDissertation.docx
SuccessfulWriting aThesis orDissertation.docx
 
KV713 Session 3
KV713 Session 3KV713 Session 3
KV713 Session 3
 
KV713 Session 3
KV713 Session 3 KV713 Session 3
KV713 Session 3
 
A Collaborative Document Ranking Model for a Multi-Faceted Search
A Collaborative Document Ranking Model for a Multi-Faceted SearchA Collaborative Document Ranking Model for a Multi-Faceted Search
A Collaborative Document Ranking Model for a Multi-Faceted Search
 
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
 
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
 
Publishing Qualitative Research
Publishing Qualitative ResearchPublishing Qualitative Research
Publishing Qualitative Research
 
Questionnaires hampshire teaching schools_final
Questionnaires hampshire teaching schools_finalQuestionnaires hampshire teaching schools_final
Questionnaires hampshire teaching schools_final
 
Greener S. - Business Research Methods (2008).pdf
Greener S. - Business Research Methods (2008).pdfGreener S. - Business Research Methods (2008).pdf
Greener S. - Business Research Methods (2008).pdf
 
How do we begin research
How do we begin researchHow do we begin research
How do we begin research
 
How to write research paper
How to write research paper How to write research paper
How to write research paper
 
KV713 Session 3
KV713 Session 3KV713 Session 3
KV713 Session 3
 
Trends in Project Management 1966-2015
Trends in Project Management 1966-2015Trends in Project Management 1966-2015
Trends in Project Management 1966-2015
 
Eight Steps Approach to Design Research Protocol
Eight Steps Approach to Design Research ProtocolEight Steps Approach to Design Research Protocol
Eight Steps Approach to Design Research Protocol
 
Introduction research(1).pptx
Introduction research(1).pptxIntroduction research(1).pptx
Introduction research(1).pptx
 
Research Methodology
Research MethodologyResearch Methodology
Research Methodology
 
Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...
 
Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...
 
Research methodology in finance l2(2020)
Research methodology in finance l2(2020)Research methodology in finance l2(2020)
Research methodology in finance l2(2020)
 

Recently uploaded

When Quality Assurance Meets Innovation in Higher Education - Report launch w...
When Quality Assurance Meets Innovation in Higher Education - Report launch w...When Quality Assurance Meets Innovation in Higher Education - Report launch w...
When Quality Assurance Meets Innovation in Higher Education - Report launch w...Gary Wood
 
Observing-Correct-Grammar-in-Making-Definitions.pptx
Observing-Correct-Grammar-in-Making-Definitions.pptxObserving-Correct-Grammar-in-Making-Definitions.pptx
Observing-Correct-Grammar-in-Making-Definitions.pptxAdelaideRefugio
 
Michaelis Menten Equation and Estimation Of Vmax and Tmax.pptx
Michaelis Menten Equation and Estimation Of Vmax and Tmax.pptxMichaelis Menten Equation and Estimation Of Vmax and Tmax.pptx
Michaelis Menten Equation and Estimation Of Vmax and Tmax.pptxRugvedSathawane
 
SPLICE Working Group: Reusable Code Examples
SPLICE Working Group:Reusable Code ExamplesSPLICE Working Group:Reusable Code Examples
SPLICE Working Group: Reusable Code ExamplesPeter Brusilovsky
 
SURVEY I created for uni project research
SURVEY I created for uni project researchSURVEY I created for uni project research
SURVEY I created for uni project researchCaitlinCummins3
 
Sternal Fractures & Dislocations - EMGuidewire Radiology Reading Room
Sternal Fractures & Dislocations - EMGuidewire Radiology Reading RoomSternal Fractures & Dislocations - EMGuidewire Radiology Reading Room
Sternal Fractures & Dislocations - EMGuidewire Radiology Reading RoomSean M. Fox
 
COMMUNICATING NEGATIVE NEWS - APPROACHES .pptx
COMMUNICATING NEGATIVE NEWS - APPROACHES .pptxCOMMUNICATING NEGATIVE NEWS - APPROACHES .pptx
COMMUNICATING NEGATIVE NEWS - APPROACHES .pptxannathomasp01
 
DEMONSTRATION LESSON IN ENGLISH 4 MATATAG CURRICULUM
DEMONSTRATION LESSON IN ENGLISH 4 MATATAG CURRICULUMDEMONSTRATION LESSON IN ENGLISH 4 MATATAG CURRICULUM
DEMONSTRATION LESSON IN ENGLISH 4 MATATAG CURRICULUMELOISARIVERA8
 
Graduate Outcomes Presentation Slides - English (v3).pptx
Graduate Outcomes Presentation Slides - English (v3).pptxGraduate Outcomes Presentation Slides - English (v3).pptx
Graduate Outcomes Presentation Slides - English (v3).pptxneillewis46
 
Major project report on Tata Motors and its marketing strategies
Major project report on Tata Motors and its marketing strategiesMajor project report on Tata Motors and its marketing strategies
Major project report on Tata Motors and its marketing strategiesAmanpreetKaur157993
 
What is 3 Way Matching Process in Odoo 17.pptx
What is 3 Way Matching Process in Odoo 17.pptxWhat is 3 Way Matching Process in Odoo 17.pptx
What is 3 Way Matching Process in Odoo 17.pptxCeline George
 
Andreas Schleicher presents at the launch of What does child empowerment mean...
Andreas Schleicher presents at the launch of What does child empowerment mean...Andreas Schleicher presents at the launch of What does child empowerment mean...
Andreas Schleicher presents at the launch of What does child empowerment mean...EduSkills OECD
 
TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT TOÁN 2024 - TỪ CÁC TRƯỜNG, TRƯỜNG...
TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT TOÁN 2024 - TỪ CÁC TRƯỜNG, TRƯỜNG...TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT TOÁN 2024 - TỪ CÁC TRƯỜNG, TRƯỜNG...
TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT TOÁN 2024 - TỪ CÁC TRƯỜNG, TRƯỜNG...Nguyen Thanh Tu Collection
 
diagnosting testing bsc 2nd sem.pptx....
diagnosting testing bsc 2nd sem.pptx....diagnosting testing bsc 2nd sem.pptx....
diagnosting testing bsc 2nd sem.pptx....Ritu480198
 
Spellings Wk 4 and Wk 5 for Grade 4 at CAPS
Spellings Wk 4 and Wk 5 for Grade 4 at CAPSSpellings Wk 4 and Wk 5 for Grade 4 at CAPS
Spellings Wk 4 and Wk 5 for Grade 4 at CAPSAnaAcapella
 
Analyzing and resolving a communication crisis in Dhaka textiles LTD.pptx
Analyzing and resolving a communication crisis in Dhaka textiles LTD.pptxAnalyzing and resolving a communication crisis in Dhaka textiles LTD.pptx
Analyzing and resolving a communication crisis in Dhaka textiles LTD.pptxLimon Prince
 
Transparency, Recognition and the role of eSealing - Ildiko Mazar and Koen No...
Transparency, Recognition and the role of eSealing - Ildiko Mazar and Koen No...Transparency, Recognition and the role of eSealing - Ildiko Mazar and Koen No...
Transparency, Recognition and the role of eSealing - Ildiko Mazar and Koen No...EADTU
 
Personalisation of Education by AI and Big Data - Lourdes Guàrdia
Personalisation of Education by AI and Big Data - Lourdes GuàrdiaPersonalisation of Education by AI and Big Data - Lourdes Guàrdia
Personalisation of Education by AI and Big Data - Lourdes GuàrdiaEADTU
 

Recently uploaded (20)

When Quality Assurance Meets Innovation in Higher Education - Report launch w...
When Quality Assurance Meets Innovation in Higher Education - Report launch w...When Quality Assurance Meets Innovation in Higher Education - Report launch w...
When Quality Assurance Meets Innovation in Higher Education - Report launch w...
 
Observing-Correct-Grammar-in-Making-Definitions.pptx
Observing-Correct-Grammar-in-Making-Definitions.pptxObserving-Correct-Grammar-in-Making-Definitions.pptx
Observing-Correct-Grammar-in-Making-Definitions.pptx
 
Michaelis Menten Equation and Estimation Of Vmax and Tmax.pptx
Michaelis Menten Equation and Estimation Of Vmax and Tmax.pptxMichaelis Menten Equation and Estimation Of Vmax and Tmax.pptx
Michaelis Menten Equation and Estimation Of Vmax and Tmax.pptx
 
SPLICE Working Group: Reusable Code Examples
SPLICE Working Group:Reusable Code ExamplesSPLICE Working Group:Reusable Code Examples
SPLICE Working Group: Reusable Code Examples
 
SURVEY I created for uni project research
SURVEY I created for uni project researchSURVEY I created for uni project research
SURVEY I created for uni project research
 
Sternal Fractures & Dislocations - EMGuidewire Radiology Reading Room
Sternal Fractures & Dislocations - EMGuidewire Radiology Reading RoomSternal Fractures & Dislocations - EMGuidewire Radiology Reading Room
Sternal Fractures & Dislocations - EMGuidewire Radiology Reading Room
 
COMMUNICATING NEGATIVE NEWS - APPROACHES .pptx
COMMUNICATING NEGATIVE NEWS - APPROACHES .pptxCOMMUNICATING NEGATIVE NEWS - APPROACHES .pptx
COMMUNICATING NEGATIVE NEWS - APPROACHES .pptx
 
DEMONSTRATION LESSON IN ENGLISH 4 MATATAG CURRICULUM
DEMONSTRATION LESSON IN ENGLISH 4 MATATAG CURRICULUMDEMONSTRATION LESSON IN ENGLISH 4 MATATAG CURRICULUM
DEMONSTRATION LESSON IN ENGLISH 4 MATATAG CURRICULUM
 
VAMOS CUIDAR DO NOSSO PLANETA! .
VAMOS CUIDAR DO NOSSO PLANETA!                    .VAMOS CUIDAR DO NOSSO PLANETA!                    .
VAMOS CUIDAR DO NOSSO PLANETA! .
 
Graduate Outcomes Presentation Slides - English (v3).pptx
Graduate Outcomes Presentation Slides - English (v3).pptxGraduate Outcomes Presentation Slides - English (v3).pptx
Graduate Outcomes Presentation Slides - English (v3).pptx
 
Major project report on Tata Motors and its marketing strategies
Major project report on Tata Motors and its marketing strategiesMajor project report on Tata Motors and its marketing strategies
Major project report on Tata Motors and its marketing strategies
 
What is 3 Way Matching Process in Odoo 17.pptx
What is 3 Way Matching Process in Odoo 17.pptxWhat is 3 Way Matching Process in Odoo 17.pptx
What is 3 Way Matching Process in Odoo 17.pptx
 
Andreas Schleicher presents at the launch of What does child empowerment mean...
Andreas Schleicher presents at the launch of What does child empowerment mean...Andreas Schleicher presents at the launch of What does child empowerment mean...
Andreas Schleicher presents at the launch of What does child empowerment mean...
 
TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT TOÁN 2024 - TỪ CÁC TRƯỜNG, TRƯỜNG...
TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT TOÁN 2024 - TỪ CÁC TRƯỜNG, TRƯỜNG...TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT TOÁN 2024 - TỪ CÁC TRƯỜNG, TRƯỜNG...
TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT TOÁN 2024 - TỪ CÁC TRƯỜNG, TRƯỜNG...
 
diagnosting testing bsc 2nd sem.pptx....
diagnosting testing bsc 2nd sem.pptx....diagnosting testing bsc 2nd sem.pptx....
diagnosting testing bsc 2nd sem.pptx....
 
Spellings Wk 4 and Wk 5 for Grade 4 at CAPS
Spellings Wk 4 and Wk 5 for Grade 4 at CAPSSpellings Wk 4 and Wk 5 for Grade 4 at CAPS
Spellings Wk 4 and Wk 5 for Grade 4 at CAPS
 
Analyzing and resolving a communication crisis in Dhaka textiles LTD.pptx
Analyzing and resolving a communication crisis in Dhaka textiles LTD.pptxAnalyzing and resolving a communication crisis in Dhaka textiles LTD.pptx
Analyzing and resolving a communication crisis in Dhaka textiles LTD.pptx
 
Transparency, Recognition and the role of eSealing - Ildiko Mazar and Koen No...
Transparency, Recognition and the role of eSealing - Ildiko Mazar and Koen No...Transparency, Recognition and the role of eSealing - Ildiko Mazar and Koen No...
Transparency, Recognition and the role of eSealing - Ildiko Mazar and Koen No...
 
Including Mental Health Support in Project Delivery, 14 May.pdf
Including Mental Health Support in Project Delivery, 14 May.pdfIncluding Mental Health Support in Project Delivery, 14 May.pdf
Including Mental Health Support in Project Delivery, 14 May.pdf
 
Personalisation of Education by AI and Big Data - Lourdes Guàrdia
Personalisation of Education by AI and Big Data - Lourdes GuàrdiaPersonalisation of Education by AI and Big Data - Lourdes Guàrdia
Personalisation of Education by AI and Big Data - Lourdes Guàrdia
 

MSR Cookbook

  • 1. The MSR Cookbook Mining a Decade of Research Hadi Hemmati, Sarah Nadi, Olga Baysal, Oleksii Kononenko, Wei Wang, Reid Holmes, Michael W. Godfrey David R. Cheriton School of Computer Science University of Waterloo, Canada MSR-2013, May 19, 2013
  • 2. Why Do We Need a Cookbook? MSR Cookbook 2
  • 3. Idea Came from MSR Vision 2020 3
  • 4. Methodology Review Open coding MSR 2004–2012 (9 yrs) 270 papers 117 full papers Comments: Generalizable observations or suggestions supported by evidence in the paper 268 comments 4 themes with 16 recommendations Recommendation: Comments supported by evidence from at least 5 papers 4
  • 6. Top 5 Recommendations Validate your assumptions and heuristics; repos are noisy [22 c, 17 p] Watch out for collinearities and skewness when synthesizing models from data [22 c, 17 p] [20 c,16 p] Manually verify outputs [12 c, 10 p] Sometimes other measures work better than precision/ recall Sometimes practicality trumps statistical rigour [16 c, 15 p] 6
  • 7. Theme/Recommendation Trends • MSR research is maturing as a field • Apparent shift from data extraction toward deeper analysis of the results and their practical use 7
  • 8. Online Forum of MSR Cookbook !"#$%%&'()*+'(,-./00*1(%2030404-4%5&.%6 8
  • 9. Take-away Message 9 • Newcomers to MSR have questions • Recommendations of best practices for conducting MSR research • Online forum to learn, discuss and contribute • Check the paper for the full list of guidelines!
  • 10. 10