SlideShare a Scribd company logo
The MSR Cookbook
Mining a Decade of Research
Hadi Hemmati, Sarah Nadi, Olga Baysal, Oleksii Kononenko,
Wei Wang, Reid Holmes, Michael W. Godfrey
David R. Cheriton School of Computer Science
University of Waterloo, Canada
MSR-2013, May 19, 2013
Why Do We Need a Cookbook?
MSR
Cookbook
2
Idea Came from MSR Vision 2020
3
Methodology
Review Open coding
MSR 2004โ€“2012 (9 yrs)
270 papers
117 full papers
Comments:
Generalizable observations
or suggestions supported
by evidence in the paper
268 comments
4 themes with
16 recommendations
Recommendation:
Comments supported
by evidence from at least
5 papers
4
Themes
5
Top 5 Recommendations
Validate your assumptions and
heuristics; repos are noisy
[22 c, 17 p]
Watch out for collinearities and
skewness when synthesizing
models from data
[22 c, 17 p]
[20 c,16 p]
Manually verify outputs
[12 c, 10 p]
Sometimes other measures
work better than precision/
recall
Sometimes practicality
trumps statistical rigour
[16 c, 15 p]
6
Theme/Recommendation Trends
โ€ข MSR research is maturing as a field
โ€ข Apparent shift from data extraction toward deeper
analysis of the results and their practical use
7
Online Forum of MSR Cookbook
!"#$%%&'()*+'(,-./00*1(%2030404-4%5&.%6
8
Take-away Message
9
โ€ข Newcomers to MSR have questions
โ€ข Recommendations of best practices for conducting
MSR research
โ€ข Online forum to learn, discuss and contribute
โ€ข Check the paper for the full list of guidelines!
10

More Related Content

Viewers also liked

Deficient Documentation Detection: A Methodology to Locate De๏ฌcient Project D...
Deficient Documentation Detection: A Methodology to Locate De๏ฌcient Project D...Deficient Documentation Detection: A Methodology to Locate De๏ฌcient Project D...
Deficient Documentation Detection: A Methodology to Locate De๏ฌcient Project D...
Chenlei Zhang
ย 
intensive metrics software evolution
intensive metrics software evolutionintensive metrics software evolution
intensive metrics software evolution
Israel Herraiz
ย 
MSR2014 opening
MSR2014 openingMSR2014 opening
MSR2014 opening
Sung Kim
ย 
Linux Variability Anomalies: What Causes Them and How do They get Fixed?
Linux Variability Anomalies: What Causes Them and How do They get Fixed?Linux Variability Anomalies: What Causes Them and How do They get Fixed?
Linux Variability Anomalies: What Causes Them and How do They get Fixed?
sarah_nadi
ย 
MSR 2015 Announcement
MSR 2015 AnnouncementMSR 2015 Announcement
MSR 2015 Announcement
Massimiliano Di Penta
ย 
MSR 2013 Preview
MSR 2013 PreviewMSR 2013 Preview
MSR 2013 Preview
Thomas Zimmermann
ย 

Viewers also liked (6)

Deficient Documentation Detection: A Methodology to Locate De๏ฌcient Project D...
Deficient Documentation Detection: A Methodology to Locate De๏ฌcient Project D...Deficient Documentation Detection: A Methodology to Locate De๏ฌcient Project D...
Deficient Documentation Detection: A Methodology to Locate De๏ฌcient Project D...
ย 
intensive metrics software evolution
intensive metrics software evolutionintensive metrics software evolution
intensive metrics software evolution
ย 
MSR2014 opening
MSR2014 openingMSR2014 opening
MSR2014 opening
ย 
Linux Variability Anomalies: What Causes Them and How do They get Fixed?
Linux Variability Anomalies: What Causes Them and How do They get Fixed?Linux Variability Anomalies: What Causes Them and How do They get Fixed?
Linux Variability Anomalies: What Causes Them and How do They get Fixed?
ย 
MSR 2015 Announcement
MSR 2015 AnnouncementMSR 2015 Announcement
MSR 2015 Announcement
ย 
MSR 2013 Preview
MSR 2013 PreviewMSR 2013 Preview
MSR 2013 Preview
ย 

Similar to MSR Cookbook

LEAD 901 Chapter 8
LEAD 901 Chapter 8LEAD 901 Chapter 8
LEAD 901 Chapter 8
BHUOnlineDepartment
ย 
SuccessfulWriting aThesis orDissertation.docx
SuccessfulWriting  aThesis orDissertation.docxSuccessfulWriting  aThesis orDissertation.docx
SuccessfulWriting aThesis orDissertation.docx
Abhinav816839
ย 
KV713 Session 3
KV713 Session 3KV713 Session 3
KV713 Session 3
kturvey
ย 
KV713 Session 3
KV713 Session 3 KV713 Session 3
KV713 Session 3
kturvey
ย 
A Collaborative Document Ranking Model for a Multi-Faceted Search
A Collaborative Document Ranking Model for a Multi-Faceted SearchA Collaborative Document Ranking Model for a Multi-Faceted Search
A Collaborative Document Ranking Model for a Multi-Faceted Search
UPMC - Sorbonne Universities
ย 
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
PROF. PAUL ALLIEU KAMARA
ย 
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
PROF. PAUL ALLIEU KAMARA
ย 
Publishing Qualitative Research
Publishing Qualitative ResearchPublishing Qualitative Research
Publishing Qualitative Research
Joel West
ย 
Questionnaires hampshire teaching schools_final
Questionnaires hampshire teaching schools_finalQuestionnaires hampshire teaching schools_final
Questionnaires hampshire teaching schools_final
Ruth Deakin Crick
ย 
Greener S. - Business Research Methods (2008).pdf
Greener S. - Business Research Methods (2008).pdfGreener S. - Business Research Methods (2008).pdf
Greener S. - Business Research Methods (2008).pdf
arpadsipos
ย 
How do we begin research
How do we begin researchHow do we begin research
How do we begin research
Angela Tironi
ย 
How to write research paper
How to write research paper How to write research paper
How to write research paper
Manish Godse, Ph.D.
ย 
KV713 Session 3
KV713 Session 3KV713 Session 3
KV713 Session 3
kturvey
ย 
Trends in Project Management 1966-2015
Trends in Project Management 1966-2015Trends in Project Management 1966-2015
Trends in Project Management 1966-2015
dbiggins
ย 
Eight Steps Approach to Design Research Protocol
Eight Steps Approach to Design Research ProtocolEight Steps Approach to Design Research Protocol
Eight Steps Approach to Design Research Protocol
netraangadi2
ย 
Introduction research(1).pptx
Introduction research(1).pptxIntroduction research(1).pptx
Introduction research(1).pptx
AlmaAtakeluargaAA
ย 
Research Methodology
Research MethodologyResearch Methodology
Research Methodology
Hamed Taherdoost
ย 
Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...
OCLC
ย 
Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...
Lynn Connaway
ย 
Research methodology in finance l2(2020)
Research methodology in finance l2(2020)Research methodology in finance l2(2020)
Research methodology in finance l2(2020)
datkumari
ย 

Similar to MSR Cookbook (20)

LEAD 901 Chapter 8
LEAD 901 Chapter 8LEAD 901 Chapter 8
LEAD 901 Chapter 8
ย 
SuccessfulWriting aThesis orDissertation.docx
SuccessfulWriting  aThesis orDissertation.docxSuccessfulWriting  aThesis orDissertation.docx
SuccessfulWriting aThesis orDissertation.docx
ย 
KV713 Session 3
KV713 Session 3KV713 Session 3
KV713 Session 3
ย 
KV713 Session 3
KV713 Session 3 KV713 Session 3
KV713 Session 3
ย 
A Collaborative Document Ranking Model for a Multi-Faceted Search
A Collaborative Document Ranking Model for a Multi-Faceted SearchA Collaborative Document Ranking Model for a Multi-Faceted Search
A Collaborative Document Ranking Model for a Multi-Faceted Search
ย 
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
ย 
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
ย 
Publishing Qualitative Research
Publishing Qualitative ResearchPublishing Qualitative Research
Publishing Qualitative Research
ย 
Questionnaires hampshire teaching schools_final
Questionnaires hampshire teaching schools_finalQuestionnaires hampshire teaching schools_final
Questionnaires hampshire teaching schools_final
ย 
Greener S. - Business Research Methods (2008).pdf
Greener S. - Business Research Methods (2008).pdfGreener S. - Business Research Methods (2008).pdf
Greener S. - Business Research Methods (2008).pdf
ย 
How do we begin research
How do we begin researchHow do we begin research
How do we begin research
ย 
How to write research paper
How to write research paper How to write research paper
How to write research paper
ย 
KV713 Session 3
KV713 Session 3KV713 Session 3
KV713 Session 3
ย 
Trends in Project Management 1966-2015
Trends in Project Management 1966-2015Trends in Project Management 1966-2015
Trends in Project Management 1966-2015
ย 
Eight Steps Approach to Design Research Protocol
Eight Steps Approach to Design Research ProtocolEight Steps Approach to Design Research Protocol
Eight Steps Approach to Design Research Protocol
ย 
Introduction research(1).pptx
Introduction research(1).pptxIntroduction research(1).pptx
Introduction research(1).pptx
ย 
Research Methodology
Research MethodologyResearch Methodology
Research Methodology
ย 
Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...
ย 
Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...
ย 
Research methodology in finance l2(2020)
Research methodology in finance l2(2020)Research methodology in finance l2(2020)
Research methodology in finance l2(2020)
ย 

Recently uploaded

RESULTS OF THE EVALUATION QUESTIONNAIRE.pptx
RESULTS OF THE EVALUATION QUESTIONNAIRE.pptxRESULTS OF THE EVALUATION QUESTIONNAIRE.pptx
RESULTS OF THE EVALUATION QUESTIONNAIRE.pptx
zuzanka
ย 
writing about opinions about Australia the movie
writing about opinions about Australia the moviewriting about opinions about Australia the movie
writing about opinions about Australia the movie
Nicholas Montgomery
ย 
Walmart Business+ and Spark Good for Nonprofits.pdf
Walmart Business+ and Spark Good for Nonprofits.pdfWalmart Business+ and Spark Good for Nonprofits.pdf
Walmart Business+ and Spark Good for Nonprofits.pdf
TechSoup
ย 
Philippine Edukasyong Pantahanan at Pangkabuhayan (EPP) Curriculum
Philippine Edukasyong Pantahanan at Pangkabuhayan (EPP) CurriculumPhilippine Edukasyong Pantahanan at Pangkabuhayan (EPP) Curriculum
Philippine Edukasyong Pantahanan at Pangkabuhayan (EPP) Curriculum
MJDuyan
ย 
Benner "Expanding Pathways to Publishing Careers"
Benner "Expanding Pathways to Publishing Careers"Benner "Expanding Pathways to Publishing Careers"
Benner "Expanding Pathways to Publishing Careers"
National Information Standards Organization (NISO)
ย 
Wound healing PPT
Wound healing PPTWound healing PPT
Wound healing PPT
Jyoti Chand
ย 
LAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UP
LAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UPLAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UP
LAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UP
RAHUL
ย 
How to Setup Warehouse & Location in Odoo 17 Inventory
How to Setup Warehouse & Location in Odoo 17 InventoryHow to Setup Warehouse & Location in Odoo 17 Inventory
How to Setup Warehouse & Location in Odoo 17 Inventory
Celine George
ย 
math operations ued in python and all used
math operations ued in python and all usedmath operations ued in python and all used
math operations ued in python and all used
ssuser13ffe4
ย 
Bร€I TแบฌP Bแป” TRแปข TIแบพNG ANH 8 Cแบข Nฤ‚M - GLOBAL SUCCESS - Nฤ‚M HแปŒC 2023-2024 (Cร“ FI...
Bร€I TแบฌP Bแป” TRแปข TIแบพNG ANH 8 Cแบข Nฤ‚M - GLOBAL SUCCESS - Nฤ‚M HแปŒC 2023-2024 (Cร“ FI...Bร€I TแบฌP Bแป” TRแปข TIแบพNG ANH 8 Cแบข Nฤ‚M - GLOBAL SUCCESS - Nฤ‚M HแปŒC 2023-2024 (Cร“ FI...
Bร€I TแบฌP Bแป” TRแปข TIแบพNG ANH 8 Cแบข Nฤ‚M - GLOBAL SUCCESS - Nฤ‚M HแปŒC 2023-2024 (Cร“ FI...
Nguyen Thanh Tu Collection
ย 
RHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem students
RHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem studentsRHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem students
RHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem students
Himanshu Rai
ย 
ISO/IEC 27001, ISO/IEC 42001, and GDPR: Best Practices for Implementation and...
ISO/IEC 27001, ISO/IEC 42001, and GDPR: Best Practices for Implementation and...ISO/IEC 27001, ISO/IEC 42001, and GDPR: Best Practices for Implementation and...
ISO/IEC 27001, ISO/IEC 42001, and GDPR: Best Practices for Implementation and...
PECB
ย 
Level 3 NCEA - NZ: A Nation In the Making 1872 - 1900 SML.ppt
Level 3 NCEA - NZ: A  Nation In the Making 1872 - 1900 SML.pptLevel 3 NCEA - NZ: A  Nation In the Making 1872 - 1900 SML.ppt
Level 3 NCEA - NZ: A Nation In the Making 1872 - 1900 SML.ppt
Henry Hollis
ย 
Gender and Mental Health - Counselling and Family Therapy Applications and In...
Gender and Mental Health - Counselling and Family Therapy Applications and In...Gender and Mental Health - Counselling and Family Therapy Applications and In...
Gender and Mental Health - Counselling and Family Therapy Applications and In...
PsychoTech Services
ย 
Prรฉsentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptx
Prรฉsentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptxPrรฉsentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptx
Prรฉsentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptx
siemaillard
ย 
Nutrition Inc FY 2024, 4 - Hour Training
Nutrition Inc FY 2024, 4 - Hour TrainingNutrition Inc FY 2024, 4 - Hour Training
Nutrition Inc FY 2024, 4 - Hour Training
melliereed
ย 
SWOT analysis in the project Keeping the Memory @live.pptx
SWOT analysis in the project Keeping the Memory @live.pptxSWOT analysis in the project Keeping the Memory @live.pptx
SWOT analysis in the project Keeping the Memory @live.pptx
zuzanka
ย 
How to deliver Powerpoint Presentations.pptx
How to deliver Powerpoint  Presentations.pptxHow to deliver Powerpoint  Presentations.pptx
How to deliver Powerpoint Presentations.pptx
HajraNaeem15
ย 
Temple of Asclepius in Thrace. Excavation results
Temple of Asclepius in Thrace. Excavation resultsTemple of Asclepius in Thrace. Excavation results
Temple of Asclepius in Thrace. Excavation results
Krassimira Luka
ย 
spot a liar (Haiqa 146).pptx Technical writhing and presentation skills
spot a liar (Haiqa 146).pptx Technical writhing and presentation skillsspot a liar (Haiqa 146).pptx Technical writhing and presentation skills
spot a liar (Haiqa 146).pptx Technical writhing and presentation skills
haiqairshad
ย 

Recently uploaded (20)

RESULTS OF THE EVALUATION QUESTIONNAIRE.pptx
RESULTS OF THE EVALUATION QUESTIONNAIRE.pptxRESULTS OF THE EVALUATION QUESTIONNAIRE.pptx
RESULTS OF THE EVALUATION QUESTIONNAIRE.pptx
ย 
writing about opinions about Australia the movie
writing about opinions about Australia the moviewriting about opinions about Australia the movie
writing about opinions about Australia the movie
ย 
Walmart Business+ and Spark Good for Nonprofits.pdf
Walmart Business+ and Spark Good for Nonprofits.pdfWalmart Business+ and Spark Good for Nonprofits.pdf
Walmart Business+ and Spark Good for Nonprofits.pdf
ย 
Philippine Edukasyong Pantahanan at Pangkabuhayan (EPP) Curriculum
Philippine Edukasyong Pantahanan at Pangkabuhayan (EPP) CurriculumPhilippine Edukasyong Pantahanan at Pangkabuhayan (EPP) Curriculum
Philippine Edukasyong Pantahanan at Pangkabuhayan (EPP) Curriculum
ย 
Benner "Expanding Pathways to Publishing Careers"
Benner "Expanding Pathways to Publishing Careers"Benner "Expanding Pathways to Publishing Careers"
Benner "Expanding Pathways to Publishing Careers"
ย 
Wound healing PPT
Wound healing PPTWound healing PPT
Wound healing PPT
ย 
LAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UP
LAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UPLAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UP
LAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UP
ย 
How to Setup Warehouse & Location in Odoo 17 Inventory
How to Setup Warehouse & Location in Odoo 17 InventoryHow to Setup Warehouse & Location in Odoo 17 Inventory
How to Setup Warehouse & Location in Odoo 17 Inventory
ย 
math operations ued in python and all used
math operations ued in python and all usedmath operations ued in python and all used
math operations ued in python and all used
ย 
Bร€I TแบฌP Bแป” TRแปข TIแบพNG ANH 8 Cแบข Nฤ‚M - GLOBAL SUCCESS - Nฤ‚M HแปŒC 2023-2024 (Cร“ FI...
Bร€I TแบฌP Bแป” TRแปข TIแบพNG ANH 8 Cแบข Nฤ‚M - GLOBAL SUCCESS - Nฤ‚M HแปŒC 2023-2024 (Cร“ FI...Bร€I TแบฌP Bแป” TRแปข TIแบพNG ANH 8 Cแบข Nฤ‚M - GLOBAL SUCCESS - Nฤ‚M HแปŒC 2023-2024 (Cร“ FI...
Bร€I TแบฌP Bแป” TRแปข TIแบพNG ANH 8 Cแบข Nฤ‚M - GLOBAL SUCCESS - Nฤ‚M HแปŒC 2023-2024 (Cร“ FI...
ย 
RHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem students
RHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem studentsRHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem students
RHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem students
ย 
ISO/IEC 27001, ISO/IEC 42001, and GDPR: Best Practices for Implementation and...
ISO/IEC 27001, ISO/IEC 42001, and GDPR: Best Practices for Implementation and...ISO/IEC 27001, ISO/IEC 42001, and GDPR: Best Practices for Implementation and...
ISO/IEC 27001, ISO/IEC 42001, and GDPR: Best Practices for Implementation and...
ย 
Level 3 NCEA - NZ: A Nation In the Making 1872 - 1900 SML.ppt
Level 3 NCEA - NZ: A  Nation In the Making 1872 - 1900 SML.pptLevel 3 NCEA - NZ: A  Nation In the Making 1872 - 1900 SML.ppt
Level 3 NCEA - NZ: A Nation In the Making 1872 - 1900 SML.ppt
ย 
Gender and Mental Health - Counselling and Family Therapy Applications and In...
Gender and Mental Health - Counselling and Family Therapy Applications and In...Gender and Mental Health - Counselling and Family Therapy Applications and In...
Gender and Mental Health - Counselling and Family Therapy Applications and In...
ย 
Prรฉsentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptx
Prรฉsentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptxPrรฉsentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptx
Prรฉsentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptx
ย 
Nutrition Inc FY 2024, 4 - Hour Training
Nutrition Inc FY 2024, 4 - Hour TrainingNutrition Inc FY 2024, 4 - Hour Training
Nutrition Inc FY 2024, 4 - Hour Training
ย 
SWOT analysis in the project Keeping the Memory @live.pptx
SWOT analysis in the project Keeping the Memory @live.pptxSWOT analysis in the project Keeping the Memory @live.pptx
SWOT analysis in the project Keeping the Memory @live.pptx
ย 
How to deliver Powerpoint Presentations.pptx
How to deliver Powerpoint  Presentations.pptxHow to deliver Powerpoint  Presentations.pptx
How to deliver Powerpoint Presentations.pptx
ย 
Temple of Asclepius in Thrace. Excavation results
Temple of Asclepius in Thrace. Excavation resultsTemple of Asclepius in Thrace. Excavation results
Temple of Asclepius in Thrace. Excavation results
ย 
spot a liar (Haiqa 146).pptx Technical writhing and presentation skills
spot a liar (Haiqa 146).pptx Technical writhing and presentation skillsspot a liar (Haiqa 146).pptx Technical writhing and presentation skills
spot a liar (Haiqa 146).pptx Technical writhing and presentation skills
ย 

MSR Cookbook

  • 1. The MSR Cookbook Mining a Decade of Research Hadi Hemmati, Sarah Nadi, Olga Baysal, Oleksii Kononenko, Wei Wang, Reid Holmes, Michael W. Godfrey David R. Cheriton School of Computer Science University of Waterloo, Canada MSR-2013, May 19, 2013
  • 2. Why Do We Need a Cookbook? MSR Cookbook 2
  • 3. Idea Came from MSR Vision 2020 3
  • 4. Methodology Review Open coding MSR 2004โ€“2012 (9 yrs) 270 papers 117 full papers Comments: Generalizable observations or suggestions supported by evidence in the paper 268 comments 4 themes with 16 recommendations Recommendation: Comments supported by evidence from at least 5 papers 4
  • 6. Top 5 Recommendations Validate your assumptions and heuristics; repos are noisy [22 c, 17 p] Watch out for collinearities and skewness when synthesizing models from data [22 c, 17 p] [20 c,16 p] Manually verify outputs [12 c, 10 p] Sometimes other measures work better than precision/ recall Sometimes practicality trumps statistical rigour [16 c, 15 p] 6
  • 7. Theme/Recommendation Trends โ€ข MSR research is maturing as a field โ€ข Apparent shift from data extraction toward deeper analysis of the results and their practical use 7
  • 8. Online Forum of MSR Cookbook !"#$%%&'()*+'(,-./00*1(%2030404-4%5&.%6 8
  • 9. Take-away Message 9 โ€ข Newcomers to MSR have questions โ€ข Recommendations of best practices for conducting MSR research โ€ข Online forum to learn, discuss and contribute โ€ข Check the paper for the full list of guidelines!
  • 10. 10