SlideShare a Scribd company logo
The MSR Cookbook
Mining a Decade of Research
Hadi Hemmati, Sarah Nadi, Olga Baysal, Oleksii Kononenko,
Wei Wang, Reid Holmes, Michael W. Godfrey
David R. Cheriton School of Computer Science
University of Waterloo, Canada
MSR-2013, May 19, 2013
Why Do We Need a Cookbook?
MSR
Cookbook
2
Idea Came from MSR Vision 2020
3
Methodology
Review Open coding
MSR 2004–2012 (9 yrs)
270 papers
117 full papers
Comments:
Generalizable observations
or suggestions supported
by evidence in the paper
268 comments
4 themes with
16 recommendations
Recommendation:
Comments supported
by evidence from at least
5 papers
4
Themes
5
Top 5 Recommendations
Validate your assumptions and
heuristics; repos are noisy
[22 c, 17 p]
Watch out for collinearities and
skewness when synthesizing
models from data
[22 c, 17 p]
[20 c,16 p]
Manually verify outputs
[12 c, 10 p]
Sometimes other measures
work better than precision/
recall
Sometimes practicality
trumps statistical rigour
[16 c, 15 p]
6
Theme/Recommendation Trends
• MSR research is maturing as a field
• Apparent shift from data extraction toward deeper
analysis of the results and their practical use
7
Online Forum of MSR Cookbook
!"#$%%&'()*+'(,-./00*1(%2030404-4%5&.%6
8
Take-away Message
9
• Newcomers to MSR have questions
• Recommendations of best practices for conducting
MSR research
• Online forum to learn, discuss and contribute
• Check the paper for the full list of guidelines!
10

More Related Content

Viewers also liked

Deficient Documentation Detection: A Methodology to Locate Deficient Project D...
Deficient Documentation Detection: A Methodology to Locate Deficient Project D...Deficient Documentation Detection: A Methodology to Locate Deficient Project D...
Deficient Documentation Detection: A Methodology to Locate Deficient Project D...
Chenlei Zhang
 
intensive metrics software evolution
intensive metrics software evolutionintensive metrics software evolution
intensive metrics software evolution
Israel Herraiz
 
MSR2014 opening
MSR2014 openingMSR2014 opening
MSR2014 opening
Sung Kim
 
Linux Variability Anomalies: What Causes Them and How do They get Fixed?
Linux Variability Anomalies: What Causes Them and How do They get Fixed?Linux Variability Anomalies: What Causes Them and How do They get Fixed?
Linux Variability Anomalies: What Causes Them and How do They get Fixed?
sarah_nadi
 
MSR 2015 Announcement
MSR 2015 AnnouncementMSR 2015 Announcement
MSR 2015 Announcement
Massimiliano Di Penta
 
MSR 2013 Preview
MSR 2013 PreviewMSR 2013 Preview
MSR 2013 Preview
Thomas Zimmermann
 

Viewers also liked (6)

Deficient Documentation Detection: A Methodology to Locate Deficient Project D...
Deficient Documentation Detection: A Methodology to Locate Deficient Project D...Deficient Documentation Detection: A Methodology to Locate Deficient Project D...
Deficient Documentation Detection: A Methodology to Locate Deficient Project D...
 
intensive metrics software evolution
intensive metrics software evolutionintensive metrics software evolution
intensive metrics software evolution
 
MSR2014 opening
MSR2014 openingMSR2014 opening
MSR2014 opening
 
Linux Variability Anomalies: What Causes Them and How do They get Fixed?
Linux Variability Anomalies: What Causes Them and How do They get Fixed?Linux Variability Anomalies: What Causes Them and How do They get Fixed?
Linux Variability Anomalies: What Causes Them and How do They get Fixed?
 
MSR 2015 Announcement
MSR 2015 AnnouncementMSR 2015 Announcement
MSR 2015 Announcement
 
MSR 2013 Preview
MSR 2013 PreviewMSR 2013 Preview
MSR 2013 Preview
 

Similar to MSR Cookbook

LEAD 901 Chapter 8
LEAD 901 Chapter 8LEAD 901 Chapter 8
LEAD 901 Chapter 8
BHUOnlineDepartment
 
SuccessfulWriting aThesis orDissertation.docx
SuccessfulWriting  aThesis orDissertation.docxSuccessfulWriting  aThesis orDissertation.docx
SuccessfulWriting aThesis orDissertation.docx
Abhinav816839
 
KV713 Session 3
KV713 Session 3KV713 Session 3
KV713 Session 3
kturvey
 
KV713 Session 3
KV713 Session 3 KV713 Session 3
KV713 Session 3
kturvey
 
A Collaborative Document Ranking Model for a Multi-Faceted Search
A Collaborative Document Ranking Model for a Multi-Faceted SearchA Collaborative Document Ranking Model for a Multi-Faceted Search
A Collaborative Document Ranking Model for a Multi-Faceted Search
UPMC - Sorbonne Universities
 
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
PROF. PAUL ALLIEU KAMARA
 
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
PROF. PAUL ALLIEU KAMARA
 
Publishing Qualitative Research
Publishing Qualitative ResearchPublishing Qualitative Research
Publishing Qualitative Research
Joel West
 
Questionnaires hampshire teaching schools_final
Questionnaires hampshire teaching schools_finalQuestionnaires hampshire teaching schools_final
Questionnaires hampshire teaching schools_final
Ruth Deakin Crick
 
Greener S. - Business Research Methods (2008).pdf
Greener S. - Business Research Methods (2008).pdfGreener S. - Business Research Methods (2008).pdf
Greener S. - Business Research Methods (2008).pdf
arpadsipos
 
How do we begin research
How do we begin researchHow do we begin research
How do we begin research
Angela Tironi
 
How to write research paper
How to write research paper How to write research paper
How to write research paper
Manish Godse, Ph.D.
 
KV713 Session 3
KV713 Session 3KV713 Session 3
KV713 Session 3
kturvey
 
Trends in Project Management 1966-2015
Trends in Project Management 1966-2015Trends in Project Management 1966-2015
Trends in Project Management 1966-2015
dbiggins
 
Eight Steps Approach to Design Research Protocol
Eight Steps Approach to Design Research ProtocolEight Steps Approach to Design Research Protocol
Eight Steps Approach to Design Research Protocol
netraangadi2
 
Introduction research(1).pptx
Introduction research(1).pptxIntroduction research(1).pptx
Introduction research(1).pptx
AlmaAtakeluargaAA
 
Research Methodology
Research MethodologyResearch Methodology
Research Methodology
Hamed Taherdoost
 
Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...
OCLC
 
Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...
Lynn Connaway
 
Research methodology in finance l2(2020)
Research methodology in finance l2(2020)Research methodology in finance l2(2020)
Research methodology in finance l2(2020)
datkumari
 

Similar to MSR Cookbook (20)

LEAD 901 Chapter 8
LEAD 901 Chapter 8LEAD 901 Chapter 8
LEAD 901 Chapter 8
 
SuccessfulWriting aThesis orDissertation.docx
SuccessfulWriting  aThesis orDissertation.docxSuccessfulWriting  aThesis orDissertation.docx
SuccessfulWriting aThesis orDissertation.docx
 
KV713 Session 3
KV713 Session 3KV713 Session 3
KV713 Session 3
 
KV713 Session 3
KV713 Session 3 KV713 Session 3
KV713 Session 3
 
A Collaborative Document Ranking Model for a Multi-Faceted Search
A Collaborative Document Ranking Model for a Multi-Faceted SearchA Collaborative Document Ranking Model for a Multi-Faceted Search
A Collaborative Document Ranking Model for a Multi-Faceted Search
 
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
 
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY   EMBRACING THE IMPORTANCE OF...
UNDERSTAND THE CONCEPTS OF RESEARCH METHODOLOGY EMBRACING THE IMPORTANCE OF...
 
Publishing Qualitative Research
Publishing Qualitative ResearchPublishing Qualitative Research
Publishing Qualitative Research
 
Questionnaires hampshire teaching schools_final
Questionnaires hampshire teaching schools_finalQuestionnaires hampshire teaching schools_final
Questionnaires hampshire teaching schools_final
 
Greener S. - Business Research Methods (2008).pdf
Greener S. - Business Research Methods (2008).pdfGreener S. - Business Research Methods (2008).pdf
Greener S. - Business Research Methods (2008).pdf
 
How do we begin research
How do we begin researchHow do we begin research
How do we begin research
 
How to write research paper
How to write research paper How to write research paper
How to write research paper
 
KV713 Session 3
KV713 Session 3KV713 Session 3
KV713 Session 3
 
Trends in Project Management 1966-2015
Trends in Project Management 1966-2015Trends in Project Management 1966-2015
Trends in Project Management 1966-2015
 
Eight Steps Approach to Design Research Protocol
Eight Steps Approach to Design Research ProtocolEight Steps Approach to Design Research Protocol
Eight Steps Approach to Design Research Protocol
 
Introduction research(1).pptx
Introduction research(1).pptxIntroduction research(1).pptx
Introduction research(1).pptx
 
Research Methodology
Research MethodologyResearch Methodology
Research Methodology
 
Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...
 
Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...Research Methods in Library and Information Science: Trends and Tips for Rese...
Research Methods in Library and Information Science: Trends and Tips for Rese...
 
Research methodology in finance l2(2020)
Research methodology in finance l2(2020)Research methodology in finance l2(2020)
Research methodology in finance l2(2020)
 

Recently uploaded

clinical examination of hip joint (1).pdf
clinical examination of hip joint (1).pdfclinical examination of hip joint (1).pdf
clinical examination of hip joint (1).pdf
Priyankaranawat4
 
Types of Herbal Cosmetics its standardization.
Types of Herbal Cosmetics its standardization.Types of Herbal Cosmetics its standardization.
Types of Herbal Cosmetics its standardization.
Ashokrao Mane college of Pharmacy Peth-Vadgaon
 
PCOS corelations and management through Ayurveda.
PCOS corelations and management through Ayurveda.PCOS corelations and management through Ayurveda.
PCOS corelations and management through Ayurveda.
Dr. Shivangi Singh Parihar
 
Digital Artefact 1 - Tiny Home Environmental Design
Digital Artefact 1 - Tiny Home Environmental DesignDigital Artefact 1 - Tiny Home Environmental Design
Digital Artefact 1 - Tiny Home Environmental Design
amberjdewit93
 
PIMS Job Advertisement 2024.pdf Islamabad
PIMS Job Advertisement 2024.pdf IslamabadPIMS Job Advertisement 2024.pdf Islamabad
PIMS Job Advertisement 2024.pdf Islamabad
AyyanKhan40
 
Top five deadliest dog breeds in America
Top five deadliest dog breeds in AmericaTop five deadliest dog breeds in America
Top five deadliest dog breeds in America
Bisnar Chase Personal Injury Attorneys
 
Chapter 4 - Islamic Financial Institutions in Malaysia.pptx
Chapter 4 - Islamic Financial Institutions in Malaysia.pptxChapter 4 - Islamic Financial Institutions in Malaysia.pptx
Chapter 4 - Islamic Financial Institutions in Malaysia.pptx
Mohd Adib Abd Muin, Senior Lecturer at Universiti Utara Malaysia
 
South African Journal of Science: Writing with integrity workshop (2024)
South African Journal of Science: Writing with integrity workshop (2024)South African Journal of Science: Writing with integrity workshop (2024)
South African Journal of Science: Writing with integrity workshop (2024)
Academy of Science of South Africa
 
CACJapan - GROUP Presentation 1- Wk 4.pdf
CACJapan - GROUP Presentation 1- Wk 4.pdfCACJapan - GROUP Presentation 1- Wk 4.pdf
CACJapan - GROUP Presentation 1- Wk 4.pdf
camakaiclarkmusic
 
Assessment and Planning in Educational technology.pptx
Assessment and Planning in Educational technology.pptxAssessment and Planning in Educational technology.pptx
Assessment and Planning in Educational technology.pptx
Kavitha Krishnan
 
A Independência da América Espanhola LAPBOOK.pdf
A Independência da América Espanhola LAPBOOK.pdfA Independência da América Espanhola LAPBOOK.pdf
A Independência da América Espanhola LAPBOOK.pdf
Jean Carlos Nunes Paixão
 
Film vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movieFilm vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movie
Nicholas Montgomery
 
How to Manage Your Lost Opportunities in Odoo 17 CRM
How to Manage Your Lost Opportunities in Odoo 17 CRMHow to Manage Your Lost Opportunities in Odoo 17 CRM
How to Manage Your Lost Opportunities in Odoo 17 CRM
Celine George
 
A Strategic Approach: GenAI in Education
A Strategic Approach: GenAI in EducationA Strategic Approach: GenAI in Education
A Strategic Approach: GenAI in Education
Peter Windle
 
Liberal Approach to the Study of Indian Politics.pdf
Liberal Approach to the Study of Indian Politics.pdfLiberal Approach to the Study of Indian Politics.pdf
Liberal Approach to the Study of Indian Politics.pdf
WaniBasim
 
Executive Directors Chat Leveraging AI for Diversity, Equity, and Inclusion
Executive Directors Chat  Leveraging AI for Diversity, Equity, and InclusionExecutive Directors Chat  Leveraging AI for Diversity, Equity, and Inclusion
Executive Directors Chat Leveraging AI for Diversity, Equity, and Inclusion
TechSoup
 
Smart-Money for SMC traders good time and ICT
Smart-Money for SMC traders good time and ICTSmart-Money for SMC traders good time and ICT
Smart-Money for SMC traders good time and ICT
simonomuemu
 
Azure Interview Questions and Answers PDF By ScholarHat
Azure Interview Questions and Answers PDF By ScholarHatAzure Interview Questions and Answers PDF By ScholarHat
Azure Interview Questions and Answers PDF By ScholarHat
Scholarhat
 
Life upper-Intermediate B2 Workbook for student
Life upper-Intermediate B2 Workbook for studentLife upper-Intermediate B2 Workbook for student
Life upper-Intermediate B2 Workbook for student
NgcHiNguyn25
 
How to Add Chatter in the odoo 17 ERP Module
How to Add Chatter in the odoo 17 ERP ModuleHow to Add Chatter in the odoo 17 ERP Module
How to Add Chatter in the odoo 17 ERP Module
Celine George
 

Recently uploaded (20)

clinical examination of hip joint (1).pdf
clinical examination of hip joint (1).pdfclinical examination of hip joint (1).pdf
clinical examination of hip joint (1).pdf
 
Types of Herbal Cosmetics its standardization.
Types of Herbal Cosmetics its standardization.Types of Herbal Cosmetics its standardization.
Types of Herbal Cosmetics its standardization.
 
PCOS corelations and management through Ayurveda.
PCOS corelations and management through Ayurveda.PCOS corelations and management through Ayurveda.
PCOS corelations and management through Ayurveda.
 
Digital Artefact 1 - Tiny Home Environmental Design
Digital Artefact 1 - Tiny Home Environmental DesignDigital Artefact 1 - Tiny Home Environmental Design
Digital Artefact 1 - Tiny Home Environmental Design
 
PIMS Job Advertisement 2024.pdf Islamabad
PIMS Job Advertisement 2024.pdf IslamabadPIMS Job Advertisement 2024.pdf Islamabad
PIMS Job Advertisement 2024.pdf Islamabad
 
Top five deadliest dog breeds in America
Top five deadliest dog breeds in AmericaTop five deadliest dog breeds in America
Top five deadliest dog breeds in America
 
Chapter 4 - Islamic Financial Institutions in Malaysia.pptx
Chapter 4 - Islamic Financial Institutions in Malaysia.pptxChapter 4 - Islamic Financial Institutions in Malaysia.pptx
Chapter 4 - Islamic Financial Institutions in Malaysia.pptx
 
South African Journal of Science: Writing with integrity workshop (2024)
South African Journal of Science: Writing with integrity workshop (2024)South African Journal of Science: Writing with integrity workshop (2024)
South African Journal of Science: Writing with integrity workshop (2024)
 
CACJapan - GROUP Presentation 1- Wk 4.pdf
CACJapan - GROUP Presentation 1- Wk 4.pdfCACJapan - GROUP Presentation 1- Wk 4.pdf
CACJapan - GROUP Presentation 1- Wk 4.pdf
 
Assessment and Planning in Educational technology.pptx
Assessment and Planning in Educational technology.pptxAssessment and Planning in Educational technology.pptx
Assessment and Planning in Educational technology.pptx
 
A Independência da América Espanhola LAPBOOK.pdf
A Independência da América Espanhola LAPBOOK.pdfA Independência da América Espanhola LAPBOOK.pdf
A Independência da América Espanhola LAPBOOK.pdf
 
Film vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movieFilm vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movie
 
How to Manage Your Lost Opportunities in Odoo 17 CRM
How to Manage Your Lost Opportunities in Odoo 17 CRMHow to Manage Your Lost Opportunities in Odoo 17 CRM
How to Manage Your Lost Opportunities in Odoo 17 CRM
 
A Strategic Approach: GenAI in Education
A Strategic Approach: GenAI in EducationA Strategic Approach: GenAI in Education
A Strategic Approach: GenAI in Education
 
Liberal Approach to the Study of Indian Politics.pdf
Liberal Approach to the Study of Indian Politics.pdfLiberal Approach to the Study of Indian Politics.pdf
Liberal Approach to the Study of Indian Politics.pdf
 
Executive Directors Chat Leveraging AI for Diversity, Equity, and Inclusion
Executive Directors Chat  Leveraging AI for Diversity, Equity, and InclusionExecutive Directors Chat  Leveraging AI for Diversity, Equity, and Inclusion
Executive Directors Chat Leveraging AI for Diversity, Equity, and Inclusion
 
Smart-Money for SMC traders good time and ICT
Smart-Money for SMC traders good time and ICTSmart-Money for SMC traders good time and ICT
Smart-Money for SMC traders good time and ICT
 
Azure Interview Questions and Answers PDF By ScholarHat
Azure Interview Questions and Answers PDF By ScholarHatAzure Interview Questions and Answers PDF By ScholarHat
Azure Interview Questions and Answers PDF By ScholarHat
 
Life upper-Intermediate B2 Workbook for student
Life upper-Intermediate B2 Workbook for studentLife upper-Intermediate B2 Workbook for student
Life upper-Intermediate B2 Workbook for student
 
How to Add Chatter in the odoo 17 ERP Module
How to Add Chatter in the odoo 17 ERP ModuleHow to Add Chatter in the odoo 17 ERP Module
How to Add Chatter in the odoo 17 ERP Module
 

MSR Cookbook

  • 1. The MSR Cookbook Mining a Decade of Research Hadi Hemmati, Sarah Nadi, Olga Baysal, Oleksii Kononenko, Wei Wang, Reid Holmes, Michael W. Godfrey David R. Cheriton School of Computer Science University of Waterloo, Canada MSR-2013, May 19, 2013
  • 2. Why Do We Need a Cookbook? MSR Cookbook 2
  • 3. Idea Came from MSR Vision 2020 3
  • 4. Methodology Review Open coding MSR 2004–2012 (9 yrs) 270 papers 117 full papers Comments: Generalizable observations or suggestions supported by evidence in the paper 268 comments 4 themes with 16 recommendations Recommendation: Comments supported by evidence from at least 5 papers 4
  • 6. Top 5 Recommendations Validate your assumptions and heuristics; repos are noisy [22 c, 17 p] Watch out for collinearities and skewness when synthesizing models from data [22 c, 17 p] [20 c,16 p] Manually verify outputs [12 c, 10 p] Sometimes other measures work better than precision/ recall Sometimes practicality trumps statistical rigour [16 c, 15 p] 6
  • 7. Theme/Recommendation Trends • MSR research is maturing as a field • Apparent shift from data extraction toward deeper analysis of the results and their practical use 7
  • 8. Online Forum of MSR Cookbook !"#$%%&'()*+'(,-./00*1(%2030404-4%5&.%6 8
  • 9. Take-away Message 9 • Newcomers to MSR have questions • Recommendations of best practices for conducting MSR research • Online forum to learn, discuss and contribute • Check the paper for the full list of guidelines!
  • 10. 10