Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MSR Cookbook


Published on

The Mining Software Repositories (MSR) research community has grown significantly since the first MSR workshop was held in 2004. As the community continues to broaden its scope and deepens its expertise, it is worthwhile to reflect on the best practices that our community has developed over the past decade of research. We identify these best practices by surveying past MSR conferences and workshops. To that end, we review all 117 full papers published in the MSR proceedings between 2004 and 2012. We extract 268 comments from these papers, and categorize them using a grounded theory methodology. From this evaluation, four high-level themes were identified: data acquisition and preparation, synthesis, analysis, and sharing/replication. Within each theme we identify several common recommendations, and also examine how these recommendations have evolved over the past decade. In an effort to make this survey a living artifact, we also provide a public forum that contains the extracted recommendations in the hopes that the MSR community can engage in a continuing discussion on our evolving best practices.

  • Be the first to comment

MSR Cookbook

  1. 1. The MSR CookbookMining a Decade of ResearchHadi Hemmati, Sarah Nadi, Olga Baysal, Oleksii Kononenko,Wei Wang, Reid Holmes, Michael W. GodfreyDavid R. Cheriton School of Computer ScienceUniversity of Waterloo, CanadaMSR-2013, May 19, 2013
  2. 2. Why Do We Need a Cookbook?MSRCookbook2
  3. 3. Idea Came from MSR Vision 20203
  4. 4. MethodologyReview Open codingMSR 2004–2012 (9 yrs)270 papers117 full papersComments:Generalizable observationsor suggestions supportedby evidence in the paper268 comments4 themes with16 recommendationsRecommendation:Comments supportedby evidence from at least5 papers4
  5. 5. Themes5
  6. 6. Top 5 RecommendationsValidate your assumptions andheuristics; repos are noisy[22 c, 17 p]Watch out for collinearities andskewness when synthesizingmodels from data[22 c, 17 p][20 c,16 p]Manually verify outputs[12 c, 10 p]Sometimes other measureswork better than precision/recallSometimes practicalitytrumps statistical rigour[16 c, 15 p]6
  7. 7. Theme/Recommendation Trends• MSR research is maturing as a field• Apparent shift from data extraction toward deeperanalysis of the results and their practical use7
  8. 8. Online Forum of MSR Cookbook!"#$%%&()*+(,-./00*1(%2030404-4%5&.%68
  9. 9. Take-away Message9• Newcomers to MSR have questions• Recommendations of best practices for conductingMSR research• Online forum to learn, discuss and contribute• Check the paper for the full list of guidelines!
  10. 10. 10