Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Msr13 mistake


Published on

Tim Menzies, MSR'13

Published in: Technology, Education
  • Be the first to comment

Msr13 mistake

  1. 1. Oops….tim@menzies.usfayolapeters@gmail.comandrian amarcus@wayne.eduMSR’13
  2. 2. Inevitable, due to the complexity &novelty of our work(But rarely reported, which is…. suspicious)What can we learn from those mistakes? 2
  3. 3. An MSR’13 paper: Cross-company learningCan “Us” can learn from “them”?• Provided “us” selects right data from “them”– Relevancy filtering: [Turhan09] (and any others)– Selection guided by structure of “us”• If “we” is small and “them” is many:– Selection guided using kernelfunctions learned from “them”– Result #1: out-performed [Turhan09].• Result #2: Result #1 was a coding error3
  4. 4. Houston, we have a problem• Mar 15: paper accepted to MSR– “Better cross-company defect prediction”• Mar 29: camera-ready submitted,• ?Apr 10: pre-prints go on-line• April 29: Hyeongmin Jeon, graduate student at Pusan Natl. Univ.,– Emailed us: can’t reproduce result• May 4: Peters, checking code, found error– Manic week of experiments ….• May11: results definitely wrong– Emails to MSR organizers4Btw, < 3 weeks. Wow…
  5. 5. Coding error• Distance between test & training instance– Remove classes– Ran a distance function– Re-inserted the classes• But…. bad re-insert– Used the training class– Not the test class5
  6. 6. Pull the paper?• In the internet age, is that even possible?– X people now have local copies of that paper– Which Google might easily stumble acrossOld pre-print,foundMay 15Old pre-print,foundMay 156
  7. 7. Authors: report your mistakes,openly and honestly• We need to expect, allow, papers with sections:“clarifications”, “errata”, “retractions”• E.g. Murphy-Hill, Parnin, Black. IEEE TSE, Jan 20127
  8. 8. Conference organizers:encourage research honesty• Need CFPs with text that encourages• Repeating and testing and challenging oldresults8
  9. 9. Researchers: Share data, checkeach other’s conclusions• Reinhart & Rogoff [2010]– “countries with debt over 90% of GDP suffer notably lowereconomic growth.”• Thomas Herndon, 3rdyear Ph.D. U.Mass.– Unable to replicate with publicly available data ,– Asked Reinhart & Rogoff for their data– Got it (Their spreadsheet)– Found errors in data on economic growth vs debt levels.• A triumph for open science– Sadly, reported in media as grave mistake– E.g.– Immature view of the nature of science9
  10. 10. Supervisors : encourage aculture of research honesty• What will you tell others about this paper?– A failure? Or a success of the open science method?– Its up to you but understand the implications• If we don’t let grad students report mistakes– Then they won’t• Students graduate,• Leave you,• The error emerges• And you are left with with the problem10
  11. 11. Specific lessons• Data mining experiments are complexsoftware prototypes– Version control(of code and data)– Code inspections– Trap and log your random number seeds– Rewrite data rarely• Pull out the class, process, put it back?• Fuhgeddaboudit• Have data headers of different types– So (say) distance measures can skip over classes11The above error does noteffect Peters & MenziesICSE’12 and TSE’13
  12. 12. Open access science• Repeatable, improvable,– and sometimes even refutable• We should not celebrate the failed paper• But we should celebrate– The open science community that finds such errors• MSR, PROMISE, etc– The grad students that struggle to reproduce results• Hyeongmin Jeon– The integrity of grad students whose first responseon finding an error was to report it• Fayola Peters 12
  13. 13. Was this a “useful” mistake?• Is this insight within this mistake?• What does it mean if using more experience makes thedefect predictor worse?• International workshop on Transfer Learning inSoftware Engineering– Nov, ASE’1313
  14. 14. 14