Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How to do better experiments in SE

451 views

Published on

Published in: Technology, Education
  • Be the first to comment

How to do better experiments in SE

  1. 1. Q: HOW TO DO BETTER EXPERIMENTS IN SE? TIM@MENZIES.US WVU, SEPT 2013
  2. 2. FROM TURKISH TOASTERS TO NASA SPACE SHIPS 2 Turhan, ESEj’09
  3. 3. Q: WHY IS THIS AN URGENT QUESTION?
  4. 4. 4
  5. 5. WHAT’S AT STAKE? • “Transfer” is a core scientific issue • Lack of transfer of causal effects is the scandal of SE • Replication is Empirical SE is rare • Conclusion instability • It all depends. • The full stop syndrome • The result? • A funding crisis 5
  6. 6. THE BAD NEWS
  7. 7. 7 WAR STORIES (DEFECT PREDICTION) Menzies:TSE’13
  8. 8. MANUAL TRANSFER (WAR STORIES) • Kitchenham, Mendes et al, TSE 2007: for = against • Zimmermann FSE, 2009: cross works in 4/600 times 8
  9. 9. THE GOOD NEWS
  10. 10. BETWEEN TURKISH TOASTERS AND NASA SPACE SHIPS 10 Turhan, ESEj’09
  11. 11. Q: HOW TO TRANSFER LESSONS LEARNED? Ignore most of the data • relevancy filtering: Turhan ESEj’09; Peters TSE’13 • variance filtering: Kocaguneli TSE’12,TSE’13 • performance similarities: He ESEM’13 Contort the data • spectral learning (working in PCA space or some other rotation) Menzies, TSE’13; Nam, ICSE’13 Buildi a bickering committee • Ensembles Minku, PROMISE’12 11
  12. 12. BTW, SOMETIMES, TRANFER BETTER THAN LOCAL 12/1/2011 12 Minku:PROMISE’12 Nam:ICSE’13 Peters:TSE’13
  13. 13. THERE IS HOPE • We’ve been looking in the wrong direction • SE project data = surface features of an underlying effect • Go beneath the surface 14
  14. 14. REFLECT LESS ON RAW DIMENSIONS 12/1/2011 15
  15. 15. WHAT’S CHANGED? Mark of the old novice: • Mostly manual analysis • Obsesses on all the raw data • Shares “the” model (the only, the single) • E.g. “Depth of inheritance is “the” most important predictor for defects.” Mark of the new expert: • Manual and automatic analysis • Combinations of Human + AI: • Each offering input and insights to the other • Filters most of the data, transforms the rest • Shares analysis methods • Cost effective methods for generating local lessons 12/1/2011 16 Most probably wrong
  16. 16. NOT EXTERNAL VALIDITY BUT “META-EXTERNAL VALIDITY” No pair programming, CMM5, agile programming, etc etc But conclusion stability, generality 10/11/2013 17
  17. 17. With new data mining technologies, true picture emerges, where we can see what is going on 12/1/2011 18 SO THERE IS HOPE
  18. 18. 19

×