• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
How to do better experiments in SE
 

How to do better experiments in SE

on

  • 209 views

 

Statistics

Views

Total Views
209
Views on SlideShare
209
Embed Views
0

Actions

Likes
1
Downloads
3
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    How to do better experiments in SE How to do better experiments in SE Presentation Transcript

    • Q: HOW TO DO BETTER EXPERIMENTS IN SE? TIM@MENZIES.US WVU, SEPT 2013
    • FROM TURKISH TOASTERS TO NASA SPACE SHIPS 2 Turhan, ESEj’09
    • Q: WHY IS THIS AN URGENT QUESTION?
    • 4
    • WHAT’S AT STAKE? • “Transfer” is a core scientific issue • Lack of transfer of causal effects is the scandal of SE • Replication is Empirical SE is rare • Conclusion instability • It all depends. • The full stop syndrome • The result? • A funding crisis 5
    • THE BAD NEWS
    • 7 WAR STORIES (DEFECT PREDICTION) Menzies:TSE’13
    • MANUAL TRANSFER (WAR STORIES) • Kitchenham, Mendes et al, TSE 2007: for = against • Zimmermann FSE, 2009: cross works in 4/600 times 8
    • THE GOOD NEWS
    • BETWEEN TURKISH TOASTERS AND NASA SPACE SHIPS 10 Turhan, ESEj’09
    • Q: HOW TO TRANSFER LESSONS LEARNED? Ignore most of the data • relevancy filtering: Turhan ESEj’09; Peters TSE’13 • variance filtering: Kocaguneli TSE’12,TSE’13 • performance similarities: He ESEM’13 Contort the data • spectral learning (working in PCA space or some other rotation) Menzies, TSE’13; Nam, ICSE’13 Buildi a bickering committee • Ensembles Minku, PROMISE’12 11
    • BTW, SOMETIMES, TRANFER BETTER THAN LOCAL 12/1/2011 12 Minku:PROMISE’12 Nam:ICSE’13 Peters:TSE’13
    • THERE IS HOPE • We’ve been looking in the wrong direction • SE project data = surface features of an underlying effect • Go beneath the surface 14
    • REFLECT LESS ON RAW DIMENSIONS 12/1/2011 15
    • WHAT’S CHANGED? Mark of the old novice: • Mostly manual analysis • Obsesses on all the raw data • Shares “the” model (the only, the single) • E.g. “Depth of inheritance is “the” most important predictor for defects.” Mark of the new expert: • Manual and automatic analysis • Combinations of Human + AI: • Each offering input and insights to the other • Filters most of the data, transforms the rest • Shares analysis methods • Cost effective methods for generating local lessons 12/1/2011 16 Most probably wrong
    • NOT EXTERNAL VALIDITY BUT “META-EXTERNAL VALIDITY” No pair programming, CMM5, agile programming, etc etc But conclusion stability, generality 10/11/2013 17
    • With new data mining technologies, true picture emerges, where we can see what is going on 12/1/2011 18 SO THERE IS HOPE
    • 19