Your SlideShare is downloading. ×
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply



Published on

Summer school, empirical studies, statistical analyses, PASED, Canada

Summer school, empirical studies, statistical analyses, PASED, Canada

Published in: Technology, Education

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Five Days of Empirical Software Engineering: The PASED Experience Massimiliano Di Penta Giuliano Antoniol Daniel M. Germán Yann-Gaël Guéhéneuc Bram Adams
  • 2. Motivation Empirical background important for graduate students Courses on statistics insufficient to provide such a background (Most) University curricula could not afford do have specific courses Some exceptions (there are others for sure!): • Easterbrook’s CSC2130: Empirical Research Methods in Software Engineering at the University of Toronto (2009) • Herbsleb’s 08-803: Empirical Methods for Socio-Technical Research at CMU (2010) • Dewayne E. Perry course, Univ. of Texas at Austin
  • 3. So students need that!
  • 4. General Info About the School • Ecole Polytechnique de Montréal, June 2011 • Funded by MITACS • low fee for students $250 all included • 44 participants, 9 countries and 25 different institutions • More on
  • 5. Learning Objectives 1. plan and conduct software engineering experiments with human subjects and collect related data 2. plan and conduct software engineering studies involving the mining of data from (un)structured software repositories 3. build prediction and classification models from the collected data, and to use these models
  • 6. Challenges Choosing Topics Combining theory and practice Dealing with heterogeneous participants
  • 7. What Topics?
  • 8. What Topics? Planning the study
  • 9. What Topics? Getting the data Planning the study
  • 10. What Topics? Getting the data Planning the study Analyzing results
  • 11. Only 5 days.... ...that’s too much!
  • 12. The Approach • Learn by example and by doing format • Experiment design principles and statistics introduced by presenting cases from studies in literature • Practical application of theoretical concepts during labs • Course material and laboratory packages available online, including course videos
  • 13. School Content PM Exp. Design Text mining Keynote AM Mining Software Archives Keynote Keynote Keynote Keynote Hands on lab Hands on lab Hands on lab Hands on lab Hands on lab Statistical Predictor Analysis Models
  • 14. Learning by doing...
  • 15. Running example I from the school • Use of UML Stereotypes in comprehension and maintenance tasks • Filippo Ricca, Massimiliano Di Penta, Marco Torchiano, Paolo Tonella, Mariano Ceccato: How Developers' Experience and Ability Influence Web Application Comprehension Tasks Supported by UML Stereotypes: A Series of Four Experiments. IEEE Trans. Software Eng. 36(1): 96-118 (2010) • Filippo Ricca, Massimiliano Di Penta, Marco Torchiano, Paolo Tonella, Mariano Ceccato: The Role of Experience and Ability in Comprehension Tasks Supported by UML Stereotypes. ICSE 2007: 375-384 • In the following briefly referred as “Conallen”
  • 16. Experiment Design from the school Group 1 Lab 1 Lab 2 Group 2 Group 3 Group 4 Sys A Sys A Sys B Sys B Sys B Sys B Sys A Sys A
  • 17. Data format: example from the school
  • 18. Boxplots: Conallen from the school
  • 19. Paired analysis: example ID F.Conallen F.UML T20 0.74 0.74 T21 0.74 0.7 0.29 T24 0.88 0.62 T25 0.75 0.8 T26 0.66 0.39 T27 0.35 0.51 T28 0.62 0.59 T29 0.57 0.68 T30 0.73 0.43 T32 0.74 >wilcox.test(F.Conallen,F.UML, paired=TRUE, alternative="greater") 0.51 T22 from the school 0.56 Wilcoxon signed rank test with continuity correction data: F.Conallen and F.UML V = 138, p-value = 0.04354 alternative hypothesis: true location shift is greater than 0 • Must have data in a paired format • or you can use a proper R script • Need to remove subjects that took part to one lab only • For parametric statistics just replace wilcox.test with t.test
  • 20. Hands on Labs • Mining software repository challenge: extract interesting facts from git • Experiment design: groups working together on designing a study • Data analysis: text mining, analyze working data sets of previous experiments and build bug predictors • Working data sets from previous experiments, PROMISE data sets • Tools: R and Weka
  • 21. Lab Script Example #UNPAIRED ANALYSIS #Analysis of single experiment #Mann-Whitney attach(tbn) wilcox.test(Correct[Fit=="yes"],Correct[Fit=="no"],paired=FALSE,alternative="greater") attach(ttrento) wilcox.test(Correct[Fit=="yes"],Correct[Fit=="no"],paired=FALSE,alternative="greater") attach(tphd) wilcox.test(Correct[Fit=="yes"],Correct[Fit=="no"],paired=FALSE,alternative="greater") #All data attach(t) wilcox.test(Correct[Fit=="yes"],Correct[Fit=="no"],paired=FALSE,alternative="greater") #Exercises: # 1) perform a two-tailed test # 2) can t-test be applied instead of Wilcoxon test? test for data normality using the Wilk-Shapiro test # 3) repeat the analysis using the t-test? # 4) repeat the analysis for the time dependent variable
  • 22. Calibrating Courses to Participants’ Profiles Excellent Excellent 1 Good Good 21 Basic 0 10 20 Good 24 10 20 30 Mining Sw repositories 30 12 Basic 25 None 1 20 1 Good 16 Basic 10 Empirical Sw Engineering Excellent 4 0 2 0 30 Statistical Analyses None 18 None 1 Excellent 20 Basic 23 None 4 7 0 10 20 30 Machine learning
  • 23. Feedbacks Guidelines on what not to do Tutorials on tools Longer Labs How to write empirical papers
  • 24. Acknowledgments • • • • • Lecturers (other than the paper authors): • • Ahmed E. Hassan, Queen’s University, Canada Andrian Marcus, Wayne State University, USA Keynote Speakers • • • • • Gail Murphy, University of British Columbia, USA Prem Devanbu, UC Davis, USA Alain Picard, Benchmark Consulting Services Inc., Canada Maria Codipiero, Peter Colligan, Kal Murtaia, SAP, Canada Marc-André Decoste, Google Montréal, Canada Student volunteers The attendees! MITACS (
  • 25. Conclusions