Successfully reported this slideshow.

Performing Large Scale Repeatable Software Engineering Studies

501 views

Published on

Talk given at the ETH Chair on Software Engineering.

  • Be the first to comment

  • Be the first to like this

Performing Large Scale Repeatable Software Engineering Studies

  1. 1. Performing Large Scale Software Engineering Studies Georgios Gousios
  2. 2. $whoami
  3. 3. SE = Empirical Sciecne
  4. 4. Observation
  5. 5. Hypothes{ie}s
  6. 6. Models
  7. 7. ValidateInvalidate
  8. 8. 30 25 20Number of Works 15 10 5 0 0 1 2 3 4 5 6 8 10 25 50 Sample sizes (number of projects)
  9. 9. Metric Gnome-VFS Evolution KDESCM History 10y 9m 12y 4m 14y Num 5522 36835 >1200000 Revisions SCM Size 105 MB 1.4 GB 60 GBNum Emails 20 240 6252(Nov 2008)
  10. 10. 400 300GB 200 100 Others KDE 0 700 projects LSE (2003-2006) GenBank Dataset
  11. 11. RCS Postfix BugzillaCVS MailMan SF.net TrackerSVN JiraDarcs Marc Gnats Git Hg
  12. 12. Researcher’s viewTime 1. Go to project.org 2. Dowload SVN, Mail, Bug, ask for IRC logs 3. .....? 4. Publish research
  13. 13. Researcher’s view (some) Project’s viewTime 1. Go to project.org 1. Hm, a new visitor 2. Hey, she is mirroring our bugzilla 2. Dowload SVN, Mail, Bug, ask for IRC logs 3. .....? 3. .....? 4. Ban her! 4. Publish research
  14. 14. Researcher’s view (some) Project’s viewTime 1. Go to project.org 1. Hm, a new visitor 2. Hey, she is mirroring our bugzilla 2. Dowload SVN, Mail, Bug, ask for IRC logs 3. .....? 3. .....? 4. Ban her! 4. Publish research
  15. 15. How is empirical research done in mature disciplines?
  16. 16. Pre-processed data
  17. 17. Result Sharing
  18. 18. Replication
  19. 19. Research Platforms
  20. 20. Our research:A platform for software engineering research
  21. 21. In our research1. We examined the current situation2. We propose a platform for large scale research3. We validated its design with 2 case studies
  22. 22. Empirical = ModelStudy + Data + Metrics / Tools + Analysis Methods + Results analysis
  23. 23. Empirical = ModelStudy + Data + Metrics / Tools + Analysis Methods + Results analysis
  24. 24. Empirical = ModelStudy + Data + Metrics / Tools + Analysis Methods + Results analysis
  25. 25. Empirical = ModelStudy + Data + Metrics / Tools + Analysis Methods + Results analysis
  26. 26. Research methods 40 35 30 25Number of Works 20 15 10 5 0 0 EXP FCS ECS CCS SUR Research Methods used
  27. 27. What sources of data are in use? 45 40 35 30Number of Works 25 20 15 10 5 0 0 BTS SRC SF ECT SCM Data Source
  28. 28. What is the examined data size (in number of projects) ? 30 25 20Number of Works 15 10 5 0 0 1 2 3 4 5 6 8 10 25 50 Sample sizes (number of projects)
  29. 29. Findings• Sample size very small • How can we extract generic results?• No experiment replication • Do we believe in each other’s work or just ignore it?• We did not check the stats...
  30. 30. @Only 20% of the tools and data reported in ICSEpapers could be retrieved a year after publication
  31. 31. We need betterempirical studies
  32. 32. Rigorous EvaluationFreely Available Empirical Data Tools and Results Sharing Better Empirical StudiesSoftware Engineering Platform
  33. 33. A software engineering research platform
  34. 34. Ready made tools Formalised data formats Easily extensibleA software engineering research platform Researcher Large scale Pre processed data community procesing
  35. 35. Alitheia Core
  36. 36. Platform
  37. 37. PlatformData
  38. 38. PlatformData Tools
  39. 39. PlatformData Tools Processing
  40. 40. Data Raw Data Metadata Tool ResultsMailing Lists BTS SCM Processed raw data
  41. 41. Mirroring Root / project. Project 1 Project 2 Project 3properties git svn mails bugs Standard Standard GIT SVN format bug<id> format .xml List 1 List 2 tmp cur new messageid.eml
  42. 42. Raw Data Mirroring Root / project. Project 1 Project 2 Project 3properties git svn mails bugs Standard Standard GIT SVN format bug<id> format .xml List 1 List 2 tmp cur new messageid.eml
  43. 43.                                                                                                                                                                                                       
  44. 44. Metadata                                                                                                                                                                                                       
  45. 45. Tools
  46. 46. Tools Metric Metric Plug-in Job Metadata Web Cluster Metric Plug-in Metric Logging Schedul Updater services Service Plug-in Activator er DB Messagi Web Plug-in Parser Data Security Service ng Admin Admin Servic Access eSQO-OSS SQO-OSS SQO-OSS Project 1 PV svn mails bugs PV PV List 1 List 2 PV Project Mirror tmp cur new Metadata Storage
  47. 47. Toolspublic interface AlitheiaPlugin { String getVersion(); String getAuthor(); Date getDateInstalled(); String getName(); String getDescription(); List<Result> getResultIfAlreadyCalculated(DAObject o, List<Metric> l); List<Result> getResult(DAObject o, List<Metric> l); List<Metric> getAllSupportedMetrics(); List<Metric> getSupportedMetrics(Class<? extends DAObject> activationType); void run(DAObject o); boolean update(); boolean install(); boolean remove(); boolean cleanup(DAObject sp); String getUniqueKey(); Set<Class<? extends DAObject>> getActivationTypes(); List<Class<? extends DAObject>> getMetricActivationTypes (Metric m); Set<PluginConfiguration> getConfigurationSchema(); Set<String> getDependencies(); Map<MetricType.Type, SortedSet<Long>> getObjectIdsToSync(StoredProject sp, Metric m);}
  48. 48. Toolspublic interface AlitheiaPlugin { String getVersion(); String getAuthor(); Date getDateInstalled(); String getName(); String getDescription(); List<Result> getResultIfAlreadyCalculated(DAObject o, List<Metric> l); List<Result> getResult(DAObject o, List<Metric> l); List<Metric> getAllSupportedMetrics(); List<Metric> getSupportedMetrics(Class<? extends DAObject> activationType); void run(DAObject o); boolean update(); boolean install(); boolean remove(); boolean cleanup(DAObject sp); String getUniqueKey(); Set<Class<? extends DAObject>> getActivationTypes(); List<Class<? extends DAObject>> getMetricActivationTypes (Metric m); Set<PluginConfiguration> getConfigurationSchema(); Set<String> getDependencies(); Map<MetricType.Type, SortedSet<Long>> getObjectIdsToSync(StoredProject sp, Metric m);}
  49. 49. Tools@MetricDeclarations(metrics = { @MetricDecl(mnemonic="MNOF", activators={ProjectDirectory.class}, descr="Number of Source Code Files in Module"), @MetricDecl(mnemonic="MNOL", activators={ProjectDirectory.class}, descr="Number of lines in module", dependencies={"Wc.loc"}), @MetricDecl(mnemonic="AMS", activators={ProjectVersion.class}, descr="Average Module Size"), @MetricDecl(mnemonic="ISSRCMOD", activators={ProjectDirectory.class}, descr="Mark for modules containing source files")})public class ModuleMetricsImplementation extends AbstractMetric { public void run(ProjectFile pf) throws AlreadyProcessingException {[...]} public void run(ProjectVersion pv) throws AlreadyProcessingException {[...]} public List<Result> getResult(ProjectFile pf, Metric m) { return getResult(pf, ProjectFileMeasurement.class, m, Result.ResultType.INTEGER); } public List<Result> getResult(ProjectVersion pv, Metric m) { return getResult(pv, ProjectVersionMeasurement.class, m, Result.ResultType.FLOAT); }}
  50. 50. public void run(ProjectFile pf) { // We do not support directories Tools if (pf.getIsDirectory()) { return; } InputStream in = fds.getFileContents(pf); if (in == null) { return; } // Create an input stream from the project files content try { // Measure the number of lines in the project file LineNumberReader lnr = new LineNumberReader(new InputStreamReader(in)); int lines = 0; while (lnr.readLine() != null) { lines++; } lnr.close(); // Store the results Metric metric = Metric.getMetricByMnemonic("LOC"); ProjectFileMeasurement locm = new ProjectFileMeasurement(); locm.setMetric(metric); locm.setProjectFile(pf); locm.setWhenRun(new Timestamp(System.currentTimeMillis())); locm.setResult(String.valueOf(lines)); db.addRecord(locm); markEvaluation(metric, pf.getProjectVersion().getProject()); } catch (IOException e) { log.error(this.getClass().getName() + " IO Error <" + e + "> while measuring: " + pf.getFileName()); } }
  51. 51. Processing - The scmmap algorithm CPU
  52. 52. Processing - the idmap algorithm CPU
  53. 53. Processing - in clusters
  54. 54. Line counting speed 60 45Minutes 30 15 0 Naive implementation Alitheia Core
  55. 55. New cluster node connects
  56. 56. Case Studies
  57. 57. Do intense conversationsaffect short-term project development?
  58. 58. How do we identify intense discussions 100000 120000 Number of messages Levels of thread depth 90000 100000 80000 70000 80000 60000 OccurencesOccurences 50000 60000 40000 40000 30000 20000 20000 10000 0 0 0 5 10 15 20 25 0 5 10 15 20 25 Number of messages per thread Thread depth
  59. 59. Hypotheses• H1: Number of messages and thread depth are dependent variables• H2: We can identify intense discussions by identifying threads in top depth and msg/ thread quartiles• H3: Intense discussions affect the repository’s source line intake
  60. 60. Method• Import projects in Alitheia Core• Develop metric plug-in to count the variables we are interested in • 3 metrics• Emails from 60 projects, ~1,2 * 10^6 emails, 679427 threads• Plug-in loc: 270 lines
  61. 61. 100 Fit function (0.55 + 1.59x) 80Number of Messages 60 40 20 0 0 5 10 15 20 25 30 35 40 45 50 Thread depth H1: R^2 = 0.70
  62. 62. 100 Fit function (0.55 + 1.59x) 80Number of Messages 60 40 20 0 0 5 10 15 20 25 30 35 40 45 50 Thread depth H1: R^2 = 0.70
  63. 63. • H2: 99 discussion threads• H3 3: Project Avogadro ΗotEffect 622 Deskbar-Applet -103 FreeBSD -34 Gnome-Network -106 Gnome-Utils -263 GTK+ 183 Sabayon -244 Vala -356 LSR 1150 Meld 27
  64. 64. • H2: 99 discussion threads• H3 3: Project Avogadro ΗotEffect 622 Deskbar-Applet -103 FreeBSD -34 Gnome-Network -106 Gnome-Utils -263 GTK+ 183 Sabayon -244 Vala -356 LSR 1150 Meld 27
  65. 65. • H2: 99 discussion threads• H3 3: Project Avogadro ΗotEffect 622 Deskbar-Applet -103 FreeBSD -34 Gnome-Network -106 Gnome-Utils -263 GTK+ 183 Sabayon -244 Vala -356 LSR 1150 Meld 27
  66. 66. Does the number ofprogrammers affectcode maintainability?
  67. 67. Hypotheses• H1: Number of programmers affects code maintainability at the project level• H2: Number of programmers affects code maintainability at the directory level.
  68. 68. Method• Per langauge • C & Java (risky)• Plug-ins that calculate • Number of developers per period of time • Halstead & McCabe (700 lines) • Omar’s Maintainability Index (240 lines)• Data from 213 projects
  69. 69. Maintainability at the project level R^2 = 0.04
  70. 70. Maintainability at the project level R^2 = 0.04
  71. 71. Maintainability at the module level R^2 = 0.05
  72. 72. Maintainability at the module level R^2 = 0.05
  73. 73. Per language C: R^2 = 0.08Java: R^2 = 0.07
  74. 74. Per language C: R^2 = 0.08Java: R^2 = 0.07
  75. 75. Why do we need large scale research?
  76. 76. Project ΗotEffect Banshee -1121 Deskbar-Applet -103 FreeBSD -34Gnome-Power-Manager -773 Gnome-Utils -263 GTranslator -230 Sabayon -244 Vala -356
  77. 77. 1 R2 0.9 0.8 0.7Correlation Co-efficient 0.6 0.5 0.4 0.3 0.2 0.1 0 R^2 at the module level for all projects
  78. 78. TODO
  79. 79. Replication of published studies
  80. 80. Platform Data validation
  81. 81. Repositories for tools, data and results
  82. 82. The SoftEng cloud project
  83. 83. Thank you!http://www.sqo-oss.org Georgios Gousios gousiosg@aueb.gr

×