Whither Software Engineering Research? (keynote talk at APSEC 2012)

22,424 views

Published on

keynote talk presented at the 19th Asia-Pacific Software Engineering Conference (APSEC 2012), Hong Kong, 6 December 2012

Published in: Technology

Whither Software Engineering Research? (keynote talk at APSEC 2012)

  1. 1. APSEC 2012 Whither Software Engineering Research? ! David S. Rosenblum! School of Computing! National University of Singapore
  2. 2. APSEC 2012 Why This Talk? Recent Events KeynoteTalks Reflecting Broadly on the Field! Carlo Ghezzi
 Reflections on 40+Years of SE Research ObservedThrough ICSE! Lionel Briand
 Useful Software Engineering Research: Leading a Double-Agent Life! Mary Shaw
 Whither Software Engineering Education?! Jeff Kramer
 Whither Software Architecture?
  3. 3. APSEC 2012 Why This Talk? Recent Events I’ll be the new Editor-in-Chief
 of ACM TOSEM beginning
 January 2013
  4. 4. APSEC 2012 0" 20" 40" 60" 80" 100" 120" 140" 36" 12" 32" 39" 41" 3" 21" 22" 24" 28" 10" 17" 7" 38" 34" 27" 16" 1" 18" 23" 14" 19" 25" 42" 30" 13" 31" 33" 6" 2" 20" 8" 9" 11" 26" 5" 29" 43" 40" 15" 4" 35" 37" Count& Topic&Number& ICSE&2013&Submissions&and&Acceptances&by&Topic& Rejected" Accepted" Why This Talk? Concerns of Under-Representation
  5. 5. APSEC 2012 0" 20" 40" 60" 80" 100" 120" 140" 36" 12" 32" 39" 41" 3" 21" 22" 24" 28" 10" 17" 7" 38" 34" 27" 16" 1" 18" 23" 14" 19" 25" 42" 30" 13" 31" 33" 6" 2" 20" 8" 9" 11" 26" 5" 29" 43" 40" 15" 4" 35" 37" Count& Topic&Number& ICSE&2013&Submissions&and&Acceptances&by&Topic& Rejected" Accepted" Why This Talk? Concerns of Under-Representation SoftwareTesting and Analysis Empirical SE
  6. 6. APSEC 2012 0" 20" 40" 60" 80" 100" 120" 140" 36" 12" 32" 39" 41" 3" 21" 22" 24" 28" 10" 17" 7" 38" 34" 27" 16" 1" 18" 23" 14" 19" 25" 42" 30" 13" 31" 33" 6" 2" 20" 8" 9" 11" 26" 5" 29" 43" 40" 15" 4" 35" 37" Count& Topic&Number& ICSE&2013&Submissions&and&Acceptances&by&Topic& Rejected" Accepted" Why This Talk? Concerns of Under-Representation SoftwareTesting and Analysis Empirical SE Architecture and Design SW Processes andWorkflows
  7. 7. APSEC 2012 Characterizing Our Field A Data-Centric Approach 1.All data were taken from the ACM Digital Library! 2.The following data were collected for each paper! • 1998 ACM CCS classifications! • Citation counts! • Abstract texts! 3.The data were cleaned and filtered! 4.The data were analyzed with R, Excel,Wordle
  8. 8. APSEC 2012 CCS Classifications Example
  9. 9. APSEC 2012 CCS Classifications Example
  10. 10. APSEC 2012 CCS Classifications Example
  11. 11. APSEC 2012 CCS Classifications Example
  12. 12. APSEC 2012 Subjects TOSEM! ICSE! FSE (including ESEC/FSE)! SIGMOD! SIGCOMM! CHI! PLDI! ACM Multimedia! ASPLOS! SOSP
  13. 13. APSEC 2012 Questions • What is the distribution of research in SE?! • What is the nature of that distribution?! • How does that distribution evolve over time?! • How does the distribution correlate with impact?! • How does the distribution compare with other fields?
  14. 14. APSEC 2012 What Would an “Ideal” Distribution Look Like? • It’s not reasonable to expect equal representation for all topics in a field! • But it’s probably not “healthy” to have extreme domination by a few topics! • At best, comparisons will be more objective than absolute judgements
  15. 15. APSEC 2012 Analyses • Breadth! • Representativeness! • Evolution! • Impact
  16. 16. APSEC 2012 Breadth ACMMM ASPLOS CHI FSE ICSE PLDI SIGCOMM SIGMOD SOSP TOSEM Additional CCS Primary CCS Count 0 50 100 150 200 250 300
  17. 17. APSEC 2012 Representativeness TOSEM Primary CCS (Top 30) D.2.1 D.2.5 D.2.4 D.2.2 D.2.7 D.2.6 D.2.9 D.2.8 D.2.11 D.2.3 D.1.5 D.2.0 D.3.4 F.3.1 D.3.3 C.2.2 C.2.4 D.1.2 D.2 D.2.13 D.2.m D.3.1 D.3.2 D.4.6 H.3.3 C.2.0 C.3 D.1.1 D.1.7 D.4.7 TOSEM 1992−2012 Primary CCS Count 0 10 20 30 40 50 60 D.2.1: Requirements and Specifications! D.2.5: Testing and Debugging! D.2.4: Software/Program Verification! D.2.2: Design Tools and Techniques! D.2.7: Distribution, Maintenance and Enhancement
  18. 18. APSEC 2012 Representativeness ICSE Primary CCS (Top 30) D.2.2 D.2.5 D.2.4 D.2.1 D.2.9 K.6.3 D.2.7 D.2.6 D.2.11 D.2.8 D.3.2 D.2 D.3.3 D.2.3 D.2.13 D.1.5 D.2.0 D.2.m K.6.1 I.2.2 D.1.3 D.4.7 F.3.1 D.2.12 F.3.2 D.3.4 D.1.2 C.2.4 H.5.2 C.3 ICSE 1976−2011 Primary CCS Count 0 50 100 150 D.2.2: Design Tools and Techniques! D.2.5: Testing and Debugging! D.2.4: Software/Program Verification! D.2.1: Requirements and Specifications! D.2.9: Management
  19. 19. APSEC 2012 Representativeness ICSE Primary CCS (All 124) 1976% 1978% 1979% 1981% 1982% 1984% 1985% 1987% 1988% 1989% 1990% 1992% 1995% 1996% 1997% 1998% 1999% 2000% 2001% 2002% 2003% 2004% 2005% 2006% 2007% 2008% 2009% 2010% 2011% 0.00%% 5.00%% 10.00%% 15.00%% 20.00%% 25.00%% 30.00%% 35.00%% 40.00%% D.2.2% D.2.4% D.2.9% D.2.7% D.2.11% D.3.2% D.3.3% D.2.13% D.2.0% K.6.1% D.1.3% F.3.1% F.3.2% D.1.2% H.5.2% D.1.0% I.2.1% C.2.1% H.2.4% K.6.5% D.3.1% H.4.1% I.2.4% J.7% C.4% E.1% G.4% H.3.5% I.2.6% J.2% C.2.0% D.1% D.1.7% D.4.1% D.4.5% D.4.9% F.3.3% G.2.2% H.2.1% H.2.7% H.4% I.2.11% I.2.8% I.6.1% J.3% K.4.1% K.8.1% B.0% C.0% D.4.3% F.1.1% F.2.2% F.4.2% H.2.8% H.3.3% H.5.4% I.2.7% I.5.3% I.6.3% I.7.0% K.3.0% K.5.0% ICSE%Distribu-on%of%Primary%CCS%Classifica-ons% 35.00%840.00%% 30.00%835.00%% 25.00%830.00%% 20.00%825.00%% 15.00%820.00%% 10.00%815.00%% 5.00%810.00%% 0.00%85.00%%
  20. 20. APSEC 2012 Representativeness SIGMOD and CHI (Top 30) H.2.4 H.2.8 H.2.1 H.3.3 H.2.3 H.2.0 H.3.1 H.2.7 H.2.2 H.2 E.1 H.2.m F.2.2 H.3.5 H.2.5 H.4.m D.4.2 G.2.2 H.3.2 C.4 F.1.2 H.4.2 D.3.4 E.5 F.4.1 G.3 H.0 H.3.4 H.4.0 H.m SIGMOD 1985−2012 Primary CCS Count 0 100 200 300 400 500 H.5.2 H.5.m H.1.2 H.5.3 H.5.1 D.2.2 H.4.3 H.m K.6.1 K.3.1 H.5.4 H.3.3 K.4.2 I.3.6 H.5.0 D.2.6 H.4.1 H.3.5 J.3 I.2.7 I.3.7 I.7.1 D.2.5 I.2.6 J.4 D.3.2 I.2.0 I.2.1 C.5.3 B.4.2 CHI 1981−2012 Primary CCS Count 0 200 400 600 800 1000 1200 Database Management
 Systems User Interfaces
  21. 21. APSEC 2012 Representativeness ACM Multimedia (All 115) 1993$ 1994$ 1995$ 1996$ 1997$ 1998$ 1999$ 2000$ 2001$ 2002$ 2003$ 2004$ 2005$ 2006$ 2007$ 2008$ 2009$ 2010$ 2011$ 0.00%$ 10.00%$ 20.00%$ 30.00%$ 40.00%$ 50.00%$ 60.00%$ 70.00%$ 80.00%$ 90.00%$ H.5.1$ H.3.1$ C.2.4$ H.5.5$ I.4.9$ H.4.3$ I.4.8$ H.2.4$ I.4.6$ E.4$ C.3$ H.3$ I.2.7$ J.3$ I.5.3$ H.5.0$ I.5.4$ I.5.1$ K.3.1$ I.4.1$ H.4$ D.4.7$ D.4.3$ D.4.1$ H.5$ I.3.5$ H.2.0$ H.3.0$ C.2.0$ I.4.0$ I.4.4$ I.4$ I.2.6$ D.2.11$ J.0$ D.1.3$ H.4.0$ J.7$ I.3.1$ H.2.3$ I.4.5$ H.4.2$ D.2.6$ H.2.7$ F.1.2$ I.5.2$ G.0$ D.4.9$ C.5.3$ D.2.m$ D.2$ K.6$ I.5$ J.2$ B.8.1$ I.5.0$ I.2.0$ J.4$ ACMMM$Distribu,on$of$Primary$CCS$Classifica,ons$ 80.00%990.00%$ 70.00%980.00%$ 60.00%970.00%$ 50.00%960.00%$ 40.00%950.00%$ 30.00%940.00%$ 20.00%930.00%$ 10.00%920.00%$ 0.00%910.00%$ Multimedia Information Systems
  22. 22. APSEC 2012 Representativeness Skewness of Topic Distributions ACMMM ASPLOS CHI FSE ICSE PLDI SIGCOMM SIGMOD SOSP TOSEM 0 2 4 6 8 10 12 AnnualSkewness
  23. 23. APSEC 2012 Representativeness Linear Regression of Annual Skewness FSE:!! ! ! 0.1079! ICSE:! ! ! 0.01419! SIGMOD:! ! 0.01232! CHI:! ! ! 0.009678! SOSP:! ! ! 0.003635! TOSEM:! ! -0.00808! SIGCOMM:!! -0.02861! PLDI:! ! ! -0.03839! ASPLOS:! ! -0.04037! ACMMM:! ! -0.3977! positive slope indicates narrowing, negative slope indicates broadening
  24. 24. APSEC 2012 Evolution Abstracts: ICSE 1976 and 2011
  25. 25. APSEC 2012 Evolution Abstracts: ICSE 1976 and 2011
  26. 26. APSEC 2012 Evolution Abstracts: SIGCOMM 1977 and 2012
  27. 27. APSEC 2012 Evolution Abstracts: SIGCOMM 1977 and 2012
  28. 28. APSEC 2012 Evolution Abstracts: SIGCOMM 1977 and 2012
  29. 29. APSEC 2012 Evolution Abstracts: SIGCOMM 1977 and 2012
  30. 30. APSEC 2012 Evolution Cosine Distance Cosine Distance between Oldest and Newest Abstracts! ICSE:! ! ! 0.7037315! CHI:! ! ! 0.6560927! SIGMOD:!! 0.6549907! PLDI:! ! ! 0.6281822! ACMMM:! ! 0.5951031! FSE:! ! ! 0.5917774! SOSP:! ! ! 0.5134387! ASPLOS:! ! 0.4987756! SIGCOMM: ! 0.4974465! TOSEM: ! ! 0.4697425
  31. 31. APSEC 2012 Impact Topics and Citations ● ● ● ●● ● ● ● ● ● Mean Citations per Paper RankCorrelation:TopicPopularityvsCitations 0 10 20 30 40 50 −1.0−0.50.00.51.0 ACMMM ASPLOS CHI FSE ICSE PLDI SIGCOMM SIGMOD SOSP TOSEM
  32. 32. APSEC 2012 The Questions Revisited • What is the distribution of research in SE?! • What is the nature of that distribution?! Very broad, but consistently skewed towards Specification,Testing and Debugging! • How does that distribution evolve over time?! Narrowing for ICSE and FSE, broadening forTOSEM
  33. 33. APSEC 2012 The Questions Revisited • How does the distribution correlate with impact?! Topic popularity correlates with higher citations! ! • How does the distribution compare with other fields?! Some are narrower, some are broader
  34. 34. APSEC 2012 Conclusion • Like many empirical studies in software engineering, these results are inconclusive! • Software engineering research appears to be healthy! • Other fields may be in worse shape! ➡! Should we be doing better?
 ! Can we do better?
 ! If so, then how?
  35. 35. APSEC 2012 Acknowledgments Prem Devanbu! Vladimir Filkov! !
  36. 36. APSEC 2012 Whither Software Engineering Research? ! David S. Rosenblum! School of Computing! National University of Singapore ThankYou!

×