keynote talk presented at the 19th Asia-Pacific Software Engineering Conference (APSEC 2012), Hong Kong, 6 December 2012

1. APSEC 2012
Whither Software
Engineering Research?
!
David S. Rosenblum!
School of Computing!
National University of Singapore

2. APSEC 2012
Why This Talk?
Recent Events
KeynoteTalks Reflecting Broadly on the Field!
Carlo Ghezzi
Reflections on 40+Years of SE Research ObservedThrough ICSE!
Lionel Briand
Useful Software Engineering Research: Leading a Double-Agent Life!
Mary Shaw
Whither Software Engineering Education?!
Jeff Kramer
Whither Software Architecture?

3. APSEC 2012
Why This Talk?
Recent Events
I’ll be the new
Editor-in-Chief
of ACM TOSEM
beginning
January 2013

4. APSEC 2012
0"
20"
40"
60"
80"
100"
120"
140"
36" 12" 32" 39" 41" 3" 21" 22" 24" 28" 10" 17" 7" 38" 34" 27" 16" 1" 18" 23" 14" 19" 25" 42" 30" 13" 31" 33" 6" 2" 20" 8" 9" 11" 26" 5" 29" 43" 40" 15" 4" 35" 37"
Count&
Topic&Number&
ICSE&2013&Submissions&and&Acceptances&by&Topic&
Rejected"
Accepted"
Why This Talk?
Concerns of Under-Representation

5. APSEC 2012
0"
20"
40"
60"
80"
100"
120"
140"
36" 12" 32" 39" 41" 3" 21" 22" 24" 28" 10" 17" 7" 38" 34" 27" 16" 1" 18" 23" 14" 19" 25" 42" 30" 13" 31" 33" 6" 2" 20" 8" 9" 11" 26" 5" 29" 43" 40" 15" 4" 35" 37"
Count&
Topic&Number&
ICSE&2013&Submissions&and&Acceptances&by&Topic&
Rejected"
Accepted"
Why This Talk?
Concerns of Under-Representation
SoftwareTesting and Analysis
Empirical SE

6. APSEC 2012
0"
20"
40"
60"
80"
100"
120"
140"
36" 12" 32" 39" 41" 3" 21" 22" 24" 28" 10" 17" 7" 38" 34" 27" 16" 1" 18" 23" 14" 19" 25" 42" 30" 13" 31" 33" 6" 2" 20" 8" 9" 11" 26" 5" 29" 43" 40" 15" 4" 35" 37"
Count&
Topic&Number&
ICSE&2013&Submissions&and&Acceptances&by&Topic&
Rejected"
Accepted"
Why This Talk?
Concerns of Under-Representation
SoftwareTesting and Analysis
Empirical SE
Architecture and Design
SW Processes andWorkflows

7. APSEC 2012
Characterizing Our Field
A Data-Centric Approach
1.All data were taken from the ACM Digital
Library!
2.The following data were collected for each paper!
• 1998 ACM CCS classifications!
• Citation counts!
• Abstract texts!
3.The data were cleaned and filtered!
4.The data were analyzed with R, Excel,Wordle

8. APSEC 2012
CCS Classifications
Example

9. APSEC 2012
CCS Classifications
Example

10. APSEC 2012
CCS Classifications
Example

11. APSEC 2012
CCS Classifications
Example

12. APSEC 2012
Subjects
TOSEM!
ICSE!
FSE (including ESEC/FSE)!
SIGMOD!
SIGCOMM!
CHI!
PLDI!
ACM Multimedia!
ASPLOS!
SOSP

13. APSEC 2012
Questions
• What is the distribution of research in SE?!
• What is the nature of that distribution?!
• How does that distribution evolve over time?!
• How does the distribution correlate with impact?!
• How does the distribution compare with other fields?

14. APSEC 2012
What Would an “Ideal”
Distribution Look Like?
• It’s not reasonable to expect equal
representation for all topics in a field!
• But it’s probably not “healthy” to have
extreme domination by a few topics!
• At best, comparisons will be more objective
than absolute judgements

15. APSEC 2012
Analyses
• Breadth!
• Representativeness!
• Evolution!
• Impact

16. APSEC 2012
Breadth
ACMMM
ASPLOS
CHI
FSE
ICSE
PLDI
SIGCOMM
SIGMOD
SOSP
TOSEM
Additional CCS
Primary CCS
Count
0
50
100
150
200
250
300

17. APSEC 2012
Representativeness
TOSEM Primary CCS (Top 30)
D.2.1
D.2.5
D.2.4
D.2.2
D.2.7
D.2.6
D.2.9
D.2.8
D.2.11
D.2.3
D.1.5
D.2.0
D.3.4
F.3.1
D.3.3
C.2.2
C.2.4
D.1.2
D.2
D.2.13
D.2.m
D.3.1
D.3.2
D.4.6
H.3.3
C.2.0
C.3
D.1.1
D.1.7
D.4.7
TOSEM 1992−2012
Primary CCS
Count
0
10
20
30
40
50
60
D.2.1: Requirements and
Specifications!
D.2.5: Testing and Debugging!
D.2.4: Software/Program
Verification!
D.2.2: Design Tools and Techniques!
D.2.7: Distribution, Maintenance
and Enhancement

18. APSEC 2012
Representativeness
ICSE Primary CCS (Top 30)
D.2.2
D.2.5
D.2.4
D.2.1
D.2.9
K.6.3
D.2.7
D.2.6
D.2.11
D.2.8
D.3.2
D.2
D.3.3
D.2.3
D.2.13
D.1.5
D.2.0
D.2.m
K.6.1
I.2.2
D.1.3
D.4.7
F.3.1
D.2.12
F.3.2
D.3.4
D.1.2
C.2.4
H.5.2
C.3
ICSE 1976−2011
Primary CCS
Count
0
50
100
150
D.2.2: Design Tools and
Techniques!
D.2.5: Testing and Debugging!
D.2.4: Software/Program
Verification!
D.2.1: Requirements and
Specifications!
D.2.9: Management

19. APSEC 2012
Representativeness
ICSE Primary CCS (All 124)
1976%
1978%
1979%
1981%
1982%
1984%
1985%
1987%
1988%
1989%
1990%
1992%
1995%
1996%
1997%
1998%
1999%
2000%
2001%
2002%
2003%
2004%
2005%
2006%
2007%
2008%
2009%
2010%
2011%
0.00%%
5.00%%
10.00%%
15.00%%
20.00%%
25.00%%
30.00%%
35.00%%
40.00%%
D.2.2%
D.2.4%
D.2.9%
D.2.7%
D.2.11%
D.3.2%
D.3.3%
D.2.13%
D.2.0%
K.6.1%
D.1.3%
F.3.1%
F.3.2%
D.1.2%
H.5.2%
D.1.0%
I.2.1%
C.2.1%
H.2.4%
K.6.5%
D.3.1%
H.4.1%
I.2.4%
J.7%
C.4%
E.1%
G.4%
H.3.5%
I.2.6%
J.2%
C.2.0%
D.1%
D.1.7%
D.4.1%
D.4.5%
D.4.9%
F.3.3%
G.2.2%
H.2.1%
H.2.7%
H.4%
I.2.11%
I.2.8%
I.6.1%
J.3%
K.4.1%
K.8.1%
B.0%
C.0%
D.4.3%
F.1.1%
F.2.2%
F.4.2%
H.2.8%
H.3.3%
H.5.4%
I.2.7%
I.5.3%
I.6.3%
I.7.0%
K.3.0%
K.5.0%
ICSE%Distribu-on%of%Primary%CCS%Classifica-ons%
35.00%840.00%%
30.00%835.00%%
25.00%830.00%%
20.00%825.00%%
15.00%820.00%%
10.00%815.00%%
5.00%810.00%%
0.00%85.00%%

20. APSEC 2012
Representativeness
SIGMOD and CHI (Top 30)
H.2.4
H.2.8
H.2.1
H.3.3
H.2.3
H.2.0
H.3.1
H.2.7
H.2.2
H.2
E.1
H.2.m
F.2.2
H.3.5
H.2.5
H.4.m
D.4.2
G.2.2
H.3.2
C.4
F.1.2
H.4.2
D.3.4
E.5
F.4.1
G.3
H.0
H.3.4
H.4.0
H.m
SIGMOD 1985−2012
Primary CCS
Count
0
100
200
300
400
500
H.5.2
H.5.m
H.1.2
H.5.3
H.5.1
D.2.2
H.4.3
H.m
K.6.1
K.3.1
H.5.4
H.3.3
K.4.2
I.3.6
H.5.0
D.2.6
H.4.1
H.3.5
J.3
I.2.7
I.3.7
I.7.1
D.2.5
I.2.6
J.4
D.3.2
I.2.0
I.2.1
C.5.3
B.4.2
CHI 1981−2012
Primary CCS
Count
0
200
400
600
800
1000
1200
Database Management
Systems
User Interfaces

21. APSEC 2012
Representativeness
ACM Multimedia (All 115)
1993$
1994$
1995$
1996$
1997$
1998$
1999$
2000$
2001$
2002$
2003$
2004$
2005$
2006$
2007$
2008$
2009$
2010$
2011$
0.00%$
10.00%$
20.00%$
30.00%$
40.00%$
50.00%$
60.00%$
70.00%$
80.00%$
90.00%$
H.5.1$
H.3.1$
C.2.4$
H.5.5$
I.4.9$
H.4.3$
I.4.8$
H.2.4$
I.4.6$
E.4$
C.3$
H.3$
I.2.7$
J.3$
I.5.3$
H.5.0$
I.5.4$
I.5.1$
K.3.1$
I.4.1$
H.4$
D.4.7$
D.4.3$
D.4.1$
H.5$
I.3.5$
H.2.0$
H.3.0$
C.2.0$
I.4.0$
I.4.4$
I.4$
I.2.6$
D.2.11$
J.0$
D.1.3$
H.4.0$
J.7$
I.3.1$
H.2.3$
I.4.5$
H.4.2$
D.2.6$
H.2.7$
F.1.2$
I.5.2$
G.0$
D.4.9$
C.5.3$
D.2.m$
D.2$
K.6$
I.5$
J.2$
B.8.1$
I.5.0$
I.2.0$
J.4$
ACMMM$Distribu,on$of$Primary$CCS$Classifica,ons$
80.00%990.00%$
70.00%980.00%$
60.00%970.00%$
50.00%960.00%$
40.00%950.00%$
30.00%940.00%$
20.00%930.00%$
10.00%920.00%$
0.00%910.00%$
Multimedia Information Systems

22. APSEC 2012
Representativeness
Skewness of Topic Distributions
ACMMM
ASPLOS
CHI
FSE
ICSE
PLDI
SIGCOMM
SIGMOD
SOSP
TOSEM
0
2
4
6
8
10
12
AnnualSkewness

23. APSEC 2012
Representativeness
Linear Regression of Annual Skewness
FSE:!! ! ! 0.1079!
ICSE:! ! ! 0.01419!
SIGMOD:! ! 0.01232!
CHI:! ! ! 0.009678!
SOSP:! ! ! 0.003635!
TOSEM:! ! -0.00808!
SIGCOMM:!! -0.02861!
PLDI:! ! ! -0.03839!
ASPLOS:! ! -0.04037!
ACMMM:! ! -0.3977!
positive slope indicates narrowing, negative slope indicates broadening

24. APSEC 2012
Evolution
Abstracts: ICSE 1976 and 2011

25. APSEC 2012
Evolution
Abstracts: ICSE 1976 and 2011

26. APSEC 2012
Evolution
Abstracts: SIGCOMM 1977 and 2012

27. APSEC 2012
Evolution
Abstracts: SIGCOMM 1977 and 2012

28. APSEC 2012
Evolution
Abstracts: SIGCOMM 1977 and 2012

29. APSEC 2012
Evolution
Abstracts: SIGCOMM 1977 and 2012

30. APSEC 2012
Evolution
Cosine Distance
Cosine Distance between Oldest and Newest Abstracts!
ICSE:! ! ! 0.7037315!
CHI:! ! ! 0.6560927!
SIGMOD:!! 0.6549907!
PLDI:! ! ! 0.6281822!
ACMMM:! ! 0.5951031!
FSE:! ! ! 0.5917774!
SOSP:! ! ! 0.5134387!
ASPLOS:! ! 0.4987756!
SIGCOMM: ! 0.4974465!
TOSEM: ! ! 0.4697425

31. APSEC 2012
Impact
Topics and Citations
●
●
●
●●
●
●
●
●
●
Mean Citations per Paper
RankCorrelation:TopicPopularityvsCitations
0 10 20 30 40 50
−1.0−0.50.00.51.0
ACMMM
ASPLOS
CHI
FSE
ICSE
PLDI
SIGCOMM
SIGMOD
SOSP
TOSEM

32. APSEC 2012
The Questions
Revisited
• What is the distribution of research in SE?!
• What is the nature of that distribution?!
Very broad, but consistently skewed towards
Specification,Testing and Debugging!
• How does that distribution evolve over time?!
Narrowing for ICSE and FSE, broadening forTOSEM

33. APSEC 2012
The Questions
Revisited
• How does the distribution correlate with impact?!
Topic popularity correlates with higher citations!
!
• How does the distribution compare with other fields?!
Some are narrower, some are broader

34. APSEC 2012
Conclusion
• Like many empirical studies in software
engineering, these results are inconclusive!
• Software engineering research appears to be
healthy!
• Other fields may be in worse shape!
➡! Should we be doing better?
! Can we do better?
! If so, then how?

35. APSEC 2012
Acknowledgments
Prem Devanbu!
Vladimir Filkov!
!

36. APSEC 2012
Whither Software
Engineering Research?
!
David S. Rosenblum!
School of Computing!
National University of Singapore
ThankYou!