We Know Less Than You Think (But We Do Know Something)


Published on

A quick summary of empirical software engineering, followed by early results from a survey of how over 1900 scientists use computers in their research.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

We Know Less Than You Think (But We Do Know Something)

  1. 1. We Know Less Than You Think (But We Do Know Something) Copyright © 2009 Gregory V. Wilson. This presentation may be freely used and distributed with attribution. Prof. Greg Wilson Dept. of Computer Science University of Toronto [email_address]
  2. 2. Computer “Science” (?) <ul><li>Some of computer science is applied mathematics </li></ul><ul><li>But what kind of “science” is software engineering? </li></ul><ul><ul><li>Most of it isn’t (yet?) as rigorous as bridge design </li></ul></ul><ul><ul><li>Practitioners would object to being called alchemists, organizational psychologists, or cheerleaders </li></ul></ul>http://www.third-bit.com 00001
  3. 3. Show Me <ul><li>Shaw 1990: “Prospects for a Discipline of Software Engineering” </li></ul><ul><ul><li>Called for evidence-based software engineering </li></ul></ul><ul><ul><li>Built on empirical work since the 1970s by Boehm, Basili, Glass, and others </li></ul></ul><ul><li>Empirical studies now expected by top journals and conferences </li></ul>http://www.third-bit.com 00010
  4. 4. There’s a Lot of Folklore… <ul><li>“ The best programmers are up to 28 times more productive than the worst.” (or 40, or 100, or…) </li></ul><ul><ul><li>Trace it back: [SAC68] had 12 (self-taught) subjects </li></ul></ul><ul><li>[BOE75]: best to worst is a factor of 5 </li></ul><ul><ul><li>Borne out by other studies </li></ul></ul><ul><ul><li>Consistent with other fields </li></ul></ul>http://www.third-bit.com 00011
  5. 5. Non-Linearity <ul><li>[WOO79]: 25% increase in problem complexity  100% increase in solution complexity </li></ul><ul><ul><li>Interactions grow as N 2 </li></ul></ul><ul><li>This can work in your favor </li></ul><ul><ul><li>Cut 1/4 of your features, problem because 1/2 as hard to solve </li></ul></ul>http://www.third-bit.com 00100
  6. 6. Glass’s Law <ul><li>Any new tool or technique initially makes you less productive </li></ul><ul><ul><li>Which is why so many senior students program with Notepad </li></ul></ul><ul><li>[GLA99]: 5-35% improvement is the best you can expect from changing tools </li></ul><ul><ul><li>Ah, but techniques… </li></ul></ul>http://www.third-bit.com 00101
  7. 7. Project Failure <ul><li>#1 cause: poor estimation </li></ul><ul><ul><li>Short iterations cure this in the small, but don’t help in the large </li></ul></ul><ul><li>#2 cause: unstable requirements </li></ul><ul><ul><li>But research code often doesn’t have “requirements” </li></ul></ul><ul><li>See [VGE91], [COL95], [HOF01] </li></ul>http://www.third-bit.com 00110
  8. 8. Boehm’s Curve <ul><li>[BOE01] and many others </li></ul>http://www.third-bit.com 00111
  9. 9. Re-Use <ul><li>“ It takes three times longer to build a reusable component than it does to build something used only once.” </li></ul><ul><li>“ If more than 25% of a component has to be revised, it’s better to rewrite it from scratch.” </li></ul><ul><li>See e.g. [THO97] </li></ul>http://www.third-bit.com 01000
  10. 10. Lines of Code <ul><li>“ Productivity and reliability depend on the length of a program’s text, independent of language level.” </li></ul><ul><li>See [PRE00] for a recent experiment </li></ul><ul><li>However, machine-independent code has machine-independent performance </li></ul>http://www.third-bit.com 01001
  11. 11. <ul><li>“ Maintenance makes up 40-80% of the total cost of a software project.” </li></ul><ul><ul><li>[BOE75] and many more since </li></ul></ul><ul><li>But roughly 60% of “maintenance” is enhancement </li></ul><ul><li>And 30% of maintenance time is spent figuring out things work </li></ul>Maintenance http://www.third-bit.com 01010
  12. 19. <ul><li>Lack of documentation was the #1 complaint for over 40% of respondents, and in the top three for almost 80% </li></ul><ul><li>… but ranked second to last in list of where respondents spend their time </li></ul>What Hurts? http://www.third-bit.com 01010
  13. 20. <ul><li>Emphasis on grids and GPUs is inappropriate and unproductive </li></ul><ul><ul><li>Improving skills will yield greater dividends on shorter timescales </li></ul></ul><ul><li>swc.scipy.org for current course </li></ul><ul><ul><li>software-carpentry.org for the next one </li></ul></ul><ul><ul><li>We’ll be webcasting July 13-31 </li></ul></ul>Thoughts http://www.third-bit.com 01011
  14. 21. References <ul><li>[BOE75] Boehm: “The High Cost of Software” </li></ul><ul><li>[COL95] Cole: “Runaway Projects—Causes and Effects” </li></ul><ul><li>[GLA99] Glass: “The Realities of Software Technology Payoffs” </li></ul><ul><li>[HOF01] Hofmann and Lehner: “Requirements Engineering as a Success Factor in Software Projects” </li></ul><ul><li>[PRE00] Prechelt: “An Empirical Comparison of Seven Programming Languages” </li></ul><ul><li>[SAC68] Sackman et al: “Exploratory Experimental Studies Comparing Online and Offline Programming Performances” </li></ul><ul><li>[THO97] Thomas et al: “An Analysis of Errors in a Reuse-Oriented Development Environment” </li></ul><ul><li>[VGE91] Van Genuchten: “Why Is Software Late?” </li></ul><ul><li>[WOO79] Woodfield: “An Experiment on Unit Increase in Problem Complexity” </li></ul>http://www.third-bit.com 01100
  15. 22. Recommended Reading <ul><li>Doar Practical Development Environments </li></ul><ul><li>Feathers Working Effectively with Legacy Code </li></ul><ul><li>Fogel Producing Open Source Software </li></ul><ul><li>Ford The Productive Programmer </li></ul><ul><li>Glass Facts and Fallacies of Software Engineering </li></ul><ul><li>Johnson GUI Bloopers </li></ul><ul><li>Kernighan almost anything </li></ul><ul><li>Kniberg Scrum and XP From the Trenches </li></ul><ul><li>Mak The Martian Principles </li></ul><ul><li>Nygard Release It! </li></ul><ul><li>Spinellis Code Reading and Code Quality </li></ul><ul><li>Weiner Time, Love, Memory </li></ul>01101 http://www.third-bit.com