Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Fact or Fiction? What Software Analytics Can Do For Us

636 views

Published on

My SPLASH-I talk @splashcon 2016 in Amsterdam, the Netherlands

Published in: Science
  • Be the first to comment

Fact or Fiction? What Software Analytics Can Do For Us

  1. 1. Fact or Fiction? What Software Analytics Can Do For Us Andy Zaidman Delft University of Technology SPLASH-I 2016, Amsterdam azaidman
  2. 2. I like to tell stories… Andy
  3. 3. EVERYBODY LIES
  4. 4. EVERYBODY LIES
  5. 5. EVERYBODY Andy Zaidman
  6. 6. ANDY LIES
  7. 7. EVERYBODY Andy Zaidman Moritz Beller Georgios Gousios Annibale Panichella Niels Spruit
  8. 8. Let’s consider software testing
  9. 9. ANDY’S LIE Grant proposal “It is well-known that testing consumes 50% of a developer’s time”
  10. 10. “it depends” “50% of your time”
  11. 11. What is the (likely) origin of the 50%?
  12. 12. The Mythical Man-Month? Brooks “observed” that testing and debugging took 50% of the time How? “Calendar research” When ± 40 years ago…
  13. 13. ANDY’S LIE FACTS Grant proposal “It is well-known that testing consumes 50% of a developer’s time [1]”
  14. 14. Chinese whispers / telephone game
  15. 15. “Truth begins in lies” -House
  16. 16. Lies begin in truth “Truth begins in lies” -House
  17. 17. Folklore/evidence test approximately 50% Lack of knowledge How much time is actually spent on testing?
  18. 18. Research Tool Software Analytics “Big Brother” in your IDE  measures testing activities • 416 Softw. Engineers, 68 countries • 5 month observation period • Java / Eclipse • 13.7 man years of development observed Immediate feedback for developers
  19. 19. WatchDog Pre-Test Questionnaire Can you estimate how much time you spend on engineering test code versus production code? 50% 50%test code production code 30% 70%test code production code
  20. 20. WatchDog Pre-Test Questionnaire Can you estimate how much time you spend on engineering test code versus production code? 48% - 52%test code production code
  21. 21. After measuring ≥ 3 months 25% - 75%test code engineering Most participants overestimate their test engineering activities production code engineering
  22. 22. Not all Java projects do unit testing Likely, an underestimation…
  23. 23. No test engineering For projects with test code: 53% of developer did not execute, read, or modify a single test within five months.
  24. 24. Do sets of unit test executions take… < 5 seconds ≥ 5 seconds
  25. 25. Swift tests Execution of set of unit tests takes 0.54 seconds (median).
  26. 26. Ad hoc test selection Developers frequently select a specific set of tests to run in the IDE. In most cases, developers execute one single test.
  27. 27. Test execution in the IDE is rare In 85% of development sessions no tests were run…
  28. 28. Tests fail 65% of test executions in the IDE fail.
  29. 29. Test Driven Development TDD is not widely practiced. Programmers who claim to do it, neither follow it strictly nor for all their modifications.
  30. 30. • Be critical: are long-standing beliefs/insights/claims • Still valid? • Valid in your context? • Triangulate • Developer opinions deviate greatly from what we objectively measured • Mixed-method research • Longitudinal studies
  31. 31. • Estimating is difficult • Estimating something that you don’t like is more difficult • Beliefs are to be questioned all the time
  32. 32. Differential diagnosis - Tests are written, but not run - Tests are written, but run on Command Line - Tests are written, but run in Continuous Integration
  33. 33. Differential diagnosis - Tests are written, but not run - (Tests are written, but run on Command Line) - Tests are written, but run in Continuous Integration
  34. 34. • 1,359 Java and Ruby projects • 2,640,825 builds
  35. 35. Testing in Continuous Integration In our population of 1,359 Java and Ruby projects
  36. 36. Builds with at least one failing test From 1,108 projects which executed tests
  37. 37. Testing #1 reason for build to break From 1,108 projects which executed tests Test failure Other failure
  38. 38. Tests are decisive for overall build From 1,108 projects which executed tests 98.3% projects Test fails  Build breaks
  39. 39. Integrating in multiple environments From 1,108 projects which executed tests 60%projects Different build result for at least one build in multi-environment setup
  40. 40. Differential diagnosis - Tests are written, but not run - (Tests are written, but run on Command Line) - Tests are written, but run in Continuous Integration - 60% failing tests in the IDE - <20% failing tests in CI - Command Line?
  41. 41. ANDY’S LIES 1. 50% of time goes to testing 2. This talk was on software analytics
  42. 42. Empirical Software Engineering Software Analytics
  43. 43. Software analytics = bait / intrinsic motivation
  44. 44. From an experienced developer… “Estimated time working on tests: 20%, actual time: 6%. Cool statistics! ” Testing analytics helps developers to get a better understanding about their own development behavior!
  45. 45. Software analytics Empowers software engineers, software teams, managers Make better tools
  46. 46. Debugging
  47. 47. WatchDog 2.0 We are looking into how developers debug Product improvement: In contact with Eclipse + IntelliJ New features coming to Eclipse based on our analytics Educational improvement: We need to teach our students to debug more efficiently (with the IDE debugger)
  48. 48. What to expect?
  49. 49. Software Analytics Challenges for the SPLASH community • What works in languages? What doesn’t? • How should we design the next version of languages? • How should we design the next generation of tools? • How should we improve education?
  50. 50. Software engineering folklore Perception w.r.t. dev. testing Measured developer testing 48% - 52% 25% - 75% Software analytics help engineers to better understand their own behavior azaidman WatchDogTeam Challenges for SPLASH
  51. 51. Pointers to literature 1. Moritz Beller, Georgios Gousios, Andy Zaidman: How (Much) Do Developers Test? ICSE 2015: 559-562 2. Moritz Beller, Georgios Gousios, Annibale Panichella, Andy Zaidman: When, how, and why developers (do not) test in their IDEs. ESEC/SIGSOFT FSE 2015: 179-190 3. Moritz Beller, Georgios Gousios, Andy Zaidman: Oops, my tests broke the build: An analysis of Travis CI builds with GitHub. PeerJ PrePrints 4: e1984 4. Fred Brooks: The Mythical Man-Month, Addison-Wesley, 1975 5. Laurent Bossavit: The Leprechauns of Software Engineering, 2012- 2014, https://leanpub.com/leprechauns

×