Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Software Analytics: The Dark Side and the Test Side


Published on

Presentation on developer testing given at the 2016 Dutch Software Engineering Symposium #sensym.

Published in: Science
  • Be the first to comment

Software Analytics: The Dark Side and the Test Side

  1. 1. Software Analytics: The Dark Side and the Test Side SEN Symposium 2016 Andy Zaidman, TU Delft
  2. 2. I like to tell stories… Andy
  3. 3. Jurgen/Frank: Can you give a keynote on software testing and software analytics at the SEN Symposium? Me: Sure I can, I’m honored by the invitation. Jurgen/Frank: Can you give us a title and abstract? Me: <<PANIC>> what should I talk about? Subject so broad... ARRGHHH...
  4. 4. 1. “Software Engineering” 2. “Software Testing” 3. “Software Analytics” I started by asking
  5. 5. I turned to again
  6. 6. The Dark Side @ work (Dec. 2015)
  7. 7.
  8. 8. Ba d
  9. 9.
  10. 10. vs. 3 - 0
  11. 11. Bring balance, you must! Test more, you should!
  12. 12. “it depends” “50% of your time”
  13. 13. What is the (likely) origin of the 50%?
  14. 14. The Mythical Man-Month? Brooks “observed” that testing and debugging took 50% of the time How? “Calendar research” When ± 40 years ago…
  15. 15. Chinese whispers / telephone game
  16. 16. Folklore/evidence test approximately 50% Lack of knowledge How much time is actually spent on testing?
  17. 17. NWO Vidi project
  18. 18. Research Tool Software Analytics “Big Brother” in your IDE  measures testing activities • 416 Softw. Engineers, 68 countries • Java / Eclipse • 13.7 man years of development observed Immediate feedback for developers
  19. 19. WatchDog Pre-Test Questionnaire Can you estimate how much time you spend on engineering test code versus production code? 50% 50%test code production code 30% 70%test code production code
  20. 20. WatchDog Pre-Test Questionnaire Can you estimate how much time you spend on engineering test code versus production code? 48% - 52%test code production code
  21. 21. After measuring ≥ 3 months 25% - 75%test code engineering Most participants overestimate their test engineering activities production code engineering
  22. 22. Not all Java projects do unit testing Likely, an underestimation…
  23. 23. No test engineering For projects with unit test code: 53% of developer did not execute, read, or modify a single test within five months.
  24. 24. Do sets of unit test executions take… < 5 seconds ≥ 5 seconds
  25. 25. Swift tests Execution of set of unit tests takes 0.54 seconds (median).
  26. 26. Ad hoc test selection Developers frequently select a specific set of tests to run in the IDE. In most cases, developers execute one single test.
  27. 27. Tests fail 65% of test executions in the IDE fail.
  28. 28. Test Driven Development TDD is not widely practiced. Programmers who claim to do it, neither follow it strictly nor for all their modifications.
  29. 29. Software Analytics? What about
  30. 30. Testing Analytics? What about
  31. 31. Software analytics = bait / intrinsic motivation
  32. 32. From an experienced developer… “Estimated time working on tests: 20%, actual time: 6%. Cool statistics! ” Testing analytics helps developers to get a better understanding about their own development behavior
  33. 33. Hawthorne effect?
  34. 34. Hawthorne effect? “Keeping up appearances” > 3 months?
  35. 35. Takeaways
  36. 36. Triangulate your results  mixed methods research Be critical towards “folklore” Core might be true, but is it (still) valid, valid in your context, etc. Don’t talk testing, but act testing Takeaways
  37. 37. Critique? • Only developer testing ( ≠ Brooks) • Only Java, only Eclipse • Only 416 developers • Only inside the IDE Extending to IntelliJ and > 1500 developers Also looking into Continuous Integration
  38. 38. Software engineering folklore Perception w.r.t. dev. testing Measured developer testing 48% - 52% 25% - 75% Software analytics help engineers to better understand their own behavior azaidman WatchDog Team
  39. 39. References • M. Beller, G. Gousios, A. Panichella, A. Zaidman. When, How and Why Developers (Do Not) Test in Their IDEs. Proceedings of the Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 179-190, 2015.