Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
When, How, and Why Developers
(Do Not) Test in Their IDEs
When, How, and Why Developers
(Do Not) Test in Their IDEs
Moritz Beller
@Inventitech
Image: https://en.wikipedia.org/wiki/Delft, CC-BY
Image: https://en.wikipedia.org/wiki/Delft, CC-BY
Image: https://en.wikipedia.org/wiki/Delft, CC-BY
Image: https://en.wikipedia.org/wiki/Province_of_Bergamo, CC-BY
Image: http://www.scarymommy.com/angry-baby/
Image: http://www.scarymommy.com/angry-baby/
I am one of
you!
Image: http://www.scarymommy.com/angry-baby/
Source: Moeez, https://commons.wikimedia.org/wiki/File:Thank_You!.jpg, CC-BY
Image: http://www.jccschools.com/news/what_s_new/district_testing_descriptions
Developer Writes Tests
Image: http://www.jccschools.com/news/what_s_new/district_testing_descriptions
Developer Writes Tests Mine Repos for Tests
Image: http://www.jccschools.com/news/what_s_new/district_testing_descriptions
Developer Writes Tests Mine Repos for Tests
Image: http://www.jccschools.com/news/what_s_new/district_testing_descriptions
?
Developer Writes Tests Mine Repos for Tests
Image: http://www.jccschools.com/news/what_s_new/district_testing_descriptio...
RQ1. How Do Developers Test?
RQ1. How Do Developers Test?
RQ2. How Do Developers Manage Failing Tests?
RQ1. How Do Developers Test?
RQ3. Do Developers Follow Test-Driven Development?
RQ2. How Do Developers Manage Failing Test...
RQ1. How Do Developers Test?
RQ3. Do Developers Follow Test-Driven Development?
RQ4. How Much Do Developers Test?
RQ2. How...
IDE!
IDE!
IDE!
Image: https://www.ckras.com/EPS-Entry-GameServer-mieten
Image: https://www.ckras.com/EPS-Entry-GameServer-mieten
→ Interval
→ Edit Distance
2013 2014 2015
Initial WatchDog
prototype with ~10
users
2013 2014 2015
Initial WatchDog
prototype with ~10
users
2013 2014 2015
1.0
Initial WatchDog
prototype with ~10
users
2013 2014 2015
1.0
40
Initial WatchDog
prototype with ~10
users
2013 2014 2015
1.0
ICSE NIER
Evaluation
with 40
students
40
Initial WatchDog
prototype with ~10
users
2013 2014 2015
1.0
ICSE NIER
Evaluation
with 40
students
40
Initial WatchDog
prototype with ~10
users
2013 2014 2015
1.0
ICSE NIER
Evaluation
with 40
students
40 416
Initial WatchDog
prototype with ~10
users
2013 2014 2015
1.0
ICSE NIER
Evaluation
with 40
students
ESEC/FSE
Studying
416 d...
Initial WatchDog
prototype with ~10
users
2013 2014 2015
1.0
ICSE NIER
Evaluation
with 40
students
ESEC/FSE
Studying
416 d...
Initial WatchDog
prototype with ~10
users
2013 2014 2015
1.0
ICSE NIER
Evaluation
with 40
students
ESEC/FSE
Studying
416 d...
416
Source: https://en.wikipedia.org/wiki/Pair_programming, CC-BY
WatchDog World Domination Tour 2015
416
416
●
Projects from industry
●
Famous and not-so-famous OSS
projects, e.g. Apache
●
Private projects
●
Anonymous projects
...
416
●
Projects from industry
●
Famous and not-so-famous OSS
projects, e.g. Apache
●
Private projects
●
Anonymous projects
...
416
●
Projects from industry
●
Famous and not-so-famous OSS
projects, e.g. Apache
●
Private projects
●
Anonymous projects
...
416
●
Projects from industry
●
Famous and not-so-famous OSS
projects, e.g. Apache
●
Private projects
●
Anonymous projects
...
RQ1. How Do Developers Test?
RQ1. How Do Developers Test?
How Common is Testing?
RQ1. How Do Developers Test?
How Common is Testing?
Projects
RQ1. How Do Developers Test?
How Common is Testing?
460
Projects
RQ1. How Do Developers Test?
How Common is Testing?
460
200 (43%)
Projects
RQ1. How Do Developers Test?
How Common is Testing?
460
200 (43%)
Projects
RQ1. How Do Developers Test?
How Common is Testing?
460
200 (43%) Actually worked on
tests
Projects
RQ1. How Do Developers Test?
How Common is Testing?
460
200 (43%) Actually worked on
tests
Projects
1) Most projects do no...
RQ1. How Do Developers Test?
What are Tests? Every class that ...
RQ1. How Do Developers Test?
What are Tests? Every class that ...
●
is a Junit test.
●
contains traces of a testing Framew...
RQ1. How Do Developers Test?
What are Tests? Every class that ...
●
is a Junit test.
●
contains traces of a testing Framew...
RQ1. How Do Developers Test?
What are Tests? Every class that ...
●
is a Junit test.
●
contains traces of a testing Framew...
RQ1. How Do Developers Test?
RQ1. How Do Developers Test?
How Common is Executing Tests?
RQ1. How Do Developers Test?
How Common is Executing Tests?
sessions
RQ1. How Do Developers Test?
How Common is Executing Tests?
3,424
sessions
RQ1. How Do Developers Test?
How Common is Executing Tests?
3,424
527 (15%)
sessions
≥ 1 test run
RQ1. How Do Developers Test?
How Common is Executing Tests?
3,424
527 (15%)
sessions
≥ 1 test run
2) Very rare, even for p...
RQ1. How Do Developers Test?
Image: TylerFrotierPhoto, http://tylerfortierphoto.deviantart.com/art/Lonely-Sheep-244657922
RQ1. How Do Developers Test?
How Many Tests Do Developers Execute?
Image: TylerFrotierPhoto, http://tylerfortierphoto.devi...
RQ1. How Do Developers Test?
How Many Tests Do Developers Execute?
Image: TylerFrotierPhoto, http://tylerfortierphoto.devi...
RQ1. How Do Developers Test?
How Many Tests Do Developers Execute?
3) One (90%), or all of them (9%).
Image: TylerFrotierP...
RQ1. How Do Developers Test?
Image: https://blog.shareaholic.com/retail-sites-page-load-time/stopwatch
RQ1. How Do Developers Test?
How Long is a Test Run?
Image: https://blog.shareaholic.com/retail-sites-page-load-time/stopw...
RQ1. How Do Developers Test?
How Long is a Test Run?
50% < 0.5s
75% < 5s
95% < 2m
Image: https://blog.shareaholic.com/reta...
RQ1. How Do Developers Test?
How Long is a Test Run?
50% < 0.5s
75% < 5s
95% < 2m
Image: https://blog.shareaholic.com/reta...
RQ1. How Do Developers Test?
How Long is a Test Run?
50% < 0.5s
75% < 5s
95% < 2m
Image: https://blog.shareaholic.com/reta...
RQ1. How Do Developers Test?
How Long is a Test Run?
50% < 0.5s
75% < 5s
95% < 2m
4) Very short.
Image: https://blog.share...
RQ1. How Do Developers Test?
RQ1. How Do Developers Test?
1) Most projects do not actively work with tests.
2) Even projects with tests execute them ve...
RQ1. How Do Developers Test?
1) Most projects do not actively work with tests.
2) Even projects with tests execute them ve...
RQ1. How Do Developers Test?
1) Most projects do not actively work with tests.
2) Even projects with tests execute them ve...
RQ1. How Do Developers Test?
1) Most projects do not actively work with tests.
2) Even projects with tests execute them ve...
RQ2. How Do Developers Manage Failing Tests?
Image: https://www.linkedin.com/pulse/admit-failure-early-james-ient
RQ2. How Do Developers Manage Failing Tests?
How Many Test Executions Fail?
Image: https://www.linkedin.com/pulse/admit-fa...
RQ2. How Do Developers Manage Failing Tests?
The majority, 65%.
How Many Test Executions Fail?
Image: https://www.linkedin...
RQ2. How Do Developers Manage Failing Tests?
RQ2. How Do Developers Manage Failing Tests?
RQ2. How Do Developers Manage Failing Tests?
RQ2. How Do Developers Manage Failing Tests?
RQ2. How Do Developers Manage Failing Tests?
RQ2. How Do Developers Manage Failing Tests?
1) Dive into production code
2) Switch away from Eclipse
3) Almost no debug p...
RQ2. How Do Developers Manage Failing Tests?
1) Dive into production code
2) Switch away from Eclipse
3) Almost no debug p...
RQ2. How Do Developers Manage Failing Tests?
1) Dive into production code
2) Switch away from Eclipse
3) Almost no debug p...
RQ2. How Do Developers Manage Failing Tests?
1) Dive into production code
2) Switch away from Eclipse
3) Almost no debug p...
RQ2. How Do Developers Manage Failing Tests?
minutes
RQ2. How Do Developers Manage Failing Tests?
minutes
RQ2. How Do Developers Manage Failing Tests?
50% of test failures are fixed within 10 minutes.
minutes
RQ2. How Do Developers Manage Failing Tests?
RQ2. How Do Developers Manage Failing Tests?
RQ2. How Do Developers Manage Failing Tests?
1) Most test executions fail.
2) Developers have certain patterns of managing...
RQ2. How Do Developers Manage Failing Tests?
1) Most test executions fail.
2) Developers have certain patterns of managing...
RQ2. How Do Developers Manage Failing Tests?
1) Most test executions fail.
2) Developers have certain patterns of managing...
RQ3. Do Developers Follow TDD?
What is TDD?
Image: http://www.agilenutshell.com/test_driven_development
RQ3. Do Developers Follow TDD?
What is TDD?
Image: http://www.agilenutshell.com/test_driven_development
RQ3. Do Developers Follow TDD?
What is TDD?
Image: http://www.agilenutshell.com/test_driven_development
RQ3. Do Developers Follow TDD?
What is TDD?
Image: http://www.agilenutshell.com/test_driven_development
RQ3. Do Developers Follow TDD?
What is TDD?
Image: http://www.agilenutshell.com/test_driven_development
RQ3. Do Developers Follow TDD?
Method?
RQ3. Do Developers Follow TDD?
Method?
RQ3. Do Developers Follow TDD?
Method?
Test phase
RQ3. Do Developers Follow TDD?
Method?
Test phase
RQ3. Do Developers Follow TDD?
Method?
Test phase Refactoring
RQ3. Do Developers Follow TDD?
Method?
Test phase Refactoring
2% of developers
RQ3. Do Developers Follow TDD?
Method?
Test phase Refactoring
2% of developers
28%
RQ3. Do Developers Follow TDD?
Method?
Test phase Refactoring
2% of developers
72%28%
RQ3. Do Developers Follow TDD?
Image: http://www.agilenutshell.com/test_driven_development
RQ3. Do Developers Follow TDD?
Image: http://www.agilenutshell.com/test_driven_development
RQ3. Do Developers Follow TDD?
(Almost) nobody follows TDD.
Image: http://www.agilenutshell.com/test_driven_development
RQ3. Do Developers Follow TDD?
“You followed TDD 38.6% of your
development changes (so, in
words, quite often).”
Sven Amann, letsdeveloper.com
RQ3. Do De...
RQ4. How Much Do Developers Test?
Software Testing takes ...
RQ4. How Much Do Developers Test?
Brooks, 1975
Software Testing takes ...
RQ4. How Much Do Developers Test?
50%
… of Project EffortBrooks, 1975
Software Testing takes ...
How (Much)
Do You
Test?
Image: http://www.agilenutshell.com/test_driven_development
RQ4. How Much Do Developers Test?
Projects
Delta(Production% Est - Real)
RQ4. How Much Do Developers Test?
Projects
Delta(Production% Est - Real)
RQ4. How Much Do Developers Test?
Projects
Delta(Production% Est - Real)
RQ4. How Much Do Developers Test?
%testingwork
RQ4. How Much Do Developers Test?
Estimated 48%
%testingwork
RQ4. How Much Do Developers Test?
Estimated 48% Reality 25%
%testingwork
RQ4. How Much Do Developers Test?
Estimated 48% Reality 25%
%testingwork
not test execution time!
Conclusion: Testing Paradox
Conclusion: Testing Paradox
1) Test executions are rare
Conclusion: Testing Paradox
1) Test executions are rare
2) Yet, 25% of work time on testing
Conclusion: Testing Paradox
1) Test executions are rare
“Running Eclipse UI Platform tests
locally blocks my machine … I
o...
@Inventitech
Moritz Beller, TU Delft
When, How, and Why Developers
(Do Not) Test in Their IDEs
@Inventitech
Moritz Beller, TU Delft
When, How, and Why Developers
(Do Not) Test in Their IDEs
@Inventitech
Moritz Beller, TU Delft
When, How, and Why Developers
(Do Not) Test in Their IDEs
@Inventitech
Moritz Beller, TU Delft
When, How, and Why Developers
(Do Not) Test in Their IDEs
When, How, and Why Developers (Do Not) Test
When, How, and Why Developers (Do Not) Test
When, How, and Why Developers (Do Not) Test
When, How, and Why Developers (Do Not) Test
When, How, and Why Developers (Do Not) Test
When, How, and Why Developers (Do Not) Test
When, How, and Why Developers (Do Not) Test
When, How, and Why Developers (Do Not) Test
When, How, and Why Developers (Do Not) Test
When, How, and Why Developers (Do Not) Test
When, How, and Why Developers (Do Not) Test
When, How, and Why Developers (Do Not) Test
When, How, and Why Developers (Do Not) Test
Upcoming SlideShare
Loading in …5
×

When, How, and Why Developers (Do Not) Test

813 views

Published on

The research community in Software Engineering and Software Testing in particular builds many of its contributions on a set of mutually shared expectations. Despite the fact that they form the basis of many publications as well as open-source and commercial testing applications, these common expectations and beliefs are rarely ever questioned. For example, Frederic Brooks’ statement that testing takes half of the development time seems to have manifested itself within the community since he first made it in the “Mythical Man Month” in 1975. With this paper, we report on the surprising results of a large-scale field study with 416 software engineers whose development activity we closely monitored over the course of five months, resulting in over 13 years of recorded work time in their integrated development environments (IDEs). Our findings question several commonly shared assumptions and beliefs about testing and might be contributing factors to the observed bug proneness of software in practice: the majority of developers in our study does not test; developers rarely run their tests in the IDE; Test-Driven Development (TDD) is not widely practiced; and, last but not least, software developers only spend a quarter of their work time engineering tests, whereas they think they test half of their time.

Published in: Science
  • Be the first to comment

  • Be the first to like this

When, How, and Why Developers (Do Not) Test

  1. 1. When, How, and Why Developers (Do Not) Test in Their IDEs
  2. 2. When, How, and Why Developers (Do Not) Test in Their IDEs Moritz Beller @Inventitech
  3. 3. Image: https://en.wikipedia.org/wiki/Delft, CC-BY
  4. 4. Image: https://en.wikipedia.org/wiki/Delft, CC-BY
  5. 5. Image: https://en.wikipedia.org/wiki/Delft, CC-BY
  6. 6. Image: https://en.wikipedia.org/wiki/Province_of_Bergamo, CC-BY
  7. 7. Image: http://www.scarymommy.com/angry-baby/
  8. 8. Image: http://www.scarymommy.com/angry-baby/
  9. 9. I am one of you! Image: http://www.scarymommy.com/angry-baby/
  10. 10. Source: Moeez, https://commons.wikimedia.org/wiki/File:Thank_You!.jpg, CC-BY
  11. 11. Image: http://www.jccschools.com/news/what_s_new/district_testing_descriptions
  12. 12. Developer Writes Tests Image: http://www.jccschools.com/news/what_s_new/district_testing_descriptions
  13. 13. Developer Writes Tests Mine Repos for Tests Image: http://www.jccschools.com/news/what_s_new/district_testing_descriptions
  14. 14. Developer Writes Tests Mine Repos for Tests Image: http://www.jccschools.com/news/what_s_new/district_testing_descriptions
  15. 15. ? Developer Writes Tests Mine Repos for Tests Image: http://www.jccschools.com/news/what_s_new/district_testing_descriptions
  16. 16. RQ1. How Do Developers Test?
  17. 17. RQ1. How Do Developers Test? RQ2. How Do Developers Manage Failing Tests?
  18. 18. RQ1. How Do Developers Test? RQ3. Do Developers Follow Test-Driven Development? RQ2. How Do Developers Manage Failing Tests?
  19. 19. RQ1. How Do Developers Test? RQ3. Do Developers Follow Test-Driven Development? RQ4. How Much Do Developers Test? RQ2. How Do Developers Manage Failing Tests?
  20. 20. IDE!
  21. 21. IDE!
  22. 22. IDE!
  23. 23. Image: https://www.ckras.com/EPS-Entry-GameServer-mieten
  24. 24. Image: https://www.ckras.com/EPS-Entry-GameServer-mieten
  25. 25. → Interval
  26. 26. → Edit Distance
  27. 27. 2013 2014 2015
  28. 28. Initial WatchDog prototype with ~10 users 2013 2014 2015
  29. 29. Initial WatchDog prototype with ~10 users 2013 2014 2015 1.0
  30. 30. Initial WatchDog prototype with ~10 users 2013 2014 2015 1.0 40
  31. 31. Initial WatchDog prototype with ~10 users 2013 2014 2015 1.0 ICSE NIER Evaluation with 40 students 40
  32. 32. Initial WatchDog prototype with ~10 users 2013 2014 2015 1.0 ICSE NIER Evaluation with 40 students 40
  33. 33. Initial WatchDog prototype with ~10 users 2013 2014 2015 1.0 ICSE NIER Evaluation with 40 students 40 416
  34. 34. Initial WatchDog prototype with ~10 users 2013 2014 2015 1.0 ICSE NIER Evaluation with 40 students ESEC/FSE Studying 416 developers 40 416
  35. 35. Initial WatchDog prototype with ~10 users 2013 2014 2015 1.0 ICSE NIER Evaluation with 40 students ESEC/FSE Studying 416 developers 1.5 40 416
  36. 36. Initial WatchDog prototype with ~10 users 2013 2014 2015 1.0 ICSE NIER Evaluation with 40 students ESEC/FSE Studying 416 developers ? 1,936 1.5 40 416
  37. 37. 416 Source: https://en.wikipedia.org/wiki/Pair_programming, CC-BY
  38. 38. WatchDog World Domination Tour 2015
  39. 39. 416
  40. 40. 416 ● Projects from industry ● Famous and not-so-famous OSS projects, e.g. Apache ● Private projects ● Anonymous projects ● 5 months
  41. 41. 416 ● Projects from industry ● Famous and not-so-famous OSS projects, e.g. Apache ● Private projects ● Anonymous projects ● 5 months
  42. 42. 416 ● Projects from industry ● Famous and not-so-famous OSS projects, e.g. Apache ● Private projects ● Anonymous projects ● 5 months ● 68 countries ● 24,255 hours of work time ● ~13.7 developer years ● 460 unique projects ● 5,665 sessions
  43. 43. 416 ● Projects from industry ● Famous and not-so-famous OSS projects, e.g. Apache ● Private projects ● Anonymous projects ● 5 months ● 68 countries ● 24,255 hours of work time ● ~13.7 developer years ● 460 unique projects ● 5,665 sessions
  44. 44. RQ1. How Do Developers Test?
  45. 45. RQ1. How Do Developers Test? How Common is Testing?
  46. 46. RQ1. How Do Developers Test? How Common is Testing? Projects
  47. 47. RQ1. How Do Developers Test? How Common is Testing? 460 Projects
  48. 48. RQ1. How Do Developers Test? How Common is Testing? 460 200 (43%) Projects
  49. 49. RQ1. How Do Developers Test? How Common is Testing? 460 200 (43%) Projects
  50. 50. RQ1. How Do Developers Test? How Common is Testing? 460 200 (43%) Actually worked on tests Projects
  51. 51. RQ1. How Do Developers Test? How Common is Testing? 460 200 (43%) Actually worked on tests Projects 1) Most projects do not work with tests.
  52. 52. RQ1. How Do Developers Test? What are Tests? Every class that ...
  53. 53. RQ1. How Do Developers Test? What are Tests? Every class that ... ● is a Junit test. ● contains traces of a testing Framework (Mockito, PowerMock, TestNG, …). ● has “Test” in its name.
  54. 54. RQ1. How Do Developers Test? What are Tests? Every class that ... ● is a Junit test. ● contains traces of a testing Framework (Mockito, PowerMock, TestNG, …). ● has “Test” in its name.
  55. 55. RQ1. How Do Developers Test? What are Tests? Every class that ... ● is a Junit test. ● contains traces of a testing Framework (Mockito, PowerMock, TestNG, …). ● has “Test” in its name.
  56. 56. RQ1. How Do Developers Test?
  57. 57. RQ1. How Do Developers Test? How Common is Executing Tests?
  58. 58. RQ1. How Do Developers Test? How Common is Executing Tests? sessions
  59. 59. RQ1. How Do Developers Test? How Common is Executing Tests? 3,424 sessions
  60. 60. RQ1. How Do Developers Test? How Common is Executing Tests? 3,424 527 (15%) sessions ≥ 1 test run
  61. 61. RQ1. How Do Developers Test? How Common is Executing Tests? 3,424 527 (15%) sessions ≥ 1 test run 2) Very rare, even for projects with tests.
  62. 62. RQ1. How Do Developers Test? Image: TylerFrotierPhoto, http://tylerfortierphoto.deviantart.com/art/Lonely-Sheep-244657922
  63. 63. RQ1. How Do Developers Test? How Many Tests Do Developers Execute? Image: TylerFrotierPhoto, http://tylerfortierphoto.deviantart.com/art/Lonely-Sheep-244657922
  64. 64. RQ1. How Do Developers Test? How Many Tests Do Developers Execute? Image: TylerFrotierPhoto, http://tylerfortierphoto.deviantart.com/art/Lonely-Sheep-244657922
  65. 65. RQ1. How Do Developers Test? How Many Tests Do Developers Execute? 3) One (90%), or all of them (9%). Image: TylerFrotierPhoto, http://tylerfortierphoto.deviantart.com/art/Lonely-Sheep-244657922
  66. 66. RQ1. How Do Developers Test? Image: https://blog.shareaholic.com/retail-sites-page-load-time/stopwatch
  67. 67. RQ1. How Do Developers Test? How Long is a Test Run? Image: https://blog.shareaholic.com/retail-sites-page-load-time/stopwatch
  68. 68. RQ1. How Do Developers Test? How Long is a Test Run? 50% < 0.5s 75% < 5s 95% < 2m Image: https://blog.shareaholic.com/retail-sites-page-load-time/stopwatch
  69. 69. RQ1. How Do Developers Test? How Long is a Test Run? 50% < 0.5s 75% < 5s 95% < 2m Image: https://blog.shareaholic.com/retail-sites-page-load-time/stopwatch
  70. 70. RQ1. How Do Developers Test? How Long is a Test Run? 50% < 0.5s 75% < 5s 95% < 2m Image: https://blog.shareaholic.com/retail-sites-page-load-time/stopwatch
  71. 71. RQ1. How Do Developers Test? How Long is a Test Run? 50% < 0.5s 75% < 5s 95% < 2m 4) Very short. Image: https://blog.shareaholic.com/retail-sites-page-load-time/stopwatch
  72. 72. RQ1. How Do Developers Test?
  73. 73. RQ1. How Do Developers Test? 1) Most projects do not actively work with tests. 2) Even projects with tests execute them very rarely. 3) Most test executions contain 1 test. 4) Most test executions are very short.
  74. 74. RQ1. How Do Developers Test? 1) Most projects do not actively work with tests. 2) Even projects with tests execute them very rarely. 3) Most test executions contain 1 test. 4) Most test executions are very short.
  75. 75. RQ1. How Do Developers Test? 1) Most projects do not actively work with tests. 2) Even projects with tests execute them very rarely. 3) Most test executions contain 1 test. 4) Most test executions are very short.
  76. 76. RQ1. How Do Developers Test? 1) Most projects do not actively work with tests. 2) Even projects with tests execute them very rarely. 3) Most test executions contain 1 test. 4) Most test executions are very short.
  77. 77. RQ2. How Do Developers Manage Failing Tests? Image: https://www.linkedin.com/pulse/admit-failure-early-james-ient
  78. 78. RQ2. How Do Developers Manage Failing Tests? How Many Test Executions Fail? Image: https://www.linkedin.com/pulse/admit-failure-early-james-ient
  79. 79. RQ2. How Do Developers Manage Failing Tests? The majority, 65%. How Many Test Executions Fail? Image: https://www.linkedin.com/pulse/admit-failure-early-james-ient
  80. 80. RQ2. How Do Developers Manage Failing Tests?
  81. 81. RQ2. How Do Developers Manage Failing Tests?
  82. 82. RQ2. How Do Developers Manage Failing Tests?
  83. 83. RQ2. How Do Developers Manage Failing Tests?
  84. 84. RQ2. How Do Developers Manage Failing Tests?
  85. 85. RQ2. How Do Developers Manage Failing Tests? 1) Dive into production code 2) Switch away from Eclipse 3) Almost no debug perspective 4) No frustration-closing of IDE
  86. 86. RQ2. How Do Developers Manage Failing Tests? 1) Dive into production code 2) Switch away from Eclipse 3) Almost no debug perspective 4) No frustration-closing of IDE
  87. 87. RQ2. How Do Developers Manage Failing Tests? 1) Dive into production code 2) Switch away from Eclipse 3) Almost no debug perspective 4) No frustration-closing of IDE
  88. 88. RQ2. How Do Developers Manage Failing Tests? 1) Dive into production code 2) Switch away from Eclipse 3) Almost no debug perspective 4) No frustration-closing of IDE
  89. 89. RQ2. How Do Developers Manage Failing Tests?
  90. 90. minutes RQ2. How Do Developers Manage Failing Tests?
  91. 91. minutes RQ2. How Do Developers Manage Failing Tests?
  92. 92. 50% of test failures are fixed within 10 minutes. minutes RQ2. How Do Developers Manage Failing Tests?
  93. 93. RQ2. How Do Developers Manage Failing Tests?
  94. 94. RQ2. How Do Developers Manage Failing Tests? 1) Most test executions fail. 2) Developers have certain patterns of managing failed test executions. 3) Most test repairs are fast.
  95. 95. RQ2. How Do Developers Manage Failing Tests? 1) Most test executions fail. 2) Developers have certain patterns of managing failed test executions. 3) Most test repairs are fast.
  96. 96. RQ2. How Do Developers Manage Failing Tests? 1) Most test executions fail. 2) Developers have certain patterns of managing failed test executions. 3) Most test repairs are fast.
  97. 97. RQ3. Do Developers Follow TDD? What is TDD? Image: http://www.agilenutshell.com/test_driven_development
  98. 98. RQ3. Do Developers Follow TDD? What is TDD? Image: http://www.agilenutshell.com/test_driven_development
  99. 99. RQ3. Do Developers Follow TDD? What is TDD? Image: http://www.agilenutshell.com/test_driven_development
  100. 100. RQ3. Do Developers Follow TDD? What is TDD? Image: http://www.agilenutshell.com/test_driven_development
  101. 101. RQ3. Do Developers Follow TDD? What is TDD? Image: http://www.agilenutshell.com/test_driven_development
  102. 102. RQ3. Do Developers Follow TDD? Method?
  103. 103. RQ3. Do Developers Follow TDD? Method?
  104. 104. RQ3. Do Developers Follow TDD? Method? Test phase
  105. 105. RQ3. Do Developers Follow TDD? Method? Test phase
  106. 106. RQ3. Do Developers Follow TDD? Method? Test phase Refactoring
  107. 107. RQ3. Do Developers Follow TDD? Method? Test phase Refactoring 2% of developers
  108. 108. RQ3. Do Developers Follow TDD? Method? Test phase Refactoring 2% of developers 28%
  109. 109. RQ3. Do Developers Follow TDD? Method? Test phase Refactoring 2% of developers 72%28%
  110. 110. RQ3. Do Developers Follow TDD? Image: http://www.agilenutshell.com/test_driven_development
  111. 111. RQ3. Do Developers Follow TDD? Image: http://www.agilenutshell.com/test_driven_development
  112. 112. RQ3. Do Developers Follow TDD? (Almost) nobody follows TDD. Image: http://www.agilenutshell.com/test_driven_development
  113. 113. RQ3. Do Developers Follow TDD?
  114. 114. “You followed TDD 38.6% of your development changes (so, in words, quite often).” Sven Amann, letsdeveloper.com RQ3. Do Developers Follow TDD?
  115. 115. RQ4. How Much Do Developers Test? Software Testing takes ...
  116. 116. RQ4. How Much Do Developers Test? Brooks, 1975 Software Testing takes ...
  117. 117. RQ4. How Much Do Developers Test? 50% … of Project EffortBrooks, 1975 Software Testing takes ...
  118. 118. How (Much) Do You Test? Image: http://www.agilenutshell.com/test_driven_development
  119. 119. RQ4. How Much Do Developers Test? Projects Delta(Production% Est - Real)
  120. 120. RQ4. How Much Do Developers Test? Projects Delta(Production% Est - Real)
  121. 121. RQ4. How Much Do Developers Test? Projects Delta(Production% Est - Real)
  122. 122. RQ4. How Much Do Developers Test? %testingwork
  123. 123. RQ4. How Much Do Developers Test? Estimated 48% %testingwork
  124. 124. RQ4. How Much Do Developers Test? Estimated 48% Reality 25% %testingwork
  125. 125. RQ4. How Much Do Developers Test? Estimated 48% Reality 25% %testingwork not test execution time!
  126. 126. Conclusion: Testing Paradox
  127. 127. Conclusion: Testing Paradox 1) Test executions are rare
  128. 128. Conclusion: Testing Paradox 1) Test executions are rare 2) Yet, 25% of work time on testing
  129. 129. Conclusion: Testing Paradox 1) Test executions are rare “Running Eclipse UI Platform tests locally blocks my machine … I offload running them to the CI.” Famous Eclipse Dude 2) Yet, 25% of work time on testing
  130. 130. @Inventitech Moritz Beller, TU Delft When, How, and Why Developers (Do Not) Test in Their IDEs
  131. 131. @Inventitech Moritz Beller, TU Delft When, How, and Why Developers (Do Not) Test in Their IDEs
  132. 132. @Inventitech Moritz Beller, TU Delft When, How, and Why Developers (Do Not) Test in Their IDEs
  133. 133. @Inventitech Moritz Beller, TU Delft When, How, and Why Developers (Do Not) Test in Their IDEs

×