Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub

143 views

Published on

Continuous Integration (CI) has become a best practice of modern software development. Yet, at present, we have a shortfall of insight into the testing practices that are common in CI-based software development. In particular, we seek quantifiable evidence on how central testing is to the CI process, how strongly the project language influences testing, whether different integration environments are valuable and if testing on the CI can serve as a surrogate to local testing in the IDE. In an analysis of 2,640,825 Java and Ruby builds on Travis CI, we find that testing is the single most important reason why builds fail. Moreover, the programming language has a strong influence on both the number of executed tests, their run time, and proneness to fail. The use of multiple integration environments leads to 10% more failures being caught at build time. However, testing on Travis CI does not seem an adequate surrogate for running tests locally in the IDE. To further research on Travis CI with GitHub, we introduce TravisTorrent.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub

  1. 1. Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub
  2. 2. Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub Moritz Beller @Inventitech
  3. 3. Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub Moritz Beller @Inventitech Georgios Gousios
  4. 4. Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub Moritz Beller @Inventitech Georgios Gousios Andy Zaidman
  5. 5. How This Paper Came To Be
  6. 6. How This Paper Came To Be
  7. 7. How This Paper Came To Be
  8. 8. Travis What?
  9. 9. Travis What?
  10. 10. Travis What?
  11. 11. P1. Using CI is a widespread best practice
  12. 12. P1. Using CI is a widespread best practice CI has become an integral quality assurance practice.
  13. 13. P. M. Duvall, S. Matyas, and A. Glover, “Continuous integration: improving software quality and reducing risk” P1. Using CI is a widespread best practice CI has become an integral quality assurance practice.
  14. 14. P1. Using CI is a widespread best practice
  15. 15. P1. Using CI is a widespread best practice
  16. 16. P1. Using CI is a widespread best practice
  17. 17. P1. Using CI is a widespread best practice
  18. 18. P1. Using CI is a widespread best practice
  19. 19. P1. Using CI is a widespread best practice
  20. 20. Around 30% of GITHUB projects that could potentially use the free Travis CI, also make use it. P1. Using CI is a widespread best practice
  21. 21. Around 30% of GITHUB projects that could potentially use the free Travis CI, also make use it. Caveat: In 2016! P1. Using CI is a widespread best practice
  22. 22. TODO: Add background with Sun P1. Using CI is a widespread best practice
  23. 23. TODO: Add background with Sun P1. Using CI is a widespread best practice Filter Criteria
  24. 24. TODO: Add background with Sun P1. Using CI is a widespread best practice Filter Criteria ● Non-toy, non-fork projects (we want the original!)
  25. 25. TODO: Add background with Sun P1. Using CI is a widespread best practice Filter Criteria ● Non-toy, non-fork projects (we want the original!) ● > 50 stars
  26. 26. TODO: Add background with Sun P1. Using CI is a widespread best practice Filter Criteria ● Non-toy, non-fork projects (we want the original!) ● > 50 stars ● A longer (> 50) Travis build history
  27. 27. TODO: Add background with Sun P1. Using CI is a widespread best practice Filter Criteria ● Non-toy, non-fork projects (we want the original!) ● > 50 stars ● A longer (> 50) Travis build history E. Kalliamvakou et al.: “An in-depth study of the promises and perils of mining GitHub”
  28. 28. TODO: Add background with Sun P1. Using CI is a widespread best practice Filter Criteria ● Non-toy, non-fork projects (we want the original!) ● > 50 stars ● A longer (> 50) Travis build history E. Kalliamvakou et al.: “An in-depth study of the promises and perils of mining GitHub”
  29. 29. TODO: Add background with Sun P1. Using CI is a widespread best practice Filter Criteria ● Non-toy, non-fork projects (we want the original!) ● > 50 stars ● A longer (> 50) Travis build history E. Kalliamvakou et al.: “An in-depth study of the promises and perils of mining GitHub”
  30. 30. TODO: Add background with Sun P1. Using CI is a widespread best practice Filter Criteria ● Non-toy, non-fork projects (we want the original!) ● > 50 stars ● A longer (> 50) Travis build history E. Kalliamvakou et al.: “An in-depth study of the promises and perils of mining GitHub” Java ● 423 projects
  31. 31. TODO: Add background with Sun P1. Using CI is a widespread best practice Filter Criteria ● Non-toy, non-fork projects (we want the original!) ● > 50 stars ● A longer (> 50) Travis build history E. Kalliamvakou et al.: “An in-depth study of the promises and perils of mining GitHub” Ruby ● 936 projects Java ● 423 projects
  32. 32. P1. Using CI is a widespread best practice ~
  33. 33. TODO: Add background with Sun P2. Testing is central to CI
  34. 34. TODO: Add background with Sun P2. Testing is central to CI CI is the best practice in which developers not only integrate their work into a shared mainline frequently, but also verify the quality continuously through testing.
  35. 35. TODO: Add background with Sun P2. Testing is central to CI CI is the best practice in which developers not only integrate their work into a shared mainline frequently, but also verify the quality continuously through testing. It is known.
  36. 36. <TL;DR>...</TL;DR>
  37. 37. <TL;DR>...</TL;DR> ------------------------------------------------------- T E S T S ------------------------------------------------------- Running nl.tudelft.watchdog.ClientVersionCheckerTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
  38. 38. <TL;DR>...</TL;DR> ------------------------------------------------------- T E S T S ------------------------------------------------------- Running nl.tudelft.watchdog.ClientVersionCheckerTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 <TL;DR>...</TL;DR>
  39. 39. <TL;DR>...</TL;DR> ------------------------------------------------------- T E S T S ------------------------------------------------------- Running nl.tudelft.watchdog.ClientVersionCheckerTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 <TL;DR>...</TL;DR>
  40. 40. <TL;DR>...</TL;DR> ------------------------------------------------------- T E S T S ------------------------------------------------------- Running nl.tudelft.watchdog.ClientVersionCheckerTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 <TL;DR>...</TL;DR>
  41. 41. <TL;DR>...</TL;DR> ------------------------------------------------------- T E S T S ------------------------------------------------------- Running nl.tudelft.watchdog.ClientVersionCheckerTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 <TL;DR>...</TL;DR>
  42. 42. <TL;DR>...</TL;DR> ------------------------------------------------------- T E S T S ------------------------------------------------------- Running nl.tudelft.watchdog.ClientVersionCheckerTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 <TL;DR>...</TL;DR> ● Parsing Log files for Java: Maven, Gradle, Ant
  43. 43. <TL;DR>...</TL;DR> ------------------------------------------------------- T E S T S ------------------------------------------------------- Running nl.tudelft.watchdog.ClientVersionCheckerTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 <TL;DR>...</TL;DR> ● Parsing Log files for Java: Maven, Gradle, Ant
  44. 44. <TL;DR>...</TL;DR> ------------------------------------------------------- T E S T S ------------------------------------------------------- Running nl.tudelft.watchdog.ClientVersionCheckerTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 <TL;DR>...</TL;DR> ● Parsing Log files for Java: Maven, Gradle, Ant
  45. 45. <TL;DR>...</TL;DR> ------------------------------------------------------- T E S T S ------------------------------------------------------- Running nl.tudelft.watchdog.ClientVersionCheckerTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 <TL;DR>...</TL;DR> ● Parsing Log files for Java: Maven, Gradle, Ant ● Parsing Log files for Ruby: bundler and rake
  46. 46. <TL;DR>...</TL;DR> ------------------------------------------------------- T E S T S ------------------------------------------------------- Running nl.tudelft.watchdog.ClientVersionCheckerTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 <TL;DR>...</TL;DR> ● Parsing Log files for Java: Maven, Gradle, Ant ● Parsing Log files for Ruby: bundler and rake
  47. 47. P2. Testing is central to CI
  48. 48. ● 69% of Java and 87.5% of the Ruby projects: test runs P2. Testing is central to CI
  49. 49. ● 69% of Java and 87.5% of the Ruby projects: test runs 1:0 for Ruby! P2. Testing is central to CI
  50. 50. ● 69% of Java and 87.5% of the Ruby projects: test runs 1:0 for Ruby! P2. Testing is central to CI ● Overall, 81% of the projects: test runs
  51. 51. ● 69% of Java and 87.5% of the Ruby projects: test runs 1:0 for Ruby! P2. Testing is central to CI ● Overall, 81% of the projects: test runs ● 96% of builds: test runs
  52. 52. P2. Testing is central to CI
  53. 53. P2. Testing is central to CI
  54. 54. P2. Testing is central to CI
  55. 55. Testing is the single most important reason for broken builds, more prevalent than compile errors, missing dependencies, build cancellations and provisioning problems together. P2. Testing is central to CI
  56. 56. Testing is the single most important reason for broken builds, more prevalent than compile errors, missing dependencies, build cancellations and provisioning problems together. P2. Testing is central to CI
  57. 57. TODO: Add background with Sun P3. Testing is language(-type) dependent Does a dynamically typed language (like Ruby) need as much testing as a statically typed language (like Java)?
  58. 58. TODO: Add background with Sun P3. Testing is language(-type) dependent Does a dynamically typed language (like Ruby) need as much testing as a statically typed language (like Java)? S. Kleinschmager, S. Hanenberg, R. Robbes, E. Tanter, A. Stefik: Do static type systems improve the maintainability of software systems? An empirical study.
  59. 59. P3. Testing is language(-type) dependent
  60. 60. P3. Testing is language(-type) dependent
  61. 61. P3. Testing is language(-type) dependent
  62. 62. P3. Testing is language(-type) dependent Tests Build Result→
  63. 63. P3. Testing is language(-type) dependent Tests Build Result→
  64. 64. P3. Testing is language(-type) dependent Tests Build Result→
  65. 65. P3. Testing is language(-type) dependent Tests Build Result→
  66. 66. Ruby projects have 10x more tests than Java projects, yet execute faster. Ruby builds are 4x more likely to fail than Java projects. P3. Testing is language(-type) dependent
  67. 67. Ruby projects have 10x more tests than Java projects, yet execute faster. Ruby builds are 4x more likely to fail than Java projects. P3. Testing is language(-type) dependent
  68. 68. Building and testing in different environments delivers additional value. P4. Are more build environments helpful?
  69. 69. Building and testing in different environments delivers additional value. We don’t know. P4. Are more build environments helpful?
  70. 70. P4. Are more build environments helpful?
  71. 71. P4. Are more build environments helpful?
  72. 72. P4. Are more build environments helpful?
  73. 73. Yes, but 8x more so in Ruby than in Java. P4. Are more build environments helpful?
  74. 74. Yes, but 8x more so in Ruby than in Java. P4. Are more build environments helpful?
  75. 75. Moritz Beller @Inventitech Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub
  76. 76. Moritz Beller @Inventitech Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub
  77. 77. Moritz Beller @Inventitech Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub
  78. 78. Moritz Beller @Inventitech Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub
  79. 79. Moritz Beller @Inventitech Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub

×