1. Oops, My Tests Broke the Build:
An Explorative Analysis of Travis CI
with GitHub

2. Oops, My Tests Broke the Build:
An Explorative Analysis of Travis CI
with GitHub
Moritz Beller
@Inventitech

3. Oops, My Tests Broke the Build:
An Explorative Analysis of Travis CI
with GitHub
Moritz Beller
@Inventitech
Georgios Gousios

4. Oops, My Tests Broke the Build:
An Explorative Analysis of Travis CI
with GitHub
Moritz Beller
@Inventitech
Georgios Gousios
Andy Zaidman

5. How This Paper Came To Be

6. How This Paper Came To Be

7. How This Paper Came To Be

8. Travis What?

9. Travis What?

10. Travis What?

11. P1. Using CI is a widespread best practice

12. P1. Using CI is a widespread best practice
CI has become an integral quality
assurance practice.

13. P. M. Duvall, S. Matyas, and A. Glover, “Continuous
integration: improving software quality and
reducing risk”
P1. Using CI is a widespread best practice
CI has become an integral quality
assurance practice.

14. P1. Using CI is a widespread best practice

15. P1. Using CI is a widespread best practice

16. P1. Using CI is a widespread best practice

17. P1. Using CI is a widespread best practice

18. P1. Using CI is a widespread best practice

19. P1. Using CI is a widespread best practice

20. Around 30% of GITHUB
projects that could
potentially use the free
Travis CI, also make use it.
P1. Using CI is a widespread best practice

21. Around 30% of GITHUB
projects that could
potentially use the free
Travis CI, also make use it.
Caveat: In 2016!
P1. Using CI is a widespread best practice

22. TODO: Add background with
Sun
P1. Using CI is a widespread best practice

23. TODO: Add background with
Sun
P1. Using CI is a widespread best practice
Filter Criteria

24. TODO: Add background with
Sun
P1. Using CI is a widespread best practice
Filter Criteria
●
Non-toy, non-fork projects
(we want the original!)

25. TODO: Add background with
Sun
P1. Using CI is a widespread best practice
Filter Criteria
●
Non-toy, non-fork projects
(we want the original!)
●
> 50 stars

26. TODO: Add background with
Sun
P1. Using CI is a widespread best practice
Filter Criteria
●
Non-toy, non-fork projects
(we want the original!)
●
> 50 stars
●
A longer (> 50) Travis build
history

27. TODO: Add background with
Sun
P1. Using CI is a widespread best practice
Filter Criteria
●
Non-toy, non-fork projects
(we want the original!)
●
> 50 stars
●
A longer (> 50) Travis build
history
E. Kalliamvakou et al.:
“An in-depth study of the
promises and perils of mining
GitHub”

28. TODO: Add background with
Sun
P1. Using CI is a widespread best practice
Filter Criteria
●
Non-toy, non-fork projects
(we want the original!)
●
> 50 stars
●
A longer (> 50) Travis build
history
E. Kalliamvakou et al.:
“An in-depth study of the
promises and perils of mining
GitHub”

29. TODO: Add background with
Sun
P1. Using CI is a widespread best practice
Filter Criteria
●
Non-toy, non-fork projects
(we want the original!)
●
> 50 stars
●
A longer (> 50) Travis build
history
E. Kalliamvakou et al.:
“An in-depth study of the
promises and perils of mining
GitHub”

30. TODO: Add background with
Sun
P1. Using CI is a widespread best practice
Filter Criteria
●
Non-toy, non-fork projects
(we want the original!)
●
> 50 stars
●
A longer (> 50) Travis build
history
E. Kalliamvakou et al.:
“An in-depth study of the
promises and perils of mining
GitHub”
Java
●
423 projects

31. TODO: Add background with
Sun
P1. Using CI is a widespread best practice
Filter Criteria
●
Non-toy, non-fork projects
(we want the original!)
●
> 50 stars
●
A longer (> 50) Travis build
history
E. Kalliamvakou et al.:
“An in-depth study of the
promises and perils of mining
GitHub”
Ruby
●
936 projects
Java
●
423 projects

32. P1. Using CI is a widespread best practice
~

33. TODO: Add background with
Sun
P2. Testing is central to CI

34. TODO: Add background with
Sun
P2. Testing is central to CI
CI is the best practice in which
developers not only integrate their
work into a shared mainline
frequently, but also verify the quality
continuously through testing.

35. TODO: Add background with
Sun
P2. Testing is central to CI
CI is the best practice in which
developers not only integrate their
work into a shared mainline
frequently, but also verify the quality
continuously through testing.
It is known.

37. <TL;DR>...</TL;DR>

38. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

39. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>

40. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>

41. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>

42. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>

43. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>
●
Parsing Log files for Java: Maven, Gradle, Ant

44. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>
●
Parsing Log files for Java: Maven, Gradle, Ant

45. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>
●
Parsing Log files for Java: Maven, Gradle, Ant

46. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>
●
Parsing Log files for Java: Maven, Gradle, Ant
●
Parsing Log files for Ruby: bundler and rake

47. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>
●
Parsing Log files for Java: Maven, Gradle, Ant
●
Parsing Log files for Ruby: bundler and rake

48. P2. Testing is central to CI

49. ●
69% of Java and 87.5% of the Ruby projects:
test runs
P2. Testing is central to CI

50. ●
69% of Java and 87.5% of the Ruby projects:
test runs
1:0 for Ruby!
P2. Testing is central to CI

51. ●
69% of Java and 87.5% of the Ruby projects:
test runs
1:0 for Ruby!
P2. Testing is central to CI
●
Overall, 81% of the projects:
test runs

52. ●
69% of Java and 87.5% of the Ruby projects:
test runs
1:0 for Ruby!
P2. Testing is central to CI
●
Overall, 81% of the projects:
test runs
●
96% of builds:
test runs

53. P2. Testing is central to CI

54. P2. Testing is central to CI

55. P2. Testing is central to CI

56. Testing is the single most important
reason for broken builds, more
prevalent than compile errors, missing
dependencies, build cancellations and
provisioning problems together.
P2. Testing is central to CI

57. Testing is the single most important
reason for broken builds, more
prevalent than compile errors, missing
dependencies, build cancellations and
provisioning problems together.
P2. Testing is central to CI

58. TODO: Add background with
Sun
P3. Testing is language(-type) dependent
Does a dynamically typed language
(like Ruby) need as much testing as a
statically typed language (like Java)?

59. TODO: Add background with
Sun
P3. Testing is language(-type) dependent
Does a dynamically typed language
(like Ruby) need as much testing as a
statically typed language (like Java)?
S. Kleinschmager, S. Hanenberg, R. Robbes, E.
Tanter, A. Stefik: Do static type systems improve the
maintainability of software systems? An empirical
study.

60. P3. Testing is language(-type) dependent

61. P3. Testing is language(-type) dependent

62. P3. Testing is language(-type) dependent

63. P3. Testing is language(-type) dependent
Tests Build Result→

64. P3. Testing is language(-type) dependent
Tests Build Result→

65. P3. Testing is language(-type) dependent
Tests Build Result→

66. P3. Testing is language(-type) dependent
Tests Build Result→

67. Ruby projects have 10x more tests
than Java projects, yet execute faster.
Ruby builds are 4x more likely to fail
than Java projects.
P3. Testing is language(-type) dependent

68. Ruby projects have 10x more tests
than Java projects, yet execute faster.
Ruby builds are 4x more likely to fail
than Java projects.
P3. Testing is language(-type) dependent

69. Building and testing in different
environments delivers additional
value.
P4. Are more build environments helpful?

70. Building and testing in different
environments delivers additional
value.
We don’t know.
P4. Are more build environments helpful?

71. P4. Are more build environments helpful?

72. P4. Are more build environments helpful?

73. P4. Are more build environments helpful?

74. Yes, but 8x more so in Ruby than in
Java.
P4. Are more build environments helpful?

75. Yes, but 8x more so in Ruby than in
Java.
P4. Are more build environments helpful?

76. Moritz Beller @Inventitech
Oops, My Tests Broke the Build:
An Explorative Analysis of Travis CI with GitHub

77. Moritz Beller @Inventitech
Oops, My Tests Broke the Build:
An Explorative Analysis of Travis CI with GitHub

78. Moritz Beller @Inventitech
Oops, My Tests Broke the Build:
An Explorative Analysis of Travis CI with GitHub

79. Moritz Beller @Inventitech
Oops, My Tests Broke the Build:
An Explorative Analysis of Travis CI with GitHub

80. Moritz Beller @Inventitech
Oops, My Tests Broke the Build:
An Explorative Analysis of Travis CI with GitHub