Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Why do Automated Builds Break? An Empirical Study (ICSME 2014)

579 views

Published on

To detect integration errors as quickly as possible, organizations use automated build systems. Such systems ensure that (1) the developers are able to integrate their parts into an executable whole; (2) the testers are able to test the built system; (3) and the release engineers are able to leverage the generated build to produce the upcoming release. The flipside of automated builds is that any incorrect change can break the build, and hence testing and releasing, and (even worse) block other developers from continuing their work, delaying the project even further. To measure the impact of such build breakage, this empirical study analyzes 3,214 builds produced in a large software company over a period of 6 months. We found a high ratio of build breakage (17.9%), and also quantified the cost of such build breakage ranging from 904.64 to 2034.92 man-hours. Interviews with 28 software engineers from the company helped to understand the circumstances under which builds are broken and the effects of build breakages on the collaboration and coordination of teams. We quantitatively investigated the main factors impacting build breakage and found that build failures correlate with the number of simultaneous contributors on branches, the type of work items performed on a branch, and the roles played by the stakeholders of the builds (for example developers vs. integrators).

http://mcis.polymtl.ca/publications/2014/icsm14_noureddine.pdf

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Why do Automated Builds Break? An Empirical Study (ICSME 2014)

  1. 1. Why do Automated Builds Break? An Empirical Study M C IS Noureddine Kerzazi Foutse Khomh Bram Adams http://imgs.steps.dragoart.com/how-to-draw-a-flying-dragon-http://www.printed-editions.com/upload/standard/Harold_Edgerton_Pigeon_Release_69.jpg dragon-in-flight-step-8_1_000000102347_5.gif
  2. 2. Imagine 2,000 developers working in the same branch, and one breaks the build, then you end up with 1,999 unhappy people! [T. Zimmermann @ SCAM 2014]
  3. 3. 3 What is "breaking the build"?
  4. 4. 3 What is "breaking the build"? trunk ...
  5. 5. branch 1 3 What is "breaking the build"? trunk ...
  6. 6. branch 1 branch 2 3 What is "breaking the build"? trunk ...
  7. 7. branch 1 branch 2 3 What is "breaking the build"? trunk ...
  8. 8. test build branch 1 branch 2 3 What is "breaking the build"? integrate trunk ...
  9. 9. test build branch 1 branch 2 3 What is "breaking the build"? integrate trunk ...
  10. 10. branch 1 branch 2 3 What is "breaking the build"? trunk integrate test build ...
  11. 11. branch 1 branch 2 3 What is "breaking the build"? trunk integrate test build ...
  12. 12. branch 1 branch 2 3 What is "breaking the build"? trunk integrate test build integrate test build down-time ...
  13. 13. branch 1 branch 2 3 What is "breaking the build"? trunk down-time team-level breakage ...
  14. 14. branch 1 branch 2 3 What is "breaking the build"? trunk down-time integrate test build team-level breakage ...
  15. 15. branch 1 branch 2 3 What is "breaking the build"? trunk down-time merge team-level breakage ...
  16. 16. branch 1 branch 2 3 What is "breaking the build"? trunk down-time merge team-level breakage integrate test build ...
  17. 17. branch 1 branch 2 3 What is "breaking the build"? trunk down-time merge down-time team-level breakage integrate test build ...
  18. 18. branch 1 branch 2 3 What is "breaking the build"? trunk merge sync down-time down-time team-level breakage X integrate test build ...
  19. 19. branch 1 branch 2 3 What is "breaking the build"? trunk merge sync down-time down-time team-level breakage X integrate test build integrate test build ...
  20. 20. branch 1 branch 2 3 What is "breaking the build"? trunk merge sync down-time down-time down-time team-level breakage X integrate test build integrate test build ...
  21. 21. branch 1 branch 2 3 What is "breaking the build"? trunk merge sync down-time down-time down-time team-level breakage X integrate test build integrate test build integrate test build ...
  22. 22. branch 1 branch 2 3 What is "breaking the build"? trunk merge sync merge down-time down-time down-time team-level breakage X integrate test build ...
  23. 23. branch 1 branch 2 3 What is "breaking the build"? trunk merge sync merge down-time down-time down-time team-level breakage X integrate test build integrate test build ...
  24. 24. branch 1 branch 2 3 What is "breaking the build"? trunk merge sync merge down-time down-time down-time team-level breakage project-level breakage X ...
  25. 25. 4 Build Breakage should be Avoided! you"are"stuck your"team" members"are"stuck testers"are" stuck (a0er"merge) other+teams" deadlines"slip are"stuck morale"drops
  26. 26. 5 RQ1. How does Build Breakage Impact a Project? RQ2. Why does the Build Break? RQ3. Which Factors Impact Build Breakage?
  27. 27. Web-Based, Financial System 6 200 employees >1.5 MLOC .NET (8,524 files) 25 branches 6 inter-dependent projects 2 geographical sites agile development across different branches 6 months of data
  28. 28. triangula9on measure interview model 7
  29. 29. 8 RQ1. How does Build Breakage Impact a Project? RQ2. Why does the Build Break? RQ3. Which Factors Impact Build Breakage?
  30. 30. Median of 8.3% Broken Builds per Day (avg. of 22.3%) 9 %broken builds 100 100" 90 90" 80 80" 70 70" 60 60" 50 50" 40 40" 30 30" 20 20" 10 10" 0" 18,07,13" 25,07,13" 01,08,13" 08,08,13" 15,08,13" 22,08,13" 29,08,13" 05,09,13" 12,09,13" 19,09,13" 26,09,13" 03,10,13" 10,10,13" 17,10,13" 24,10,13" 31,10,13" 07,11,13" 14,11,13" 21,11,13" 28,11,13" 05,12,13" 12,12,13" 19,12,13" 26,12,13" 02,01,14" 09,01,14" 16,01,14" 23,01,14" 30,01,14" 0
  31. 31. 1000 1000" 900 900" 800 800" 700 700" 600 600" 500 500" 400 400" 300 300" 200 200" 100" 0" Architecture) Branch-2) Branch-3) Branch-4) Branch-5) Branch-6) Branch-7) Branch-8) Branch-9) QuickFix) Branch-18) PreRelease) Branch-10) Branch-11) Branch-12) Branch-13) Branch-14) Branch-15) Branch-16) Branch-20) Branch-21) Trunk) Branch-23) Branch-24) Branch-25) DownDme)by)branch)(min)) Average Median Downtime of 1 Hour per Branch 10 #min. re-architecting can cause very long downtime 100
  32. 32. Estimating Cost of Team-Level Build Breakage for Branch 11 X integrate test build integrate test build time integrate test build integrate + test build time + ... $$$(branch) in man-hours=
  33. 33. Estimating Cost of Team-Level Build Breakage for Project 12 $$$(project) in man-hours = $$$(branch 1) in man-hours + $$$(branch 2) in man-hours + ...
  34. 34. Total of 1.25 to 2.83 Man- Months Wasted (in 6 months) 13 0 100 200 300 400 500 Branch−18 Branch−2 Branch−5 QuickFix Branch−12 Branch−21 Branch−10 Branch−25 Branch−3 Branch−11 Branch−23 Branch−15 Branch−7 Branch−14 Branch−24 PreRelease Branch−4 Branch−6 Branch−20 Branch−13 Branch−16 Branch−8 Arch. Branch−9 Trunk re-architecting fixing trunk upper bound lower bound M/H
  35. 35. 14 1 hour of downtime (avg.) per branch >=1.25 man-months lost
  36. 36. RQ2. Why does the Build Break? 1 hour of downtime (avg.) per branch 14 >=1.25 man-months lost
  37. 37. Semi-Structured Interviews 15 architects technical leads integrator front-end developers back-end developers testers >3 years of experience 20th percentile most successful/ broken builds 50 minutes
  38. 38. Analysis of Interviews 16 awareness (Q1-Q4) causes (Q5-Q6) prevention (Q7-Q8) X Q1. Q2. ... card sort
  39. 39. 25% of Interviewees Track the Build Results I switched off the Build notifications to avoid any distraction. I check the results of my builds only in predefined time windows within the day. I’m always aware about the build notification, especially the red ones. When a build notification for a branch shows up, I examine not only that build, but also the history of build failure on that branch. 17
  40. 40. Awareness of a Breakage Takes on Average 3 Hours When we see in the work items system tracker that the developer is done and we do not see the build result, then we have to communicate with the concerned developer to verify where the latest code to test is. 18
  41. 41. Breakage Takes on Average 1 Hour to Resolve > 19 concerned about project's overall consistency concerned about own feature/bug fix
  42. 42. 3 Major Causes of Build Breakage 20 forgetting to commit certain files (libraries, templates, configuration files, ...) committing from the wrong workspace (branch) feature libA 1.0 libB 2.1 feature libA 1.0 libB 1.5 build scripts do not specify transitive library dependencies
  43. 43. Developers Active in 4 to 15 Branches in Parallel I was assigned bugs on a specific project within the overall solution. I unload all other projects from the solution to focus only on one project. I am a nomad moving from a branch to another. I map one workspace by branch to avoid check-in work in progress related to other branch. 21
  44. 44. Only 12% has Routine for Avoiding Breakages Before I check- in, I always compare diligently my pending changes with the latest version from the server. In this way I am not wasting team members’ time with broken builds. New developers joining a development project should be required to learn the basic practices of code committing, as part of the process known as onboarding 22
  45. 45. 23 1 hour of downtime (avg.) per branch >=1.25 man-months lost 25% track CI results detection takes 3 h caused by missing file (dependencie)s and accidental commits active in 4-15 branches in parallel
  46. 46. 23 RQ3. Which Factors Impact Build Breakage? 1 hour of downtime (avg.) per branch >=1.25 man-months lost 25% track CI results detection takes 3 h caused by missing file (dependencie)s and accidental commits active in 4-15 branches in parallel
  47. 47. Statistical Model of Impact 24 vs. vs. vs. vs. vs. local vs. remote code/file churn commit vs. merge feature vs. bug fix vs. integration P1 P2 P3 P4 day of week/time of day breakage?
  48. 48. Factors having Higher Breakage Role NumberofContributors Work_Items Build_Type Timeline.P1_P4. Churn..Code. Churn..Files. Working_Day Working_Hour Geog_Distance ● ● mtcars.rf ● ● ● ● ● ● ● ● 10 20 30 40 50 25 larger team %incMSE AUC: 0.78
  49. 49. Factors having Higher Breakage Role NumberofContributors Work_Items Build_Type Timeline.P1_P4. Churn..Code. Churn..Files. Working_Day Working_Hour Geog_Distance ● ● mtcars.rf ● ● ● ● ● ● ● ● 10 20 30 40 50 25 P1 P2 P3 P4 larger team P1 %incMSE AUC: 0.78
  50. 50. Factors having Higher Breakage Role NumberofContributors Work_Items Build_Type Timeline.P1_P4. Churn..Code. Churn..Files. Working_Day Working_Hour Geog_Distance ● ● mtcars.rf ● ● ● ● ● ● ● ● 10 20 30 40 50 25 P1 P2 P3 P4 time of day larger team P1 late commits, longer downtime %incMSE AUC: 0.78
  51. 51. Factors having Higher Breakage Role NumberofContributors Work_Items Build_Type Timeline.P1_P4. Churn..Code. Churn..Files. Working_Day Working_Hour Geog_Distance ● ● mtcars.rf ● ● ● ● ● ● ● ● 10 20 30 40 50 25 vs. vs. vs. vs. vs. P1 P2 P3 P4 time of day larger team P1 late commits, longer downtime %incMSE architects AUC: 0.78
  52. 52. Factors having Higher Breakage Role NumberofContributors Work_Items Build_Type Timeline.P1_P4. Churn..Code. Churn..Files. Working_Day Working_Hour Geog_Distance ● ● mtcars.rf ● ● ● larger team ● ● ● ● ● 10 20 30 40 50 25 vs. vs. vs. vs. vs. feature vs. bug fix vs. integration P1 P2 P3 P4 time of day integration > feature > fix P1 late commits, longer downtime %incMSE architects AUC: 0.78
  53. 53. Factors having Higher Breakage Role NumberofContributors Work_Items Build_Type Timeline.P1_P4. Churn..Code. Churn..Files. Working_Day Working_Hour Geog_Distance ● ● mtcars.rf ● ● ● larger team ● ● ● ● ● 10 20 30 40 50 25 vs. vs. vs. vs. vs. feature vs. bug fix vs. integration commit vs. merge P1 P2 P3 P4 time of day integration > feature > fix merge > commit P1 late commits, longer downtime %incMSE architects AUC: 0.78
  54. 54. Factors having Higher Breakage Role NumberofContributors Work_Items Build_Type Timeline.P1_P4. Churn..Code. Churn..Files. Working_Day Working_Hour Geog_Distance ● ● mtcars.rf ● ● ● larger team ● ● ● ● ● 10 20 30 40 50 25 vs. vs. vs. vs. vs. feature vs. bug fix vs. integration commit vs. merge P1 P2 P3 P4 code churn file churn time of day integration > feature > fix merge > commit more churn P1 late commits, longer downtime %incMSE architects AUC: 0.78
  55. 55. Factors having Higher Breakage Role NumberofContributors Work_Items Build_Type Timeline.P1_P4. Churn..Code. Churn..Files. Working_Day time of day Working_Hour Geog_Distance ● ● mtcars.rf ● ● ● larger team ● ● ● ● ● 10 20 30 40 50 25 vs. vs. vs. vs. vs. feature vs. bug fix vs. integration commit vs. merge P1 P2 P3 P4 code churn file churn local vs. remote integration > feature > fix merge > commit more churn local teams P1 late commits, longer downtime %incMSE architects AUC: 0.78
  56. 56. Factors having Higher Breakage Role NumberofContributors Work_Items Build_Type Timeline.P1_P4. Churn..Code. Churn..Files. day of week time of day Working_Day Working_Hour Geog_Distance ● ● mtcars.rf ● ● ● larger team ● ● ● ● ● 10 20 30 40 50 25 vs. vs. vs. vs. vs. feature vs. bug fix vs. integration commit vs. merge P1 P2 P3 P4 code churn file churn local vs. remote integration > feature > fix merge > commit more churn local teams P1 late commits, longer downtime %incMSE architects AUC: 0.78
  57. 57. Factors having Higher Breakage Role NumberofContributors Work_Items Build_Type Timeline.P1_P4. Churn..Code. Churn..Files. day of week time of day Working_Day Working_Hour Geog_Distance ● ● mtcars.rf ● ● ● larger team ● ● ● ● ● 10 20 30 40 50 25 vs. vs. vs. vs. vs. feature vs. bug fix vs. integration commit vs. merge P1 P2 P3 P4 code churn file churn local vs. remote integration > feature > fix merge > commit more churn local teams P1 late commits, longer downtime %incMSE architects AUC: 0.78
  58. 58. 26 1 hour of downtime (avg.) per branch >=1.25 man-months lost 25% track CI results detection takes 3 h caused by missing file (dependencie)s and accidental commits active in 4-15 branches in parallel ro l e team size in branch type of work item/ build timeline
  59. 59. branch 1 branch 2 Trunk- vs. Branch-based Development 3 What is "breaking the build"? trunk merge sync merge down-time down-time down-time team-level breakage project-level breakage X ... triangula9on measure interview model 7 Total of 1.25 to 2.83 Man- Months Wasted (in 6 months) upper bound lower 13 0 100 200 300 400 500 bound Branch−18 Branch−2 Branch−5 QuickFix Branch−12 Branch−21 Branch−10 Branch−25 Branch−3 Branch−11 Branch−23 Branch−15 Branch−7 Branch−14 Branch−24 PreRelease Branch−4 Branch−6 Branch−20 Branch−13 Branch−16 Branch−8 Arch. Branch−9 Trunk 26 1 hour of downtime (avg.) per branch >=1.25 man-months lost 25% track CI results detection takes 3 h caused by missing file (dependencie)s and accidental commits active in 4-15 branches in parallel role team size in branch type of work item/ build timeline

×