Software Build processes and Git


Published on

A presentation given at DevOps DownUnder May 2010. Code at

Updated Nov 2010 for OSDC Melbourne

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Software Build processes and Git

  1. 1. Build Audit and Processes A line of sight from DEV to PROD
  2. 2. Alec Clews Voga Consulting Licensed under Creative Commons Attribution-Share Alike 2.5 Australia License
  3. 3. Summary <ul><li>Build software in an audited and managed environment as part of Configuration Management (CM)
  4. 4. Using Open Source Tools (Make and Git) as a framework for a controlled build process </li></ul>
  5. 5. Agenda <ul><li>The different types of build and the place of the system build
  6. 6. Auditing and logging requirements for system build
  7. 7. Implementing a CM System Build Framework </li><ul><li>The Makefile
  8. 8. The Driver script
  9. 9. The output </li></ul></ul>
  10. 10. Motivation <ul><li>This paper takes a “commercial” view on the process of software change management
  11. 11. Desire to provide a business value quickly
  12. 12. Communicate effectively between team members
  13. 13. Reduce the amount of unsatisfying project work </li><ul><li>rework, report writing,etc... </li></ul></ul>
  14. 14. The Change Lifecyle <ul><li>Document and agree the change
  15. 15. Implement the change (hack the code and desktop build/testing)
  16. 16. Integration build and unit test
  17. 17. System Build in a controlled environment
  18. 18. System Testing (or UAT, or beta, or ...)
  19. 19. Release for distribution or production </li></ul>
  20. 20. Continuous Integration vs. System Builds <ul><li>Continuous Integration (c.f. Martin Fowler) </li><ul><li>Builds rapidly and frequently
  21. 21. Runs unit tests
  22. 22. Sets up for unit testing framework
  23. 23. Happens automagically
  24. 24. May only build small pieces
  25. 25. May include code instrumentation and additional debugging options </li></ul></ul>
  26. 26. System Builds <ul><li>Build for production, or pre-production </li><ul><li>Compiler optimisation , different configurations etc. </li></ul><li>Runs on a different schedule
  27. 27. Needs documentation
  28. 28. May have large scope </li></ul>
  29. 29. How might the system build be done? <ul><li>Fetch the sources and build them
  30. 30. Ask the developer for already built software!
  31. 31. Issues with these approaches. How can we answer such questions as? </li><ul><li>Which source versions were used
  32. 32. Under what configuration was this built? </li><ul><li>Compiler switches, which libraries and tools (the build configuration ) </li></ul><li>Can we “re-create the build”? </li></ul></ul>
  33. 33. “ re-create the build” is a myth <ul><li>After problems often asked to re-create the environment for forensic testing
  34. 34. But if we re-run the build process that does not assure that we will get the same result </li><ul><li>Version No, dates and time change: trivial difference masks possibly other major changes in our files
  35. 35. We usually don't have a record of the previous build configuration and can't re-create it anyway </li></ul><li>Result: we can't reliably get the same result twice </li></ul>
  36. 36. Consistent Build Process <ul><li>It is possible to define a consistent build process by using tools such as Make, Ant or CONS and some additional scaffolding
  37. 37. Must be done in a controlled environment
  38. 38. Must be done with pre-defined scripts and documented process
  39. 39. Scripts should be under version control
  40. 40. A record should be keep of all system builds </li></ul>
  41. 41. Baseline build results <ul><li>Use the consistent process
  42. 42. After the system build save the build directory tree (source code, makefiles, build log, built files) into an archive (consider burning a CD) </li><ul><li>This is our built baseline. We can always take this off the shelf and see what software we used to “re-create the build”. </li></ul><li>However it's often hard to identify source versions and what build environment was used </li></ul>
  43. 43. Manage Build Environments <ul><li>Consider building on virtual hosts
  44. 44. The host image can be archived as changes are made </li></ul>
  45. 45. There is a better way <ul><li>As we build new files identify </li><ul><li>Source files and their version
  46. 46. Tools and their settings
  47. 47. Configuration used (platforms, packages etc. ) </li></ul><li>At the completion of a successful build we can preserve the outputs in Version Control </li><ul><li>Including a complete audit trail of the environment , the build log and the input files </li></ul></ul>
  48. 48. The result <ul><li>In our version control system we have a complete audit trail of each system build
  49. 49. We can extract our built files at any time – now we can “re-create the build” just by fetching files from version control
  50. 50. If our change ticket system is integrated into version control we can identify which changes when into a build </li></ul>
  51. 51. Why do we want to do this <ul><li>We want to close the change management loop </li><ul><li>A “Line of sight” from change request to deployment </li></ul><li>Improved compliance </li><ul><li>N.B. May be a legal requirement to keep the external auditors happy. Sometimes process auditors can shut down the business!
  52. 52. Our KPI may depend on SLAs and process outcomes </li></ul></ul>
  53. 53. Outcomes <ul><li>Better forensics – what is really running in production?
  54. 54. Solid documentation
  55. 55. Reduced rework </li></ul>
  56. 56. Example Implementation <ul><li>There are a variety of ways to this
  57. 57. Following example uses </li><ul><li>A simple C project (5 source files)
  58. 58. Make
  59. 59. Git distributed version control tool </li></ul><li>Commercial products also available. </li><ul><li>e.g. Atlassian Bamboo, Electric Cloud </li></ul></ul>
  60. 60. Git <ul><li>Now a major player in Version Control market
  61. 61. Baselines are created using tags. A tag is a label attached to a commit </li></ul>
  62. 62. Some concepts and words <ul><li>Build Tags. Tags created for each build we perform. c.f. Build Baseline
  63. 63. A Build Configuration. Documentation of the environment in which the build is being done (e.g. Platform, compiler, library versions...) </li></ul>
  64. 64. The Makefile <ul><li>To create the audit trail we must know ALL dependencies </li><ul><li>i.e. All the source files used to build a target file </li></ul><li>Peter Miller already did the hard work for me </li><ul><li>His paper explains an effective way to implement makefiles with a complete dynamic dependencies
  65. 65. A side affect is that your makefile is more efficient! </li></ul></ul>
  66. 66. Dependencies in C <ul><li>A list of input files that an output file depends on. e.g. </li><ul><li>a.o: a.c.a.h b.h
  67. 67. main: main.o a.o b.o </li></ul><li>Dependencies can change any time we edit the source code, including editing any header files
  68. 68. We must re-calculate the dependencies at make time if the source files or header files change </li></ul>
  69. 69. Dynamic dependency <ul><li># Generate a dependency file
  70. 70. $(DEP_DIR)/%.d: $(SRC_DIR)/%.c </li><ul><li>$(CC) $(CFLAGS) -MM -MG -MT $(OBJ_DIR)/$*.o -MT
  71. 71. $(DEP_DIR)/$*.d -MF $@ $< </li></ul><li>Creates a file that contains
  72. 72. o/a.o d/a.d: src/a.c h/a.h
  73. 73. Include the dependency file in the makfile
  74. 74. # Find the name of all the Dynamic dependency file and include them
  75. 75. include $(patsubst $(OBJ_DIR)/%, $(DEP_DIR)/%, $(patsubst %.o,%.d, $(OBJ))) </li></ul>
  76. 76. Built From List (BFL) <ul><li>When we preserve a new target we want to record </li></ul><ul><ul><li>Unique sha1 of each source file used
  77. 77. Sha1 of settings and environment </li></ul></ul><ul><li>This is the Built From List </li><ul><li>The list the target is built from </li></ul></ul>
  78. 78. Calculating BFL <ul><li>When a file is build from complete dependency list we can also discover the CM data for the dependencies our target is built from </li></ul>echo -n &quot;BLD_FLAGS [$(CC) $(CFLAGS)]:&quot; > b/$(@F).bfl echo $(CC) $(CFLAGS) | git hash-object --stdin >> b/$(@F).bfl echo &quot;BLD_ENV: $(BUILDCONFIG)&quot; >> b/$(@F).bfl for i in $^ $(MAKEFILE_LIST); do echo -n &quot;$$i:&quot; >> b/$(@F).bfl git hash-object $$i >> b/$(@F).bfl done
  79. 79. Example BFL BLD_FLAGS [cc -ansi] a275bd53898b96ff56682992cfef600c8d9172b6 BLD_ENV 8b137891791fe96927ad78e64b0aad7bded08bdc src/a.c fbb2a1faee180bd3c81d6d00bb39d1bb68b51725 h/a.h 8ed2304c36b99ef1fd9a1f22a41ee2afe4eb995e makefile 34e0b0adb203123c8ebfdf2c4fb059ff0c942e27 d/main.d 3045b0da533f900bf1ab044de22725475d4d171c d/a.d 8bbf937c5cc911242a3b3fdc531c5d1a5cf7f37c d/b.d bba81a7805d40e267b139a01f69c15c676d0a82d
  80. 80. Storing the target & BFL <ul><li>Every time we commit a new built target to the repository </li><ul><li>Add the BFL to the repository
  81. 81. Add the new target to the repository </li></ul></ul>git add b/$(@F).bfl git add $(@)
  82. 82. Driver script sets up <ul><li>Build ID – derived from sha1 + timestamp
  83. 83. The Build Configuration. A simple string representing the build environment
  84. 84. host name of build machine – usually not important unless problem </li></ul>
  85. 85. DEMO
  86. 86. Further work <ul><li>Better Error Checking!!
  87. 87. Save and re-use built intermediate files </li><ul><li>(but see constraints later ) </li></ul><li>Update tickets with “Fixed In Build” information </li></ul>
  88. 88. Further work contd. <ul><li>Integrate with meta make tools </li><ul><li>e.g. cmake, autoconf/automake, ... </li></ul><li>Automate the production of release notes and other reports from the BFL and other version control information </li></ul>
  89. 89. How to re-use targets in later Builds <ul><li>Validate sha1 values on source files on original BFL against source files in Build areas – must match
  90. 90. Check build configuration for target on build tag against current environment – must match
  91. 91. Check command line switches – must match
  92. 92. We don't care about dates and time </li></ul>
  93. 93. Outstanding Questions <ul><li>Does this scale? </li><ul><li>More complex directory structures
  94. 94. Multiple languages (e.g. grammar files & assembler) </li></ul><li>What about languages and tools with hard to calculate dependencies (e.g. Java, code generators,...) </li></ul>
  95. 95. Outstanding Questions contd <ul><li>Which other version control tools work </li><ul><li>SVN version implemented in 2006 </li></ul><li>Will this work across platforms </li><ul><li>differences in tools
  96. 96. problems because text files change? (tool dependent)
  97. 97. .... </li></ul></ul>
  98. 98. Further reading <ul><li>Example scripts written for this paper </li></ul> <ul><li>“ Recursive Make Considered Harmful” by Peter Miller
  99. 99. “ Continuous Integration” by Martin Fowler </li></ul>