The document outlines the '10 Commandments of Release Engineering' by Dinah McNutt, emphasizing the importance of structured release processes in software development. It highlights key practices such as using source control, automating builds, managing upgrades, and ensuring reproducibility. The commandments serve as guidelines for release engineers to enhance collaboration and efficiency in moving software from development to operations.

1. The 10 Commandments of
Release Engineering
Dinah McNutt
Google, Inc.

2. Release Engineering
“Accelerating the path from
development to operations”

3. Overview
● These commandments are FROM test engineers/system
administrators TO release engineers
● Concepts apply to software products for both internal and
external customers
● Ideas presented are my own, not necessarily Google's

4. Background
● Release processes are usually an afterthought
● Most systems do the minimum required to "get it done"
● Release processes should be treated as products in their
own right
● There is often a big disconnect between the developer
writing the code, the person writing tests, and the system
admin who installs it

5. Steps in Release Process
Check out
code
Compile
Test
Release

6. The Real Process
Check out
code
Bug Fixes
Compile
Unit Tests
Canaries
Deployment
Package
System
Tests
More System
Tests

7. The Real, Real Process
Build Artifacts
Check out
code
Bug Fixes
Compile
Reports
Unit Tests
Package
Canaries
System
Tests
Alerts/Monitoring
Deployment
More System
Tests

8. Release Process Features
● Reproducible
● Tracking of changes and the ability to understand what is in
a new version of the product or product component
● An identification mechanism (e.g. build ID) that uniquely
identifies what is contained in a package or product
● Implementation and enforcement of policy and procedures
● Management of upgrades and patch releases

9. I - Thou shalt use a source code
control system.
● Everything needed to release should be under source
control
○ source code
○ build files
○ build tools
○ documentation
● Doesn't matter what you use, just use something!

10. Reproducible Build Environment
● Not usually checked into a SCR, but still may need to be
recreated:
○ Operating System
○ Compilers
○ Build tools
● Possible solutions:
○ Backups
○ Installation servers

11. II - Thou shalt use the right tool(s) for
the job.
Complex projects may require multiple build tools
Examples:
● make for C and C++ - the dependency checking is crucial
● ant for java
● scripting languages (bash, python, etc.)
Unnecessary complexity is a sin.

12. III - Thou shalt write portable and lowmaintenance build files
● Plan to support multiple architectures and OS versions
● Use centralized configuration files for definitions common to
build files
○ Compiler options will change between architectures
○ Editing hundreds of files for a single change is no fun
● Provide template files so developers can easily create new
build files
Measure twice, cut once.

13. IV - Thou shalt use a release process
that is reproducible
And automated...
And unattended...
And reproducible...
● Adopt a continuous build policy
● Leverage open source tools like Jenkins and Puppet
Automation is a virtue.

14. V - Thou shalt use a unique build ID
● Must provide enough information so the build can be
uniquely identified and reproduced
● Examples:
○ Date
○ Repository revision
○ Release version
● Must be easily obtainable
○ Included in packaging
○ Embedded in binaries
Knowing where you came from is a virtue.

15. Build IDs
● 20131008_RC05
● 164532_20131008_2_RC00
● 164532_0_RC02

16. VI - Thou shalt use a package manager
● Auditing
● Leverage installation/upgrade/removal capabilities
● Package summary (who, what, when, etc.)
● Built-in version tracking and dependency checking
● Manifest (ok, tar -tf gives you that.)
● Use native package managers when possible
tar is not a package manager...

17. VII - Thou shalt design an upgrade
process before releasing version 1.0
● Packaging decisions can affect the ability to upgrade
● Design an upgrade process at the same time you are
designing an installation process
Not thinking ahead is a sin.

18. VIII - Thou shalt provide a detailed log
of what thou hath done to my machine
● Installing/Patching/Upgrading/Removing software should
provide a detailed log of what is happening
● Provide the ability to unpack and inspect the packages
without installing
● Ideally there should be a "do nothing" option so I can see
what is going to happen first

19. IX - Thou shalt provide a complete
install/upgrade/patch/uninstall process
● Totally automated process
● Rollback AND roll forward
● Packages should be relocatable
Playing nice with others is a virtue.

20. X - Test Engineer: Thou shalt apply
these laws to thyself
● All of these commandments can be applied to development
and execution of tests

21. Shameless Plug
The 2014 USENIX Summit on Release Engineering
June 20, 2014 Philadelphia
“From Dev to DevOps”