Oleg Chorny discusses Google's approach to release engineering, which aims to reduce manual toil through automation. Some key aspects include establishing frequent release cycles to catch bugs early, empowering individual teams through self-service tools and best practices, integrating automated testing, packaging, deployment and configuration management into the release process, and defining clear policies for approving and controlling releases. The overall goal is to make software releases as simple as pressing a button by establishing the right automated systems and collaboration between developers and release engineers.

1. Oleg Chorny
@Ciklum
Release Engineering:
Google's approach

2. Agenda
Motivation
Engineering
Release Engineering
Automated Release System
Configuration Management

3. Motivation: Toil
● If a machine could accomplish the task just as well as a human, or
the need for the task could be designed away, that task is toil
● If you’re performing a task for the first time ever, or even the second
time, this work is not toil. Toil is work you do over and over
● Toil is interrupt-driven and reactive, rather than strategy-driven and
proactive

4. Motivation: Why toil is bad
● Your career progress will slow down or grind to a halt if you spend
too little time on projects. Too much toil leads to low morale
● Excessive toil makes a team less productive. A product’s feature
velocity will slow if the team is too busy with manual work
● If you build too much toil into your team’s procedures, you motivate
the team’s best engineers to start looking elsewhere for a more
rewarding job

5. Engineering: The work of reducing toil
● Requires human judgment
● Produces a permanent improvement in your service
● Guided by a strategy
● Creative and innovative
● Helps your team or organization handle a larger service, or more
services, with the same level of staffing

6. Release Engineering: High Velocity
● Aim to roll out customer-facing features as quickly as possible
● Frequent releases result in fewer changes between versions. This
approach makes testing and troubleshooting easier
● Some teams perform hourly builds and then select the version to
actually deploy to production
● Other teams have adopted a “Push on Green” release model and
deploy every build that passes all tests

7. Release Engineering: Self-Service Model
● Individual teams can decide how often and when to release new
versions of their products
● Release engineering has developed best practices and tools that
allow product development teams to control and run their own
release processes
● Releases are truly automatic, and only require engineer involvement if
and when problems arise

8. Release Engineering: Budgeting
● Managers don’t always plan and budget for release engineering in
the early stages of a project
● Teams should budget for release engineering resources at the
beginning of the product development cycle
● It’s cheaper to put good practices and process in place early, rather
than have to retrofit your system later

9. Automated Release System: Branching
● All code is checked into the main branch of the source code tree
● Most major projects don’t release directly from the main branch
● Each release has own branch
● Release branch never merges changes back into the main branch
● Bug fixes are submitted to the main branch and then cherry picked
into the release branch for inclusion in the release

10. Automated Release System: Building
● If two people attempt to build the same product at the same revision
number in the source code repository on different machines, we
expect identical results
● The build process is self-contained and must not rely on services that
are external to the build environment
● Build tools must allow us to ensure consistency and repeatability

11. Automated Release System: Testing
● A continuous test system runs unit tests against the code in the main
branch each time a change is submitted
● An independent testing environment that runs system-level tests on
packaged build artifacts
● Because of cherry picks, the release branch may contain a version of
the code that doesn’t exist anywhere on the main branch. During the
release, unit tests are run once again using the release branch

12. Automated Release System: Packaging
● Packages are named (e.g., search/frontend), versioned with a unique
hash, and signed to ensure authenticity
● Labels can be applied to package to indicate a package’s location in
the release process (e.g., dev, canary, or production)
● If a package is labeled as canary, someone subsequently installing
the canary version of that package will automatically receive the
newest version of the package with the label canary

13. Automated Release System: Deployment
● Performed by a general-purpose rollout automation framework
● Framework provides a set of Python classes that can be extended to
support any deployment process
● It has a dashboard that allows for finer control on how the rollout is
performed and provides a way to monitor the rollout’s progress
● A rollout is a logical unit of work that is composed of one or more
individual tasks

14. Automated Release System: Policies
● Several layers of security and access control determine who can
perform specific operations when releasing a project:
○ Approving source code changes
○ Specifying the actions to be performed during the release
○ Creating a new release
○ Deploying a new release
○ Making changes to a project’s build configuration

15. Configuration Management
● In order to simplify deployment, include configuration files and
binaries in the same package
● If you want to release binaries and configuration files independently,
put configuration files into “configuration packages”
● If projects have configuration files that need to change frequently or
dynamically, read configuration files from external store
● Using the mainline for configuration can lead us to skew between the
checked-in and running version of the configuration file

16. Summary
● It is essential that Developers and Release Engineers work together
● Release Engineer needs to understand the intention of how the code
should be built and deployed. Developers shouldn’t build and “throw
the results over the fence” to be handled by Release Engineer
● When equipped with the right tools, proper automation, and well-
defined policies, we shouldn’t have to worry about releasing
software. Releases can be as painless as simply pressing a button

17. Thank you
Oleg Chorny
oleg.chorny@gmail.com