The document discusses key topics from a session on build and release engineering, including continuous integration, continuous delivery, and continuous deployment. It defines these terms and lists the main steps and principles associated with each. Specific topics covered include software release principles, configuring management, deployment pipelines, testing approaches, and managing environments and releases of complex systems. The target audience for these concepts is listed as various engineering and operations roles.

1. ì
Build
&
Release
Engineering
By:
Pranesh
Vi.al
CG
h.p://linkedin.com/in/praneshvi.al

2. Main
Agenda
of
this
session
h"p://linkedin.com/in/praneshvi"al
2

3. Key
Takeaways
from
this
session
ì So<ware
Release
Principles
&
its
importance.
ì Not
to
expect
results
from
“Day
1”.
It’s
a
slow
process.
Takes
months
or
at
Lmes
even
years
to
reach
perfecLon
levels.
ì What
is
“ConLnuous
IntegraLon”?
ì Steps
associated
with
“ConLnuous
IntegraLon”?
ì What
is
a
“ConLnuous
Delivery”?
ì Steps
associated
with
“ConLnuous
Delivery”?
ì What
is
a
“ConLnuous
Deployment”?
ì Release
Engineering
Steps.
ì Main
Focus
of
Release
Engineering.
h"p://linkedin.com/in/praneshvi"al
3

4. Key
Takeaways
from
this
session
ì ConfiguraLon
Management.
ì Principles
of
Managing
ApplicaLon
ConfiguraLon.
ì Its
not
just
about
the
tools.
Its
about
the
big
“Cultural
Change”
that
everyone
has
to
go
through
.
ì Release
Engineering
Architecture.
ì Deep
dive
into
commit
/
component
builds.
ì AnL-­‐Pa.erns
in
commit
/
component
builds.
ì Aim
of
Deployment
Pipeline.
ì AnL-­‐Pa.erns
of
Deployment
Pipeline.
ì Best
qualiLes
of
a
Deployment
Pipeline.
h"p://linkedin.com/in/praneshvi"al
4

5. Key
Takeaways
from
this
session
ì Deployment
on
Large
Scale
ProducLon
Environments.
ì Why
Automated
Deployment
is
an
indispensable
goal
???
ì Deep
dive
into
TesLng
Quadrant.
ì Deep
Dive
into
Automated
Acceptance
Tests.
ì Few
more
Generic
AnL-­‐Pa.erns.
ì AdapLon
of
Release
Engineering.
ì Tools
Used
at
each
of
the
stages.
ì Managing
Environments.
ì Managing
Releases
of
complex
systems.
h"p://linkedin.com/in/praneshvi"al
5

6. Key
Takeaways
from
this
session
ì Steps
involved
to
build
a
deployment
pipeline.
ì Everyone
is
an
equal
contributor
to
reach
“ConLnuous
Deployment”
stage.
ì MulL-­‐Tenancy
Cloud.
ì MulL-­‐Instance
ApplicaLons.
ì Zero
DownLme
Deployment.
ì ExtracLng
the
server
logs.
ì Next
generaLon
tools
in
the
DevOps
world.
ì PreparaLon
for
the
Release.
ì Maturity
of
a
“Deployment
Pipeline”.
h"p://linkedin.com/in/praneshvi"al
6

7. Target
Audience
ì Product
Managers
/
Business
Analysts.
ì Engineering
Managers.
ì Development
Teams.
ì Quality
Engineering
Teams.
ì System
Admins.
ì Service
Engineering
/
ProducLon
Engineering
Teams.
ì On-­‐site
Coordinators.
h"p://linkedin.com/in/praneshvi"al
7

8. Software
Release
Principles
&
its
importance.
ì Should
be
Fast
/
Predictable
/
Repeatable.
ì Automate
almost
everything.
ì Keep
everything
in
Version
Control.
ì If
it
hurts,
do
it
more
o<en.
ì Improve
the
quality
of
the
Builds.
Push
the
boundaries
of
test
automaLon
so
that
quality
is
not
compromised.
ì Done
means
the
so<ware
is
released
and
has
reached
the
producLon.
h"p://linkedin.com/in/praneshvi"al
8

9. What
is
Deployment?
ì Run
the
same
commands
in
several
hosts.
ì Install
the
required
so<ware
from
a
central
repository
so
that
the
end
state
of
every
host
is
same.
ì Based
on
the
host-­‐type,
install
the
required
packages.
ì End
goal
is
to
bring
all
the
hosts
of
the
respecLve
host-­‐type
in
the
same
state.
h"p://linkedin.com/in/praneshvi"al
9

10. What
is
“Continuous
Integration”?
ì Building,
deploying
and
tesLng
the
applicaLon.
h"p://linkedin.com/in/praneshvi"al
10

11. Steps
associated
with
“Continuous
Integration”?
ì Integrate
all
the
components
at
regular
intervals.
ì Binaries
built
only
once
and
used
in
all
the
environments.
ì Execute
Unit
Tests
before
the
packages
are
generated.
ì Deploy
the
latest
packages
onto
an
INT
environment.
ì Run
the
automated
tests,
validate
the
test
results.
ì Track
the
Code
quality
criteria
such
as
staLc
code
analysis,
test
coverage.
ì Deploying
to
other
environments
with
a
pre-­‐defined
frequency.
ì Repeat
the
process.
ì It’s
a
pracGce,
not
a
tool
h"p://linkedin.com/in/praneshvi"al
11

12. What
is
a
“Continuous
Delivery”?
ì Building,
deploying,
tesLng,
promoLon
of
the
applicaLon
to
the
next
environment.
ì Image
credit:
h.p://www.axonivy.com/
h"p://linkedin.com/in/praneshvi"al
12

13. Steps
associated
with
“Continuous
Delivery”?
ì Steps
followed
in
“ConLnuous
IntegraLon”.
ì AutomaLc
promoLon
/
deployment
to
other
environments
(QA,
Staging,
Performance
etc…)
upon
successful
execuLon
of
automated
tests.
ì ApplicaLon
is
always
ready
to
deploy
to
producLon
through
a
largely
automated
process.
h"p://linkedin.com/in/praneshvi"al
13

14. What
is
a
“Continuous
Deployment”?
ì Building,
deploying,
tesLng,
promoLon
of
the
applicaLon
to
the
producLon.
ì Image
credit:
Yassal
Sundman
h"p://linkedin.com/in/praneshvi"al
14

15. Release
Engineering
Principles
&
its
importance.
ì Minimize
Cycle
Time
from
check-­‐in
to
producLon
push.
ì Everybody
is
responsible
for
the
Delivery
Process.
ì ConLnuous
Improvement.
ì AcLviLes
such
as
SCM,
Release
planning,
Automated
Deployment,
Acceptance
TesLng,
Performance
TesLng
are
NOT
SECONDARY.
ì Generates
no
revenue
Lll
its
in
the
hands
of
the
end-­‐users.
ì Customer
SaLsfacLon
through
early
&
conLnuous
delivery
of
so<ware.
h"p://linkedin.com/in/praneshvi"al
15

16. Release
Engineering
Steps
ì SCM
(Git,
SVN,
Clearcase).
ì Automated
Commit
/
Component
Builds
(Generate
Binaries).
ì Generate
Code
Quality
Metrics.
ì Build
Deployment
Pipeline.
ì Automated
Acceptance
TesLng.
ì Automated
PromoLon
to
subsequent
Environments.
ì Single
click
or
automated
push
to
producLon.
h"p://linkedin.com/in/praneshvi"al
16

17. Main
Focus
of
Release
Engineering
ì Build
-­‐>
Deploy
-­‐>
Test
-­‐>
Release
ì Every
single
check-­‐in
should
potenLally
be
in
a
releasable
state.
h"p://linkedin.com/in/praneshvi"al
17

18. Configuration
Management
ì Using
Version
Control
and
commit
everything.
ì Check-­‐in
the
changes
at
regular
frequency.
ì Use
meaningful
messages
for
every
commit.
ì Managing
Dependencies
(External
Libraries,
Components).
ì Managing
the
Infrastructure
(Consistent
network
topology,
firewall,
OS
configuraLon,
patches
etc...
across
all
the
environments).
ì Managing
the
Environments
&
the
tools
to
be
used
for
the
same.
ì Managing
the
Change
Process.
h"p://linkedin.com/in/praneshvi"al
18

19. Principles
of
Managing
Application
Configuration
ì Good
understanding
of
the
stage
in
which
the
configuraLon
should
be
injected
(Assembly,
Deployment,
Restart
etc…)
ì ConfiguraLon
sefngs
for
the
applicaLon
to
be
in
the
project’s
root
directory.
ì Should
always
be
automated
and
values
checked
into
repository.
ì Use
clear
naming
convenLon.
ì Avoid
Over-­‐engineering.
Keep
it
as
simple
as
possible.
ì Ensure
that
the
configuraLon
is
tested
properly
a<er
deployment.
h"p://linkedin.com/in/praneshvi"al
19

20. Release
Engineering
Architecture
ì Its
all
about
building
the
“Build
&
Release”
pipeline.
ì Built
using
Jenkins
or
similar
ConLnuous
IntegraLon
Tools.
ì IllustraLon
of
a
“Delivery
Pipeline”
(Image
Courtesy:
“ConLnuous
Delivery”
by
Jez
Humble
&
David
Farley).
h"p://linkedin.com/in/praneshvi"al
20

21. Deep
dive
into
commit
/
component
builds:
ì Aim:
ì Eliminate
builds
that
are
unfit
for
producLon.
ì Steps:
ì Compile
the
code.
ì Run
a
set
of
commit
tests.
ì Run
Lint
based
tests
or
some
of
the
basic
staLc
code
analysis
tasks.
ì Publish
the
results.
ì Once
above
steps
are
done,
build
the
package
and
upload
the
binary
to
a
centralized
repository.
ì Add
tests
on
an
ongoing
basis.
ì Keep
a
watch
on
the
build
execuLon
Lme
and
opLmize
frequently.
ì Explore
the
‘Pre-­‐Commit’
or
‘Pre-­‐Flight’
build
opLons
provided
by
some
of
the
CI
tools.
h"p://linkedin.com/in/praneshvi"al
21

22. Anti-­‐Patterns
in
commit
/
component
builds:
ì Don’t
check-­‐in
new
code
on
a
broken
build.
The
only
fix
that
has
to
go
are
the
changes
that
fix
the
broken
build.
ì Always
run
commit
tests
locally
before
the
commit.
ì Always
wait
for
commit
tests
to
pass
before
next
check-­‐in.
ì Developers
who
have
commi.ed
their
code
are
responsible
for
the
build
process
to
succeed.
ì Never
go
home
on
a
broken
build.
ì Time-­‐box
during
every
failed
build.
Give
about
10-­‐20
minutes
to
fix,
else
rollback
the
changes.
ì Don’t
EVER
comment
failing
tests.
ì Take
ownership
for
all
the
breakages
that
result
from
your
changes
and
fix
them
accordingly.
h"p://linkedin.com/in/praneshvi"al
22

23. Aim
of
Deployment
Pipeline
ì Every
Step
should
be
visible
(Results
in
be.er
collaboraLon).
ì Deliver
useful,
working
so<ware
to
the
users
as
early
as
possible.
ì Early
feedback
(IdenLfy
&
resolve
issues
in
the
early
stages).
ì Enable
teams
to
deploy
&
release
any
version
of
their
so<ware
at
any
point
to
any
of
the
environments.
ì Avoid
last
day
heroics
on
the
release
day.
ì Release
in
small
chunks
(Avoid
“Big
Bang”
release).
ì Should
be
driven
by
tools
&
not
by
sviduals.
ì Ability
for
every
change
to
be
transparent
&
propagate
them
through
the
pipeline.
ì Ability
to
roll-­‐back
to
a
previous
stable
state
in
case
of
any
failures.
h"p://linkedin.com/in/praneshvi"al
23

24. Anti-­‐Patterns
of
Deployment
Pipeline
ì Manual
Deployment
of
So<ware.
ì Deployment
performed
by
mulLple
teams.
ì The
order
of
steps
are
not
defined.
ì No
proper
“Run
books”
for
failures.
ì Too
much
of
reliance
of
manual
tesLng.
ì CorrecLon
to
the
release
process
during
the
actual
producLon
release.
ì ConfiguraLon
across
all
the
environments
varies
to
a
large
extent.
ì Manual
ConfiguraLon
management
of
ProducLon
Environment.
ì Deployment
to
a
producLon-­‐like
environment
is
done
only
development
is
100%
complete.
h"p://linkedin.com/in/praneshvi"al
24

25. Best
qualities
of
a
Deployment
Pipeline
ì Deployments
should
be
automated.
ì Deployments
should
be
done
frequently
(If
possible,
for
every
single
check-­‐in).
ì Provide
early
feedback
so
that
the
development
team
can
act
on
the
failures
in
a
faster
way.
ì Good
AutomaLon
Coverage
to
test
early.
ì Should
be
scalable.
ì Should
enable
one-­‐click
deployment.
h"p://linkedin.com/in/praneshvi"al
25

26. Deployment
on
Large
Scale
Production
Environments
ì What
are
host-­‐types?
ì What
are
recipes
/
cookbooks?
ì Tools
like
Pogo
OR
other
agent
based
deployment
tools.
$
ssh
<target-­‐host-­‐name>
'whoami;’
praneshv
h"p://linkedin.com/in/praneshvi"al
26

27. Why
Automated
Deployment
is
an
indispensable
goal
???
ì On
most
occasions,
the
deployment
documentaLon
is
outdated.
ì Logging
all
the
steps
in
the
form
of
scripts
is
very
beneficial
for
book
keeping
purposes.
ì Works
seamlessly
if
all
the
steps
are
taken
care
of.
ì Even
non-­‐experts
should
be
able
to
deploy
effortlessly.
ì Every
team
using
automated
deployment
scripts
results
in
its
maturity
&
less
prone
to
errors.
ì Take
off
the
dependency
from
the
deployment
expert
and
uLlize
them
for
more
challenging
tasks.
ì Its
fast,
cheap.
Facilitates
in
early
tesLng
&
faster
feedback
cycles.
ì Deployment
Logs
are
auditable
for
future
references.
h"p://linkedin.com/in/praneshvi"al
27

28. Deep
dive
into
Testing
Quadrant
Image
Credit:
Concept
defined
by
‘Brian
Marick’
(Diagram
obtained
from
“ConLnuous
Delivery”
by
‘Jez
Humble’
&
‘David
Farley’.
h"p://linkedin.com/in/praneshvi"al
28

29. Deep
Dive
into
Automated
Acceptance
Tests
ì Aim:
ì Avoid
manual
tests
to
whatever
extent
possible.
ì Steps
to
be
followed:
ì Tests
to
be
performed
in
producLon-­‐like
environment.
ì If
sefng
up
environment
is
expensive,
use
a
scaled-­‐down
environment.
ì Setup
the
actual
user’s
environment
on
a
grid.
ì Use
Business
language
in
the
scripts
instead
of
technical
jargon
(‘Place
Order’
instead
of
‘Clicking
on
bu.on
XYZ’).
ì Well-­‐defined
exit
criteria
for
Pass
/
Fail
of
tests.
ì Don’t
fail
the
test
due
to
minor
issues.
h"p://linkedin.com/in/praneshvi"al
29

30. Deep
Dive
into
Automated
Acceptance
Tests
continued…
ì Steps
to
be
followed:
ì Include
a
few
tests
to
assert
external
systems.
ì If
you
don’t
care
about
a
parLcular
field,
don’t
bother
tesLng
it.
ì These
tests
should
be
run
a<er
every
deployment.
ì Block
the
pipeline
when
there
are
massive
failures.
ì Parallelize
the
tests
to
whatever
extent
possible.
ì Outcome
of
this
step
are
the
so<ware
packages
that
have
fought
against
all
the
tests
&
challenges
like
a
mythical
hero
and
ready
to
take
on
the
world
!!!
h"p://linkedin.com/in/praneshvi"al
30

31. Few
more
Generic
Anti-­‐Patterns
ì TesLng
done
on
developer
machines.
ì Service
Engineering
team
hasn’t
even
seen
the
applicaLon
Lll
the
actual
release
date.
ì Installers,
ConfiguraLons
etc…
not
done
Lll
the
release
date.
ì No
collaboraLon
between
development
team
&
SE
teams.
ì Late
setup
of
Staging
Environment.
ì Release
too
many
changes,
unable
to
figure
out
what
caused
the
failure.
ì Changing
the
configuraLons
directly
on
ProducLon
o<en
resulLng
in
disasters.
h"p://linkedin.com/in/praneshvi"al
31

32. Adaption
of
Release
Engineering
ì Change
in
the
mindset
of
the
team
members.
ì All
aspects
of
development,
tesLng,
staging,
producLon
environments
(most
importantly
configuraLon
management)
to
be
managed
through
a
VCS.
ì Integrate
development,
tesLng,
release
teams
as
a
part
of
the
development
process.
ì Manage
the
Infrastructure
to
build
environments
in
no
Lme.
ì Beef
up
the
test
automaLon
coverage
so
that
there’s
not
much
reliance
on
Manual
tesLng.
ì Rehearse
the
release
&
rollback
process
so
that
its
easy
to
do
it
Lme
&
again.
ì Reach
a
stage
wherein
“So<ware
Release”
is
in
self-­‐service
mode.
h"p://linkedin.com/in/praneshvi"al
32

33. Tools
Used
at
each
of
the
stages.
ì Version
control
–
Git
ì Build
Tools
–
Ant,
Nant,
MSBuild,
gmake,
Maven,
Buildr,
Psake.
ì Agile
/
Scrum
–
Jira.
ì StaLc
validaLon
–
Coverity,
SonarQube
ì Unit
tesLng
–
junit
ì Test
automaLon
–
Protractor,
Selenium
ì ConLnuous
IntegraLon
–
Jenkins,
Atlassian
Bamboo,
Thoughtworks
Go,
UrbanCode
AntHillPro
ì Deployment
–
Pogo,
Chef
ì Environment
provisioning
–
Puppet,
Chef
ì IAAS,
PAAS
–
AWS,
Azure,
Docker,
Vagrant
h"p://linkedin.com/in/praneshvi"al
33

34. Tools
Used
at
each
of
the
stages.
ì Image
Credit
:
h.p://maestrodev.com
h"p://linkedin.com/in/praneshvi"al
34

35. Managing
Environments
ì No
ApplicaLon
is
an
Island.
ì Poor
ConfiguraLon
Management
results
in
significant
waste
of
Lme,
addiLonal
costs,
tech
debt
etc…
ì Should
always
be
cheaper
to
create
a
new
environment
than
repairing
a
broken
environment.
ì Few
of
the
names
used
for
Environments
ì Development
ì IntegraLon
ì QA
/
TesLng
ì Staging
/
Pre-­‐ProducLon
ì Performance
ì Canary
/
Bucket
TesLng
ì ProducLon
h"p://linkedin.com/in/praneshvi"al
35

36. Managing
Releases
of
complex
systems
h"p://linkedin.com/in/praneshvi"al
36
Image
Credit:
Slide
deck
available
at
h.ps://learn.chef.io/

37. Managing
Releases
of
complex
systems
h"p://linkedin.com/in/praneshvi"al
37
OrchestraLon:
ì Automated
arrangement,
coordinaLon,
and
management
of
complex
computer
systems,
middleware,
and
services.
ì Aligning
the
business
request
with
the
applicaLons,
data,
and
infrastructure
ì Deployment
Rhythm
ì Its
about
gefng
the
perfect
configuraLon,
checking
in
these
values
and
automate
the
release
using
these
values.
ì Have
the
right
sequence
of
steps
in
the
configuraLon
file.
ì Have
the
right
kind
of
checks
&
balances
at
every
stage.
ì PracLce
roll-­‐backs
in
case
of
issues.
ì Implement
fail-­‐safe
methodologies
in
case
of
issues.
ì Build
environments
wherein
the
ProducLon
traffic
can
be
replayed
(Use
of
access
logs
to
simulate
user
acLons).

38. Steps
involved
to
build
a
deployment
pipeline
ì Model
your
value
stream
&
create
a
walking
skeleton.
ì Automate
the
build
&
deployment
process.
ì Automate
unit
tests
&
code
analysis.
ì Automate
Acceptance
Tests.
ì Automate
Releases.
h"p://linkedin.com/in/praneshvi"al
38

39. Multi-­‐Tenant
Applications
ì Customers
share
the
same
cloud
plavorm
and
infrastructure
and
their
data
is
commingled.
ì Single
code-­‐base
for
the
applicaLon.
ì Upgrades
are
executed
easily
by
the
SaaS
provider.
ì All
the
tenants
are
using
the
same
version.
ì Data
of
different
tenants
is
stored
in
the
same
place,
however,
measures
taken
to
make
it
inaccessible
to
other
tenants.
ì BuckeLze
the
access
based
on
the
IP
Range
/
Cookie
range
/
Header
InformaLon
etc...
ì Toggle
a
Feature
using
configuraLon.
ì Hard
to
maintain
different
versions
for
different
tenants.
h"p://linkedin.com/in/praneshvi"al
39

40. Multi-­‐Instance
Applications
ì MulLple
customers
run
their
own
separate
instance
of
an
applicaLon
and
OS
running
on
a
separate
VM.
ì Avoid
comingling
of
data.
ì Develop
&
use
their
own
logical
piece
of
the
cloud
service.
ì Allows
for
greater
flexibility
and
control
of
configuraLon,
customizaLon,
updates
and
upgrades.
ì Ability
to
migrate
instance
to
an
on-­‐premise
server,
or
to
another
cloud
hosLng
provider
h"p://linkedin.com/in/praneshvi"al
40

41. Zero
Downtime
Deployment
ì What
is
‘Zero
DownLme
Deployment’?
ì What
is
‘Out
Of
RotaLon’
(OOR)
?
ì What
is
‘Concurrency’
?
ì Colo
(Data-­‐Center)
based
deployments.
h"p://linkedin.com/in/praneshvi"al
41

42. Extracting
the
server
logs
ì Standardize
the
logging
messages.
Involves
lot
of
effort
from
the
development
teams.
ì Setup
Splunk
alerts
for
‘FATAL’
errors
in
the
code.
h"p://linkedin.com/in/praneshvi"al
42

43. Next
generation
tools
in
the
DevOps
world
ì Chef
ì Puppet
ì Cfengine
ì Salt
h"p://linkedin.com/in/praneshvi"al
43

44. Preparation
for
the
Release
ì Any
issues
during
the
release,
stop
or
postpone
the
release.
Stability
takes
the
highest
priority.
ì Prepare
Release
Plan
or
Runbook
with
correct
sets
and
if
possible
automate
them.
ì IdenLfy
the
error-­‐prone
steps
and
automate
them.
ì Perform
the
drill
for
performing
rolling
back
of
the
releases.
ì Should
be
as
simple
as
selecLng
a
value
and
clicking
a
bu.on.
ì Let
one
team
perform
all
the
releases.
(Too
many
cooks
spoil
the
broth).
h"p://linkedin.com/in/praneshvi"al
44

45. Maturity
of
a
“Deployment
Pipeline”
?
h"p://linkedin.com/in/praneshvi"al
45
ì Image
Credit:
h.p://www.praqma.com/

46. Q
&
A
h"p://linkedin.com/in/praneshvi"al
46

47. Thank
You
h"p://linkedin.com/in/praneshvi"al
47

48. References:
ì Details
about
each
of
the
phases
from
“ConLnuous
Delivery”
by
Jez
Humble,
David
Farley
ì Source
of
some
of
the
generic
images
used
–
“Unknown”.
ì RespecLve
credits
given
to
the
images
used
in
the
same
slide.
ì h.ps://gigaom.com/2014/01/26/mulL-­‐tenant-­‐or-­‐mulL-­‐instance-­‐cloud-­‐lets-­‐
do-­‐both/
ì h.p://diginomica.com/2013/12/20/mulL-­‐tenant-­‐mulL-­‐instance-­‐saas-­‐
spectrum/
ì h.ps://msdn.microso<.com/en-­‐us/library/hh534478.aspx
ì h.ps://www.youtube.com/watch?v=CE4hLYhfmBE
-­‐
Deployment
using
pogo.
h"p://linkedin.com/in/praneshvi"al
48