Bert Jan Schrijver presented on Malmberg's journey to continuous delivery using open source tools and cloud services. Malmberg is an educational publisher that builds e-learning applications using Java, Vert.x, AngularJS and MongoDB on Amazon cloud services. They faced issues with slow development and operations communication and difficult production problem analysis using a traditional model. To address this, Malmberg established an expert DevOps team, defined key principles like keeping the master branch releasable at all times and managing all automation through Jenkins, and overcame challenges like gaining developer skills and hitting AWS limits to achieve benefits like increased agility, availability and cost reductions.

1. Swimming upstream
in the container revolution
Practical continuous delivery
with open source and cloud services
Bert
Jan
Schrijver
@bjschrijverbertjan@jpoint.nl

2. Bert Jan Schrijver
L
e
t
‘
s
m
e
e
t
@bjschrijver

3. Outline
S o w h a t ‘ s n e x t ?
Approach
Principles
Introduction
Definitions
Challenges
and
lessons
learned
Business
benefits
Looking
ahead

4. Every
change
goes
through
the
build/test
pipeline
and
automatically
gets
put
into
production.
Continuous Deployment
Development
and
operations
engineers
participate
together
in
the
entire
product
lifecycle.
DevOps
Building
and
testing
software
in
such
a
way
that
the
software
can
be
released
to
production
at
any
time.
"Ship
early,
ship
often,
sacrificing
features,
never
quality"
-­‐
Kyle
Neath
Continuous Delivery
Team
members
integrate
their
work
frequently.
Commits
are
verified
by
automated
builds
and
tests.
Continuous Integration
Definitions
Who’s
who
in
CD

5. Malmberg
is
an
educational
publisher
in
the
Netherlands.
Malmberg
is
building
modern,
rich
and
scalable
e-­‐learning
applications
using
Java
8,
Vert.x,
AngularJS
and
MongoDB,
running
on
Amazon
cloud
services.
About Malmberg

7. Differences lead to issues
Communication between
development and operations was
slow, problem analysis in production
was difficult and releases took a lot
of time.
Traditional operations
Production environments managed
by an external operations partner.
Differences in infrastructure
between development and
production.
Modern development culture
Modern tools and lots of
automation. Test environments are
managed by developers.
History
Some
months
ago
Things needed to change
Issues and differences between
development and operations were
slowing us down. We needed to
shift strategies to keep progressing.

8. …said no product manager ever.
J.
Paul
Reed
Let’s spend the next few months..
..working
on
automated
testing
and
build/release
infrastructure,
and
redesigning
how
our
application
is
written.
We
can
postpone
our
feature
development.
“
”

9. Build a complete new setup to
allow development teams to
transform to the new
situation at their own pace.
Keep the shop open
Build a dedicated team of
Devs, Ops and Cloud experts.
Expert team
Define key points that identify
your approach and help you
set goals.
Define principles
Approach
How
we
initiated
change.

10. Principles

11. Master branch is always releasable
Principle 1
Every change is developed and tested in a feature branch.
Don’t merge it until it’s done.

12. Unit/integration (Java & JavaScript), mutation, end-to-end (FitNesse/BrowserStack),
Sonar for quality and coverage reporting.
Rely on multiple layers of tests.
Each commit is tested extensively
Principle 2

13. Manage builds, tests, QA and deployments from a single place.
Jenkins as the heart of the delivery process.
Every delivery step is a Jenkins job
Principle 3

14. After deploying 6 new features, when one has an issue, why roll back 5 good features?
Don’t. Just roll out a fix quickly.
Keep moving ahead.
Deployments are roll-forward only
Principle 4

15. No logging in to servers. Need a change or upgrade? Just update the recipe.
Hands off.
Infrastructure as code - for everything
Principle 5

16. The flexibility and resilience is well worth it.
What about using containers? The EC2 instance is our container.
Even when you don’t need to scale.
Put everything in auto scaling groups
Principle 6

17. You can’t explain maintenance windows to modern end users anymore.
Our end users are the Facebook generation.
No downtime in production
Principle 7

18. Make sure you find the problem before it finds you.
Work proactive, not reactive.
Eyes and ears in production
Principle 8

19. Repeating tasks such as viewing logs and doing deployments must be common jobs.
Specialisms are OK, but only for incidental tasks.
Repeating tasks are executable for all team members
Principle 9

20. Differences between teams are OK. A team that’s dependent on external help is not.
Give teams the freedom to work in a way that works for them.
DevOps teams work on a self service basis
Principle 10

21. Don't assume that cultural change
won't be an issue. It will.
Resistance
It turns out that something in
recent AWS Linux kernels triggers
slave disconnects.
We found out the hard way.
Jenkins slaves can be a pain
Not all devs are comfortable with
managing infrastructure and
middleware.
Devs need to step up their game
This kills team progress.
Don’t depend on availability
of Ops
Challenges
02
03
04
05
06
01
When you automate everything and
keep growing, chances are you’re
going to hit limits.
Amazon has limits
All environments are provisioned
automatically. Challenge: how to
prevent testing directly in
production.
How to test Puppet changes
and
lessons
learned

22. Auto scaling and pro-active monitoring
boost availability. A lot.
Availability
Business benefits
High level of automation results in
shorter release cycles and faster time to
market.
Agility
Automated provisioning makes sure that
every environment can be re-built from
scratch in minutes.
Continuity
Lower operations costs due to scheduling
and scaling. Lower maintenance costs due
to high degree of automation.
Cost reduction
Faster problem analysis and solution.
Better reaction speed
How to sell this to your boss.

23. Get
the
teams
the
information
they
need,
readily
available
on
a
dashboard
visible
from
their
desks.
Better monitoring and dashboards
Looking ahead
01
Nightly
performance
runs
on
test
environments
and
continuous
end-­‐user
performance
monitoring
in
production.
Continuous performance testing03
The
only
way
to
be
really
prepared
for
failure
is
to
make
sure
that
things
will
fail
by
making
it
fail
yourself.
Automated resilience testing02
This
is
a
tough
one.
We’re
still
trying
to
work
out
what
works
for
us.
Continuous security testing04
The near future

24. Questions?
@bjschrijver

25. Thanks for your time.
Liked
it?
Tweet
it!
@bjschrijver