Presentation given at OSCON 2015 Open Cloud Day on 7/21/2015 For open source projects such as OpenStack, being able to effectively build, release, and deploy upstream code into a production environment is as much art as science. Using the experience gained from more than three years in OpenStack operations, we will share best practices and tools to create a sustainable and repeatable release engineering process for your own open source development needs. http://www.oscon.com/open-source-2015/public/schedule/detail/44790

1. Release Engineering Downstream
of an OpenStack Project
Best Practices for the Path from Idea to Production with
an Open Source Code Base
Rainya Mosher
@rainyamosher | www.rainyamosher.com

2. People don’t want…a quarter inch
drill. They want a quarter inch hole.
Clayton M. Christensen
The Innovator’s Solution

5. About Rainya
• 10+ Years Web App Devoplment
(ASP & .NET C#)
• 3+ Years OpenStack Operations
• Master of Science in Technology
Commercialization
• Musical Theatre Nerd, Singer, and
Actor
• Twitter: @rainyamosher
• Blog: rainyamosher.com That about sums it up.
ENFP

6. What is Release Engineering?

7. Release Engineering (RelEg / RE) is the sub-
discipline of software engineering concerned
with the compilation, assembly, and delivery of
source code into finished products or other
software components.
https://en.wikipedia.org/wiki/Release_engineering

8. Release
Engineering
UPSTREAM
PDLC / SDLC TOOLS
CULTURE

9. What is your relationship with the
upstream codebase?

10. Current Dates & Milestones: https://wiki.openstack.org/wiki/Releases

11. http://docs.openstack.org/infra/manual/developers.html
• Clone the code from github
• Branch your change
• Make fixes
• Submit review
• Submit patch revisions until you
get passing tests and a +2
• Change is merged
Average Review Wait Times
as of 7/20/2015
• Nova – 22 days
• Neutron – 9 days
• Glance – 26 days
http://stackalytics.com

12. Pull
OpenStack
Master
Consume
Third-Party
Distro
Internal Integrations & Configs
More ownership
over release
cadence
More in-house
resources for
packaging
More flexiblity
with to
integration of
internal
changes into
python source
code
Third party
guides release
cadence
Fewer in-house
packaging
needs
Less flexibility in
integrating
internal
changes into
python source
code
No matter the upstream relationship,
you’ll need to account for your
internal configurations and system
integrations!

13. Even understanding
the upstream
landscape of the
project, there will be
times when someone
– usually a product
or engineering
manager – will say “I
don’t care about the
community, just ship
the feature/bug
fix/widget.” What can
you do?

14. How will you manage the product
and software development lifecycles?

15. PDLC
SDLC
AGILE
Scrum,
Kanban,
XP, etc.
PDLC is the management of a
good or service from inception,
design, engineering, build, sale,
maintenance, and disposal
through the integration of
people, data, process, and
business systems to make an
optimal experience for a
customer (internal or external).
SDLC is a framework of steps
and phases providing a model
for development teams to follow
when creating a piece of
software to improve the quality
of the software and the overall
process.
AGILE SDLCs are a collection
of methodologies valuing
flexibility, collaboration,
communication, and rapid
iterations in the development
process.

16. Simple Easy
One role / task /
concept / dimension
vs Complex
Familiar, part of our
skillset, near at hand
vs Hard

17. Source Image: Rich Hickey, “Simple Made Easy”, 11/2012
Early on, the easy path gives
you speed and a sense of doing.
Over time, easy ignores
complexity, slows you down, and
leads to painful technical debt.

18. Being intentional with your PDLC & SDLC approach can help
you get from mad baby to successful sand eating baby.

19. What tools will you use for build,
release, and deploy activities?

20. BUILD
RELEASE
DEPLOY
SYSTEMS
Pipeline
Config
Management
Packaging
Orchestration

21. BUILD
RELEASE
DEPLOY
SYSTEMS
Pipeline
• Invest in your non-production environments
• Engineers cannot deliver stable software to production if the places
they have to develop and validate are unstable

22. Dev
• Prod-like
functionality
(as much as
possible)
• Unit tests at a
minimum
Integration
• First place for
Dev to
integrate code
with upstream
packages
• Unit tests
• Basic smoke
tests
Test
• Full smoke
tests
• High level
integration
testing with
tightly coupled
services
Pre-Prod
• Mirrors Prod
• Full
integration
testing with all
services
• Load and
performance
testing when
major
changes
Production
• Run baseline
tests prior to
an upgrade
• Automate the
deploy to
keep human
error out of
the way
• Don’t touch it!
Speed: Find failures quickly Accuracy: Find all failures Stability!

23. BUILD
RELEASE
DEPLOY
SYSTEMS
Config
Management
• Make all system and software
changes through a config
management system
• Start with immutable servers and
services (it’s harder to give up root
later)
• Ansible, Puppet, Chef, Salt
all have strengths &
weaknesses – take the
time to determine which is
best for your situation!

24. BUILD
RELEASE
DEPLOY
SYSTEMS
Packaging
• Successful packaging has been
done with RPMs and DEBs on
physical hosts and VMs
• Containers are rapidly maturing into
a viable approach for the control
plane
• Finding an approach everyone
can agree to is perhaps the
biggest challenge!
• Configs can be bundled
with the source code or
distributed separately
based on frequency of
updates and scale of
system

25. BUILD
RELEASE
DEPLOY
SYSTEMS
Orchestration
• Thoughtful orchestration for deploy-
time activities can overcome many
shortfalls of software
• Dedicate time and energy to find the
best order of operations for
introducing changes to services
• Deploy patterns - blue-green, rolling,
shotgun – are limited by the
architecture of the software being
deployed, and that’s ok!

26. How will your organization adapt to
the changes necessary for success?

27. We need to create a culture that reinforces
the value of taking risks and learning from
failure and the need for repetition and
practice to create mastery.
Gene Kim
The Phoenix Project

28. CI &
DevOps
Release
Management
Engineering Practices
Always code as if the person who
ends up maintaining your code will
be a violent psychopath who
knows where you live.
~ John F. Woods, c. 1991
Until zero-downtime deploys are a
reality, CI/CD may as well stand for
continual impact and constant
downtime. Adopt a release
management process to stay sane!
Automation first! This is difficult for many
organizations to do. It often means slowing
down, taking the long way rather than the
easy way, and telling people “not right now.”

29. Be Intentional
Adapt to Overcome
Simplicity First

30. References
 Images
- Gripp & Drouhard. Space Unicorn. (2012). http://youtu.be/17o1OlroNSE
- Miltonious Blog Unicorn of Technical Difficulties, Source Unknown
- Random Internet Meme’s found via Google Search
- Hickey, R. 2012. Simple Made Easy. http://www.slideshare.net/evandrix/simple-made-easy
 Quotes
- Christensen, C. M. (2003). The Innovator’s Solution: Creating and Sustaining Successful Growth
- Kim, G., Behre, K., Spafford, G. (2013). The Phoenix Project: A Novel about IT, DevOps, and
Helping Your Business Win

31. References
 Useful Links
- OpenStack Dev: http://docs.openstack.org/infra/manual/developers.html
- Review Wait Times: http://stackalytics.com/
- OpenStack Releases: https://wiki.openstack.org/wiki/Releases
- Rel Eng Overview: https://en.wikipedia.org/wiki/Release_engineering

32. Release Engineering Downstream
of an OpenStack Project
Discovering the Path from Idea to Production with an
Open Source Code Base
Rainya Mosher
@rainyamosher | www.rainyamosher.com