As development teams move toward a more agile development process coupled with continuous delivery (CD), the role of operations has evolved from a support organization to an integral part of the product delivery ecosystem. Today, operations organizations need to be agile in their feature delivery, but as requests come from different stakeholders, operations must balance their investments in support tasks, urgent tooling requests, and strategic projects. Sumedha Ganjoo explains why it is important for an operations organization to move toward CD and identifies process, tools, and development changes required for its implementation. Sumedha shares changes her team made to be more agile while planning and prioritizing work, as well as changes to development practices that helped the team move toward CD. To explore the benefits of an operations organization adopting continuous delivery and to see how you can start moving toward it, join Sumedha for this session.
4. 11/8/17
2
What are we not going to talk
about?
Team Structure
DevOps Vs Operations
Context
• Global team
• Tools and infrastructure used by 200-400 devs
• More than hundred thousand tests and ~50-100
submissions per day
5. 11/8/17
3
One year back…
• Projects were long and by the time they finished,
requirements or some tools had changed
• Developers were running into issues and were
complaining about tools
• Fixes to tools couldn’t be made available to developers
fast as roll outs were risky and took time
9. 11/8/17
7
Keep the mainline healthy
• Release branch as close to mainline as possible
Cherry-picking the fix
from master
Main line
Release branch
X is fixed quickly
Feature X is bad
Scenario 1
Bug Y is fixed in main line much
later
Feature X is bad
R1 branch
R2 branch
Feature X is reverted
from Release branch to
be able to release and
bug Y is filed
Time of stabilizing the
release branch
increases.
Main line
z
z
Getting to a healthy mainline takes much longer
now:
• Longer the time from introduction, longer the
bug fix takes
• Other features build on top of buggy code
Scenario 2
Automated testing
• Standardized acceptance tests
• Unit testing
10. 11/8/17
8
Make code more testable
• Refactor & Re-architect
• Componentize
Low tolerance for failure
• Quality first
• Low infrastructure failure
12. 11/8/17
10
Balance long term vs short
term
• Support vs feature requests
• Short term enhancements vs strategic investments
• Focused & regular prioritization
Balance long term vs short
term
• Support vs feature requests
• Short term enhancements vs strategic investments
• Focused & regular prioritization
13. 11/8/17
11
Balance long term vs short
term
• Support vs feature requests
• Short term enhancements vs strategic investments
• Focused & regular prioritization
Minimal viable products
• Groom features into MVPs
• Deliverable every iteration
15. 11/8/17
13
Delivery Pipeline Practices
• Only Build Your Binaries Once
• Smoke-Test Your Deployments
• Deploy into a Copy of Production
• Each Change Should Propagate through the Pipeline
Instantly
• If Any Part of the Pipeline Fails, Stop the Pipeline
Pipeline
Manual
Extensive
Testing
Smoke
Testing
Build
Installers
CI 2
CI
1
CI 3
SCC
Pre GC
16. 11/8/17
14
Build Binaries
Build
Installers
Build VMs
Pipeline
Pre
SCC
Commit
& Test
Staged Automated
Testing
Asynchronous Stages
Manual
Deploy
Reproducible, Reliable &
Scalable Infrastructure
Build Binaries Build
Installers
Build VMs
Pre
SCC
Commit &
Test
Staged Automated Testing
Asynchronous Stages
Manual
Deploy
>600 Systems
>100
>400
>100
18. 11/8/17
16
Why should an Operations
organization adopt CD practices?
• Easier to adjust to changing priorities and requests
• Faster turn around time on requests = Happy clients
• Higher confidence in release and more autonomy
• Less stress
• More efficient team
Benefits