This document discusses lessons learned from continuous delivery practices. Key points include: automating infrastructure provisioning and application deployment; treating infrastructure as code; adopting a dynamic "cattle not pets" approach to infrastructure; implementing revision control and automated testing for infrastructure and applications; overcoming cultural challenges through collaboration, Lean practices, and a blameless culture; managing risk through experimentation; and shifting activities earlier in the process. Containerization and tools that facilitate team collaboration are also trends in DevOps.

1. Lessons Learned From Continuous
Delivery
Out of the Frying Pan, Into the Fire
Mandi Walls
Leeds DevOps
January 31, 2017

2. Mandi Walls
Technical Community Manager for EMEA
@lnxchk
mandi@chef.io

3. Who Is Chef?

4. Why Continuous Delivery?
• Used as a proxy for success with DevOps transformations
• Blame John Allspaw and Paul Hammond
• “10+ Deploys A Day at Flickr”

5. The rewards?*
*source: Dr. Nicole Forsgren research on DevOps
More deployments
Ship faster
Faster MTTR after issues
More profits, market share, and productivity
Market cap goes up

6. Technical Lessons
The Fun Stuff

7. Lesson: Automation
• Automated, full-stack application policies
• Package and service installation
• Versionable, testable, repeatable workflow
• Scalable application policies
• Management of interdependencies across nodes

8. Lesson: Dynamic Infrastructure
• Provisioning and setting up environments
• Dynamic scaling of compute resources
• Migrating legacy workloads to the cloud
• Multi cloud and hybrid cloud deployment
• Support for heterogeneous environments

9. Do you HAVE to go to the Cloud?
• Well….
• You need cloud-approximate features
• APIs
• Resources-on-Demand

10. Infrastructure as Code
• Programmatically provision and configure
components
• Treat like any other code base
• Reconstruct business from code repository,
data backup, and compute resources
• Partner with dynamic infrastructure
• “Cattle not pets” or “Crops not house plants”

11. How Chef Got Into DevOps
• System automation is often a first step for transitioning to DevOps work practices
• With or without previous experience in Agile
• Agile left Operations concerns in the dark
• Config management presents practical value
• Displays easy wins, demonstrable and measurable outcomes
• MTTR

12. The game changer: rapid time to value
Innovation
Quality/
Complian
ce
Dynamic
Infrastructure
Infrastructure as Code
Automate the Stack
DevOps
+ +

13. Trust The Automation

14. Lesson: Revision Control
• Findings from State of DevOps report
• Everyone must use version control
• Dev
• Ops
• You
• Me
• The dog your hipster office mate brings to work

15. Lesson: Automated Testing
• Difficult tasks – automating the testing of applications developed internally and
COTS or contract work
• Different tools for different languages and platforms
• Delivers a lot of value, but costs a lot to get going
• Unfortunately met a lot of orgs that struggle with this or choose to postpone

16. Testing Infra Code Too
describe 'apache::default' do
context 'When all attributes are default, on an unspecified platform' do
let(:chef_run) do
runner = ChefSpec::ServerRunner.new
runner.converge(described_recipe)
end
it 'converges successfully' do
expect { chef_run }.to_not raise_error
end
it 'installs apache' do
expect(chef_run).to install_package 'apache2'
end
end
end

17. Culture Lessons
The Messy Stuff

18. Lesson: Collaboration
• Ideological shift for Operations folks used to working alone
• Also difficult for specialized techs in silos – Storage Ops, NetOps, Security
• Talking to people vs just putting in tickets and waiting
• The stereotypical “throw it over the wall”

19. Lesson: CAMS to CALMS
• CAMS: Culture, Automation, Metrics, Sharing (c. 2010)
• CALMS: Culture, Automation, Lean, Metrics, Sharing
• Using Lean to break through the logjam of enterprise processes

20. Lean
• Eliminate non-value-added action – Don’t do things that don’t add value just
because you’ve always done them
• Pull over Push – Spend time on the things customers want to use. Don’t spend
time thinking about ways to force them to use things they don’t want
• Kaizen – Continuously improve people, processes, tools
• Kaikaku – Be willing to change the whole thing if it isn’t working
• Small Batch + Experimentation – Often learned from Agile

21. Business Stuff
The “Culture + Tools = ££££” Stuff

22. Failure
• Accepting failure as part of experimentation
• Practice recognizing how much risk is involved in each experiment
• The cost of an individual failure

23. Blameless Culture
• Recognizing that complex systems have complex problems
• The system may be too complex for one human to fully understand
• Research in other systems fields
• Aerospace and aviation, nuclear power generation, etc
• “Human error” is often cited but not often the actual problem
• The humans didn’t know your system didn’t have redundant error handling
• The humans didn’t know there wasn’t alerting on a subsystem they weren’t told about

24. Lesson: Business Value
• Fits in with Lean
• Work on the things that will add value to users, even if they aren’t the most fun
• “Eating your vegetables”
• Minimize NIH in favor of reuse

25. Managing Risk
• Small batches to manage the near-term costs
• Validate with customers in short timeframes
• Introducing near-term volatility to mitigate long-term risk
We spent a year building this thing but no one liked it!

26. Lesson: Skills x Contractors x Employees
• DevOps is a way of life
• Getting to Continuous Delivery takes TIME
• Managing the skill matrix of the people working on the project is important
• Harder to do when folks are leaving after 6 months to find something more
interesting

27. Trends
What’s up with DevOps in 2017+

28. Containers
• Of course.
• Because containers.
• And microservices.
• And whatever comes next as the smallest compartment of compute

29. Shifting “To the Left”
• Pushing more and more activities, discussion,
acknowledgement of complexity to earlier stages of
the process
• Security, compliance, testing, performance
http://www.rebalanceclinic.co.nz/wp-content/uploads/2013/05/keep-left.png

30. Tools For All Teams
• Complement collaboration, communication, transparency
• Integrate dynamic infrastructure, testing, reporting

31. We Are Chef
• https://chef.io
• Find us! https://events.chef.io/
• learn.chef.io