50 production deployments a day, at least

1. www.techarchday.fi www.techarchday.fi 50 production deployments a day, at least Oscar Renalias Senior Technology Architect v10
2. @oscarrenalias github.com/oscarrenalias github.com/Accenture oscar.renalias@accenture.com www.linkedin.com/oscarrenalias www.slideshare.net/oscarrenalias
3. 2009 2014 2015 50+ deploys per day 10 deploys per day Deploys to production every 11.6 seconds
4. Requirements Design & Build QA & Test Operate
5. QA & Test Operate Best Guess Design & Build
6. Design & Build QA & Test Operate Best Guess Feedback Experiments
7. QA & Test Operate Best Guess Design & Build 1 YEAR6 MONTHS1 MONTH2 WEEKS1 WEEKS1 DAY1 HOURMINUTES
8. Run over 200 concurrent experiments on any given day http://www.exp-platform.com/Pages/ControlledExperimentsAtLargeScale.aspx
9. THIS MANY EXPERIMENTS
10. Our objective: 50 deployments a day with no service impact
11. Requirements Design & Build QA & Test Operate Continuous Improvement
12. Requirements Design & Build QA & Test Operate Continuous Improvement
13. SAY CONTINUOUS IMPROVEMENT ONE MORE TIME
14. Continuous Delivery Automation Architecture Culture
15. Value Stream Idea Value
16. Release Ideas Values
19. Value Released Time Guessed Well
21. Value Released Time Guessed Well Guessed Badly
22. Value Released Time Value Stream Idea Value Value Stream Idea Value Value Stream Idea Value Rapid Feedback Optimise
23. Release Ideas Value “Hotfixes”
24. Ideas Value Overhead Release
25. Requirements QA & Test Operate Design & Build Continuous Integration
26. CI – what good looks like… Changes constantly pushed to version control Fixing a broken build is always the priority Nobody goes home on a broken build
27. Requirements 29 Design & Build Operate `Continuous Test QA & Test
28. Pipeline! Pipeline! Pipeline! Build Static Analysis ST Regression Build Static Analysis ST Regression Build Static Analysis ST Regression Performance Test Security Test
29. Pipeline! Pipeline! Pipeline! Build Static Analysis ST Regression Build Static Analysis ST Regression Build Static Analysis ST Regression Performance Test Security Test Effort and schedule Test Phase Comprehensive Fast
30. Pipeline! Pipeline! Pipeline! Build Static Analysis ST Regression Build Static Analysis ST Regression Build Static Analysis ST Regression Performance Test Security Test Effort and schedule Test Phase Pre-commit 1 min Component 1 hour Acceptance 4-8 hours Commit 10 mins
31. Pipeline! Pipeline! Pipeline! Build Static Analysis ST Regression Build Static Analysis ST Regression Build Static Analysis ST Regression Performance Test Security Test Effort and schedule Test Phase Stop the assembly line!
32. What about tools?
33. THEY WANTED MORE AGILITY, SO WE SOLD THEM $2M IN DEVOPS TOOLS
34. Continuous Delivery Principles Each check-in is a candidate production release The same processes and tools in ALL environments A failure at any stage stops the production line It’s not about the tools!
36. I CAN HAZ CONTINUOUS DELIVERY?
37. Fail-fast Microservices Metrics and Logs Independently deployable and scalable components Failure is unavoidable. Deal with it. Proactive monitoring
39. RELEASE TIME!
40. Oscar’s unscientific observations of release failures Human failures
41. ELIMINATE ALL HUMANS
42. Pets vs Cattle
43. Build Static Analysis ST Regression Performance Test Security Test Deployment Test & QA
44. Automation Infrastructure as code Test, test, test Eliminate the human factor
46. WORKED FINE IN DEV OPS PROBLEM NOW
47. Smooth Collaboration Fully Embedded Infrastructure-as-a-Service DevOps as a Service Separate DevOps Team DevOps Culture: Which one is right for you? Source: http://www.devopstopologies.com
48. FIRED YOUR EXISTING TEAM AND REPLACED THEM WITH A DEVOPS TOOL? THAT MUST BE GOING WELL
49. Organisation and Culture Align to business priorities Quality is everybody’s responsibility Define your own DevOps Culture People over processes and tools
50. http://bit.ly/AccentureDevOpsPlatform

Editor's Notes

Deployment every 11.6 seconds is equivalent to (24*60*60)/11.6 = 7448 deployments/day

Accelerating value delivery.

Do I need you to do 50 deploymens a day? No. But you need to desing your DevOps capabilities as if you have to.

Finally perhaps a more subtle problem is that when releases are infrequent call them whatever you want, smaller releases take place in between
These are usually harder to test, harder to track, they cause merge work and they make main releases much harder

When a deployment fails, 95% of the time is due to human error

Pets are cared for, and taken to the vet when they get ill.
When cattle are sick, they’re shot.
Same applies for infrastructure.

Automate everything

Smooth Collaboration
This is the ‘promised land’ of DevOps: smooth collaboration between Dev teams and Ops teams, each specialising where needed, but also sharing where needed. There are likely many separate Dev teams, each working on a separate or semi-separate product stack.
My sense is that this Type 1 model needs quite substantial organisational change to establish it, and a good degree of competence higher up in the technical management team. Dev and Ops must have a clearly expressed and demonstrably effective shared goal (‘Delivering Reliable, Frequent Changes’, or whatever). Ops folks must be comfortable pairing with Devs and get to grips with test-driven coding and Git, and Devs must take operational features seriously and seek out Ops people for input into logging implementations, and so on, all of which needs quite a culture change from the recent past.

Fully Shared Ops Reponsibilities
Where operations people have been integrated in product development teams, we see a Type 2 topology. There is so little separation between Dev and Ops that all people are highly focused on a shared purpose; this is arguable a form of Type 1 (Dev and Ops Collaboration), but it has some special features.
Organisations such as Netflix and Facebook with effectively a single web-based product have achieved this Type 2 topology, but I think it’s probably not hugely applicable outside a narrow product focus, because the budgetary constraints and context-switching typically present in an organisation with multiple product streams will probably force Dev and Ops further apart (say, back to a Type 1 model). This topology might also be called ‘NoOps‘, as there is no distinct or visible Operations team (although the Netflix NoOps might also be Type 3 (Ops as IaaS)).
Where operations people have been integrated in product development teams, we see a Type 2 topology. There is so little separation between Dev and Ops that all people are highly focused on a shared purpose; this is arguable a form of Type 1 (Dev and Ops Collaboration), but it has some special features.
Organisations such as Netflix and Facebook with effectively a single web-based product have achieved this Type 2 topology, but I think it’s probably not hugely applicable outside a narrow product focus, because the budgetary constraints and context-switching typically present in an organisation with multiple product streams will probably force Dev and Ops further apart (say, back to a Type 1 model). This topology might also be called ‘NoOps‘, as there is no distinct or visible Operations team (although the Netflix NoOps might also be Type 3 (Ops as IaaS)).

Ops as Infrastructure-as-a-Service
For organisations with a fairly traditional IT Operations department which cannot or will not change rapidly [enough], and for organisations who run all their applications in the public cloud (Amazon EC2, Rackspace, Azure, etc.), it probably helps to treat Operations as a team who simply provides the elastic infrastructure on which applications are deployed and run; the internal Ops team is thus directly equivalent to Amazon EC2, or Infrastructure-as-a-Service.
A team (perhaps a virtual team) within Dev then acts as a source of expertise about operational features, metrics, monitoring, server provisioning, etc., and probably does most of the communication with the IaaS team. This team is still a Dev team, however, following standard practices like TDD, CI, iterative development, coaching, etc.
The IaaS topology trades some potential effectiveness (losing direct collaboration with Ops people) for easier implementation, possibly deriving value more quickly than by trying for Type 1 (Dev and Ops Collaboration) which could be attempted at a later date.
For organisations with a fairly traditional IT Operations department which cannot or will not change rapidly [enough], and for organisations who run all their applications in the public cloud (Amazon EC2, Rackspace, Azure, etc.), it probably helps to treat Operations as a team who simply provides the elastic infrastructure on which applications are deployed and run; the internal Ops team is thus directly equivalent to Amazon EC2, or Infrastructure-as-a-Service.
A team (perhaps a virtual team) within Dev then acts as a source of expertise about operational features, metrics, monitoring, server provisioning, etc., and probably does most of the communication with the IaaS team. This team is still a Dev team, however, following standard practices like TDD, CI, iterative development, coaching, etc.
The IaaS topology trades some potential effectiveness (losing direct collaboration with Ops people) for easier implementation, possibly deriving value more quickly than by trying for Type 1 (Dev and Ops Collaboration) which could be attempted at a later date.

50 production deployments a day, at least

You are reading a preview.

50 production deployments a day, at least

More Related Content

Similar to 50 production deployments a day, at least

More from Oscar Renalias

Featured

Related Books

Related Audiobooks