Successfully reported this slideshow.

50 production deployments a day, at least

1

Share

Loading in …3
×
1 of 53
1 of 53

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Related Audiobooks

Free with a 14 day trial from Scribd

See all

50 production deployments a day, at least

  1. 1. www.techarchday.fi www.techarchday.fi 50 production deployments a day, at least Oscar Renalias Senior Technology Architect v10
  2. 2. @oscarrenalias github.com/oscarrenalias github.com/Accenture oscar.renalias@accenture.com www.linkedin.com/oscarrenalias www.slideshare.net/oscarrenalias
  3. 3. 2009 2014 2015 50+ deploys per day 10 deploys per day Deploys to production every 11.6 seconds
  4. 4. Requirements Design & Build QA & Test Operate
  5. 5. QA & Test Operate Best Guess Design & Build
  6. 6. Design & Build QA & Test Operate Best Guess Feedback Experiments
  7. 7. QA & Test Operate Best Guess Design & Build 1 YEAR6 MONTHS1 MONTH2 WEEKS1 WEEKS1 DAY1 HOURMINUTES
  8. 8. Run over 200 concurrent experiments on any given day http://www.exp-platform.com/Pages/ControlledExperimentsAtLargeScale.aspx
  9. 9. THIS MANY EXPERIMENTS
  10. 10. Our objective: 50 deployments a day with no service impact
  11. 11. Requirements Design & Build QA & Test Operate Continuous Improvement
  12. 12. Requirements Design & Build QA & Test Operate Continuous Improvement
  13. 13. SAY CONTINUOUS IMPROVEMENT ONE MORE TIME
  14. 14. Continuous Delivery Automation Architecture Culture
  15. 15. Value Stream Idea Value
  16. 16. Release Ideas Values
  17. 17. Release Ideas Values
  18. 18. Release Ideas Values
  19. 19. Value Released Time Guessed Well
  20. 20. Release Ideas Values
  21. 21. Value Released Time Guessed Well Guessed Badly
  22. 22. Value Released Time Value Stream Idea Value Value Stream Idea Value Value Stream Idea Value Rapid Feedback Optimise
  23. 23. Release Ideas Value “Hotfixes”
  24. 24. Ideas Value Overhead Release
  25. 25. Requirements QA & Test Operate Design & Build Continuous Integration
  26. 26. CI – what good looks like… Changes constantly pushed to version control Fixing a broken build is always the priority Nobody goes home on a broken build
  27. 27. Requirements 29 Design & Build Operate `Continuous Test QA & Test
  28. 28. Pipeline! Pipeline! Pipeline! Build Static Analysis ST Regression Build Static Analysis ST Regression Build Static Analysis ST Regression Performance Test Security Test
  29. 29. Pipeline! Pipeline! Pipeline! Build Static Analysis ST Regression Build Static Analysis ST Regression Build Static Analysis ST Regression Performance Test Security Test Effort and schedule Test Phase Comprehensive Fast
  30. 30. Pipeline! Pipeline! Pipeline! Build Static Analysis ST Regression Build Static Analysis ST Regression Build Static Analysis ST Regression Performance Test Security Test Effort and schedule Test Phase Pre-commit 1 min Component 1 hour Acceptance 4-8 hours Commit 10 mins
  31. 31. Pipeline! Pipeline! Pipeline! Build Static Analysis ST Regression Build Static Analysis ST Regression Build Static Analysis ST Regression Performance Test Security Test Effort and schedule Test Phase Stop the assembly line!
  32. 32. What about tools?
  33. 33. THEY WANTED MORE AGILITY, SO WE SOLD THEM $2M IN DEVOPS TOOLS
  34. 34. Continuous Delivery Principles Each check-in is a candidate production release The same processes and tools in ALL environments A failure at any stage stops the production line It’s not about the tools!
  35. 35. Continuous Delivery Automation Architecture Culture
  36. 36. I CAN HAZ CONTINUOUS DELIVERY?
  37. 37. Fail-fast Microservices Metrics and Logs Independently deployable and scalable components Failure is unavoidable. Deal with it. Proactive monitoring
  38. 38. Continuous Delivery Automation Architecture Culture
  39. 39. RELEASE TIME!
  40. 40. Oscar’s unscientific observations of release failures Human failures
  41. 41. ELIMINATE ALL HUMANS
  42. 42. Pets vs Cattle
  43. 43. Build Static Analysis ST Regression Performance Test Security Test Deployment Test & QA
  44. 44. Automation Infrastructure as code Test, test, test Eliminate the human factor
  45. 45. Continuous Delivery Automation Architecture Culture
  46. 46. WORKED FINE IN DEV OPS PROBLEM NOW
  47. 47. Smooth Collaboration Fully Embedded Infrastructure-as-a-Service DevOps as a Service Separate DevOps Team DevOps Culture: Which one is right for you? Source: http://www.devopstopologies.com
  48. 48. FIRED YOUR EXISTING TEAM AND REPLACED THEM WITH A DEVOPS TOOL? THAT MUST BE GOING WELL
  49. 49. Organisation and Culture Align to business priorities Quality is everybody’s responsibility Define your own DevOps Culture People over processes and tools
  50. 50. http://bit.ly/AccentureDevOpsPlatform

Editor's Notes

  • Deployment every 11.6 seconds is equivalent to (24*60*60)/11.6 = 7448 deployments/day
  • Accelerating value delivery.
  • Do I need you to do 50 deploymens a day? No. But you need to desing your DevOps capabilities as if you have to.
  • Finally perhaps a more subtle problem is that when releases are infrequent call them whatever you want, smaller releases take place in between
    These are usually harder to test, harder to track, they cause merge work and they make main releases much harder
  • Finally perhaps a more subtle problem is that when releases are infrequent call them whatever you want, smaller releases take place in between
    These are usually harder to test, harder to track, they cause merge work and they make main releases much harder
  • When a deployment fails, 95% of the time is due to human error
  • Pets are cared for, and taken to the vet when they get ill.
    When cattle are sick, they’re shot.
    Same applies for infrastructure.
  • Automate everything
  • Smooth Collaboration
    This is the ‘promised land’ of DevOps: smooth collaboration between Dev teams and Ops teams, each specialising where needed, but also sharing where needed. There are likely many separate Dev teams, each working on a separate or semi-separate product stack.
    My sense is that this Type 1 model needs quite substantial organisational change to establish it, and a good degree of competence higher up in the technical management team. Dev and Ops must have a clearly expressed and demonstrably effective shared goal (‘Delivering Reliable, Frequent Changes’, or whatever). Ops folks must be comfortable pairing with Devs and get to grips with test-driven coding and Git, and Devs must take operational features seriously and seek out Ops people for input into logging implementations, and so on, all of which needs quite a culture change from the recent past.

    Fully Shared Ops Reponsibilities
    Where operations people have been integrated in product development teams, we see a Type 2 topology. There is so little separation between Dev and Ops that all people are highly focused on a shared purpose; this is arguable a form of Type 1 (Dev and Ops Collaboration), but it has some special features.
    Organisations such as Netflix and Facebook with effectively a single web-based product have achieved this Type 2 topology, but I think it’s probably not hugely applicable outside a narrow product focus, because the budgetary constraints and context-switching typically present in an organisation with multiple product streams will probably force Dev and Ops further apart (say, back to a Type 1 model). This topology might also be called ‘NoOps‘, as there is no distinct or visible Operations team (although the Netflix NoOps might also be Type 3 (Ops as IaaS)).
    Where operations people have been integrated in product development teams, we see a Type 2 topology. There is so little separation between Dev and Ops that all people are highly focused on a shared purpose; this is arguable a form of Type 1 (Dev and Ops Collaboration), but it has some special features.
    Organisations such as Netflix and Facebook with effectively a single web-based product have achieved this Type 2 topology, but I think it’s probably not hugely applicable outside a narrow product focus, because the budgetary constraints and context-switching typically present in an organisation with multiple product streams will probably force Dev and Ops further apart (say, back to a Type 1 model). This topology might also be called ‘NoOps‘, as there is no distinct or visible Operations team (although the Netflix NoOps might also be Type 3 (Ops as IaaS)).

    Ops as Infrastructure-as-a-Service
    For organisations with a fairly traditional IT Operations department which cannot or will not change rapidly [enough], and for organisations who run all their applications in the public cloud (Amazon EC2, Rackspace, Azure, etc.), it probably helps to treat Operations as a team who simply provides the elastic infrastructure on which applications are deployed and run; the internal Ops team is thus directly equivalent to Amazon EC2, or Infrastructure-as-a-Service.
    A team (perhaps a virtual team) within Dev then acts as a source of expertise about operational features, metrics, monitoring, server provisioning, etc., and probably does most of the communication with the IaaS team. This team is still a Dev team, however, following standard practices like TDD, CI, iterative development, coaching, etc.
    The IaaS topology trades some potential effectiveness (losing direct collaboration with Ops people) for easier implementation, possibly deriving value more quickly than by trying for Type 1 (Dev and Ops Collaboration) which could be attempted at a later date.
    For organisations with a fairly traditional IT Operations department which cannot or will not change rapidly [enough], and for organisations who run all their applications in the public cloud (Amazon EC2, Rackspace, Azure, etc.), it probably helps to treat Operations as a team who simply provides the elastic infrastructure on which applications are deployed and run; the internal Ops team is thus directly equivalent to Amazon EC2, or Infrastructure-as-a-Service.
    A team (perhaps a virtual team) within Dev then acts as a source of expertise about operational features, metrics, monitoring, server provisioning, etc., and probably does most of the communication with the IaaS team. This team is still a Dev team, however, following standard practices like TDD, CI, iterative development, coaching, etc.
    The IaaS topology trades some potential effectiveness (losing direct collaboration with Ops people) for easier implementation, possibly deriving value more quickly than by trying for Type 1 (Dev and Ops Collaboration) which could be attempted at a later date.
  • ×