Download to read offline
Uploaded byEoin Woods
Getting Your System to Production and Keeping it There
This document discusses the challenges and requirements of managing production systems, emphasizing the need for correctness, stability, capacity, and security. It outlines specific solutions, design principles, and processes for achieving these goals while highlighting the complexities and unpredictability of production environments. The author also notes that successful production management is about collaboration, automation, and continuous improvement.
More Related Content
Similar to Getting Your System to Production and Keeping it There
Getting Your System to Production and Keeping it There
- 1. Getting a Systemto Production ... and keeping it there Eoin Woods eoin.woods@endava.com www.eoinwoods.info 1
- 2. Content Introducing Production Systems WhatGoes Wrong in Production? Solutions for Production Systems Conclusions 2
- 3.
- 4. What is aproduction system? 4 Any system being used for real work
- 5. Why is ProductionisationHard? No one teaches you about production who do you talk to? what do they want? what is the definition of “done” ? Production is difficult for developers hard to access, interrogate, debug, change, ... 5
- 6. A new castof characters 6 Developers Development Users
- 7. A new castof characters 6 Production Users Developers Auditors Operations Acquirers Infrastructure Business Management
- 8. Production is constrained Highlycontrolled Content is all valuable Change can be difficult 7
- 9.
- 10. Production is highlyvisible! 9
- 11. You don’t ownproduction 10
- 12.
- 13. Performance surprises Interactive load Batchtime surprises System abusers! “all transactions this year”, “average since 1967”, ... 12
- 14. Environment bombshells Constraints andcontention Unexpected behaviour Integration points 13
- 15. Failures happen Software defects Platformfailures Environment failures 14
- 16. Security tangles Security issimple in Development Much more complex in Production! 15
- 17.
- 18. Key requirements forproduction Functionally correct does what the business process requires Stability behaves predictably in all situations Capacity can process the workload required (at all times) Security limits access to those who are authorised to have it 17
- 19. Solution Framework Correctness StabilityCapacity Security Design Principles Technology Processes Simplicity Resource Governor Threat Modelling 18
- 20. General Principles One Team Automate Measureand Improve (feedback loops) Good Enough over Perfection 19 Timeless principles … today led by CD and DevOps
- 21.
- 22. Stability - designprinciples Fail quickly fail fast, timeouts Isolate problems flow control, circuit breakers, bulkheads, asynchronous integration Ensure steady state operation housekeeping, predictable resource allocation, governors, throttling 21
- 23. Stability - technologysolutions Timeouts Circuit Breaker Fail fast Bulkhead Governor Housekeeping 22
- 24. Example - CircuitBreaker Clear Checking Tripped err_returned timeout err_returned && err_count > 10err_returned 23
- 25. Stability - processprinciples Repeatability defined processes, practice scenarios, prelive environments Automation automate the routine, automate the difficult allow the human back in the loop on demand Transparency logging, monitoring, alerts, trends 24
- 26. Stability - processautomation Logging & Metrics Monitoring Automation 25
- 27.
- 28. Capacity - designprinciples Minimise workload efficiency is important Flatten the peaks move workload around Design for the large (scalability) understand where the time goes multiply by a million 27
- 29. Capacity - technologysolutions Measure and minimise understand where the work is Caching and pre-computing reduce the work to be done Sharding and partitioning separate workload to allow scale 28
- 30. Capacity - solutions Segment Timings Staticcache Lookaside cache Precompute Result set caching Phased batch 29
- 31. Moving Work Around Utilisation 0 25 50 75 100 01 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Utilisation 0 25 50 75 100 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 30
- 32. Capacity - processprinciples Model and estimate Test capacity on realistic environments allows model calibration Monitoring and trend analysis tests theory against reality spots impending storms before they hit 31
- 33.
- 34. Security - designprinciples What they don’t have won’t hurt you least privilege - grant the minimum needed Security needs simplicity what you can’t analyse you can’t be sure about Don’t put your eggs in one basket separate privileges to avoid total breaches Fail safely 33
- 35. Security - solutions Authentication &Roles Least privilege / separation Privacy (TLS) Isolation (firewalls & zones) Trust (certs) 34
- 36. Security - processes Modelthreats to identify mitigation Define policy to know what to protect Test security as well as functions 35
- 37.
- 38.
- 39. Production is justdifferent it’s not yours and you need to respect that Production is demanding Correctness Stability Capacity Security Summary 38
- 40. Summary (ii) Solutions foreach requirement by area principles technologies processes 39
- 41. Summary (iii) Production requirementsand principles go back to the age of the mainframe CD and DevOps the latest incarnation welcome attention from developers new tech enabling new possibilities breaking down silos to make it happen 40
- 42. Books SoftwareSystems ArchitectureSecondEdition NICK ROZANSKI •EOIN WOODS Working with Stakeholders Using Viewpoints and Perspectives Second Edition 41
- 43.