This document discusses the importance of understanding production operations from an architect and developer perspective to create resilient and operable systems. It highlights the critical roles of monitoring, alerting, and service delivery in maintaining a live service environment, emphasizing the need for proper procedures and automation. Additionally, it covers the balance between resilience, performance, and operability, as well as the financial impact of service outages.

1. ProductionOperationsAn Architect and Developer Perspective“Understanding the production environment and the core functions of the live service Operations and Service Delivery teams helps us to design and build solutions that can be better operated and supported during live service and beyond.”

2. About Mewww.design-build-run.net

3. About This SessionThis session provides practical insights and proven principles for architects and developers to help design, build and implement solutions that are resilient and operable.

4. The Production Service is a critical part of running and sustaining the business.

5. The Production Environmentis the execution foundation for the live service systemand needs to be resilient and operable.

6. Data Centers arefacilities where all the productionservers, networking components and infrastructure are stored.Courtesy of Wikipedia

7. Live Service Management includes Operations, Service Deliveryand Application Maintenance.

8. The Production Operations team is responsible for monitoring and operatingall the systems in live service.Courtesy of Wikipedia

9. Monitoring Softwareprovides a viewon the currentstateof the organization’s production applications andenvironments.

10. Monitoring Rulesare used to filter and control what information is collected and what alerts get raised on the bridge.

11. Informational Alerts indicate something of interest has happened.INFORMATION – SOMETHING OF INTEREST HAS HAPPENED

12. Warning Alerts indicate something that might need attention.INFORMATION – SOMETHING OF INTEREST HAS HAPPENEDWARNING – A ERROR IS ABOUT TO OCCUR (MAYBE)

13. Error Alerts indicate a critical situation has occurred.INFORMATION – SOMETHING OF INTEREST HAS HAPPENEDWARNING – A ERROR IS ABOUT TO OCCUR (MAYBE)CRITICAL ERROR – A CRITICAL ERROR HAS OCCURRED

14. Consider: “All servers will have processor utilization, measured over any 5 minute period, of less than 70% for 95% of the time.”Information, Warning or Alert?CPU Spike

15. Unnecessary alertsare justnoise and will be ignored.

16. Hardware Fault Tolerance improves availability and can increase performance and scalability.

17. Software Fault Tolerance improves availabilityand provides non-stop processing.NOSTOPPING

18. Wait-Retrylogic can help ensure your software isresilient.

19. Data Replication and Recovery helps increase availability and reduce failover time

20. We need to carefully BalanceResilience along withPerformance and Operability.

21. Alertsshould contain information that is useful from an operational and investigative point of view.

22. The Operations Procedures are the most important “documents” for Live Service Support.OperationsProcedures

23. Automate corrective actions(where possible) to reducemanual effort.

24. Service Delivery is responsible for supporting the production service.

25. Batch and house-keeping routines ensure the system is in good operational state.

26. Reporting allows us to monitor and trend live service performance and can help with Incident Investigation.

27. Incident InvestigationInvestigate

28. Remember Live Service Outages and Down Time cost money.

29. Thank YouQuestions?