Download to read offline
Uploaded bySubbu Allamaraju
Engineering operations
The document describes the evolution of engineering operations at eBay from a small manually maintained dev/test cloud to a large automated production infrastructure running the business. It went from a dev to ops ratio of 1:0 with no automation and thousands of nodes distributed across several availability zones operated 24x7 with a dev to ops ratio of 5:1. The document recommends treating infrastructure as code, managing drift through automation and audits, measuring everything as another product feature, and fostering a culture of shared accountability between dev and ops.
More Related Content
Engineering operations
- 1. Engineering Operations SubbuAllamaraju @sallamar eBay Inc.
- 2. 2012 2014 Adev/test cloud Less than a rack of compute Handcrafted by an engineer Supported by another engineer Zero automation Dev to op ratio = 1:0 Thousands of nodes Distributed across several AZs Automated Operated 24x7 Running the business Dev to op ratio: 5:1
- 3.
- 4. 1 2 34 Treat infrastructure as code
- 5. 1. Fully automatedeployments o Well known principle 2. Treat automation artifacts like you treat code o Source control → code reviews → tests → deployment 3. Take automation as a product feature o Road map, sprints, bugs, backlog, releases 4. Measure outcomes with KPIs o Time to deploy, time to recover, time to rollout a change
- 6. 1 2 34 Manage drift
- 7. System is ina state other than the desired state! - Incidents waiting to happen - Impacts time to recover - Impacts customers
- 8. Drift Automation Gaps Habits Transitional Debugging Incidents
- 9. Accept that drifthappens - and manage drift mitigation 1. Automated audits 2. Drift tracking 3. Mitigation as a planned routine activity 4. Culture - reward right habits
- 10. 1 2 34 Awareness of systems and operations
- 11. Measure everything Business KPIs Config management Alerts Another product Drift feature!
- 12. 1 2 34 Culture of shared accountability
- 13. Ops Dev Workingon what’s running the business Working on (wants to work on) new things Knows how the system fails Knows how the system is supposed to work Worries about TTR Wants to understand why Make TTR a shared goal!
- 14.
- 15.