More Related Content
Similar to 190 Countries and 5 SREs
Similar to 190 Countries and 5 SREs (20)
Recently uploaded
Recently uploaded (20)
190 Countries and 5 SREs
- 1. @jonahhorowitz 190 Countries and 5 SREs SRE at Scale
- 3. @jonahhorowitz SREs are expensive & hard to find
- 4. @jonahhorowitz SRE at Netflix C loud O perations R eliability E ngineering
- 5. @jonahhorowitz How big is Netflix?
- 7. @jonahhorowitz Extreme DevOps You write it, you run it.
- 9. @jonahhorowitz Is Your Service Production Ready? session_events session_events 100 session_logs session_logs-Argus 92 api api-prod 92 dhs dhs 92 playback_history playback_history 92 session_logs session_logs 92 session_summary session_summary-replay 92 apiproxy apiproxy-website 83 map map-dataprocessor 83 map map 83 apiproxy apiproxy-padme 83
- 10. @jonahhorowitz The last line of defense
- 13. @jonahhorowitz Developers can run Ops* *If provided the tools and support
- 14. @jonahhorowitz What are great tools?
- 15. @jonahhorowitz https://netflix.github.io/ https://jobs.netflix.com/ @altobey Incident Management and Chatops Today, Track 2, 2:30pm to 3:25pm (right now) @brendangregg Performance Checklists for SREs Tomorrow, 4:00pm to 5:00pm
Editor's Notes
- Hit next
- Audience Question: How many people here currently have an open rec for an SRE? Leave your hands up if that’s been open for more than 6 months? A year? Permanently? Are you basically always looking for an SRE?
- SREs always trying to automate themselves out of a job. The cloud was supposed to eliminate the need for “Ops”, but instead, we need SREs with even more specialized skills.
- Netflix CORE Team Only SRE team at Netflix Part of Operations Engineering ~70 people, mostly software engineers building tools, also PNE and performance
- 2400+ Employees 1500+ Engineers 100,000+ instances 700+ Microservices 30TBit/s+ at peak
- Culture of Freedom and Responsibility, Hire Smart People, Get out of the way. Provide a “paved road” for developers to follow. Allow developers to go “off road” if they determine that is the best choice. Discovery Java NIWS/Ribbon Hystrix
- Puppet, Chef, Other DSLs The value and power of Baked AMIs
- At Netflix, we leverage our SREs as consultants
- Production ready looks at several things that SREs have always looked at: Deployment Pipelines Canaries Dashboards Actionable Alerts Etc.
- The Netflix SRE team handles production issues that go uncaught by other teams - if they affect customers or cause significant risk. We also triage production issues when the cause is unknown or caused by an external service. DNS provider, AWS, Sony, Microsoft, Samsung.
- Manage incidents, get the right people in the room, make sure that documentation is kept, schedule and MC the post-mortem.
- We also write a bunch of tools - Al is working on an Incident Managment Bot Dave is working on great anomaly detection
- Netflix writes a lot of tools. Traditionally the developer writes the software, and hands it off to the “operator” to run the software. In our world, the developer writes the software, and the tools run it. It’s like a moving from a world where the ops was a chauffeur, to a world where the developer has a Tesla - it doesn’t quite drive itself, but we make sure the lane markings are clear and it will get you most of the way to work.
- Engineering Tools Team Spinnaker Dashboards Alerts UI Canary Traffic and Chaos Team Chaos Monkey, Etc. FIT Great tools are ones developers can use