Netflix Women Living on the "Edge" - WiT event
•Download as PPTX, PDF•
0 likes•310 views
Women of Edge Engineering @Netflix talking about the work we do
Recommended
Recommended
More Related Content
Similar to Netflix Women Living on the "Edge" - WiT event
Similar to Netflix Women Living on the "Edge" - WiT event (20)
More from Sangeeta Narayanan
More from Sangeeta Narayanan (13)
Recently uploaded
Recently uploaded (20)
Netflix Women Living on the "Edge" - WiT event
- 1. Women Living On The Edge Women in Tech Dinner March 22, 2018 Netflix HQ, Los Gatos
- 2. 190+ Countries 117+ Million Members
- 4. 140+ Million Hours Streamed DAILY
- 7. Edge powers core product experiences
- 8. Edge powers core product experiences
- 9. Edge powers core product experiences
- 10. Edge powers core product experiences
- 11. Compelling product Reliable experience
- 12. Compelling product Reliable experience
- 13. Zuul Cloud Gateway Mid-Tier Services Netflix Edge Architecture API Service Layer NodeQuark Edge Device Services AuthN Services DEVELOPER PRODUCTIVITY OPERATIONAL INSIGHTS Playback Licensing Netflix Micro- services
- 14. Gaya Varadarajan Senior Software Engineer Cloud Gateway Kim Trott Engineering Director Edge Device Services Karen Casella Engineering Leader Edge & Playback Access Haripriya Murthy Senior Software Engineer Playback Licensing Sangeeta Narayanan Engineering Director Edge Dev Experience Daniela Enyedi Senior Software Engineer DNA API Meet the Women Living on the Edge
- 16. Zuul Cloud Gateway Mid-Tier Services API Service Layer NodeQuark Edge Device Services AuthN Services DEVELOPER PRODUCTIVITY OPERATIONAL INSIGHTS Playback Licensing Netflix Micro- services Cloud Gateway
- 17. Persistence Tier Business services Tier Presentation Tier Website API DB Traffic steering, routing, and insights
- 18. Resiliency Persistence Tier Business services Tier Presentation Tier Website API DB
- 19. Resiliency Persistence Tier Business services Tier Presentation Tier Website API DB
- 20. Resiliency Persistence Tier Business services Tier Presentation Tier Website API DB Persistence Tier Business services Tier Presentation Tier Website API DB Persistence Tier Business services Tier Presentation Tier Website API DB
- 21. Zuul to Backend Debugging Throttling Contextual Alerting Sticky Canary Zuul to Device DDOS Mitigation Anomaly detection Security Protocols Canarying Cross Region Resiliency Smart Load Balancing Flexibility Alerts Load Shedding Resiliency Self Service Routing Custom Canary Push Messages
- 22. Debugging Throttling Contextual Alerting Sticky Canary Zuul to Device DDOS Mitigation Anomaly detection Security Protocols Canarying Smart Load Balancing Flexibility Alerts Load Shedding Resiliency Self Service Routing Custom Canary Cross Region Resiliency Zuul to Backend Push Messages
- 26. Reject Spam Configure timeouts Sticky Canary Custom Canary Primary Routing Ramp new Backend Route Overrides Squeeze testing Self Service Routing
- 27. Reject Spam Configure timeouts Sticky Canary Custom Canary Primary Routing Ramp new Backend Route Overrides Squeeze testing Self Service Routing
- 29. Reject Spam Configure timeouts Sticky Canary Custom Canary Primary Routing Ramp new Backend Route Overrides Squeeze testing Self Service Routing
- 31. Reject Spam Configure timeouts Sticky Canary Custom Canary Primary Routing Ramp new Backend Route Overrides Squeeze testing Self Service Routing
- 33. Reject Spam Configure timeouts Sticky Canary Custom Canary Primary Routing Ramp new Backend Route Overrides Squeeze testing Self Service Routing
- 35. Reject Spam Configure timeouts Sticky Canary Custom Canary Primary Routing Ramp new Backend Route Overrides Squeeze testing Self Service Routing
- 37. Reject Spam Configure timeouts Sticky Canary Custom Canary Primary Routing Ramp new Backend Route Overrides Squeeze testing Self Service Routing
- 38. Reject Spam Configure timeouts Sticky Canary Custom Canary Primary Routing Ramp new Backend Route Overrides Squeeze testing Self Service Routing
- 42. Talk to Me About... ● Self Service Routing ● Traffic Shaping ● DDOS Mitigation ● Availability & Reliability ● Insights
- 43. Edge Device Services Kim Trott
- 44. Zuul Cloud Gateway Mid-Tier Services API Service Layer NodeQuark Edge Device Services AuthN Services DEVELOPER PRODUCTIVITY OPERATIONAL INSIGHTS Playback Licensing Netflix Micro- services NodeQuark
- 47. What’s similar? What’s different?
- 48. I MY BFF
- 53. Platform as a Service Allows engineers “...to develop, run, and manage applications without the complexity of building and maintaining infrastructure…” - wikipedia
- 54. Isolation
- 56. Isolation ● Isolate failures ● Independent root cause
- 57. Isolation ● Isolate failures ● Independent root cause ● Fine-grained capacity management
- 58. Isolation ● Isolate failures ● Independent root cause ● Fine-grained capacity management ● Better regression / performance testing
- 60. Talk to Me About... ● Node.js ● Observability ● Availability and Reliability ● gRPC ● Diagnostics Tools
- 61. API Service Layer Karen Casella
- 62. Zuul Cloud Gateway Mid-Tier Services API Service Layer NodeQuark Edge Device Services AuthN Services DEVELOPER PRODUCTIVITY OPERATIONAL INSIGHTS Playback Licensing Netflix Micro- services API Service Layer
- 64. Sign-Up Discovery API Service Layer
- 65. Sign-Up Discovery Playback API Service Layer
- 66. Sign-Up Discovery Playback API Service Layer ● Orchestration
- 67. Sign-Up Discovery Playback API Service Layer ● Orchestration ● Availability Protection
- 68. Sign-Up Discovery Playback API Service Layer ● Orchestration ● Availability Protection ● Abstraction
- 70. Challenges & Opportunities Time to ResolveObservability
- 71. Challenges & Opportunities Time to Resolve Blast RadiusObservability
- 72. Breaking Sharding the Monolith “The price of reliability is the pursuit of the utmost simplicity” - C.A.R. Hoare, Turing Award Lecture, 1980
- 73. Sharding Strategies CRITICAL NON CRITICAL Traffic Criticality Traffic Volume (Function or Source)
- 74. Mid-Tier Services Example API Sharding Mobile API Shard Ratings Smart TV API Shard Bookmarks Account
- 75. Results ↓Time to Detect & Debug
- 76. Results ↓Time to Resolve ↓Time to Detect & Debug
- 77. Results ↓Time to Resolve ↓ Blast Radius ↑ Reliability ↓Time to Detect & Debug
- 78. Trade-Offs ● ↑ Configuration Overhead ● ↑ Operational Complexity Operational Overhead Availability
- 79. Talk to Me About... ● Playback Licensing / DRM Tech ● Authentication Services ● Availability & Reliability ● Inclusion & Diversity ● Operations @ Scale ● Availability & Reliability ● Service Orchestration
- 82. Zuul Cloud Gateway Mid-Tier Services API Service Layer NodeQuark Edge Device Services AuthN Services DEVELOPER PRODUCTIVITY OPERATIONAL INSIGHTS Playback Licensing Netflix Micro- services Playback Licensing
- 85. Chunking & Encryption CDN Key Store Playback Licensing Playback API 1 2
- 87. Mid-Tier Services Playback Licensing API Service Layer Playback Licensing License Server License Server License Server
- 88. Mid-Tier Services Playback Licensing API Service Layer Playback Licensing License Server Offline License Server License Server Offline Key Server Key Server Netflix Micro- services
- 89. Mid-Tier Services Playback Licensing API Service Layer Playback Licensing License Server Offline License Server License Server Offline Key Server Key Server Netflix Micro- services
- 93. Mid-Tier Services Playback Licensing API Service Layer Playback Licensing License Server Offline License Server License Server Offline Key Server Key Server Netflix Micro- services
- 94. Talk to Me About... ● Playback Licensing / DRM Tech ● New Engineer @ Netflix ● Chaos Engineering ● Service Reliability
- 95. Edge Developer Experience Sangeeta Narayanan
- 96. Zuul Cloud Gateway Mid-Tier Services API Service Layer NodeQuark Edge Device Services AuthN Services DEVELOPER PRODUCTIVITY OPERATIONAL INSIGHTS Playback Licensing Netflix Micro- services Productivity & Insights
- 97. High rate of change
- 100. Example: Rapid local iterations
- 103. Detect Recover Root cause
- 105. Example: Domain specific insights
- 106. Build in observability BE SAFE
- 109. MOVE FAST, BE SAFE, STAY CALM Compelling product Reliable experience
- 110. Learn more The Netflix Tech Blog https://netflix.github.io/
- 111. Talk to Me About... ● Developer Experience ● Observability ● PaaS, Containers, Serverless ● Engineering Leadership
- 112. Thank you!
- 113. Join Us for Dinner! #SheRules
Editor's Notes
- Edge provides functionality and metadata to power core Netflix product experiences
- Edge provides functionality and metadata to power core Netflix product experiences
- Edge provides functionality and metadata to power core Netflix product experiences
- Edge provides functionality and metadata to power core Netflix product experiences
- Zuul: Internet facing service, provides routing, traffic shaping, security and more NQ: Translation layer to optimize experience for each device API: Orchestration and abstraction over other Netflix services PBL: Handles content licensing EDX: Tools and infrastructure to enable engineers to develop and operate complex systems at scale
- Handles traffic from nflx customer all over the world supporting 1000’s of device types, proxying 10s of billions of requests in a day Zuul is the front door to nflx server infra
- Zuul is analogous to traffic steering, routing, and insights into Netflix’s cloud systems
- Example of tv scenario of a bad config change and how zuul scaled well
- Example of tv scenario of a bad config change and zuul scaled well
- Cross Region Resiliency when a backend gets into trouble in one region.
- Zuul offers a lot of cool features But I am going to focus on self service routing
- At the begining, few requests were coming in for route changes
- Getting Gateway out of the way for route changes Self Service UI
- Assign her primary route
- Ramp a new backend slowly to productionize it
- Override a small % of traffic to a single instance cluster for debugging
- squeeze some traffic to establish benchmark of CPU to RPS
- Configure Security rules to reject spam
- She can Assign her primary route She can Override a small % of traffic to a a single instance cluster for debugging She can squeeze some traffic the performance of the
- My name is Kim and I’ve been at Netflix for over 10 years. I started out as a UI Engineer working on the Website back when that’s the only platform we had. Now I’m deep on the server infrastructure side.
- Netflix is continuously innovating to deliver the best possible customer experience. You may be familiar with our TV UI.
- And this is our iOS mobile experience.
- While they’re both Netflix, they’re actually quite different. The form factors of TVs and mobile devices are different and input (touch vs. remote control). Also, the size and orientation of the imagery and the metadata fields making up the screen. Powering the UI takes a lot of data (and personalization algorithms), but both of these UIs have very different data needs. That’s why….
- No I’m not talking about best friends, though for some engineers this may be their best friend
- I’m talking about Backends for Frontends. Sam Newman wrote about this pattern in 2015. It’s fun when the thing you’ve been doing for years gets a name! Resources: https://samnewman.io/patterns/architectural/bff/ https://nordicapis.com/building-a-backend-for-frontend-shim-for-your-microservices/
- With a BFF, each UI team can have their own backend service for completely customizing the data needs for their UI/device application. Giving them complete control over the request-response lifecycle between the device and server.
- Translation layer enables: Customization, business logic: Get the data they need and send it back in the format that best suits that device. Rapid iteration and A/B testing: Change what data is returned without needing to coordinate with other teams Creates thinner, more focused services
- The core charter of our UI engineering team is to rapidly iterate on the user experience. Edge provides a platform that enables devices teams to rapidly and easily deploy services for their front-end application, without having to deal with the complexity of server infrastructure, high availability, fault tolerance, etc. We provide the platform and manage the infrastructure. They bring their code / scripts. Use Node.js as the technology as best overall fit for UI teams.
- Isolation: Isolate failures Independent root cause of issues Independent autoscaling Better regression / performance testing
- Isolation: Isolate failures Independent root cause of issues Independent autoscaling Better regression / performance testing
- Isolation: Isolate failures Independent root cause of issues Independent autoscaling Better regression / performance testing
- Isolation: Isolate failures Independent root cause of issues Independent autoscaling Better regression / performance testing
- Isolation: Isolate failures Independent root cause of issues Independent autoscaling Better regression / performance testing
- If every BFF had to talk to the hundreds of microservices at Netflix, it would overwhelm our UI teams and prevent them from rapidly iterating on the user experience. That’s why we have the API Service Layer to aggregate and orchestrate all the mid-tier at Netflix and insulate and abstract that layer from UI Engineering teams. That’s what Karen Casella is going to talk about next.
- API service layer Traffic sharding to improve availability
- Orchestration: Zuul -> NQ -> API: orchestrates / owns order of operations / fetches data from back-end systems, aggregates & returns data to upstream
- Availability Protection: priority is to favor streaming over all other functionality, API can not go down entirely, Hystrix fault tolerance pattern as a library, handle errors with fallbacks, may result in degraded customer experience, but at least they can stream, throttling
- Abstraction: shield upstream teams from downstream system knowledge & changes, provides APIs not provided by downstream services, caching, batch APIs
- Monolithic architecture challenges Observability Reduce time to detect & debug issues Flamegraphs have too much information
- Time to Resolve Monolith, high start-up times Inhibits fast releases / rollbacks Image credits https://www.gannett-cdn.com/-mm-/c7c72be3b5ba5526bd2a95f450ca45139f4b704f/c=0-79-1483-1191&r=x404&c=534x401/local/-/media/2015/08/06/Indianapolis/B9318361924Z.1_20150806155035_000_GPCBIAR98.1-0.png https://www.netbraintech.com/wp-content/uploads/2017/07/saas-mttr-300x187.png
- Blast Radius Isolation in failure scenarios Bad code push / downstream service unavailable Image credits https://www.gannett-cdn.com/-mm-/c7c72be3b5ba5526bd2a95f450ca45139f4b704f/c=0-79-1483-1191&r=x404&c=534x401/local/-/media/2015/08/06/Indianapolis/B9318361924Z.1_20150806155035_000_GPCBIAR98.1-0.png https://www.netbraintech.com/wp-content/uploads/2017/07/saas-mttr-300x187.png
- http://devopsagenda.techtarget.com/opinion/Break-the-monolith-Loosely-coupled-architecture-brings-DevOps-success
- Benefits of sharding greatly outweigh the complexity and up-front investment
- https://tenor.com/view/netflix-loading-gif-6089689
- https://dailyplateofcrazy.com/wp-content/uploads/2016/04/The-Key-to-Finding-the-Missing-Piece-of-the-Puzzle.jpg
- https://www.youtube.com/watch?v=WhzDEhdOeic https://github.com/Netflix/Hystrix Bulk heading and fault tolerance
- https://www.pinterest.com/pin/504825439451023649/
- http://brookscanavesi.com/blog/mobile-app-development/technology-trends/chaos-engineering-breaking-things-purpose/
- 100s of changes in a 24 hour period. How to sustain this velocity?
- From idea to release in shortest amount of time
- Reducing friction in the development process enables our engineers to move fast.
- Moving fast is risky at our scale and complexity. Failure is inevitable.
- Moving fast is risky at our scale and complexity. We embrace failure and focus on minimizing the time to detect, recover from and root cause issues.
- Our observability suite includes various capabilities such as the ones listed here.
- Granular insights into system behavior - per device, per request, in near real time
- http://bit.ly/2Dsbjsz
- Our business has seen impressive growth over the past few years. We work hard to ensure our systems can scale to support this type of growth.
- Bur not just enough that we scale our systems. Scaling ourselves is equally important.
- An emerging area of focus is the human factors involved in operating in an environment such as ours. We believe that is key to maintaining the balance between velocity and reliability.
- Join the women of Edge Eng and our allies for dinner
- Zuul: Internet facing service, provides routing, traffic shaping, security and more NQ: Translation layer to optimize experience for each device API: Orchestration and abstraction over other Netflix services PBL: Handles content licensing EDX: Tools and infrastructure to enable engineers to develop and operate complex systems at scale