Downloaded 387 times
Uploaded byMikey Cohen - Hiring Amazing Engineers
Rethinking Cloud Proxies
The document discusses Netflix's gateway architecture, emphasizing its evolution from a simple proxy to a sophisticated gateway handling billions of requests daily. It highlights key features such as dynamic routing, load balancing, and the importance of insights, while also noting the challenges and lessons learned throughout the transition. Additionally, it details strategic principles for managing service traffic and ensuring system resiliency under varying conditions.
More Related Content
![[OpenStack] 공개 소프트웨어 오픈스택 입문 & 파헤치기](https://cdn.slidesharecdn.com/ss_thumbnails/opensource-skimmingopenstackwithkoreausergroup-180824144737-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Rethinking Cloud Proxies
![[QCon London 2020] The Future of Cloud Native API Gateways - Richard Li](https://cdn.slidesharecdn.com/ss_thumbnails/thefutureofcloud-nativeapigateways-200305225127-thumbnail.jpg?width=640&height=640&fit=bounds)
Rethinking Cloud Proxies
- 1. A Cloud Gateway- A Large Scale Company’s First Line of Defense Mikey Cohen Manager - Edge Gateway Netflix
- 2. Today, more than36% of North America’s internet traffic is controlled by systems in the Amazon Cloud
- 4. Global Streaming ofTV Shows and Movies
- 5. Nearly 70 MillionSubscribers In over 80 Countries
- 6. Netflix accounts forover 36% of Downstream Traffic in North America
- 7. From the Internetto Services in the Cloud Gateway Gateway ????? Origin (API) Origin (API) API Origin (API) Origin (API) Website
- 8. Our Edge Gateway@ Netflix Handles most netflix.com hosts Over 20 production Zuul clusters ~ 50 elbs Gateway handles ~10 origin services
- 9. Netflix Gateway Scale Tensof billions of requests per day 3 AWS regions Over 1000 device types Hundreds of permutations of protocols and device versions
- 10.
- 11. So What!? -Change your perspective!!
- 12. Traditional Cloud ProxyMission Simple static rule-based routing API portal Request authentication Throttling - request caps Monitoring Caching
- 13. The Gateway -a grown-up proxy! ●Dynamic routing ●Deep Insights ●Load balancing ●Availability focused ●Service protection ●Quality assurance tool
- 14. Evolving to aGateway
- 15. Netflix’s Public API Late2008 Mashery Datacenter
- 16. Streaming Devices usingpublic API Early Streaming Devices - 2009 Windows Media Center XBox PS3
- 17. Migration to AWS 2010 Sonoa/ Apigee proxy Device traffic, not public Controlling DC -> cloud migration Running in AWS Under Netflix control
- 18.
- 19. Anti-patterns of mostcloud proxies Static configurations Service push needed to change behavior Limited range of functionality Limited to HTTP
- 20. Zuul Created 2012 Dynamically injectedand compiled filters Manipulate requests and responses Headers / Body / etc Change routing Add metrics and other functions Built on Netflix’s OSS stack Open Sourced
- 21. Zuul - AVictim of Success Easy and convenient Instant results High adoption Happy customers Business logic in proxy Affects system resiliency Zuul team in critical path
- 22.
- 23. Principles of Netflix’sGateway Strategy Creative Routing Dynamic Routing Delivery Focused Traffic Shaping React Fast Insights
- 24. Creative Routing -Subclusters with Purpose Gateway Gateway Gateway Origin (API) v1 v2 test debug Instrumented squeeze “sticky” canarybaseline “sticky” baseline v1 v2 test debug baseline canary “sticky” canary “sticky” baselineFIT Instrumented squeeze
- 25. Red / GreenDeployments Gateway Gateway Gateway Origin (API) v1 v2 test debug canary Instrumented squeeze “sticky” canarybaseline “sticky” baseline v1 v2 test debug baseline canary “sticky” canary “sticky” baselineFIT Instrumented Instrumented squeeze squeeze
- 26. Developer Test Branches Gateway Gateway Gateway Origin(API) v1 v2 test debug canary Instrumented squeeze “sticky” canarybaseline “sticky” baseline v1 v2 test debug baseline canary “sticky” canary “sticky” baselineFIT Instrumented Instrumented squeeze squeeze
- 27. Instrumented Clusters Gateway Gateway Gateway Origin (API) v1 v2 test debug canary Instrumented squeeze “sticky” canarybaseline “sticky” baseline v1 v2 test debug baselinecanary “sticky” canary “sticky” baselineFIT Instrumented squeeze squeeze
- 28. Squeeze Testing Gateway Gateway Gateway Origin (API) v1 v2 test debug canary Instrumented squeeze “sticky” canarybaseline “sticky” baseline v1 v2 test debug baselinecanary “sticky” canary “sticky” baselineFIT Instrumented squeeze
- 29. Targeted Routing Gateway Gateway Gateway Origin (API) v1 v2 test debug canary Instrumented squeeze “sticky” canarybaseline “sticky” baseline v1 v2 test debu g baselinecanary “sticky” canary “sticky” baselineFIT Instrumented squeeze
- 30. Service “Canarying” Gateway Gateway Gateway Origin (API) v1 v2 test debug canary Instrumented squeeze “sticky” canarybaseline “sticky” baseline v1 v2 test debug baselinecanary “sticky” canary “sticky” baselineFIT Instrumented squeeze squeeze
- 31. “Sticky” Canary Gateway Gateway Gateway Origin (API) v1 v2 test debug canary Instrumented squeeze “sticky” canarybaseline “sticky” baseline v1 v2 test debug baselinecanary “sticky” canary “sticky” baselineFIT Instrumented squeeze squeeze
- 32. Failure Injection Testing Gateway Gateway Gateway Origin(API) v1 v2 test debug Instrumented squeeze “sticky” canarybaseline “sticky” baseline v1 v2 test debug baseline canary “sticky” canary “sticky” baselineFIT Instrumented squeeze squeeze
- 33. Degraded Experience Testing Gateway Gateway Gateway Origin(API) v1 v2 test debug Instrumented squeeze “sticky” canarybaseline “sticky” baseline v1 v2 test debug baseline canary “sticky” canary “sticky” baselineFIT Instrumented squeeze squeeze
- 34.
- 35. A Global CloudDeployment Persistence Tier Business services Tier Presentation Tier Network Tier Websites API Proxy DB Persistence Tier Business services Tier Presentation Tier Network Tier Websites API Proxy DB Persistence Tier Business services Tier Presentation Tier Network Tier Websites API Proxy DB
- 36. Global Cloud Routing PersistenceTier Business services Tier Presentation Tier Network Tier Websites API Proxy DB Persistence Tier Business services Tier Presentation Tier Network Tier Websites API Proxy DB Persistence Tier Business services Tier Presentation Tier Network Tier Websites API Proxy DB
- 37. A Failing region PersistenceTier Business services Tier Presentation Tier Network Tier Websites API Proxy DB Persistence Tier Business services Tier Presentation Tier Network Tier Websites API Proxy DB Persistence Tier Business services Tier Presentation Tier Network Tier Websites API Proxy DB
- 38. Gateway routing toother regions Persistence Tier Business services Tier Presentation Tier Network Tier Websites API Proxy DB Persistence Tier Business services Tier Presentation Tier Network Tier Websites API Proxy DB Persistence Tier Business services Tier Presentation Tier Network Tier Websites API Proxy DB
- 39. Attack prevention Gateway Gateway Gateway Origin (API) Origin(API) API Origin (API) Origin (API) Website
- 40.
- 41. Smart Load Balancing- Bad Nodes Gateway Gateway Gateway Origin (API)
- 42. Gateway Backoff andBlacklists Bad Nodes Gateway Gateway Gateway Origin (API)
- 43. Zone Failure -Blacklist the Zone automatically Gateway Gateway Gateway Origin (API)
- 44. React Quickly -Runtime Filter changes Gateway Gateway Gateway Origin (API) Origin (API) API Origin (API) Origin (API) Website Runtime Policy Injection
- 45. A Room witha View - Insights Gateway Gateway Gateway Origin (API) Origin (API) API Origin (API) Origin (API) Website Insights
- 46. What’s Next forNetflix’s Gateway? Gateway as a service Self-service dynamic routing / route validation Control APIs for special routing functions Netty Based Zuul (using RxNetty) Handling persistent connections non-blocking, async Transport protocol agnostic routing Reactive Socket http://reactivesocket.io/
- 47. Top Ten LessonsLearned
- 48.
- 49.
- 50.
- 51. Shard to ReduceBlast Radius
- 52. Devices are Weird Protocolsare Weird
- 53. Devices are Forever Protocolsare Forever
- 54. It will bebuilt “wrong”
- 55. Keep Business Logicout of your Gateway
- 56. For More Info... ZuulOSS Netflix Tech Blog RxNetty Jobs
Editor's Notes
- #12 Our gateway strategy will change the way you think about resiliency, debugging, continuous delivery, service operations, and insights.
- #19 Devices slow to update Need emergency policies Fast action
- #20 Limited range of functionality Hard to program Authentication Authorization Static responses / Origin specific headers Why? Federation of logic across systems creates complexity Minimize gateway dependencies to maximize availability
- #24 Origin services run many clusters Route to service clusters based on dynamic routing rules Shape or reject traffic based on service, regional health, or attack React fast in emergencies Realtime analytics and insights Ensures request delivery from internet to services running in the cloud Dynamically changes routing behaviors Routes to services Services have multiple clusters Clusters have dynamically changing nodes Bridges multiple cloud regions and data centers Provides system Insights
- #25 Same service: Subclusters for many purposes Set up by filters in Zuul Self serviceable by cluster owners Automated Quality assurance / Test Automation Targeted debugging Test Automation Canary / Baseline A/B testing of service behavior per build Squeeze Testing Service capacity testing Trickle traffic Instrumented builds Sticky Canary A/B testing of client behavior per origin build
- #28 Trickling traffic into clusters High Overhead profiling tools “Coalmine” verbose logging
- #29 Server capacity testing Gateway gradually increases traffic until performance degradation is detected Automated or manual
- #30 Isolate requests by customer, route, type of device, or any routing rule Debug node(s) are often instrumented to give verbose logging Custom Request Routing
- #31 Compare server behavior and metrics Equal traffic rates hit both clusters Automated part of production push process Error rates CPU for equivalent work Automated metrics analysis returns a score of how well the canary cluster performed A poor score stops the push process
- #32 Servers may be healthy data may be bad API changes that affect devices Data changes certain devices can’t interpret Protocol and transport changes that some devices can’t accept Testing 1000’s of types of devices would be a time consuming, tedious process. Sticky Canary idea - Stick all requests for a small subset of customers for a limited time to a “sticky canary” or “sticky baseline” If servers are equivalent, there should be no behavioral differences. Insights can help find these anomalies Limited scope of impact - a very small subset of customers could be affected but only for a short period of time
- #37 Reroute to the closer region to the client - DNS accuracy issues, etc Reroute due to region failure.
- #40 Speedbump Dynamic DDOS prevention