Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale
LinkedIn has evolved from serving live traffic out of one data center to four data centers spread geographically. Serving live traffic from four data centers at the same time has taken the company from a disaster recovery model to a disaster avoidance model, where an unhealthy data center can be taken out of rotation and its traffic redistributed to the healthy data centers within minutes, with virtually no visible impact to users. As LinkedIn transitioned from big monolithic applications to microservices, it was difficult to determine capacity constraints of individual services to handle extra load during disaster scenarios. Stress testing individual services using artificial load in a complex microservices architecture wasn’t sufficient to provide enough confidence in data center’s capacity. To solve this problem, LinkedIn leverages live traffic to stress services site-wide by shifting traffic to simulate a disaster load. Michael Kehoe and Anil Mallapur discuss how LinkedIn uses traffic shifts to mitigate user impact by migrating live traffic between its data centers and stress test site-wide services for improved capacity handling and member experience.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale
Similar to Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale (20)
More from Michael Kehoe
More from Michael Kehoe (20)
Recently uploaded
Recently uploaded (20)
Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale
- 1. TrafficShift: Avoiding Disasters at Scale Jeff Weiner Chief Executive Officer Michael Kehoe Staff SRE Anil Mallapur Sr SRE
- 2. Today’s agenda 1 Introductions 2 Evolution of the Infrastructure 3 Planning for Disaster 4 LinkedIn Traffic-Tier 5 TrafficShift 6 Load Testing 7 Q&A
- 3. Key Takeaways • Design infrastructure to facilitate disaster recovery • Test regularly • Automate everything
- 5. World’s largest professional network Largest global network of professionals 500+M members Serving users world- wide 200+ Countries
- 6. Who are we? PRODUCTION-SRE TEAM AT LINKEDIN • Assist in restoring stability to services during site-critical issues • Develop applications to improve MTTD and MTTR • Provide direction and guidelines for site monitoring • Build tools for efficient site-issue detection, correlation & troubleshooting,
- 8. Terminologies • Fabric/Colo Data Center with full application stack deployed • PoP/ Edge Entry point to LinkedIn network (TCP/ SSL Termination) • Load Test Planned stress testing of data centers
- 10. Evolution of the Infrastructure 2003 2010 2011 2013 2014 2017 Active & Passive Active & Active Multi-colo 3-way Active & Active Multi-colo n-way Active & Active
- 11. 2017 4 Data Centers 13 PoPs 1000+ services
- 13. Why care about Disasters ?
- 14. What are Disasters Service Degradation Infrastructure Issues Human Error Data Center on Fire
- 15. One Solution for all Disasters • TrafficShift – Reroute user traffic to different datacenters without any user interruption.
- 18. LinkedIn Traffic-Tier Border Router IPVS ATS ATS Frontend EDGE FABRIC Stickyrouting
- 19. LinkedIn Traffic-Tier ATS EDGE FABRIC DC1 DC2 DC1 in Cookie Got DC2 as primary fabric Gets primary fabric for user Stickyrouting
- 20. LinkedIn Traffic-Tier Fabric Buckets 1 91 2 3 10 92 93 100
- 21. How Stickyrouting assigns users to a fabric? Capacity of a Datacenter Geographic distance to users Hadoop
- 22. Advantages of Stickyrouting Less Latency Store data where needed Control over capacity
- 23. TrafficShift
- 24. Site Traffic and Disaster Recovery DC2 DC3 DC1 DC4 EDGE 30% Distributed Load 50% Distributed Load 50% Distributed Load 10% Distributed Load Traffic stops being served to offline fabrics when we mark buckets offline Traffic is shifted to online fabrics as ATS redirects those users to their secondary fabric DC1 DC4
- 26. TrafficShift Architecture Web application Salt master Stickyrouting ServiceCouchbase Backend Worker Processes FABRIC BUCKETS
- 27. Load Testing
- 28. What is Load Testing? 3x a week Peak hour traffic Fixed SLA
- 30. Benefits of Load testing Capacity Planning Stress Test Identify Bugs Confidence
- 31. Big Red Buttom • Kill-switch for a datacenter • Failout of a datacenter & PoP in minutes • Minimal user impact
- 32. Key Takeaways
- 33. Key Takeaways • Design infrastructure to facilitate disaster recovery • Stress test regularly to avoid surprises • Automate everything to reduce time to mitigate impact
- 34. Q & A