LinkedIn has evolved from serving live traffic out of one data center to four data centers spread geographically. Serving live traffic from four data centers at the same time has taken us from a disaster recovery model to a disaster avoidance model where we can take an unhealthy data center out of rotation and redistribute its traffic to the healthy data centers within minutes, with virtually no visible impact to users
As we transitioned from big monolithic application to micro-services, we witnessed pain in determining capacity constraints of individual services to handle extra load during disaster scenarios. Stress testing individual services using artificial load in a complex micro-services architecture wasn’t sufficient to provide enough confidence in a data center’s capacity. To solve this problem, we at LinkedIn leverage live traffic to stress services site-wide by shifting traffic to simulate a disaster load.
This talk provide details on how LinkedIn uses Traffic Shifts to mitigate user impact by migrating live traffic between its data centers and stress test site-wide services for improved capacity handling and member experience.
World’s largest professional network
Who are we ?
Production-SRE team at LinkedIn
● Assist in restoring stability to services
during site critical issues
● Developing applications to improve
MTTD and MTTR
● Provide direction and guidelines for site
● Build tools for efficient site issue
troubleshooting, issue detection &
Fabric/Colo Data Center with full application stack deployed
PoP/Edge Entry point to LinkedIn network (TCP/ SSL
Load Test Planned stress testing of data centers
2003 2010 2011 2013 2014 2015
way Active &
way Active &
How StickyRouting assigns users to a colo ?
Capacity of Fabric
Offline job to
Advantages of sticky routing
Less latency for users
Store data where it’s necessary
Provides precise control over capacity allotment
When to TrafficShift ?
Site Traffic and Disaster Recovery
Traffic stops being
served to offline
Traffic is shifted to online
What is Load Testing ?
3 times a week
Peak hour traffic
LinkedIn’s PoP Architecture
• Using IPVS - Each PoP announces a unicast address and a regional anycast
• APAC, EU and NAMER anycast regions
• Use GeoDNS to steer users to the ‘best’ PoP
• DNS will either provide users with an anycast or unicast address for
• US and EU members is nearly all anycast
• APAC is all unicast
LinkedIn’s PoP DR
• Sometimes need to fail out of PoP’s
• 3rd party provider issues (e.g. transit links
• Infrastructure maintenance
• Withdraw anycast route announcements
• Fail healthchecks on proxy to drain unicast