Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
How	
  LinkedIn	
  used	
  TCP	
  Anycast	
  to	
  make	
  
the	
  site	
  faster	
  
Ritesh	
  Maheshwari	
  	
  	
  	
  ...
Anycast	
  
•  Anycast	
  provides	
  a	
  distributed	
  service	
  via	
  rou8ng.	
  
•  It	
  is	
  not	
  really	
  di...
SF	
  
CHI	
  
NYC	
  
Bob	
  
www.linkedin.com	
  
2001:db8::1/56	
  
www.linkedin.com	
  
2001:db8::1/56	
  
www.linkedi...
Anycast	
  with	
  ECMP	
  
•  Not	
  a	
  real	
  issue	
  in	
  today’s	
  internet	
  
•  Consistent	
  flow	
  rou8ng	
...
Anycast	
  Complica8ons	
  
•  Broken	
  MTU	
  Challenges	
  
•  ICMP	
  message	
  may	
  not	
  reach	
  the	
  intende...
 
But!	
  
How	
  to	
  measure	
  Anycast	
  effec8veness?	
  
What	
  is	
  RUM?	
  
	
  
JavaScript	
  (Client-­‐code)	
  to	
  measure	
  
performance	
  
	
   •  DNS	
  Time	
  
•  ...
What	
  are	
  PoPs?	
  
	
  
Point	
  of	
  Presence	
  /	
  PoP	
  
•  Small-­‐scale	
  data	
  centers	
  
•  Proxy	
  ...
Without	
  PoPs	
  
Browser	
   Data	
  Center	
  
connec8on	
  8me	
   250ms	
  
Without	
  PoPs	
  
Browser	
   Data	
  Center	
  
connec8on	
  8me	
  
server	
  
compute	
  
8me	
  
250ms	
  
500ms	
  
Without	
  PoPs	
  
Browser	
   Data	
  Center	
  
connec8on	
  8me	
  
3-­‐5	
  round	
  trips	
  
first	
  	
  
byte	
  
...
With	
  PoPs	
  
Browser	
   Data	
  Center	
  PoP	
  
100ms	
  
250ms	
  
With	
  PoPs	
  
Browser	
   Data	
  Center	
  PoP	
  
100ms	
  connec8on	
  8me	
  
Old	
  TCP	
  Connec8on	
  
With	
  PoPs	
  
Browser	
   Data	
  Center	
  PoP	
  
100ms	
  connec8on	
  8me	
  
one	
  round	
  trip	
  
first	
  	
  ...
With	
  PoPs	
  
Browser	
   Data	
  Center	
  PoP	
  
100ms	
  connec8on	
  8me	
  
one	
  round	
  trip	
  
5	
  RTTs	
 ...
How	
  are	
  users	
  assigned	
  to	
  PoPs?	
  
Through	
  DNS:	
  	
  
	
  IP	
  handed	
  based	
  on	
  user’s	
  re...
Should	
  India	
  connect	
  to	
  Singapore	
  or	
  
Dublin?	
  
	
  
How	
  to	
  assure	
  op,mal	
  PoPs	
  assignme...
RUM	
  beacons	
  
Fetch	
  a	
  8ny	
  object	
  from	
  each	
  candidate	
  PoP	
  
	
  
For each pop_name,
1.  Start t...
We	
  can	
  assign	
  countries	
  to	
  new	
  PoPs!	
  
Country	
   PoP	
  
Median	
  Beacon	
  
Time(ms)	
  
China	
  ...
We	
  can	
  audit	
  current	
  assignment!	
  
Country	
   Is	
  PoP	
  op8mal?	
   Current	
  PoP	
   Op8mal	
  PoP	
  ...
0%	
  
5%	
  
10%	
  
15%	
  
20%	
  
25%	
  
30%	
  
India	
   Pakistan	
   Singapore	
   Russia	
   Brazil	
  
Percentag...
Plot	
  Twist:	
  	
  
Assignment	
  far	
  from	
  op8mal	
  
•  About	
  31%	
  of	
  US	
  traffic	
  gets	
  assigned	
 ...
DNS	
  PoP	
  assignment	
  is	
  subop8mal	
  
•  Assignment	
  based	
  on	
  Resolver	
  IP,	
  not	
  Client	
  IP	
  ...
DNS	
  PoP	
  assignment	
  is	
  subop8mal	
  
•  Assignment	
  based	
  on	
  Resolver	
  IP,	
  not	
  Client	
  IP	
  ...
Story	
  so	
  far	
  
1.  We	
  built	
  PoPs	
  
2.  …used	
  RUM	
  to	
  assign	
  users	
  to	
  Op8mal	
  PoPs	
  
3...
Accurate	
  PoP	
  assignment	
  Problem	
  
•  Bug	
  our	
  DNS	
  providers	
  (31%	
  -­‐>	
  27%)	
  
•  Run	
  our	
...
Anycast	
  –	
  One	
  IP,	
  Mul8ple	
  Servers	
  
PoP	
  A	
  
PoP	
  B	
  
PoP	
  C	
  
Bob	
  
1.1.1.1	
  
1.1.1.1	
 ...
How	
  does	
  Anycast	
  compare	
  to	
  DNS?	
  
	
  
Will	
  anycast	
  send	
  more	
  users	
  to	
  op,mal	
  PoP?	...
RUM	
  to	
  rescue	
  
	
  
For	
  each	
  PoP:	
  
1.  Announce	
  same	
  anycast	
  IP	
  (108.174.13.10)	
  
2.  Confi...
RUM	
  to	
  rescue	
  
For	
  each	
  page	
  view:	
  
1.  RUM	
  downloads	
  a	
  8ny	
  object	
  :	
  	
   	
   	
  ...
Results	
  J	
  
Region	
  or	
  	
  
Country	
  
DNS	
  %	
  Op8mal	
  
Assignment	
  
Anycast	
  %	
  Op8mal	
  	
  
As...
Results	
  L	
  
Region	
  or	
  	
  
Country	
  
DNS	
  %	
  Op8mal	
  
Assignment	
  
Anycast	
  %	
  Op8mal	
  	
  
As...
Fewer	
  hops	
  !=	
  Lower	
  Latency	
  
•  Carriers	
  prefer	
  to	
  haul	
  packets	
  within	
  
their	
  own	
  n...
Maybe	
  DNS	
  wasn’t	
  so	
  bad	
  
	
  
Con8nent-­‐level	
  assignments	
  	
  
	
  
City	
  /	
  State	
  level	
  a...
“Regional”	
  Anycast	
  
DNS-­‐based	
  
1	
  anycast	
  IP	
  per	
  con8nent	
  
Ran	
  a	
  RUM	
  experiment,	
  	
  ...
USA	
  Ramp	
  Results	
  
50.00	
  
55.00	
  
60.00	
  
65.00	
  
70.00	
  
75.00	
  
80.00	
  
85.00	
  
90.00	
  
95.00...
Story	
  so	
  far	
  
1.  We	
  built	
  PoPs	
  
2.  …used	
  RUM	
  to	
  assign	
  users	
  to	
  Op8mal	
  PoPs	
  
3...
Story:	
  The	
  End	
  
Learnings	
  
•  Clients	
  are	
  your	
  
measurement	
  agents	
  
•  Trust,	
  but	
  verify	...
©2014 LinkedIn Corporation. All Rights Reserved.©2014 LinkedIn Corporation. All Rights Reserved.
How LinkedIn used TCP Anycast to make the site faster
How LinkedIn used TCP Anycast to make the site faster
How LinkedIn used TCP Anycast to make the site faster
How LinkedIn used TCP Anycast to make the site faster
Upcoming SlideShare
Loading in …5
×

How LinkedIn used TCP Anycast to make the site faster

2,131 views

Published on

TCP anycast effectiveness in public internet and measurement by real user monitoring (RUM) to improve user experience,

Published in: Internet
  • Be the first to comment

How LinkedIn used TCP Anycast to make the site faster

  1. 1. How  LinkedIn  used  TCP  Anycast  to  make   the  site  faster   Ritesh  Maheshwari                              Shawn  Zandi  
  2. 2. Anycast   •  Anycast  provides  a  distributed  service  via  rou8ng.   •  It  is  not  really  different  than  unicast.   •  NLRI  object  with  mul8ple  next-­‐hops.   •  It  simply  works  for  both  TCP  and  UDP  applica8ons.  (use   with  cau8ons!)        
  3. 3. SF   CHI   NYC   Bob   www.linkedin.com   2001:db8::1/56   www.linkedin.com   2001:db8::1/56   www.linkedin.com   2001:db8::1/56  
  4. 4. Anycast  with  ECMP   •  Not  a  real  issue  in  today’s  internet   •  Consistent  flow  rou8ng  is  required  (per  packet  load   balancing  breaks  Anycast)  –  Pre_y  Much  Standard   •  Most  BGP  implementa8ons  do  not  load  balance  across   different  AS-­‐PATHs  even  with  same  size.    
  5. 5. Anycast  Complica8ons   •  Broken  MTU  Challenges   •  ICMP  message  may  not  reach  the  intended  receiver   to  report  MTU  problem.  Adjus8ng  MSS  can  help.   •  RPF  Checks   •  Mul8ple  covering  prefixes  -­‐  Only  one  Service  Address   should  be  covered  by  each  adver8sed  prefix  /24  or  /56   •  Monitoring!          
  6. 6.   But!   How  to  measure  Anycast  effec8veness?  
  7. 7. What  is  RUM?     JavaScript  (Client-­‐code)  to  measure   performance     •  DNS  Time   •  Connec8on  8me   •  First  Byte  Time   •  Download  Time   •  Page  Load  Time  
  8. 8. What  are  PoPs?     Point  of  Presence  /  PoP   •  Small-­‐scale  data  centers   •  Proxy  servers  at  LinkedIn  (ATS)  
  9. 9. Without  PoPs   Browser   Data  Center   connec8on  8me   250ms  
  10. 10. Without  PoPs   Browser   Data  Center   connec8on  8me   server   compute   8me   250ms   500ms  
  11. 11. Without  PoPs   Browser   Data  Center   connec8on  8me   3-­‐5  round  trips   first     byte    8me     +   page   download   8me   5  RTTs  =  5x250ms  =  1250ms   server   compute   8me   250ms   Total  =  2000ms   500ms  
  12. 12. With  PoPs   Browser   Data  Center  PoP   100ms   250ms  
  13. 13. With  PoPs   Browser   Data  Center  PoP   100ms  connec8on  8me   Old  TCP  Connec8on  
  14. 14. With  PoPs   Browser   Data  Center  PoP   100ms  connec8on  8me   one  round  trip   first     byte    8me     +   page   download   8me   Old  TCP  Connec8on   server   compute   8me   500ms  
  15. 15. With  PoPs   Browser   Data  Center  PoP   100ms  connec8on  8me   one  round  trip   5  RTTs  =  5x100ms  =  500ms   Total  =  1100ms  900  ms  gain!   first     byte    8me     +   page   download   8me   Old  TCP  Connec8on   500ms   server   compute   8me  
  16. 16. How  are  users  assigned  to  PoPs?   Through  DNS:      IP  handed  based  on  user’s  resolver  country       #  Spain   $  dig  @109.69.8.51  +short  www.linkedin.com   91.225.248.80     #  California   $  dig  +short  www.linkedin.com   216.52.242.80    
  17. 17. Should  India  connect  to  Singapore  or   Dublin?     How  to  assure  op,mal  PoPs  assignment?      
  18. 18. RUM  beacons   Fetch  a  8ny  object  from  each  candidate  PoP     For each pop_name, 1.  Start timer 2.  Fetch {pop_name}.perf.linkedin.com/pop/admin 3.  Stop timer Send data back to our servers •  Millions  of  agents!   •  Analyze  data  to  find  “op8mal”  PoP  per  country  
  19. 19. We  can  assign  countries  to  new  PoPs!   Country   PoP   Median  Beacon   Time(ms)   China   Hong  Kong   434   China   Dublin   1216   China   Singapore   515   India   Hong  Kong   1368   India   Dublin   1042   India   Singapore   898  
  20. 20. We  can  audit  current  assignment!   Country   Is  PoP  op8mal?   Current  PoP   Op8mal  PoP   India   TRUE   Singapore   Singapore   Pakistan   FALSE   Singapore   Dublin   Spain   TRUE   Dublin   Dublin   Brazil   FALSE   US  West  Coast   US  East  Coast   Netherlands   TRUE   Dublin   Dublin   UAE   FALSE   US  West  Coast   Dublin   Italy   TRUE   Dublin   Dublin   Mexico   TRUE   US  West  Coast   US  West  Coast   Russia   FALSE   US  West  Coast   Dublin  
  21. 21. 0%   5%   10%   15%   20%   25%   30%   India   Pakistan   Singapore   Russia   Brazil   Percentage  Improvement   LinkedIn  Homepage  Download  Time  Improvement   Median  Improvement   90th  Percen8le  Improvement  
  22. 22. Plot  Twist:     Assignment  far  from  op8mal   •  About  31%  of  US  traffic  gets  assigned  to  a   subop8mal  PoP.   – 45%  of  East  Coast   •  About  10%  of  traffic  globally  gets  assigned  to  a   subop8mal  PoP.  
  23. 23. DNS  PoP  assignment  is  subop8mal   •  Assignment  based  on  Resolver  IP,  not  Client  IP   DNS   Resolver   PoP   US   East   PoP   US   West   New  York  California  
  24. 24. DNS  PoP  assignment  is  subop8mal   •  Assignment  based  on  Resolver  IP,  not  Client  IP   •  Bad  IP  to  Geo  databases   – Resolver  really  in  NY,  but  database  says  CA  
  25. 25. Story  so  far   1.  We  built  PoPs   2.  …used  RUM  to  assign  users  to  Op8mal  PoPs   3.  …found  DNS  based  assignment  is  subop8mal  
  26. 26. Accurate  PoP  assignment  Problem   •  Bug  our  DNS  providers  (31%  -­‐>  27%)   •  Run  our  own  DNS     How  about  Anycast?  
  27. 27. Anycast  –  One  IP,  Mul8ple  Servers   PoP  A   PoP  B   PoP  C   Bob   1.1.1.1   1.1.1.1   1.1.1.1   ü Client  IP,  not  Resolver  IP  used!   ü No  Geo-­‐IP  Databases    
  28. 28. How  does  Anycast  compare  to  DNS?     Will  anycast  send  more  users  to  op,mal  PoP?     Ø Lets  test  it!  
  29. 29. RUM  to  rescue     For  each  PoP:   1.  Announce  same  anycast  IP  (108.174.13.10)   2.  Configure  a  domain   ac.perf.linkedin.com  to  point  to   108.174.13.10  
  30. 30. RUM  to  rescue   For  each  page  view:   1.  RUM  downloads  a  8ny  object  :          ac.perf.linkedin.com/pop/admin 2.  Read    X-Li-Pop response  header  to  record  which  PoP  served   the  object   3.  Send  this  back  to  LinkedIn  with  RUM  data   Data:   1.  For  each  user,  the  anycast  PoP   2.  For  each  user,  the  op8mal  PoP  (from  pop  beacons)  
  31. 31. Results  J   Region  or     Country   DNS  %  Op8mal   Assignment   Anycast  %  Op8mal     Assignment   Illinois   70   90   Florida   73   95   Georgia   75   93   Pennsylvania   85   95  
  32. 32. Results  L   Region  or     Country   DNS  %  Op8mal   Assignment   Anycast  %  Op8mal     Assignment   Arizona   60   39   Brazil   88   33   New  York   77   74  
  33. 33. Fewer  hops  !=  Lower  Latency   •  Carriers  prefer  to  haul  packets  within   their  own  network   •  Peering  can  create  inter-­‐con8nental   short  cuts   Z   X   Alice   Y   inter-­‐con8nental  link   1.1.1.1   1.1.1.1   1.1.1.1  
  34. 34. Maybe  DNS  wasn’t  so  bad     Con8nent-­‐level  assignments       City  /  State  level  assignments  
  35. 35. “Regional”  Anycast   DNS-­‐based   1  anycast  IP  per  con8nent   Ran  a  RUM  experiment,     all  was  fine   Z   X   Alice   Y   2.2.2.2   1.1.1.1   1.1.1.1   inter-­‐con8nental  link  
  36. 36. USA  Ramp  Results   50.00   55.00   60.00   65.00   70.00   75.00   80.00   85.00   90.00   95.00   100.00   20141207  20141208  20141209  20141210  20141211  20141212  20141213  20141214  20141215  20141216  20141217   %  Traffic  going  to  Op8mal  PoP   Date   Illinois   Florida   North  Carolina   Indiana   NY   NJ   VA   WV   LA   Ramp  outside  USA     In  progress  
  37. 37. Story  so  far   1.  We  built  PoPs   2.  …used  RUM  to  assign  users  to  Op8mal  PoPs   3.  …found  DNS  based  assignment  is  subop8mal   4.  …evaluated  Anycast  as  a  solu8on  using  RUM   5.  …now  using  Anycast  to  assign  users  to  PoPs   Next  play:   •  Build  more  PoPs!  
  38. 38. Story:  The  End   Learnings   •  Clients  are  your   measurement  agents   •  Trust,  but  verify   •  You  can  have  a  bigger   impact  if  you  collaborate   Next  Play   •  Keep  evalua8ng  Anycast   •  Keep  building  new  PoPs  
  39. 39. ©2014 LinkedIn Corporation. All Rights Reserved.©2014 LinkedIn Corporation. All Rights Reserved.

×