Disaster Recovery in distant DC; people, hardware, latency - PLNOG 200909

359 views

Published on

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
359
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
8
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Disaster Recovery in distant DC; people, hardware, latency - PLNOG 200909

  1. 1. Disaster Recovery in distant DC; people, hardware, latency.
  2. 2. • What is really the Allegro.pl? • Distaster Recovery - true story • Latency - why 20 ms matters • Data Centers topology agenda
  3. 3. Allegro ... QXL ? Serwisy Aukcyjne Serwisy Ogłoszeniowe Płatności .pl .cz .ro .ua.bg .hu .sk .lt .pl .pl Sklepy Internetowe
  4. 4. Allegro... beginings
  5. 5. o Allegro... today
  6. 6. o Allegro... today about 1000 servers
  7. 7. o Allegro... today about 1000 servers about 300 IT staff
  8. 8. o Allegro... today about 1000 servers about 300 IT staff about 40 in infrastructure
  9. 9. we are growing Source: http://www.aukcjostat.pl/
  10. 10. why bother?
  11. 11. Qvo vadis?
  12. 12. Qvo vadis?
  13. 13. Qvo vadis?
  14. 14. Qvo vadis?
  15. 15. Qvo vadis?
  16. 16. Qvo vadis?
  17. 17. 2 hours by plane... ~800 km Go west...
  18. 18. 2 hours by plane... ~800 km Go west...
  19. 19. WWCKD ?
  20. 20. WWCKD ? Hope is the mother of lost battles No guts, no glory. No pain, no game. No risk, no fun.
  21. 21. WWCKD ? Hope is the mother of lost battles No guts, no glory. No pain, no game. tiaaaa... No risk, no fun.
  22. 22. Building DC3
  23. 23. Building DC3
  24. 24. DC net topology
  25. 25. DC net topology ISP 1 ISP 2 DC2 , Poznań
  26. 26. DC net topology ISP 1 ISP 2 ISP 1 ISP 2 DC2 , Poznań DC3 , Frankfurt
  27. 27. DC net topology ISP 1 ISP 2 ISP 1 ISP 2 DC2 , Poznań DC3 , Frankfurt 2G, EoMPLS
  28. 28. DC net topology ISP 1 ISP 2 ISP 1 ISP 2 DC2 , Poznań DC3 , Frankfurt 2G, EoMPLS 10G, DWDM
  29. 29. Global Traffic Manager DC2 , Poznań DNS req: www.allegro.pl DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl DNS DNS $ host -t ns allegro.pl allegro.pl name server gtmdc3.allegro.pl. allegro.pl name server gtmdc2.allegro.pl.
  30. 30. Global Traffic Manager DC2 , Poznań DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl
  31. 31. Global Traffic Manager DC2 , Poznań DNS req: www.allegro.pl DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl
  32. 32. Global Traffic Manager DC2 , Poznań DNS req: www.allegro.pl DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl DNS request
  33. 33. Global Traffic Manager DC2 , Poznań DNS req: www.allegro.pl DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl DNS request
  34. 34. Global Traffic Manager DC2 , Poznań DNS req: www.allegro.pl DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl DNS request
  35. 35. Global Traffic Manager DC2 , Poznań DNS req: www.allegro.pl DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl DNS request
  36. 36. Global Traffic Manager DC2 , Poznań DNS req: www.allegro.pl DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl DNS request
  37. 37. Global Traffic Manager DC2 , Poznań DNS req: www.allegro.pl DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl DNS request
  38. 38. Global Traffic Manager DC2 , Poznań DNS req: www.allegro.pl DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl DNS request
  39. 39. Global Traffic Manager DC2 , Poznań DNS req: www.allegro.pl DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl DNS response
  40. 40. Global Traffic Manager DC2 , Poznań DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl DNS req: www.allegro.pl HTTP traffic
  41. 41. Global Traffic Manager DC2 , Poznań DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl
  42. 42. Global Traffic Manager DC2 , Poznań DNS req: www.allegro.pl DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl
  43. 43. Global Traffic Manager DC2 , Poznań DNS req: www.allegro.pl DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl
  44. 44. Global Traffic Manager DC2 , Poznań DNS req: www.allegro.pl DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl
  45. 45. Global Traffic Manager DC2 , Poznań DNS req: www.allegro.pl DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl
  46. 46. Global Traffic Manager DC2 , Poznań DNS req: www.allegro.pl DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl
  47. 47. Global Traffic Manager DC2 , Poznań DNS req: www.allegro.pl DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl
  48. 48. Global Traffic Manager DC2 , Poznań DNS req: www.allegro.pl DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl
  49. 49. Global Traffic Manager DC2 , Poznań DNS req: www.allegro.pl DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl
  50. 50. Global Traffic Manager DC2 , Poznań DNS req: www.allegro.pl DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl
  51. 51. Global Traffic Manager DC2 , Poznań DC3 , Frankfurt www.allegro.pl F5 LTM F5 LTM F5 LTM F5 LTM F5 GTM F5 GTM F5 GTM F5 GTM www.allegro.pl DNS req: www.allegro.pl HTTP traffic
  52. 52. Latency vs tennis
  53. 53. SYN SYN/ACK ACK TCP/IP explained FIN ACK ACK DATA DATA ACK Control packets Data packets bandwidth RTT / latency window size = 64 kBytes Bandwidth-delay Product BDP (bytes) = bandwidth (KBytes/sec) x RTT (ms)
  54. 54. 0,0005sec SYN SYN/ACK ACK TCP/IP explained FIN ACK 0,0005sec 0,0005sec ACK DATA DATA ACK 0,0005sec 0,0005sec 0,0005sec 0,0005sec 0,0005sec 0,0005sec Control packets Data packets
  55. 55. 0,0005sec SYN SYN/ACK ACK TCP/IP explained FIN ACK 0,0005sec 0,0005sec ACK DATA DATA ACK 0,0005sec 0,0005sec 0,0005sec 0,0005sec 0,0005sec 0,0005sec Control packets Data packets Scenario #1 • bandwidth = 10Gbit/sec • RTT / latency = 1 ms (1/1000 sec) • window size = 64 kBytes • data to transfer = 640 000 kBytes (1CD) What is the time to deliver CD from point A to B?
  56. 56. 0,0005sec SYN SYN/ACK ACK TCP/IP explained FIN ACK 0,0005sec 0,0005sec ACK DATA DATA ACK 0,0005sec 0,0005sec 0,0005sec 0,0005sec 0,0005sec 0,0005sec Control packets Data packets Scenario #1 • bandwidth = 10Gbit/sec • RTT / latency = 1 ms (1/1000 sec) • window size = 64 kBytes • data to transfer = 640 000 kBytes (1CD) What is the time to deliver CD from point A to B? 1 sec !!! (640 000 kBytes) / (64 kBytes/ 1ms) = 1000 ms
  57. 57. 0,5sek SYN SYN/ACK ACK TCP/IP explained FIN ACK 0,5sek 0,5sek ACK DATA DATA ACK 0,5sek 0,5sek 0,5sek 0,5sek 0,5sek 0,5sek Control packets Data packets
  58. 58. 0,5sek SYN SYN/ACK ACK TCP/IP explained FIN ACK 0,5sek 0,5sek ACK DATA DATA ACK 0,5sek 0,5sek 0,5sek 0,5sek 0,5sek 0,5sek Control packets Data packets Scenario #2 • bandwidth = 10Gbit/sec • RTT / latency = 1 sek • window size = 64 kBytes • data to transfer = 640 000 kBytes (1CD) What is the time to deliver CD from point A to B?
  59. 59. 0,5sek SYN SYN/ACK ACK TCP/IP explained FIN ACK 0,5sek 0,5sek ACK DATA DATA ACK 0,5sek 0,5sek 0,5sek 0,5sek 0,5sek 0,5sek Control packets Data packets Scenario #2 • bandwidth = 10Gbit/sec • RTT / latency = 1 sek • window size = 64 kBytes • data to transfer = 640 000 kBytes (1CD) What is the time to deliver CD from point A to B? 1000 sec !!! (640 000 kBytes) / (64 kBytes/1000ms) = 1000 sec
  60. 60. 800km distance...
  61. 61. 800km distance... storage data base application / transactions AIX Oracle MySQLbackend daemons file transfers server deployment Linux
  62. 62. 800km distance... just 20ms...
  63. 63. co-operation
  64. 64. co-operation developers
  65. 65. co-operation developers infrastructure
  66. 66. co-operation developers infrastructure
  67. 67. co-operation developers infrastructure
  68. 68. Recipe for success Disaster recovery ingredients
  69. 69. Recipe for success
  70. 70. Recipe for success
  71. 71. Recipe for success
  72. 72. Recipe for success
  73. 73. Thank You, Q&A

×