Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Linux con berlin-2016-presentation-deitchera

498 views

Published on

LinuxCon/ContainerCon Berlin 2016 - Networking (Containers) in Ultra-Low-Latency Environments

Published in: Technology

Linux con berlin-2016-presentation-deitchera

  1. 1. Networking (Containers) in Ultra- Low-Latency Environments Avi Deitcher avi@atomicinc.com
  2. 2. Who Am I? Avi Deitcher avi@atomicinc.com
  3. 3. Who Am I? Avi Deitcher avi@atomicinc.com
  4. 4. Who Am I? Avi Deitcher avi@atomicinc.com (not 24601)
  5. 5. Who Am I? •  Life in tech business: –  10 yrs large-scale mission-criKcal IT –  10+ yrs consulKng & training –  Some startups on the way •  Avid (if not very good) ice hockey player •  Long-Kme lover of great engineering…. when used to make a real difference •  Atomic Inc: –  Generalist pracKKoner •  Network just one element –  Product : engineering : operaKons Avi Deitcher avi@atomicinc.com (not 24601)
  6. 6. A LiVle History Avi Deitcher avi@atomicinc.com
  7. 7. A LiVle History Summer 2015 •  Fintech X: “Containerize us!” –  Hint: It is harder than you think… and worth it –  Culture/process > technology •  QuesKon: Networking? •  Answer: ScienKfic method Avi Deitcher avi@atomicinc.com
  8. 8. A LiVle History Summer 2015 •  Fintech X: “Containerize us!” –  Hint: It is harder than you think… and worth it –  Culture/process > technology •  QuesKon: Networking? •  Answer: ScienKfic method Fall 2016 •  Good pracKce demands: 1.  Redo tests with new opKons and versions 2.  Make tests available 3.  Explain it all well Avi Deitcher avi@atomicinc.com
  9. 9. What Is “Ultra-Low” Latency? Avi Deitcher avi@atomicinc.com
  10. 10. What Is “Ultra-Low” Latency? 1.  hVp://home.blarg.net/%7Eglinden/StanfordDataMining.2006-11-29.ppt Avi Deitcher avi@atomicinc.com “every 100ms of delay costs 1% of sales”[1]
  11. 11. What Is “Ultra-Low” Latency? “extra 0.5s in search page generaKon Kme dropped traffic by 20%”[2] 1.  hVp://home.blarg.net/%7Eglinden/StanfordDataMining.2006-11-29.ppt 2.  hVp://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html Avi Deitcher avi@atomicinc.com “every 100ms of delay costs 1% of sales”[1]
  12. 12. What Is “Ultra-Low” Latency? “extra 0.5s in search page generaKon Kme dropped traffic by 20%”[2] 1.  hVp://home.blarg.net/%7Eglinden/StanfordDataMining.2006-11-29.ppt 2.  hVp://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html Avi Deitcher avi@atomicinc.com Not. Even. Close. “every 100ms of delay costs 1% of sales”[1]
  13. 13. Ultra-Low Latency 38 messages in 7 milliseconds 1 message (avg) every 184 𝓊-sec! Avi Deitcher avi@atomicinc.com
  14. 14. Two Types of Networking… Direct Avi Deitcher avi@atomicinc.com
  15. 15. Two Types of Networking… Direct Fabric+Overlay Avi Deitcher avi@atomicinc.com
  16. 16. … maybe four Workload Awareness Avi Deitcher avi@atomicinc.com
  17. 17. … maybe four Workload Awareness Fabric Awareness Avi Deitcher avi@atomicinc.com
  18. 18. Networking OpKons Direct Metal macvlan Bridge/vSwitch (no NAT) net=host SR-IOV Overlay Flannel Weave Docker Overlay Calico (IPIP) Workload Awareness Docker bridge (NAT) Fabric Awareness Calico (NaKve) Avi Deitcher avi@atomicinc.com
  19. 19. Our Tests What We Tested •  netperf ⇒ netserver •  UDP & TCP round-robin •  Sizes: 300, 500, 1024, 2048 •  No orchestraKon = complete control •  50000 iteraKons –  Law of large numbers •  Latency (Avg, %iles), CPU •  DifferenQals, not absolutes How We Tested •  .net –  Because it had to be metal –  Wicked smart team •  Complete test run –  Network changes –  Hardware variaKons, errors hVps://github.com/deitch/network-tests Avi Deitcher avi@atomicinc.com
  20. 20. Local vs. Remote Avi Deitcher avi@atomicinc.com
  21. 21. Avi Deitcher avi@atomicinc.com
  22. 22. Avi Deitcher avi@atomicinc.com
  23. 23. Avi Deitcher avi@atomicinc.com
  24. 24. Local Networking Summary •  SR-IOV horrible latency but great CPU –  Hold that thought… •  net=host on par with metal •  macvlan closest virtualized to metal •  Rest in same range: –  Latency: 5-10 𝓊-sec overhead –  CPU: negligible difference •  Calico (IPIP & naKve) & Docker overlay slightly more performant (margin of error?) •  Watch out for very large TCP packets Avi Deitcher avi@atomicinc.com
  25. 25. Avi Deitcher avi@atomicinc.com
  26. 26. Avi Deitcher avi@atomicinc.com
  27. 27. Avi Deitcher avi@atomicinc.com
  28. 28. Remote Networking Summary •  Weave (sleeve) adds latency and CPU – Reason for “fast datapath” •  Again, macvlan best virtualized •  All the rest: – Latency: within 50 𝓊-sec of each other, except SR- IOV with very large TCP packets – CPU: similar, but keep an eye on Flannel (UDP) Avi Deitcher avi@atomicinc.com
  29. 29. About that SR-IOV Type 1: Intel I350 1Gbps Type 3: Mellanox MT27500 ConnectX-3 10Gbps Avi Deitcher avi@atomicinc.com
  30. 30. SR-IOV SR-IOV does not automaQcally mean beWer •  Switch in network card •  Trades host CPU for card processor •  Quality varies drama3cally –  Even Mellanox far worse locally •  My 2€: SR-IOV falls further behind due to: –  Speed of iteraKon –  Open-source –  Sosware + CPU Avi Deitcher avi@atomicinc.com
  31. 31. What else could we do? Ø  Other hardware types Ø  Other network fabrics Ø  Other network overlay versions (we have the data…) Ø  Docker macvlan network driver Ø  ipvlan Ø  IPv6 Ø  Kernel and network stack tuning Ø  Distant networks Ø  Other traffic paVerns (mulKcast vs unicast) Ø  Other host-to-host encrypKon Ø  Other kernel versions Ø  Other OSes (Illumos-based?) Ø  A whole lot more… Avi Deitcher avi@atomicinc.com
  32. 32. Headaches (and Thanks) •  Headaches –  Weave SYN-(nothing) –  etcd is “touchy” –  Packet L3 network is powerful but… unique •  Macvlan, weave, flannel: all required pings for mac •  Sexng up bridge w/o NAT, Calico, macvlan was “different” –  SR-IOV is complicated and flaky, especially Mellanox –  netperf with UDP packets can get stuck (Calico-ipip) –  And a whole lot more (ask me offline) •  And thanks: –  Bryan Boreham, Adam Harrison at weave.works –  Zac Smith, Adam, Aaron, Andy, Lucas, everyone at Packet Avi Deitcher avi@atomicinc.com
  33. 33. Conclusions •  SR-IOV: most of the Kme, just not worth it •  Performance: –  Metal (+ net=host): always performs best –  Direct network++: macvlan is your friend –  Others: Roughly similar, careful of Weave (sleeve) •  What’s your use case? –  ULL: Metal/net=host > macvlan > calico > overlay –  Everything else: Focus on your architecture and skills Pick intelligently: easier, not simple Avi Deitcher avi@atomicinc.com
  34. 34. Conclusions •  SR-IOV: most of the Kme, just not worth it •  Performance: –  Metal (+ net=host): always performs best –  Direct network++: macvlan is your friend –  Others: Roughly similar, careful of Weave (sleeve) •  What’s your use case? –  ULL: Metal/net=host > macvlan > calico > overlay –  Everything else: Focus on your architecture and skills Pick intelligently: easier, not simple Avi Deitcher avi@atomicinc.com
  35. 35. QuesKons and help: @avideitcher avi@atomicinc.com

×