Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Networking	(Containers)	in	Ultra-
Low-Latency	Environments	
Avi	Deitcher	
avi@atomicinc.com
‫אכסניא‬
Avi	Deitcher		avi@atomicinc.com
‫אכסניא‬
Akh-san-ya	?aksnaja? n.	(ancient	Aramaic,	
from	Ancient	Greek	xénos)	1:	Hospitality,	
lodging;	2:	Host.	
	
Avi	De...
‫אכסניא‬
Akh-san-ya	?aksnaja? n.	(ancient	Aramaic,	
from	Ancient	Greek	xénos)	1:	Hospitality,	
lodging;	2:	Host.	
	
		:‫אכ...
Who	Am	I?	
Avi	Deitcher		avi@atomicinc.com
Who	Am	I?	
Avi	Deitcher		avi@atomicinc.com	
(not	24601)
Who	Am	I?	
•  Life	in	tech	business:	
–  10	yrs	financial	services	IT	
–  10+	yrs	consulWng	&	training	
–  Some	startups	on...
A	Lile	History	
Avi	Deitcher		avi@atomicinc.com
A	Lile	History	
Summer	2015	
•  Fintech	X:	“Help	us	
containerize!”	
–  Hint:	It	is	harder	than	you	
think…	and	worth	it	
...
A	Lile	History	
Summer	2015	
•  Fintech	X:	“Help	us	
containerize!”	
–  Hint:	It	is	harder	than	you	
think…	and	worth	it	
...
What	Is	“Ultra-Low”	Latency?	
Avi	Deitcher		avi@atomicinc.com
What	Is	“Ultra-Low”	Latency?	
1.  hp://home.blarg.net/%7Eglinden/StanfordDataMining.2006-11-29.ppt	
Avi	Deitcher		avi@atom...
What	Is	“Ultra-Low”	Latency?	
“extra	0.5s	in	search	page	generaWon	
Wme	dropped	traffic	by	20%”[2]	
	
1.  hp://home.blarg.ne...
What	Is	“Ultra-Low”	Latency?	
“extra	0.5s	in	search	page	generaWon	
Wme	dropped	traffic	by	20%”[2]	
	
1.  hp://home.blarg.ne...
Ultra-Low	Latency	
38	messages	in	7	milliseconds	
	
1	message	(avg)	every	184	𝓊-sec!	
Avi	Deitcher		avi@atomicinc.com
Networking	Workloads	
•  Networked	Workloads:	
	“things	that	do	work	and	must	talk”	
•  Same	principles	for	all	workloads:...
Two	Types	of	Networking…	
Direct	
Avi	Deitcher		avi@atomicinc.com
Two	Types	of	Networking…	
Direct	 Fabric+Overlay	
Avi	Deitcher		avi@atomicinc.com
…	maybe	four	
Workload	Awareness		
Avi	Deitcher		avi@atomicinc.com
…	maybe	four	
Workload	Awareness		 Fabric	Awareness	
Avi	Deitcher		avi@atomicinc.com
Networking	OpWons	
Direct	
Metal	
macvlan	
Bridge/vSwitch		
									(no	NAT)	
net=host	
SR-IOV	
Overlay	
Flannel	
Weave	
...
Our	Tests	
What	We	Tested	
•  netperf	⇒	netserver	
•  UDP	&	TCP	round-robin	
•  Sizes:	300,	500,	1024,	2048	
•  No	orchest...
Local	vs.	Remote	
Avi	Deitcher		avi@atomicinc.com
Avi	Deitcher		avi@atomicinc.com
Avi	Deitcher		avi@atomicinc.com
Avi	Deitcher		avi@atomicinc.com
Local	Networking	Summary	
•  SR-IOV	horrible	latency	but	great	CPU	
–  Hold	that	thought…	
•  net=host	on	par	with	metal	
...
Avi	Deitcher		avi@atomicinc.com
Avi	Deitcher		avi@atomicinc.com
Avi	Deitcher		avi@atomicinc.com
Remote	Networking	Summary	
•  Weave	(sleeve)	adds	latency	and	CPU	
– Reason	for	“fast	datapath”	
•  Again,	macvlan	best	vi...
About	that	SR-IOV	
Type	1:	Intel	I350	1Gbps	
Type	3:	Mellanox	MT27500	ConnectX-3	10Gbps		
Avi	Deitcher		avi@atomicinc.com
SR-IOV	
SR-IOV	does	not	automaRcally	mean	beXer	
•  Switch	in	network	card	
•  Trades	host	CPU	for	card	processor	
•  Qual...
Headaches	(and	Thanks)	
•  Headaches	
–  Weave	SYN-(nothing)	
–  etcd	is	“touchy”	
–  Packet	L3	network	is	powerful	but…	u...
What	else	could	we	do?	
Ø Other	hardware	types	
Ø Other	network	fabrics	
Ø Docker	macvlan	network	driver	(experimental)	
Ø...
Conclusions	
•  SR-IOV:	most	of	the	Wme,	just	not	worth	it	
•  Performance:	
–  Metal	(+	net=host):	always	performs	best	
...
Conclusions	
•  SR-IOV:	most	of	the	Wme,	just	not	worth	it	
•  Performance:	
–  Metal	(+	net=host):	always	performs	best	
...
QuesWons	and	help:	
@avideitcher							avi@atomicinc.com
Upcoming SlideShare
Loading in …5
×

LinuxCon/ContainerCon Japan 2016 "Networking Containers in Ultra-Low Latency Environments"

363 views

Published on

Presentation by Avi Deitcher at LinuxCon/ContainerCon Japan 2016. "Networking Containers in Ultra-Low Latency Environments".

Published in: Technology
  • Be the first to comment

  • Be the first to like this

LinuxCon/ContainerCon Japan 2016 "Networking Containers in Ultra-Low Latency Environments"

  1. 1. Networking (Containers) in Ultra- Low-Latency Environments Avi Deitcher avi@atomicinc.com
  2. 2. ‫אכסניא‬ Avi Deitcher avi@atomicinc.com
  3. 3. ‫אכסניא‬ Akh-san-ya ?aksnaja? n. (ancient Aramaic, from Ancient Greek xénos) 1: Hospitality, lodging; 2: Host. Avi Deitcher avi@atomicinc.com
  4. 4. ‫אכסניא‬ Akh-san-ya ?aksnaja? n. (ancient Aramaic, from Ancient Greek xénos) 1: Hospitality, lodging; 2: Host. :‫אכסניא‬ ‫בכבוד‬ ‫פותחים‬ Ancient Jewish custom to begin public speaking by honouring or thanking the hosts. Avi Deitcher avi@atomicinc.com
  5. 5. Who Am I? Avi Deitcher avi@atomicinc.com
  6. 6. Who Am I? Avi Deitcher avi@atomicinc.com (not 24601)
  7. 7. Who Am I? •  Life in tech business: –  10 yrs financial services IT –  10+ yrs consulWng & training –  Some startups on the way •  Avid (if not very good) ice hockey player •  Long-Wme lover of great engineering…. when used to make a real difference •  Atomic Inc: –  ConsulWng –  Training Avi Deitcher avi@atomicinc.com (not 24601)
  8. 8. A Lile History Avi Deitcher avi@atomicinc.com
  9. 9. A Lile History Summer 2015 •  Fintech X: “Help us containerize!” –  Hint: It is harder than you think… and worth it –  Culture/process > technology •  QuesWon: Networking? •  Answer: ScienWfic method Avi Deitcher avi@atomicinc.com
  10. 10. A Lile History Summer 2015 •  Fintech X: “Help us containerize!” –  Hint: It is harder than you think… and worth it –  Culture/process > technology •  QuesWon: Networking? •  Answer: ScienWfic method Summer 2016 •  Good pracWce demands: 1.  Redo tests with new opWons and versions 2.  Make tests available 3.  Explain it all well Avi Deitcher avi@atomicinc.com
  11. 11. What Is “Ultra-Low” Latency? Avi Deitcher avi@atomicinc.com
  12. 12. What Is “Ultra-Low” Latency? 1.  hp://home.blarg.net/%7Eglinden/StanfordDataMining.2006-11-29.ppt Avi Deitcher avi@atomicinc.com “every 100ms of delay costs 1% of sales”[1]
  13. 13. What Is “Ultra-Low” Latency? “extra 0.5s in search page generaWon Wme dropped traffic by 20%”[2] 1.  hp://home.blarg.net/%7Eglinden/StanfordDataMining.2006-11-29.ppt 2.  hp://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html Avi Deitcher avi@atomicinc.com “every 100ms of delay costs 1% of sales”[1]
  14. 14. What Is “Ultra-Low” Latency? “extra 0.5s in search page generaWon Wme dropped traffic by 20%”[2] 1.  hp://home.blarg.net/%7Eglinden/StanfordDataMining.2006-11-29.ppt 2.  hp://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html Avi Deitcher avi@atomicinc.com Not. Even. Close. “every 100ms of delay costs 1% of sales”[1]
  15. 15. Ultra-Low Latency 38 messages in 7 milliseconds 1 message (avg) every 184 𝓊-sec! Avi Deitcher avi@atomicinc.com
  16. 16. Networking Workloads •  Networked Workloads: “things that do work and must talk” •  Same principles for all workloads: – VMs – Cloud – Serverless – Containers Avi Deitcher avi@atomicinc.com
  17. 17. Two Types of Networking… Direct Avi Deitcher avi@atomicinc.com
  18. 18. Two Types of Networking… Direct Fabric+Overlay Avi Deitcher avi@atomicinc.com
  19. 19. … maybe four Workload Awareness Avi Deitcher avi@atomicinc.com
  20. 20. … maybe four Workload Awareness Fabric Awareness Avi Deitcher avi@atomicinc.com
  21. 21. Networking OpWons Direct Metal macvlan Bridge/vSwitch (no NAT) net=host SR-IOV Overlay Flannel Weave Docker Overlay Calico (IPIP) Workload Awareness Docker bridge (NAT) Fabric Awareness Calico (NaWve) Avi Deitcher avi@atomicinc.com
  22. 22. Our Tests What We Tested •  netperf ⇒ netserver •  UDP & TCP round-robin •  Sizes: 300, 500, 1024, 2048 •  No orchestraWon = complete control •  50000 iteraWons –  Law of large numbers •  Latency (Avg, %iles), CPU •  DifferenRals, not absolutes How We Tested •  .net –  Because it had to be metal –  Wicked smart team •  Complete test run –  Network changes –  Hardware variaWons, errors hps://github.com/deitch/network-tests Avi Deitcher avi@atomicinc.com
  23. 23. Local vs. Remote Avi Deitcher avi@atomicinc.com
  24. 24. Avi Deitcher avi@atomicinc.com
  25. 25. Avi Deitcher avi@atomicinc.com
  26. 26. Avi Deitcher avi@atomicinc.com
  27. 27. Local Networking Summary •  SR-IOV horrible latency but great CPU –  Hold that thought… •  net=host on par with metal •  macvlan closest virtualized to metal •  Rest in same range: –  Latency: 5-10 𝓊-sec overhead –  CPU: negligible difference •  Calico (IPIP & naWve) & Docker overlay slightly more performant •  Watch out for very large TCP packets Avi Deitcher avi@atomicinc.com
  28. 28. Avi Deitcher avi@atomicinc.com
  29. 29. Avi Deitcher avi@atomicinc.com
  30. 30. Avi Deitcher avi@atomicinc.com
  31. 31. Remote Networking Summary •  Weave (sleeve) adds latency and CPU – Reason for “fast datapath” •  Again, macvlan best virtualized •  All the rest: – Latency: within 50 𝓊-sec of each other, except SR- IOV with very large TCP packets – CPU: similar, but keep an eye on Flannel (UDP) Avi Deitcher avi@atomicinc.com
  32. 32. About that SR-IOV Type 1: Intel I350 1Gbps Type 3: Mellanox MT27500 ConnectX-3 10Gbps Avi Deitcher avi@atomicinc.com
  33. 33. SR-IOV SR-IOV does not automaRcally mean beXer •  Switch in network card •  Trades host CPU for card processor •  Quality varies drama5cally –  Even Mellanox far worse locally •  My 2¥: SR-IOV falls further behind due to: –  Speed of iteraWon –  Open-source –  Sowware + CPU Avi Deitcher avi@atomicinc.com
  34. 34. Headaches (and Thanks) •  Headaches –  Weave SYN-(nothing) –  etcd is “touchy” –  Packet L3 network is powerful but… unique •  Macvlan, weave, flannel: all required pings for mac •  Se{ng up bridge w/o NAT, Calico, macvlan was “different” –  SR-IOV is complicated and flaky, especially Mellanox –  netperf with UDP packets can get stuck (Calico-ipip) –  And a whole lot more (ask me offline) •  And thanks: –  Bryan Boreham, Adam Harrison at weave.works –  Zac Smith, Adam, Aaron, Andy, Lucas, everyone at Packet Avi Deitcher avi@atomicinc.com
  35. 35. What else could we do? Ø Other hardware types Ø Other network fabrics Ø Docker macvlan network driver (experimental) Ø Ipvlan Ø Other packet sizes Ø Kernel and network stack tuning Ø Distant (and VPN) networks Ø Other traffic paerns Ø Other host-to-host encrypWon Ø A whole lot more… Avi Deitcher avi@atomicinc.com
  36. 36. Conclusions •  SR-IOV: most of the Wme, just not worth it •  Performance: –  Metal (+ net=host): always performs best –  Direct network++: macvlan is your friend –  Others: Roughly similar, careful of Weave (sleeve) •  What’s your use case? –  ULL: Metal/net=host > macvlan > calico > overlay –  Everything else: Focus on your architecture and skills Pick intelligently: easier, not simple Avi Deitcher avi@atomicinc.com
  37. 37. Conclusions •  SR-IOV: most of the Wme, just not worth it •  Performance: –  Metal (+ net=host): always performs best –  Direct network++: macvlan is your friend –  Others: Roughly similar, careful of Weave (sleeve) •  What’s your use case? –  ULL: Metal/net=host > macvlan > calico > overlay –  Everything else: Focus on your architecture and skills Pick intelligently: easier, not simple Avi Deitcher avi@atomicinc.com
  38. 38. QuesWons and help: @avideitcher avi@atomicinc.com

×