Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Networking!
Carl Perry
caperry@edolnx.net
Twitter/Reddit/Steam/Freenode: edolnx
whoami
●
Principal Linux Administrator @ Ncsoft
– We make videogames (Guild Wars, Blade & Soul)
●
Founder of Sphere Cube L...
Three Basic Types of Networking
●
Inter-site:
– “Transit”, MPLS, ISP Links
●
Intra-site:
– Ethernet, Fiber Channel, Infini...
Let's talk about layers
Let's talk about layers
Let's talk about layers
Let's talk about layers
OSI Model
Layer Protocol
Data Unit
Function Examples
Host
7 Application
Data
High-level APIs, incl...
Really Esoteric
●
Appletalk
●
Token Ring (unless you work at IBM)
●
FDDI
●
BRI/PRI
●
Frame Relay
●
SONET
Fiber Channel
●
Mostly used for storage
●
Some other applications
●
Topologies: Point 2 Point, Switched, Arbitrated
Loop
●...
Infiniband
●
Primarily developed for HPC
●
2.5, 5, 10, 14.0625, 25, and soon 50 Gbps
native data rate (much more when agg)...
Hail the king
Ethernet
Oh wait
“...commercially introduced in 1980 and first
standardized in 1983 as IEEE 802.3...”
Got room for one more?
802.1q 802.11 802.1x 802.1AB 802.1AC 802.3a 802.3i 802.3z 802.3ab 802.1AX
802.3ae 802.3af 802.3ak 8...
THE CONNECTION IS MADE
THE CONNECTION IS MADE
THE CONNECTION IS MADE
But where do the frames go?
Duh
Bow before your new god
The worst kept secret
●
Nexus 3000, 5000, 7000, 9000 (pizza boxes)
●
Juniper QFX, OFX
●
Arista 7000
●
Every whitebox:
– Ed...
Trident(+)
●
Up to 56 (64) ports
– 48x10GbE, 4x40GbE
●
Cut through
●
960 Mpps
●
560 (640) Gbps
Trident 2
●
Up to 128 Ports
– 32x40GbE
– 48x10GbE, 6x40GbE
●
Cut through
●
1Bpps
●
1.44Tbs
Where do we go from here?
SFP+25
&
QSFP+28
100GbE?!?!?!
Tomahawk
●
Up to 128 Ports
– 32x100GbE
●
Cut through or Store
& Forward
●
3.2Tbps
●
SFP+25, QSFP+28
1GbE
●
Various manufacturers of ASIC parts
●
Various features options
●
All the way from super cheap (D-Link,
Netgear) to ...
TP vs TwinAx vs Fiber
●
SFP+ Modules, $20/ea (MM-OM4) & 2-5w
– SM is more depending on length
●
TP (CAT6a) adds $20-40 & 7...
Exampe Deployment
Closeup
Lessons Learned
●
Use active TwinAx or MPO fiber for QSFP+
spine-spine connections
●
Cable management is mandatory
●
LLDP ...
Virtual Networking
●
Frame Forwarding
– VMware vSwitch (Stand alone & distributed)
– Linux Bridge
– MAC VTAP
●
OpenFlow
– ...
Routers & Firewalls
●
If you are rich, buy one
●
Otherwise, build one:
– Keep an eye on PCI Express bandwidth
– Moving bet...
NICs
About dem NICs
●
Intel NICs are great at 1GbE
●
Intel and Broadcom are descent at 10GbE
●
Broadcom sucks at 1GbE
●
Realtek...
Upcoming SlideShare
Loading in …5
×

Bare Metal Club ATX: Networking Discussion

159 views

Published on

Here is a presentation I gave on the 24th of May, 2016 for the Bare Metal Club of Austin on Networking

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Bare Metal Club ATX: Networking Discussion

  1. 1. Networking! Carl Perry caperry@edolnx.net Twitter/Reddit/Steam/Freenode: edolnx
  2. 2. whoami ● Principal Linux Administrator @ Ncsoft – We make videogames (Guild Wars, Blade & Soul) ● Founder of Sphere Cube LLC – Privacy and Security focused Infastructure hosting on esoteric platforms (ARM now, RISC-V later) ● Previously: Chef Software, BlueBox, RedHat, eNovance, Midokura, DreamHost, Rackspace ... ● I work a lot with networking stuff
  3. 3. Three Basic Types of Networking ● Inter-site: – “Transit”, MPLS, ISP Links ● Intra-site: – Ethernet, Fiber Channel, Infiniband ● Virtual: – vswitch, tunneling, pass-through
  4. 4. Let's talk about layers
  5. 5. Let's talk about layers
  6. 6. Let's talk about layers
  7. 7. Let's talk about layers OSI Model Layer Protocol Data Unit Function Examples Host 7 Application Data High-level APIs, including resource sharing, remote file access, directory services and virtual terminals DotNetFtpLibrary, SMTP web API, SSH.NET, SnmpSharpNet, HTML Class, HTTP API server 6 Presentation Translation of data between a networking service and an application; including character encoding, data compression and encryption/decryption CSS, GIF, HTML, XML, JSON, S/MIME 5 Session Managing communication sessions, i.e. continuous exchange of information in the form of multiple back- and-forth transmissions between two nodes RPC, SCP, NFS, PAP, TLS, FTP, HTTP, HTTPS, SMTP, SSH 4 Transport Segment (TCP) / Datagram (UDP) Reliable transmission of data segments between points on a network, including segmentation, acknowledgement and multiplexing NBF, TCP, UDP Media 3 Network Packet Structuring and managing a multi-node network, including addressing, routing and traffic control AppleTalk, ICMP, IPsec, IPv4, IPv6 2 Data Link Frame Reliable transmission of data frames between two nodes connected by a physical layer IEEE 802.2, L2TP, LLDP, MAC, PPP, ATM, MPLS 1 Physical Bit Transmission and reception of raw bit streams over a physical medium DOCSIS, DSL, Ethernet physical layer, ISDN, RS-232 https://en.wikipedia.org/wiki/OSI_model
  8. 8. Really Esoteric ● Appletalk ● Token Ring (unless you work at IBM) ● FDDI ● BRI/PRI ● Frame Relay ● SONET
  9. 9. Fiber Channel ● Mostly used for storage ● Some other applications ● Topologies: Point 2 Point, Switched, Arbitrated Loop ● Available in 1, 2, 4, 8, 16, and 32 Gbps native data rates ● Loosing Ground to iSCSI (SCSI over IP)
  10. 10. Infiniband ● Primarily developed for HPC ● 2.5, 5, 10, 14.0625, 25, and soon 50 Gbps native data rate (much more when agg) ● Switched fabric topology ● Still used in a lot of storage ● Has lots of nifty features like RDMA
  11. 11. Hail the king Ethernet
  12. 12. Oh wait “...commercially introduced in 1980 and first standardized in 1983 as IEEE 802.3...”
  13. 13. Got room for one more? 802.1q 802.11 802.1x 802.1AB 802.1AC 802.3a 802.3i 802.3z 802.3ab 802.1AX 802.3ae 802.3af 802.3ak 802.3an 802.3at 802.3ba 802.3bm 802.3bq 802.3by
  14. 14. THE CONNECTION IS MADE
  15. 15. THE CONNECTION IS MADE
  16. 16. THE CONNECTION IS MADE
  17. 17. But where do the frames go?
  18. 18. Duh
  19. 19. Bow before your new god
  20. 20. The worst kept secret ● Nexus 3000, 5000, 7000, 9000 (pizza boxes) ● Juniper QFX, OFX ● Arista 7000 ● Every whitebox: – EdgeCore (they make Dell) – Quanta – Supermicro – PenguinComputing – Agema – HPE (some H3C)
  21. 21. Trident(+) ● Up to 56 (64) ports – 48x10GbE, 4x40GbE ● Cut through ● 960 Mpps ● 560 (640) Gbps
  22. 22. Trident 2 ● Up to 128 Ports – 32x40GbE – 48x10GbE, 6x40GbE ● Cut through ● 1Bpps ● 1.44Tbs
  23. 23. Where do we go from here? SFP+25 & QSFP+28
  24. 24. 100GbE?!?!?!
  25. 25. Tomahawk ● Up to 128 Ports – 32x100GbE ● Cut through or Store & Forward ● 3.2Tbps ● SFP+25, QSFP+28
  26. 26. 1GbE ● Various manufacturers of ASIC parts ● Various features options ● All the way from super cheap (D-Link, Netgear) to super awesome (Broadcom Hurricane 2 based) ● Find what matches your price/req
  27. 27. TP vs TwinAx vs Fiber ● SFP+ Modules, $20/ea (MM-OM4) & 2-5w – SM is more depending on length ● TP (CAT6a) adds $20-40 & 7w per port ● TwinAx is passive, $20/2m @ 2.5w TOTAL
  28. 28. Exampe Deployment
  29. 29. Closeup
  30. 30. Lessons Learned ● Use active TwinAx or MPO fiber for QSFP+ spine-spine connections ● Cable management is mandatory ● LLDP is your friend ● OSPF bad, BGP good
  31. 31. Virtual Networking ● Frame Forwarding – VMware vSwitch (Stand alone & distributed) – Linux Bridge – MAC VTAP ● OpenFlow – VMware NSX – Open vSwitch ● Pass through – SR-IOv – MAC VTAP (depends on hardware capabilities)
  32. 32. Routers & Firewalls ● If you are rich, buy one ● Otherwise, build one: – Keep an eye on PCI Express bandwidth – Moving between NUMA nodes is bad – LOTS OF RAM (at least 16GB to hold Internet routing table) – NICs matter
  33. 33. NICs
  34. 34. About dem NICs ● Intel NICs are great at 1GbE ● Intel and Broadcom are descent at 10GbE ● Broadcom sucks at 1GbE ● Realtek to be avoided at all costs ● Mellanox is supposed to be great now with Connect-X3 for 10/40GbE (FB uses them)

×