Here is a presentation I gave on the 24th of May, 2016 for the Bare Metal Club of Austin on Networking

1. Networking!
Carl Perry
caperry@edolnx.net
Twitter/Reddit/Steam/Freenode: edolnx

2. whoami
●
Principal Linux Administrator @ Ncsoft
– We make videogames (Guild Wars, Blade & Soul)
●
Founder of Sphere Cube LLC
– Privacy and Security focused Infastructure hosting on
esoteric platforms (ARM now, RISC-V later)
●
Previously: Chef Software, BlueBox, RedHat,
eNovance, Midokura, DreamHost, Rackspace ...
●
I work a lot with networking stuff

3. Three Basic Types of Networking
●
Inter-site:
– “Transit”, MPLS, ISP Links
●
Intra-site:
– Ethernet, Fiber Channel, Infiniband
●
Virtual:
– vswitch, tunneling, pass-through

4. Let's talk about layers

5. Let's talk about layers

6. Let's talk about layers

7. Let's talk about layers
OSI Model
Layer Protocol
Data Unit
Function Examples
Host
7 Application
Data
High-level APIs, including resource sharing, remote file
access, directory services and virtual terminals
DotNetFtpLibrary,
SMTP web API,
SSH.NET,
SnmpSharpNet, HTML
Class, HTTP API server
6 Presentation Translation of data between a networking service and
an application; including character encoding, data
compression and encryption/decryption
CSS, GIF, HTML, XML,
JSON, S/MIME
5 Session Managing communication sessions, i.e. continuous
exchange of information in the form of multiple back-
and-forth transmissions between two nodes
RPC, SCP, NFS, PAP,
TLS, FTP, HTTP,
HTTPS, SMTP, SSH
4 Transport Segment
(TCP) /
Datagram
(UDP)
Reliable transmission of data segments between
points on a network, including segmentation,
acknowledgement and multiplexing
NBF, TCP, UDP
Media
3 Network Packet Structuring and managing a multi-node network,
including addressing, routing and traffic control
AppleTalk, ICMP, IPsec,
IPv4, IPv6
2 Data Link Frame Reliable transmission of data frames between two
nodes connected by a physical layer
IEEE 802.2, L2TP,
LLDP, MAC, PPP, ATM,
MPLS
1 Physical Bit Transmission and reception of raw bit streams over a
physical medium
DOCSIS, DSL, Ethernet
physical layer, ISDN,
RS-232
https://en.wikipedia.org/wiki/OSI_model

8. Really Esoteric
●
Appletalk
●
Token Ring (unless you work at IBM)
●
FDDI
●
BRI/PRI
●
Frame Relay
●
SONET

9. Fiber Channel
●
Mostly used for storage
●
Some other applications
●
Topologies: Point 2 Point, Switched, Arbitrated
Loop
●
Available in 1, 2, 4, 8, 16, and 32 Gbps native
data rates
●
Loosing Ground to iSCSI (SCSI over IP)

10. Infiniband
●
Primarily developed for HPC
●
2.5, 5, 10, 14.0625, 25, and soon 50 Gbps
native data rate (much more when agg)
●
Switched fabric topology
●
Still used in a lot of storage
●
Has lots of nifty features like RDMA

11. Hail the king
Ethernet

12. Oh wait
“...commercially introduced in 1980 and first
standardized in 1983 as IEEE 802.3...”

13. Got room for one more?
802.1q 802.11 802.1x 802.1AB 802.1AC 802.3a 802.3i 802.3z 802.3ab 802.1AX
802.3ae 802.3af 802.3ak 802.3an 802.3at 802.3ba 802.3bm 802.3bq 802.3by

14. THE CONNECTION IS MADE

15. THE CONNECTION IS MADE

16. THE CONNECTION IS MADE

17. But where do the frames go?

18. Duh

19. Bow before your new god

20. The worst kept secret
●
Nexus 3000, 5000, 7000, 9000 (pizza boxes)
●
Juniper QFX, OFX
●
Arista 7000
●
Every whitebox:
– EdgeCore (they make Dell)
– Quanta
– Supermicro
– PenguinComputing
– Agema
– HPE (some H3C)

21. Trident(+)
●
Up to 56 (64) ports
– 48x10GbE, 4x40GbE
●
Cut through
●
960 Mpps
●
560 (640) Gbps

22. Trident 2
●
Up to 128 Ports
– 32x40GbE
– 48x10GbE, 6x40GbE
●
Cut through
●
1Bpps
●
1.44Tbs

23. Where do we go from here?
SFP+25
&
QSFP+28

24. 100GbE?!?!?!

25. Tomahawk
●
Up to 128 Ports
– 32x100GbE
●
Cut through or Store
& Forward
●
3.2Tbps
●
SFP+25, QSFP+28

26. 1GbE
●
Various manufacturers of ASIC parts
●
Various features options
●
All the way from super cheap (D-Link,
Netgear) to super awesome (Broadcom
Hurricane 2 based)
●
Find what matches your price/req

27. TP vs TwinAx vs Fiber
●
SFP+ Modules, $20/ea (MM-OM4) & 2-5w
– SM is more depending on length
●
TP (CAT6a) adds $20-40 & 7w per port
●
TwinAx is passive, $20/2m @ 2.5w TOTAL

28. Exampe Deployment

29. Closeup

30. Lessons Learned
●
Use active TwinAx or MPO fiber for QSFP+
spine-spine connections
●
Cable management is mandatory
●
LLDP is your friend
●
OSPF bad, BGP good

31. Virtual Networking
●
Frame Forwarding
– VMware vSwitch (Stand alone & distributed)
– Linux Bridge
– MAC VTAP
●
OpenFlow
– VMware NSX
– Open vSwitch
●
Pass through
– SR-IOv
– MAC VTAP (depends on hardware capabilities)

32. Routers & Firewalls
●
If you are rich, buy one
●
Otherwise, build one:
– Keep an eye on PCI Express bandwidth
– Moving between NUMA nodes is bad
– LOTS OF RAM (at least 16GB to hold Internet
routing table)
– NICs matter

33. NICs

34. About dem NICs
●
Intel NICs are great at 1GbE
●
Intel and Broadcom are descent at 10GbE
●
Broadcom sucks at 1GbE
●
Realtek to be avoided at all costs
●
Mellanox is supposed to be great now with
Connect-X3 for 10/40GbE (FB uses them)