Slide presentation from the April 16th, 2015 Downtown NY Tech Meetup hosted at Control Group and presented by Adam Rothschild from Packet (www.packet.net) The Docker and Linux container space is causing a re-evaluation of best practices around network models. The standard solutions around overlay networks and encapsulation have serious ramifications and drawbacks, but there are other alternatives. Adam gives an overview of the current networking landscape and what other potential solutions are in the context of Project Calico, which is a scale-out networking fabric for bare metal, container, VM, and hybrid environments. About Adam Rothschild Adam Rothschild is Packet’s Senior Vice President of Network and Datacenter Infrastructure; he is responsible for all facets of the company’s next-generation infrastructure strategy and engineering. Adam was previously a member of Voxel dot Net’s founding technical team, where he served as Vice President of Network Architecture. Following Voxel’s sale to Internap in 2011, he held several leadership positions at Internap, and was most recently Senior Manager of Network Architecture and Product Development. - See more at: https://www.packet.net/about/team/adam-rothschild

1. Let’s Talk Container
Networking: Packet
and Project Calico
Adam Rothschild <asr@packet.net>
Co-Founder and SVP, Infrastructure
New York, NY - April 16, 2015

2. AGENDA
● Packet and Container Networking Use Cases
● Container Networking 101
● Leading Projects and Architectures
● To Overlay or Not to Overlay
● Review and Calico Intro

3. ABOUT PACKET
And what’s with the robot...

4. PACKET’S MANIFESTO
● Provide high-performance bare metal compute nodes, available globally in <5 minutes.
● Ensure high-performance networking at all layers:
○ Bonded NICs at 2x1G or 2x10G
○ Dense 10G/40G on the backbone (soon: 100G)
○ Backbone network designed with copious capacity and eyeball delivery in mind
○ High availability: no SPOF on the top of rack, or core network!
● Design for all of the above with “Internet scale” in mind:
○ Support many thousands of hosts inside a datacenter
○ Support many thousands of disparate customers
○ Support backend networking between datacenters
○ Support IP mobility, global load balancing, between datacenters
○ Legacy-free layer-3 datacenter network to meet these needs; zero VLANs and STP
throughout our entire production network (* some on our parallel management network,
out of full disclosure :-)

5. OUR INTERNETS; L3
EVERYWHERE
to POP to POP
Optical (n*10G)
Copper (1G)
2*1G bonded 2*1G bonded
BBR
DSR DSR
ESR ESR ESR
Servar Servar
BBR
Packet
Datacenter
Carrier Hotel POPCarrier Hotel POP
ESR
to NSP/peering
CSR CSR
to NSP/peering

6. CUSTOMER NEEDS
Stripped down to the essentials, Packet’s ecosystem and “Internet scale”
customers want:
● Simple node discovery, ability to move a service/container between hosts
without provider coordination
● Secure and free/cheap back-end transfer
● IP takeover (e.g. heartbeat) for HA/DR use cases
● Security zoning between server tiers
● VPN between diverse providers or cloud environments (e.g. mix and match
AWS, DO, SL compute)

7. CONTAINER NETWORKING
101
Defaults and Concepts

8. DOCKER NETWORKING
The Problem:

9. DOCKER NETWORKING
● Remember, Containers != Small VM’s
● Environments can be expected to have rapid creation / deletion of
containers
● Containers are expected to move around alot
● Finally something that is cross provider -- but what about the network?
● Security? What security?

10. DOCKER NETWORKING
To recap, if you’re running containers today, your networking options include:
● Run docker separately on each box; exposing ports on public or private interfaces so containers
can talk to each other.
● Run in-between / overlay solutions like Weave to fully abstract the networking.
● Run "ready-to-go" multi-host platforms for docker like Deis, Flynn, or Rancher.
● Build a shared bridge on a meshed network among your boxes and get your Docker services to
spawn containers there.

11. LEADING PROJECTS
Open Source Projects and Architectures

12. Route based
THE LANDSCAPE
FLANNEL
WEAVE
Overlay / tunnel based

13. ROCKET (PROPOSED)

14. WEAVE

15. RANCHER
1. Creates a full-mesh network between containers using IPSEC tunnels
2. Rancher server orchestrates IPSEC setup and key management. Fully
automatic. No manual steps required
3. IPSEC tunnels are created on demand to reduce overhead
4. Rancher server manage DHCP server and IP address allocation.
Allocates 10.42/16 address globally by default.
5. Rancher takes over container linking from Docker
6. Works with Docker native API and command line. Rancher listens for
Docker events and sets up networking in the background.
7. Optionally disable encryption to improve performance

16. THE APPROACHES: OVS

17. OCTO TECHNOLOGY
...seriously?!

18. TO OVERLAY OR NOT TO OVERLAY
That’s the Question

19. OVERLAY CONCERNS
I am an infrastructure operator, not a software engineer, by trade, and I see
many operational challenges:
● Performance bottleneck as additional encapsulations (e.g. VXLAN, GRE,
MPLS) and crypto (IPSec, SSL/TLS) are handled by the host
○ Not a problem with VMs, as network is constrained before compute
○ Very much a problem with powerful bare metal
● Overlay agents are required on the host/hypervisor layer
● Topology is extremely complex, difficult to understand and troubleshoot; is
the connectivity problem with the overlay or underlay?
● IP multicast required for some implementations; “BUM” flooding
● Annoying defaults: [double] NAT, hardcoded subnets (which could overlap)
● Scaling to large host count is still an issue with many implementations!
● All of this complexity for questionable gain, as we are about to see...

20. NEEDS REVISITED
Do we actually need an overlay for these?
● Secure and free/cheap back-end transfer
● Simple node discovery, ability to move a service/container between hosts
without provider coordination
● IP takeover (e.g. heartbeat) for HA/DR use cases
● Security zoning between server tiers
● VPN between diverse providers or cloud environments (e.g. mix and match
AWS, DO, SL compute)
VPN is the only real use case today, and it’s becoming decreasingly relevant as
the cloud interconnection market takes off. See: Equinix Cloud Exchange,
SoftLayer and Amazon Direct Connect, et al.

21. GET INVOLVED!
● Rocket Networking Spec: https://github.
com/coreos/rkt/issues/273
● Docker Networking Spec: https://github.
com/docker/docker/issues/8951

22. Questions or
comments
welcome:
asr@packet.net