Presented at Rackspace Austin (downtown) on July 27th, 2016.
An inherent to component to any distributed application, networking is one of the most complicated and expansive infrastructure technologies. Container networking needs to be developer-friendly. Application-driven and portable. With developers busily adopting container technologies, the time has come for network engineers and operators to prepare for the unique challenges brought on by cloud native applications. What container networking specifications bring to the table and how to leverage them.
same demands and measurements
developer-friendly and application-driven
simple to use and deploy for developers and operators
better or at least on par with their existing virtualized data
but no need to actually know these
Container Network Model
...is a speciﬁcation proposed by Docker,
adopted by projects such as
Plugins built by projects such as ,
Project Calico Kuryr
Container Network Interface
...is a speciﬁcation proposed by
CoreOS and adopted by projects such
as , , ,
Plugins created by projects such as
, , and
rkt Kurma Kubernetes Cloud
Foundry Apache Mesos
Weave Project Calico Contiv
Container Networking Speciﬁcations
Container Network Model
Remote DriversLocal Drivers
Container Network Interface
1. Container runtime needs to:
1. allocate a network namespace to the container and assign a container ID
2. pass along a number of parameters (CNI conﬁg) to network driver.
2. Network driver attaches container to a network and then
reports the assigned IP address back to the container runtime
(via JSON schema)
CNI and CNM
Similar in that each...
...are driver-based, and therefore
democratize the selection of which type of container networking
...allow multiple network drivers to be active and used
each provide a one-to-one mapping of network to that network’s driver
...allow containers to join one or more networks.
...allow the container runtime to launch the network in its
segregate the application/business logic of connecting the container to
the network to the network driver.
CNI and CNM
Diﬀerent in that...
CNI supports any container runtime
CNM only support Docker runtime
CNI is simpler, has adoption beyond its creator
CNM acts as a broker for conﬂict resolution
CNI is still considering its approach to arbitration
Types of Container Networking
Links and Ambassadors
container receives a network stack, but
lacks an external network interface.
it does, however, receive a loopback
facilitate single host connectivity
"discovery" via /etc/hosts or env vars
facilitate multi-host connectivity
uses a tcp port forwarder (socat)
one container reuses (maps to) the networking
namespace of another container.
may only be invoked when running a docker
container (cannot be deﬁned in Dockerﬁle):
Ah, yes, docker0
default networking for Docker
uses a host-internal network
leverages iptables for network address translation
(NAT) and port-mapping
container created shares its network namespace with the host
default Mesos networking mode
easy to understand and troubleshoot
suﬀers port conﬂicts
use networking tunnels to delivery communication
Most useful in hybrid cloud scenarios
or when shadow IT is needed
Many tunneling technologies exist
VXLAN being the most commonly used
Requires distributed key-value store
K/V Store for Overlay
Docker - requires K/V store (built-in as experimental as of
WeaveMesh - does not require K/V store
WeaveNet - limited to single network; requires K/V store
Flannel - requires K/V store
Plumgrid - requires K/V store; built-in and not pluggable
Midokura - requires K/V store; built-in and not pluggable
Calico - requires K/V store
expose host interfaces (i.e. the physical network interface at
eth0) directly to containers running on the host
not necessarily public cloud friendly
Default rkt networking mode
Uses NAT (IPMASQ) by default
Creates a virtual ethernet pair
placing one on the host and the other into the container pod
leverages iptables to provide port-forwarding
for inbound traﬃc to the pod
internal communication between other
containers in the pod over the loopback
allows creation of multiple virtual network interfaces behind
the host’s single physical interface
Each virtual interface has unique MAC and IP addresses
with restriction: the IP address needs to be in the same broadcast
domain as the physical interface
eliminates the need for the Linux bridge, NAT and port-
allowing you to connect directly to physical interface
allows creation of multiple virtual network interfaces behind the host’s single
Each virtual interface has unique IP addresses assigned
Same MAC address used for all containers
L2-mode containers must be on same network as host (similar to MACvlan)
L3-mode containers must be on diﬀerent network than host
Network advertisement and redistribution into the network still needs to be done.
MACvlan and IPvlan
While multiple modes of networking are supported on a given host, MACvlan
and IPvlan can’t be used on the same physical interface concurrently.
ARP and broadcast traﬃc, the L2 modes of these underlay drivers operate
just as a server connected to a switch does by ﬂooding and learning using
IPvlan L3-mode - No multicast or broadcast traﬃc is allowed in.
In short, if you’re used to running trunks down to hosts, L2 mode is for you.
If scale is a primary concern, L3 has the potential for massive scale.
Beneﬁts of pushing past L2 to L3
resonates with network engineers
leverage existing network infrastructure
use routing protocols for connectivity; easier to interoperate with existing
data center across VMs and bare metal servers
More granular control over ﬁltering and isolating network traﬃc
Easier traﬃc engineering for quality of service
Easier to diagnose network issues
a way of gaining access to many more IP addresses, expanding from one assigned IP
address to 250 more IP addresses
“address expansion” - multiplies the number of available IP addresses on the
host, providing an extra 253 usable addresses for each host IP
Fan addresses are assigned as subnets on a virtual bridge on the host,
IP addresses are mathematically mapped between networks
uses IP-in-IP tunneling; high performance
particularly useful when running containers in a public cloud
where a single IP address is assigned to a host and spinning up additional networks is prohibitive or
running another load-balancer instance is costly
Network Capabilities and
IPAM, multicast, broadcast, IPv6, load-balancing, service discovery, policy, quality
of service, advanced ﬁltering and performance are all additional considerations to
account for when selecting networking that ﬁts your needs.
IPv6 and IPAM
lack of support for IPv6 in the top public clouds
reinforces the need for other networking types (overlays and fan networking)
some tier 2 public cloud providers oﬀer support for IPv6
most container runtime engines default to host-local for assigning addresses
to containers as they are connected to networks.
Host-local IPAM involves deﬁning a ﬁxed block of IP addresses to be selected.
DCHP is universally supported across the container networking projects.
CNM and CNI both have IPAM built-in and plugin frameworks for integration
with IPAM systems