This document discusses several network overlay options in Docker: Weave, Flannel, and Libnetwork. Weave creates a custom bridge and uses encapsulation to connect containers across hosts, but has low throughput due to packet processing in userspace. Flannel assigns each host a subnet and supports backends like VxLAN, with higher throughput than Weave by using the kernel driver. Libnetwork is integrated with Docker and supports custom drivers like Weave; it defines networks and services and allows containers to attach across hosts, with throughput close to Flannel due to using the VxLAN driver.

1. Network Overlay Options in
Docker
Syed Mushtaq Ahmed
syed.ahmed4@mail.mcgill.ca

2. Networking in Docker
● When docker daemon starts, it sets up the docker0 bridge which is the entry point
to all container traffic.
● Communication between local containers works but anything outside should be
port forwarded.
● Can cause problems if multiple containers want to communicate over the same
port.
● Overlay networks allow the possibility of seamless communication between
multiple containers without jumping multiple hoops.
● We examine three overly networking options that are available in Docker. Weave,
Flannel, Libnetwork.

3. Setup

4. Weave
● “Weave creates a virtual network that connects Docker containers deployed
across multiple hosts and enables their automatic discovery.”[1]
● Weave creates a custom bridge to which each container connects.
● Uses a “router” container which intercepts packets destined to the bridge,
encapsulates them and sends it over to the right peer router.
● Each router learns which mac addresses belong to which peer router and is also
aware of the overall topology.
● Uses a custom encapsulation format and batches multiple frames in a single UDP
payload
[1] https://github.com/weaveworks/weave#readme

5. Weave Setup
#install
curl -L git.io/weave -o /usr/local/bin/weave
chmod a+x /usr/local/bin/weave
#start
weave launch [$PEER_IP]
eval $(weave env)
#run
docker run --name c1 -it ubuntu

6. Weave
Overlay Throughput
Native (Inter-VM) 3.2 Gb/s
Weave 147 Mb/s
Weave is slow because the router container uses PCAP to capture packets and encapsulate
them in userspace which is every expensive.

7. Flannel
● “Flannel is a virtual network that gives a subnet to each host for use with
container runtimes.”[2]
● Each host gets a subnet assigned to it and IP allocations to a container happen
from that subnet.
● Uses etcd for storing configuration.
● Can have multiple “backends” (UDP, VxLAN, AWS-VPC)
● docker0 is kept the default bridge so no extra interfaces in the container.
● Supports multi-network mode but is static and still experimental.
[2] https://github.com/coreos/flannel

8. # Setup Etcd ...
#Build flannel
git clone https://github.com/coreos/flannel.git
cd flannel
docker run -v `pwd`:/opt/flannel -i -t google/golang /bin/bash -c "cd /opt/flannel &&
./build"
#push network config to etcd
curl -L http://127.0.0.1:2379/v2/keys/coreos.com/network/config -XPUT -d value='{
"Network": "10.0.0.0/8",
"SubnetLen": 20,
"SubnetMin": "10.10.0.0",
"SubnetMax": "10.99.0.0",
"Backend": {
"Type": "vxlan",
"Port": 7890
}
}'
Flannel Setup (kernel > 3.15)

9. #start flannel
cd flannel/bin
./flanneld -etcd-endpoints="http://127.0.0.1:2379"
# start docker with the flannel (you may have to change the docker0's IP
service docker stop
source /run/flannel/subnet.env
docker -d --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}
#start containers normally
docker run -it ubuntu
Flannel Setup (kernel > 3.15)

10. Flannel
Overlay Throughput
Native (Inter-VM) 3.2 Gb/s
Weave 147 Mb/s
Flannel 1.22 Gb/s
Flannel is faster than Weave because it uses the kernel Vxlan driver thus avoiding packet copy to
user space.

11. Libnetwork
● Currently in active development.
● Integrated tightly with Docker, it provides native multi-host networking.
● Flexible to support any external drivers (Weave for example).
● Defines networks and services as top level objects.
● We can dynamically create multiple networks, services belonging to different
networks and attach them to containers.

12. #build docker binary with experimental support
git clone https://github.com/docker/docker.git; cd docker
DOCKER_EXPERIMENTAL=1 make
#setup a Key-Value store (using Consul here)
#host1
consul agent -server -bootstrap -data-dir /tmp/consul -bind=<host-1-ip-address>
#host2
consul agent -data-dir /tmp/consul -bind <host-2-ip-address>
consul join <host-1-ip-address>
#start docker
docker -d --kv-store=consul:localhost:8500 --
label=com.docker.network.driver.overlay.bind_interface=eth0
[--label=com.docker.network.driver.overlay.neighbor_ip=<host-1-ip-address>]
Libnetwork Setup (kernel > 3.15)

13. #Create network with overlay driver
docker network create -d overlay mynet
#create a service under the network
#host1
docker service publish svc1.mynet
#host2
docker service publish svc2.mynet
#start a container and attach the service to it
#host1
CID=$(docker run -itd ubuntu)
docker service attach $CID svc1.mynet
#host2
CID=$(docker run -itd ubuntu)
docker service attach $CID svc2.mynet
Libnetwork Setup (kernel > 3.15)

14. Libnetwork
Overlay Throughput
Native (Inter-VM) 3.2 Gb/s
Weave 147 Mb/s
Flannel 1.22 Gb/s
Libnetwork 1.32 Gb/s
Libnetwork uses the same Vxlan driver as Flannel. It has a slightly higher throughput possibly
because Flannel sets a slightly lower MTU (1450 instead of 1500) on the docker bridge.

15. Other approaches
● Rancher uses IPSec tunnels between different hosts to implement their overlay.
● Socketplane used Open VSwitch as their container bridge and used its VxLAN
tunneling capability.

16. Questions?