This document summarizes Matt Johnson's presentation on container networking at Cisco LIVE! 2015. The presentation discussed: 1) The default Docker networking model of using NAT which has limits like containers only having private IPs tied to the host IP. 2) Emerging solutions like Kubernetes that provide services for containers but still rely on tunnels. 3) The benefits of integrating containers as "first class citizens" on the real network using standards like BGP without NAT or tunnels so containers can have public IPs and mobility. 4) Cisco's work to implement these ideas in an open source project called the Cisco Microservices Environment based on Mesos to provide a preconfigured environment for real workloads.

1. Matt Johnson
#vBrownBag at Cisco LIVE! 2015
9th June 2015
Container networking for real workloads.
Why Default isn’t Best
matjohn2@cisco.com
@mattdashj
Presentation recording available at:
https://www.youtube.com/watch?v=z-GD-ctjCRc

2. 2© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public
Hard to say ‘container’
nowadays without…
So we’ll start our story here.

3. 3© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Quick Overview: Docker Networking
host docker0
172.17.42.1/16
veth049e89e
eth0
172.17.0.21/16
veth049e88f veth049e87af
eth0
192.168.1.10
eth0
192.168.1.10
eth0
192.168.1.10
host eth0
docker run --net=bridge (default) docker run --net=host
Container shares the host’s IP address and MAC
address.
vEth pair linked into container from docker0 bridge.
DNAT provides inbound ports to containerized apps.
• Default IP address is dynamically allocated from unused range
(ie 172.17.0.0/16.)
can be set with –fixed-cidr on docker daemon
• Default GW for containers is the IP addr of the host docker0 interface
can be set with –bip on docker daemon
eth0
172.17.0.21/16
eth0
172.17.0.21/16 containers
3
iptables (NAT)iptables (NAT)

4. 4© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public
Why NAT?
- Works out of the box on a host with a single IP
Dockers’ rise to prominence due to User Experience.
Not new tech.
- LXC, Existed
- Containers, Nothing new (Solaris Zones Anyone?)
The difference is it had never been as easy to
use/explain/build/share.

5. Not without it’s limits:
- Container IP tied to Host.
- Only one container can have a ‘common’ port per host.
- Careful manual port assignment.
- Dynamic assignment with support in the app to support it.
- App’s need to be told to read ‘real’ network info from $ENV’s

6. 6© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Extending Networking: Kubernetes
Pod - a co-located group of Docker containers with
shared volumes. They're the smallest deployable units
that can be created, scheduled, and managed with
Kubernetes.
Service - provide a single, stable name and address
for a set of pods. They act as basic load balancers.
Label - are used to organize and select groups of
objects based on key/value pairs.
Replication Controller - are used to organize
and select groups of objects based on key/value pairs.
master server
apiserver
etcd
controller manager
scheduler
skydns
pod
C11
C12
C13
minion
pod
C21
C22
C23
pod
C41
C42
C43
minion
pod
C14
C15
pod
C44
minion
pod
C24
pod
C44
C45
minion
pod
C24
pod
C11
C12
C13
minion
pod
C21
C22
C23
service(IP addr/DN)replication Controller service(IP addr/DN)
cluster
6

7. 7© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Extending Networking: Kubernetes
7

8. Tunnels are great… Right?

9. 9© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public
Tunnels
We don’t have to deal with
whatever the underlying
network is.

10. 10© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public
Tunnels
Although we do have to deal
with…
Ramps! On/Off the overlay.
Potential bottleneck / scalability issues
An Interesting L2 Design.
Subject to implementation, may lack multicast
support, treat broadcast strangely etc.
Debug overhead.
I only have 50 VM’s, but which one of my
1,225 tunnels (full mesh) is causing the issue.

11. 11© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public
Tunnels
Although we do have to deal
with…
MTU and Frag. avoidance.
It is a tunnel after all!
Security.
Subject to implementation, spanning ARP
really isn’t ideal. Lots of containers running
privileged.

12. 12© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Future Docker Networking: libnetwork
The Container Network Model (CMN)
12
Docker Container Docker Container Docker Container
Network Sandbox Network Sandbox Network Sandbox
Endpoint Endpoint Endpoint Endpoint
Backend Network Frontend Network
• Network Sandbox -- isolated environment where container network configuration lives
• Endpoint – network interface tied to a specific network
• Network – a uniquely identifiable collection of Endpoints that are able to communicate with each other
A pluggable interface. Expected to first ship in Docker 1.7. Distributed bridge plugin under development.

13. 13© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Future Docker Networking: Overlay/L2 Centric
The Container Network Model (CMN)
13

14. 14© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public
So wouldn’t it be great if…
We could network containers without NAT,
Without requiring a ‘tunnels everywhere’ approach,
Without on or off ramps,
With full IPv4 and IPv6 support,
And at Internet-scale?

15. 15© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public
We Can!
.. If we don’t treat containers
as Unique/Special

16. 16© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Scalable DataCenter topologies have existed for
quite a while…

17. And no, this isn’t about to turn into a Cisco
sales pitch… promise!

18. 18© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public
Standards
Use Networking standards
tried and tested by the rest of
the industry. Already used
every day at unprecedented
Scale.
Don’t reinvent the wheel.
Integrate and benefit from
existing network team skillsets.

19. 19© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Proposed solution with Docker / Container Networking

20. 20© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Containers as a first class citizen of the real network
• Devolve routing down into each host.
• BGP as the saleable, proven control plane.
• Standard, proven linux networking stack.
• Containers have IP’s..
• No NAT.
• DNS works properly.
• No tunnels necessary.
• Support for legacy applications & clustering.
• Support for multicast.
• IPv4 & IPv6 Support.
• Standard network protocols that the rest of the industry uses
• Service providers and Enterprises can easily integrate.

21. 21© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public
Buy Today…
And we’ll throw in container mobility for
free!
IP’s no longer bound to the IP of the host,
If a workload moves, the IP can go with it.
Containerized data services will benefit
massively from this.

22. 22© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public
So where is this work going?
Validated ideas through other
container Industry
supporters.
Not the only people thinking
this way.
Implementation is happening
now!
Which brings me onto a
side note…

23. Matt Johnson
#vBrownBag at Cisco LIVE! 2015
9th June 2015
This isn’t a real title.
The ‘Nothing really to do with
networking’ part of the talk
@mattdashj

24. Snowflakes!

25. 25© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
GLUEFlakes!

26. Snowflakes!

27. 27© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public
Avoid GlueFlakes!
There’s something to this
whole ‘user experience’
thing!
MicroServices Infrastructure
release 0.3 TODAY!
we’re changing the name to something
less sucky soon too!

28. 28© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Cisco Microservices Environment based on Mesos.
• Fully open source.
• Framework support in Mesos leveraged to deploy scalable data,
monitoring and assurance services all-in-one.
• We’ve designed, written and tested the glue.
• ‘Dogfooding’ internally. We’ve built it to solve real problems.
• Contribute vs Creating a new GlueFlake.
• ‘Out of the Box’ environment for real-life workloads.
Enable the automatic deployment of development
dependencies!

29. 29© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public
So back to that network thing…
You guessed it… It’s going into MicroServices Infrastructure.
Preconfigured to work with data and marathon workloads
out of the box.
/ciscocloud/microservices-infrastructure
Roadmap updates and official announcements soon…

30. Thanks for Listening!
Questions?