This document discusses container orchestration integration between OpenStack Kuryr and Apache Mesos. It begins with introductions to Docker and Mesos, describing how they are used to distribute workloads across container hosts. It then reviews the history of networking in Docker and Mesos, how libnetwork was developed for Docker networking, and how Mesos uses IPAM and network isolator modules. Finally, it describes how OpenStack Kuryr acts as a translator between Neutron and libnetwork, allowing Neutron networking APIs to manage containers running on Mesos.

1. Container Orchestration
Integration:
OpenStack Kuryr
and
Apache Mesos
@takufukushima

2. @takufukushima
• MidoNet contributor sometimes
• OpenStack Kuryr contributor these
days

3. Agenda
1. Introduction to Docker and Apache
Mesos
2. The history of Docker and Apache
Mesos Networking
3. OpenStack Kuryr as the building block
4. Summary

4. 1. Introduction to Docker
and Apache Mesos

5. We need the cluster manager
• We distribute workloads to containers on
hosts or VM instances
• Docker and other containers are building
blocks
• We want to manage them from the bird’s-
eye view

6. Container cluster managers
• Docker native
• Docker Swarm
• Apache Mesos
• Docker based
• Google Kubernetes

7. rocks
• Blazing fast (VM? Huh?)
• Great ecosystem
• e.g., DockerHub, Meetups
• Golang dev hipsters
And nice art works
(seriously)

8. The dark side of
• “fundamentally flawed”
• “It’s The Future”
• “So I just need to split my simple CRUD app into 12 microservices, each with
their own APIs which call each others’ APIs but handle failure resiliently, put
them into Docker containers, launch a fleet of 8 machines which are Docker
hosts running CoreOS, “orchestrate” them using a small Kubernetes cluster
running etcd, figure out the “open questions” of networking and storage,
and then I continuously deliver multiple redundant copies of each microservice
to my fleet. Is that it?”

9. rocks
• The core of Mesosphere DCOS
• Originally research project of UCB RAD (AMP) lab
• Great ecosystem and use cases
• Twitter, Apple, Airbnb, eBay and so on
• Pluggable frameworks
• Apache Aurora, Chronos, Marathon

10. architecture
Retrieved from http://radar.oreilly.com/2015/10/swarm-v-fleet-v-kubernetes-v-mesos.html

11. and
• Mesos has few containerizers
• cgroups and namespace based containerizer
• Docker containerizer
• External containerizer

12. 2. The history of Docker and
Apache Mesos Networking

13. Docker networking
• docker0 bridge
• veth pairs and netns
• --icc and --link
• --net
• bridge, container, host, none
• NAT by iptables

14. Extended Docker networking
• CoreOS flannel
• For Kubernetes
• Weave
• SocketPlane
• pipeworks

18. libnetwork
• Networking component as a plugin
• docker network command
• Drivers separated from Docker core
• bridge
• overlay
• none
• Remote driver opened up for everyone

19. overlay driver
• SocketPlane
• Container communication over the hosts
• VXLAN
• libkv for storing the network state in the
distributed datastore
• --cluster-store and --cluster-
advertise
• etcd, Consul and ZooKeeper

20. network.CreateEndpoint()
controller.NewNetwork()
InterfaceInfo
endpoint.Join() endpoint.Leave()
endpoint.Delete()
network.Delete()
sandbox.Info
e.g., netns
Network Controller
Driver
Built-in Remote
Endpoint
Bridge
Host
None
Overlay
Calico
Kuryr
Weave
Network Network
Endpoint Endpoint
IP addresses
MAC addresses
Routes
DNS entries
Container Container
Option
Label
CLI
Sandbox Sandbox
Container Container
libnetwork CNM

21. networking
• Almost the same as Docker
• especially if you’re using Docker as the
containerizer
• Containers share the IP of the slaves
• NAT and netns

22. integration point
• External Containerizer Program (ECP)
• Slaves delegate the containerising to ECP
• It’s just building the Docker command
• Protobuf data is passed through stdin and
stdout
• ENV vars can be used for additional data

23. new networking
• IPAM server
• IPAM client on masters and slaves
• Network Isolator Module (NIM) on slaves
• Cleanup Module on masters

24. new networking
Retrieved from https://github.com/apache/mesos/blob/master/docs/networking-for-mesos-managed-containers.md

25. 3. OpenStack Kuryr as
a building block

26. OpenStack?
Why is it
related with
Docker?

27. OpenStack and Docker
• OpenStack and Docker are exclusive for each other at
this point
• Multi tenancy
• Strict resource isolation
• OpenStack Magnum
• Docker managed by OpenStack
• Docker containers on VM instances
• OpenStack Kolla

28. Revisiting OpenStack Neutron
• Neutron is a networking component of OpenStack
• Networking resource allocation through the API
• Vendor agnostic APIs
• Many network controllers supporting these APIs
• The model of libnetwork is getting close to
Neutron’s one

29. Maximizing the
developers effects:
Investment for the
most effective way

30. OpenStack Kuryr
• A new component in “Neutron Stadium”
• A translator between Neutron and libnetwork
• Map the API calls on the remote driver into
Neutron’s API calls

31. OpenStack Kuryr
• A new component in “Neutron Stadium”
• A translator between Neutron and libnetwork
• Map the API calls on the remote driver into
Neutron’s API calls

32. OpenStack Kuryr
• A new component in “Neutron Stadium”
• A translator between Neutron and libnetwork
• Map the API calls on the remote driver into
Neutron’s API calls

33. Kuryr architcture
Neutron
Daemon
Host
Distributed
Datastore
Distributed
DatastoreDistributed
Datastore
Container
Container
Container
Daemon
Host
Container
Container
Container
Daemon
Host
Container
Container
Container
Kuryr Kuryr Kuryr
Keystone

35. Kuryr as a translator

36. Kuryr as a translator

37. 4. Summary

38. Container networking made easy
• Container networking had some issues
• The new networking models and APIs are
emerging
• OpenStack Kuryr can be the common
building block

39. Kuryr as a translator
The end of slides.
Any questions?