DockerCon EU 2015: The Missing Piece: when Docker networking unleashing soft architecture 2.0
•Download as PPTX, PDF•
6 likes•4,592 views
Presented by Adrien Blind, DevOps Coach, Socîeté Générale and Laurent Grangeau, Solutions Architect, Finaxys Docker now provides several building blocks, combining engine, clustering, and componentization, while the new networking and service features enable many new usecases such as multi-tenancy. In this session, you will first discover the new experimental networking and service features expected soon, and then drift rapidly to software architecture, explaining how a complete Docker stack unleashes microservices paradigms.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to DockerCon EU 2015: The Missing Piece: when Docker networking unleashing soft architecture 2.0
Similar to DockerCon EU 2015: The Missing Piece: when Docker networking unleashing soft architecture 2.0 (20)
More from Docker, Inc.
More from Docker, Inc. (20)
Recently uploaded
Recently uploaded (20)
DockerCon EU 2015: The Missing Piece: when Docker networking unleashing soft architecture 2.0
- 1. The missing piece: when Docker networking unleashes software architecture 2.0 A. Blind DevOps coach Société Générale @adrienblind L. Grangeau Solutions architect Finaxys @laurentgrangeau
- 2. Agenda 2 - Starters Docker networking & volume features discovered 3 - Dessert Taste-an-app 1 - Apetizer Back on current Docker paradigms 3 - Main course Application architecture shifts
- 3. Back on current docker paradigms
- 4. Back on Docker paradigms ‘’A universal, self-sufficient and standard artifact embedding an app module, and its subsequent infrastructure configuration’’ Immutable Versionned Light Portable Disposable Programatic Social Incremental It’s mainly focused on enclosing computing capabilities: what about storage ? Network ?
- 5. Networking & volume features discovered
- 8. Docker networking The Container Network Model (CNM) A docker container Endpoint A docker container Endpoint A docker container EndpointEndpoint Network sandbox Network sandbox Network sandbox Front network Back network
- 9. Docker networking $ docker network create mynetwork 5000dec7c180a63d87031de7e6bfcf2b25cf1e5daef6338f16fbd4 451210a938 $ docker network create –d overlay multihostnetwork e6537b859359843bc02392245ab226070f79dbf87be2d492969c84 3f89fb6de6
- 10. Docker networking $ docker network inspect mynetwork [ { "Name": "mynetwork", "Id": "5000dec7c180a63d87031de7e6bfcf2b25cf1e5daef6338f16fbd4451 210a938", "Scope": "local", "Driver": "bridge", "IPAM": { "Driver": "default", "Config": [ {} ] }, "Containers": {}, "Options": {} } ]
- 11. Docker networking Host Host Host Host SDNs SDN 1 SDN 2 SDN 3
- 13. Docker networking Docker Compose evolved to embrace new networking features $ docker-compose --x-networking --x-network-driver=overlay up $ docker-compose up
- 14. Docker volumes
- 16. Docker volumes
- 17. Docker volumes $ docker volume ls DRIVER VOLUME NAME local cf872ca21d27843f6b6319ac1a34390… local f19f50251f48c64a6b33a5c637c2330… $ docker volume inspect cf872ca21d27843f6b6319ac1a34390dd38d94… [ { "Name": "cf872ca21d27843f6b6319ac1a34390dd38d94…", "Driver": "local", "Mountpoint": "/mnt/sda1/var/lib/docker/volumes/[…]/_data" } ]
- 18. Docker volumes $ docker volume create –d volplugin --name pool/name Cf872ca21d27843f6b6319ac1a34390dd38d94ed4649cd985456d5 23fb05d4cc $ docker run –d –p 8080:8080 –v pool/name:/var/jenkins_home jenkins 96aec6f4e45e050dfb4f75a1009e7f105bced5b406752e62d47061 5d07348b07
- 19. Take-away
- 21. Application Compute (Run containers) Storage (Volumes) ‘’Immutability of containers led to data Externalization in separate objects’’
- 22. ‘’Fine-grained granularity of containers led to closely interconnect them’’ Application Compute (Run containers) Storage (Volumes) Transport (Network)
- 23. ‘’The whole topology can now be described’’ Application Compute (Run containers) Storage (Volumes) Transport (Network) Topology (Compose)
- 24. ‘’Docker finally shifted to object-oriented infra. architecture’’ Application Compute (Run containers) Storage (Volumes) Transport (Network) Topology (Compose) CaaS platform (Swarm, Machine...)
- 27. Security paradigms shifts Your IT opens up • Externalization • Cloud (IaaS/PaaS/SaaS) Open up your IS • B2B, services exposition • Multi tenancy More & more breaches appears in your Great Wall of China! Security is an app feature, not just an outer infra. concern Onboard security guys in your feature team SecDevOps
- 28. Security paradigms shifts The necessary porosity of your IS requires to stick security closer to each application: sandbox your apps and expose protected interfaces! Network is part of application topology Authentication & authorization is the key
- 29. Network paradigms shifts VM VM VM VM VM VM VM VM VM Internet Internet DMZ Physical overview Logical overview Tenant #1 Tenant #2 LAN LAN DMZ1 DMZ2 Traditional networks relies a lot on low layers (L2, etc.) Application topologies are quite different from physical ones
- 30. Network paradigms shifts SDNs proposes network solutions embracing cloud paradigms Massively multi-tenant Thousands tenants, massively scalable Easy & fast (de)provisioning Infra as code, API centric Infrastructure agnostic L3, does not stick with lower levels (physical designs, vlans & co) Decouple infrastructure & tenants lifecycles Cross technology, vendor agnostic
- 31. From Enterprise Services buses to full-mesh topologies ESB Service Service Service Service Service > ServiceService Service Service Service Micro services
- 32. Fine-grained, highly decoupled and atomic purpose centric services Designed for failure Multi-versioned Scalable Micro services Stateless Share-nothing Immutable Continuously delivered Distributed
- 33. Service consumer Service provider Registry 2. Find 1. Publish 3. Bind Leverage on a Service registry to discover where are services located Micro services
- 34. Resilience & scalability: apps problem now! Vertical > horizontal Dumber infrastructure Apps designed for failure & scalability Data to be externalized Structured: MongoDB, Hadoop, Cassandra, Elastic Search... Binaries: object storage with Ceph, OpenStack Swift... Helpful patterns: stateless, share nothing, loose coupling... Infrastructure rationalization Low-cost, poor-SLA commodity
- 35. « Organizations which design systems... are constrained to produce designs which are copies of the communication structures of these organizations ». - M. Conway, 1968 Consider shifting your organization if you wish to shift your architecture Forget about the central architects myth of organizing, integrating everything Consider changing your organization to expect changing the architecture! promote feature teams Organization
- 36. Docker suits perfectly new applications challenges Create docker networks to isolate applications Docker container properties fits micro-services challenges Resilience & scalability is mostly about multiplying containers Expect to discuss roles shift in organization
- 37. Taste-an-app
- 38. Execution platform Registry Docker-machine Docker-swarm The registrator discovers new containers and feeds the registry
- 39. Application design Provider micro serviceConsumers The python app module exposes a REST service searching information in the MongoDB The NGINX reverse proxy forward app. requests on one of the python instance registered in Consul Find
- 40. Application topology & runtime The whole application topology is stored as: docker-compose yaml file docker-compose args (aka --x-networking & --x-network-driver) You can scale up or down the python instances of the micro- service using traditionnal docker-compose scale command
- 41. Network view Only the load balancer VIP is exposed externally A WAF instance could secure this entrypoint SDN « myapp » Host network Provider micro serviceConsumers
- 42. Network view - advanced Provider micro service Consumers SDN « front » SDN « back » Host network Back Middle Front ‘’To enhance security you may decouple each application tier’’
- 43. Zoom on the registry usages At infrastructure level, the registry is used by swarm (internally) to be aware of the cluster’s participants At container level, the registrator enable to registers any container instances, grouped per type At application level, the consumers asks the registry where the micro-service (the NGINX front-end) is located Noticed the three different usages of the registry ? You may consider using different registries for each usage : for example an internal registry for the micro service internal topology
- 44. Conclusions
- 45. Docker shifted from universal containers to object-oriented infrastructure Security is an app concern Software is eating the world: application architecture is the key, infrastructure is commodity
- 46. Thank you! A. Blind @adrienblind adrien.blind@sgcib.com L. Grangeau @laurentgrangeau laurent.grangeau@gmail.com
Editor's Notes
- LGR
- ABL
- ABL Docker c’est avant tout le paradigme du conteneur Plein de caractéristiques intéressantes (approfondir l’une ou l’autre) Mais c’est surtout là pour embarquer un morceau de code à exécuter Hors, une application, ca va au delà : c’est différents morceaux de code différents, c’est de la data, le tout doit interagir... On se propose de creuser ensemble cette vision « applicative », on va voir comment docker intègre cette vue
- ABL
- LGR
- LGR Let’s go back briefly in the past. On April, after buying Socketplane one month before, Docker made a promise : that any containers can speak freely and easily on a network without any limitation. By this promise is born libnetwork and the container network model, wich was release with Docker 1.9 earlier this month. With this new library, Docker can do SDN seemlessly. Now the network is more tight to the application, since developpers can create and update networks as they want. The network is part of an application architecture. Libnetwork is based on the container network model, which we will talk in the next slide
- LGR This CNM defines 3 terms : Network sandbox : an isolated environment where the Networking configuration for a Docker Container lives. Endpoint : A network interface that can be used for communication over a specific network. Endpoints join exactly one network and multiple endpoints can exist within a single Network Sandbox. Network : A network is a uniquely identifiable group of endpoints that are able to communicate with each other. You could create a “Frontend” and “Backend” network and they would be completely isolated. The CNM provides the following contract between networks and containers. - All containers on the same network can communicate freely with each other - Multiple networks are the way to segment traffic between containers and should be supported by all drivers - Multiple endpoints per container are the way to join a container to multiple networks - An endpoint is added to a network sandbox to provide it with network connectivity
- LGR With the new networking feature, you can now create a simple network with Docker natively, even across all the hosts in the cluster with Swarm $ docker network create mynetwork You can also choose different drivers. Docker comes with 2 drivers: Bridge : Default driver. It provides the same sort of networking via veth bridge devices Overlay : Native multi-host network for docker cluster $ docker network create –d overlay multihostnetwork Other drivers can be used with plugins like Weave, Microsoft, Cisco and even VMWare ==================================== You can connect or disconnect a running container ! $ docker network connect mynetwork mysql $ docker network disconnect mynetwork mysql Containers can be attach to as many as network you want The overlay driver uses VxLAN to route traffic Overlay network driver provides out of the box, multi-host network connectivity for docker containers in a cluster You can then use this network to run containers $ docker run -itd --net=multi-host-network busybox There is also a toolbox to manage networking like ls, rm, info. SDN provides now more efficient isolation for a cluster of containers and better security as discussed later.
- LGR You can also inspect a network, like you would do with the docker inspect command. With this, you can see which driver this network use. You can also which containers are attached to this network. IPAM driver is the same as the bridge network, and does not currently support DHCP. Some people are working on this support and you will be able to add support of DHCP soon
- LGR With the new networking feature, you can now create a simple network with Docker natively, even across all the hosts in the cluster with Swarm $ docker network create -d overlay multi-host-network Overlay network driver provides out of the box, multi-host network connectivity for docker containers in a cluster You can then use this network to run containers $ docker run -itd --net=multi-host-network busybox There is also a toolbox to manage networking like ls, rm, info. SDN provides now more efficient isolation for a cluster of containers and better security as discussed later. With the definition of micro-network between containers, we can slowly drift from monolithic applications to micro-services. Docker networks take care of isolation between containers outside a network. Applications, or services, are now a whole, and can evolve independently But how can we make containers communicate with each others if they can’t speak outside a network ?
- LGR If we want containers communicate outside a network, we have to add this containers into the same network. As we have seen before, a container can have multiple endpoint, so it can belong to multiples networks, but with differents IP addresses Links can now be considered obsolete. Docker network is the new way to link containers. Network use the native docker autodiscover services to find containers based on their name. Name have to be unique in the network, even across a Swarm cluster. This feature is extremely helpful if you restart a container for example. It will be assigned a different IP, but the name will be the same. It will be registered into the KV store, and others containers can find its IP address baed on its name. -------------------------------------------------- Multi instanciation de conteneurs pour un même service ? Round robin ? comment le client fait ? est ce que ca retire le besoin de créer des load balancer pour équilibrer ? c’est une alternative ? Ca complete ? Cote positif : pas de spof sur le LB (design full mesh et non étoile), et ca fait un composant en moins. Mais pas de heartbeat / healtcheck ? quid si l’un des conteneurs est down ?... ? Quid de la partie nommage ? y a du dns intégré ? completement indépendant ? Que renseigne la commande service /qu’est ce qu’on consomme ? Comment on l’utilise au final ? A RETIRER ON DEVELOPPE CA PLUS TARD
- Now let’s talk about docker-compose. We have seen that we can create a network and attach or detach running containers. We have seen that we can also create a multihost network changing only the driver. We then have seen that we can create networks even on a swarm cluster, and the default driver on a cluster is a multi host driver. Don’t you think it will be cool to do the same with a simple file ? It’s here that comes Docker-compose and the new experimental features ON LE MET A LA FIN : on PARLE DES MAJ DE NETWK ET VOLUME, COMPOSE est un sous-jacent qui évolue. Ou pas de slide dédié, on intègre directement dans les slides networks
- ABL
- ABL I will speak now briefly about volume. Like network, volume appears in 1.9 release of Docker. Prior to 1.9, volumes used to be simple container, usually the same container as the service we want to run. Some use a busybox container to have the smallest fingerprint on the system. This container don’t usually run, but is used only to manage data. Others map the volume directly on the host. But what is we want to run containers on Swarm, or we want to change host ? What about resiliency ?
- ABL It’s now a simple Docker object, and can be extend with plugins, like the networking. Currently, plugins are provide by ClusterHQ, Ceph, Blockbridge, EMC, and Portworx
- ABL We can list existing volume, and inspect them easily with built in commands
- ABL We can also create volumes with different plugins. For example, we can create a volume on a Ceph cluster. Then, we can start a container, mapping the volume exposed on the Ceph volume created. The data will live even if the container is destroy and recreated. QUID VOLUMES avec COMPOSE ? QUID VOLUMES et LES CLUSTERS ? Développer le fait que le volume portte de moins en moins la payload et proxifie vers des supports plus appropriés : ceph & co
- ABL
- ABL
- ABL
- ABL
- On a parlé du quoi, mais pourqiuoi docker à fait tout ca ? Prenons un peu de recul
- ABL Let’s first discuss security shifts Traditionnaly, perimetric security was the key : everything inside your territory was protected, and the world beyond your borders was unfair The point was to control efficiently the few checkpoints
- ABL But... Your IS is spreading more & more. You have parts in your IT, but also in the cloud... You have some apps on paas, and even consume SaaS services. It becomes complex to control the gateways Forget about your great firewall of china
- ABL
- LGR Before going deep dive into the new network model, let’s talk briefly about « lecacy network » Currently, apps live in a traditionnal network, with switch or vlan that isolate network between them. In some big companies, the network is still flat, without any limitation between hosts. The network is separated from the application. Network administrators still define networks and isolation between them. Applications are not « aware » of underlying network and it does not change software architecture. With the appearance of cloud and networking as a service, developpers can automate the creation of network. Now, apps can be more easily isolated, but they still communicate only through fixed networks. There is still some limitations : network is infrastructure centric, it cannot be easily handled and easily adapted. We only create fixed network. Network administrators rarely create network for an application. Then a new concept emerged, the SDN (Software Define Networking). SDN allows network administrators to manage network services through abstration of higher-level functionnality. It’s designed to be dynamic, manageable, cost effective and adaptable.
- LGR
- LGR Before, there was ESB. And before that, there was EAI. All this ecosystem was driven by solutions architects who have global vision of the IS. Now, the applications are more and more agile. Development are faster, and releases are more frequent. We’re now talking about feature tems who wants to be as fast as possible. Applications are also API centric. Developers want ------------------------------------- le mythe de l’esb, parler d’eai, glisser vers le full mesh qui sert de fait de justification aux services ESB euipes transverses, vs équipes en peer-to-peer mythe de l’architecte avec vision d’ensemble apps plus api centric une app est une api qui envloppe des apis qui descendent jusqu’à l’infra. Relation de besoins apps intéressées : pas besoin d’un tiers Agilité, itérabilité des produits indépendants les un des autres: découpler les temporalité des applications : loose coupling, versionned api, multiversioning
- LGR Parler du continuous delivery
- ABL
- ABL Les archis de stockage de data dont on parle ici s’appliquent elles même la capacité de scaler horizontalement. Ex avec Ceph à l’extrême même plus besoin de faire du RAID pour la perf ou la résilience, c’est l’appli elle même qui le porte Apps constraints are no longer visible in the datacenters
- ABL Et docker est sexy pour ca : il permet de modeler les roles/resp efficacement : devs dans le conteneru ops dehors
- ABL
- LGR I already prepare my host to show you the demo. There is a Swam cluster, created with Docker-machine, and a Consul container which provides the KV store needed by both the Swarm cluster and the networking feature
- Question : insider consul ? Outsider : consider the scope Atention consul côté client pour trouver le service !
- LGR
- ABL Question : insider consul ? Outsider : consider the scope Atention consul côté client pour trouver le service !
- ABL Question : insider consul ? Outsider : consider the scope Atention consul côté client pour trouver le service !
- ABL
- ABL/LGR Devs rulez the world / commo infra, app autosuffisante / infra totalement masquée ???? Software is eating the world : cloud pub/infra pu un débat de l’appli. On arrive presque à modeler l’archi de l’appli avec des boites
- ABL