The Docker network overlay driver relies on several technologies: network namespaces, VXLAN, Netlink and a distributed key-value store. This talk will present each of these mechanisms one by one along with their userland tools and show hands-on how they interact together when setting up an overlay to connect containers.
The talk will continue with a demo showing how to build your own simple overlay using these technologies.
The Docker network overlay driver relies on several technologies: network namespaces, VXLAN, Netlink and a distributed key-value store. This talk will present each of these mechanisms one by one along with their userland tools and show hands-on how they interact together when setting up an overlay to connect containers.
The talk will continue with a demo showing how to build your own simple overlay using these technologies.
Linux offers an extensive selection of programmable and configurable networking components from traditional bridges, encryption, to container optimized layer 2/3 devices, link aggregation, tunneling, several classification and filtering languages all the way up to full SDN components. This talk will provide an overview of many Linux networking components covering the Linux bridge, IPVLAN, MACVLAN, MACVTAP, Bonding/Team, OVS, classification & queueing, tunnel types, hidden routing tricks, IPSec, VTI, VRF and many others.
Docker Networking with New Ipvlan and Macvlan DriversBrent Salisbury
Docker Networking presentation at ONS2016.
Docker Macvlan and Ipvlan Networking Drivers Experimental Readme:
github.com/docker/docker/blob/master/experimental/vlan-networks.md
Kernel requirements for Ipvlan mode is v4.2+, Macvlan mode is v3.19.
If using Virtualbox to test with, use NAT mode interfaces unless you have multiple MAC addresses working in your setup. Use the 172.x.x.x subnet and gateway used by the VBox NAT network. Vmware Fusion works out of the box.
Here is a screenshot of a VirtualBox NAT interface:
https://www.dropbox.com/s/w1rf61n18y7q4f1/Screenshot%202016-03-20%2001.55.13.png?dl=0
Content caching is one of the most effective ways to dramatically improve the performance of a web site. In this webinar, we’ll deep-dive into NGINX’s caching abilities and investigate the architecture used, debugging techniques and advanced configuration. By the end of the webinar, you’ll be well equipped to configure NGINX to cache content exactly as you need.
View full webinar on demand at http://nginx.com/resources/webinars/content-caching-nginx/
The Docker network overlay driver relies on several technologies: network namespaces, VXLAN, Netlink and a distributed key-value store. This talk will present each of these mechanisms one by one along with their userland tools and show hands-on how they interact together when setting up an overlay to connect containers.
The talk will continue with a demo showing how to build your own simple overlay using these technologies.
Linux offers an extensive selection of programmable and configurable networking components from traditional bridges, encryption, to container optimized layer 2/3 devices, link aggregation, tunneling, several classification and filtering languages all the way up to full SDN components. This talk will provide an overview of many Linux networking components covering the Linux bridge, IPVLAN, MACVLAN, MACVTAP, Bonding/Team, OVS, classification & queueing, tunnel types, hidden routing tricks, IPSec, VTI, VRF and many others.
Docker Networking with New Ipvlan and Macvlan DriversBrent Salisbury
Docker Networking presentation at ONS2016.
Docker Macvlan and Ipvlan Networking Drivers Experimental Readme:
github.com/docker/docker/blob/master/experimental/vlan-networks.md
Kernel requirements for Ipvlan mode is v4.2+, Macvlan mode is v3.19.
If using Virtualbox to test with, use NAT mode interfaces unless you have multiple MAC addresses working in your setup. Use the 172.x.x.x subnet and gateway used by the VBox NAT network. Vmware Fusion works out of the box.
Here is a screenshot of a VirtualBox NAT interface:
https://www.dropbox.com/s/w1rf61n18y7q4f1/Screenshot%202016-03-20%2001.55.13.png?dl=0
Content caching is one of the most effective ways to dramatically improve the performance of a web site. In this webinar, we’ll deep-dive into NGINX’s caching abilities and investigate the architecture used, debugging techniques and advanced configuration. By the end of the webinar, you’ll be well equipped to configure NGINX to cache content exactly as you need.
View full webinar on demand at http://nginx.com/resources/webinars/content-caching-nginx/
Taking Security Groups to Ludicrous Speed with OVS (OpenStack Summit 2015)Thomas Graf
Open vSwitch (OVS) has long been a critical component of the Neutron's reference implementation, offering reliable and flexible virtual switching for cloud environments.
Being an early adopter of the OVS technology, Neutron's reference implementation made some compromises to stay within the early, stable featureset OVS exposed. In particular, Security Groups (SG) have been so far implemented by leveraging hybrid Linux Bridging and IPTables, which come at a significant performance overhead. However, thanks to recent developments and ongoing improvements within the OVS community, we are now able to implement feature-complete security groups directly within OVS.
In this talk we will summarize the existing Security Groups implementation in Neutron and compare its performance with the Open vSwitch-only approach. We hope this analysis will form the foundation of future improvements to the Neutron Open vSwitch reference design.
Tutorial: Using GoBGP as an IXP connecting routerShu Sugimoto
- Show you how GoBGP can be used as a software router in conjunction with quagga
- (Tutorial) Walk through the setup of IXP connecting router using GoBGP
Cilium - Network security for microservicesThomas Graf
Cilium is open source software for providing and transparently securing network connectivity and loadbalancing between application containers and services deployed using Linux container management platforms like Docker and Kubernetes.
A new Linux kernel technology called eBPF is at the foundation of Cilium, which enables the dynamic insertion of BPF bytecode into the Linux kernel. Cilium generates eBPF programs for each individual application container to provide networking, security, loadbalancing and visibility.
Cilium - Fast IPv6 Container Networking with BPF and XDPThomas Graf
We present a new open source project which provides IPv6 networking for Linux Containers by generating programs for each individual container on the fly and then runs them as JITed BPF code in the kernel. By generating and compiling the code, the program is reduced to the minimally required feature set and then heavily optimised by the compiler as parameters become plain variables. The upcoming addition of the Express Data Plane (XDP) to the kernel will make this approach even more efficient as the programs will get invoked directly from the network driver.
This presentation covers the basics about OpenvSwitch and its components. OpenvSwitch is a Open Source implementation of OpenFlow by the Nicira team.
It also also talks about OpenvSwitch and its role in OpenStack Networking
BPF & Cilium - Turning Linux into a Microservices-aware Operating SystemThomas Graf
Container runtimes cause Linux to return to its original purpose: to serve applications interacting directly with the kernel. At the same time, the Linux kernel is traditionally difficult to change and its development process is full of myths. A new efficient in-kernel programming language called eBPF is changing this and allows everyone to extend existing kernel components or glue them together in new forms without requiring to change the kernel itself.
Deep dive in container service discoveryDocker, Inc.
Service discovery and traffic load-balancing in the container ecosystem relies on different technologies, such as IPVS and iptables, and container orchestrators use different approaches. This talk will present in details how Docker Swarm and Kubernetes achieve this. The talk will continue with a demo showing how applications that are not managed by Kubernetes can take advantage of its native load-balancing. Finally, it will compare these approaches to service-mesh solutions.
netfilter is a framework provided by the Linux kernel that allows various networking-related operations to be implemented in the form of customized handlers.
iptables is a user-space application program that allows a system administrator to configure the tables provided by the Linux kernel firewall (implemented as different netfilter modules) and the chains and rules it stores.
Many systems use iptables/netfilter, Linux's native packet filtering/mangling framework since Linux 2.4, be it home routers or sophisticated cloud network stacks.
In this session, we will talk about the netfilter framework and its facilities, explain how basic filtering and mangling use-cases are implemented using iptables, and introduce some less common but powerful extensions of iptables.
Shmulik Ladkani, Chief Architect at Nsof Networks.
Long time network veteran and kernel geek.
Shmulik started his career at Jungo (acquired by NDS/Cisco) implementing residential gateway software, focusing on embedded Linux, Linux kernel, networking and hardware/software integration.
Some billions of forwarded packets later, Shmulik left his position as Jungo's lead architect and joined Ravello Systems (acquired by Oracle) as tech lead, developing a virtual data center as a cloud-based service, focusing around virtualization systems, network virtualization and SDN.
Recently he co-founded Nsof Networks, where he's been busy architecting network infrastructure as a cloud-based service, gazing at internet routes in astonishment, and playing the chkuku.
Using eBPF for High-Performance Networking in CiliumScyllaDB
The Cilium project is a popular networking solution for Kubernetes, based on eBPF. This talk uses eBPF code and demos to explore the basics of how Cilium makes network connections, and manipulates packets so that they can avoid traversing the kernel's built-in networking stack. You'll see how eBPF enables high-performance networking as well as deep network observability and security.
Intro to Project Calico: a pure layer 3 approach to scale-out networkingPacket
Slide presentation from the April 16th, 2015 Downtown NY Tech Meetup hosted at Control Group and presented by Christopher Liljenstolpe from Project Calico (www.projectcalico.org)
Project Calico is a scale-out networking fabric for bare metal, container, VM, and hybrid environments. Project Calico leverages the same networking techniques used to scale out the Internet to present a highly scaleable, L3 network for those environments without the use of tunnels, overlays, or other complex constructs. We'll also do a demo of a Calico enabled Docker environment, and have plenty of time for q&a during and after.
About Christopher Liljenstolpe
Christopher is the original architect of Project Calico and one of the project's evangelists. In his day job, he's the director of solutions architecture at Metaswitch Networks. Prior to Calico/Metaswitch, he's designed and run some bio-informatics OpenStack clusters, done some SDN architecture work at Big Switch Networks, Run architecture at two large carriers (Telstra - AS1221, and Cable & Wireless/iMCI - AS3561) and been the IP CTO for Alcatel in Asia. He's also run networks in Antarctica (hint, bend radius becomes REALLY important at -50C), and been foolish enough to do a stint as a wg co-chair in the IETF. Occasionally you can have the (mis-)fortune of hearing him speak at conferences and the like.
Deep Dive in Docker Overlay Networks - Laurent Bernaille - Architect, D2SIDocker, Inc.
The Docker network overlay driver relies on several technologies: network namespaces, VXLAN, Netlink and a distributed key-value store. This talk will present each of these mechanisms one by one along with their userland tools and show hands-on how they interact together when setting up an overlay to connect containers. The talk will continue with a demo showing how to build your own simple overlay using these technologies.
Deeper Dive in Docker Overlay NetworksDocker, Inc.
The Docker network overlay driver relies on several technologies: network namespaces, VXLAN, Netlink and a distributed key-value store. This talk will present each of these mechanisms one by one along with their userland tools and show hands-on how they interact together when setting up an overlay to connect containers. The talk will continue with a demo showing how to build your own simple overlay using these technologies. Finally, it will show how we can dynamically distribute IP and MAC information to every hosts in the overlay.
Taking Security Groups to Ludicrous Speed with OVS (OpenStack Summit 2015)Thomas Graf
Open vSwitch (OVS) has long been a critical component of the Neutron's reference implementation, offering reliable and flexible virtual switching for cloud environments.
Being an early adopter of the OVS technology, Neutron's reference implementation made some compromises to stay within the early, stable featureset OVS exposed. In particular, Security Groups (SG) have been so far implemented by leveraging hybrid Linux Bridging and IPTables, which come at a significant performance overhead. However, thanks to recent developments and ongoing improvements within the OVS community, we are now able to implement feature-complete security groups directly within OVS.
In this talk we will summarize the existing Security Groups implementation in Neutron and compare its performance with the Open vSwitch-only approach. We hope this analysis will form the foundation of future improvements to the Neutron Open vSwitch reference design.
Tutorial: Using GoBGP as an IXP connecting routerShu Sugimoto
- Show you how GoBGP can be used as a software router in conjunction with quagga
- (Tutorial) Walk through the setup of IXP connecting router using GoBGP
Cilium - Network security for microservicesThomas Graf
Cilium is open source software for providing and transparently securing network connectivity and loadbalancing between application containers and services deployed using Linux container management platforms like Docker and Kubernetes.
A new Linux kernel technology called eBPF is at the foundation of Cilium, which enables the dynamic insertion of BPF bytecode into the Linux kernel. Cilium generates eBPF programs for each individual application container to provide networking, security, loadbalancing and visibility.
Cilium - Fast IPv6 Container Networking with BPF and XDPThomas Graf
We present a new open source project which provides IPv6 networking for Linux Containers by generating programs for each individual container on the fly and then runs them as JITed BPF code in the kernel. By generating and compiling the code, the program is reduced to the minimally required feature set and then heavily optimised by the compiler as parameters become plain variables. The upcoming addition of the Express Data Plane (XDP) to the kernel will make this approach even more efficient as the programs will get invoked directly from the network driver.
This presentation covers the basics about OpenvSwitch and its components. OpenvSwitch is a Open Source implementation of OpenFlow by the Nicira team.
It also also talks about OpenvSwitch and its role in OpenStack Networking
BPF & Cilium - Turning Linux into a Microservices-aware Operating SystemThomas Graf
Container runtimes cause Linux to return to its original purpose: to serve applications interacting directly with the kernel. At the same time, the Linux kernel is traditionally difficult to change and its development process is full of myths. A new efficient in-kernel programming language called eBPF is changing this and allows everyone to extend existing kernel components or glue them together in new forms without requiring to change the kernel itself.
Deep dive in container service discoveryDocker, Inc.
Service discovery and traffic load-balancing in the container ecosystem relies on different technologies, such as IPVS and iptables, and container orchestrators use different approaches. This talk will present in details how Docker Swarm and Kubernetes achieve this. The talk will continue with a demo showing how applications that are not managed by Kubernetes can take advantage of its native load-balancing. Finally, it will compare these approaches to service-mesh solutions.
netfilter is a framework provided by the Linux kernel that allows various networking-related operations to be implemented in the form of customized handlers.
iptables is a user-space application program that allows a system administrator to configure the tables provided by the Linux kernel firewall (implemented as different netfilter modules) and the chains and rules it stores.
Many systems use iptables/netfilter, Linux's native packet filtering/mangling framework since Linux 2.4, be it home routers or sophisticated cloud network stacks.
In this session, we will talk about the netfilter framework and its facilities, explain how basic filtering and mangling use-cases are implemented using iptables, and introduce some less common but powerful extensions of iptables.
Shmulik Ladkani, Chief Architect at Nsof Networks.
Long time network veteran and kernel geek.
Shmulik started his career at Jungo (acquired by NDS/Cisco) implementing residential gateway software, focusing on embedded Linux, Linux kernel, networking and hardware/software integration.
Some billions of forwarded packets later, Shmulik left his position as Jungo's lead architect and joined Ravello Systems (acquired by Oracle) as tech lead, developing a virtual data center as a cloud-based service, focusing around virtualization systems, network virtualization and SDN.
Recently he co-founded Nsof Networks, where he's been busy architecting network infrastructure as a cloud-based service, gazing at internet routes in astonishment, and playing the chkuku.
Using eBPF for High-Performance Networking in CiliumScyllaDB
The Cilium project is a popular networking solution for Kubernetes, based on eBPF. This talk uses eBPF code and demos to explore the basics of how Cilium makes network connections, and manipulates packets so that they can avoid traversing the kernel's built-in networking stack. You'll see how eBPF enables high-performance networking as well as deep network observability and security.
Intro to Project Calico: a pure layer 3 approach to scale-out networkingPacket
Slide presentation from the April 16th, 2015 Downtown NY Tech Meetup hosted at Control Group and presented by Christopher Liljenstolpe from Project Calico (www.projectcalico.org)
Project Calico is a scale-out networking fabric for bare metal, container, VM, and hybrid environments. Project Calico leverages the same networking techniques used to scale out the Internet to present a highly scaleable, L3 network for those environments without the use of tunnels, overlays, or other complex constructs. We'll also do a demo of a Calico enabled Docker environment, and have plenty of time for q&a during and after.
About Christopher Liljenstolpe
Christopher is the original architect of Project Calico and one of the project's evangelists. In his day job, he's the director of solutions architecture at Metaswitch Networks. Prior to Calico/Metaswitch, he's designed and run some bio-informatics OpenStack clusters, done some SDN architecture work at Big Switch Networks, Run architecture at two large carriers (Telstra - AS1221, and Cable & Wireless/iMCI - AS3561) and been the IP CTO for Alcatel in Asia. He's also run networks in Antarctica (hint, bend radius becomes REALLY important at -50C), and been foolish enough to do a stint as a wg co-chair in the IETF. Occasionally you can have the (mis-)fortune of hearing him speak at conferences and the like.
Deep Dive in Docker Overlay Networks - Laurent Bernaille - Architect, D2SIDocker, Inc.
The Docker network overlay driver relies on several technologies: network namespaces, VXLAN, Netlink and a distributed key-value store. This talk will present each of these mechanisms one by one along with their userland tools and show hands-on how they interact together when setting up an overlay to connect containers. The talk will continue with a demo showing how to build your own simple overlay using these technologies.
Deeper Dive in Docker Overlay NetworksDocker, Inc.
The Docker network overlay driver relies on several technologies: network namespaces, VXLAN, Netlink and a distributed key-value store. This talk will present each of these mechanisms one by one along with their userland tools and show hands-on how they interact together when setting up an overlay to connect containers. The talk will continue with a demo showing how to build your own simple overlay using these technologies. Finally, it will show how we can dynamically distribute IP and MAC information to every hosts in the overlay.
The Docker network overlay driver relies on several technologies: network namespaces, VXLAN, Netlink and a distributed key-value store. This talk will present each of these mechanisms one by one along with their userland tools and show hands-on how they interact together when setting up an overlay to connect containers. The talk will continue with a demo showing how to build your own simple overlay using these technologies. Finally, it will show how we can dynamically distribute IP and MAC information to every hosts in the overlay using BGP EVPN
Docker Meetup: Docker Networking 1.11 with Madhu VenugopalDocker, Inc.
In this talk, Madhu Venugopal will present Docker Networking & Service Discovery features shipped in 1.11 and new Experimental VLAN network drivers introduced in 1.11.
Docker 1.11 Meetup: Networking ShowcaseDocker, Inc.
In this talk, Madhu Venugopal will present Docker Networking & Service Discovery features shipped in 1.11 and new Experimental Vlan network drivers introduced in 1.11.
Docker Meetup: Docker Networking 1.11, by Madhu VenugopalMichelle Antebi
In this talk, Madhu Venugopal will present Docker Networking & Service Discovery features shipped in 1.11 and new Experimental Vlan network drivers introduced in 1.11.
Chris Swan ONUG Academy - Container Networks TutorialCohesive Networks
Slides from Chris Swan's ONUG Academy "Hands-On Container Networks" on May 12, 2015
This hands on session will begin by looking at how Docker modifies a Linux host to enable containers to be connected to a network. It will then go through how applications running in containers can be connected together, and the different options for interconnectivity on a host and between hosts. Finally we will take a look at running network application services inside of containers.
Syllabus
Learn what Docker does to your Linux host on installation.
Connect applications running across multiple containers using configuration metadata and compositing tools.
Understand the different Docker networking modes (host, container, none).
Using Pipework to customise network configuration.
Connecting containers across VMs using Open vSwitch.
Using containers for application network services sush as proxies, load balancers and for TLS termination
Learning Objective 1: Understand how containers relate to the host network, and the consequences that has for services running within containers
Learning Objective 2: Understand the different ways that containers can be networked and internetworked.
Learning Objective 3: Use containers to run network application services.
About the topic:
Containers aren’t a new thing, but the Docker project has made them a hot topic as organisations look at new ways to build, ship and run their applications. This brings new challenges for the network as containers are likely to be ten times as numerous as virtual machines. At the same time there is regulatory pressure to move away from the flat LAN model and deliver greater separation and segregation. This presentation will look at how these two forces are coming together, firstly by examining how containers are networked and some of the new approaches and challenges that come with that. This will be followed by a look at how overlay networks are being deployed to achieve ‘microsegmentation’, and ultimately drive a shift towards application centric networking. Of course these forces will collide, bringing us to contained networks of containers.
Docker Networking Meetup - Intro to Docker NetworkingDhananjay Sampath
We will have two parts to this meetup - in the first part we will cover the lay of the land for how networks are setup, configured and managed for containers and follow that up with a hands-on session.
6:00p-8:00p :
Introduction to Container Networking - DJ
Pipeworks - Jérôme Petazzoni
Docker + OpenvSwitch demo - Sriram
Handson Instructions: https://bitbucket.org/snrism/containet
Q&As: Online
Food: We will have pizzas and sodas.
Technical information: We will provide usb sticks with the required scripts. For this hands-on, only pre-req is to install "vagrant" in your laptop.
Chris Swan's presentation on Docker Networking from Container.Camp in London 12 September 2014
A look at how stock Docker does networking, and how containers can be connected together. Introduction to libchan and pipework projects, and a look at container internetworking using Open vSwitch and kernel VXLAN. Docker can also be used as a place to run layer 4-7 network application services like SSL termination, proxying, load balancing, content caching and intrusion detection.
JDO 2019: Tips and Tricks from Docker Captain - Łukasz LachPROIDEA
This session covers a bunch of tips and tricks for getting the most out of Docker. The tips were inspired by suggestions, blogs, and presentations and everyday challenges encountered by other Docker Captains but also the members of the Docker community. Come and see the unobvious and unexpected in terms of orchestration, image creation and management, also networking and volumes!
Introduction to Docker Networking options. We give in-depth description of the different options with single host examples. See our other presentations for multi-host, IPv6, and CoreOS Flannel descriptions.
Running Kubernetes at scale is challenging and you can often end up in situations where you have to debug complex and unexpected issues. This requires understanding in detail how the different components work and interact with each other. Over the last 3 years, Datadog migrated most of its workloads to Kubernetes and now manages dozens of clusters consisting of thousands of nodes each. During this journey, engineers have debugged complex issues with root causes that were sometimes very surprising. In this talk Laurent and Tabitha will share some of these stories, including a favorite: how a complex interaction between familiar Kubernetes components allowed an OOM-killer invocation to trigger the deletion of a namespace.
DNS is one of the Kubernetes core systems and can quickly become a source of issues when you’re running clusters at scale. For over a year at Datadog, we’ve run Kubernetes clusters with thousands of nodes that host workloads generating tens of thousands of DNS queries per second. It wasn’t easy to build an architecture able to handle this load, and we’ve had our share of problems along the way.
This talk starts with a presentation of how Kubernetes DNS works. It then dives into the challenges we’ve faced, which span a variety of topics related to load, connection tracking, upstream servers, rolling updates, resolver implementations, and performance. We then show how our DNS architecture evolved over time to address or mitigate these problems. Finally, we share our solutions for detecting these problems before they happen—and identifying misbehaving clients.
Kube-proxy enables access to Kubernetes services (virtual IPs backed by pods) by configuring client-side load-balancing on nodes. The first implementation relied on a userspace proxy which was not very performant. The second implementation used iptables and is still the one used in most Kubernetes clusters. Recently, the community introduced an alternative based on IPVS. This talk will start with a description of the different modes and how they work. It will then focus on the IPVS implementation, the improvements it brings, the issues we encountered and how we fixed them as well as the remaining challenges and how they could be addressed. Finally, the talk will present alternative solutions based on eBPF such as Cilium.
The Kubernetes audit logs are a rich source of information: all of the calls made to the API server are stored, along with additional metadata such as usernames, timings, and source IPs. They help to answer questions such as “What is overloading my control plane?” or “Which sequence of events led to this problematic situation?”. These questions are hard to answer otherwise—especially in large clusters. At Datadog, we have been running clusters with 1000+ nodes for more than a year and during that time, the audit logs have proved invaluable.
In this presentation, we will first introduce the audit logs, explain how they are configured, and review the type of data they store. Finally, we will describe in detail several scenarios where they have helped us to diagnose complex problems.
Running large Kubernetes clusters is difficult. Datadog has been running large-scale Kubernetes clusters (thousands of nodes) for more than a year and has learned several lessons the hard way.
Laurent Bernaille examines the challenges Datadog faced during this journey. He dives into problems that arise when you run large clusters—and, crucially, how to address them—by providing detailed examples based on Datadog’s experience across different cloud providers. You’ll explore complex runtime and networking issues: at scale you discover complex issues in low-level components that are very rare but happen regularly when you have a large number of nodes.
Additionally, Laurent provides examples of how to improve the architecture of clusters to increase scalability and performance, both on the control plane and the data plane (communication between pods and ingress traffic). If scale can be hard on the control plane, it’s even harder on tools from the ecosystem, which have rarely been tested on very large clusters. He explains several examples of the tools Datadog uses and how it had to improve them to handle its scale. And you’ll leave with practical advice on how to build a good relationship with the community and start contributing back.
Running large Kubernetes clusters is challenging. At large scales, practitioners need to adapt and tune both their architectures and component configurations in specialized ways.
Our organisation has been running large scale Kubernetes clusters (up to 2000 nodes, and growing) for more than a year, and we have learned several lessons the hard way. This talk will dive into complex runtime and networking issues that occur when running Kubernetes in production at scale. We will provide examples of how to improve the architecture of clusters to increase scalability and performance, both on the control plane and the data plane. Further, tools from the greater ecosystem will be examined, as they are rarely tested within the context of very large clusters.
Finally, the talk will also discuss the mutually beneficial relationship we built with the larger Kubernetes community by providing feedback on the tools and contributing both fixes and improvements upstream.
10 ways to shoot yourself in the foot with kubernetes, #9 will surprise you! ...Laurent Bernaille
Kubernetes is a very powerful and complicated system, and many users don’t understand the underlying systems. Come learn how your users can abuse container runtimes, overwhelm your control plane, and cause outages - it’s actually quite easy!
In the last year, we have containerized hundreds of applications and deployed them in large scale clusters (more than 1000 nodes). The journey was eventful and we learned a lot along the way. We’ll share stories of our ten favorite Kubernetes foot guns, including the dangers of cargo culting, rolling updates gone wrong, the pitfalls of initContainers, and nightmarish daemonset upgrades. The talk will present solutions we adopted to avoid or work around some these problems and will finally show several improvements we plan deploy in the future.
Similar to the Kubecon talk with the same title with a few new incidents.
10 ways to shoot yourself in the foot with kubernetes, #9 will surprise you!Laurent Bernaille
Kubernetes is a very powerful and complicated system, and many users don’t understand the underlying systems. Come learn how your users can abuse container runtimes, overwhelm your control plane, and cause outages - it’s actually quite easy!
In the last year, we have containerized hundreds of applications and deployed them in large scale clusters (more than 1000 nodes). The journey was eventful and we learned a lot along the way. We’ll share stories of our ten favorite Kubernetes foot guns, including the dangers of cargo culting, rolling updates gone wrong, the pitfalls of initContainers, and nightmarish daemonset upgrades. The talk will present solutions we adopted to avoid or work around some these problems and will finally show several improvements we plan deploy in the future.
Running large Kubernetes clusters is challenging. This talk focus on how you can optimize your network setup in clusters with 1000-2000 nodes. It discusses standard ingresses solutions and their drawbacks as well as potential solutions
This presentation describes the challenges we faced building, scaling and operating a Kubernetes cluster of more than 1000 nodes to host the Datadog applications
Serverless architectures are promising and will play an important role in the coming years but the ecosystem around serverless is still pretty young. We have been operating Lambda based applications for about a year and faced several challenges. In this presentation we share these challenges and propose some solutions to work around them.
Most tools to recognize the application associated with network connections use well-known signatures as basis for their classification. This approach is very effective in enterprise and campus networks to pinpoint forbidden applications (peer to peer, for instance) or security threats. However, it is easy to use encryption to evade these mechanisms. In particular, Secure Sockets Layer (SSL) libraries such as OpenSSL are widely available and can easily be used to encrypt any type of traffic. In this paper, we propose a method to detect applications in SSL encrypted connections. Our method uses only the size of the first few packets of an SSL connection to recognize the application, which enables an early classification. We test our method on packet traces collected on two campus networks and on manually-encrypted traces. Our results show that we are able to recognize the application in an SSL connection with more than 85% accuracy.
The automatic detection of applications associated with network traffic is an essential step for network security and traffic engineering. Unfortunately, simple port-based classification methods are not always efficient and systematic analysis of packet payloads is too slow. Most recent research proposals use flow statistics to classify traffic flows once they are finished, which limit their applicability for online classification. In this paper, we evaluate the feasibility of application identification at the beginning of a TCP connection. Based on an analysis of packet traces collected on eight different networks, we find that it is possible to distinguish the behavior of an application from the observation of the size and the direction of the first few packets of the TCP connection. We apply three techniques to cluster TCP connections: K-Means, Gaussian Mixture Model and spectral clustering. Resulting clusters are used together with assignment and labeling heuristics to design classifiers. We evaluate these classifiers on different packet traces. Our results show that the first four packets of a TCP connection are sufficient to classify known applications with an accuracy over 90% and to identify new applications as unknown with a probability of 60%.
0x01 - Newton's Third Law: Static vs. Dynamic AbusersOWASP Beja
f you offer a service on the web, odds are that someone will abuse it. Be it an API, a SaaS, a PaaS, or even a static website, someone somewhere will try to figure out a way to use it to their own needs. In this talk we'll compare measures that are effective against static attackers and how to battle a dynamic attacker who adapts to your counter-measures.
About the Speaker
===============
Diogo Sousa, Engineering Manager @ Canonical
An opinionated individual with an interest in cryptography and its intersection with secure software development.
Sharpen existing tools or get a new toolbox? Contemporary cluster initiatives...Orkestra
UIIN Conference, Madrid, 27-29 May 2024
James Wilson, Orkestra and Deusto Business School
Emily Wise, Lund University
Madeline Smith, The Glasgow School of Art
Have you ever wondered how search works while visiting an e-commerce site, internal website, or searching through other types of online resources? Look no further than this informative session on the ways that taxonomies help end-users navigate the internet! Hear from taxonomists and other information professionals who have first-hand experience creating and working with taxonomies that aid in navigation, search, and discovery across a range of disciplines.
This presentation, created by Syed Faiz ul Hassan, explores the profound influence of media on public perception and behavior. It delves into the evolution of media from oral traditions to modern digital and social media platforms. Key topics include the role of media in information propagation, socialization, crisis awareness, globalization, and education. The presentation also examines media influence through agenda setting, propaganda, and manipulative techniques used by advertisers and marketers. Furthermore, it highlights the impact of surveillance enabled by media technologies on personal behavior and preferences. Through this comprehensive overview, the presentation aims to shed light on how media shapes collective consciousness and public opinion.
This presentation by Morris Kleiner (University of Minnesota), was made during the discussion “Competition and Regulation in Professions and Occupations” held at the Working Party No. 2 on Competition and Regulation on 10 June 2024. More papers and presentations on the topic can be found out at oe.cd/crps.
This presentation was uploaded with the author’s consent.
Acorn Recovery: Restore IT infra within minutesIP ServerOne
Introducing Acorn Recovery as a Service, a simple, fast, and secure managed disaster recovery (DRaaS) by IP ServerOne. A DR solution that helps restore your IT infra within minutes.
4. Environment
Bastion
Public subnets
NAT
GW
Public subnets Public subnets
consulagent
(UI)
consul0
docker0 docker1 docker2
consul1 consul2
10.200.0.0/16
dockerd -H fd:// --cluster-store=consul://consul0:8500 --cluster-advertise=eth0:2376
10.200.128.0/17
What is in consul? Not much for now just metadata tree
5. docker0:~$ docker network create --driver overlay
--internal
--subnet 192.168.0.0/24 dockercon
c4305b67cda46c2ed96ef797e37aed14501944a1fe0096dacd1ddd8e05341381
docker1:~$ docker network ls
NETWORK ID NAME DRIVER SCOPE
bec777b6c1f1 bridge bridge local
c4305b67cda4 dockercon overlay global
3a4e16893b16 host host local
c17c1808fb08 none null local
Let's create an overlay network
6. docker0:~$ docker run -d --ip 192.168.0.100 --net dockercon --name C0 debian sleep 3600
docker1:~$ docker run --net dockercon debian ping 192.168.0.100
PING 192.168.0.100 (192.168.0.100): 56 data bytes
64 bytes from 192.168.0.100: seq=0 ttl=64 time=1.153 ms
64 bytes from 192.168.0.100: seq=1 ttl=64 time=0.807 ms
docker1:~$ ping 192.168.0.100
PING 192.168.0.100 (192.168.0.100) 56(84) bytes of data.
^C--- 192.168.0.100 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3024ms
Does it work?
7. Overlay
What did we build?
consul
docker0
C0
eth0
docker1
C1
eth0PING
192.168.0.100 192.168.0.Y
10.200.128.100 10.200.129.100
9. docker0:~$ docker exec C0 ip addr show
58: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP
inet 192.168.0.100/24 scope global eth0
docker0:~$ docker exec C0 ip -details link show dev eth0
58: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group
default
veth
How does it work? Let's look inside containers
11. docker0:~$ ip link show >> Nothing, it must be in another Namespace
docker0:~$ sudo ip netns ls
8-c4305b67cd
docker0:~$ docker network inspect dockercon -f {{.Id}}
c4305b67cda46c2ed96ef797e37aed14501944a1fe0096dacd1ddd8e05341381
docker0:~$ overns=8-c4305b67cd
docker0:~$ sudo ip netns exec $overns ip -d link show
2: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default
bridge
62: vxlan1: <..> mtu 1450 qdisc noqueue master br0 state UNKNOWN mode DEFAULT group default
vxlan id 256 srcport 10240 65535 dstport 4789 proxy l2miss l3miss ageing 300
59: veth2: <...> mtu 1450 qdisc noqueue master br0 state UP mode DEFAULT group default
Where is the other end of the veth?
13. • Tunneling technology over UDP (L2 in UDP)
• Developed for cloud SDN to create multitenancy
• Without the need for L2 connectivity
• Without the normal VLAN limit (4096 VLAN Ids)
• In Linux
• Started with Open vSwitch
• Native with Kernel >= 3.7 and >=3.16 for Namespace support
What is VXLAN?
Outer IP packet
UDP
dst: 4789
VXLAN
Header
Original L2
14. docker0:~$ sudo tcpdump -nn -i eth0 "port 4789"
docker1:~$ docker run --net dockercon debian ping 192.168.0.100
PING 192.168.0.100 (192.168.0.100): 56 data bytes
64 bytes from 192.168.0.100: seq=0 ttl=64 time=1.153 ms
64 bytes from 192.168.0.100: seq=1 ttl=64 time=0.807 ms
docker0:~$
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:05:04.041366 IP 10.200.129.98.34922 > 10.200.128.130.4789: VXLAN, flags [I] (0x08), vni 256
IP 192.168.0.2 > 192.168.0.100: ICMP echo request, id 256, seq 62903, length 64
14:05:04.041429 IP 10.200.128.130.59164 > 10.200.129.98.4789: VXLAN, flags [I] (0x08), vni 256
IP 192.168.0.100 > 192.168.0.2: ICMP echo reply, id 256, seq 62903, length 64
Let's verify this
15. Full connectivity with VXLAN
consul
docker0
eth0
192.168.0.100
C0 Namespace
br0
vxlanveth
eth0
docker1
C1 Namespace
br0
vxlanveth
eth0PING
eth0
192.168.0.Y
10.200.128.100 10.200.129.100
IP
src: 10.200.129.X
dst: 10.120.128.X
UDP
src: X
dst: 4789
VXLAN
VNI
Original L2
src: 192.168.0.Y
dst: 192.168.0.100
16. docker0:~$ sudo ip netns exec $overns ip neighbor show
192.168.0.2 dev vxlan1 lladdr 02:42:c0:a8:00:02 PERMANENT
docker0:~$ sudo ip netns exec $overns bridge fdb show br br0
02:42:c0:a8:00:02 dev vxlan1 dst 10.200.129.98 self permanent
docker1:~$ docker run -d --ip 192.168.0.200 --net dockercon --name C1 debian sleep 3600
docker0:~$ sudo ip netns exec $overns ip neighbor show
192.168.0.2 dev vxlan1 lladdr 02:42:c0:a8:00:02 PERMANENT
192.168.0.20 dev vxlan1 lladdr 02:42:c0:a8:00:14 PERMANENT
docker0:~$ sudo ip netns exec $overns ip neighbor show
02:42:c0:a8:00:02 dev vxlan1 dst 10.200.129.98 self permanent
02:42:c0:a8:00:14 dev vxlan1 dst 10.200.129.98 self permanent
How does docker0 know about containers on C1
17. VXLAN L2/L3 resolution - Option 1: Multicast
vxlan vxlan
vxlan
Multicast
239.x.x.x
ARP: Who has 192.168.0.2?
L2 discovery: where is 02:42:c0:a8:00:02 ?
Use a multicast group to send traffic for unknown L3/L2 addresses
• PROS: simple and efficient
• CONS: Multicast connectivity not always available (on public clouds for instance)
18. vxlan vxlan
Remote IP: point-to-point
Send everything to remote IP
VXLAN L2/L3 resolution - Option 2: Point-to-point
Configure a remote IP address where to send traffic for unknown addresses
• PROS: simple, not need for multicast, very good for two hosts
• CONS: difficult to manage with more than 2 hosts
19. vxlan vxlan
daemon daemon
Manual (with a daemon modifying ARP/FDB)
ARP: Do you know 192.168.0.2?
L2: where is 02:42:c0:a8:00:02 ?
vxlan
daemon
VXLAN L2/L3 resolution - Option 3: Manual
Do nothing, provide ARP / FDB information from outside
• PROS: very flexible
• CONS: requires a daemon and a centralized database of addresses
25. ip netns add overns
ip netns exec overns ip link add dev br0 type bridge
ip netns exec overns ip addr add dev br0 192.168.0.1/24
ip link add dev vxlan1 type vxlan id 42 proxy learning l2miss l3miss dstport 4789
ip link set vxlan1 netns overns
ip netns exec overns ip link set vxlan1 master br0
ip netns exec overns ip link set vxlan1 up
ip netns exec overns ip link set br0 up
Create overlay components
create overlay NS
create bridge in NS
create VXLAN interface
move it to NS
add it to bridge
bring all interfaces up
27. docker0
docker run -d --net=none --name=demo debian sleep 3600
ctn_ns_path=$(docker inspect --format="{{ .NetworkSettings.SandboxKey}}" demo)
ctn_ns=${ctn_ns_path##*/}
ip link add dev veth1 mtu 1450 type veth peer name veth2 mtu 1450
ip link set dev veth1 netns overns
ip netns exec overns ip link set veth1 master br0
ip netns exec overns ip link set veth1 up
ip link set dev veth2 netns $ctn_ns
ip netns exec $ctn_ns ip link set dev veth2 name eth0 address 02:42:c0:a8:00:02
ip netns exec $ctn_ns ip addr add dev eth0 192.168.0.2
ip netns exec $ctn_ns ip link set dev eth0 up
docker1
Same with 192.168.0.2 / 02:42:c0:a8:00:02
Create containers and connect them
Create container without net
Create veth
Send veth1 to overlay NS
Attach it to overlay bridge
Send veth2 to container
Rename & Configure
Get NS for container
29. Does it ping?
docker0:~$ docker exec -it demo ping 192.168.0.3
PING 192.168.0.3 (192.168.0.3): 56 data bytes
92 bytes from 192.168.0.2: Destination Host Unreachable
docker0:~$ sudo ip netns exec overns ip neighbor show
docker0:~$ sudo ip netns exec overns ip neighbor add 192.168.0.3 lladdr 02:42:c0:a8:00:03 dev vxlan1
docker0:~$ sudo ip netns exec overns bridge fdb add 02:42:c0:a8:00:03 dev vxlan1 self dst 10.200.129.98
vni 42 port 4789
docker0:~$ docker exec -it demo ping 192.168.0.3
docker1: Look at ARP / FDB tables
=> FDB has "learned" about 02:42:c0:a8:00:02 but ARP does not know about 192.168.0.2
docker1:~$ sudo ip netns exec overns ip neighbor add 192.168.0.2 lladdr 02:42:c0:a8:00:02 dev vxlan1
32. Recreate overlay without static entries
docker0:~$ sudo ip netns delete overns
Recreate overlay Namespace, bridge and VXLAN
Connect demo container to overlay NS bridge
docker0:~$ sudo ip netns exec overns ip link show
docker0:~$ sudo ip netns exec overns ip neighbor show
docker0:~$ sudo ip netns exec overns bridge fdb show
33. Catching network events: NETLINK
• Kernel interface for communication between Kernel and userspace
• Designed to transfer networking info (used by iproute2)
• Several protocols
• NETLINK_ROUTE
• NETLINK_FIREWALL
• Several notification types, for NETLINK_ROUTE for instance:
• LINK
• NEIGHBOR
• Many events
• LINK: NEWLINK, GETLINK
• NEIGHBOR: GETNEIGH <= information on ARP, L2 discovery queries
35. Catching L2/L3 misses
docker0-1:~$ sudo ip netns exec overns python/l2l3miss.py
docker0-2:~$ docker exec -it demo ping 192.168.0.3
docker0-1:~$
INFO:root:L3Miss: Who has IP: 192.168.0.3?
Add MAC entry for other container
docker0-1:~$
INFO:root:L2Miss: Who has MAC: 02:42:c0:a8:00:03?
Add FDB entry >> It pings
36. Storing MAC, FDB info in Consul
Recreate overlay, attach container
docker0:~$ sudo python/arpd-consul.py
docker0:~$ docker exec -it demo ping 192.168.0.3
INFO Starting new HTTP connection (1): consul1
INFO L3Miss on vxlan1: Who has IP: 192.168.0.3?
INFO Populating ARP table from Consul: IP 192.168.0.3 is 02:42:c0:a8:00:03
INFO L2Miss on vxlan1: Who has Mac Address: 02:42:c0:a8:00:03?
INFO Populating FIB table from Consul: MAC 02:42:c0:a8:00:03 is on host 10.200.129.98
38. Quick summary on VLXAN options
ip link add dev vxlan1 type vxlan
id 42
proxy
learning
l2miss l3miss
dstport 4789
VNI: to multiplex multiple VXLANs between hosts
Allows for multiple isolated overlay
ARP proxy: interface answers ARP queries
This is how our ARP queries got answered
Learn MAC addresses from frames for FDB
UDP port for tunneling
Generate Neighbor events on Netlink
39. A few tricky implementation details
• ip netns commands do not work by default with docker net NS
• Workaround: nsenter or symlink /var/run/docker/netns to /var/run/netns
• vxlan must be created in host Network NS and moved in the overlay NS
• Keeps a link with the host eth0 interface
• Otherwise vxlan will not be able to go outside the host
• l2miss / l3miss
• By default GETNEIGH events are not sent on Netlink
• Alternative: use /proc/sys/net/ipv4/neigh/eth0/app_solicit
• Consul python script runs in host network namespace
• If it runs in the overlay namespace it can not access consul
• The script binds the Netlink socket inside the overlay Namespace