IP Routing, AWS, and Docker

•

4 likes•1,473 views

Project Quadra is a PaaS built on Docker that can run on EC2, OpenStack, or bare metal. It uses GRE tunnels over an overlay network to extend a company's internal network to EC2 instances, but there are some MTU issues to address. BGP is used with Quagga for dynamic routing between the overlay network and EC2 instances, with route filtering and policies configured. Docker networking uses bridges, veth interfaces, and NAT rules to connect containers.

Project Quadra
- PaaS built on Docker
- Python
- Run on EC2, OpenStack, Bare Metal

Amazon EC2
- VPC
- direct connect
- extends our internal network
- secondary ip address limitations

Overlay network
- GRE
- MTU issues
- EC2 instances with jumbo frame support
CC2, C3, R3, CG1, CR1
G2, HS1, HI1, I2, M3

GRE configuration
#ip tunnel del gre1
ip tunnel add gre1 mode gre remote 67.215.78.24 local 10.70.74.187 ttl 255
ip tunnel add gre1 mode gre remote 10.1.1.1 local 10.70.74.187 ttl 255
ip link set gre1 up
ip addr add 10.71.0.2/30 dev gre1
echo 200 quadranet >> /etc/iproute2/rt_tables
ip rule add from 10.71.1.0/24 table quadranet
ip route add default via 10.71.0.1 dev gre1 table quadranet

BGP: Border Gateway Protocol
● dynamic routing protocol
● core internet routing protocol
● quagga, bird, exabgp

Quagga configuration
router bgp 65535
bgp router-id 10.70.74.187
redistribute kernel
timers bgp 3 20
neighbor 10.71.0.1 remote-as 36692
neighbor 10.71.0.1 prefix-list DOCKER out
neighbor 10.71.0.1 next-hop-self
neighbor 10.71.0.1 route-map anycast-out out
!
ip prefix-list DOCKER seq 5 permit 10.71.1.0/24 le 32
ip prefix-list DOCKER seq 10 deny any
!
route-map anycast-out permit 10
set local-preference 100
set origin igp

Docker Networking
- docker0 bridge
- veth interfaces
- 172.17.0.0/16
- iptables nat rules

$Linux Network Namespaces # Add the address to the containers loopback interface ip netns exec "${NSPID}" ip addr add "${GUEST_IPADDR}" dev lo # Add a route to the host's table for the guest's new IP ip route add "${GUEST_IPADDR}/32" via "${DOCKER_IPADDR}"$

Whats next?
● OSPF?
● fully meshed network
● automated GRE tunnels
● public ip addresses

Running Kubernetes at scale is challenging and you can often end up in situations where you have to debug complex and unexpected issues. This requires understanding in detail how the different components work and interact with each other. Over the last 3 years, Datadog migrated most of its workloads to Kubernetes and now manages dozens of clusters consisting of thousands of nodes each. During this journey, engineers have debugged complex issues with root causes that were sometimes very surprising. In this talk Laurent and Tabitha will share some of these stories, including a favorite: how a complex interaction between familiar Kubernetes components allowed an OOM-killer invocation to trigger the deletion of a namespace.

[오픈소스컨설팅] Linux Network Troubleshooting

Open Source Consulting

Building Docker images with Puppet

Nick Jones

Kubernetes has the concept of resource requests and limits. Pods get scheduled on the nodes based on their requests and optionally limited in how much of the resource they can consume. Understanding and optimizing resource requests/limits is crucial both for reducing resource "slack" and ensuring application performance/low-latency. This talk shows our approach to monitoring and optimizing Kubernetes resources for 80+ clusters to achieve cost-efficiency and reducing impact for latency-critical applications. All shown tools are Open Source and can be applied to most Kubernetes deployments.

Evolution of kube-proxy (Brussels, Fosdem 2020)

Laurent Bernaille

Kube-proxy enables access to Kubernetes services (virtual IPs backed by pods) by configuring client-side load-balancing on nodes. The first implementation relied on a userspace proxy which was not very performant. The second implementation used iptables and is still the one used in most Kubernetes clusters. Recently, the community introduced an alternative based on IPVS. This talk will start with a description of the different modes and how they work. It will then focus on the IPVS implementation, the improvements it brings, the issues we encountered and how we fixed them as well as the remaining challenges and how they could be addressed. Finally, the talk will present alternative solutions based on eBPF such as Cilium.

[OpenInfra Days Korea 2018] Day 2 - E6 - OpenInfra monitoring with Prometheus

OpenStack Korea Community

Kubernetes the Very Hard Way. Lisa Portland 2019

Laurent Bernaille

Running large Kubernetes clusters is challenging. At large scales, practitioners need to adapt and tune both their architectures and component configurations in specialized ways. Our organisation has been running large scale Kubernetes clusters (up to 2000 nodes, and growing) for more than a year, and we have learned several lessons the hard way. This talk will dive into complex runtime and networking issues that occur when running Kubernetes in production at scale. We will provide examples of how to improve the architecture of clusters to increase scalability and performance, both on the control plane and the data plane. Further, tools from the greater ecosystem will be examined, as they are rarely tested within the context of very large clusters. Finally, the talk will also discuss the mutually beneficial relationship we built with the larger Kubernetes community by providing feedback on the tools and contributing both fixes and improvements upstream.

KubeCon EU 2016: Using Traffic Control to Test Apps in Kubernetes

KubeAcademy

Testing applications is important, as shown by the rise of continuous integration and automated testing. In this talk, I will focus on one area of testing that is difficult to automate: poor network connectivity. Developers usually work within reliable networking conditions so they might not notice issues that arise in other networking conditions. I will give examples of software that would benefit from test scenarios with varying connectivity. I will explain how traffic control on Linux can help to simulate various network connectivity. Finally, I will run a demo showing how an application running in Kubernetes behaves when changing network parameters. Sched Link: http://sched.co/6Bb3

What's new in Ansible 2.0

Allan Denot

Kubernetes Networking

CJ Cullen

Kubernetes the Very Hard Way. Velocity Berlin 2019

Laurent Bernaille

Running large Kubernetes clusters is difficult. Datadog has been running large-scale Kubernetes clusters (thousands of nodes) for more than a year and has learned several lessons the hard way. Laurent Bernaille examines the challenges Datadog faced during this journey. He dives into problems that arise when you run large clusters—and, crucially, how to address them—by providing detailed examples based on Datadog’s experience across different cloud providers. You’ll explore complex runtime and networking issues: at scale you discover complex issues in low-level components that are very rare but happen regularly when you have a large number of nodes. Additionally, Laurent provides examples of how to improve the architecture of clusters to increase scalability and performance, both on the control plane and the data plane (communication between pods and ingress traffic). If scale can be hard on the control plane, it’s even harder on tools from the ecosystem, which have rarely been tested on very large clusters. He explains several examples of the tools Datadog uses and how it had to improve them to handle its scale. And you’ll leave with practical advice on how to build a good relationship with the community and start contributing back.

CoreOS @Codetalks Hamburg

Timo Derstappen

What makes AWS invincible? from JAWS Days 2014Emma Haruka Iwao

Build Your Own CaaS (Container as a Service)

HungWei Chiu

Small, Simple, and Secure: Alpine Linux under the Microscope

Docker, Inc.

Ensuring Kubernetes Cost Efficiency across (many) Clusters - DevOps Gathering...

Henning Jacobs

Talk held at DevOps Gathering 2019 in Bochum on 2019-03-13. Abstract: This talk will address one of the most common challenges of organizations adopting Kubernetes on a medium to large scale: how to keep cloud costs under control without babysitting each and every deployment and cluster configuration? How to operate 80+ Kubernetes clusters in a cost-efficient way for 200+ autonomous development teams? This talk provides insights on how Zalando approaches this problem with central cost optimizations (e.g. Spot), cost monitoring/alerting, active measures to reduce resource slack, and automated cluster housekeeping. We will focus on how to ingrain cost efficiency in tooling and developer workflows while balancing rigid cost control with developer convenience and without impacting availability or performance. We will show our use case running Kubernetes on AWS, but all shown tools are open source and can be applied to most other infrastructure environments.

LISA18: Hidden Linux Metrics with Prometheus eBPF Exporter

Ivan Babrou

Presented at LISA18: https://www.usenix.org/conference/lisa18/presentation/huynh While there are plenty of readily available metrics for monitoring Linux kernel, many gems remain hidden. With the help of recent developments in eBPF, it is now possible to run safe programs in the kernel to collect arbitrary information with little to no overhead. A few examples include: * Disk latency and io size histograms * Run queue (scheduler) latency * Page cache efficiency * Directory cache efficiency * LLC (aka L3 cache) efficiency * Kernel timer counters * System-wide TCP retransmits Practically any event from "perf list" output and any kernel function can be traced, analyzed and turned into a Prometheus metric with almost arbitrary labels attached to it. If you are already familiar with BCC tools, you may think if ebpf_exporter as bcc tools turned into prometheus metrics. In this tutorial we’ll go over eBPF basics, how to write programs and get insights into a running system.

Packet Walk(s) In Kubernetes

Don Jayakody

When it comes to networking inside Kubernetes, selecting the correct networking solution may be one of the most important decisions you may face. This is especially true if you are trying to run a Kubernetes cluster in production. Therefore it's beneficial to have a good understanding of different CNI options out there and most importantly how these networking options are different from each other. This presentation goes over packet by packet-level details of how the network plumbing is happening with different CNI plugins including, Flannel, Calico & Cilium.

Load Balancing 101

HungWei Chiu

ATO Linux Performance 2018

Brendan Gregg

Talk by Brendan Gregg for All Things Open 2018. "At over one thousand code commits per week, it's hard to keep up with Linux developments. This keynote will summarize recent Linux performance features, for a wide audience: the KPTI patches for Meltdown, eBPF for performance observability and the new open source tools that use it, Kyber for disk I/O sc heduling, BBR for TCP congestion control, and more. This is about exposure: knowing what exists, so you can learn and use it later when needed. Get the most out of your systems with the latest Linux kernels and exciting features."

KubeCon EU 2016: Getting the Jobs Done With Kubernetes

KubeAcademy

When you hear words such as Kubernetes or OpenShift you immediately start thinking about long running processes you can easily scale at will. However, Kubernetes includes a lesser known feature which allows you to run pretty much anything from simple tasks up to highly-complicated ones. During this presentation, the author of the Job resource in Kubernetes will guide you through several techniques for performing anything ranging from simple Pi calculations to rendering a movie. No matter if you're a data scientist running large scale calculations across several data centers or a hobby programmer running simple day-to-day tasks, this presentation is to teach you how to efficiently use Kubernetes Jobs on their own or as the building blocks of something bigger. This presentation will feature a number of live demos to help illustrate the various ways that you can put Jobs to work. Don’t miss out on learning about one of the coolest features of Kubernetes! Sched Link: http://sched.co/6BUw

Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latency

Henning Jacobs

Talk given at JAX DevOps London on 2019-05-15 Kubernetes has the concept of resource requests and limits. Pods get scheduled on the nodes based on their requests and optionally limited in how much of the resource they can consume. Understanding and optimizing resource requests/limits is crucial both for reducing resource "slack" and ensuring application performance/low-latency. This talk shows our approach to monitoring and optimizing Kubernetes resources for 90+ clusters to achieve cost-efficiency and reducing impact for latency-critical applications. All shown tools are open source and can be applied to most Kubernetes deployments. Topics covered in the talk include: understanding resource requests and limits, cgroups and CFS quota behavior, contributing factors to cluster costs (in public clouds), and best practices for managing Kubernetes resources.

Kubernetes at Datadog Scale - Ara Pulido

PROIDEA

Kubernetes is currently the most popular container orchestration platform, and while many organizations are migrating their workloads to it, Kubernetes is still relatively immature. New corner cases, errors, and quirks are regularly discovered as users push the boundaries of size and scale. Joining Datadog allowed me to discover Kubernetes at a larger scale. In this session I will share what I’ve learned, how it has changed my perspective about the Kubernetes ecosystem, and how you can apply those learnings when scaling your cluster to hundreds or thousands of nodes.

Deeper Dive in Docker Overlay Networks

Docker, Inc.

The Docker network overlay driver relies on several technologies: network namespaces, VXLAN, Netlink and a distributed key-value store. This talk will present each of these mechanisms one by one along with their userland tools and show hands-on how they interact together when setting up an overlay to connect containers. The talk will continue with a demo showing how to build your own simple overlay using these technologies. Finally, it will show how we can dynamically distribute IP and MAC information to every hosts in the overlay.

Deeper dive in Docker Overlay Networks

Laurent Bernaille

What's hot

Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...

Henning Jacobs

Evolution of kube-proxy (Brussels, Fosdem 2020)

Laurent Bernaille

[OpenInfra Days Korea 2018] Day 2 - E6 - OpenInfra monitoring with Prometheus

OpenStack Korea Community

Kubernetes the Very Hard Way. Lisa Portland 2019

Laurent Bernaille

KubeCon EU 2016: Using Traffic Control to Test Apps in Kubernetes

KubeAcademy

What's new in Ansible 2.0

Allan Denot

Kubernetes Networking

CJ Cullen

Kubernetes the Very Hard Way. Velocity Berlin 2019

Laurent Bernaille

CoreOS @Codetalks Hamburg

Timo Derstappen

What makes AWS invincible? from JAWS Days 2014Emma Haruka Iwao

Build Your Own CaaS (Container as a Service)

HungWei Chiu

Small, Simple, and Secure: Alpine Linux under the Microscope

Docker, Inc.

Ensuring Kubernetes Cost Efficiency across (many) Clusters - DevOps Gathering...

Henning Jacobs

LISA18: Hidden Linux Metrics with Prometheus eBPF Exporter

Ivan Babrou

Packet Walk(s) In Kubernetes

Don Jayakody

Load Balancing 101

HungWei Chiu

ATO Linux Performance 2018

Brendan Gregg

KubeCon EU 2016: Getting the Jobs Done With Kubernetes

KubeAcademy

Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latency

Henning Jacobs

Kubernetes at Datadog Scale - Ara Pulido

PROIDEA

What's hot (20)

Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...

Evolution of kube-proxy (Brussels, Fosdem 2020)

[OpenInfra Days Korea 2018] Day 2 - E6 - OpenInfra monitoring with Prometheus

Kubernetes the Very Hard Way. Lisa Portland 2019

KubeCon EU 2016: Using Traffic Control to Test Apps in Kubernetes

What's new in Ansible 2.0

Kubernetes Networking

Kubernetes the Very Hard Way. Velocity Berlin 2019

CoreOS @Codetalks Hamburg

What makes AWS invincible? from JAWS Days 2014

Build Your Own CaaS (Container as a Service)

Small, Simple, and Secure: Alpine Linux under the Microscope

Ensuring Kubernetes Cost Efficiency across (many) Clusters - DevOps Gathering...

LISA18: Hidden Linux Metrics with Prometheus eBPF Exporter

Packet Walk(s) In Kubernetes

Load Balancing 101

ATO Linux Performance 2018

KubeCon EU 2016: Getting the Jobs Done With Kubernetes

Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latency

Kubernetes at Datadog Scale - Ara Pulido

Similar to IP Routing, AWS, and Docker

Deeper Dive in Docker Overlay Networks

Docker, Inc.

Deeper dive in Docker Overlay Networks

Laurent Bernaille

Docker Setting for Static IP allocation

Ji-Woong Choi

Multicloud connectivity using OpenNHRP

Bob Melander

Implementing an IPv6 Enabled Environment for a Public Cloud Tenant

Shixiong Shang

Deep Dive in Docker Overlay Networks

Laurent Bernaille

Kubernetes at Datadog Scale

Docker, Inc.

Ara Pulido, Datadog - Container technologies, although not new, have increased their popularity in the past few years, with container orchestrators allowing companies around the world to adopt these technologies to help them ship and scale microservices with precision and velocity. Kubernetes is currently the most popular container orchestration platform, and while many organizations are migrating their workloads to it, Kubernetes is still relatively immature. New corner cases, errors, and quirks are regularly discovered as users push the boundaries of size and scale. When Datadog adopted Kubernetes we discovered some of these boundaries the hard way, and we continuously challenge and modify our infrastructure decisions in order to fit our use case. Join me in this talk for our story on what we learned while we scaled our Kubernetes clusters, the contributions to Kubernetes we made along the way, and how you can apply those learnings when growing your Kubernetes clusters from a handful to hundreds or thousands of nodes.

Deep Dive in Docker Overlay Networks - Laurent Bernaille - Architect, D2SI

Docker, Inc.

Osnug meetup-tungsten fabric - overview.pptx

M.Qasim Arham

Linux Networking Commands

tmavroidis

Docker SDN (software-defined-networking) JUG

Piotr Kieszczyński

Kubernetes internals (Kubernetes 해부하기)

DongHyeon Kim

Konfigurasi Server Gateway dengan fitur PROXY, WEBSERVER dan DHCP

Walid Umar