Downloaded 18 times
Uploaded byOntico
Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
This document discusses using traffic control to test microservices applications running in containers. It describes using tools like Kubernetes, Weave Scope, and netem to configure network emulation scenarios like bandwidth limiting, latency injection and packet dropping. A demo application is shown running in Kubernetes pods with Weave Scope visualizing network traffic and a traffic control plugin modifying network behavior for testing purposes. Advanced filtering options using eBPF and custom plugins are proposed to define more complex traffic classes and collect test statistics.
More Related Content
![[OpenInfra Days Korea 2018] Day 2 - E5-1: "Invited Talk: Kubicorn - Building ...](https://cdn.slidesharecdn.com/ss_thumbnails/e60955buildingsimplekubernetes-180705043459-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
- 1. In containers Alban Crequy Testingapplications with traffic control HighLoad++ - November 2016https://goo.gl/knaz6m
- 2. Alban Crequy ∘ Workedon traffic control for multimedia applications in cars (tcmmd) ∘ Worked on the container run-time rkt ∘ Working on Weave Scope https://github.com/alban
- 3. Berlin-based software companybuilding foundational Linux technologies Some examples of what we worked on... OSTreegit for operating system binaries
- 4.
- 5. Find out moreabout us… Blog: https://kinvolk.io/blog Github: https://github.com/kinvolk Twitter: https://twitter.com/kinvolkio Email: hello@kinvolk.io
- 6. ∘ What istraffic control and how does it work on Linux ∘ How it can be used for testing a microservices application ∘ Demo ∘ With Minikube, Kubernetes, Weave Scope, Weave Socks Plan
- 7. What is trafficcontrol? How does it work on Linux?
- 8. Traffic control, why? webserver client client client THE INTERNET ∘ fair distribution of bandwidth ∘ reserve bandwidth to specific applications ∘ avoid bufferbloat
- 9. ∘ Network schedulingalgorithm ∘ which packet to emit next? ∘ when? ∘ Configurable at run-time: ∘ /sbin/tc ∘ Netlink ∘ Default on new network interfaces: sysctl net.core.default_qdisc Queuing disciplines (qdisc) eth0 THE INTERNETqdisc
- 10. Stochastic Fairness Queueing (sfq) eth0 THEINTERNET FIFO n FIFO 1 FIFO 0 ... round robin
- 11. Traffic control fortesting?
- 12. Developers & Internet Developersusually have good Internet connection - Low latency - High bandwidth - Not much packet loss or packet corruption Difficult to see problems
- 13. Network emulator (netem) eth0 THEINTERNETnetem bandwidth latency packet loss corrupt ...
- 14. Testing with containers container1 container 2 eth0eth0 Testing framework configure “netem” qdiscs: bandwidth, latency, packet drop...
- 15.
- 16.
- 17.
- 18.
- 19. Pods Group of container(s)running together, in the same context Kubernetes node 1 Kubernetes node 2 pod container container pod container pod container pod container Each pod has - Its own network namespace - Its own IP address
- 20. Replica sets - Controlthe number of “Pods” replicas running - Deploy pods Kubernetes node 1 Kubernetes node 2 pod catalogue pod front-end pod orders In this example: - 2x front-end - 1x catalogue - 1x orders pod front-end
- 21. Replica set “catalogue” Services Directing thetraffic to “Pods” - Cluster IP - DNAT catalogue1 catalogue2 catalogue3 Service “catalogue” front-end
- 22. Testing with trafficcontrol in Kubernetes Kubernetes node 1 pod pod Kubernetes node 2 pod pod tc tc controls ∘ Latency ∘ Bandwidth ∘ Packet drop ∘ configure network simulator ∘ play scenarios
- 23.
- 24.
- 25. Testing with WeaveScope Kubernetes node 1 tc Scope Probe pod pod pod pod Kubernetes Node 2 tc Scope Probe pod pod pod pod Scope App
- 26. <Demo> Reproduce the demoyourself: https://github.com/kinvolk/demo
- 47. </Demo> Reproduce the demoyourself: https://github.com/kinvolk/demo
- 48. Testing framework for webapps Selenium
- 49. Plugins in Scope tcplugin Scope Probe Scope App report reportcontrol control - Unix socket in /var/run/scope/plugins/ - Protocols: report and control - Write your own plugins for your testing needs
- 51. Testing more complex scenarios (my“wishlist”)
- 52. Add latency ona specific connection front-end Firefox catalogue ordersorders-db payment latency=100ms
- 53. How to defineclasses of traffic eth0 netem interface latency=100ms dest_ip=10.0.4.* dest_ip=10.0.5.* other
- 54. u32: filter oncontent eth0 HTB HTB HTBHTB HTB netemnetem netem interface root qdisc (type = HTB) root class (type = HTB) leaf qdiscs (type = netem) leaf classes (type = HTB) filters (type=u32) otherip=10.0.5.*ip=10.0.4.* latency=10ms
- 55. Filtering with cBPF/eBPF eth0 BPF netemnetem kernel userspace BPF_JMP... BPF_LD... BPF_RET... if(skb->protocol…) return TC_H_MAKE(TC_H_ROOT, mark); compilation clang... -march=bpf upload in the kernel: - bpf() - Netlink x86_64 code JIT compilation
- 56. eBPF maps eth0 BPF netemnetem kernel userspace x86_64 code eBPF map tc ∘Build statistics ∘ Make them available to the testing framework
- 57. - Another WeaveScope plugin could handle more filters: u32 or eBPF Advanced tc plugin Another plugin Scope Probe Scope App control control eth0 u32, eBPF, ...
- 58. The End Try thedemos yourself: https://github.com/kinvolk/demo Weave Scope’s traffic control plugin: https://github.com/weaveworks-plugins/scope-traffic-control The slides: https://goo.gl/knaz6m
- 59.