This document discusses using traffic control to test microservices applications running in containers. It describes using tools like Kubernetes, Weave Scope, and netem to configure network emulation scenarios like bandwidth limiting, latency injection and packet dropping. A demo application is shown running in Kubernetes pods with Weave Scope visualizing network traffic and a traffic control plugin modifying network behavior for testing purposes. Advanced filtering options using eBPF and custom plugins are proposed to define more complex traffic classes and collect test statistics.

1. In containers
Alban Crequy
Testing applications
with traffic control
HighLoad++ - November 2016https://goo.gl/knaz6m

2. Alban Crequy
∘ Worked on traffic control for multimedia
applications in cars (tcmmd)
∘ Worked on the container run-time rkt
∘ Working on Weave Scope
https://github.com/alban

3. Berlin-based software company building
foundational Linux technologies
Some examples of what we worked on...
OSTreegit for operating
system binaries

4. Who works with...

5. Find out more about us…
Blog: https://kinvolk.io/blog
Github: https://github.com/kinvolk
Twitter: https://twitter.com/kinvolkio
Email: hello@kinvolk.io

6. ∘ What is traffic control and how does it work on Linux
∘ How it can be used for testing a microservices application
∘ Demo
∘ With Minikube, Kubernetes, Weave Scope, Weave Socks
Plan

7. What is traffic control?
How does it work on Linux?

8. Traffic control, why?
web server client
client
client
THE
INTERNET
∘ fair distribution
of bandwidth
∘ reserve
bandwidth to
specific
applications
∘ avoid
bufferbloat

9. ∘ Network scheduling algorithm
∘ which packet to emit next?
∘ when?
∘ Configurable at run-time:
∘ /sbin/tc
∘ Netlink
∘ Default on new network interfaces: sysctl net.core.default_qdisc
Queuing disciplines
(qdisc)
eth0 THE INTERNETqdisc

10. Stochastic Fairness
Queueing (sfq)
eth0
THE INTERNET
FIFO n
FIFO 1
FIFO 0
...
round robin

11. Traffic control for testing?

12. Developers & Internet
Developers usually have good Internet connection
- Low latency
- High bandwidth
- Not much packet loss or packet corruption
Difficult to see problems

13. Network emulator
(netem)
eth0 THE INTERNETnetem
bandwidth
latency packet loss
corrupt
...

14. Testing with containers
container 1 container 2
eth0eth0
Testing framework
configure “netem” qdiscs:
bandwidth, latency, packet drop...

15. The demo application

16. microservices-demo
https://github.com/microservices-demo/microservices-demo

17. Some micro-services
front-end Firefox
catalogue
ordersorders-db
payment

18. Kubernetes

19. Pods
Group of container(s) running together, in the same context
Kubernetes
node 1
Kubernetes
node 2
pod
container
container
pod
container
pod
container
pod
container
Each pod has
- Its own network namespace
- Its own IP address

20. Replica sets
- Control the number of “Pods” replicas running
- Deploy pods
Kubernetes
node 1
Kubernetes
node 2
pod
catalogue
pod
front-end
pod
orders
In this example:
- 2x front-end
- 1x catalogue
- 1x orders
pod
front-end

21. Replica set
“catalogue”
Services
Directing the traffic to “Pods”
- Cluster IP
- DNAT
catalogue1
catalogue2
catalogue3
Service
“catalogue”
front-end

22. Testing with traffic control in
Kubernetes
Kubernetes
node 1
pod
pod
Kubernetes
node 2
pod
pod
tc tc
controls
∘ Latency
∘ Bandwidth
∘ Packet drop
∘ configure network
simulator
∘ play scenarios

23. Weave Scope

24. Weave Scope

25. Testing with Weave Scope
Kubernetes
node 1
tc
Scope
Probe
pod pod
pod pod
Kubernetes
Node 2
tc
Scope
Probe
pod pod
pod pod
Scope
App

26. <Demo>
Reproduce the demo yourself: https://github.com/kinvolk/demo

47. </Demo>
Reproduce the demo yourself: https://github.com/kinvolk/demo

48. Testing framework
for web apps
Selenium

49. Plugins in Scope
tc plugin
Scope
Probe
Scope
App
report
reportcontrol
control
- Unix socket in /var/run/scope/plugins/
- Protocols: report and control
- Write your own plugins for your testing needs

51. Testing more complex
scenarios
(my “wishlist”)

52. Add latency on a specific
connection
front-end Firefox
catalogue
ordersorders-db
payment
latency=100ms

53. How to define classes of traffic
eth0
netem
interface
latency=100ms
dest_ip=10.0.4.* dest_ip=10.0.5.* other

54. u32: filter on content
eth0
HTB
HTB
HTBHTB HTB
netemnetem netem
interface
root qdisc (type = HTB)
root class (type = HTB)
leaf qdiscs (type = netem)
leaf classes (type = HTB)
filters
(type=u32)
otherip=10.0.5.*ip=10.0.4.*
latency=10ms

55. Filtering with cBPF/eBPF
eth0
BPF
netemnetem
kernel
userspace
BPF_JMP...
BPF_LD...
BPF_RET...
if (skb->protocol…)
return TC_H_MAKE(TC_H_ROOT,
mark); compilation
clang... -march=bpf
upload
in the kernel:
- bpf()
- Netlink
x86_64 code
JIT compilation

56. eBPF maps
eth0
BPF
netemnetem
kernel
userspace
x86_64 code
eBPF
map
tc
∘ Build statistics
∘ Make them available to
the testing framework

57. - Another Weave Scope plugin could handle
more filters: u32 or eBPF
Advanced tc plugin
Another
plugin
Scope
Probe
Scope
App
control
control
eth0
u32,
eBPF, ...

58. The End
Try the demos yourself: https://github.com/kinvolk/demo
Weave Scope’s traffic control plugin:
https://github.com/weaveworks-plugins/scope-traffic-control
The slides: https://goo.gl/knaz6m

59. Questions?