Automating linux network performance testing
•
1 like•361 views
The new virtualization technologies and cloud environments are a big challenge for testing network performance. We need a new approach for testing, using realistic scenarios and flexible tools that allow us to generate packets at high speed. Trex is an Open Source network generator with all these batteries included.
More Related Content
What's hot
What's hot (20)
Similar to Automating linux network performance testing
Similar to Automating linux network performance testing (20)
Recently uploaded
Recently uploaded (20)
Automating linux network performance testing
- 1. Automating Linux Network Performance Testing Antonio Ojea, SUSE, Team Lead Openstack aojeagarcia@suse.com
- 2. Who am I aojeagarcia@suse.com https://github.com/aojea
- 4. Link Layer Internet Layer Transport Layer Application Link Layer Internet Layer Transport Layer ApplicationData Data Data Data Transport Header Transport Header Transport Header Internet Header Internet Header Link Layer Header GET / HTTP/1.1 Host: www.opensuse.org ... HTTP/1.1 200 OK … Content-Type: text/html How networks works: TCP/IP Model
- 5. GET / HTTP/1.1 Host: www.opensuse.org ... HTTP/1.1 200 OK … Content-Type: text/html How networks works: pre-virtualization era
- 6. GET / HTTP/1.1 Host: www.opensuse.org ... HTTP/1.1 200 OK … Content-Type: text/html How networks works: virtualization era
- 7. GET / HTTP/1.1 Host: www.opensuse.org ... HTTP/1.1 200 OK … Content-Type: text/html How networks works: container era
- 8. Performance The word performance in computer performance means the same thing that performance means in other contexts, that is, it means "How well is the computer doing the work it is supposed to do?" 1 Computer Performance Analysis with Mathematica by Arnold O. Allen, Academic Press, 1994. $1.1 Introduction, pg 1. COMPUTER PERFORMANCE NETWORK PERFORMANCE
- 10. Traffic Generator DATAPLANE CONTROL PLANE MANAGEMENT PLANE ● Packets per second ● Connections per second ● Maximum concurrent connections per second ● ... ● Throughput ● Latency ● Frame Loss Rate ● Back to Back frames ● System Recovery ● Reset Network PerformanceRFC 2544
- 11. Linux Networking Performance: 10 Gbps maths Inter-frame gap 12 bytes MAC Preamble 8 bytes MAC header 14 bytes Min Payload Size 46 bytes CRC 4 bytes - Memory access: 70 nsec - Cache miss takes 32 nsec - L2 cache access: 4.3 nsec - L3 cache access: 7.9 nsec - 3 Ghz CPU ~ 3 nsec/inst * Ethernet frame size Transmission delay ((frame size+20)*8)/ ((10000*10^6))*10^9 Peak packet rate (10*10^9) bits/sec / (frame size + 20* 8) 64 bytes (minimum size) 67.2 ns 14,880,952 pps 1514 bytes 1227.2 ns 812,744 pps http://netoptimizer.blogspot.com/2014/05/the-calculations-10gbits-wirespeed.html?q=cycles Minimum ethernet size
- 12. Network Performance: Applications (UDP and TCP) ● UDP Throughput is Not Impacted by Latency ○ UDP fires and forget ○ This lack of control causes that packet loss impacts the application ● TCP is directly impacted by latency ○ When latency is high, it means that the sender spends more time idle (not sending any new packets), which reduces how fast throughput grows. ● TCP is Impacted by Retransmission and Packet Loss ○ Packets will need to be retransmitted (even if only the acknowledgment packet got lost and the packets got delivered) ○ The TCP congestion window size will not permit an optimal throughput http://smutz.us/techtips/NetworkLatency.html Round trip latency TCP throughput with no packet loss Round trip latency TCP throughpu t with 2% packet loss 0 ms 93.5 Mbps 3.72 Mbps 30 ms 16.2 Mbps 1.63 Mbps 60 ms 8.7 Mbps 1.33 Mbps 90 ms 5.32 Mbps 0.85 Mbps
- 13. Network Performance: Distributed systems The fallacies of distributed computing are a set of assertions made by L Peter Deutsch and others at Sun Microsystems describing false assumptions that programmers new to distributed applications invariably make. ● The network is reliable ● Latency is zero ● Bandwidth is infinite ● The network is secure ● Topology doesn't change ● There is one administrator ● Transport cost is zero ● The network is homogeneous COMPUTER NETWORK DISTRIBUTED SYSTEM https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing
- 16. Network Performance Testing: A Different Approach The word performance in computer performance means the same thing that performance means in other contexts, that is, it means "How well is the computer doing the work it is supposed to do?" 1 Computer Performance Analysis with Mathematica by Arnold O. Allen, Academic Press, 1994. $1.1 Introduction, pg 1. Let’s test how well is doing what it is supposed to do We need realistic scenarios
- 17. The tool: T-Rex (Realistic) Traffic Generator ● TRex is an open source, low cost, stateful and stateless traffic generator fuelled by DPDK. ● It generates L4-7 traffic based on pre-processing and smart replay of real traffic templates. ● TRex amplifies both client and server side traffic and can scale up to 200Gb/sec. ● TRex Stateless functionality includes support for multiple streams, the ability to change any packet field and provides per stream statistics, latency and jitter. ● New Advanced Stateful functionality includes support for emulating L7 traffic with fully-featured scalable TCP layer. ● Support testing clusters and complex network scenario https://github.com/cisco-system-traffic-generator/trex-core
- 18. Testbed Albali Marfak eth1 eth3 eth1 eth4 1.1.1.2 2.2.2.2 1.1.1.1 2.2.2.1 16.0.0.0/16 48.0.0.0/16 Marfak eth1eth4 REALITY MARFAK’s ”REALITY” Thanks @mkubecek
- 19. DEMO: Saturate 10 Gbps with http traffic Albali Marfak eth1 eth3 eth1 eth4 1.1.1.2 2.2.2.2 1.1.1.1 2.2.2.1 16.0.0.0/16 48.0.0.0/16 https://asciinema.org/a/yKc3063byjyC6J7uVdhJ5LqOl
- 20. The testing framework: Pytest ● Trex has a Python API https://trex-tgn.cisco.com/trex/doc/cp_stl_docs/#how-to-use ● … and lot of examples on how to use it https://trex-tgn.cisco.com/trex/doc/trex_cookbook/index.html ● Pytest is easy to use and I was used to it, but any other framework can be used https://github.com/aojea/pytest-trex
- 21. DEMO: A simple pytest example https://github.com/aojea/network-performance-automation/blob /master/labs/tests/test_simple_service.py https://asciinema.org/a/269496
- 22. Performance Testing Continuous Integration Unit, Functional, … Deploy Environment Run Performance Tests Check for regressions Performance Results
- 23. Lessons learned ● Don’t trust benchmarks blindly ○ Understand if the benchmark adds any value and what’s really benchmarking ● Test performance as close as possible to production ○ virtual = mock != real HW ● Treat your testing code as your normal code ○ Having tests on tree improves coordination and supportability ● Test performance periodically and don’t allow regressions ○ Use a subset of tests for gating and run the full suite nightly ● Iterate, review, measure and adjust ○ Review periodically your thresholds ○ Add policies to implement tests with bugfixes and features affecting performance ● Test need to be reproducible ○ Don’t make assumptions, verify everything ○ (A ~=B ) != (A == B)
- 24. Q&A
- 25. Marfak BONUS DEMO: A Microservice application using Linkerd (Service Mesh) on Kubernetes Thanks Rafael @ereslibre K8s node1 K8s node2 Ref: https://trex-tgn.cisco.com/trex/doc/trex_astf_vs_nginx.html
- 26. Testbed Traffic Generator DUT eth1 eth3 eth1 eth4 1.1.1.2 1.1.1.1 16.0.0.0/16 dummy 1 7 2 . 1 6 . 0 . 0 / 2 4 10.96.0.0/16 https://github.com/GoogleCloudPlatform/microservices-demo BONUS DEMO: A Microservice application using Linkerd (Service Mesh) on Kubernetes
- 27. BONUS DEMO: A Microservice application using Linkerd (Service Mesh) on Kubernetes Property Values HTTP request size 100B HTTP response size 32KB Request/Response per TCP connection 1 Average number of packets per TCP connection 32 Number of clients 200 Number of servers 1 initwnd 2 delay-ack 100msec https://youtu.be/_n8YNiYx4HU