Your SlideShare is downloading. ×
0
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Packet Capture in 10-Gigabit Ethernet Environments Using ...
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Packet Capture in 10-Gigabit Ethernet Environments Using ...

1,039

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,039
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
14
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Packet Capture in 10-Gigabit Ethernet Environments Using Contemporary Commodity Hardware Fabian Schneider J¨rg Wallerich Anja Feldmann o {fabian,joerg,anja}@net.t-labs.tu-berlin.de Technische Universtit¨t Berlin a Deutsche Telekom Laboratories Passive and Active Measurement Conference 5th April 2007 Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 1 / 20
  • 2. Introduction Motivation Motivation Example Scenario: Network security tool at the edge of your network • need access to packet level data for application layer analysis • High-speed networks ⇒ high data and packet rate Challenge: capture full packets without missing any packet • One approach: specialized hardware • e.g. Monitoring cards from Endace • Drawbacks: high costs, single purpose Question: Is it feasible to capture traffic with commodity hardware? Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 2 / 20
  • 3. Monitoring 10-Gigabit Outline 1 Monitoring 10-Gigabit Approach Link Bundling 2 Comparing 1-Gigabit Monitoring Systems 3 Results 4 Summary Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 3 / 20
  • 4. Monitoring 10-Gigabit Approach Approach for 10-Gigabit Monitoring • Problem: No recent host bus or disk system can handle the bandwidth needs 10GigE of 10-Gigabit environments • Solution: split up traffic and distribute the load (e.g. 10-Gigabit on multiple 1-Gigabit links) • Use a switch: e.g. link bundling feature 10 x 1GigE • Use specialized hardware • Keep corresponding data together! Monitor Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 4 / 20
  • 5. Monitoring 10-Gigabit Link Bundling Link Bundling Feasibility Etherchannel (Cisco) feature enables link-bundling for: • higher bandwidth, redundancy, . . . • or load-balancing e. g. for Webservers Feasibility test: • Tested on a Cisco 3750 • 1-Gigabit Ethernet link split on eight FastEthernet (100 Mbit/s) links. • Assign packets to links based on both IP addresses. ⇒ It works with real traffic! Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 5 / 20
  • 6. Monitoring 10-Gigabit Link Bundling Link Bundling Load-Balancing • Simple switches use only MAC addresses ⇒ Not useful for a router-to-router link • On a Cisco 3750: any combination of IP and/or MAC addresses ⇒ is sufficient for our example scenario • On a Cisco 65xx: MAC’s, IP’s, and/or Port Numbers Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 6 / 20
  • 7. Comparing 1-Gigabit Monitoring Systems Comparing 1-Gigabit Monitoring Systems 1 Monitoring 10-Gigabit 2 Comparing 1-Gigabit Monitoring Systems Methodology System under Test Measurement Setup 3 Results 4 Summary Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 7 / 20
  • 8. Comparing 1-Gigabit Monitoring Systems Methodology Methodology • Comparable priced systems with • Different processor architectures • Different operating systems • Task of those systems: • Capture full packets • Do not analyze them (Out-of-Scope) • Workload: • All system are subject to identical input • Increase bandwidth up to a fully loaded Gigabit link • Realistic packet size distribution • Measurement Categories: • Capturing Rate: number of captured packets (simple libpcap app) • System Load: CPU usage while capturing (simple top like app) Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 8 / 20
  • 9. Comparing 1-Gigabit Monitoring Systems System under Test Systems under Test Two examples of any of the systems: • One installed with Linux • The other with FreeBSD First set of systems purchased in 2005: • 2x AMD Opteron 244 (1 MB Cache, 1.8 GHz), • 2x Intel Xeon (Netburst, 512 kB Cache, 3.06 GHz), Second set purchased in 2006: • 2x Dual Core AMD Opteron 270 (1 MB Cache, 2.0 GHz) All: 2 Gbytes of RAM, optical Intel Gigabit Ethernet card, RAID array Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 9 / 20
  • 10. Comparing 1-Gigabit Monitoring Systems Measurement Setup Measurement Setup eth0 SNMP Interface Generator (LKPG) Counter Queries eth2 eth1 Cisco C3500XL Workload -> optical Splitter (mulitiplies every Signal) Linux/ Linux/ FreeBSD/ FreeBSD/ AMD Opteron Intel Xeon AMD Opteron Intel Xeon (Netburst) (Netburst) Control Network Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 10 / 20
  • 11. Results 1 Monitoring 10-Gigabit 2 Comparing 1-Gigabit Monitoring Systems 3 Results Using multiple processors? first set of systems Increasing buffer sizes measurements Additional Insights (I) Write to disk second set of systems Additional Insights (II) measurements 4 Summary Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 11 / 20
  • 12. Results Using multiple processors? (32) no SMP, no HT, std. buffers, 1 app, no filter, no load Linux/AMD Linux/Intel FreeBSD/AMD Single processor, 1st Set FreeBSD/Intel Capturing Rate [%] CPU usage [%] 100 Upper Part: 90 Capturing Rate Capturing Rate [%] 80 70 60 50 40 30 20 Lower Part: CPU Usage 10 0 SP: 100% corresponds to one fully utilised processor 100 90 MP: 50% corresponds to one fully utilised processor 80 CPU usage [%] 70 60 50 40 30 X-Axis: Generated Bandwidth 20 10 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 Datarate [Mbit/s] Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 12 / 20
  • 13. Results Using multiple processors? (32) no SMP, no HT, std. buffers, 1 app, no filter, no load Linux/AMD Linux/Intel FreeBSD/AMD Single processor, 1st Set FreeBSD/Intel Capturing Rate [%] CPU usage [%] 100 90 Capturing Rate [%] 80 70 60 50 40 30 20 10 0 100 90 80 CPU usage [%] 70 60 50 40 30 20 10 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 Datarate [Mbit/s] Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 12 / 20
  • 14. Results Using multiple processors? (32) no SMP, no HT, std. buffers, 1 app, no filter, no load Linux/AMD Linux/Intel FreeBSD/AMD Single processor, 1st Set FreeBSD/Intel Capturing Rate [%] CPU usage [%] 100 90 Capturing Rate [%] 80 70 60 50 40 30 20 Sharp decline at high data rates 10 0 100 90 Opteron/FreeBSD system performs best 80 CPU usage [%] 70 60 50 40 30 20 10 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 Datarate [Mbit/s] Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 12 / 20
  • 15. Results Using multiple processors? (31) SMP, no HT, std. buffers, 1 app, no filter, no load Linux/AMD Linux/Intel FreeBSD/AMD Multiprocessor (SMP), 1st Set FreeBSD/Intel Capturing Rate [%] CPU usage [%] 100 90 Capturing Rate [%] 80 70 60 50 40 30 20 10 0 100 90 80 CPU usage [%] 70 60 50 40 30 20 10 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 Datarate [Mbit/s] Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 13 / 20
  • 16. Results Using multiple processors? (31) SMP, no HT, std. buffers, 1 app, no filter, no load Linux/AMD Linux/Intel FreeBSD/AMD Multiprocessor (SMP), 1st Set FreeBSD/Intel Capturing Rate [%] CPU usage [%] 100 90 Capturing Rate [%] 80 70 60 50 All systems are benefitting . . . 40 30 20 10 0 . . . even though the second 100 90 processor is not used extensively 80 CPU usage [%] 70 60 50 40 30 20 10 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 Datarate [Mbit/s] Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 13 / 20
  • 17. Results Increasing buffer sizes Increasing Buffer Sizes ? Setup: • First Set of systems • Dual processor • Increased buffer sizes Operating system buffers: FreeBSD 6.x: sysctl’s net.bpf.bufsize and net.bpf.maxbufsize FreeBSD 5.x: sysctl’s debug.bpf bufsize and debug.maxbpf bufsize Linux: /proc/sys/net/core/rmem default, /proc/sys/net/core/rmem max, and /proc/sys/net/core/netdev max backlog Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 14 / 20
  • 18. Results Increasing buffer sizes (17) SMP, no HT, inc. buffers, 1 app, no filter, no load Linux/AMD Linux/Intel FreeBSD/AMD increased buffers, 1st Set FreeBSD/Intel Capturing Rate [%] CPU usage [%] 100 90 Capturing Rate [%] 80 70 60 50 40 30 20 10 0 100 90 80 CPU usage [%] 70 60 50 40 30 20 10 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 Datarate [Mbit/s] Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 15 / 20
  • 19. Results Increasing buffer sizes (17) SMP, no HT, inc. buffers, 1 app, no filter, no load Linux/AMD Linux/Intel FreeBSD/AMD increased buffers, 1st Set FreeBSD/Intel Capturing Rate [%] CPU usage [%] 100 90 Capturing Rate [%] 80 70 60 50 40 30 The capturing rate could be increased again 20 10 0 100 90 80 CPU usage [%] 70 60 50 40 30 20 10 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 Datarate [Mbit/s] Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 15 / 20
  • 20. Results Additional Insights (I) Additional Insights First set of measurements • Filtering is cheap with respect to its benefit (reduced packet processing) • Running multiple capturing applications concurrently leads to bad performance. • Measurement with additional compression show some advantage for Intel Systems • Intel Hyperthreading does not change the performance • using the memory-map patch from Phil Woods (Linux only) does help Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 16 / 20
  • 21. Results Write to disk Writing packets to disk? preliminary measurements have shown that • newer system do not lose any packet: with buffers, SMP, etc. • disk writing speed is not the bottleneck Setup: • Newer systems: 2x dual core AMD systems ⇒ CPU usage: 25% correspond to one fully utilized processor • Increased buffer sizes • No filter • Linux vs. FreeBSD • 32bit vs. 64bit OS’es Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 17 / 20
  • 22. Results Write to disk (2-8) SMP, no HT, inc. buffers, 1 app, no filter, 32bit FreeBSD/Opteron writing to disk 64bit FreeBSD/Opteron 32bit Linux/Opteron Writing to disk, 2nd Set 64bit Linux/Opteron Capturing Rate [%] CPU usage [%] 100 90 Capturing Rate [%] 80 70 60 50 40 30 20 10 0 100 90 80 CPU usage [%] 70 60 50 40 30 20 10 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 Datarate [Mbit/s] Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 18 / 20
  • 23. Results Write to disk (2-8) SMP, no HT, inc. buffers, 1 app, no filter, 32bit FreeBSD/Opteron writing to disk 64bit FreeBSD/Opteron 32bit Linux/Opteron Writing to disk, 2nd Set 64bit Linux/Opteron Capturing Rate [%] CPU usage [%] 100 90 Capturing Rate [%] 80 70 60 50 40 30 20 Feasible up to 600-700 Mbit/s! 10 0 100 90 80 CPU usage [%] 70 60 50 40 30 20 10 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 Datarate [Mbit/s] Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 18 / 20
  • 24. Results Additional Insights (II) Additional Insights Second set of measurements • additional load (copying the packets in memory) shows significantly better performance for FreeBSD • 64bit systems drop more packets • Using 4 cores (2x Dual Core) is slightly better than 2 cores (1x Dual Core) Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 19 / 20
  • 25. Summary Summary • Split up 10-Gigabit on multiple 1-Gigabit monitoring systems • FreeBSD/AMD Opteron combination in general performs best • Utilizing multiple processors proves to be benefitting • Choosing large enough buffer size is important • Capturing full traces to disk is feasible up to about 600-700 Mbit/s For further information see: High Performance Packet Capture http://www.net.t-labs.tu-berlin.de/research/hppc/ Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 20 / 20

×