Packet Capture in 10-Gigabit Ethernet Environments Using ...
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Packet Capture in 10-Gigabit Ethernet Environments Using ...

  • 1,334 views
Uploaded on

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,334
On Slideshare
1,331
From Embeds
3
Number of Embeds
1

Actions

Shares
Downloads
12
Comments
0
Likes
0

Embeds 3

http://www.slideshare.net 3

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Packet Capture in 10-Gigabit Ethernet Environments Using Contemporary Commodity Hardware Fabian Schneider J¨rg Wallerich Anja Feldmann o {fabian,joerg,anja}@net.t-labs.tu-berlin.de Technische Universtit¨t Berlin a Deutsche Telekom Laboratories Passive and Active Measurement Conference 5th April 2007 Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 1 / 20
  • 2. Introduction Motivation Motivation Example Scenario: Network security tool at the edge of your network • need access to packet level data for application layer analysis • High-speed networks ⇒ high data and packet rate Challenge: capture full packets without missing any packet • One approach: specialized hardware • e.g. Monitoring cards from Endace • Drawbacks: high costs, single purpose Question: Is it feasible to capture traffic with commodity hardware? Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 2 / 20
  • 3. Monitoring 10-Gigabit Outline 1 Monitoring 10-Gigabit Approach Link Bundling 2 Comparing 1-Gigabit Monitoring Systems 3 Results 4 Summary Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 3 / 20
  • 4. Monitoring 10-Gigabit Approach Approach for 10-Gigabit Monitoring • Problem: No recent host bus or disk system can handle the bandwidth needs 10GigE of 10-Gigabit environments • Solution: split up traffic and distribute the load (e.g. 10-Gigabit on multiple 1-Gigabit links) • Use a switch: e.g. link bundling feature 10 x 1GigE • Use specialized hardware • Keep corresponding data together! Monitor Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 4 / 20
  • 5. Monitoring 10-Gigabit Link Bundling Link Bundling Feasibility Etherchannel (Cisco) feature enables link-bundling for: • higher bandwidth, redundancy, . . . • or load-balancing e. g. for Webservers Feasibility test: • Tested on a Cisco 3750 • 1-Gigabit Ethernet link split on eight FastEthernet (100 Mbit/s) links. • Assign packets to links based on both IP addresses. ⇒ It works with real traffic! Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 5 / 20
  • 6. Monitoring 10-Gigabit Link Bundling Link Bundling Load-Balancing • Simple switches use only MAC addresses ⇒ Not useful for a router-to-router link • On a Cisco 3750: any combination of IP and/or MAC addresses ⇒ is sufficient for our example scenario • On a Cisco 65xx: MAC’s, IP’s, and/or Port Numbers Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 6 / 20
  • 7. Comparing 1-Gigabit Monitoring Systems Comparing 1-Gigabit Monitoring Systems 1 Monitoring 10-Gigabit 2 Comparing 1-Gigabit Monitoring Systems Methodology System under Test Measurement Setup 3 Results 4 Summary Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 7 / 20
  • 8. Comparing 1-Gigabit Monitoring Systems Methodology Methodology • Comparable priced systems with • Different processor architectures • Different operating systems • Task of those systems: • Capture full packets • Do not analyze them (Out-of-Scope) • Workload: • All system are subject to identical input • Increase bandwidth up to a fully loaded Gigabit link • Realistic packet size distribution • Measurement Categories: • Capturing Rate: number of captured packets (simple libpcap app) • System Load: CPU usage while capturing (simple top like app) Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 8 / 20
  • 9. Comparing 1-Gigabit Monitoring Systems System under Test Systems under Test Two examples of any of the systems: • One installed with Linux • The other with FreeBSD First set of systems purchased in 2005: • 2x AMD Opteron 244 (1 MB Cache, 1.8 GHz), • 2x Intel Xeon (Netburst, 512 kB Cache, 3.06 GHz), Second set purchased in 2006: • 2x Dual Core AMD Opteron 270 (1 MB Cache, 2.0 GHz) All: 2 Gbytes of RAM, optical Intel Gigabit Ethernet card, RAID array Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 9 / 20
  • 10. Comparing 1-Gigabit Monitoring Systems Measurement Setup Measurement Setup eth0 SNMP Interface Generator (LKPG) Counter Queries eth2 eth1 Cisco C3500XL Workload -> optical Splitter (mulitiplies every Signal) Linux/ Linux/ FreeBSD/ FreeBSD/ AMD Opteron Intel Xeon AMD Opteron Intel Xeon (Netburst) (Netburst) Control Network Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 10 / 20
  • 11. Results 1 Monitoring 10-Gigabit 2 Comparing 1-Gigabit Monitoring Systems 3 Results Using multiple processors? first set of systems Increasing buffer sizes measurements Additional Insights (I) Write to disk second set of systems Additional Insights (II) measurements 4 Summary Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 11 / 20
  • 12. Results Using multiple processors? (32) no SMP, no HT, std. buffers, 1 app, no filter, no load Linux/AMD Linux/Intel FreeBSD/AMD Single processor, 1st Set FreeBSD/Intel Capturing Rate [%] CPU usage [%] 100 Upper Part: 90 Capturing Rate Capturing Rate [%] 80 70 60 50 40 30 20 Lower Part: CPU Usage 10 0 SP: 100% corresponds to one fully utilised processor 100 90 MP: 50% corresponds to one fully utilised processor 80 CPU usage [%] 70 60 50 40 30 X-Axis: Generated Bandwidth 20 10 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 Datarate [Mbit/s] Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 12 / 20
  • 13. Results Using multiple processors? (32) no SMP, no HT, std. buffers, 1 app, no filter, no load Linux/AMD Linux/Intel FreeBSD/AMD Single processor, 1st Set FreeBSD/Intel Capturing Rate [%] CPU usage [%] 100 90 Capturing Rate [%] 80 70 60 50 40 30 20 10 0 100 90 80 CPU usage [%] 70 60 50 40 30 20 10 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 Datarate [Mbit/s] Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 12 / 20
  • 14. Results Using multiple processors? (32) no SMP, no HT, std. buffers, 1 app, no filter, no load Linux/AMD Linux/Intel FreeBSD/AMD Single processor, 1st Set FreeBSD/Intel Capturing Rate [%] CPU usage [%] 100 90 Capturing Rate [%] 80 70 60 50 40 30 20 Sharp decline at high data rates 10 0 100 90 Opteron/FreeBSD system performs best 80 CPU usage [%] 70 60 50 40 30 20 10 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 Datarate [Mbit/s] Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 12 / 20
  • 15. Results Using multiple processors? (31) SMP, no HT, std. buffers, 1 app, no filter, no load Linux/AMD Linux/Intel FreeBSD/AMD Multiprocessor (SMP), 1st Set FreeBSD/Intel Capturing Rate [%] CPU usage [%] 100 90 Capturing Rate [%] 80 70 60 50 40 30 20 10 0 100 90 80 CPU usage [%] 70 60 50 40 30 20 10 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 Datarate [Mbit/s] Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 13 / 20
  • 16. Results Using multiple processors? (31) SMP, no HT, std. buffers, 1 app, no filter, no load Linux/AMD Linux/Intel FreeBSD/AMD Multiprocessor (SMP), 1st Set FreeBSD/Intel Capturing Rate [%] CPU usage [%] 100 90 Capturing Rate [%] 80 70 60 50 All systems are benefitting . . . 40 30 20 10 0 . . . even though the second 100 90 processor is not used extensively 80 CPU usage [%] 70 60 50 40 30 20 10 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 Datarate [Mbit/s] Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 13 / 20
  • 17. Results Increasing buffer sizes Increasing Buffer Sizes ? Setup: • First Set of systems • Dual processor • Increased buffer sizes Operating system buffers: FreeBSD 6.x: sysctl’s net.bpf.bufsize and net.bpf.maxbufsize FreeBSD 5.x: sysctl’s debug.bpf bufsize and debug.maxbpf bufsize Linux: /proc/sys/net/core/rmem default, /proc/sys/net/core/rmem max, and /proc/sys/net/core/netdev max backlog Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 14 / 20
  • 18. Results Increasing buffer sizes (17) SMP, no HT, inc. buffers, 1 app, no filter, no load Linux/AMD Linux/Intel FreeBSD/AMD increased buffers, 1st Set FreeBSD/Intel Capturing Rate [%] CPU usage [%] 100 90 Capturing Rate [%] 80 70 60 50 40 30 20 10 0 100 90 80 CPU usage [%] 70 60 50 40 30 20 10 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 Datarate [Mbit/s] Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 15 / 20
  • 19. Results Increasing buffer sizes (17) SMP, no HT, inc. buffers, 1 app, no filter, no load Linux/AMD Linux/Intel FreeBSD/AMD increased buffers, 1st Set FreeBSD/Intel Capturing Rate [%] CPU usage [%] 100 90 Capturing Rate [%] 80 70 60 50 40 30 The capturing rate could be increased again 20 10 0 100 90 80 CPU usage [%] 70 60 50 40 30 20 10 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 Datarate [Mbit/s] Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 15 / 20
  • 20. Results Additional Insights (I) Additional Insights First set of measurements • Filtering is cheap with respect to its benefit (reduced packet processing) • Running multiple capturing applications concurrently leads to bad performance. • Measurement with additional compression show some advantage for Intel Systems • Intel Hyperthreading does not change the performance • using the memory-map patch from Phil Woods (Linux only) does help Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 16 / 20
  • 21. Results Write to disk Writing packets to disk? preliminary measurements have shown that • newer system do not lose any packet: with buffers, SMP, etc. • disk writing speed is not the bottleneck Setup: • Newer systems: 2x dual core AMD systems ⇒ CPU usage: 25% correspond to one fully utilized processor • Increased buffer sizes • No filter • Linux vs. FreeBSD • 32bit vs. 64bit OS’es Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 17 / 20
  • 22. Results Write to disk (2-8) SMP, no HT, inc. buffers, 1 app, no filter, 32bit FreeBSD/Opteron writing to disk 64bit FreeBSD/Opteron 32bit Linux/Opteron Writing to disk, 2nd Set 64bit Linux/Opteron Capturing Rate [%] CPU usage [%] 100 90 Capturing Rate [%] 80 70 60 50 40 30 20 10 0 100 90 80 CPU usage [%] 70 60 50 40 30 20 10 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 Datarate [Mbit/s] Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 18 / 20
  • 23. Results Write to disk (2-8) SMP, no HT, inc. buffers, 1 app, no filter, 32bit FreeBSD/Opteron writing to disk 64bit FreeBSD/Opteron 32bit Linux/Opteron Writing to disk, 2nd Set 64bit Linux/Opteron Capturing Rate [%] CPU usage [%] 100 90 Capturing Rate [%] 80 70 60 50 40 30 20 Feasible up to 600-700 Mbit/s! 10 0 100 90 80 CPU usage [%] 70 60 50 40 30 20 10 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 Datarate [Mbit/s] Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 18 / 20
  • 24. Results Additional Insights (II) Additional Insights Second set of measurements • additional load (copying the packets in memory) shows significantly better performance for FreeBSD • 64bit systems drop more packets • Using 4 cores (2x Dual Core) is slightly better than 2 cores (1x Dual Core) Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 19 / 20
  • 25. Summary Summary • Split up 10-Gigabit on multiple 1-Gigabit monitoring systems • FreeBSD/AMD Opteron combination in general performs best • Utilizing multiple processors proves to be benefitting • Choosing large enough buffer size is important • Capturing full traces to disk is feasible up to about 600-700 Mbit/s For further information see: High Performance Packet Capture http://www.net.t-labs.tu-berlin.de/research/hppc/ Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 20 / 20