Ringmap slides

933 views
865 views

Published on

Ringmap is a complete FreeBSD packet capturing stack specialized for very high-speed networks. Conventional packet capturing software often suffers packet loss at high data rates close to 1Gb/sec or greater. Ringmap takes significant measures to minimize both packet loss and overall system load during packet capture. The goal of this project is to further develop the new packet capturing software for FreeBSD and integrate it with the generic network drivers and libpcap.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
933
On SlideShare
0
From Embeds
0
Number of Embeds
109
Actions
Shares
0
Downloads
13
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Ringmap slides

  1. 1. Outline Motivation Background Solution Performance evaluation SummaryRingmap Capturing Stack for High Performance Packet Capturing in FreeBSD Alexander Fiveg afiveg@freebsd.org September 28, 2010 Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  2. 2. Outline Motivation Background Solution Performance evaluation SummaryOutline 1 Motivation 2 Background 3 Solution 4 Performance evaluation 5 Summary Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  3. 3. Outline Motivation Background Problem Solution Goal of the ProjectPerformance evaluation Summary Motivation Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  4. 4. Outline Motivation Background Problem Solution Goal of the Project Performance evaluation SummaryProblem Problems arising during packet capturing: High bit rates and packet rates ⇒ high CPU load and packet loss 100 100 Packet loss (%) CPU usage (%) 80 80 60 60 40 40 20 20 0 0 100 200 300 400 500 600 100 200 300 400 500 600 Bit-Rate (MBit/s) Bit-Rate (MBit/s) Figure: Capturing 64-Bytes packets. (Bezier curves) Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  5. 5. Outline Motivation Background Problem Solution Goal of the Project Performance evaluation SummaryThe reason of the Problems Inefficiency of standard capturing software To many “expensive” operations in terms of CPU cycles: System calls Packet copy operations Memory allocations etc. . . Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  6. 6. Outline Motivation Background Problem Solution Goal of the Project Performance evaluation SummaryGoal of the project Increase the capturing performance in FreeBSD Design and implementation the new capturing software to minimize the packet loss and CPU load during capturing. Supported Hardware: The following Intel GbE Controllers: 8254x, 8257x, 8259x Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  7. 7. Outline Motivation Background Problem Solution Goal of the Project Performance evaluation SummaryApproach Eliminating the packet copy operations by using shared memory buffers (memory mapping) Eliminating the memory allocations by using ring buffers Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  8. 8. Outline Motivation Background Problem Solution Goal of the Project Performance evaluation SummaryApproach Eliminating the packet copy operations by using shared memory buffers (memory mapping) Eliminating the memory allocations by using ring buffers ⇒ ring + mapping = ringmap Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  9. 9. Outline Motivation Packet Capturing Background How does it work Solution DisadvantagesPerformance evaluation Summary Background Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  10. 10. Outline Motivation Packet Capturing Background How does it work Solution Disadvantages Performance evaluation SummaryPacket Capturing: Hardware View 1 Receiving network packets CPU’s receive at network adapter DMA transfer in RAM 2 2 Filtering the received 3 mass storage packets RAM Berkeley Packet Filter 1 3 Storing to the hard disk using a system call Network adapter Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  11. 11. Outline Motivation Packet Capturing Background How does it work Solution Disadvantages Performance evaluation SummaryFreeBSD Packet Capturing Stack Network driver Receiving packets Berkley Packet Filter (BPF) Filtering packets User-space application Accessing received packets Initiates: Storing packets to hard drive Output packets information to the terminal Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  12. 12. Outline Motivation Packet Capturing Background How does it work Solution Disadvantages Performance evaluation SummaryHow does standard packet capturing work Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  13. 13. Outline Motivation Packet Capturing Background How does it work Solution Disadvantages Performance evaluation SummaryFreeBSD Standard Packet Capturing Stack Packet is received and saved in Adapter-FIFO Adapter-FIFO Network adapter Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  14. 14. Outline Motivation Packet Capturing Background How does it work Solution Disadvantages Performance evaluation SummaryFreeBSD Standard Packet Capturing Stack DMA-Transfer Paket-Buffer Packet is received and saved in Adapter-FIFO 1 Adapter-FIFO Network adapter Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  15. 15. Outline Motivation Packet Capturing Background How does it work Solution Disadvantages Performance evaluation SummaryFreeBSD Standard Packet Capturing Stack Interrupt DMA-Transfer Paket-Buffer Packet is received and saved in Adapter-FIFO 1 Adapter-FIFO Network adapter Interrupt Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  16. 16. Outline Motivation Packet Capturing Background How does it work Solution Disadvantages Performance evaluation SummaryFreeBSD Standard Packet Capturing Stack Interrupt Service Routine Interrupt Kernel-Thread DMA-Transfer Packet-Buffer Packet is received and saved xN in Adapter-FIFO Network driver ISR 1 Adapter-FIFO Network adapter Interrupt Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  17. 17. Outline Motivation Packet Capturing Background How does it work Solution Disadvantages Performance evaluation SummaryFreeBSD Standard Packet Capturing Stack Userspace Packet filtering Kernelspace BPF-Buffer BPF Interrupt Service Routine Interrupt 2 Kernel-Thread DMA-Transfer Packet-Buffer Packet is received and saved xN in Adapter-FIFO Network driver ISR 1 Adapter-FIFO Network adapter Interrupt Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  18. 18. Outline Motivation Packet Capturing Background How does it work Solution Disadvantages Performance evaluation SummaryFreeBSD Standard Packet Capturing Stack User-Application User-Buffer Access packets Userspace Libpcap-Functions Packet filtering 3 Kernelspace BPF-Buffer BPF Interrupt Service Routine Interrupt 2 Kernel-Thread DMA-Transfer Packet-Buffer Packet is received and saved xN in Adapter-FIFO Network driver ISR 1 Adapter-FIFO Network adapter Interrupt Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  19. 19. Outline Motivation Packet Capturing Background How does it work Solution Disadvantages Performance evaluation SummaryDisadvantages of Standard Packet Capturing Stack Three copies per packet 1 DMA: FIFO ⇒ Packet Buffer 2 Packet Buffer ⇒ BPF Buffer 3 BPF Buffer ⇒ Userspace Buffer (*) Memory allocations For each new received packet will be a new mbuf allocated System calls User-space application receives the packets using read(2)(*) Saving packets to the hard disk (*) Using Zero-Copy BPF eliminates the last copy and system call Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  20. 20. Outline Motivation Ringmap Capturing Stack Background From generic to ringmap Solution OverviewPerformance evaluation Summary Solution Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  21. 21. Outline Motivation Ringmap Capturing Stack Background From generic to ringmap Solution Overview Performance evaluation SummaryRingmap Packet Capturing Stack Ringmap: The new packet capturing stack based on standard FreeBSD packet capturing stack: based on generic network drivers and libpcap but network driver and libpcap require small modifications Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  22. 22. Outline Motivation Ringmap Capturing Stack Background From generic to ringmap Solution Overview Performance evaluation SummaryWhat did we change in generic User-Application Libpcap-Functions Userspace Kernelspace BPF TCP/IP-Stack Kernel-Thread xN ISR Network driver Software Network adapter Interrupt Hardware Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  23. 23. Outline Motivation Ringmap Capturing Stack Background From generic to ringmap Solution Overview Performance evaluation SummaryWhat did we change in generic User-Application Libpcap-Functions Userspace Disabled TCP/IP-Stack Kernelspace Only packet capturing BPF TCP/IP-Stack Kernel-Thread xN ISR Network driver Software Network adapter Interrupt Hardware Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  24. 24. Outline Motivation Ringmap Capturing Stack Background From generic to ringmap Solution Overview Performance evaluation SummaryWhat did we change in generic User-Application Libpcap-Functions Userspace Disabled TCP/IP-Stack Kernelspace Only packet capturing BPF BPF is accessible in both TCP/IP-Stack kernel and user-space Kernel-Thread xN ISR Network driver Software Network adapter Interrupt Hardware Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  25. 25. Outline Motivation Ringmap Capturing Stack Background From generic to ringmap Solution Overview Performance evaluation SummaryWhat did we change in generic User-Application Libpcap-Functions Userspace Disabled TCP/IP-Stack Kernelspace Only packet capturing BPF BPF is accessible in both TCP/IP-Stack kernel and user-space Kernel-Thread xN Network driver and Libpcap ISR is modified Network driver Software BPF is unchanged Network adapter Interrupt Hardware Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  26. 26. Outline Motivation Ringmap Capturing Stack Background From generic to ringmap Solution Overview Performance evaluation SummaryFrom generic to ringmap Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  27. 27. Outline Motivation Ringmap Capturing Stack Background From generic to ringmap Solution Overview Performance evaluation SummaryFrom generic to ringmap User-Application User-Buffer generic Userspace Libpcap-Functions Kernelspace BPF-Buffer BPF Kernel-Thread Packet-Buffer xN ISR Network driver Adapter-FIFO Network adapter Interrupt Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  28. 28. Outline Motivation Ringmap Capturing Stack Background From generic to ringmap Solution Overview Performance evaluation SummaryFrom generic to ringmap User-Application User-Buffer generic Userspace Libpcap-Functions Eliminate packet copies and Kernelspace syscalls BPF-Buffer BPF Kernel-Thread Packet-Buffer xN ISR Network driver Adapter-FIFO Network adapter Interrupt Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  29. 29. Outline Motivation Ringmap Capturing Stack Background From generic to ringmap Solution Overview Performance evaluation SummaryFrom generic to ringmap User-Application User-Buffer generic Libpcap-Functions Eliminate packet copies and Userspace syscalls BPF-Buffer Kernelspace BPF BPF-Buffer is then not necessary Kernel-Thread Packet filtering is possible in both kernel and Packet-Buffer xN user-space ISR Network driver Adapter-FIFO Network adapter Interrupt Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  30. 30. Outline Motivation Ringmap Capturing Stack Background From generic to ringmap Solution Overview Performance evaluation SummaryFrom generic to ringmap User-Application User-Buffer generic Libpcap-Functions Eliminate packet copies and Userspace syscalls Kernelspace BPF BPF-Buffer is then not necessary Kernel-Thread Packet filtering is possible in both kernel and Packet-Buffer xN user-space ISR Mapping the Packet-Buffer Network driver to user-space Network adapter Interrupt Adapter-FIFO Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  31. 31. Outline Motivation Ringmap Capturing Stack Background From generic to ringmap Solution Overview Performance evaluation SummaryFrom generic to ringmap User-Applications Packet-Buffer generic Libpcap-Functions Eliminate packet copies and Userspace syscalls Kernelspace BPF BPF-Buffer is then not necessary Kernel-Thread Packet filtering is possible in both kernel and Packet-Buffer xN user-space ISR Mapping the Packet-Buffer Network driver to user-space Network adapter Interrupt Adapter-FIFO ⇒ ringmap Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  32. 32. Outline Motivation Ringmap Capturing Stack Background From generic to ringmap Solution Overview Performance evaluation SummaryOverview Packet Copy Functions Packet Copy Functions User-Applications User-Applications User-Buffer Packet-Buffer Libpcap-Functions Libpcap-Functions Userspace Kernelspace Userspace BPF-Buffer BPF BPF Kernelspace Kernel-Thread Kernel-Thread Packet-Buffer Packet-Buffer xN xN ISR ISR Network driver Network driver Adapter-FIFO Network adapter Interrupt Adapter-FIFO Network adapter Interrupt standard Capturing Stack ringmap Capturing Stack Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  33. 33. Outline Motivation Measurement setup Background Testing sequence Solution Results Performance evaluation SummaryMeasurements and performance evaluation Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  34. 34. Outline Motivation Measurement setup Background Testing sequence Solution Results Performance evaluation SummaryGoal of measurements Conducting performance evaluations of ringmap capturing stack CPU-Load and packet loss during capturing Benchmarking: generic vs. ringmap Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  35. 35. Outline Motivation Measurement setup Background Testing sequence Solution Results Performance evaluation SummaryTestbed Capturer Packet generator FreeBSD Linux-SMP control traffic Linux generated packets Test control system Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  36. 36. Outline Motivation Measurement setup Background Testing sequence Solution Results Performance evaluation SummaryMeasurement setup Traffic generation Linux Kernel Packet Generator (pktgen) generates network packets at very high speed with different: packet sizes bit-rate and packet-rate Capturing ringmap and generic stacks Libpcap-application accessing packets through call pcap loop() counting the captured packets Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  37. 37. Outline Motivation Measurement setup Background Testing sequence Solution Results Performance evaluation SummaryCalculation of results System load Percentage proportion of time the CPU spent in system mode Interrupt load is not considered because of interrupt-throttling it is always constant for generic and ringmap syst = 1 − intr − user − nice − idle Packet loss Difference between generated and captured packets Packetsloss = Packetssend − Packetsreceived Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  38. 38. Outline Motivation Measurement setup Background Testing sequence Solution Results Performance evaluation SummaryTesting sequence 1 Login to the capturer to start of capturing start of system load measurement applications 2 Login to the Packet-Generator to generate traffic with: a certain number of packets a certain bit-rate and packet-rate 3 Login to the capturer to stop capturing- and measurement-applications 4 Storing of measured values Each experiment is repeated five times The mean and standard deviation is calculated Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  39. 39. Outline Motivation Measurement setup Background Testing sequence Solution ResultsPerformance evaluation Summary Results Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  40. 40. Outline Motivation Measurement setup Background Testing sequence Solution Results Performance evaluation Summarygeneric vs. ringmap: Packet loss 100 64-bytes packets ringmap 200-bytes packets ringmap 300-bytes packets ringmap 64-bytes packets generic 200-bytes packets generic 80 300-bytes packets generic Standard deviation 60 packet loss (%) 40 20 0 0 100 200 300 400 500 600 700 800 900 1000 Bit-Rate (MBit/s) Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  41. 41. Outline Motivation Measurement setup Background Testing sequence Solution Results Performance evaluation Summarygeneric vs. ringmap: System-load 100 64-bytes packets ringmap 200-bytes packets ringmap 90 300-bytes packets ringmap 64-bytes packets generic 200-bytes packets generic 80 300-bytes packets generic Standard deviation 70 60 Systemload (%) 50 40 30 20 10 0 0 100 200 300 400 500 600 700 800 900 1000 Bit-Rate (MBit/s) Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  42. 42. Outline Motivation Background Achieved Goals Solution Future WorksPerformance evaluation Summary Summary Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  43. 43. Outline Motivation Background Achieved Goals Solution Future Works Performance evaluation SummaryAchieved goals Enhanced Capturing-Performance very low system load (below 12%) and very low packet loss (below 0.02%) only during capturing smallest 64-Bytes packets for maximal reached packet-rate (> 450000pkts/sec) Stability and usability of implemented software during all experiments, no kernel panics and no segmentation faults occurred very simple install and remove two Shell-Scripts to installing and removing the ringmap Capturing Stacks Libpcap-applications don’t require modification in order to run with the ringmap Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  44. 44. Outline Motivation Background Achieved Goals Solution Future Works Performance evaluation SummaryAchieved goals 2 Ringmap is easily portable to the other network controllers The software contains hardware dependent and hardware independent parts. Only hardware dependent part require modifications. The generic driver and libpcap should contain a few hooks for calling ringmap-functions. Packet Filtering Packet filtering can be accomplished using both libpcap- and kernel-BPF. Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  45. 45. Outline Motivation Background Achieved Goals Solution Future Works Performance evaluation SummaryAchieved goals 3 Multithreaded Capturing Multiple applications can capture from the same interface. Partly ported to the 10GbE controller Currently only one queue is used while capturing. The work on supporting multiqueue is in progress. Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
  46. 46. Outline Motivation Background Achieved Goals Solution Future Works Performance evaluation SummaryFuture works Benchmarking: Zero-Copy BPF vs. ringmap Support for hardware time stamping Writing the packets to the disc from within the kernel Enabling TCP/IP stack 10-GbE-Packet-Capturing: Multiqueue support Support for hardware packet filtering Extending ringmap for packet transmission Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur

×