Your SlideShare is downloading. ×
Network processing by pid
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Network processing by pid

2,223
views

Published on

Network monitoring of processes in Linux, using Linux dynamic Kernel instrumentation (KProbes) …

Network monitoring of processes in Linux, using Linux dynamic Kernel instrumentation (KProbes)

Monitoring network interactions of one process accessing the network is not always simple and it has some performance issues.
A Linux Kernel Module was developed, which uses dynamic instrumentation and monitors the target user process for interactions and registers the information to a repository.
When packets pass through the network interface the repository is queried to decide if the packet should be captured for further analysis.
To control this monitoring mechanism an interface was developed which can be modified through files in the virtual filesystem, DebugFS.
To use this monitoring mechanism it is necessary to have the Linux Kernel Module loaded and have a user process running that performs the network monitoring (such as TCPDump). This monitoring process can use this mechanism without changing its own source code.

Published in: Technology, Education

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,223
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
23
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Energia Open Source Network Monitoring by process id Nuno Martins nuno.martins@caixamagica.pt
  • 2. Energia Open Source Monitoring  Understand programs actual behaviour when running  Analyse resource usage  Create a usage profile, to evaluate performance and correctness  Can be done actively (polling the monitor) or passively  Through dynamic or static instrumentation  Analysis of the data can be done online (during capture) or offline (post mortem)27.02.2012 Network Monitoring by Process id 2
  • 3. Energia Open Source Network Monitoring  Network monitoring is done passively by capturing packets  Generally it is done using PCap (Packet Capture) Library with LSF (Linux Socket Filtering)  Analysis of communication protocols  Analysis of the interactions between distributed entities  Error detection, performance rating, troubleshooting, etc.  The reduction in obtained data is done through filters  This way the overhead is reduced (because it doesnt have to copy so much data)  Only capture the relevant packets for analysis27.02.2012 Network Monitoring by Process id 3
  • 4. Energia Open Source Actual Network Monitoring User TCPDump User APP User APP User APP ... LibPCAP Kernel Network TCP/IP stack PACKET stack Before sending or receiving packets Hardware NIC27.02.2012 Network Monitoring by Process id 4
  • 5. Energia Open Source Actual Monitoring Mechanism  A filter is a set of rules  These rules can be based on ports, addresses, protocols, etc.  Or on a specific set of bytes  Rules are combined with logical operands  Changing the filter with libpcap  Necessary to drain the socket  Possible to lose packets  When filters are more complex they cant be applied on kernel space  All packets are captured and the filter is applied in userspace  Without specifying a filter the monitor will capture all packets27.02.2012 Network Monitoring by Process id 5
  • 6. Energia Open Source Challenges of network monitoring by process id  Monitoring the network activity of a process in user space is limited to some cases and can overload the system  Changing the filter by using libpcap has non-negligible latency Compile and Draining and pcap filter optimise attach the new filter 0 Time Latency when attaching a new filter27.02.2012 Network Monitoring by Process id 6
  • 7. Energia Open Source Network Monitoring by Process ID  New Approach towards the reduction of data for analysis  More specific analysis  Performance and overhead issues  When we only want a subset of the packets flowing on the network card  Can simplify the use of bpf filters27.02.2012 Network Monitoring by Process id 7
  • 8. Energia Open Source Why we need this  Analysis of closed protocols  No access to source code  Not easy to understand / follow some network protocols  Debug protocols when creating new ones  Determining if a program is leaking information  In production machines  Cannot stop servers for debug  Troubleshooting specific threads27.02.2012 Network Monitoring by Process id 8
  • 9. Energia Open Source Kernel Space Changes  Two Main Parts  Main kernel code  Created a hook to be attached by the filter function  Changed the filter_function at the end to call the hook  Module (MRoP – “Monitorização de Rede orientada ao Processo” [Pt])  KProbes handler functions  Repository (a RB-Tree)  Filter function  User space interface (through DebugFS)27.02.2012 Network Monitoring by Process id 9
  • 10. Energia Open Source Kernel Components  The Kernel module developed has 4 components  Syscalls hooked handlers Repository (of socket information)  ports and addresses Filter function DebugFS interface  Communication to/from user land  Statistics and control27.02.2012 Network Monitoring by Process id 10
  • 11. Energia Open Source New Filter Mechanism (by process id) Root user App DebugFS Monitor App tcpdump User appUser space PID, stats, etc Sending Kernel or receiving Packet Kernel packets stack filter module27.02.2012 Network Monitoring by Process id 11
  • 12. Energia Open Source Generic Process TCPDump Monitor Pcap Library Control API AF_PACKET KProbes Packets Instrumented Syscalls handlers Hook Repository Packet Filter NIC Function Driver NIC27.02.2012 Network Monitoring by Process id 12
  • 13. Energia Open Source KProbes  Dynamic Instrumentation Mechanism on kernel space  Different types of instrumentation based on what want to be achieved  Kprobe, Jprobe, KretProbe  int 3 instruction (trap)  Does not need Debug information  Uses kallsyms  To locate symbols  Overhead of 0.6 microseconds  per probe hit  Its a mechanism not a tool27.02.2012 Network Monitoring by Process id 13
  • 14. Energia Open Source Syscalls hooked handlers  Connect, accept, bind, recvfrom, sendto and sock_close function  KProbes is a kernel mechanism for instrumentation KRetProbes are probes that get the return value of the functions KRetProbes use a trampolin to catch the return value  Inside the handlers the computation must be very quick27.02.2012 Network Monitoring by Process id 14
  • 15. Energia Open Source Syscalls hooked handlers (II)  Inside handlers  Need to get the socket information  The sockets information is on the parameters of the syscall or on socket descriptor  The registers have the value of the socket descriptor  The socket information is written to the repository  The are two handlers, one on the entry of the function and the other on the return  This way on the return handler using the return value we are sure if the call was successful, if not the information on the repository regarding that socket is removed27.02.2012 Network Monitoring by Process id 15
  • 16. Energia Open Source Filter Function  Packets that will be accepted by the bpf program filter are evaluated by the module filter function  The filter function only evaluates TCP and UDP packets  Search the repository for the packet port If it finds verifies the protocol and the address27.02.2012 Network Monitoring by Process id 16
  • 17. Energia Open Source Repository  This repository is for socket information so that the filter can know if a specific packet belongs to the target process  Implemented using a Red and Black Tree  Mainly performance (must be searched once per packet)27.02.2012 Network Monitoring by Process id 17
  • 18. Energia Open Source Repository II27.02.2012 Network Monitoring by Process id 18
  • 19. Energia Open Source DebugFS interface to user land  Created a directory on the DebugFS Files for controlling the filtering mechanism Search the process structure and add information to the repository Clear the repository Identify which process to monitor Files for statistics/logging purposes How many packets passed/dropped by the filter How many times the handlers functions were called How many elements have the repository27.02.2012 Network Monitoring by Process id 19
  • 20. Energia Open Source Evaluation  Functional evaluation  Created small and specific programs to verify socket information on both sides (user space and kernel space)  Transfered data using http, ftp and iperf protocols  Data transferred with http and ftp was monitored and saved to a file  It was visualized on Wireshark  Application layer data was compared with the data sent using md5 and sha1 checksums27.02.2012 Network Monitoring by Process id 20
  • 21. Energia Open Source Evaluation II  Performance Evaluation  Evaluated the overhead using a 1GB transfer  While transferring data it was monitored and saved to a file  Overhead introduced:  Dynamic instrumentation  Managing the repository27.02.2012 Network Monitoring by Process id 21
  • 22. Energia Open Source Performance Evaluation  Two machines connected directly on a 100 Mbit/s link  Transferring 1GB data through that link using:  ftp, http protocols and iperf tool  Capturing only one flow of data  Capturing two flows of data (one being the relevant one)  Capturing only one flow using MRoP VS  Capturing two flows  Measured times:  Without monitoring (transfer only)  Using standard monitoring  Using MRoP (tcpdump + developed kernel module)27.02.2012 Network Monitoring by Process id 22
  • 23. Energia Open Source Performance Evaluation II27.02.2012 Network Monitoring by Process id 23
  • 24. Energia Open Source Performance Evaluation III 3.5%27.02.2012 Network Monitoring by Process id 24
  • 25. Energia Open Source Network Monitoring by Process ID  Created a new kernel module to extend functionality of network monitoring of a Process  Only captures the relevant packets for analysis  Low overhead  Maintains compatibility with old bpf filters  Shows better results when the analysis is focused on a subset of the network packets27.02.2012 Network Monitoring by Process id 25
  • 26. Energia Open Source More Integrated (Work in Progress)  This approach has some integration issues  It is necessary to use the debugfs to introduce process ids  Not integrated with libpcap  Filters still dont have a pid mnemonic  The core functionality will be kept27.02.2012 Network Monitoring by Process id 26
  • 27. Energia Open Source Generic Process TCPDump Monitor PCap Library Control API AF_PACKET KProbes Packets Instrumented Syscalls handlers Hook Repository Packet Filter NIC Function Driver NIC27.02.2012 Network Monitoring by Process id 27
  • 28. Energia Open Source Generic Process TCPDump Monitor Pcap Library Control API (as debug) AF_PACKET KProbes Packets Control API Instrumented Syscalls handlers Hook Packet Filter NIC Repository Function Driver NIC27.02.2012 Network Monitoring by Process id 28
  • 29. Energia Open Source Final Considerations  These changes benefit the new monitoring system based on process id  The mechanism of applying a new filter function may also benefit other kernel developers  Wireshark developers desire a mechanism to filter packets based on application id (they have it in their wish list)  Maybe it can be used to detect malware (since the instrumentation is done below the userspace)27.02.2012 Network Monitoring by Process id 29

×