0
Internet Traffic Monitoring and Analysis 홍 원 기 포항공과대학교 컴퓨터공학과  분산처리 및 네트워크관리 연구실 [email_address] http://dpnm.postech.ac.kr...
Table of Contents <ul><li>Introduction </li></ul><ul><li>Real-World Applications of Traffic Monitoring and Analysis </li><...
1. Introduction –  Growth of Internet Use <ul><li>The number of Internet users is growing </li></ul>Source : Nua Inc. Inte...
1. Introduction -  Evolving IP Network Environment <ul><li>WAN : SONET/SDH (OC3, OC12, OC48, OC192), ATM, WDM/DWDM </li></...
1. Introduction –  Reliance on Internet <ul><li>The Internet generated revenue has been increasing rapidly!  </li></ul>Sou...
<ul><li>Traditional Internet Applications </li></ul><ul><ul><li>Web, FTP, Email, Telnet, etc. </li></ul></ul><ul><li>Emerg...
<ul><li>Peer-to-Peer (P2P) </li></ul><ul><ul><li>New concept between file sharing and transferring </li></ul></ul><ul><ul>...
<ul><li>Bursty data transfer vs. Streaming data transfer </li></ul>1. Introduction –  Types of Traffic <ul><li>Static sess...
1. Introduction –  Motivation <ul><li>Needs of Customers </li></ul><ul><ul><li>Want to get their money’s worth </li></ul><...
1. Introduction –  Application Areas <ul><li>Network Problem Determination and Analysis </li></ul><ul><li>Traffic Report G...
1. Introduction –  Problems <ul><li>Capturing Packets </li></ul><ul><ul><li>How to capture all packets from high-speed, hi...
2. Real-World Applications -  Network Usage Analysis <ul><li>WAN Traffic Usage View </li></ul><ul><ul><li>Using MRTG </li>...
2. Real-World Applications -  Network Planning <ul><li>Network grows  in complexity and increases in usage   </li></ul><ul...
2. Real-World Applications -  Network Weather Service (Abilene) <ul><li>Abilene Network weather map of the traffic load on...
2. Real-World Applications -  Network Weather Service (AT&T) <ul><li>Network Performance Map on AT&T backbone network </li...
2. Real-World Applications -  SLA Monitoring <ul><li>Service Level Agreement (SLA) is a  contract between a network servic...
2. Real-World Applications -  Usage-based Billing <ul><li>On a typical broadband network, 5% of the customers consume over...
2. Real-World Applications -  CRM <ul><li>Customer Relationship Management ( CRM) is a  discipline  as well as  a set of d...
2. Real-World Applications -  Security <ul><li>Network Security Threats </li></ul><ul><ul><li>Reconnaissance   </li></ul><...
Code Red Worm (July 19, 2001) <ul><li>A famous example of a  TCP flood attack </li></ul><ul><li>Infected over 350,000 host...
Code Red Worm (July 19, 2001) <ul><li>Damages </li></ul><ul><ul><li>Various service & network outages throughout the world...
Sapphire/Slammer Worm (Jan 25, 2003) <ul><li>A famous example of an  UDP flood attack </li></ul><ul><li>Exploited a buffer...
Sapphire/Slammer Worm (Jan 25, 2003) <ul><li>Damages </li></ul><ul><ul><li>Since the worm did not contain malicious payloa...
3. POSTECH R&D Activities in Traffic Monitoring <ul><li>MRTG+ </li></ul><ul><li>WebTrafMon I </li></ul><ul><li>WebTrafMon ...
MRTG+ <ul><li>Network  link utilization  monitoring, analysis & reporting system  </li></ul><ul><li>Extended Multi-Router ...
MRTG+ Architecture
MRTG+ Network Sensitive Map (1997)
Link Utilization Output
WebTrafMon <ul><li>Web -based IP Network  Tra ffic  Mon itoring System  </li></ul><ul><li>Developed at DPNM Lab, POSTECH <...
WebTrafMon-I Features <ul><li>Web-based User Interface </li></ul><ul><li>Real-time and short-term analysis  </li></ul><ul>...
WebTrafMon-I Architecture
WebTrafMon-I User Interface
WebTrafMon-I Limitations <ul><li>All in one system </li></ul><ul><ul><li>cause  packet loss  and  response and analysis ti...
WebTrafMon-II Requirements <ul><li>Improve the limitations of WebTrafMon-I </li></ul><ul><li>No packet loss  in probe </li...
WebTrafMon-II Architecture database Traffic  analyzer (minutely,  hourly, daily, monthly, yearly) probe network point prom...
WebTrafMon-II User Interface
WebTrafMon-II Limitations <ul><li>Takes long time to analyze high-speed, high-volume traffic </li></ul><ul><li>Takes long ...
4. NG-MON <ul><li>History </li></ul><ul><ul><li>MRTG+  (1996-97) </li></ul></ul><ul><ul><ul><li>Traffic load analysis with...
NG-MON -  Requirements <ul><li>Distributed, load-balancing architecture  for scalability </li></ul><ul><ul><li>subdivide m...
NG-MON -  Design <ul><li>NG-MON is composed of 5 phases </li></ul><ul><ul><li>Packet Capture </li></ul></ul><ul><ul><li>Fl...
NG-MON -  Packet Capture Network Link Splitting Device divided raw packet pkt header messages <ul><li>Distribution of raw ...
NG-MON -  Flow Generation <ul><li>Distribution of packet header information </li></ul><ul><ul><li>5-tuple based hashing in...
NG-MON -  Flow Store <ul><li>Separation of  write operations  from  read operations </li></ul><ul><ul><li>the destination ...
NG-MON -  Traffic Analysis & Presentation <ul><li>Analyzer extracts information from Flow Stores and can perform applicati...
NG-MON -  Implementation Phase Packet Capture Flow Generator Flow Store Analyzer Presenter Development Tool pcap library C...
NG-MON -  Deployment at POSTECH http://ngmon.postech.ac.kr Packet Capture Flow  Generator Flow  Store Analyzer Presenter 1...
NG-MON -  Host Data Received Minute View
NG-MON -  Host Data Exchanged Minute View
NG-MON -  Detailed Subnet Data Sent Minute View
NG-MON -   Application Protocol Minute View
NG-MON -   Time Series Minute View
5. Summary <ul><li>Internet is continuously growing  in terms of: # of users & hosts, traffic loads & types </li></ul><ul>...
NG-Mon Demo <ul><li>http://ngmon.postech.ac.kr </li></ul>
Upcoming SlideShare
Loading in...5
×

Internet Traffic Monitoring and Analysis

6,856

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
6,856
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
71
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Abstract Most Internet networking devices are now equipped with a Web server for providing Web-based element management so that an administrator may take advantage of this enhanced and powerful management interface. On the other hand, for network management, an administrator normally buys and deploys SNMP-based network management platform to be customized to his network. Each management scheme has mutually exclusive advantages; consequently, two schemes coexist in the real world. This results in both a high development cost and a dual management interface for administrator. We propose an embedded Web server (EWS)-based network management architecture as an alternative to an SNMP based network management and to leverage on already existing embedded web server. We extend EWS-based element management architecture to the network management architecture. Our proposed architecture uses HTTP as a communication protocol with management information and operation encoding. Further we designed a management system on the basis of our proposed architecture that supports basic management functions.
  • ------------------------------------------------------------------------------------------------------------------------------------------------------------- [http://http://www.caida.org/outreach/metricswg/faq.xml] CAIDA  Outreach  Network Measurement FAQ 2.1. Why should I measure my network&apos;s behaviour? If you don&apos;t measure it, you have no objective record or benchmark of how it behaves. This could make it difficult to judge whether changes in the network have improved its performance, or degraded it. If you are buying Internet connectivity from an ISP you need to understand the kind of service being offered, and you need to measure the actual performance so as to verify that you&apos;re getting what you pay for. ------------------------------------------------------------------------------------------------------------------------------------------------------------- [KRNET Tutorial] http://dpnm.postech.ac.kr/webboard/ Internet Traffic Monitoring &amp; Analysis User’s Needs * Monitor the performance experienced by one ’ s application - Why is the web page download so slow? - Why is my multicast video stream jerky? * Check if level of service meets one ’ s need - Do I have enough b/w? * Check if one experiences intrusions and attacks - Is someone attacking me? Service provider ’ s needs * Monitor the current level of activity * Enforce SLAs(service level agreements) * Detect faults and failures * Engineer the network for better performance * Plan for future capacity * Feedback to customers -----------------------------------------------------------------------------------------------------------------------
  • ------------------------------------------------------------------------------------------------------------------------------------------------------------- [http://http://www.caida.org/outreach/metricswg/faq.xml] CAIDA  Outreach  Network Measurement FAQ 2.1. Why should I measure my network&apos;s behaviour? If you don&apos;t measure it, you have no objective record or benchmark of how it behaves. This could make it difficult to judge whether changes in the network have improved its performance, or degraded it. If you are buying Internet connectivity from an ISP you need to understand the kind of service being offered, and you need to measure the actual performance so as to verify that you&apos;re getting what you pay for. ------------------------------------------------------------------------------------------------------------------------------------------------------------- [KRNET Tutorial] http://dpnm.postech.ac.kr/webboard/ Internet Traffic Monitoring &amp; Analysis User’s Needs * Monitor the performance experienced by one ’ s application - Why is the web page download so slow? - Why is my multicast video stream jerky? * Check if level of service meets one ’ s need - Do I have enough b/w? * Check if one experiences intrusions and attacks - Is someone attacking me? Service provider ’ s needs * Monitor the current level of activity * Enforce SLAs(service level agreements) * Detect faults and failures * Engineer the network for better performance * Plan for future capacity * Feedback to customers -----------------------------------------------------------------------------------------------------------------------
  • ------------------------------------------------------------------------------------------------------------------------------------------------------------- [http://http://www.caida.org/outreach/metricswg/faq.xml] CAIDA  Outreach  Network Measurement FAQ 2.1. Why should I measure my network&apos;s behaviour? If you don&apos;t measure it, you have no objective record or benchmark of how it behaves. This could make it difficult to judge whether changes in the network have improved its performance, or degraded it. If you are buying Internet connectivity from an ISP you need to understand the kind of service being offered, and you need to measure the actual performance so as to verify that you&apos;re getting what you pay for. ------------------------------------------------------------------------------------------------------------------------------------------------------------- [KRNET Tutorial] http://dpnm.postech.ac.kr/webboard/ Internet Traffic Monitoring &amp; Analysis User’s Needs * Monitor the performance experienced by one ’ s application - Why is the web page download so slow? - Why is my multicast video stream jerky? * Check if level of service meets one ’ s need - Do I have enough b/w? * Check if one experiences intrusions and attacks - Is someone attacking me? Service provider ’ s needs * Monitor the current level of activity * Enforce SLAs(service level agreements) * Detect faults and failures * Engineer the network for better performance * Plan for future capacity * Feedback to customers -----------------------------------------------------------------------------------------------------------------------
  • ------------------------------------------------------------------------------------------------------------------------------------------------------------- [http://http://www.caida.org/outreach/metricswg/faq.xml] CAIDA  Outreach  Network Measurement FAQ 2.1. Why should I measure my network&apos;s behaviour? If you don&apos;t measure it, you have no objective record or benchmark of how it behaves. This could make it difficult to judge whether changes in the network have improved its performance, or degraded it. If you are buying Internet connectivity from an ISP you need to understand the kind of service being offered, and you need to measure the actual performance so as to verify that you&apos;re getting what you pay for. ------------------------------------------------------------------------------------------------------------------------------------------------------------- [KRNET Tutorial] http://dpnm.postech.ac.kr/webboard/ Internet Traffic Monitoring &amp; Analysis User’s Needs * Monitor the performance experienced by one ’ s application - Why is the web page download so slow? - Why is my multicast video stream jerky? * Check if level of service meets one ’ s need - Do I have enough b/w? * Check if one experiences intrusions and attacks - Is someone attacking me? Service provider ’ s needs * Monitor the current level of activity * Enforce SLAs(service level agreements) * Detect faults and failures * Engineer the network for better performance * Plan for future capacity * Feedback to customers -----------------------------------------------------------------------------------------------------------------------
  • ------------------------------------------------------------------------------------------------------------------------------------------------------------- [http://http://www.caida.org/outreach/metricswg/faq.xml] CAIDA  Outreach  Network Measurement FAQ 2.1. Why should I measure my network&apos;s behaviour? If you don&apos;t measure it, you have no objective record or benchmark of how it behaves. This could make it difficult to judge whether changes in the network have improved its performance, or degraded it. If you are buying Internet connectivity from an ISP you need to understand the kind of service being offered, and you need to measure the actual performance so as to verify that you&apos;re getting what you pay for. ------------------------------------------------------------------------------------------------------------------------------------------------------------- [KRNET Tutorial] http://dpnm.postech.ac.kr/webboard/ Internet Traffic Monitoring &amp; Analysis User’s Needs * Monitor the performance experienced by one’s application - Why is the web page download so slow? - Why is my multicast video stream jerky? * Check if level of service meets one’s need - Do I have enough b/w? * Check if one experiences intrusions and attacks - Is someone attacking me? Service provider’s needs * Monitor the current level of activity * Enforce SLAs(service level agreements) * Detect faults and failures * Engineer the network for better performance * Plan for future capacity * Feedback to customers -----------------------------------------------------------------------------------------------------------------------
  • ------------------------------------------------------------------------------------------------------------------------------------------------------------- [http://http://www.caida.org/outreach/metricswg/faq.xml] CAIDA  Outreach  Network Measurement FAQ 2.1. Why should I measure my network&apos;s behaviour? If you don&apos;t measure it, you have no objective record or benchmark of how it behaves. This could make it difficult to judge whether changes in the network have improved its performance, or degraded it. If you are buying Internet connectivity from an ISP you need to understand the kind of service being offered, and you need to measure the actual performance so as to verify that you&apos;re getting what you pay for. ------------------------------------------------------------------------------------------------------------------------------------------------------------- [KRNET Tutorial] http://dpnm.postech.ac.kr/webboard/ Internet Traffic Monitoring &amp; Analysis User’s Needs * Monitor the performance experienced by one ’ s application - Why is the web page download so slow? - Why is my multicast video stream jerky? * Check if level of service meets one ’ s need - Do I have enough b/w? * Check if one experiences intrusions and attacks - Is someone attacking me? Service provider ’ s needs * Monitor the current level of activity * Enforce SLAs(service level agreements) * Detect faults and failures * Engineer the network for better performance * Plan for future capacity * Feedback to customers -----------------------------------------------------------------------------------------------------------------------
  • To monitor high speed network such 10Gpbs link, the NG-MON should consider these 5 significant requirements. The first one, as stated, NG-MON needs distributed, load-balancing architecture. To distribute the processing load , we should divide monitoring and analysis task into several functional units, and we also need an efficient load sharing mechanism within each phase. For load distribution method , we considered the pipeline and parallel methods. The second is lossless packet capture . NG-MON should capture all packets without a loss to provide all the required information to various analysis applications. The fourth one is, to reduce processing load , flow based analysis is essential. by the flow-based analysis, NG-MON can aggregate packet information into flows for efficient processing. Also, limited storage at each phase should be considered. By the consideration of these requirements we designed the architecture of NG-MON.
  • This is an overall architecture of NG-MON design. The key feature in our design is an pipelined distribution and load balancing technique. Whole tasks are divided into 5 phases like this. Packet capture, Flow Generation, Flow Store, Traffic Analysis and Presentation phase. The entire raw packets are captured in the Packet Capture phase. And packet header information extracted from raw packets are delivered to the second phase: Flow Generation phase, The flow information is generated in this Flow Generation phase. the flow information is stored in the Flow Store phase. Traffic Analyzer queries to Flow Store and store analyzed data, provide them to Presenter. Load distribution mechanism used in each phase will be explained in the following slides in detail.
  • This slide shows the first phase of our NG-MON design: packet capture phase. Large bulk traffic on the network links is distributed over probe systems and sent to next phase, Flow Generation. In the distribution of raw packets we can use one of these methods. First one is by using splitting function provided by an optical splitter. And Using mirroring functions provided by network devices is the second one. These probe systems captures incoming packets and extract packet header information form layered headers of each raw packet, then push into the export buffer-queues by packet header’s 5-tuple based hashing. Each probe system maintain the same number of buffer queues corresponding to the number of flow generators. If a buffer queue becomes full , probe constructs packet header messages then export to next phase. The raw packets with the same color indicates that they belong to the same flow. As you can see, packets which belong to the same flow put together into the same packet header messages. ( 5-tuple : src &amp; dst address, protocol number, src &amp; dst port number )
  • This and next slides shows the second phases of our NG-MON design. In this phase, packet headers are compressed into flows. For the distribution of packet header information, we used 5-tuple based hashing and buffer queue for each flow generator. Therefore the packet header information of potentially the same flow get delivered to the same flow generator. There can’t be the case that same flow is generated in different flow generator at a certain moment. Flow generators simply generate flow messages from incoming packet header messages, then exports these to next phase, flow store.
  • This slide shows the third phase of our NG-MON architecture: Flow Store phase The main role of Flow Store phase is to store flow information and handle the request from analyzer: those are write operation and read operation . For the load distribution and efficient processing , we considered a method that prevent write operations from occurring with read operations at the same time in a single flow store system. In order to do this, the destination address of flow messages should be changed over to Flow Store sequentially depending on the time slot changes. While one or more flow stores are inserting flow data, the other flow stores are queried by the traffic analyzers. As you can see here , at the time slot t1, Flow Store 1 only receives flow messages and the other Flow Stores are processing queries from Analyzers. Before the time slot changes from t1 to t2, queries to Flow Store 2 should be finished. Then the time slot becomes t2, flow messages will go into the Flow Store 2, and queries to Flow Store 1 will be started. In our earlier work , we realized that one of the bottleneck of the monitoring process is a huge storage space required. So, Flow Store keeps flow information for only several time slots, and then discard them when they are finished an analysis by traffic analyzers. Therefore, flow store only requires a small and fixed amount of disk space. Flow store provides traffic information to support various analysis applications and provide an analysis API to analyzers.
  • This slide shows the fourth and fifth phases of our NG-MON architecture. These two phases are tightly coupled according to the analysis purpose; such as Traffic Throughput Analysis, Usage-based billing analysis, DDOS and DOS attack analysis, such like that. Analyzer extracts information from Flow Stores and can perform application specific analysis . Separate analyzer is needed for each application. we separated the presenter from traffic analyzer, because more than one systems tend to be allocated in the traffic analysis phase.
  • In this summer We implemented a prototype of NG-MON and deployed our system in our campus backbone network. In the implementation, we used Net Optics’ Gigabit Fiber Optic tap to split the traffic and used GE Card to get it. The hardware configuration we used are, P-III 800MHz, 256 Mbytes memory, 20Gbytes HD. And we developed our system on Redhat Linux 7.2 OS. And used C language with pcap library in Packet Capture phase. In the Flow Store, we used MySQL Database to store flows. Presenter uses PHP with jpgraph library to present the analysis result through the web.
  • This and other two slides show some selected screen shots of our prototype implementation. Our analyzer shows various throughput information according to the HOST, SUBNET, and PROTOCOL. This screen shot shows the throughput of host received in one minute, and total throughput changes by the time in one hour is illustrated in the form of line graph.
  • This and other two slides show some selected screen shots of our prototype implementation. Our analyzer shows various throughput information according to the HOST, SUBNET, and PROTOCOL. This screen shot shows the throughput of host received in one minute, and total throughput changes by the time in one hour is illustrated in the form of line graph.
  • This is a detailed subnet data sent view in a certain minute.
  • Left one is an application protocol view in a certain minute. And right one is Time series graph of the throughput at each protocol layer during a certain hour.
  • Left one is an application protocol view in a certain minute. And right one is Time series graph of the throughput at each protocol layer during a certain hour.
  • Transcript of "Internet Traffic Monitoring and Analysis"

    1. 1. Internet Traffic Monitoring and Analysis 홍 원 기 포항공과대학교 컴퓨터공학과 분산처리 및 네트워크관리 연구실 [email_address] http://dpnm.postech.ac.kr/ Tel: 054-279-2244
    2. 2. Table of Contents <ul><li>Introduction </li></ul><ul><li>Real-World Applications of Traffic Monitoring and Analysis </li></ul><ul><li>POSTECH R&D Activities in Traffic Monitoring and Analysis </li></ul><ul><li>NG-MON: N ext G eneration Network Traffic MON itoring and Analysis System </li></ul><ul><li>Summary </li></ul>
    3. 3. 1. Introduction – Growth of Internet Use <ul><li>The number of Internet users is growing </li></ul>Source : Nua Inc. Internet traffic has increased dramatically Source: America’s Network
    4. 4. 1. Introduction - Evolving IP Network Environment <ul><li>WAN : SONET/SDH (OC3, OC12, OC48, OC192), ATM, WDM/DWDM </li></ul><ul><li>LAN : 10/100 Mbps to 1 Gbps to 10 Gbps Ethernet </li></ul><ul><li>Broadband Internet Access : Cable Modem, ADSL, VDSL </li></ul><ul><li>Wireless Access : WLAN (IEEE 802.11), Wireless Internet </li></ul><ul><li>Wired/Wireless Convergence : Softswitch, Media Gateway, NGCN </li></ul>
    5. 5. 1. Introduction – Reliance on Internet <ul><li>The Internet generated revenue has been increasing rapidly! </li></ul>Source : Active Media. <ul><li>Internet’s importance and reliance are increasing! </li></ul>
    6. 6. <ul><li>Traditional Internet Applications </li></ul><ul><ul><li>Web, FTP, Email, Telnet, etc. </li></ul></ul><ul><li>Emerging Internet applications </li></ul><ul><ul><li>Online games, shopping, banking, stock trading, network storage </li></ul></ul><ul><ul><li>VOD, EOD, VoIP </li></ul></ul><ul><ul><li>P2P applications – instant messaging, file sharing </li></ul></ul>1. Introduction – Internet Applications Online game VoIP VOD
    7. 7. <ul><li>Peer-to-Peer (P2P) </li></ul><ul><ul><li>New concept between file sharing and transferring </li></ul></ul><ul><ul><li>Generates high volume of traffic </li></ul></ul>1. Introduction – Structure of Applications <ul><li>Structures of applications are changing! </li></ul><ul><li>Client-Server </li></ul><ul><ul><li>Traditional structure </li></ul></ul>client server peer discovery, content, transfer query peer peer
    8. 8. <ul><li>Bursty data transfer vs. Streaming data transfer </li></ul>1. Introduction – Types of Traffic <ul><li>Static sessions vs. Dynamic sessions </li></ul>packet network packet <ul><li>Types of traffic are various and increasing! </li></ul>Negotiate & allocate connect disconnect use dynamic protocol, port data connect disconnect control use static protocol, port network
    9. 9. 1. Introduction – Motivation <ul><li>Needs of Customers </li></ul><ul><ul><li>Want to get their money’s worth </li></ul></ul><ul><ul><li>Fast, reliable, high-quality, secure, virus-free Internet access </li></ul></ul><ul><li>Needs of Service Providers </li></ul><ul><ul><li>Understand the behavior of their networks </li></ul></ul><ul><ul><li>Provide fast, high-quality, reliable service to satisfy customers and thus reduce churn rate </li></ul></ul><ul><ul><li>Plan for network deployment and expansion </li></ul></ul><ul><ul><li>SLA monitoring </li></ul></ul><ul><ul><li>Network security attack detection and prevention </li></ul></ul>
    10. 10. 1. Introduction – Application Areas <ul><li>Network Problem Determination and Analysis </li></ul><ul><li>Traffic Report Generation </li></ul><ul><li>Intrusion & Hacking Attack (e.g., DoS, DDoS) Detection </li></ul><ul><li>Service Level Monitoring (SLM) </li></ul><ul><li>Network Planning </li></ul><ul><li>Usage-based Billing </li></ul><ul><li>Customer Relationship Management (CRM) </li></ul><ul><li>Marketing </li></ul>
    11. 11. 1. Introduction – Problems <ul><li>Capturing Packets </li></ul><ul><ul><li>How to capture all packets from high-speed, high volume networks (Mbps  Gbps  Tbps)? </li></ul></ul><ul><li>Flow Generation & Storage </li></ul><ul><ul><li>What packet information to save to perform various analysis? </li></ul></ul><ul><ul><li>How to minimize storage requirements? </li></ul></ul><ul><li>Analysis </li></ul><ul><ul><li>How to analyze and generate data needed quickly? </li></ul></ul><ul><ul><li>Streaming media (Windows Media, Real, Quicktime) </li></ul></ul><ul><ul><li>P2P traffic </li></ul></ul><ul><ul><li>Network Security Attacks </li></ul></ul>
    12. 12. 2. Real-World Applications - Network Usage Analysis <ul><li>WAN Traffic Usage View </li></ul><ul><ul><li>Using MRTG </li></ul></ul><ul><ul><li>At Internet Junction </li></ul></ul><ul><li>Time Series Data View </li></ul><ul><ul><li>Daily </li></ul></ul><ul><ul><li>Monthly </li></ul></ul><ul><ul><li>Weekly </li></ul></ul><ul><ul><li>Yearly </li></ul></ul>Internet Traffic Usage View
    13. 13. 2. Real-World Applications - Network Planning <ul><li>Network grows in complexity and increases in usage </li></ul><ul><ul><li>difficult to predict usage trends and loading on individual segments </li></ul></ul><ul><ul><li>Previously, SNMP was the only tool available to service providers seeking access to usage statistics - severely limited </li></ul></ul><ul><li>For accurate network capacity planning </li></ul><ul><ul><li>Service providers must have access to in-depth info about their networks </li></ul></ul><ul><ul><li>Network bottleneck details broken down into bandwidth used vs. bandwidth available </li></ul></ul><ul><ul><li>Detailed network usage history reports </li></ul></ul><ul><ul><li>A complete view of current use </li></ul></ul><ul><ul><li>Analytical tools to analyze and predict usage trends </li></ul></ul>
    14. 14. 2. Real-World Applications - Network Weather Service (Abilene) <ul><li>Abilene Network weather map of the traffic load on the core links </li></ul><ul><li>Measurement Method: SNMP </li></ul><ul><li>http://loadrunner.uits.iu.edu/weathermaps/abilene/ </li></ul>Courtesy of the Abilene Network Operations Center, Indiana University
    15. 15. 2. Real-World Applications - Network Weather Service (AT&T) <ul><li>Network Performance Map on AT&T backbone network </li></ul><ul><ul><li>http://ipnetwork.bgtmo.ip.att.net/ </li></ul></ul><ul><li>Measured Metrics </li></ul><ul><ul><li>Round Trip Delay </li></ul></ul><ul><ul><li>Packet Loss </li></ul></ul><ul><ul><li>Availability </li></ul></ul><ul><li>Measurement Method </li></ul><ul><ul><li>ICMP based tools </li></ul></ul><ul><ul><ul><li>Ping, Traceroute </li></ul></ul></ul><ul><ul><li>Every 30 minutes </li></ul></ul><ul><li>The left figure shows the latency and loss rate from Atlanta to all the other major cities in USA </li></ul>
    16. 16. 2. Real-World Applications - SLA Monitoring <ul><li>Service Level Agreement (SLA) is a contract between a network service provider and a customer that specifies, usually in measurable terms, what services the network service provider will furnish. </li></ul><ul><li>SLA Life Cycle </li></ul><ul><li>SLA Negotiation </li></ul><ul><ul><li>Using QoS Parameters </li></ul></ul><ul><li>SLA Implementation (provisioning) </li></ul><ul><ul><li>Network provisioning using QoS technology such as Diffserv, Intserv, MPLS, etc. </li></ul></ul><ul><ul><li>Service configuration </li></ul></ul><ul><li>SLA Execution and Monitoring </li></ul><ul><ul><li>QoS Parameter to Network Performance Metric Mapping </li></ul></ul><ul><ul><li>SLA violation handling </li></ul></ul><ul><ul><li>Real-time reporting </li></ul></ul>Product/Service Development Negotiation Sales Implementation Execution Monitoring Assessment
    17. 17. 2. Real-World Applications - Usage-based Billing <ul><li>On a typical broadband network, 5% of the customers consume over 50% of the bandwidth. </li></ul>Gas Telephone Electricity Can you imagine your telephone , electricity and gas not being metered and priced by usage? What about the services provided by current NSP and ISP? Such as VPN, broadband Internet (xDSL, Cable Modem) These services are charged using a flat-fee billing model . Is this situation is reasonable?
    18. 18. 2. Real-World Applications - CRM <ul><li>Customer Relationship Management ( CRM) is a discipline as well as a set of discrete software and technologies that focus on automating and improving the business processes associated with managing customer relationships in the areas of sales, marketing, customer service and support. </li></ul><ul><li>Business Objectives of CRM </li></ul><ul><ul><li>Increased efficiency through automation </li></ul></ul><ul><ul><li>The ability to provide faster response to customer inquiries </li></ul></ul><ul><ul><li>Having a deeper knowledge of customer needs </li></ul></ul><ul><ul><li>Generating more marketing or cross-selling opportunities </li></ul></ul><ul><ul><li>Better information for better management </li></ul></ul><ul><ul><li>Reduced cost of sales and increased productivity of sales representatives </li></ul></ul><ul><ul><li>Receiving customer feedback that leads to new and improved products or services </li></ul></ul><ul><li>Traffic Monitoring for CRM </li></ul><ul><ul><li>basic technology to get the customer’s network usage pattern and using for target marketing </li></ul></ul>
    19. 19. 2. Real-World Applications - Security <ul><li>Network Security Threats </li></ul><ul><ul><li>Reconnaissance </li></ul></ul><ul><ul><ul><li>probing or mapping the network to identify targets (e.g., ping and port scans, usually a precursor to an actual exploit attempt) </li></ul></ul></ul><ul><ul><li>Denial of Service (DoS) </li></ul></ul><ul><ul><ul><li>Attempts to consume bandwidth or computing resources in order to prevent a host communicating on the network (e.g., Smurf attacks or SYN floods) </li></ul></ul></ul><ul><ul><li>Distributed DoS (DDoS) </li></ul></ul><ul><ul><ul><li>Very similar to DoS, except that the attack originates from multiple machines </li></ul></ul></ul><ul><ul><li>Exploits </li></ul></ul><ul><ul><ul><li>Attempts to gain access to or compromise systems on the network, often seen as repeated failed login attempts </li></ul></ul></ul><ul><ul><li>Misuse </li></ul></ul><ul><ul><ul><li>Attempts to violate organizational policy (e.g., using disallowed services or including unauthorized content in e-mail or ftp transfers) </li></ul></ul></ul>
    20. 20. Code Red Worm (July 19, 2001) <ul><li>A famous example of a TCP flood attack </li></ul><ul><li>Infected over 350,000 hosts over a week </li></ul><ul><ul><li>The infection rate was doubling in about 37 minutes </li></ul></ul><ul><ul><li>The first incarnation of the Code-Red worm (CRv1) began to infect hosts running unpatched versions of MS IIS web server on July 12th, 2001. The first version of the worm uses a static seed for it's random number generator. </li></ul></ul><ul><ul><li>Then, around 10:00 UTC in the morning of July 19th, 2001, a random seed variant of the Code-Red worm (CRv2) appeared and spread. </li></ul></ul><ul><li>MS Windows machines were vulnerable </li></ul><ul><ul><li>Microsoft web servers MS Windows NT 4.0 IIS 4.0, Windows 2000 IIS 5.0 and Windows XP beta IIS 6.0 were all susceptible to the Index Server ISAPI vulnerability which could be used to take control of a server by specially formatting a web page request. </li></ul></ul><ul><li>The worm's original purpose was to perform a denial-of-service attack against www.whitehouse.gov. </li></ul>
    21. 21. Code Red Worm (July 19, 2001) <ul><li>Damages </li></ul><ul><ul><li>Various service & network outages throughout the world </li></ul></ul><ul><ul><li>The economic cost of the original Code Red worm and its more malicious cousin, Code Red II, was more than $2 Billion US, according to Computer Economics. </li></ul></ul><ul><ul><li>&quot;the most expensive virus in the history of the Internet&quot; </li></ul></ul>
    22. 22. Sapphire/Slammer Worm (Jan 25, 2003) <ul><li>A famous example of an UDP flood attack </li></ul><ul><li>Exploited a buffer overflow vulnerability in computers on the Internet running MS SQL Server or MSDE 2000 (MS Desktop Engine) </li></ul><ul><ul><li>Sent UDP packet (376 bytes size of very small worm) with destination UDP port 1434 . </li></ul></ul><ul><ul><li>No response required from the receiving machine. </li></ul></ul><ul><li>Spreading strategy was based on random IP scanning </li></ul><ul><ul><li>It selects IP addresses at random to infect, eventually finding all susceptible hosts. </li></ul></ul><ul><ul><li>Infected more than 90% of vulnerable hosts in the world within 10 minutes. The worm infected at least 100,000 hosts </li></ul></ul><ul><ul><li>Propagation was two orders of magnitude faster than the Code Red </li></ul></ul><ul><ul><li>“ The fastest spreading worm in the history of the Internet” </li></ul></ul>
    23. 23. Sapphire/Slammer Worm (Jan 25, 2003) <ul><li>Damages </li></ul><ul><ul><li>Since the worm did not contain malicious payload, it fortunately did not damage the data on the compromised machines </li></ul></ul><ul><ul><li>Saturated network links causing network and service outages </li></ul></ul><ul><ul><li>Caused big financial damages (hundreds of millions of dollars) to Internet-based businesses (such as Internet shopping malls, on-line paid contents services - games, movies) </li></ul></ul>
    24. 24. 3. POSTECH R&D Activities in Traffic Monitoring <ul><li>MRTG+ </li></ul><ul><li>WebTrafMon I </li></ul><ul><li>WebTrafMon II </li></ul><ul><li>NG-MON </li></ul>
    25. 25. MRTG+ <ul><li>Network link utilization monitoring, analysis & reporting system </li></ul><ul><li>Extended Multi-Router Traffic Grapher (MRTG) </li></ul><ul><ul><li>added security , threshold reporting & sensitive map </li></ul></ul><ul><li>uses Web browser, Web server & SNMP agents </li></ul><ul><li>generates HTML pages containing GIF images which provide a LIVE visual representation traffic </li></ul><ul><li>based on Perl and C </li></ul><ul><li>being used to monitor POSTECH, POSCO enterprise networks since 1997 </li></ul>
    26. 26. MRTG+ Architecture
    27. 27. MRTG+ Network Sensitive Map (1997)
    28. 28. Link Utilization Output
    29. 29. WebTrafMon <ul><li>Web -based IP Network Tra ffic Mon itoring System </li></ul><ul><li>Developed at DPNM Lab, POSTECH </li></ul><ul><li>In 1998, WebTrafMon I was first designed and developed to complement MRTG+ </li></ul><ul><li>In 1999, upgrading and performance tuning of WebTrafMon I </li></ul><ul><li>In 2000-2001, WebTrafMon II with a new architecture and a new implementation </li></ul><ul><li>In 2002, NG-MON is being developed </li></ul><ul><li>WebTrafMon has been deployed at the LAN-Internet junction of POSTECH campus network </li></ul><ul><li>WebTrafMon provides comprehensive information </li></ul><ul><ul><li>Spatial, temporal and composition analysis </li></ul></ul><ul><ul><li>Detailed analysis of traffic by minute, hour, day, month, year </li></ul></ul><ul><ul><li>By protocols (network, transport, application & service) </li></ul></ul>
    30. 30. WebTrafMon-I Features <ul><li>Web-based User Interface </li></ul><ul><li>Real-time and short-term analysis </li></ul><ul><li>Packet capture with sampling (1/10, 1/100, etc.) </li></ul><ul><li>Analysis Feature </li></ul><ul><ul><li>MAC Layer: Packet Size </li></ul></ul><ul><ul><li>Network Layer : IP, ARP, RARP </li></ul></ul><ul><ul><li>Transport Layer: TCP, UDP </li></ul></ul><ul><ul><li>Application Layer: Telnet, FTP, HTTP, SMTP, DNS… </li></ul></ul>
    31. 31. WebTrafMon-I Architecture
    32. 32. WebTrafMon-I User Interface
    33. 33. WebTrafMon-I Limitations <ul><li>All in one system </li></ul><ul><ul><li>cause packet loss and response and analysis time delay </li></ul></ul>analysis network interface user network traffic data packet header information analyzed information capture presentation All in a single server Long Analysis Time Response Time Delay Packet Loss
    34. 34. WebTrafMon-II Requirements <ul><li>Improve the limitations of WebTrafMon-I </li></ul><ul><li>No packet loss in probe </li></ul><ul><li>Real-time and long-term analysis </li></ul><ul><li>Reduced analysis and response time </li></ul><ul><li>Distributed Load Sharing Architecture </li></ul>capture presentation user network interface packet header information network traffic data distributed environment analysis
    35. 35. WebTrafMon-II Architecture database Traffic analyzer (minutely, hourly, daily, monthly, yearly) probe network point promiscuous mode packet capture hash log format and save into DB user distributed environment request response packet header information log file log format port information port information make short term, long term traffic data minutely minutely hourly, daily, monthly, yearly statistics network traffic data analyzer Flow generator
    36. 36. WebTrafMon-II User Interface
    37. 37. WebTrafMon-II Limitations <ul><li>Takes long time to analyze high-speed, high-volume traffic </li></ul><ul><li>Takes long time to generate presentation pages </li></ul><ul><li>Analyzer does not support multiple probes </li></ul><ul><li>High overhead in the NFS file system </li></ul>Need for NG-Mon (Next Generation Monitoring) System
    38. 38. 4. NG-MON <ul><li>History </li></ul><ul><ul><li>MRTG+ (1996-97) </li></ul></ul><ul><ul><ul><li>Traffic load analysis with sensitive map </li></ul></ul></ul><ul><ul><li>WebTrafMon-I (1997-98) </li></ul></ul><ul><ul><ul><li>Traffic type analysis on a single monolithic system (up to 10 Mbps) </li></ul></ul></ul><ul><ul><li>WebTrafMon-II (1999-2001) </li></ul></ul><ul><ul><ul><li>Traffic type analysis using a distributed architecture (up to 100 Mbps) </li></ul></ul></ul><ul><li>NG-MON (2002-present) </li></ul><ul><ul><li>N ext G eneration Network Traffic MON itoring and Analysis System </li></ul></ul><ul><ul><li>Targeting 10 Gbps or higher networks </li></ul></ul><ul><ul><li>To support various analysis applications </li></ul></ul><ul><ul><ul><li>Streaming media, multimedia conferencing, P2P, game traffic analysis </li></ul></ul></ul><ul><ul><ul><li>Network security attack detection and analysis </li></ul></ul></ul><ul><ul><ul><li>SLA monitoring </li></ul></ul></ul><ul><ul><ul><li>Usage-based billing </li></ul></ul></ul><ul><ul><ul><li>Customer relationship management </li></ul></ul></ul>
    39. 39. NG-MON - Requirements <ul><li>Distributed, load-balancing architecture for scalability </li></ul><ul><ul><li>subdivide monitoring system into several functional components </li></ul></ul><ul><ul><li>efficient load sharing between phases and within each phase </li></ul></ul><ul><ul><li>pipelined and parallel architecture </li></ul></ul><ul><li>Lossless packet capture </li></ul><ul><li>Flow-based analysis </li></ul><ul><ul><li>aggregate packet information into flows for efficient processing </li></ul></ul><ul><li>Support for various applications </li></ul><ul><li>Considerations for small storage requirements </li></ul>
    40. 40. NG-MON - Design <ul><li>NG-MON is composed of 5 phases </li></ul><ul><ul><li>Packet Capture </li></ul></ul><ul><ul><li>Flow Generation </li></ul></ul><ul><ul><li>Flow Store </li></ul></ul><ul><ul><li>Traffic Analysis </li></ul></ul><ul><ul><li>Presentation & Reporting </li></ul></ul>Packet Capturer Flow Generator Flow Store Traffic Analyzer Presenter Web Server Network Device User Interface Web browser stored flows analyzed data raw packet packet header information flow information
    41. 41. NG-MON - Packet Capture Network Link Splitting Device divided raw packet pkt header messages <ul><li>Distribution of raw packets </li></ul><ul><ul><li>by using splitting function provided by an optical splitter </li></ul></ul><ul><ul><li>by using mirroring function provided in network devices </li></ul></ul><ul><li>Probe </li></ul><ul><ul><li>captures all packets coming into probe </li></ul></ul><ul><ul><li>export buffer-queues : one to one with flow generators </li></ul></ul><ul><ul><li>fills buffer-queues with packet header’s 5-tuple based hashing </li></ul></ul><ul><ul><li>collect the scattered packets in the same flow into the same buffer-queue </li></ul></ul>Probe #1 Probe #2 Probe #3
    42. 42. NG-MON - Flow Generation <ul><li>Distribution of packet header information </li></ul><ul><ul><li>5-tuple based hashing in the probe </li></ul></ul><ul><ul><li>Packet header messages of potentially the same flow get delivered to the same flow generator </li></ul></ul><ul><li>Flow generator receives packet header messages and generates flows and exports flow messages to flow store </li></ul>pkt header messages flow messages Flow Generator #1 Flow Generator #2 Flow Generator #3 Flow Generator #4
    43. 43. NG-MON - Flow Store <ul><li>Separation of write operations from read operations </li></ul><ul><ul><li>the destination address of flow message is assigned to the flow store according to the time </li></ul></ul><ul><ul><li>While one or more flow stores are inserting flow data, the other flow stores are queried by the traffic analyzers </li></ul></ul><ul><li>Flow store provides traffic information to support various analysis applications </li></ul><ul><ul><li>provides an analysis API to analyzers </li></ul></ul>t 2 t 3 Database Query / Response Traffic Analyzer #1 Traffic Analyzer #2 flow messages Write operations Read operations t 1 Flow Store #1 Flow Store #2 Flow Store #3
    44. 44. NG-MON - Traffic Analysis & Presentation <ul><li>Analyzer extracts information from Flow Stores and can perform application specific analysis </li></ul><ul><li>Separate analyzer is needed for each application </li></ul>Flow Store #1 Presenter Traffic Throughput Analyzer Usage-based billing application DDoS or DoS Attack Analyzer Other applications Flow Store #2 Flow Store #3 Web Server
    45. 45. NG-MON - Implementation Phase Packet Capture Flow Generator Flow Store Analyzer Presenter Development Tool pcap library C language C language C language MySQL C language MySQL PHP jpgraph library Hardware System <ul><li>Xeon 2.4 GHz 2 CPUs </li></ul><ul><li>1 Gbytes memory </li></ul><ul><li>2-1000 Mbps NICs </li></ul><ul><li>80 GB hard disk </li></ul><ul><li>Pentium-III 800 GHz CPU </li></ul><ul><li>256 Mbytes memory </li></ul><ul><li>2-100 Mbps NICs </li></ul><ul><li>20GB hard disk </li></ul>OS Redhat Linux 7.2
    46. 46. NG-MON - Deployment at POSTECH http://ngmon.postech.ac.kr Packet Capture Flow Generator Flow Store Analyzer Presenter 141.223.182. 40 EnterFLEX at Computer Center Flow Store 141.223. 182.[31,32,33,34] POSTECH Computer Center 141.223.182. 38 EnterFLEX at Computer Center 141.223.182. 37 EnterFLEX at Computer Center 141.223.182. 36 EnterFLEX at Computer Center INTERNET 1Gbps Optical link NetOptics 1Gbps Optical Splitter Packet Capture Flow Generator Packet Capture Flow Generator Packet Capture Flow Generator POSTECH Gigabit Campus Network Router Router
    47. 47. NG-MON - Host Data Received Minute View
    48. 48. NG-MON - Host Data Exchanged Minute View
    49. 49. NG-MON - Detailed Subnet Data Sent Minute View
    50. 50. NG-MON - Application Protocol Minute View
    51. 51. NG-MON - Time Series Minute View
    52. 52. 5. Summary <ul><li>Internet is continuously growing in terms of: # of users & hosts, traffic loads & types </li></ul><ul><li>ISPs and enterprises need to monitor their networks for various purposes (e.g., Problem Detection, Workload Characterization, Planning, SLA, Billing, Security, CRM) </li></ul><ul><li>This talk introduced monitoring approaches, flow generation and analysis methods, tools, R&D/standards activities, NG-MON , and real-world applications </li></ul><ul><li>NG-MON </li></ul><ul><ul><li>Scalable and cost-effective architecture </li></ul></ul><ul><ul><li>Spatial, temporal, composition analysis </li></ul></ul><ul><ul><li>P2P, multimedia service, game traffic analysis </li></ul></ul><ul><ul><li>Network security attack analysis </li></ul></ul><ul><li>Network monitoring and analysis is essential for service providers and enterprise network administrators but it is not easy and still needs a lot of work to do it right! </li></ul>
    53. 53. NG-Mon Demo <ul><li>http://ngmon.postech.ac.kr </li></ul>
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×