Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Can not produce (in the sender) or consume (in the receiver) efficiently in the user space. In the sender, TCP has few new data to send; in the receiver, TCP receiver-buffer fill up. New technologies: multi-core CPU, PCI-Express etc
  • Accurate bandwidth estimation is difficult Network provide very little network status information to network end systems. Network condition change instantaneously.
  • When networks are congested, router/switch buffer overflows lead to packet drops. such as dirty fibers, malfunctioning line cards, flawed Ethernet connections
  • ppt

    1. 1. Wide Area Network Performance Analysis Methodology Wenji Wu, Phil DeMar, Mark Bowden Fermilab ESCC/Internet2 Joint Techs Workshop 2007 [email_address] , [email_address] , [email_address]
    2. 2. Topics <ul><li>Problems </li></ul><ul><li>End-to-End Network Performance Analysis </li></ul><ul><ul><li>TCP transfer throughput </li></ul></ul><ul><ul><li>TCP throughput is network-end-system limited </li></ul></ul><ul><ul><li>TCP throughput is network-limited </li></ul></ul><ul><li>Network Performance Analysis Methodology </li></ul><ul><ul><li>Performance Analysis Network Architecture </li></ul></ul><ul><ul><li>Performance Analysis Steps </li></ul></ul>
    3. 3. 1. Problems <ul><li>What, Where, and How are the performance bottlenecks of network applications in wide area networks? </li></ul><ul><li>How to diagnose network/application performance quickly and efficiently? </li></ul>
    4. 5. 2. End-to-End Network/Application Performance Analysis
    5. 6. 2.1 TCP transfer throughput <ul><li>An end-to-end TCP connection can be separated into: the sender, the networks, and the receiver. </li></ul><ul><li>TCP adaptive windowing scheme consists of a send-window (W s ), congestion-window (CWND), and receive-window (W R ). </li></ul><ul><ul><li>Congestion Control: congestion-window </li></ul></ul><ul><ul><li>Flow Control: receive window </li></ul></ul><ul><li>The overall end-to-end performance of TCP throughput is decided by the sender, the network, and the receiver, which are modeled and symbolized in the sender as W s, CWND, and W R. Assume the round trip time RTT, the instantaneous TCP throughput at time t: </li></ul><ul><ul><li>Throughput (t) = min{ W s (t) , CWND(t), W R (t) }/RTT(t) </li></ul></ul>
    6. 7. 2.1 TCP transfer throughput (cont) <ul><li>If any of the three windows is small, especially when such conditions last for a relatively long period of time, the overall TCP throughput would be seriously degraded. </li></ul><ul><ul><li>The TCP throughput is network-end-system-limited for the duration T, if it has: </li></ul></ul><ul><ul><li>The TCP throughput is network-limited for the duration T, if it has: </li></ul></ul>
    7. 8. 2.2 TCP throughput is network-end-system limited <ul><li>User/Kernel space split </li></ul><ul><ul><li>Network application in user space, in process context </li></ul></ul><ul><ul><li>Protocol processing in kernel, in the interrupt context </li></ul></ul><ul><li>Interrupt-driven operating system </li></ul><ul><ul><li>Hardware interrupt -> Software interrupt -> process </li></ul></ul>
    8. 9. 2.2 TCP throughput is network-end-system limited (cont) <ul><li>Factors leading to a relatively small window of W S (t) & W R (t) </li></ul><ul><ul><li>Poorly-designed network application </li></ul></ul><ul><ul><li>Performance-limited hardware </li></ul></ul><ul><ul><ul><li>CPU, disk I/O subsystem, system buses, memory </li></ul></ul></ul><ul><ul><li>Heavily-loaded network end systems </li></ul></ul><ul><ul><ul><li>System interrupt loads are too high </li></ul></ul></ul><ul><ul><ul><ul><li>Interrupt coalescing, Jumbo Frame </li></ul></ul></ul></ul><ul><ul><ul><li>System process load are too high </li></ul></ul></ul><ul><ul><li>Poorly configured TCP protocol parameters </li></ul></ul><ul><ul><ul><li>TCP Send/Receive buffer size </li></ul></ul></ul><ul><ul><ul><li>TCP window scaling in high speed, long distance networks. </li></ul></ul></ul>
    9. 10. 2.3 TCP throughput is network-limited <ul><li>Two Facts: </li></ul><ul><ul><li>TCP sender tries to estimate the available bandwidth in the networks, and represents it as CWND with congestion control algorithms. </li></ul></ul><ul><ul><li>TCP assumes packet drops are caused by network congestion. Any packet drops will lead to a reduction in CWND. </li></ul></ul><ul><li>Two determining factors for CWND </li></ul><ul><ul><li>Congestion control algorithm </li></ul></ul><ul><ul><li>Network Conditions (Packet drops) </li></ul></ul>
    10. 11. 2.3 TCP throughput is network-limited (cont) <ul><li>TCP congestion control algorithm is evolving </li></ul><ul><ul><li>Standard TCP congestion control (Reno/NewReno) </li></ul></ul><ul><ul><ul><li>Slow start, congestion avoidance, retransmission timeouts, fast retransmit and fast recovery </li></ul></ul></ul><ul><ul><ul><li>AIMD scheme for congestion avoidance </li></ul></ul></ul><ul><ul><ul><ul><li>Perform well in traditional networks </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Cause under-utilized problem in high-speed and long-distance networks </li></ul></ul></ul></ul><ul><ul><li>High-speed TCP variants: FAST TCP, HTCP, HSTCP, BIC, and CUBIC </li></ul></ul><ul><ul><ul><ul><li>Modify the AMID congestion avoidance scheme of standard TCP to be more aggressive, </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Keep the same fast retransmit and fast recovery algorithm </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Solve the under-utilized problem in high speed and long distance networks </li></ul></ul></ul></ul>
    11. 12. 2.3 TCP throughput is network-limited (cont) <ul><li>With high-speed TCP variants, it is mainly the packet drops that lead to a relatively small CWND </li></ul><ul><li>The following conditions could lead to packet drops </li></ul><ul><ul><li>Network congestion. </li></ul></ul><ul><ul><li>Network infrastructure failures. </li></ul></ul><ul><ul><li>Network end systems. </li></ul></ul><ul><ul><ul><li>Packet drops in Layer 2 queues due to limited queue size. </li></ul></ul></ul><ul><ul><ul><li>Packet dropped in ring buffer due to system memory pressure. </li></ul></ul></ul><ul><ul><li>Routing changes. </li></ul></ul><ul><ul><ul><li>When a route changes, the interaction of routing policies, iBGP, and the MRAI timer may lead to transient disconnectivity. </li></ul></ul></ul><ul><ul><li>Packet reordering. </li></ul></ul><ul><ul><ul><li>Packet reordering will cause duplicate ACKs to the sender. RFC 2581 suggest a TCP sender should consider three or more dupACKs as an indication of packet loss. With severe packet reordering, TCP might misinterpret it as packet losses. </li></ul></ul></ul>
    12. 13. 2.3 TCP throughput is network-limited (cont) <ul><li>Congestion window is manipulated on the unit of Maximum Segment Size (MSS). Larger MSS entails higher TCP throughput. </li></ul><ul><li>Larger MSS is efficient for both networks and network end systems. </li></ul>
    13. 14. 3. Network Performance Analysis Methodology
    14. 16. Network Performance Analysis Methodology <ul><li>An end-to-end network/application performance is viewed as </li></ul><ul><ul><li>Application-related problems, </li></ul></ul><ul><ul><ul><li>Beyond the scope of any standardized problem analysis </li></ul></ul></ul><ul><ul><li>Network end system problems </li></ul></ul><ul><ul><li>Network path problems </li></ul></ul><ul><li>Network performance analysis methodology </li></ul><ul><ul><li>Analyze and appropriately tune the network end systems </li></ul></ul><ul><ul><li>Network path analysis, with remediation of detected problems where feasible </li></ul></ul><ul><ul><li>If network end system and network path analysis do not uncover significant problems or concerns, packet trace analysis will be conducted. </li></ul></ul><ul><ul><ul><li>Any performance bottlenecks will manifest themselves in the packet traces. </li></ul></ul></ul>
    15. 17. 3.1 Network Performance Analysis Network Architecture
    16. 18. <ul><li>Network end system diagnosis server. </li></ul><ul><ul><li>We use Network Diagnostic Tool (NDT). </li></ul></ul><ul><ul><ul><li>collect various TCP network parameters in the network end systems, and identify their configuration problems </li></ul></ul></ul><ul><ul><ul><li>Identify local network infrastructure problems such as faulted Ethernet connections, malfunctioning NICs, and Ethernet duplex mismatch. </li></ul></ul></ul><ul><li>Network path diagnosis server </li></ul><ul><ul><li>We use OWAMP applications to collect and diagnose one-way network path statistics. </li></ul></ul><ul><ul><ul><li>The forward and reverse path might not be symmetric </li></ul></ul></ul><ul><ul><ul><li>The forward and reverse path traffic loads likely not symmetric </li></ul></ul></ul><ul><ul><ul><li>The forward and reverse path might have different Qos schemes </li></ul></ul></ul><ul><ul><li>Other tools such as Ping, traceroute, pathneck, iperf, and PerfSONAR etc could be used. </li></ul></ul>3.1 Network Performance Analysis Network Architecture (cont)
    17. 19. 3.1 Network Performance Analysis Network Architecture (cont) <ul><li>Packet trace diagnosis server </li></ul><ul><ul><li>Directly connect to the border router, can port-mirror any port in the border router </li></ul></ul><ul><ul><li>TCPDump, used to record packet traces </li></ul></ul><ul><ul><li>TCPTrace, used to analyze the recorded packet traces </li></ul></ul><ul><ul><li>Xplot, used to examine the recorded traces visually </li></ul></ul>
    18. 20. <ul><li>Step 1: Definition of the problem space </li></ul><ul><li>Step 2: Collect of network end system information & network path characteristics </li></ul><ul><li>Step 3: Network end system diagnosis </li></ul><ul><li>Step 4: Network path performance analysis </li></ul><ul><ul><li>Route changes frequently? </li></ul></ul><ul><ul><li>Network congestion: delay variance large? Bottleneck location? </li></ul></ul><ul><ul><li>Infrastructure failures: examine the counter one by one </li></ul></ul><ul><ul><li>Packet reordering: load balancing? Parallel processing? </li></ul></ul><ul><li>Step 5: Evaluate packet trace pattern </li></ul>3.2 Network/Application Performance Analysis Steps
    19. 21. Collection of network end system information
    20. 22. Collection of network path characteristics <ul><li>Network path characteristics </li></ul><ul><ul><li>Round-trip time (ping) </li></ul></ul><ul><ul><li>Sequence of routers along the paths (traceroute) </li></ul></ul><ul><ul><li>One-way delay, delay variance (owamp) </li></ul></ul><ul><ul><li>One-way packet drop rate (owamp) </li></ul></ul><ul><ul><li>Packet reordering (owamp) </li></ul></ul><ul><ul><li>Current achievable throughput (iperf) </li></ul></ul><ul><ul><li>Bandwidth bottleneck location (pathneck) </li></ul></ul>
    21. 23. Traffic trace from Fermi to OEAW What happened?
    22. 24. Traffic trace from Fermi to Brazil What happened?
    23. 25. Conclusion <ul><li>Fermilab is working on developing a performance analysis methodology </li></ul><ul><ul><li>Objective is to put structure into troubleshooting network performance problems </li></ul></ul><ul><ul><li>Project is in early stages of development </li></ul></ul><ul><li>We welcome collaboration & feedback </li></ul><ul><ul><li>Biweekly Wide-Area-Working-Group (WAWG) meeting on alternate Friday mornings </li></ul></ul><ul><ul><ul><li>Send email to WAWG@FNAL.GOV </li></ul></ul></ul>