View Carlson Presentation Part 3


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • * These are the values we are currently we are planning on using, but this is not set in stone.
  • View Carlson Presentation Part 3

    1. 1. The Performance Bottleneck Application, Computer, or Network Richard Carlson Internet2 Part 3
    2. 2. Outline <ul><li>Why there is a problem </li></ul><ul><li>What can be done to find/fix problems </li></ul><ul><li>Tools you can use </li></ul><ul><li>Ramblings on what’s next </li></ul>
    3. 3. Active Measurement Tools <ul><li>Tools that inject packets into the network to measure some value </li></ul><ul><ul><li>Available Bandwidth </li></ul></ul><ul><ul><li>Delay/Jitter </li></ul></ul><ul><ul><li>Loss </li></ul></ul><ul><li>Requires bi-directional traffic or synchronized hosts </li></ul>
    4. 4. Passive Measurement Tools <ul><li>Tools that monitor existing traffic on the network and extract some information </li></ul><ul><ul><li>Bandwidth used </li></ul></ul><ul><ul><li>Jitter </li></ul></ul><ul><ul><li>Loss rate </li></ul></ul><ul><li>May generate some privacy and/or security concerns </li></ul>
    5. 5. Tools, Tools, Tools <ul><li>Ping </li></ul><ul><li>Traceroute </li></ul><ul><li>Iperf </li></ul><ul><li>Tcpdump </li></ul><ul><li>Tcptrace </li></ul><ul><li>BWCTL </li></ul><ul><li>NDT </li></ul><ul><li>OWAMP </li></ul><ul><li>AMP </li></ul><ul><li>Advisor </li></ul><ul><li>Thrulay </li></ul><ul><li>Web100 </li></ul><ul><li>MonaLisa </li></ul><ul><li>pathchar </li></ul><ul><li>NPAD </li></ul><ul><li>Pathdiag </li></ul><ul><li>Surveyor </li></ul><ul><li>Ethereal </li></ul><ul><li>CoralReef </li></ul><ul><li>MRTG </li></ul><ul><li>Skitter </li></ul><ul><li>Cflowd </li></ul><ul><li>Cricket </li></ul><ul><li>Net100 </li></ul>
    6. 6. Network Diagnostic Tool (NDT) <ul><li>Measure performance to users desktop </li></ul><ul><li>Identify real problems for real users </li></ul><ul><ul><li>Network infrastructure is the problem </li></ul></ul><ul><ul><li>Host tuning issues are the problem </li></ul></ul><ul><li>Make tool simple to use and understand </li></ul><ul><li>Make tool useful for users and network administrators </li></ul>
    7. 7. <ul><li>10 Mbps NIC </li></ul><ul><ul><li>Throughput 6.8/6.7 mbps send/receive </li></ul></ul><ul><ul><li>RTT 20 ms </li></ul></ul><ul><ul><li>Retransmission/Timeouts 25/3 </li></ul></ul><ul><li>100 Mbps NIC </li></ul><ul><ul><li>Throughput 84/86 mbps send/receive </li></ul></ul><ul><ul><li>RTT 10 ms </li></ul></ul><ul><ul><li>Retransmission/Timeouts 0/0 </li></ul></ul>Different Host, same switch port
    8. 8. Web100 Project <ul><li>Joint PSC/NCAR project funded by NSF </li></ul><ul><li>‘First step’ to gather TCP data </li></ul><ul><ul><li>Kernel Instrument Set (KIS) </li></ul></ul><ul><li>Developed patches Linux kernel </li></ul><ul><li>Geared toward wide area network performance </li></ul><ul><li>Future steps will automate tuning to improve application performance </li></ul>
    9. 9. NDT’s Web100 Based Approach <ul><li>Simple bi-directional test to gather E2E data </li></ul><ul><li>Gather multiple data variables (a breadth of measurements) </li></ul><ul><li>Compare measured performance to analytical values </li></ul><ul><li>Translate network values into plain text messages </li></ul><ul><li>Geared toward campus area network </li></ul>
    10. 10. SC’04 Real Life Example <ul><li>Booth having trouble getting application to run from Amsterdam to Pittsburgh </li></ul><ul><li>Tests between remote SGI and local PC showed throughput limited to < 20 Mbps </li></ul><ul><li>Assumption is: PC buffers too small </li></ul><ul><li>Question: How do we set WinXP send/receive buffer </li></ul>
    11. 11. SC’04 Determine WinXP info
    12. 12. SC’04 Confirm PC settings <ul><li>DrTCP reported 16 MB buffers, but test program still slow, Q: How to confirm? </li></ul><ul><li>Run test to SC NDT server (PC has Fast Ethernet Connection) </li></ul><ul><ul><li>Client-to-Server: 90 Mbps </li></ul></ul><ul><ul><li>Server-to-Client: 95 Mbps </li></ul></ul><ul><ul><li>PC Send/Recv Buffer size: 16 Mbytes (wscale 8) </li></ul></ul><ul><ul><li>NDT Send/Recv Buffer Size: 8 Mbytes (wscale 7) </li></ul></ul><ul><ul><li>Reported TCP RTT: 46.2 msec </li></ul></ul><ul><ul><ul><li>approximately 600 Kbytes of data in TCP buffer </li></ul></ul></ul><ul><ul><li>Min buffer size / RTT: 1.3 Gbps </li></ul></ul>
    13. 13. SC’04 Local PC Configured OK <ul><li>No problem found </li></ul><ul><li>Able to run at line rate </li></ul><ul><li>Confirmed that PC’s TCP buffers were set correctly </li></ul>
    14. 14. SC’04 Remote SGI <ul><li>Run test from remote SGI to SC show floor (SGI is Gigabit Ethernet connected) . </li></ul><ul><ul><li>Client-to-Server: 17 Mbps </li></ul></ul><ul><ul><li>Server-to-Client: 16 Mbps </li></ul></ul><ul><ul><li>SGI Send/Recv Buffer size: 256 Kbytes (wscale 3) </li></ul></ul><ul><ul><li>NDT Send/Recv Buffer Size: 8 Mbytes (wscale 7) </li></ul></ul><ul><ul><li>Reported RTT: 106.7 msec </li></ul></ul><ul><ul><li>Min Buffer size / RTT: 19 Mbps </li></ul></ul>
    15. 15. SC’04 Remote SGI Results <ul><li>Needed to download and compile command line client </li></ul><ul><li>SGI TCP buffer is too small to fill transatlantic pipe (19 Mbps max) </li></ul><ul><li>User reluctant to make changes to SGI network interface from SC show floor </li></ul><ul><li>NDT client tool allows application to change buffer (setsockopt() function call) </li></ul>
    16. 16. SC’04 Remote SGI (tuned) <ul><li>Re-run test from remote SGI to SC show floor. </li></ul><ul><ul><li>Client-to-Server: 107 Mbps </li></ul></ul><ul><ul><li>Server-to-Client: 109 Mbps </li></ul></ul><ul><ul><li>SGI Send/Recv Buffer size: 2 Mbytes (wscale 5) </li></ul></ul><ul><ul><li>NDT Send/Recv Buffer Size: 8 Mbytes (wscale 7) </li></ul></ul><ul><ul><li>Reported RTT: 104 msec </li></ul></ul><ul><ul><li>Min Buffer size / RTT: 153.8 Mbps </li></ul></ul>
    17. 17. SC’04 Debugging Results <ul><li>Team spent over 1 hour looking at Win XP config, trying to verify Buffer size </li></ul><ul><li>Single NDT test verified this in under 30 seconds </li></ul><ul><li>10 minutes to download and install NDT client on SGI </li></ul><ul><li>15 minutes to discuss options and run client test with set buffer option </li></ul>
    18. 18. SC’04 Debugging Results <ul><li>8 Minutes to find SGI limits and determine maximum allowable buffer setting (2 MB) </li></ul><ul><li>Total time 34 minutes to verify problem was with remote SGIs’ TCP send/receive buffer size </li></ul><ul><li>Network path verified but Application still performed poorly until it was also tuned </li></ul>
    19. 19. NDT Benefits <ul><li>End-user based view of network </li></ul><ul><li>Can be used to identify performance bottlenecks (could be host problem) </li></ul><ul><li>Provides some ‘hard evidence’ to users and network administrators to reduce finger pointing </li></ul><ul><li>Doesn’t rely on historical data </li></ul>
    20. 20. Web Based Performance tool <ul><li>Operates on Any client with a Java enabled Web browser </li></ul><ul><li>What it can do </li></ul><ul><ul><li>Positively state if Sender, Receiver, or Network is operating properly </li></ul></ul><ul><ul><li>Provide accurate application tuning info </li></ul></ul><ul><ul><li>Suggest changes to improve performance </li></ul></ul>
    21. 21. Web base Performance tool <ul><li>What it can’t do </li></ul><ul><ul><li>Tell you where in the network the problem is </li></ul></ul><ul><ul><li>Tell you how other servers perform </li></ul></ul><ul><ul><li>Tell you how other clients will perform </li></ul></ul>
    22. 22. NDT methodology <ul><li>Identify specific problem(s) that affect end users </li></ul><ul><li>Analyze problem to determine ‘Network Signature’ for this problem </li></ul><ul><li>Provide testing tool to automate detection process </li></ul>
    23. 23. Duplex Mismatch Detection <ul><li>Developing analytical model to describe how network operates (no prior art?) </li></ul><ul><li>Expanding model to describe UDP and TCP flows </li></ul><ul><li>Test models in LAN, MAN, and WAN environments </li></ul><ul><li>NIH/NLM grant funding </li></ul>
    24. 24. Test environment <ul><li>Receiver is put is various states </li></ul><ul><ul><li>Switch = full & Host = full or half </li></ul></ul><ul><ul><li>Switch = half & Host = full or half </li></ul></ul>Switch NDT Srv Source 100 Mbps Full Duplex NDT Clt Receiver 100 Mbps Mismatch
    25. 25. Ethernet transmission strategy <ul><li>Half Duplex </li></ul><ul><ul><li>Use carrier sense signal to determine if link in use </li></ul></ul><ul><ul><li>If not, send frame at head of queue </li></ul></ul><ul><ul><li>Else, wait for frame to end and send frame </li></ul></ul><ul><ul><li>Use collision detection signal to determine if other station also sends </li></ul></ul><ul><li>Full Duplex </li></ul><ul><ul><li>Send packet at head of queue </li></ul></ul><ul><ul><li>Disable carrier sense </li></ul></ul><ul><ul><li>Disable collision detection </li></ul></ul>
    26. 26. Analytical Loss-Model Loss vs Transmission rate
    27. 27. Analytical Loss-Model Low speed loss vs send rate
    28. 28. UDP Loss-Model Sender view – normal operation
    29. 29. UDP Loss-Model Sender view – lost data pkts
    30. 30. UDP Loss-Model Sender view – lost acks
    31. 31. TCP Operation on LAN <ul><li>Observed behavior depends on direction of TCP flow and direction of mismatch </li></ul><ul><ul><li>Data and ACK packets delivered </li></ul></ul><ul><ul><li>Data packets lost and ACKs delayed </li></ul></ul><ul><ul><li>ACKs packets lost and Data delayed </li></ul></ul><ul><li>Losing ACKs has bigger effect than losing Data packets </li></ul><ul><li>Web100 details are only available when NDT server is source and client is sink </li></ul>
    32. 32. Four Cases of Duplex Setting FD-FD FD-HD HD-FD HD-HD
    33. 33. Duplex Mismatch Switch is Full & Host is Half
    34. 34. Tentative Mismatch Detection <ul><li>Full to Half mismatch detection </li></ul><ul><ul><li>Large percentage of duplicate ACKs </li></ul></ul><ul><ul><li>Connection spends majority of the time in CwndLimited state </li></ul></ul><ul><ul><li>Asymmetric throughput </li></ul></ul><ul><ul><ul><li>opposite direction is less </li></ul></ul></ul>
    35. 35. Duplex Mismatch Switch is Half & Host is Full
    36. 36. Tentative Mismatch Detection <ul><li>Half to Full mismatch detection </li></ul><ul><ul><li>Large number of timeouts causes long idle time (RTO x timeout value) </li></ul></ul><ul><ul><li>Connection spends majority of the time in CwndLimited state </li></ul></ul><ul><ul><li>Asymmetric throughput </li></ul></ul><ul><ul><ul><li>opposite direction is greater </li></ul></ul></ul>
    37. 37. Bottleneck Link Detection <ul><li>What is the slowest link in the end-2-end path? </li></ul><ul><ul><li>Monitors packet arrival times using libpacp routine </li></ul></ul><ul><ul><li>Use TCP dynamics to create packet pairs </li></ul></ul><ul><ul><li>Quantize results into link type bins (no fractional or bonded links) </li></ul></ul><ul><li>Cisco URP grant work </li></ul>
    38. 38. Normal congestion detection <ul><li>Shared network infrastructures will cause periodic congestion episodes </li></ul><ul><ul><li>Detect/report when TCP throughput is limited by cross traffic </li></ul></ul><ul><ul><li>Detect/report when TCP throughput is limited by own traffic </li></ul></ul>
    39. 39. Faulty Hardware/Link Detection <ul><li>Detect non-congestive loss due to </li></ul><ul><ul><li>Faulty NIC/switch interface </li></ul></ul><ul><ul><li>Bad Cat-5 cable </li></ul></ul><ul><ul><li>Dirty optical connector </li></ul></ul><ul><li>Preliminary works shows that it is possible to distinguish between congestive and non-congestive loss </li></ul>
    40. 40. Full/Half Link Duplex setting <ul><li>Detect half-duplex link in E2E path </li></ul><ul><ul><li>Identify when throughput is limited by half-duplex operations </li></ul></ul><ul><li>Preliminary work shows detection possible when link transitions between blocking states </li></ul>
    41. 41. Additional Functions and Features <ul><li>Provide basic tuning information </li></ul><ul><li>Basic Features </li></ul><ul><ul><li>Basic configuration file </li></ul></ul><ul><ul><li>FIFO scheduling of tests </li></ul></ul><ul><ul><li>Simple server discovery protocol </li></ul></ul><ul><ul><li>Federation mode support </li></ul></ul><ul><ul><li>Command line client support </li></ul></ul><ul><li>Created project page </li></ul>
    42. 42. Internet2 piPEs Project <ul><li>Develop E2E measurement infrastructure capable of finding network problems </li></ul><ul><li>Tools include </li></ul><ul><ul><li>BWCTL: Bandwidth Control wrapper for NLANR Iperf </li></ul></ul><ul><ul><li>OWAMP: One-Way Active Measurement </li></ul></ul><ul><ul><li>NDT: Network Diagnostic Tool </li></ul></ul>
    43. 44. BWCTL Design Goals <ul><li>Bandwidth Control Server </li></ul><ul><li>Wrapper for Dast Iperf tool </li></ul><ul><li>Performs scheduled tests between 11 peers </li></ul><ul><li>Supports on-demand tests between peer nodes </li></ul>
    44. 45. Architecture
    45. 46. Specific difficulties <ul><li>UDP </li></ul><ul><ul><li>Iperf doesn’t always send at requested rate </li></ul></ul><ul><ul><li>Iperf sender hangs (likely Linux/iperf interaction – could be due to signal handling of the bwctl level) </li></ul></ul><ul><ul><li>End of session is difficult to detect, which is problematic for a “scheduled” timeslot </li></ul></ul><ul><ul><li>Iperf sometimes takes large amounts of time to finish </li></ul></ul>
    46. 47. Specific difficulties <ul><li>TCP </li></ul><ul><ul><li>Large pipe to small pipe </li></ul></ul><ul><ul><ul><li>Launch a large window </li></ul></ul></ul><ul><ul><ul><li>Test waits until completion </li></ul></ul></ul><ul><ul><ul><li>Terminate test to remain within schedule </li></ul></ul></ul><ul><ul><ul><li> Sets of incomplete tests to interpret </li></ul></ul></ul><ul><ul><li>Full mesh presents difficulties for window size selection (and other path specific characteristics) </li></ul></ul><ul><ul><ul><li>bwctl uses the peer to peer server connection to deduce a “reasonable” window </li></ul></ul></ul><ul><ul><ul><li>If at all possible path specific parameters need to be dynamically configured </li></ul></ul></ul>
    47. 48. Future Possibilities <ul><li>Server-less client side for end hosts </li></ul><ul><li>Closer integration with test engine (iperf API?) </li></ul><ul><ul><li>Better error detection </li></ul></ul><ul><ul><li>Better timing control (begin and end of test is currently a problem) </li></ul></ul><ul><li>3-party tests (client not on one of the endpoints) </li></ul><ul><li>Open source development </li></ul>
    48. 49. Availability <ul><li>Beta version currently available </li></ul><ul><ul><li> </li></ul></ul><ul><li>Mail lists: </li></ul><ul><li>bwctl-users </li></ul><ul><li>bwctl-announce </li></ul><ul><li> </li></ul>
    49. 50. OWAMP Design Goals <ul><li>One-Way-Active-Measurement-Protocol </li></ul><ul><ul><li>Possible due to growing availability of good time sources </li></ul></ul><ul><ul><li>Wide deployment of “open” servers would allow measurement of one-way delay to become as commonplace as measurement of RTT using ICMP tools such as ping. </li></ul></ul><ul><ul><li>Current Draft: draft-ietf-ippm-owdp-07.txt </li></ul></ul><ul><ul><ul><li>Shalunov,Teitelbaum,Karp,Boote,Zekauskas </li></ul></ul></ul><ul><ul><ul><li>RFC just released </li></ul></ul></ul><ul><ul><li>Sample implementation under development </li></ul></ul><ul><ul><ul><li>Alpha code currently available </li></ul></ul></ul>
    50. 51. Abilene OWAMP deployment <ul><li>2 overlapping full meshes (IPv4 & IPv6) </li></ul><ul><ul><li>11 measurement nodes = 220 ongoing tests </li></ul></ul><ul><ul><li>UDP singletons </li></ul></ul><ul><ul><li>Rate: 10 packets/second* </li></ul></ul><ul><ul><li>Packetsize: (32 byte payload)* </li></ul></ul><ul><ul><li>Results are continuously streamed back to “Measurement Portal” for long-term archive and data dissemination (Near real-time) </li></ul></ul>
    51. 52. OWAMP Errors <ul><li>Preliminary Findings: </li></ul><ul><ul><li>Min error estimates look to be in the 55-60 usec range. </li></ul></ul><ul><ul><li>Serialization Delay: ~5usec x 2 </li></ul></ul><ul><ul><li>Get Timestamp: ~15usec x 2 </li></ul></ul><ul><ul><li>Additional error is: </li></ul></ul><ul><ul><ul><li>Time from userland “send” to 1 st byte hits the wire </li></ul></ul></ul><ul><ul><ul><li>Time from kernel has packet to userland “recv” returns </li></ul></ul></ul><ul><ul><ul><li>Potentially recv process data processing before calling “recv” </li></ul></ul></ul>
    52. 53. OWAMP implementation status <ul><li>Sample implementation </li></ul><ul><li> </li></ul><ul><ul><li>Alpha Release ver 1.6c: </li></ul></ul><ul><ul><ul><li>No “policy” </li></ul></ul></ul><ul><ul><ul><li>No authentication/encryption </li></ul></ul></ul><ul><ul><ul><li>Tested on FreeBSD & Linux </li></ul></ul></ul>
    53. 54. Sample Traceroute Tree map R6 R7 R5 R4 R3 R2 R1 S2 S3 S7 S8 S1 S6 S5 S4
    54. 55. Sample Traceroute from S1 to Client Traceroute to Client R5 R4 R1 S1 Rc Rb Ra Client
    55. 56. Sample Traceroute Tree map S2 S3 S7 S8 S1 S6 S5 S4 R6 R7 R5 R4 R3 R2 R1 R5 R4 R1 S1 Rc Rb Ra Client
    56. 57. Client re-directed to S6 for test Rc Rb Ra Client R5 S6