Finding Network Problems That Influence Applications


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Introduce yourself. --- These slides were prepared by Matt Zekauskas, using original material inspired by Matt Mathis, NLANR DAST and experience; example material created by Rich Carlson and Russ Hobby, with information from the e-VLBI project at MIT Haystack Observatory. Copyright © 2004, Internet2. All Rights Reserved, except that permission is expressly granted for others to use in noncommercial educational materials as long as attribution is given to Internet2 and the authors. [[Replace with blessed text]]
  • Finding Network Problems That Influence Applications

    1. 1. Finding Network Problems that Influence Applications: Measurement Tools Internet2 Performance Workshop
    2. 2. Outline <ul><li>Problems, typical causes, diagnostic strategies </li></ul><ul><li>Examples showing usage of the tools we’ll be talking about today </li></ul><ul><li>End-to-End Measurement Infrastructure </li></ul>
    3. 3. We Would Like Your Help <ul><li>What problems are you experiencing? </li></ul><ul><li>Have you used a good tool? </li></ul><ul><li>Give us the benefit of your experience: successful problem resolution! </li></ul>
    4. 4. What Are The Problems? (1) <ul><li>Packet loss </li></ul><ul><li>Jitter </li></ul><ul><li>Out-of-order packets (extreme jitter) </li></ul><ul><li>Duplicated packets </li></ul><ul><li>Excessive latency </li></ul><ul><ul><li>Interactive applications </li></ul></ul><ul><ul><li>TCP’s control system </li></ul></ul>
    5. 5. For TCP <ul><li>Eliminating loss is the goal </li></ul><ul><li>Non-congestive losses especially tricky </li></ul><ul><li>TCP: 100 Mbit Ethernet coast-to-coast: </li></ul><ul><ul><li>Full size packets… need 10 -6 P loss [Mathis] </li></ul></ul><ul><ul><li>Less than 1 loss every 83 seconds </li></ul></ul><ul><li> </li></ul><ul><li>GigE: 10 -8 , 1 loss every 497 seconds </li></ul>
    6. 6. What Are The Problems? (2) <ul><li>TCP: lack of buffer space </li></ul><ul><ul><li>Forces protocol into stop-and-wait </li></ul></ul><ul><ul><li>Number one TCP-related performance problem. </li></ul></ul><ul><ul><li>70ms * 1Gbps = 70*10^6 bits, or 8.4MB </li></ul></ul><ul><ul><li>70ms * 100Mbps = 855KB </li></ul></ul><ul><ul><li>Many stacks default to 64KB, or 7.4Mbps </li></ul></ul>
    7. 7. What Are The Problems? (3) <ul><li>Video/Audio: lack of buffer space </li></ul><ul><ul><li>Makes broadcast streams very sensitive to previous problems </li></ul></ul><ul><li>Application behaviors </li></ul><ul><ul><li>Stop-and-wait behavior; Can’t stream </li></ul></ul><ul><ul><li>Lack of robustness to network anomalies </li></ul></ul>
    8. 8. The Usual Suspects <ul><li>Host configuration errors (TCP buffers) </li></ul><ul><li>Duplex mismatch (Ethernet) </li></ul><ul><li>Wiring/Fiber problem </li></ul><ul><li>Bad equipment </li></ul><ul><li>Bad routing </li></ul><ul><li>Congestion </li></ul><ul><ul><li>“ Real” traffic </li></ul></ul><ul><ul><li>Unnecessary traffic (broadcasts, multicast, denial of service attacks) </li></ul></ul>
    9. 9. Strategy <ul><li>Most problems are local… </li></ul><ul><li>Test ahead of time! </li></ul><ul><li>Is there connectivity & reasonable latency? (ping -> OWAMP) </li></ul><ul><li>Is routing reasonable (traceroute) </li></ul><ul><li>Is host reasonable (NDT; Web100) </li></ul><ul><li>Is path reasonable (iperf -> BWCTL) </li></ul>
    10. 10. One Technique: Problem Isolation via Divide and Conquer
    11. 11. Outline <ul><li>Problems, typical causes, diagnostic strategies </li></ul><ul><li>Examples showing usage of the tools we’ll be talking about today </li></ul><ul><li>End-to-End Measurement Infrastructure </li></ul>
    12. 12. Tool Examples <ul><li>When to use NDT </li></ul><ul><li>NDT in action at SC’04 </li></ul><ul><li>When to use BWCTL </li></ul><ul><li>BWCTL in action with e-VLBI </li></ul><ul><li>When to use OWAMP </li></ul><ul><li>OWAMP in action with Abilene </li></ul>
    13. 13. When to use NDT <ul><li>When you want to know about last mile and host problems </li></ul><ul><li>When you want a quick and easy test to provide clues at possible problem cause </li></ul><ul><li>When you want to understand large segments of the path from the host view point </li></ul><ul><li>When a user wants to test their own host </li></ul>
    14. 14. Technique <ul><li>Start by testing to the nearest NDT server from each end of the problem path </li></ul><ul><li>This will help you with a majority of problems </li></ul><ul><li>If test both indicate good performance, test to a distant NDT server </li></ul><ul><li>If tests still indicate good performance, suspect a problem in the application, not the host or network. </li></ul>
    15. 15. SC’04 Real Life Example <ul><li>Booth having trouble getting application to run from Amsterdam to Pittsburgh </li></ul><ul><li>Tests between Amsterdam SGI and Pittsburgh PC showed throughput limited to < 20 Mbps </li></ul><ul><li>Assumption is: PC buffers too small </li></ul><ul><li>Question: How do we set WinXP send/receive buffer </li></ul>
    16. 16. SC’04 Determine WinXP info
    17. 17. SC’04 Confirm PC settings <ul><li>DrTCP reported 16 MB buffers, but test program still slow, Q: How to confirm? </li></ul><ul><li>Run test to SCInet NDT server (PC has Fast Ethernet Connection) </li></ul><ul><ul><li>Client-to-Server: 90 Mbps </li></ul></ul><ul><ul><li>Server-to-Client: 95 Mbps </li></ul></ul><ul><ul><li>PC Send/Recv Buffer size: 16 Mbytes (wscale 8) </li></ul></ul><ul><ul><li>NDT Send/Recv Buffer Size: 8 Mbytes (wscale 7) </li></ul></ul><ul><ul><li>Reported TCP average RTT: 46.2 msec </li></ul></ul><ul><ul><ul><li>approximately 600 Kbytes of data in TCP buffer </li></ul></ul></ul><ul><ul><li>Min buffer size / RTT: 1.3 Gbps </li></ul></ul>
    18. 18. SC’04 Local PC Configured OK <ul><li>No problem found </li></ul><ul><li>Able to run at line rate </li></ul><ul><li>Confirmed that PC’s TCP buffers were set correctly </li></ul>
    19. 19. SC’04 Amsterdam SGI <ul><li>Run test from remote SGI to SC show floor (SGI is Gigabit Ethernet connected) . </li></ul><ul><li>Downloaded and built command line tool on SGI IRIX </li></ul><ul><ul><li>Client-to-Server: 17 Mbps </li></ul></ul><ul><ul><li>Server-to-Client: 16 Mbps </li></ul></ul><ul><ul><li>SGI Send/Recv Buffer size: 256 Kbytes (wscale 3) </li></ul></ul><ul><ul><li>NDT Send/Recv Buffer Size: 8 Mbytes (wscale 7) </li></ul></ul><ul><ul><li>Average RTT: 106.7 msec </li></ul></ul><ul><ul><li>Min Buffer size / RTT: 19 Mbps </li></ul></ul>
    20. 20. SC’04 Amsterdam SGI (tuned) <ul><li>Re-run test from remote SGI to SC show floor with –b # option. </li></ul><ul><ul><li>Client-to-Server: 107 Mbps </li></ul></ul><ul><ul><li>Server-to-Client: 109 Mbps </li></ul></ul><ul><ul><li>SGI Send/Recv Buffer size: 2 Mbytes (wscale 5) </li></ul></ul><ul><ul><li>NDT Send/Recv Buffer Size: 8 Mbytes (wscale 7) </li></ul></ul><ul><ul><li>Reported average RTT: 104 msec </li></ul></ul><ul><ul><li>Min Buffer size / RTT: 153.8 Mbps </li></ul></ul>
    21. 21. SC’04 Debugging Results <ul><li>Team spent over 1 hour looking at Win XP config, trying to verify Buffer size </li></ul><ul><ul><li>2 tools used gave different results </li></ul></ul><ul><li>Single NDT test verified this in under 30 seconds </li></ul><ul><li>10 minutes to download and install NDT client on SGI </li></ul><ul><li>15 minutes to discuss options and run client test with set buffer option </li></ul>
    22. 22. SC’04 Debugging Results <ul><li>8 Minutes to find SGI limits and determine maximum allowable buffer setting (2 MB) </li></ul><ul><li>Total time 34 minutes to verify problem was with remote servers’ TCP send/receive buffer size </li></ul><ul><li>Network path verified but Application still performed poorly until it was also tuned </li></ul>
    23. 23. When to use BWCTL <ul><li>You want to understand segments of the path </li></ul><ul><li>You want to know if each segment can handle flows of a specific size </li></ul><ul><li>You want to know parameters such as bandwidth, packet loss and latency </li></ul><ul><li>To help design or tune an application based on available performance </li></ul>
    24. 24. Technique <ul><li>Divide and Conquer! </li></ul><ul><li>Look for segments with performance less that required by the application </li></ul>
    25. 25. e-VLBI Case Study <ul><li>The e-VLBI project needed to move massive amounts of data between a number of sites around the world </li></ul><ul><li>They found that performance from some sites was only in the 1 Mbps range </li></ul><ul><li>They needed to understand why </li></ul>
    26. 26. e-VBLI test infrastructure <ul><li>David Lapsley, one of the research engineers, established BWCTL servers at the sites of the project. </li></ul><ul><ul><li>Japan: Kashima Observatory </li></ul></ul><ul><ul><li>Sweden: Onsala Observatory </li></ul></ul><ul><ul><li>US: Haystack (BOS) </li></ul></ul><ul><li>He performed a full mesh of tests between all of the servers </li></ul>
    27. 27. e-VLBI Results #1 <ul><li>They used Abilene nodes to divide the problem path </li></ul><ul><li>David found that there was considerable packet loss in the area of Haystack Observatory </li></ul><ul><li>Working with network folk from the area the problem was isolated and resolved </li></ul>
    28. 28. e-VLBI Results #2 <ul><li>For one site that was using a commodity Internet only 1 Mbps was regularly seen </li></ul><ul><li>The application was changed to locate caching to reduce dependence on that site. </li></ul>
    29. 29. e-VLBI Regular Testing <ul><li>They found the testing to be very useful in understanding the network status </li></ul><ul><li>They established a regular testing schedule </li></ul><ul><li>They established a web site for reporting the results </li></ul><ul><li>All researchers can check the network status </li></ul><ul><li> </li></ul>
    30. 30. When to use OWAMP <ul><li>Want baseline “heartbeat” information </li></ul><ul><li>Asymmetric routes can make problem location more difficult </li></ul><ul><li>OWAMP can provide detailed performance on one direction in the path </li></ul><ul><li>When you want to know precise latency information </li></ul><ul><li>Good for helping real-time applications </li></ul>
    31. 31. Why use OWAMP <ul><li>It is very sensitive to minor network changes </li></ul><ul><ul><li>Route changes </li></ul></ul><ul><ul><li>Packet queuing </li></ul></ul><ul><li>It tells you about one-direction of the path </li></ul>
    32. 32. OWAMP Case Study Queuing on Abilene <ul><li>Tuesday, 2004-08-17, 16:05-16:20 UTC </li></ul><ul><li>That’s 11:05 to 11:20 EDT </li></ul><ul><li>Caltech to CERN performing 10GE </li></ul><ul><li>throughput experiment </li></ul><ul><li>• Single adapter to date, PCI-X </li></ul><ul><li>• Theoretical limit of ~8.5 Gbps </li></ul><ul><li>• Practical limit closer to 7.5 Gbps </li></ul><ul><li>• Exactly what was tested at that time is unkown </li></ul><ul><li>“ Worst 10” delay list had some larger than </li></ul><ul><li>normal variances… to date, software issues </li></ul>
    33. 33. One Links History <ul><li>The Denver to KSCY Link </li></ul>
    34. 34. What It Shows <ul><li>Only paths that traverse DNVR>KSCY showed additional delay </li></ul><ul><li>Some delayed by ~ an extra 35msec </li></ul><ul><li>Probable cause – Router started queuing packets create a small delay </li></ul><ul><li>It tells you that there is congestion on the link. </li></ul>
    35. 35. Outline <ul><li>Problems, typical causes, diagnostic strategies </li></ul><ul><li>Examples showing usage of the tools we’ll be talking about today </li></ul><ul><li>End-to-End Measurement Infrastructure </li></ul>
    36. 36. End-to-End Measurement Infrastructure Vision <ul><li>Ongoing monitoring to test major elements, and end-to-end paths. </li></ul><ul><ul><li>Elements: gigaPoP links, peering, … </li></ul></ul><ul><ul><li>Utilization </li></ul></ul><ul><ul><li>Delay </li></ul></ul><ul><ul><li>Loss </li></ul></ul><ul><ul><li>Occasional throughput </li></ul></ul><ul><ul><li>Multicast connectivity </li></ul></ul>
    37. 37. End-to-End Measurement Infrastructure Vision II <ul><li>Many more end to end paths than can be monitored. </li></ul><ul><li>Diagnostic tools available on-demand (with authorization) </li></ul><ul><ul><li>Show routes </li></ul></ul><ul><ul><li>Perform flow tests (perhaps app tests) </li></ul></ul><ul><ul><li>Parse/debug flows (a-la tcpdump or OCXmon with heuristic tools) </li></ul></ul>
    38. 38. What Campuses Can Do <ul><li>Export SNMP data </li></ul><ul><ul><li>I have an “Internet2 list”, can add you </li></ul></ul><ul><ul><li>Monitor loss as well as throughput </li></ul></ul><ul><li>Performance test point at campus edge </li></ul><ul><ul><li>Hopefully, the result of today’s workshop </li></ul></ul><ul><ul><li>Possibly also traceroute “looking glass” </li></ul></ul><ul><ul><li>Commercial (e.g., NetIQ) complements </li></ul></ul><ul><ul><li>We have a master list </li></ul></ul>
    39. 39. Strategy (references) (1) <ul><li>See also </li></ul><ul><ul><li> Look at stories, documents, tools </li></ul></ul><ul><ul><li> Pointer to the tool, and using it for debugging the last mile </li></ul></ul>
    40. 40. Strategy (references) (2) <ul><ul><li> How to tweak OS parameters (also scp pointer) </li></ul></ul><ul><ul><li> TCP debugging the detailed way </li></ul></ul><ul><ul><li> Tips for app writers </li></ul></ul><ul><ul><li> And some checking to do by hand & debugging. </li></ul></ul>
    41. 41.
    42. 42. Acknowledgements <ul><li>The original presentation by Matt   Zekauskas using ideas inspired by material from NLANR DAST, Matt   Mathis, and others. </li></ul><ul><li>Copyright Internet2 2005, All Rights Reserved. </li></ul>
    43. 43. Background: Detailed Tools Discussion
    44. 44. Bakground: Tools Outline <ul><li>Tools: First mile, host issues </li></ul><ul><li>Tools: Path issues </li></ul><ul><li>Tools: Others to be aware of </li></ul><ul><li>Tools within Abilene </li></ul>
    45. 45. Internet2 Detective <ul><li>A simple “is there any hope” tool </li></ul><ul><ul><li>Windows “tray” application </li></ul></ul><ul><ul><li>Red/green lights, am I on Internet2 </li></ul></ul><ul><ul><li>Multicast available </li></ul></ul><ul><ul><li>IPv6 available </li></ul></ul><ul><li> </li></ul>
    46. 46. NLANR Performance Advisor <ul><li>Geared for the naive user </li></ul><ul><li>Run at both ends, and see if a standard problem is detected. </li></ul><ul><li>Can also work with intermediate servers </li></ul><ul><li> </li></ul>
    47. 47. NDT <ul><li>Network Debugging Tool </li></ul><ul><li>Java applet </li></ul><ul><li>Connects to server in middle, runs tests, and evaluates heuristics looking for host and first mile problems. </li></ul><ul><li>Has detailed output. </li></ul><ul><li>You’ll see lots of detail later today. </li></ul><ul><li>A commercial tool that tests for TCP buffer problems: </li></ul>
    48. 48. Host/OS Tuning: Web100 <ul><li>Goal: TCP stack, tuning not bottleneck </li></ul><ul><li>Large measurement component </li></ul><ul><ul><li>TCP performance not what you expect? Ask TCP why! </li></ul></ul><ul><ul><ul><li>Receiver bottleneck (out of receiver window) </li></ul></ul></ul><ul><ul><ul><li>Sender bottleneck (no data to send) </li></ul></ul></ul><ul><ul><ul><li>Path bottleneck (out of congestion window) </li></ul></ul></ul><ul><ul><ul><li>Path anomalies (duplicate, out of order, loss) </li></ul></ul></ul><ul><li> </li></ul>
    49. 49. Reference Servers (Beacons) <ul><li>H.323 conferencing </li></ul><ul><ul><li>Goal: portable machines that tell you if system likely to work (and if not, why?) </li></ul></ul><ul><ul><li>Moderate-rate UDP of interest </li></ul></ul><ul><ul><li>E.g., H.323 Beacon </li></ul></ul><ul><ul><li>ViDeNet Scout, </li></ul></ul>
    50. 50. Background: Tools Outline <ul><li>Tools: First mile, host issues </li></ul><ul><li>Tools: Path issues </li></ul><ul><li>Tools: Others to be aware of </li></ul><ul><li>Tools within Abilene </li></ul>
    51. 51. OWAMP – Latency/Loss <ul><li>One-Way Active Measurement Protocol </li></ul><ul><li>Requires NTP-Synchronized clocks </li></ul><ul><li>Look for one-way latency, loss </li></ul><ul><li>Authentication and Scheduling </li></ul><ul><li>Again, lots more later today </li></ul>
    52. 52. BWCTL -- Throughput <ul><li>A tool for throughput testing that includes scheduling and authentication. </li></ul><ul><li>Currently uses iperf for actual tests. </li></ul><ul><li>Can assign users (or IP addresses) to classes, give classes different throughput limits or time limits. </li></ul><ul><li>Periodic and on-demand testing. </li></ul><ul><li>Lots more later today. </li></ul>
    53. 53. Background: Tools Outline <ul><li>Tools: First mile, host issues </li></ul><ul><li>Tools: Path issues </li></ul><ul><li>Tools: Others to be aware of </li></ul><ul><li>Tools within Abilene </li></ul>
    54. 54. Some Commercial Tools <ul><li>Caveat: only a partial list, give me more! </li></ul><ul><li>Spirent (nee Netcom/Adtech): </li></ul><ul><ul><li>SmartBits: test at low & high rates, QoS; test components or end-to-end path </li></ul></ul><ul><li>NetIQ: Chariot/Pegasus </li></ul><ul><li>Agilent (like SmartBits, and FireHunter) </li></ul><ul><li>Ixia (like SmartBits/Spirent) </li></ul><ul><li>Brix Networks (like AMP/Owamp, for ‘QoS’) </li></ul><ul><li>Apparent Networks: path debugger </li></ul>
    55. 55. Some Noncommercial Tools <ul><li>Iperf: </li></ul><ul><ul><li>See also </li></ul></ul><ul><ul><li> </li></ul></ul><ul><li>Flowscan: </li></ul><ul><ul><li> </li></ul></ul><ul><ul><li> </li></ul></ul><ul><li>SLAC’s traceroute perl script: </li></ul><ul><ul><li> </li></ul></ul><ul><li>One large list: </li></ul><ul><ul><li> </li></ul></ul>
    56. 56. Background: Tools Outline <ul><li>Tools: First mile, host issues </li></ul><ul><li>Tools: Path issues </li></ul><ul><li>Tools: Others to be aware of </li></ul><ul><li>Tools within Abilene </li></ul>
    57. 57. Abilene: Measurements from the Center <ul><li>Active (latency, throughput) </li></ul><ul><ul><li>Measurement within Abilene </li></ul></ul><ul><ul><li>Measurements to the edge </li></ul></ul><ul><li>Passive </li></ul><ul><ul><li>SNMP stats (esp. core Abilene links) </li></ul></ul><ul><ul><li>Variables via router proxy </li></ul></ul><ul><ul><li>Router configuration </li></ul></ul><ul><ul><li>Route state </li></ul></ul><ul><ul><li>Characterization of traffic </li></ul></ul><ul><ul><ul><li>Netflow; OCxMON </li></ul></ul></ul>
    58. 58. Goal <ul><li>Abilene goal to be an exemplar </li></ul><ul><ul><li>Measurements open </li></ul></ul><ul><ul><li>Tests possible to router nodes </li></ul></ul><ul><ul><li>Throughput tests routinely through backbone </li></ul></ul><ul><ul><li>… as well as existing utilization, etc. </li></ul></ul><ul><ul><li>The “Abilene Observatory” </li></ul></ul>
    59. 59. Abilene: Machines <ul><li>GigE connected high-performance tester </li></ul><ul><ul><li>bwctl, “nms1”, 9000 byte MTU </li></ul></ul><ul><li>Latency tester </li></ul><ul><ul><li>owamp, “nms4”, 100bT </li></ul></ul><ul><li>Stats collection </li></ul><ul><ul><li>SNMP, flow-stats, “nms3”, 100bT </li></ul></ul><ul><li>Ad-hoc tests </li></ul><ul><ul><li>NDT server, “nms2”, gigE, 1500 byte MTU </li></ul></ul>
    60. 60. Throughput <ul><li>Take tests 1/hr, 20 seconds each </li></ul><ul><ul><li>IPv4 TCP </li></ul></ul><ul><ul><li>IPv6 TCP (no discernable difference) </li></ul></ul><ul><ul><li>IPv4 UDP (on our platforms flakey at 1G) </li></ul></ul><ul><ul><li>IPv6 UDP (ditto) </li></ul></ul><ul><li>Others test to our nodes </li></ul><ul><li>Others test amongst themselves </li></ul><ul><li>Net result: 25% of traffic (NOT capacity) is measurement </li></ul>
    61. 61. Latency <ul><li>CDMA used to synchronize NTP </li></ul><ul><ul><li> </li></ul></ul><ul><li>Test among all router node pairs </li></ul><ul><li>10/sec </li></ul><ul><li>IPv4 and IPv6 </li></ul><ul><li>Minimal sized packets </li></ul><ul><li>Poisson schedule </li></ul>
    62. 62. Passive - Utilization <ul><li>The Abilene NOC takes </li></ul><ul><ul><li>Packets in,out </li></ul></ul><ul><ul><li>Bytes in,out </li></ul></ul><ul><ul><li>Drops/Errors </li></ul></ul><ul><ul><li>..for all interfaces, publishes internal links & peering points (at 5 min intervals) </li></ul></ul><ul><ul><li>..via SNMP polling – every 60 sec </li></ul></ul><ul><li> </li></ul>
    63. 64. Abilene Pointers <ul><li> </li></ul><ul><ul><li>Monitoring </li></ul></ul><ul><ul><li>Tools </li></ul></ul><ul><li> </li></ul><ul><li> (summaries) </li></ul>