Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Introduce yourself. --- These slides were prepared by Matt Zekauskas, using original material inspired by Matt Mathis, NLANR DAST and experience; example material created by Rich Carlson and Russ Hobby, with information from the e-VLBI project at MIT Haystack Observatory. Copyright © 2004, Internet2. All Rights Reserved, except that permission is expressly granted for others to use in noncommercial educational materials as long as attribution is given to Internet2 and the authors. [[Replace with blessed text]]
  • Before we really start, I want to say that we don’t have all the answers. I’m going to tell you about common problems, and tools and techniques we’ve found useful. You’ll learn more about specific tools over the course of the day. However, if you have a common problem, or a particularly difficult problem, we’d like to hear about it. In fact, we collect “war stories” for publication on our web site. In addition, if you have a tool or technique that we don’t talk about today, please do speak up during the day, or send us details after the workshop.
  • I’ll start off with a list of network problems that we find affect performance of most applications. Packet loss slows TCP (bulk data transfer), and causes dropouts with voice and “jaggies” with video. Jitter, or the change in the rate that packets arrive, can also cause TCP to slow down, or at least react to problems more slowly; excessive jitter in a real-time application can cause some packets to be treated as if they were lost, causing dropouts and video problems. If you have an interactive application, say remote control of a scanning-tunneling microscope, jitter makes it hard for humans to react. We can learn how to deal with latency, but we can’t adjust for arbitrary changes in latency. An Ohio state study of h.323 codecs found that jitter caused more problems than loss (up to some point). Out-of-order packets can be viewed as extreme jitter; beside the problems already listed, many applications are not written well (for example, early MPEG-2 and HDTV codecs), and out of order packets can cause more problems than lost packets. Applications should be written to be tolerant of out-of-order packets (since parallelism in the network can generate them naturally, and they actually occur quite frequently), but for now, reducing the number of out of order packets will improve application efficiency. Duplicate packets waste bandwidth, and in extreme cases can cause TCP to slow down or confuse real-time applications. Excessive latency makes it difficult for interactive applications (video conferencing, remote instrument control), although humans can compensate to some extent. TCP’s control system also has more trouble as latency increases, since it reacts more slowly. In general, it is best to engineer paths so that latency is minimized.
  • Let’s look at how “vanilla” Reno TCP (still the most commonly deployed TCP stack) reacts to losses to see how important limiting loss along a path is. If a path is congested, it is obvious (at least once the problem link is found) because link utilization is high. However losses can be caused for other reasons (which we will get to in a moment), and these “non-congestive” losses are especially hard to track down. However, those are the ones that are also important to eradicate as much as possible. Let’s say our goal is modest (for modern workstations): send 100 megabits from coast-to-coast. With full size Ethernet packets (1500 bytes for 100Mbps interfaces) you need a probability of packet loss on the order of one in a million. That’s one loss every 83 seconds. [[Note: this is an equation to give you a general idea; exact semantics depend on the version of TCP that is deployed; a single loss may get covered by “fast retransmit”, and the sender may never slow down.]] How about if you have gigabit Ethernet? Then the loss probability must be less than one in ten to the negative eighth power, or one loss every 497 seconds. The situation gets better if you can use so-called “jumbo-frames”, or 9000 byte packets, it goes back to close to the 100 megabit case. That’s one reason to try and make high-performance paths “9000-byte clean”. The situation also gets better with some of the newer TCP algorithms (high-speed TCP, BIC TCP, FAST TCP) which is why there’s a lot of research into new bulk-transfer control algorithms. But we would still like a way to find and remove non-congestive losses as much as possible.
  • OK, we talked about network problems. However, you also should know that the number one reason for TCP not running at “full-speed” is for it to be starved for buffer space. Vendors ship TCP stacks with buffers that are tuned for the commercial Internet. If the buffer is too small, TCP, which uses a “sliding window” for flow control, must wait for packets to be acknowledged in order to advance the window and send more data. Essentially the sender is forced to stop and wait. You need to be able to buffer the number of bits you can send in one round-trip time at your desired speed. [[Note that there are also send and receive buffers; if the either is too small you can end up with this problem.]] For example, with a 70 millisecond round-trip time (more-or-less trans-continental North America), to sustain one gigabit per second you need 8.4 megabytes of buffer space. For 100Mbps at the same distance you need 855 kilobytes. Many stacks default to 64 kilobytes, which only allows 7.4 Mbps. One word of caution: network kilobits, megabits, gigabits are powers of 10. Memory kilobytes and megabytes are in powers of two, a kilobyte being 1024 bytes (2^10) and a megabyte being 1,048,576 bytes (2^20). More detail on TCP behavior from Rich: TCP has 2 buffers, send and receive. The sliding window is always used, not just if the buffer is too small. Since TCP delivers a reliable, in-order packet delivery service it needs to detect and recover from loss and mis-ordered packets. The sender must retain a copy of all packet sent in the event that IP packets are lost. If this buffer fills up, the sender must stop sending until ACKs are received. The receiver must also deal with out-of-order packets, so it maintains a reassembly buffer. This buffer also holds packets when the application is unable to process them. If this receive buffer fills up, the sender must stop sending until the application can process the data. So either buffer filling up can cause the sender to stop. Both sender and receiver must be tuned to eliminate buffer stalls. In some cases it may not be possible for the local host or sys-admin to fix the problem (i.e., the remote host has mis-set buffers).
  • I also want to mention that the same problem carries up to the applications themselves. We won’t be speaking more about this today, but for video and audio (streaming media) the lack of buffer space in the application (in our world, MPEG-2 based applications are especially bad) means the application is very sensitive to packet loss or reordering. Of course, if your application is interactive, then increased buffering can lead to lag in response, which is not desirable, either. This generalizes to bad network application behavior, so that they are not robust to network changes or anomalies. Drops will occur. Reordering will occur. Even if only very occasionally. Even applications that would like to use TCP to do bulk transfer can do things like not hand enough data to TCP to allow it to stream over long distances. One that was brought to light recently is scp (and therefore ssh; this problem can also occur with standard FTP); popular versions of scp do not provide large enough buffers for TCP to stream. (There is a pointer to a good version of scp off a tcp tuning page at PSC mentioned later.)
  • So what causes these problems? Here’s a laundry-list of the “usual suspects”. First on the list, and most common is a bad host configuration. As we just mentioned, this is usually because operating systems ship tuned to the commercial internet, and we have very different paths over the Internet2 infrastructure (in particular the “bandwidth delay product” is much greater). Second is duplex mismatch, usually due to autoconfiguration failure, with one side believing it is full-duplex (can send and receive simultaneously), and the other side believing it is half-duplex (can only send or receive one at a time). This is a legacy of how the Ethernet standard has evolved. This is the major cause of “non-congestive” packet losses. Wiring or fiber problems can cause non-congestive packet losses. Bad equipment (anything from host interfaces that cannot run full-speed, to host, switch, router, or fiber equipment failure) can cause excessive delays, jitter, or non-congestive packet loss. Bad routing can cause excessive latency, or sometimes jitter due to multiple different length paths being used. Congestion causes varying delays and packet loss.
  • OK. This slide illustrates one way to attack a performance problem. First and foremost, if you are planning a demo, or other event, do test ahead of time. If you have a concerned application community, this may mean periodic testing among points close to key equipment. For example, all of the VLBI sites may test among each other. It may also mean periodic testing within your network to points in Abilene, or other campuses you talk to frequently. Now say you have a problem that the periodic testing did not pick up (there are just too many paths to test them all). The first question – do you have connectivity and reasonable latency? Ping will give you round-trip times, assuming it isn’t blocked along the way. We’ll describe a tool, owamp, that measures one-way delay, which allows you to disambiguate problems that might occur asymmetrically – asymmetric routing, asymmetric traffic queuing, a dirty fiber can cause asymmetric problems (since each fiber transmits light in one direction). Are you seeing many losses with these low-rate tests? If so, there’s something terribly wrong. If the latency is not what you expect, there may be a routing problem. The best-known tool is traceroute, and you can use that to make sure the path looks reasonable. It goes through your campus, possibly through a gigapop, across Abilene and down to the other side in a reasonable fashion (not taking a scenic tour of the US, for example). Remember that you have to test in the opposite direction; the Abilene router proxy and traceroute servers can help. Has the host been tuned? Is there potentially a duplex-mismatch one of the local Ethernet connections? Here, running NDT, also to be described today, can point out a series of common problems. NDT itself relies on web100, which instruments the Linux kernel. You might consider installing a web100 machine (or using machines with web100 code); there are additional diagnostics you can run using the web100-provided variables, and the kernel itself is better “out of the box”: it can automatically tune buffers on some TCP connections. If routing looks reasonable, and the host is reasonable, you may have a problem in the path. (Large losses in the low rate tests also indicate path problems, assuming it isn’t a duplex mismatch problem, local congestion--perhaps a denial of service attack--or even broken network hardware on the end system.) Iperf is a tool to run synthetic TCP streams (memory-to-memory) between two machines. Bwctl is a tool that we will talk about today that adds authentication and scheduling to iperf, and allows you to test to multiple points, including midpoints within Abilene.
  • And, for path problems, the best strategy is usually to “divide and conquer”; test to a midpoint, and see which side the problem is on, and then test to a midpoint on one side, until you’ve exhausted your midpoints and have localized the problem as much as you can. We’re working on tools to automate this process, but for now it’s manual. This picture shows testing to a bunch of points that you have access to.
  • Quickest of the Tools to use Client can be used on almost any host Servers are almost always close to some point in the path in Internet2
  • This begins a section on how the NDT was used to find a problem in a real live situation. SC is the annual Supercomputing conference and it was held in Pittsburgh PA in November 2004. Brief problem description given and sys admins assumption as to what was wrong. Question is what Rich Carlson was asked
  • Rich pointed the sys admin to this tool to check/set the Windows buffer size. Note: in Windows, TCP transmit buffer tracks receive buffer, so setting 1 sets them both. This is the output from a laptop computer.
  • Results when run on sys Admin’s XP system, but they didn’t believe it as their test program kept saying 8K buffers were being used. Test from XP to NDT on conference network. Report clearly shows that PC buffers are being set to 16 MB. Also note that with that buffer size and the reported RTT, the link should be able to sustain 1.3 Gbs and the limit is the physical 100 Mbps Fast E interface in the PC.
  • Conclusions from test. Note that we now know that the PC is operating properly.
  • Next run test from remote SGI. This host was located at the admin’s home university in Amsterdam. They needed to download and build the command line client before this test could be done. Note that the SGI has a small (256KB) buffer and that this buffer limits the throughput to ~19 Mbps, which is in line with the measured results. Now we know that the SGI needs to be tuned to run over the transatlantic path User reluctant to make changes to SGI network interface from SC show floor NDT client tool allows application to change buffer (setsockopt() function call)
  • This slide show what happens when the user increased the SGI buffer size. Experiments showed that 2 MB was the max value the SGI would allow. It would require a system configuration change to go above this value. Notice that we now get 100 Mbps, which is good enough because the client is only connected vi a FastE link. The admin now knows that the network path can support full line rate between the convention floor and the remote site in Amsterdam.
  • Observations and conclusions. The admin spent over an hour trying to figure out why the XP host wasn’t operating properly. The tools he was using didn’t give the correct data or the data wasn’t believed because the test results didn’t match to expected results. Highlight that a single NDT test showed that everything was working as configured. It took about 10 minutes to download and build the NDT command line client on the SGI. This was done remotely from the SC convention floor via an ssh terminal session. After running the test it took another 15 minutes or so to evaluate what it meant, what our options were, and how to tell if things would get better if the default buffer sizes were increased.
  • Brief summary of where the time went. Note that about ½ hour elapses since the NDT tests began. Contrast this with the 1 hour already spent looking for a non-existent XP config problem. Final note, even after the network path was verified, the user application failed to operate as expected. It was eventually tuned to perform better.
  • Other reasons to use BWCTL: You do not have access to the end hosts (so you can still get partial information) BWCTL allows testing only between know entities (other allowed BWCTL servers) BWCTL insures that only one test at a time is done between servers BWCTL servers will have been tuned to insure that the results reflect the network performance In order to use BWCTL you need to get access to intermediate servers.
  • Infrastructure revealed problems. Abilene nodes were in the middle – so they can act to see which side of the path you should focus on. Problems are often easy (at least conceptually) to fix, once you can focus on the problem segments (Sometimes requires the application of funds (upgrade equipment or links)
  • [[Verify URL still valid]]
  • It also gives a long-term baseline, so you can note trends and changes.
  • Background on what we think was going on…
  • The minimum never varies – typical of a link, unless it is fully congested. The 95 th percentile is rising, and occasionally the 50 th percentile is rising too – so a “typical” packet will see some queuing. For research links, we ideally want to keep the 95 th percentile equal to the minimum, unless very large tests are being run.
  • OK. So, we’ve talked about problems. Diagnostic strategies. And we showed the use of some of the tools to implement those strategies. I’d like to take a moment to revisit the end-to-end performance initiative vision, and talk a bit about what campuses can do.
  • Add separate slide for application communities? Not yet, so say one of the important set of end-to-end paths is the set that a particular application community cares about. Nuclear physics sites. Medical sites. Astronomy sites.
  • But, because there are many many paths, there will have to be some ability to do tests on the fly. So make them available. In the long run, provide a tool to do most of it for you, and just hand back the results. Probably skip: This is not limited to just the US; we work with folks in Europe, Asia, and elsewhere. So there has to be a way to interoperate. That’s one of the things Internet2 is working on now.
  • So, what can you do? A few simple things. Make utilization data available, at least at the edge of your campus. Monitor not only utilization, but things that can cause losses… packet drop and error counters. Placing points at the edge of your campus will allow you to test ad-hoc from within your campus to the edge, and allow you to constantly monitor campus connectivity you think is important. (There will be some more use cases later; one possibility is just making sure your university to university traffic goes over the high-performance network--as long as it’s up). Thanks for listening to the overview.
  • References for debugging strategies and application design. Flip through these quickly, they are here so participants can look later.
  • Just because my name ‘ll probably come off the front if it becomes standard course material
  • Let’s start with tools that check out the hosts, and the network connections near the hosts.
  • Very rudimentary (NOTE: Do not read any of the items on this page – flip through quickly)
  • I want to mention this one because it’s fresh. But we haven’t had time to extensively evaluate it. (NOTE: Do not read any of the items on this page – flip through quickly)
  • This is the one we’re actively developing. More on this later today. (NOTE: Do not read any of the items on this page – flip through quickly)
  • As an aside, I mentioned Web100 earlier in a bullet. Here’s what Web100 is, and why you might want to put it on systems you use, if you can. (NOTE: Do not read any of the items on this page – flip through quickly) KEEP THIS TEXT FOR FOLKS WHO DOWNLOAD THE SLIDES: It is a kernel modification, currently to Linux 2.6 series kernels. There is a TCP MIB draft in the IETF to try and standardize the export-TCP-state part of Web100, and we expect Microsoft and others to pick that up. (Microsoft already has some of the elements in recent Windows server versions)
  • Rather than the generic NDT tool, there are also specific tools for videoconferencing. Moderate-rate UDP is a substitute, but the H.323 Beacon from Ohio State (free) and ViDeNet Scout (uses licensed software) actually run the protocol and capture behavior. (NOTE: Do not read any of the items on this page – flip through quickly)
  • OK, what about tools to help us resolve path problems?
  • Here’s a brief description of OWAMP just to get you oriented. More on this tool later today. (NOTE: Do not read any of the items on this page – flip through quickly)
  • Likewise, a brief description of BWCTL. More on this tool later today, also. (NOTE: Do not read any of the items on this page – flip through quickly)
  • Finally, here are some pointers to other tools that are used. They are here more for reference than for detailed explanation now.
  • Here are some commercial tools that we know of. (NOTE: show this slide but don’t read through the items) Spirent makes testers that can rigorously evaluate routers (and paths), and work at line rate. NetIQ has little drones that you can run with a command and control console. It can simulate some application behavior, and also has a capture then replay ability. Agilent makes testers like spirent, and also has a product called FireHunter that is used by ISPs. It does things like pings, and FTP fetches, and Web fetches, and can issue alerts when things go out of spec. Ixia makes boxes like Agilent and Spirent. Brix Networks is interesting, because they make measurement points that you can deploy, and then run tests like owamp among them, as well as other tests specifically design to probe QoS parameters and limits. We already mentioned Apparent Networks.
  • Here are a couple of noncommercial tools, along with a pointer to a whole bunch more. (NOTE: show this slide but don’t read through the items) The direct pointer to iperf, which bwctl uses. Flowscan is a tool to process netflow output and create pretty aggregate graphs. There is another set of tools, called “flow-tools” from ohio state, that Abilene uses (note: Ohio State is an Internet2 technology evaluation center). SLAC has a perl script that can be used with a web server to provide traceroutes. Les Cottrell and his group at SLAC also has a huge list of tools.
  • OK, let’s see what’s in Abilene.
  • Abilene does both active tests, and passive tests. Of particular interest is the router proxy. You can give mediated commands to the router to query state. This can be very useful. You can also issue traceroutes and pings from the Abilene routers.
  • Why does Abilene take all these measurements, and publish the results? (NOTE: Do not read ll of the items on this page – flip through quickly)
  • We currently have four machines at each router node. Here are their roles. (NOTE: show this slide but don’t read through the items) Add slide probably: 1.4ghz PIII, dual bank. Whatever the chipset. Fairly slow, but fastest we could get with a 48VDC supply off the shelf when we were building Abilene.
  • Abilene uses BWCTL for throughput. (NOTE: show this slide but don’t read through the items)
  • Abilene uses OWAMP for latency. (NOTE: show this slide but don’t read through the items)
  • A word on Abilene’s utilization. (NOTE: Do not read any of the items on this page – flip through quickly)
  • And you’ve all seen the weathermap. This is an older version (no Chicago!)
  • There are lots of tools at the NOC page Netflow data is currently at Ohio State And we make weekly summaries to try and understand what traffic is passing over the network, and watch for trends.
  • PPT

    1. 1. Finding Network Problems that Influence Applications: Measurement Tools Internet2 Performance Workshop
    2. 2. Outline <ul><li>Problems, typical causes, diagnostic strategies </li></ul><ul><li>Examples showing usage of the tools we’ll be talking about today </li></ul><ul><li>End-to-End Measurement Infrastructure </li></ul>
    3. 3. We Would Like Your Help <ul><li>What problems are you experiencing? </li></ul><ul><li>Have you used a good tool? </li></ul><ul><li>Give us the benefit of your experience: successful problem resolution! </li></ul>
    4. 4. What Are The Problems? (1) <ul><li>Packet loss </li></ul><ul><li>Jitter </li></ul><ul><li>Out-of-order packets (extreme jitter) </li></ul><ul><li>Duplicated packets </li></ul><ul><li>Excessive latency </li></ul><ul><ul><li>Interactive applications </li></ul></ul><ul><ul><li>TCP’s control system </li></ul></ul>
    5. 5. For TCP <ul><li>Eliminating loss is the goal </li></ul><ul><li>Non-congestive losses especially tricky </li></ul><ul><li>TCP: 100 Mbit Ethernet coast-to-coast: </li></ul><ul><ul><li>Full size packets… need 10 -6 P loss [Mathis] </li></ul></ul><ul><ul><li>Less than 1 loss every 83 seconds </li></ul></ul><ul><li>http://www.psc.edu/~mathis/papers/JTechs200105/ </li></ul><ul><li>GigE: 10 -8 , 1 loss every 497 seconds </li></ul>
    6. 6. What Are The Problems? (2) <ul><li>TCP: lack of buffer space </li></ul><ul><ul><li>Forces protocol into stop-and-wait </li></ul></ul><ul><ul><li>Number one TCP-related performance problem. </li></ul></ul><ul><ul><li>70ms * 1Gbps = 70*10^6 bits, or 8.4MB </li></ul></ul><ul><ul><li>70ms * 100Mbps = 855KB </li></ul></ul><ul><ul><li>Many stacks default to 64KB, or 7.4Mbps </li></ul></ul>
    7. 7. What Are The Problems? (3) <ul><li>Video/Audio: lack of buffer space </li></ul><ul><ul><li>Makes broadcast streams very sensitive to previous problems </li></ul></ul><ul><li>Application behaviors </li></ul><ul><ul><li>Stop-and-wait behavior; Can’t stream </li></ul></ul><ul><ul><li>Lack of robustness to network anomalies </li></ul></ul>
    8. 8. The Usual Suspects <ul><li>Host configuration errors (TCP buffers) </li></ul><ul><li>Duplex mismatch (Ethernet) </li></ul><ul><li>Wiring/Fiber problem </li></ul><ul><li>Bad equipment </li></ul><ul><li>Bad routing </li></ul><ul><li>Congestion </li></ul><ul><ul><li>“ Real” traffic </li></ul></ul><ul><ul><li>Unnecessary traffic (broadcasts, multicast, denial of service attacks) </li></ul></ul>
    9. 9. Strategy <ul><li>Most problems are local… </li></ul><ul><li>Test ahead of time! </li></ul><ul><li>Is there connectivity & reasonable latency? (ping -> OWAMP) </li></ul><ul><li>Is routing reasonable (traceroute) </li></ul><ul><li>Is host reasonable (NDT; Web100) </li></ul><ul><li>Is path reasonable (iperf -> BWCTL) </li></ul>
    10. 10. One Technique: Problem Isolation via Divide and Conquer
    11. 11. Outline <ul><li>Problems, typical causes, diagnostic strategies </li></ul><ul><li>Examples showing usage of the tools we’ll be talking about today </li></ul><ul><li>End-to-End Measurement Infrastructure </li></ul>
    12. 12. Tool Examples <ul><li>When to use NDT </li></ul><ul><li>NDT in action at SC’04 </li></ul><ul><li>When to use BWCTL </li></ul><ul><li>BWCTL in action with e-VLBI </li></ul><ul><li>When to use OWAMP </li></ul><ul><li>OWAMP in action with Abilene </li></ul>
    13. 13. When to use NDT <ul><li>When you want to know about last mile and host problems </li></ul><ul><li>When you want a quick and easy test to provide clues at possible problem cause </li></ul><ul><li>When you want to understand large segments of the path from the host view point </li></ul><ul><li>When a user wants to test their own host </li></ul>
    14. 14. Technique <ul><li>Start by testing to the nearest NDT server from each end of the problem path </li></ul><ul><li>This will help you with a majority of problems </li></ul><ul><li>If test both indicate good performance, test to a distant NDT server </li></ul><ul><li>If tests still indicate good performance, suspect a problem in the application, not the host or network. </li></ul>
    15. 15. SC’04 Real Life Example <ul><li>Booth having trouble getting application to run from Amsterdam to Pittsburgh </li></ul><ul><li>Tests between Amsterdam SGI and Pittsburgh PC showed throughput limited to < 20 Mbps </li></ul><ul><li>Assumption is: PC buffers too small </li></ul><ul><li>Question: How do we set WinXP send/receive buffer </li></ul>
    16. 16. SC’04 Determine WinXP info http://www.dslreports.com/drtcp
    17. 17. SC’04 Confirm PC settings <ul><li>DrTCP reported 16 MB buffers, but test program still slow, Q: How to confirm? </li></ul><ul><li>Run test to SCInet NDT server (PC has Fast Ethernet Connection) </li></ul><ul><ul><li>Client-to-Server: 90 Mbps </li></ul></ul><ul><ul><li>Server-to-Client: 95 Mbps </li></ul></ul><ul><ul><li>PC Send/Recv Buffer size: 16 Mbytes (wscale 8) </li></ul></ul><ul><ul><li>NDT Send/Recv Buffer Size: 8 Mbytes (wscale 7) </li></ul></ul><ul><ul><li>Reported TCP average RTT: 46.2 msec </li></ul></ul><ul><ul><ul><li>approximately 600 Kbytes of data in TCP buffer </li></ul></ul></ul><ul><ul><li>Min buffer size / RTT: 1.3 Gbps </li></ul></ul>
    18. 18. SC’04 Local PC Configured OK <ul><li>No problem found </li></ul><ul><li>Able to run at line rate </li></ul><ul><li>Confirmed that PC’s TCP buffers were set correctly </li></ul>
    19. 19. SC’04 Amsterdam SGI <ul><li>Run test from remote SGI to SC show floor (SGI is Gigabit Ethernet connected) . </li></ul><ul><li>Downloaded and built command line tool on SGI IRIX </li></ul><ul><ul><li>Client-to-Server: 17 Mbps </li></ul></ul><ul><ul><li>Server-to-Client: 16 Mbps </li></ul></ul><ul><ul><li>SGI Send/Recv Buffer size: 256 Kbytes (wscale 3) </li></ul></ul><ul><ul><li>NDT Send/Recv Buffer Size: 8 Mbytes (wscale 7) </li></ul></ul><ul><ul><li>Average RTT: 106.7 msec </li></ul></ul><ul><ul><li>Min Buffer size / RTT: 19 Mbps </li></ul></ul>
    20. 20. SC’04 Amsterdam SGI (tuned) <ul><li>Re-run test from remote SGI to SC show floor with –b # option. </li></ul><ul><ul><li>Client-to-Server: 107 Mbps </li></ul></ul><ul><ul><li>Server-to-Client: 109 Mbps </li></ul></ul><ul><ul><li>SGI Send/Recv Buffer size: 2 Mbytes (wscale 5) </li></ul></ul><ul><ul><li>NDT Send/Recv Buffer Size: 8 Mbytes (wscale 7) </li></ul></ul><ul><ul><li>Reported average RTT: 104 msec </li></ul></ul><ul><ul><li>Min Buffer size / RTT: 153.8 Mbps </li></ul></ul>
    21. 21. SC’04 Debugging Results <ul><li>Team spent over 1 hour looking at Win XP config, trying to verify Buffer size </li></ul><ul><ul><li>2 tools used gave different results </li></ul></ul><ul><li>Single NDT test verified this in under 30 seconds </li></ul><ul><li>10 minutes to download and install NDT client on SGI </li></ul><ul><li>15 minutes to discuss options and run client test with set buffer option </li></ul>
    22. 22. SC’04 Debugging Results <ul><li>8 Minutes to find SGI limits and determine maximum allowable buffer setting (2 MB) </li></ul><ul><li>Total time 34 minutes to verify problem was with remote servers’ TCP send/receive buffer size </li></ul><ul><li>Network path verified but Application still performed poorly until it was also tuned </li></ul>
    23. 23. When to use BWCTL <ul><li>You want to understand segments of the path </li></ul><ul><li>You want to know if each segment can handle flows of a specific size </li></ul><ul><li>You want to know parameters such as bandwidth, packet loss and latency </li></ul><ul><li>To help design or tune an application based on available performance </li></ul>
    24. 24. Technique <ul><li>Divide and Conquer! </li></ul><ul><li>Look for segments with performance less that required by the application </li></ul>
    25. 25. e-VLBI Case Study <ul><li>The e-VLBI project needed to move massive amounts of data between a number of sites around the world </li></ul><ul><li>They found that performance from some sites was only in the 1 Mbps range </li></ul><ul><li>They needed to understand why </li></ul>
    26. 26. e-VBLI test infrastructure <ul><li>David Lapsley, one of the research engineers, established BWCTL servers at the sites of the project. </li></ul><ul><ul><li>Japan: Kashima Observatory </li></ul></ul><ul><ul><li>Sweden: Onsala Observatory </li></ul></ul><ul><ul><li>US: Haystack (BOS) </li></ul></ul><ul><li>He performed a full mesh of tests between all of the servers </li></ul>
    27. 27. e-VLBI Results #1 <ul><li>They used Abilene nodes to divide the problem path </li></ul><ul><li>David found that there was considerable packet loss in the area of Haystack Observatory </li></ul><ul><li>Working with network folk from the area the problem was isolated and resolved </li></ul>
    28. 28. e-VLBI Results #2 <ul><li>For one site that was using a commodity Internet only 1 Mbps was regularly seen </li></ul><ul><li>The application was changed to locate caching to reduce dependence on that site. </li></ul>
    29. 29. e-VLBI Regular Testing <ul><li>They found the testing to be very useful in understanding the network status </li></ul><ul><li>They established a regular testing schedule </li></ul><ul><li>They established a web site for reporting the results </li></ul><ul><li>All researchers can check the network status </li></ul><ul><li>http://web.haystack.mit.edu/staff/dlapsley/tsev7.html </li></ul>
    30. 30. When to use OWAMP <ul><li>Want baseline “heartbeat” information </li></ul><ul><li>Asymmetric routes can make problem location more difficult </li></ul><ul><li>OWAMP can provide detailed performance on one direction in the path </li></ul><ul><li>When you want to know precise latency information </li></ul><ul><li>Good for helping real-time applications </li></ul>
    31. 31. Why use OWAMP <ul><li>It is very sensitive to minor network changes </li></ul><ul><ul><li>Route changes </li></ul></ul><ul><ul><li>Packet queuing </li></ul></ul><ul><li>It tells you about one-direction of the path </li></ul>
    32. 32. OWAMP Case Study Queuing on Abilene <ul><li>Tuesday, 2004-08-17, 16:05-16:20 UTC </li></ul><ul><li>That’s 11:05 to 11:20 EDT </li></ul><ul><li>Caltech to CERN performing 10GE </li></ul><ul><li>throughput experiment </li></ul><ul><li>• Single adapter to date, PCI-X </li></ul><ul><li>• Theoretical limit of ~8.5 Gbps </li></ul><ul><li>• Practical limit closer to 7.5 Gbps </li></ul><ul><li>• Exactly what was tested at that time is unkown </li></ul><ul><li>“ Worst 10” delay list had some larger than </li></ul><ul><li>normal variances… to date, software issues </li></ul>
    33. 33. One Links History <ul><li>The Denver to KSCY Link </li></ul>
    34. 34. What It Shows <ul><li>Only paths that traverse DNVR>KSCY showed additional delay </li></ul><ul><li>Some delayed by ~ an extra 35msec </li></ul><ul><li>Probable cause – Router started queuing packets create a small delay </li></ul><ul><li>It tells you that there is congestion on the link. </li></ul>
    35. 35. Outline <ul><li>Problems, typical causes, diagnostic strategies </li></ul><ul><li>Examples showing usage of the tools we’ll be talking about today </li></ul><ul><li>End-to-End Measurement Infrastructure </li></ul>
    36. 36. End-to-End Measurement Infrastructure Vision <ul><li>Ongoing monitoring to test major elements, and end-to-end paths. </li></ul><ul><ul><li>Elements: gigaPoP links, peering, … </li></ul></ul><ul><ul><li>Utilization </li></ul></ul><ul><ul><li>Delay </li></ul></ul><ul><ul><li>Loss </li></ul></ul><ul><ul><li>Occasional throughput </li></ul></ul><ul><ul><li>Multicast connectivity </li></ul></ul>
    37. 37. End-to-End Measurement Infrastructure Vision II <ul><li>Many more end to end paths than can be monitored. </li></ul><ul><li>Diagnostic tools available on-demand (with authorization) </li></ul><ul><ul><li>Show routes </li></ul></ul><ul><ul><li>Perform flow tests (perhaps app tests) </li></ul></ul><ul><ul><li>Parse/debug flows (a-la tcpdump or OCXmon with heuristic tools) </li></ul></ul>
    38. 38. What Campuses Can Do <ul><li>Export SNMP data </li></ul><ul><ul><li>I have an “Internet2 list”, can add you </li></ul></ul><ul><ul><li>Monitor loss as well as throughput </li></ul></ul><ul><li>Performance test point at campus edge </li></ul><ul><ul><li>Hopefully, the result of today’s workshop </li></ul></ul><ul><ul><li>Possibly also traceroute “looking glass” </li></ul></ul><ul><ul><li>Commercial (e.g., NetIQ) complements </li></ul></ul><ul><ul><li>We have a master list </li></ul></ul>
    39. 39. Strategy (references) (1) <ul><li>See also </li></ul><ul><ul><li>http://e2epi.internet2.edu/ Look at stories, documents, tools </li></ul></ul><ul><ul><li>http://e2epi.internet2.edu/ndt/ Pointer to the tool, and using it for debugging the last mile </li></ul></ul>
    40. 40. Strategy (references) (2) <ul><ul><li>http://www.psc.edu/networking/projects/tcptune/ How to tweak OS parameters (also scp pointer) </li></ul></ul><ul><ul><li>http://www.ncne.org/research/tcp/ TCP debugging the detailed way </li></ul></ul><ul><ul><li>http://dast.nlanr.net/Guides/WritingApps/ Tips for app writers </li></ul></ul><ul><ul><li>http://dast.nlanr.net/Guides/GettingStarted And some checking to do by hand & debugging. </li></ul></ul>
    41. 41. www.internet2.edu
    42. 42. Acknowledgements <ul><li>The original presentation by Matt   Zekauskas using ideas inspired by material from NLANR DAST, Matt   Mathis, and others. </li></ul><ul><li>Copyright Internet2 2005, All Rights Reserved. </li></ul>
    43. 43. Background: Detailed Tools Discussion
    44. 44. Bakground: Tools Outline <ul><li>Tools: First mile, host issues </li></ul><ul><li>Tools: Path issues </li></ul><ul><li>Tools: Others to be aware of </li></ul><ul><li>Tools within Abilene </li></ul>
    45. 45. Internet2 Detective <ul><li>A simple “is there any hope” tool </li></ul><ul><ul><li>Windows “tray” application </li></ul></ul><ul><ul><li>Red/green lights, am I on Internet2 </li></ul></ul><ul><ul><li>Multicast available </li></ul></ul><ul><ul><li>IPv6 available </li></ul></ul><ul><li>http://detective.internet2.edu/ </li></ul>
    46. 46. NLANR Performance Advisor <ul><li>Geared for the naive user </li></ul><ul><li>Run at both ends, and see if a standard problem is detected. </li></ul><ul><li>Can also work with intermediate servers </li></ul><ul><li>http://dast.nlanr.net/Projects/Advisor </li></ul>
    47. 47. NDT <ul><li>Network Debugging Tool </li></ul><ul><li>Java applet </li></ul><ul><li>Connects to server in middle, runs tests, and evaluates heuristics looking for host and first mile problems. </li></ul><ul><li>Has detailed output. </li></ul><ul><li>You’ll see lots of detail later today. </li></ul><ul><li>A commercial tool that tests for TCP buffer problems: http://www.dslreports.com/tweaks/ </li></ul>
    48. 48. Host/OS Tuning: Web100 <ul><li>Goal: TCP stack, tuning not bottleneck </li></ul><ul><li>Large measurement component </li></ul><ul><ul><li>TCP performance not what you expect? Ask TCP why! </li></ul></ul><ul><ul><ul><li>Receiver bottleneck (out of receiver window) </li></ul></ul></ul><ul><ul><ul><li>Sender bottleneck (no data to send) </li></ul></ul></ul><ul><ul><ul><li>Path bottleneck (out of congestion window) </li></ul></ul></ul><ul><ul><ul><li>Path anomalies (duplicate, out of order, loss) </li></ul></ul></ul><ul><li>www.web100.org </li></ul>
    49. 49. Reference Servers (Beacons) <ul><li>H.323 conferencing </li></ul><ul><ul><li>Goal: portable machines that tell you if system likely to work (and if not, why?) </li></ul></ul><ul><ul><li>Moderate-rate UDP of interest </li></ul></ul><ul><ul><li>E.g., H.323 Beacon http://www.osc.edu/oarnet/itecohio.net/beacon/ </li></ul></ul><ul><ul><li>ViDeNet Scout, http://scout.video.unc.edu/ </li></ul></ul>
    50. 50. Background: Tools Outline <ul><li>Tools: First mile, host issues </li></ul><ul><li>Tools: Path issues </li></ul><ul><li>Tools: Others to be aware of </li></ul><ul><li>Tools within Abilene </li></ul>
    51. 51. OWAMP – Latency/Loss <ul><li>One-Way Active Measurement Protocol </li></ul><ul><li>Requires NTP-Synchronized clocks </li></ul><ul><li>Look for one-way latency, loss </li></ul><ul><li>Authentication and Scheduling </li></ul><ul><li>Again, lots more later today </li></ul>
    52. 52. BWCTL -- Throughput <ul><li>A tool for throughput testing that includes scheduling and authentication. </li></ul><ul><li>Currently uses iperf for actual tests. </li></ul><ul><li>Can assign users (or IP addresses) to classes, give classes different throughput limits or time limits. </li></ul><ul><li>Periodic and on-demand testing. </li></ul><ul><li>Lots more later today. </li></ul>
    53. 53. Background: Tools Outline <ul><li>Tools: First mile, host issues </li></ul><ul><li>Tools: Path issues </li></ul><ul><li>Tools: Others to be aware of </li></ul><ul><li>Tools within Abilene </li></ul>
    54. 54. Some Commercial Tools <ul><li>Caveat: only a partial list, give me more! </li></ul><ul><li>Spirent (nee Netcom/Adtech): </li></ul><ul><ul><li>SmartBits: test at low & high rates, QoS; test components or end-to-end path </li></ul></ul><ul><li>NetIQ: Chariot/Pegasus </li></ul><ul><li>Agilent (like SmartBits, and FireHunter) </li></ul><ul><li>Ixia (like SmartBits/Spirent) </li></ul><ul><li>Brix Networks (like AMP/Owamp, for ‘QoS’) </li></ul><ul><li>Apparent Networks: path debugger </li></ul>
    55. 55. Some Noncommercial Tools <ul><li>Iperf: dast.nlanr.net/Projects/iperf </li></ul><ul><ul><li>See also http://www-itg.lbl.gov/nettest/ </li></ul></ul><ul><ul><li>http://www-didc.lbl.gov/NCS/ </li></ul></ul><ul><li>Flowscan: </li></ul><ul><ul><li>http://www.caida.org/tools/utilities/flowscan/ </li></ul></ul><ul><ul><li>http://net.doit.wisc.edu/~plonka/FlowScan/ </li></ul></ul><ul><li>SLAC’s traceroute perl script: </li></ul><ul><ul><li>http://www.slac.stanford.edu/comp/net/wan-mon/traceroute-srv.html </li></ul></ul><ul><li>One large list: </li></ul><ul><ul><li>http://www.slac.stanford.edu/xorg/nmtf/nmtf-tools.html </li></ul></ul>
    56. 56. Background: Tools Outline <ul><li>Tools: First mile, host issues </li></ul><ul><li>Tools: Path issues </li></ul><ul><li>Tools: Others to be aware of </li></ul><ul><li>Tools within Abilene </li></ul>
    57. 57. Abilene: Measurements from the Center <ul><li>Active (latency, throughput) </li></ul><ul><ul><li>Measurement within Abilene </li></ul></ul><ul><ul><li>Measurements to the edge </li></ul></ul><ul><li>Passive </li></ul><ul><ul><li>SNMP stats (esp. core Abilene links) </li></ul></ul><ul><ul><li>Variables via router proxy </li></ul></ul><ul><ul><li>Router configuration </li></ul></ul><ul><ul><li>Route state </li></ul></ul><ul><ul><li>Characterization of traffic </li></ul></ul><ul><ul><ul><li>Netflow; OCxMON </li></ul></ul></ul>
    58. 58. Goal <ul><li>Abilene goal to be an exemplar </li></ul><ul><ul><li>Measurements open </li></ul></ul><ul><ul><li>Tests possible to router nodes </li></ul></ul><ul><ul><li>Throughput tests routinely through backbone </li></ul></ul><ul><ul><li>… as well as existing utilization, etc. </li></ul></ul><ul><ul><li>The “Abilene Observatory” http://abilene.internet2.edu/observatory </li></ul></ul>
    59. 59. Abilene: Machines <ul><li>GigE connected high-performance tester </li></ul><ul><ul><li>bwctl, “nms1”, 9000 byte MTU </li></ul></ul><ul><li>Latency tester </li></ul><ul><ul><li>owamp, “nms4”, 100bT </li></ul></ul><ul><li>Stats collection </li></ul><ul><ul><li>SNMP, flow-stats, “nms3”, 100bT </li></ul></ul><ul><li>Ad-hoc tests </li></ul><ul><ul><li>NDT server, “nms2”, gigE, 1500 byte MTU </li></ul></ul>
    60. 60. Throughput <ul><li>Take tests 1/hr, 20 seconds each </li></ul><ul><ul><li>IPv4 TCP </li></ul></ul><ul><ul><li>IPv6 TCP (no discernable difference) </li></ul></ul><ul><ul><li>IPv4 UDP (on our platforms flakey at 1G) </li></ul></ul><ul><ul><li>IPv6 UDP (ditto) </li></ul></ul><ul><li>Others test to our nodes </li></ul><ul><li>Others test amongst themselves </li></ul><ul><li>Net result: 25% of traffic (NOT capacity) is measurement </li></ul>
    61. 61. Latency <ul><li>CDMA used to synchronize NTP </li></ul><ul><ul><li>www.endruntechnologies.com </li></ul></ul><ul><li>Test among all router node pairs </li></ul><ul><li>10/sec </li></ul><ul><li>IPv4 and IPv6 </li></ul><ul><li>Minimal sized packets </li></ul><ul><li>Poisson schedule </li></ul>
    62. 62. Passive - Utilization <ul><li>The Abilene NOC takes </li></ul><ul><ul><li>Packets in,out </li></ul></ul><ul><ul><li>Bytes in,out </li></ul></ul><ul><ul><li>Drops/Errors </li></ul></ul><ul><ul><li>..for all interfaces, publishes internal links & peering points (at 5 min intervals) </li></ul></ul><ul><ul><li>..via SNMP polling – every 60 sec </li></ul></ul><ul><li>http://loadrunner.uits.iu.edu/weathermaps/abilene/abilene.html </li></ul>
    63. 64. Abilene Pointers <ul><li>http://www.abilene.iu.edu/ </li></ul><ul><ul><li>Monitoring </li></ul></ul><ul><ul><li>Tools </li></ul></ul><ul><li>http://www.itec.oar.net/abilene-netflow </li></ul><ul><li>http://netflow.internet2.edu/weekly/ (summaries) </li></ul>