Troubleshooting the run-of-the-mill IP protocols is fairly straightforward. Your web server provides copious logging of hits and errors. You can debug on the browser side. The same goes for email – you have logs to tell you what’s going on, you get bounces on errors. When one of your endpoints is a phone though (with a user on the end) things change. Does that fast busy mean that you’re out of trunks or some other error has occurred? Your PBX may have some logging to help, but that’s only one half of the conversation. PBXes also log so that vendors and PBX people can understand them. What about data people? And what about voice? Most of the protocols we deal with day to day run over TCP, where error correction is provided for us. If that FTP transfer take an extra second, who notices? But when we get data problems in the voice stream, people notice. How to find the cause of the problem?
Wireshark is hopefully a familiar tool to you. It has some advanced capabilities to help you interpret voice traffic and make sense of the complex signaling that goes on in the background. With some general know-how and Wireshark, you can debug voice problems just as easily as you do HTTP and SMTP. Over the course of this seminar you will learn the various ways that Wireshark helps expose VoIP problems, which will go a long way to solving them. Note that I said “expose”. Wireshark won’t solve your problems, but it will certainly help you find out where they are.
This presentation is structured around the steps you might use to solve a VoIP problem. First you will capture the traffic. I’ll show you how capturing VoIP traffic is slightly different than other forms of traffic that you might be used to. Next I’ll go over some of the ways that Wireshark dissects individual packets. Sometimes the problem is obvious. Other times it’s not, so you will use some of the analysis tools to look at the capture and find the packets you need. Finally we’ll look at the voice traffic itself. Wireshark has some helpful tools to pull statistics from the voice stream, including replaying the conversation.
I’ve targeted this presentation to people who have used Wireshark before, but may be new to VoIP. Who has/hasn’t used WS before? - Who has/hasn’t deployed VoIP? - Who is using VoIP over the corporate network vs the Internet?
The first step to analyzing VoIP traffic is to capture it. This may seem obvious, but there are several important things to keep in mind so that you end up with a good trace. It is just as important to know how you are measuring something as it is to know how to read the measurements. This first step of capturing the traffic is therefore vital.
Like real estate, a good location from which to capture the traffic is necessary. Your location dictates whether or not you see all the traffic, and how the trace reflects network conditions.
This is a fairly simple VoIP network that may or may not look like anything you have. It was designed to be simple but to highlight many of the challenges in network analysis of VoIP traffic. The cloud in the middle can be the corporate WAN or the Internet. Circle objects are routers, the boxes with arrows are switches. The top two blue boxes with the arrows and phones are PBXes. Take a guess which icon is the phone. I haven’t shown any PSTN gateways, but they act much the same as a phone.
Perhaps the most critical thing to understand about VoIP is that the signaling traffic and the voice traffic are separate. The telephone signals its pbx to set up a call, which might set off more signaling traffic. Eventually the phone stops signaling, and starts speaking real time protocol (RTP). **analogy to POTS ** The path of the signaling depends on how the PBXes are configured to handle the traffic.
In this case the phone talks RTP to the PBX, who talks RTP to the other PBX, who talks RTP to the phone. This is three separate conversations that have been bridged together. Nothing is inherently wrong about this configuration, you will see if often. However, consider what would happen if you took your packet trace on one of the parts of the network that sees the traffic twice. Seeing traffic twice isn’t necessarily a bad thing unless it causes you to misinterpret your observations.
When analyzing VoIP you are not only interested in the contents of the packets, but the time at which they are received. These packets contain parts of a real time voice conversation. One particular problem happens when you are looking at the same conversation from two different vantage points. You will not see any jitter on one side of the conversation because the packet has not traversed enough elements to experience jitter. This can be helpful though. With a binary search you can find the source of problems. You don’t need to use Wireshark directly to capture data. Running tcpdump and tethereal on a remote host on a SPANned port and copying your capture file locally work just as well. SPANning is a Cisco term that means that a port or VLAN is mirrored to a different port. Most managed switches support this feature, but will call it a something else. If you are using Cisco equipment, make sure to learn about RSPAN. Remote SPAN lets you redirect the captured traffic to a port on a different switch, which makes it easier to quickly monitor a port anywhere on your LAN.
To further complicate the issue, you might have NAT involved. NAT changes the source or the destination address of the packet in order to alter routing, or to hide devices behind a gateway. NAT often causes problems with VoIP network because of this change in address, especially when transiting the Internet. You must understand how the addresses are being changed, and adjust your capture filters accordingly depending on which side of the NAT gateway you’re on.
Wireshark uses two types of filters, capture and display. Capture filters describe what data is pulled off the wire. Display filters allow you to show or hide data that’s already been captured. Capturing too much data means that you’ll have a lot of stuff to sift through as you troubleshoot. However, data that you don’t capture can’t be analyzed. To make matters worse, some VoIP protocols don’t run on predictable ports. RTP runs on random UDP ports. SIP normally runs on 5060/UDP, but if you have a two line phone, it can also run on 5061. H.323 runs on many different ports. A capture filter of “udp” is generally good enough for SIP because there’s usually not much background UDP traffic. Filtering by hosts is also good, but you have to make sure that you know which hosts are involved in the conversation.
The packet list window shows a great amount of detail about the packets that Wireshark can decode.
You will find information about the protocol under the protocol heading. Some protocols have sub protocols, such as the Session Description Packet within SIP, and those are displayed here (see the green box) The info column usually has some more details about what’s going on in the particular packet. Look at the red box, you can see that an invitation is made, with a response requiring authentication. This repeats continually. The problem here is an authentication problem.
Any network where Voice competes with other traffic needs quality of service to guarantee the needs of voice. QoS tags a packet, usually at layer 3, with a value of 2e hex to tag the packet for expedited forwarding. The TOS byte in the IP header is where this value goes. There is a corresponding three bits at layer 2, but it is not carried over a router hop or over a WAN. When you tag a packet with a particular value, every network element can handle the packet accordingly instead of the usual first-in-first-out treatment. On a router, this might mean putting Voice at the head of the queue, and ensuring Citrix gets 25% of the remaining traffic. The possibilities are endless with QoS, but only if traffic is tagged properly.
Here, I’ve set a coloring rule to make any packet set for expedited forwarding have a red foreground and a white background. From View, coloring rules, you can add a color filter. By default the new filter will go at the bottom, so you want to push it up above the UDP rule of black on blue. Once the coloring rule is in, untagged voice traffic shows up easily. Here, traffic from 192.168.1.156 is tagged EF but the return traffic isn’t. You would then start your troubleshooting at 192.168.1.1. The color method is much superior to digging into each packet and looking through the IP header for the QoS tag. (show on “voip call through lesnet with sip and asymetric qos.cap”) Misconfigured QoS usually manifests itself as poor voice quality.
Wireshark gave some information about the RTP traffic in the packet list pane, much like it did for the signaling traffic earlier. However, Wireshark knew the signaling traffic was SIP because it was using UDP port 5060. RTP does not have a defined port, so Wireshark uses the signaling details to figure out which port is being used for RTP, and then decodes the packets accordingly. What if you have a proprietary phone system that uses RTP, but has its own signaling? You can tell Wireshark to look more closely at UDP packets to determine if they are RTP by checking the “Try to decode RTP outside of conversations” box. Demonstrate on “rtp with no signaling”
Pop up lesnet and show details about SIP
Explain the flow of a typical call Some signaling protocols are more detailed Some protocols are more stupid (SCCP)
Baseline – open voip call through lesnet with SIP Pick out some packets (ie #4), not easy to find stuff or select between conversations Statistics -> VoIP calls Pick a conversation, show some of the fields Use “prepare filter” to generate a filter based on frame number Back to Statistics->VoIP calls, pick the conversation and click graph Show the handshaking, explain SIP process Click on packet to show the packet in the display Select authentication phase Explain role of SDP Open nat problems capture. User is complaining of one way audio Stats->VoIP calls. Follow the sequence. Looks OK One way audio confirmed, note RTP is to a different host Look back at SDP, specifically the C and M lines Show H.323 and Skinny. For skinny show some of the display filters
Fax/modem calls can’t have any loss… consider relaying
Statistics->VoIP->Player, look at the graph, play back Statistics->RTP->Show All - directions - next non-ok - save payload
Walberg-expose vo_ip problems with wireshark
Exposing VoIP problemswith WiresharkApril 2, 2008Sean WalbergNetwork Guy | CanwestSHARKFEST 08Foothill CollegeMarch 31 - April 2, 2008 SHARKFEST 08 | Foothill College | March 31 - April 2, 2008
Signaling protocols SIP (from the IETF) H.323 (from the ITU) MGCP IAX SS7 (Telco) GSM (Telco/Cell) SCCP (Cisco Skinny) Vendor specific
The role of signaling Indicate to the remote end that a call is coming Establish the codec to be used for voice Establish the addresses of the endpoints Get out of the way Tear down the connection once it’s done
The properties of RTP RTP simulates the real time voice normally carried over a wire 4KHz voice bandwidth = 8KHz sampling rate (Nyquist) 8 bits/sample * 8KHz = 64,000bps (DS0) A Codec (G.711u/A law, G.729, G.726, etc) Most codecs use 20ms voice samples = 50pps Even with compression, you have a fairly consistent packet rate, only the size changes
Three factors that affect voice quality Latency <= 150ms (one way) Jitter <= 20ms Packet loss <= 0.1%
Latency <= 150ms (one way) Jitter buffer, Transcoding Path delay delay Serialization delay Hi, how are you? Hello? Oops, sorry, go ahead Fine, I oh hello, go ahead
Packet Loss <= 0.1% Hi Bo *POP* How *POP*e you? Hi Bo How you?