2. Chapter Content A few words about troubleshooting tools Wireshark โ basics Wireshark โ advanced features Case studies
3.
4.
5.
6.
7.
8.
9.
10. Were to Locate the Wireshark? To ISP For server monitoring: Connect the laptop to the LAN switch, with port mirror to the monitored server For WAN monitoring: Connect the laptop to the LAN switch, with port mirror to the monitored router For Internet connectivity monitoring: Before or after the Firewall
11. Chapter Content A few words about troubleshooting tools Wireshark โ basics Wireshark โ advanced features Case studies
15. TCP/IP Protocol Stack - Reminder IP ICMP TCP UDP Telnet SNMP HTTP FTP DNS SMTP ARP OSI Layer 1/2 OSI Layer 3 OSI Layer 4 OSI Layer 5-7 T.R. F.R. Ethernet DialUp ISDN ATM
16. Data Structure Over- head Data Layer 4 Err (Op.) Data Over- head Layer 3 Err (Op.) Data Layer 1 Over- head Data Layer 2 Err (Op.) Over- head Data Layer 5-7 Err (Op.)
18. Data Flow Server Router Router Public Data Network Eth. Eth. Host Bit stream OH Data E IP (L3) OH Data E TCP (L4) OH Data E HTTP (L-5/6/7) OH Data E Ethernet (L2) Bit stream OH Data E OH Data E OH Data E OH Data E FR (L2) Bit stream OH Data E OH Data E OH Data E OH Data E
19. Frame Format โ Ethernet II / 802.3 bytes Dest. Address Source Address Type 6 6 2 IP IPX AppleTalk CRC 4 Pad Data PA 8 Ethernet II IEEE 802.3 Dest. Address Source Address Length 6 6 2 CRC 4 Pad Length Data PA SFD 7 1
21. IP Datagram Format H Data E Ethernet (L2) H Data IP (L3) H Data E TCP (L4) H Data E HTTP (L-5/6/7) This is the IP header Bit stream
22. IP Datagram Format Ver Length 32 bits Data (variable length, typically a TCP or UDP segment) 16-bit identifier Internet checksum Time to live 32 bit source IP address Head. len Type of service flgs Fragment offset Upper layer 32 bit destination IP address Options (if any) IP protocol version number Header Length (in bytes โ Typeโ of data Total datagram length (in bytes For fragmentation and reassembly Max. no. remaining hops (decremented at each router) Upper layer protocol to which payload is delivered E.g. timestamp, record route taken, specify list of routers to visit
26. TCP Message Structure source port # dest port # 32 bits application data (variable length) sequence number acknowledgement number rcvr window size ptr urgent data checksum F S R P A U head len not used Options (variable length) URG โ Urgent data (generally not used ACK: ACK # valid PSH - Push data now RST โ Connection RESET Ack numbers to confirm data arrival # of bytes rcvr is willing to accept SYNC โ Start session FIN โ End session In case of URG pointer, indicates the data location Options Numbering of sent data Port Numbers
30. And You Will Get: Packet List Packet Details Packet Bytes
31. Or โ Define Capture Options Buffer size โ in order not to fill your laptop disk Capture all packets on the network Capture filter Capture in multiple files When to automatically stop the capture Display options Name resolution options
32. And if you want to see some details: Example (W-LAN): Received Signal Strength Indication (RSSI) and Link speed (BW)
47. Statistics โ Conversations - What can we do with it? On Layer-2 (Ethernet) โ To find and isolate broadcast storms And then to go to the switch, and find the troublemaker
48. Statistics โ Conversations - What can we do with it? On Layer-3/4 (TCP/IP) โ To connect in parallel to the Internet router port, and check who is loading the line to the ISP And then to go to him/her, and ask questions โฆโฆ
69. Example A - Stable Performance File Transfer A stable throughput of around 1MB/8Mb per second It is important to test in parallel with SNMP tool for channel capacity
71. Example B โ Non-Stable Performance File Transfer Something happened here (After ~5.25 Seconds)
72. Example B โ Non-Stable Performance File Transfer 5.25 seconds after start of stream, we donโt see any connectivity problems โ probably slow server/applications
82. Case #3 โ Slow DB Response 10.2.1.105 10.1.1.7 Frame Relay Network (Year 2000)
83. Case #3 โ Slow DB Response Connection Establishment
84. Case #3 โ Slow DB Response And more packets (900+ since beginning of connection) โฆ..
85. Case #3 โ Slow DB Response And more packets (2000+ since beginning of connection) โฆ.. 40mS delay between packets 2000Packets * 40mSec = 80Sec application delay !!!
There are three major types of debugging tools: Standard PC (or operating systems) tools โ all the standard applications that you can run from the standard command line on your PC or on the UNIX machine Access to communication devices โ switches, routers, etc Protocol analyzers โ applications that analyze packets and protocols that runs on the network SNMP tools โ applications and software's that monitors MIB (Management Information Base) continuously, and therefore can be used also for network troubleshooting Special tools โ Netflow, Solarwinds and other tools for engineering and special case monitoring
CLI tools, like ping, tracert (or traceroute โ depends on the OS), will give you an initial โfeelingโ of the network. You can get delay, jitter and packet loss, with simple ping, and reachability tests with trace.
Telnet or web connectivity to communication devices will give you much more data. You will be able to get the number of input and output packets on an interface, number of errors, CPU utilization, packet size distribution and much more
Wireshark โ well, this is the purpose of our seminar, so you will see โฆโฆ.
SNMP tools, like SNMPc, MRTG, Whatsup Gold, HPOV-NNM and others, are installed on a dedicated platfor, that continuously monitors the network, gives us a networks map, event browser and other features, depends on the software. For troubleshooting purposes, we will use the monitoring features, that will gove us continues monitoring of network parameters.
There are special tools like Netflow, Loggers etc. For example, in Netflow, we can get accurate statistics, of who is using the network (by IP address), what is he doing (by port numbers โ http, mail etc.) and more. There are many tools for these purposes.
IF you need more then this, for example simulating network conditions, you can use software tools (for example Shunra), od hardware devices that will simulate error patterns, load, application loads and more.
The wireshark, which is a protocol analyzer, will be located to monitor specific traffic flow in the network. It can be located to monitor a server, a router, a group of users etc.
Monitor or Mirror port are a configurable ports on the switch, and is configured in a way that: The laptop is connected to the physical port that was configured as Monitor/Mirror port The server, router or any monitored device are connected to the Monitored/Mirrored port Every port can be configured to be a Monitor or Monitored port. Depends on the vendor, monitoring can be enhanced to several ports, and filters can be set on incoming/outgoing ports, MAC addresses and more.
Wireshark 1.2.0 was released in June 15, 2009. This is the new stable release branch of Wireshark with many new features. One of the main differences is the web look-like interface, that gives you easier access to the Wireshark functions.
Here are some examples people use Wireshark for: network administrators use it to troubleshoot network problems network security engineers use it to examine security problems developers use it to debug protocol implementations people use it to learn network protocol internals Beside these examples, Wireshark can be helpful in many other situations too.
If weโll go back to the OSI-RM definitions, layers 1 and 2 are the LAN and WAN protocols. TCP works on any of them. In layer 3, the protocols that provides end to end connectivity is the IP โ Internet Protocol. In parallel to the IP, there are other special purpose protocols, like ICMP (Ping command) ARP (Address Resolution Protocol) is used for address resolution between layer-2 LAN and layer-3 IP protocols In layer 4 we have two protocols for application connectivity โ TCP (Transport Control Protocol) which is a connection-oriented, reliable protocol, and UDP (User Datagram Protocol), which is an unreliable, connection-less protocol. In layers 5 to 7, the โupper layersโ, we have two types of protocols: Those who requires reliability, like FTP, HTTP and others โ they work on the top of reliable TCP infrastructure. Of course, working over TCP slows the operation Those who does not requires reliability, or does require speed โ they work on the top of the faster, unreliable UDP.
In data networks, the basic data unit is called PDU โ Packet Data Unit The formal definition is: Frame - PDU in layer 2 Packet โ PDU in layer 3 Message โ PDU in layer 4 and above For convenience, In the seminar we will refer to all of them as โpacketsโ The important thing is that every data field in a packet carries the upper layer packet: Layer 1 is bits and therefore no packet is defined Layer 2 data field carries layer 3 packet Layer 3 data field carries layer 4 packet Layer 4 data field carries layer (5-6)+7 packet (layer 5 is a session management layer and layer 6 is data definition layer, and therefore they donโt have any dedicated packets) Every packet will have the following fields as overhear (head or tail): Start of Frame (in some cases also End of Frame) Layer identifier. In layer 2 โ layer two addresses (for example MAC address), in layer 3 โ layer three addresses (for example IP address), in layer 4 โ layer four addresses (for example TCP/UDP port numbers)
Wireshark's main window consists of parts that are commonly known from many other GUI programs. The menu is used to start actions. The main toolbar provides quick access to frequently used items from the menu. The filter toolbar provides a way to directly manipulate the currently used display filter. The packet list pane displays a summary of each packet captured. By clicking on packets in this pane you control what is displayed in the other two panes. The packet details pane displays the packet selected in the packet list pane in more detail. The packet bytes pane displays the data from the packet selected in the packet list pane, and highlights the field selected in the packet details pane. The statusbar shows some detailed information about the current program state and the captured data.
Now look at what happens, when we will click www.cellcom.co.il on our web browser. On the sender side: The sender opens the web browser and type http://www.cellcom.co.il The PC creates an HTTP frame (layer-7), with the http parameters The HTTP frame is inserted into layer-4 TCP frame. The PC TCP software marks a destination port number with code 80 (HTTP), and with a random source port (to tell the receiver to which port to send the answer) The TCP frame is inserted into layer-3 IP frame which add source and destination IP addresses The IP frame is inserted into layer-2 Ethernet frame that adds MAC addresses, and takes the packet through the LAN to the router, on the way to the destination. On the way: The routers on the way to the destination, opens the packets to layer-3, looks at the destination IP address, makes routing decisions, and forward the frames to the destination The routers, in case of different interfaces, also takes the IP packet out of the source layer-2 frame, and insert it into the destination layer-2 frame At the destination: The receiving server, gets the Ethernet packet from the network. It looks at the Ethernet header (Type field), and see that the layer-3 protocol is IP. The server extract the IP frame from the Ethernet frame, and forward into the IP process that runs on it. The IP process looks at the IP packet, and see that the layer-4 protocol is TCP. It extract the layer-4 data from it, and forward it to the TCP process on the server. The layer-4 TCP process, the server looks at the port number, see 80, which indicates โHTTPโ, and forward it to the HTTP server
Ethernet-2: Used for TCP/IP as well as many other protocols. In general - most common frame type Ethernet Header = 18 Bytes [Dst Mac(6) + Src Mac(6) + Frame Type (2) +CRC(4)] Minimum Data Portion = 46 Bytes Minimum Ethernet Frame Size = 64 Bytes Ethernet II type field values are greater than 0x5FF (1535) while 802.3 length field values are less then 0x5EE (1518). This is how Ethernet II and 802.3 frames are differentiated. IEEE 802.3: Ethernet Header = 18 Bytes [Dst Mac(6) + Src Mac(6) + Length (2) +CRC(4)] Minimum Data Portion = 46 Bytes Minimum Ethernet Frame Size = 64 Bytes There can only be one L3 protocol that uses 802.3-only encapsulation on a host because there isn't anything in the 802.3 header to differentiate between different L3 packets.
Here we can see an example for a packet carrying FTP data. We have opemed the Ethernet frame to see itโe details.
Now, we will discuss the IP heder.
Version The first header field in an IP packet is the four-bit version field. For IPv4, this has a value of 4 (hence the name IPv4). Internet Header Length (IHL)ย The second field (4 bits) is the Internet Header Length (IHL) telling the number of 32-bit words in the header. Since an IPv4 header may contain a variable number of options, this field specifies the size of the header (this also coincides with the offset to the data). The minimum value for this field is 5 (RFC 791), which is a length of 5ร32 = 160 bits. Being a 4-bit value, the maximum length is 15 words or 480 bits. Differentiated Services (DS) Originally defined as the TOS field, this field is now defined by RFC 2474 for Differentiated services (DiffServ) and by RFC 3168 for Explicit Congestion Notification (ECN). The original intention of the Type of Services (TOS) field was for a sending host to specify a preference for how the datagram would be handled as it made its way through an internet. For instance, one host could set its IPv4 datagrams TOS field value to prefer low delay, while another might prefer high reliability.
Total Length This 16-bit field defines the entire datagram size, including header and data, in bytes. The minimum-length datagram is 20 bytes (20-byte header + 0 bytes data) and the maximum is 65,535 โ the maximum value of a 16-bit word. The minimum size datagram that any host is required to be able to handle is 576 bytes, but most modern hosts handle much larger packets. Sometimes subnetworks impose further restrictions on the size, in which case datagrams must be fragmented. Fragmentation is handled in either the host or packet switch in IPv4. Identificationย This field is an identification field and is primarily used for uniquely identifying fragments of an original IP datagram. Some experimental work has suggested using the ID field for other purposes, such as for adding packet-tracing information to datagrams in order to help trace back datagrams with spoofed source addresses. Flags Are used for the purpose of packet fragmentation. Fragment Offsetย The fragment offset field, measured in units of eight-byte blocks, is 13 bits long and specifies the offset of a particular fragment relative to the beginning of the original unfragmented IP datagram. The first fragment has an offset of zero. This allows a maximum offset of (213 โ 1) ร 8 = 65,528 which would exceed the maximum IP packet length of 65,535 with the header length included. Time To Live (TTL) An eight-bit time to live (TTL) field helps prevent datagrams from persisting (e.g. going in circles) on an internet. This field limits a datagram's lifetime. It is specified in seconds, but time intervals less than 1 second are rounded up to 1. In latencies typical in practice, it has come to be a hop count field. Each router that a datagram crosses decrements the TTL field by one. When the TTL field hits zero, the packet is no longer forwarded by a packet switch and is discarded. Typically, an ICMP message (specifically the time exceeded) is sent back to the sender that it has been discarded. The reception of these ICMP messages is at the heart of how traceroute works. Protocol This field defines the protocol used in the data portion of the IP datagram. The Internet Assigned Numbers Authority maintains a list of Protocol numbers which was originally defined in RFC 790. Common protocols and their decimal values are shown below. Header Checksum The 16-bit checksum field is used for error-checking of the header. At each hop, the checksum of the header must be compared to the value of this field. If a header checksum is found to be mismatched, then the packet is discarded. Note that errors in the data field are up to the encapsulated protocol to handle โ indeed, both UDP and TCP have checksum fields. Since the TTL field is decremented on each hop and fragmentation is possible at each hop then at each hop the checksum will have to be recomputed. The method used to compute the checksum is defined within RFC 791. Source addressย The IP address of the sender of the packet Destination addressย The IP address of the intent receiver of the apacket. Options Additional header fields may follow the destination address field, but these are not often used.
And some example for an IP packet.
Source Port - The 16-bit port number of the process that originated the UDP message on the source device. This will normally be an ephemeral (client) port number for a request sent by a client to a server, or a well-known/registered (server) port number for a reply sent by a server to a client. Destination Port - The 16-bit port number of the process that is the ultimate intended recipient of the message on the destination device. This will usually be a well-known/registered (server) port number for a client request, or an ephemeral (client) port number for a server reply. Length - The length of the entire UDP datagram, including both header and Data fields. Checksum - An optional 16-bit checksum computed over the entire UDP datagram plus a special โpseudo headerโ of fields.
Source Port The 16-bit source port number, used by the receiver to reply. Destination Port The 16-bit destination port number. Sequence Number The sequence number of the first data byte in this segment. If the SYN control bit is set, the sequence number is the initial sequence number (n) and the first data byte is n+1. Acknowledgment Number If the ACK control bit is set, this field contains the value of the next sequence number that the receiver is expecting to receive. Data Offset The number of 32-bit words in the TCP header. It indicates where the data begins. Reserved Six bits reserved for future use; must be zero.
And some example for TCP packet.
One of the following methods can be used to start capturing packets with Wireshark: You can get an overview of the available local interfaces using the " Capture Interfaces" dialog box. You can start a capture from this dialog box, using (one of) the "Capture" button(s). You can start capturing using the "Capture Options" dialog box. If you have selected the right capture options before, you can immediately start a capture using the " Capture Start" menu / toolbar item. The capture process will start immediately.
And this is what you will get.
In its simple form a packet sniffer simply captures all of the packets of data that pass through a given network interface. Typically, the packet sniffer would only capture packets that were intended for the machine in question. However, if placed into promiscuous mode, the packet sniffer is also capable of capturing ALL packets traversing the network regardless of destination.
When you select Details from the Capture Interface menu, Wireshark pops up the "Interface Details" dialog box as shown in the figure. This dialog shows various characteristics and statistics for the selected interface.
Here we see what happens when we open an HTTP session: First packet is SYNC sequence The second packet is SYNC and ACK The third packet is another ACK
And to get the flow from the Wireshark, you have to choose Flow Graph from Statistics, and you will get โฆ
A graphical representation of what youโve tried so much to draw!
What are Retransmissions, Duplicate Ack, Previous segment loss โฆ. We will see later.
Examples: Capture only traffic to or from IP address 172.18.5.4 : host 172.18.5.4 Capture traffic to or from a range of IP addresses : net 192.168.0.0/24 or net 192.168.0.0 mask 255.255.255.0 Capture traffic from a range of IP addresses : src net 192.168.0.0/24 or src net 192.168.0.0 mask 255.255.255.0 Capture traffic to a range of IP addresses : dst net 192.168.0.0/24 or dst net 192.168.0.0 mask 255.255.255.0
Capture only DNS ( port 53 ) traffic : port 53 Capture non - HTTP and non - SMTP traffic on your server ( both are equivalent ): host www . example . com and not ( port 80 or port 25 ) host www . example . com and not port 80 and not port 25 Capture except all ARP and DNS traffic : port not 53 and not arp Capture traffic within a range of ports ( tcp [ 2:2 ] > 1500 and tcp [ 2:2 ] < 1550 ) or ( tcp [ 4:2 ] > 1500 and tcp [ 4:2 ] < 1550 ) or, with newer versions of libpcap ( 0.9.1 and later: tcp portrange 1501-1549 Capture only Ethernet type EAPOL : ether proto 0x888e Reject ethernet frames towards the Link Layer Discovery Protocol Multicast group : not ether dst 01:80 : c2:00:00:0e Capture only IP traffic - the shortest filter, but sometimes very useful to get rid of lower layer protocols like ARP and STP : ip Capture only unicast traffic - useful to get rid of noise on the network if you only want to see traffic to and from your machine, not, for example, broadcast and multicast announcements : not broadcast and not multicast
Example: Host www.ynet.co.il
Wireshark provides a simple but powerful display filter language that allows you to build quite complex filter expressions. You can compare values in packets as well as combine expressions into more specific expressions. The following sections provide more information on doing this. There is a rich display filter options that you can use.
ip.addr == 172.16.100.111 and ip.addr == 172.16.100.12
In order to monitor all traffic to the server, we will simply define a filter with the IP address of the server
ip.addr == 192.168.101.253
This is a tree of all the protocols in the capture. You can collapse or expand subtrees, by clicking on the plus / minus icons. By default, all trees are expanded. Each row contains the statistical values of one protocol. The Display filter will show the current display filter. The following columns containing the statistical values are available: Protocol : this protocol's name % Packets : the percentage of protocol packets, relative to all packets in the capture Packets : the absolute number of packets of this protocol Bytes : the absolute number of bytes of this protocol MBit/s : the bandwidth of this protocol, relative to the capture time End Packets : the absolute number of packets of this protocol (where this protocol was the highest protocol to decode) End Bytes : the absolute number of bytes of this protocol (where this protocol was the highest protocol to decode) End MBit/s : the bandwidth of this protocol, relative to the capture time (where this protocol was the highest protocol to decode)
A network conversation is the traffic between two specific endpoints. For example, an IP conversation is all the traffic between two IP addresses. The conversations window is similar to the endpoint Window. Along with addresses, packet counters, and byte counters the conversation window adds four columns: the time in seconds between the start of the capture and the start of the conversation (&quot;Rel Start&quot;), the duration of the conversation in seconds, and the average bits (not bytes) per second in each direction.
User configurable graph of the captured network packets. You can define up to five differently colored graphs. The user can configure the following things: Graphs Graph 1-5 : enable the specific graph 1-5 (only graph 1 is enabled by default) Color : the color of the graph (cannot be changed) Filter : a display filter for this graph (only the packets that pass this filter will be taken into account for this graph) Style : the style of the graph (Line/Impulse/FBar/Dot) X Axis Tick interval : an interval in x direction lasts (10/1 minutes or 10/1/0.1/0.01/0.001 seconds) Pixels per tick : use 10/5/2/1 pixels per tick interval View as time of day : option to view x direction labels as time of day instead of seconds or minutes since beginning of capture Y Axis Unit : the unit for the y direction (Packets/Tick, Bytes/Tick, Bits/Tick, Advanced...) [XXX - describe the Advanced feature.] Scale : the scale for the y unit (Logarithmic,Auto,10,20,50,100,200,500,...)
You can save captured packets simply by using the Save As... menu item from the File menu under Wireshark. You can choose which packets to save and which file format to be used.
We can save the data into an Excel file, and then we can manipulate it as required.
If you are working with TCP based protocols it can be very helpful to see the data from a TCP stream in the way that the application layer sees it. Perhaps you are looking for passwords in a Telnet stream, or you are trying to make sense of a data stream. Maybe you just need a display filter to show only the packets of that TCP stream. If so, Wireshark's ability to follow a TCP stream will be useful to you. Simply select a TCP packet in the packet list of the stream/connection you are interested in and then select the Follow TCP Stream menu item from the Wireshark Tools menu (or use the context menu in the packet list). Wireshark will set an appropriate display filter and pop up a dialog box with all the data from the TCP stream laid out in order, as shown in the figure.
You can see that only a part of the original packets are presented.
A very useful mechanism available in Wireshark is packet colorization. You can set-up Wireshark so that it will colorize packets according to a filter. This allows you to emphasize the packets you are (usually) interested in. You will find a lot of Coloring Rule examples at the Wireshark Wiki Coloring Rules page at http://wiki.wireshark.org/ColoringRules
There are two types of coloring rules in Wireshark. Temporary ones that are only used until you quit the program. And permanent ones that will be saved to a preference file so that they are available on a next session. Temporary coloring rules can be added by selecting a packet and pressing the <ctrl> key together with one of the number keys. This will create a coloring rule based on the currently selected conversation. It will try to create a conversation filter based on TCP first, then UDP, then IP and at last Ethernet. Temporary filters can also be created by selecting the &quot;Colorize with Filter > Color X&quot; menu items when rightclicking in the packet-detail pane.
And you will get โฆ
In this case, we can clearly see and TLS connection established.
Whenever a TCP segment is transmitted, a copy of it is also placed on the retransmission queue. When the segment is placed on the queue, a retransmission timer is started for the segment, which starts from a particular value and counts down to zero. It is this timer that controls how long a segment can remain unacknowledged before the sender gives up, concludes that it is lost and sends it again. The length of time we use for retransmission timer is thus very important. If it is set too low, we might start retransmitting a segment that was actually received, because we didn't wait long enough for the acknowledgment of that segment to arrive. Conversely, if we set the timer too long, we waste time waiting for an acknowledgment that will never arrive, reducing overall performance. A TCP receiver should send an immediate duplicate ACK when an out-of-order segment arrives; this is to inform that a segment was received out-of-order and which sequence number is expected (caused by dropping, reordering or duplication in the network). In addition, a TCP receiver should send an immediate ACK when the incoming segment fills in all or part of a gap in the sequence space. This will generate more timely information for the sender recovery.
RTT Vs. Sequence numbers gives us the time that take to Ack every packet. In case of variations, it can cause DUPACKs and even Retransmissions Usually will happen on communications lines: Over the Internet Over cellular networks
Time / Sequence representes how sequence numbers advances with time In a good connection (like in the example), the line will be linear The angle of the line indicates the speed of the connection. In this example โ fast connection
In this case, we see a non-contiguous graph Can be due to: Severe packet loss Server response (processing) time
A stable throughput of around 1MB/8Mb per second It is important to test in parallel with SNMP tool for channel capacity
Something happened here (After ~5.25 Seconds). What can it be?
5.25 seconds after start of stream, we donโt see any connectivity problems โ probably slow server/applications
RTP, Short for R eal-Time T ransport P rotocol, an Internet protocol for transmitting real-time data such as audio and video. RTP itself does not guarantee real-time delivery of data, but it does provide mechanisms for the sending and receiving applications to support streaming data. Typically, RTP runs on top of the UDP protocol, although the specification is general enough to support other transport protocols.