tcpdump overview

1. F5 TCPDUMP
BY: ALEX WADE

2. BASICS

3. IN-LINE LOAD BALANCER
With the in-line method the servers are behind the F5 and the F5 becomes the default gateway for the servers.
This method preserves the source IP which is one of the best methods for non-HTTP applications and will also ease
troubleshooting.

4. WHAT IS TCPDUMP
TCPDUMP is a NETWORK PACKET ANALYZER. tcpdump uses libpacp/winpcap to
capture data and uses it extensive protocol definitions build inside to analyze
the captured packets. Its mainly used to debug the protocol of the captured
packet which in turn reveals the network traffic characteristics.

5. CAPTURE FULL TRAFFIC FLOW
Capturing traffic with TMM information for a specific traffic flow
Beginning in BIG-IP 11.2.0, you can use the 'p' interface modifier with the 'n' modifier to
capture traffic with TMM information for a specific flow, and its related peer flow. The p
modifier allows you to capture a specific traffic flow through the BIG-IP system from end to
end, even when the configuration uses a Secure Network Address Translation (SNAT) or
OneConnect. For example, the following command searches for traffic to or from client
10.0.0.1 on interface 0.0:
tcpdump -ni 0.0:nnnp -s0 -c 100000 -w /var/tmp/capture.dmp host 10.0.0.1
Once tcpdump identifies a related flow, the flow is marked in TMM, and every subsequent packet
in the flow (on both sides of the BIG-IP system) is written to the capture file.

6. TCPDUMP WITH SNAT -P FLAG TO THE RESCUE
tcpdump -ni 0.0:nnnp -s 0 host client-ip -w /var/tmp/traffic_from_client.pcap
Ex.
tcpdump -ni 0.0:nnnp -s 0 host 10.100.100.102 -w /var/tmp/traffic_from_client.pcap
Note: with the “-p” flag, you can narrow down by all traffic to that VIP as well if you put
tcpdump -ni 0.0:nnnp -s 0 host <vip-ip> and port <vip-port> -w /var/tmp/traffic_to_vip.pcap
Ex.
tcpdump -ni 0.0:nnnp -s 0 host 1.1.1.1 and port 443 -w /var/tmp/traffic_to_vip.pcap
you can always filter on either the client or server IP. so if you want to tcpdump traffic "behind" the F5 and
can't use the client IP due to SNAT you can filter on the Node IP address(es).

7. SSL DUMP
SSL handshake and other SSL record messages
SSL connections are established on top of an existing TCP connection using an SSL handshake that accomplishes the following:
• The client and server negotiate security capabilities, such as the public-key algorithm, the symmetric key algorithm, and compression
algorithms.
• The server transmits its certificate to the client, allowing the client to validate the identity of the server.
• The client and server exchange session key information.
• The client may also send its certificate to the server, allowing the server to validate the identity of the client.
The handshake transactions consist of a number of SSL record messages. These messages can be examined by executing the ssldump
utility using the -r option to specify the path and name of the tcpdump capture file to be examined. Other useful options include the
following:
• -n Do not resolve host names.
• -A Print all fields (ssldump, by default, prints only the most interesting).
• -e Print absolute timestamps.
• -d Display application data, including traffic before session initiates.
• -M Output a pre-master secret log file (v. 11.2.0 and later)

8. THE FOLLOWING COMMAND DISPLAYS ALL OF THE SSL RECORD MESSAGES FOUND IN THE TCPDUMP CAPTURE FILE
NAMED WWW-SSL-CLIENT.CAP:
SSLDUMP -NR /VAR/TMP/WWW-SSL-CLIENT.CAP
THE SSL RECORDS PRINTED BY THE SSLDUMP UTILITY APPEAR SIMILAR TO THE FOLLOWING EXAMPLE:
New TCP connection #2: 172.16.31.22(32866) <->
192.168.1.8(8389)
2 1 0.0002 (0.0002) C>S Handshake
ClientHello
Version 3.0
resume [32]=
a3 ca ad 46 95 5d 64 bb 33 ec b5 12 91 21 a3 50
d2 c0 c5 f6 67 c3 cc 9e c0 4a 71 1b 92 dc 58 55
cipher suites
SSL_DHE_RSA_WITH_AES_256_CBC_SHA
SSL_DHE_DSS_WITH_AES_256_CBC_SHA
SSL_RSA_WITH_AES_256_CBC_SHA
SSL_DHE_RSA_WITH_AES_128_CBC_SHA
SSL_DHE_DSS_WITH_AES_128_CBC_SHA
************************************
ServerHello
Version 3.0
session_id[32]=
a3 ca ad 46 95 5d 64 bb 33 ec b5 12 91 21 a3 50
d2 c0 c5 f6 67 c3 cc 9e c0 4a 71 1b 92 dc 58 55
cipherSuite SSL_RSA_WITH_3DES_EDE_CBC_SHA
compressionMethod NULL
2 3 0.0277 (0.0000) S>C ChangeCipherSpec
2 4 0.0277 (0.0000) S>C Handshake
2 5 0.0282 (0.0005) C>S ChangeCipherSpec
2 6 0.0282 (0.0000) C>S Handshake
2 7 0.0282 (0.0000) C>S application_data
2 8 0.0289 (0.0006) S>C application_data
2 9 0.0289 (0.0000) S>C application_data
2 10 0.0292 (0.0003) C>S application_data
2 11 0.0296 (0.0003) S>C application_data
2 12 0.0296 (0.0000) S>C application_data

9. TYPICAL TCPDUMP SCENARIO
A typical scenario would be to run simultaneous dumps on the internal and external VLANs (assuming the
LTM is in a classic Layer 3 implementation), filtering on client IP or whatever else you are searching for.
If packets appear on the external VLAN but not the internal, then there's likely a problem with the LTM
configuration (particularly if you are seeing RST packets). If you see SYN packets on both the external
and internal VLANs, but no ACK packets returning, then it is likely that the server either does not have
its' gateway set to the BIG-IP, or it has another route (possibly via a 2nd NIC) to whatever the source
network of the client is.
Those are just a couple of examples of how tcpdump can help quickly resolve issues. Do be aware that
the PVA (ASIC) can be handling some L4 traffic in hardware that won't show up in a regular tcpdump on
an LTM VLAN. If need be you can dump on the 0.0 interface to see all traffic going through an LTM, but
be careful doing that on a busy box if you don't appropriately filter the tcpdump by host

10. VIEW ENCRYPTED TRAFFIC INSIDE WIRESHARK
• On Linux systems WireShark must be compiled against Gnu-TLS and GCrypt, not OpenSSL or some other
encryption suite; not something to worry about on Windows systems.
• The private key used to encrypt the data must be available on the system running Wireshark.
• The private key file must be in the PEM or PKCS12 format; if it’s not you can use OpenSSL to convert what you
have as appropriate, just Google it.
• The private key file should only contain the private key, not the public key (aka the certificate). Files frequently
contain both, check by viewing the file in a true text editor. You only need the text delimited by this;
• Header:
• —–BEGIN RSA PRIVATE KEY—–
• Footer:
• —–END RSA PRIVATE KEY—–
• The capture must include both ‘sides’ of a conversation. In other words, the capture must include the full
client and server exchange.

11. Configuring Wireshark to Decrypt Data
In Wireshark click Edit>Preferences…
Select and expand Protocols, scroll down (or just type ssl) and select SSL
Click the RSA Keys List Edit… button, click New and then enter the following information;
IP Address is the IP address of the host that holds the private key used to decrypt the data and serves the certificate (i.e. the
decrypting host, the server)
Port is the destination port used to communicate with the host that holds the private key used to decrypt the data and serves
the certificate (i.e. the decrypting host, the server)
Protocol is the upper-layer protocol encrypted by SSL/TLS, for instance, the protocol encrypted over a HTTPS web connection is
HTTP
Key File – select as necessary
Password is the passphrase used to protect the private key file, if any
I believe a wildcard IP address of 0.0.0.0 and wildcard port of 0 or data can be used.
Optionally, enter the path and file name of a debug file that you might find useful in helping you diagnose any issues with the
decryption. Note this may slow down the initial load of the capture file.
Then simply open the capture and, if you’ve met all the requirements, you should find the application data has been
unencrypted. If the standard SSL/TLS port isn’t being used you may need to select a relevant packet and then click Analyse >
Decode As… and then select SSL.

12. Important:
The capture must include the initial SSL/TLS session establishment. In other words, the CLIENTHELLO and
SERVERHELLO exchange. Beware captures taken where a session has been resumed. Ideally, ensure
any capture either a) is of packets related to an entirely new device connecting or b) where a device
that has already previously established a session is used, it is used after a considerable time after the
last session was established.

13. Things to Come:
performance monitoring with SolarWinds.
Overview: Configuring network monitoring with sFlow
sFlow is an industry-standard technology for monitoring high-speed
switched networks. You can configure the BIG-IP® system to poll
internal data sources and send data samples to an sFlow receiver.
You can then use the collected data to analyze the traffic that
traverses the BIG-IP system. This analysis can help you understand
traffic patterns and system usage for capacity planning and
chargeback, troubleshooting network and application issues, and
evaluate the effectiveness of your security policies.

14. • F5 Wireshark Plugin (Requires login) [Download plugin,
Instructions, additional information]
• https://devcentral.f5.com/articles/getting-started-with-the-f5-
wireshark-plugin-on-windows
• https://support.f5.com/csp/article/K10209
• https://support.f5.com/csp/article/K13637
• https://thecciejourney.wordpress.com/2014/11/23/wireshark-f5-
plugins-make-for-easier-troubleshooting/
ADDITIONAL LINKS