Abandon Decades-old
TCPdump for Modern
Troubleshooting
Nathan McMahon, Avi Networks
TCPdump to the Rescue
7 6 5 4 3 2 1 0 0*2 + 0*2 + 0*2 + 1*2 + 0*2 + 0*2 + 1*2 + 0*2 = 18
Now we can't just use 'tcp[13] == 18' in the tcpdump filter expression, because that would select only those packets that have
SYN-ACK set, but not those with only SYN set. Remember that we don't care if ACK or any other control bit is set as long as
SYN is set.
In order to achieve our goal, we need to logically AND the binary value of octet 13 with some other value to preserve the
SYN bit. We know that we want SYN to be set in any case, so we'll logically AND the value in the 13th octet with the binary
value of a SYN:
00010010 SYN-ACK 00000010 SYN AND 00000010 (we want SYN) AND 00000010 (we want SYN) -------- -------- = 00000010 = 00000010
We see that this AND operation delivers the same result regardless whether ACK or another TCP control bit is set. The
decimal representation of the AND value as well as the result of this operation is 2 (binary 00000010), so we know that for
packets with SYN set the following relation must hold true:
( ( value of octet 13 ) AND ( 2 ) ) == ( 2 )
This points us to the tcpdump filter expression
tcpdump -i xl0 'tcp[13] & 2 == 2'
Some offsets and field values may be expressed as names rather than as numeric values. For example tcp[13] may be
replaced with tcp[tcpflags]. The following TCP flag field values are also available: tcp-fin, tcp-syn, tcp-rst, tcp-push, tcp-ack,
tcp-urg.
This can be demonstrated as:
tcpdump -i xl0 'tcp[tcpflags] & tcp-push != 0'
Note that you should use single quotes or a backslash in the expression to hide the AND ('&') special character from the
shell.
UDP Packets
UDP format is illustrated by this rwho packet:https://www.tcpdump.org/manpages/tcpdump.1.html
• Powerful
• Detailed
Triage
• You must know an issue exists in order to find it with TCPdump
• You must capture packets while the issue is occurring
• The issue must be in the network segment you are capturing
• You must have all the correct parameters, such as snaplen
App has incorrect
permissions set for
an HTTP imageTap Tap
Things That Have Been Around a While
Researchers Find 3,000 Year Old Cheese
https://www.mirror.co.uk/news/world-news/archaeologists-discover-worlds-oldest-cheese-13100880
Poll Question…
Intel 486 processor
DEC Alpha
Fax machine
TCPdump
10BASE-T Ethernet
Which came first?
1990
1992
1843
1988
1990
TCPdump: 1988 - 2018
TCPdump Is Old…
TCPdump may be old, but does that matter?
• Application proliferation as apps move from bare metal to virtual machine to containers
• Network teams are asked to do more with the same number of people
• Network analysis in public clouds and containers requires different tools
Technology Has Changed: HTTP/2
• Multiple streams (requests) are multiplexed over a single connection
• Headers are compressed with HPACK
• Most browsers require HTTP/2
use modern TLS encryption
• HTTP/2 has strict requirements for TLS cipher
suites, preferring connections over PFS
HTTP/2 connection to www.google.com
Technology Has Changed: SSL / TLS
• TCPdump and Wireshark can decrypt SSL with the private keys
• Modern TLS is moving to ephemeral key exchange (forward secrecy)
– The private key is rotated, often every day
– The client and server can still decrypt the connection and view clear text
– Man in the middle devices, such as NPMs, aren’t able to view traffic
Tap
NPM
Technology Has Changed for the Better
• Artificial Intelligence
• Visibility / Analytics
• Shift from data points to actionable information
Technology Has Changed for the Better: Artificial Intelligence
• Anomaly detection algorithms swiftly sift through data to provide intelligent insights
• Automation detects and even correct problems before end users feel or report them
Anomaly detection: A slow server is degrading end user experience of a virtual service
Technology Has Changed for the Better: Visibility / Analytics
Better to show than tell…
Modern Distributed Architecture
CONTROLLER
SERVICE
ENGINE
SEPARATE CONTROL
& DATA PLANE
ELASTICITY
BARE METAL VIRTUALIZED CONTAINERSON PREMISES PUBLIC CLOUDVIRTUALIZED CONTAINERS
Analytics Drives Intelligence
SERVICE
ENGINE
INTELLIGENCE
• Connection log analytics
• Security insights: DDoS
• App performance metrics
• End user experience
CONTROLLER
END TO END TIMING
TOTAL RESPONSE TIME
End User
Client
RTT
Server
RTT
App
Response
Load
Balancer
Server App
SEPARATE CONTROL
& DATA PLANE
ELASTICITY
Demo
TCPdump: Tool of Last Resort
• What if I still need to do TCPdump on Avi?
• Available within the Avi Controller UI
– Perform traffic capture on a virtual service
– Traffic capture is executed on all Service Engines hosting the VS
– A single PCAP file is created from traffic aggregated across all Service Engines
CONTROLLER
SERVICE
ENGINE
VS1 VS1 VS1
Network Analytics with TLS and Forward Secrecy
• What if I still need to do capture traffic elsewhere on the network?
• Avi Service Engines can mirror or clone traffic to NPM or network analytics tools
• Cloned traffic may be clear text or re-encrypted with non-PFS TLS for wire to wire encryption
• Traffic sent to public networks is encrypted with modern TLS encryption
NPM
Modernize Your World
• Troubleshoot faster
• Do more with less… even as network and apps grow
Happy birthday TCPdump…
but it’s time to let go of those red balloons
Thank You!
Nathan McMahon, Avi Networks
nathan@avinetworks.com
avinetworks.com
Next Webinar:
“Top 10 iRules That You Can Migrate
to a Modern Policy Platform”
Try out Avi:
Request a demo @ avinetworks.com
Learn more:
avinetworks.com/workshops

Abandon Decades-Old TCPdump for Modern Troubleshooting

  • 1.
    Abandon Decades-old TCPdump forModern Troubleshooting Nathan McMahon, Avi Networks
  • 2.
    TCPdump to theRescue 7 6 5 4 3 2 1 0 0*2 + 0*2 + 0*2 + 1*2 + 0*2 + 0*2 + 1*2 + 0*2 = 18 Now we can't just use 'tcp[13] == 18' in the tcpdump filter expression, because that would select only those packets that have SYN-ACK set, but not those with only SYN set. Remember that we don't care if ACK or any other control bit is set as long as SYN is set. In order to achieve our goal, we need to logically AND the binary value of octet 13 with some other value to preserve the SYN bit. We know that we want SYN to be set in any case, so we'll logically AND the value in the 13th octet with the binary value of a SYN: 00010010 SYN-ACK 00000010 SYN AND 00000010 (we want SYN) AND 00000010 (we want SYN) -------- -------- = 00000010 = 00000010 We see that this AND operation delivers the same result regardless whether ACK or another TCP control bit is set. The decimal representation of the AND value as well as the result of this operation is 2 (binary 00000010), so we know that for packets with SYN set the following relation must hold true: ( ( value of octet 13 ) AND ( 2 ) ) == ( 2 ) This points us to the tcpdump filter expression tcpdump -i xl0 'tcp[13] & 2 == 2' Some offsets and field values may be expressed as names rather than as numeric values. For example tcp[13] may be replaced with tcp[tcpflags]. The following TCP flag field values are also available: tcp-fin, tcp-syn, tcp-rst, tcp-push, tcp-ack, tcp-urg. This can be demonstrated as: tcpdump -i xl0 'tcp[tcpflags] & tcp-push != 0' Note that you should use single quotes or a backslash in the expression to hide the AND ('&') special character from the shell. UDP Packets UDP format is illustrated by this rwho packet:https://www.tcpdump.org/manpages/tcpdump.1.html • Powerful • Detailed
  • 3.
    Triage • You mustknow an issue exists in order to find it with TCPdump • You must capture packets while the issue is occurring • The issue must be in the network segment you are capturing • You must have all the correct parameters, such as snaplen App has incorrect permissions set for an HTTP imageTap Tap
  • 4.
    Things That HaveBeen Around a While Researchers Find 3,000 Year Old Cheese https://www.mirror.co.uk/news/world-news/archaeologists-discover-worlds-oldest-cheese-13100880
  • 5.
    Poll Question… Intel 486processor DEC Alpha Fax machine TCPdump 10BASE-T Ethernet Which came first? 1990 1992 1843 1988 1990
  • 6.
  • 7.
    TCPdump Is Old… TCPdumpmay be old, but does that matter? • Application proliferation as apps move from bare metal to virtual machine to containers • Network teams are asked to do more with the same number of people • Network analysis in public clouds and containers requires different tools
  • 8.
    Technology Has Changed:HTTP/2 • Multiple streams (requests) are multiplexed over a single connection • Headers are compressed with HPACK • Most browsers require HTTP/2 use modern TLS encryption • HTTP/2 has strict requirements for TLS cipher suites, preferring connections over PFS HTTP/2 connection to www.google.com
  • 9.
    Technology Has Changed:SSL / TLS • TCPdump and Wireshark can decrypt SSL with the private keys • Modern TLS is moving to ephemeral key exchange (forward secrecy) – The private key is rotated, often every day – The client and server can still decrypt the connection and view clear text – Man in the middle devices, such as NPMs, aren’t able to view traffic Tap NPM
  • 10.
    Technology Has Changedfor the Better • Artificial Intelligence • Visibility / Analytics • Shift from data points to actionable information
  • 11.
    Technology Has Changedfor the Better: Artificial Intelligence • Anomaly detection algorithms swiftly sift through data to provide intelligent insights • Automation detects and even correct problems before end users feel or report them Anomaly detection: A slow server is degrading end user experience of a virtual service
  • 12.
    Technology Has Changedfor the Better: Visibility / Analytics Better to show than tell…
  • 13.
  • 14.
    BARE METAL VIRTUALIZEDCONTAINERSON PREMISES PUBLIC CLOUDVIRTUALIZED CONTAINERS Analytics Drives Intelligence SERVICE ENGINE INTELLIGENCE • Connection log analytics • Security insights: DDoS • App performance metrics • End user experience CONTROLLER END TO END TIMING TOTAL RESPONSE TIME End User Client RTT Server RTT App Response Load Balancer Server App SEPARATE CONTROL & DATA PLANE ELASTICITY
  • 15.
  • 16.
    TCPdump: Tool ofLast Resort • What if I still need to do TCPdump on Avi? • Available within the Avi Controller UI – Perform traffic capture on a virtual service – Traffic capture is executed on all Service Engines hosting the VS – A single PCAP file is created from traffic aggregated across all Service Engines CONTROLLER SERVICE ENGINE VS1 VS1 VS1
  • 17.
    Network Analytics withTLS and Forward Secrecy • What if I still need to do capture traffic elsewhere on the network? • Avi Service Engines can mirror or clone traffic to NPM or network analytics tools • Cloned traffic may be clear text or re-encrypted with non-PFS TLS for wire to wire encryption • Traffic sent to public networks is encrypted with modern TLS encryption NPM
  • 18.
    Modernize Your World •Troubleshoot faster • Do more with less… even as network and apps grow Happy birthday TCPdump… but it’s time to let go of those red balloons
  • 19.
    Thank You! Nathan McMahon,Avi Networks nathan@avinetworks.com avinetworks.com Next Webinar: “Top 10 iRules That You Can Migrate to a Modern Policy Platform” Try out Avi: Request a demo @ avinetworks.com Learn more: avinetworks.com/workshops

Editor's Notes

  • #14 Next, I am going to talk about how Avi delivers these fundamental values. There are 5 key building blocks. Software defined principle – our architecture separates the control plane and data plane. It allows centralized management, policies and orchestration across distributed data plane referred to as SEs. Elasticity: SEs can scale out and in automatically – seamless scaling based on workloads. Automation is really the invisible secret sauce that makes this possible. And to make full-stack automation, it’s key to integrate into a rich set of ecosystem with 100% clean REST APIs.
  • #15 4. Intelligence: LB is at a prime location – between users and mission critical applications. Metrics about networks and information about clients and applications are all precious data. Ultimately, what’s driving all these is based on Avi’s built in analytics. It helps your teams work smarter, your infrastructure to react to changing demands faster and enable you to make wiser decisions. App performance metrics End-to-end timing Connection log analytics Security information such SSL certifications and security policies etc.