Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
IPv4 Network debugging
An introduction
Jacco van Buuren <jacco@bjvb.nl>
http://bjvb.nl/
Intro
• Who am I
• Who are you?
What you should know
• Basic IP connectivity concepts:
▫ Know what an IP-address and netmask is
(IPv4)
What you will learn
• Common layer-2 network protocols and their
daily use
• Basic IPv4 routing and problem identification...
Contents
• Introduction
• What you should know
• What you will learn
• When connectivity fails…
• Before you begin
• The t...
When connectivity fails…
• First: What would you do?
When connectivity
fails…
• Oh really?
• Listen…
• …Listen carefully…
• …LISTEN…
• ((Gracefully take any insults, it’s-just...
Before you begin
• Can I test/simulate this SOMEWHERE ELSE
• Baseline performance figures! (normal behavior)
• Zero-load p...
The tools
• KNOW THY TOOLS… THOROUGHLY
• Learn tools in test environment.
• Do that again…
• …and again…
• …and again…
• R...
The tools
• BSD/Linux
• arp / ethtool / ping / etc.
• Tcpdump
• Wireshark
• Nmap(!)
• Hping[23] / traceroute
• Dig / Drill...
About the tests
• We’ll be following the OSI layers:
1. People
2. Do
3. Need
4. To
5. See
6. Pamala
7. Anderson
bits
frame...
…In theory – as short as possible
• Hardware, NIC, MAC *
• VLAN, ports & tags *
• Spanning tree *
• TCP/IP, IPv4 address s...
Ethernet frame
https://en.wikipedia.org/wiki/Ethernet_frame
IPv4 + TCP header
Source: Wikipedia
The tests – But first…
1. Identify “problem-chain”
(if more than one, pick any, all if possible)
 Documentation…?... Or…
...
Network host discovery
• Going boldly where no packet
has gone before…
▫ (ze)nmap!
▫ Zmap?
▫ Masscan??
▫ Milder: Zabbix ho...
The tests
2. Check settings at both ends, and – if possible
3. EVERYWHERE IN BETWEEN
(( Check interfaces autosense/autoneg...
The tests
4. Check the ARP cache
root@io:~# arp -an
? (192.168.223.1) at 08:00:27:60:05:2a [ether] on eth1
? (10.0.2.2) at...
The tests
5. Check ICMP Echo/Echo reply a.k.a. “PING”
- Local interface
- Local network
- Ping broadcast address
- Default...
The tests
6. Check “distances” with variable Time-To-Live
(TTL) packets
(ping)
The tests
7. Check fragments with variable MTU sizes at
distant networks.
- Set “Don’t fragment” bit…
(ping)
The tests
8. Check “port/host unreachable” with UDP ports
at distant network.
((h)ping)
The tests
• Check nameresolving for relevant hostnames
C:WINDOWSsystem32>nslookup.exe
Default Server:
nlhag999a21ads.ww002...
The tests
• Check timesynchronisation (NTP)
root@io:~# ntpdc -np
remote local st poll reach delay offset disp
============...
Where are we?
We’ve just tested
1. Network settings (layer 1) and…
2. Parameters (layer 2), including…
3. Basic connectivi...
More tests
INTRUSIVE TESTS, THIS WILL NOT GO
UNNOTICED! PROCEED WITH
CAUTION!!
The tests
• Check available bandwidth and latency
▫ Check on high QoS ports (SIP: 5060/5061 tcp)
•  iperf
•  ftp(!)
The results
Latency:
• Localhost
▫ <1 ms latency
• Localnet
▫ <10 ms latency
• Distant net
▫ …yeah… fuzzy…
• Bandwidth sho...
…And now what?
• …Move along now, nothing to see here(?)
…or is it…?
Network measurements
• Port monitor at network edge (tcpdump)
• Port monitor at server farm (tcpdump)
• Routers&switches: ...
Post processing
WARNING! CODING SKILLS REQUIRED!
PROCEED WITH CAUTION!
• What am I looking for?
▫ Spikes = High Bandwidth ...
Questions?
Further reading
• Perl / Python for log/text parsing
• Unix command line skills (grep/ngrep)
• SNMP Monitoring:
MRTG / Cac...
The end
• Jacco van Buuren CISSP <jacco@bjvb.nl>
• http://bjvb.nl/
Upcoming SlideShare
Loading in …5
×

netw-debug

130 views

Published on

  • Be the first to comment

  • Be the first to like this

netw-debug

  1. 1. IPv4 Network debugging An introduction Jacco van Buuren <jacco@bjvb.nl> http://bjvb.nl/
  2. 2. Intro • Who am I • Who are you?
  3. 3. What you should know • Basic IP connectivity concepts: ▫ Know what an IP-address and netmask is (IPv4)
  4. 4. What you will learn • Common layer-2 network protocols and their daily use • Basic IPv4 routing and problem identification, meaning… ▫ You don’t have to solve problems you encounter, just be able to pin-point them. • Common IP services, like nameresolving and timesynchronisation • Opensource network tools and some basic Unix hacking skills
  5. 5. Contents • Introduction • What you should know • What you will learn • When connectivity fails… • Before you begin • The tools • Network plan/map • The tests • The results • Network measurements • Post processing • Questions Soft-skills “Direct” tests Difficult cases Procesintime
  6. 6. When connectivity fails… • First: What would you do?
  7. 7. When connectivity fails… • Oh really? • Listen… • …Listen carefully… • …LISTEN… • ((Gracefully take any insults, it’s-just-work-you-know)) • …repeat
  8. 8. Before you begin • Can I test/simulate this SOMEWHERE ELSE • Baseline performance figures! (normal behavior) • Zero-load performance figures! (single user performance) • Peak hours? Spikes? Notorious: Batch processing/backups • Who is involved in this? ▫ Users? ▫ Managers? ▫ 3rd parties? • What do “they” expect from me? ▫ Follow procedures? (impact=?) ▫ Document(s)? • Begin with the end in mind ▫ Setup test-tree (“if-this-works” then “test-that”)
  9. 9. The tools • KNOW THY TOOLS… THOROUGHLY • Learn tools in test environment. • Do that again… • …and again… • …and again… • Repeat
  10. 10. The tools • BSD/Linux • arp / ethtool / ping / etc. • Tcpdump • Wireshark • Nmap(!) • Hping[23] / traceroute • Dig / Drill / Nslookup • Ntpdate / ntpdc / ntpq • Iperf • Syslog • (SNMP)
  11. 11. About the tests • We’ll be following the OSI layers: 1. People 2. Do 3. Need 4. To 5. See 6. Pamala 7. Anderson bits frames packets 1. Princess 2. Diana 3. Never 4. Tried 5. Shagging 6. Prince 7. Andrew 1. Port 2. Drinking 3. Now 4. Together 5. Standing 6. People 7. All 1. Processing 2. Data 3. Need 4. To 5. Seem 6. People 7. All
  12. 12. …In theory – as short as possible • Hardware, NIC, MAC * • VLAN, ports & tags * • Spanning tree * • TCP/IP, IPv4 address space, netmask calculations • ARP, ICMP • UDP, TCP three-way handshake • TLS/SSL, PKI • NTP, DNS * https://en.wikipedia.org/wiki/User:Jaccovanbuuren/Books/Layer_2
  13. 13. Ethernet frame https://en.wikipedia.org/wiki/Ethernet_frame
  14. 14. IPv4 + TCP header Source: Wikipedia
  15. 15. The tests – But first… 1. Identify “problem-chain” (if more than one, pick any, all if possible)  Documentation…?... Or…  …Document-It-Yourself (DIY)  BUILD A MAP!
  16. 16. Network host discovery • Going boldly where no packet has gone before… ▫ (ze)nmap! ▫ Zmap? ▫ Masscan?? ▫ Milder: Zabbix host discovery • … but rarely done as part of troubleshooting Just because you can, doesn’t mean that you should
  17. 17. The tests 2. Check settings at both ends, and – if possible 3. EVERYWHERE IN BETWEEN (( Check interfaces autosense/autonegotiate, line speed and duplex settings )) ((( Layer 2 intermezzo: MAC, CDP/LLDP, STP! )))
  18. 18. The tests 4. Check the ARP cache root@io:~# arp -an ? (192.168.223.1) at 08:00:27:60:05:2a [ether] on eth1 ? (10.0.2.2) at 52:54:00:12:35:02 [ether] on eth0
  19. 19. The tests 5. Check ICMP Echo/Echo reply a.k.a. “PING” - Local interface - Local network - Ping broadcast address - Default gateway - Host on other network
  20. 20. The tests 6. Check “distances” with variable Time-To-Live (TTL) packets (ping)
  21. 21. The tests 7. Check fragments with variable MTU sizes at distant networks. - Set “Don’t fragment” bit… (ping)
  22. 22. The tests 8. Check “port/host unreachable” with UDP ports at distant network. ((h)ping)
  23. 23. The tests • Check nameresolving for relevant hostnames C:WINDOWSsystem32>nslookup.exe Default Server: nlhag999a21ads.ww002.siemens.net Address: 139.10.220.20 > set type=txt > set class=chaos > version.bind Server: nlhag999a21ads.ww002.siemens.net Address: 139.10.220.20 version.bind.ww002.siemens.net text = "Microsoft DNS 6.1.7601 (1DB1557D)" > exit
  24. 24. The tests • Check timesynchronisation (NTP) root@io:~# ntpdc -np remote local st poll reach delay offset disp ======================================================================= =91.208.160.226 10.0.2.15 16 64 0 0.00000 0.000000 3.99217 =91.189.89.199 10.0.2.15 16 64 0 0.00000 0.000000 3.99217 =129.70.132.34 10.0.2.15 16 64 0 0.00000 0.000000 3.99217 =85.214.111.180 10.0.2.15 16 64 0 0.00000 0.000000 3.99217 =178.63.14.131 10.0.2.15 16 64 0 0.00000 0.000000 3.99217
  25. 25. Where are we? We’ve just tested 1. Network settings (layer 1) and… 2. Parameters (layer 2), including… 3. Basic connectivity with ICMP and ICMP error- codes (layer 3). …And at layer 7 1. Nameresolving (DNS 53/udp) 2. Timesynchronisation (NTP 123/udp)
  26. 26. More tests INTRUSIVE TESTS, THIS WILL NOT GO UNNOTICED! PROCEED WITH CAUTION!!
  27. 27. The tests • Check available bandwidth and latency ▫ Check on high QoS ports (SIP: 5060/5061 tcp) •  iperf •  ftp(!)
  28. 28. The results Latency: • Localhost ▫ <1 ms latency • Localnet ▫ <10 ms latency • Distant net ▫ …yeah… fuzzy… • Bandwidth should be within 10% Must o Should Could o Would
  29. 29. …And now what? • …Move along now, nothing to see here(?) …or is it…?
  30. 30. Network measurements • Port monitor at network edge (tcpdump) • Port monitor at server farm (tcpdump) • Routers&switches: SNMP graphics! • Server farm: show me your SYSLOG
  31. 31. Post processing WARNING! CODING SKILLS REQUIRED! PROCEED WITH CAUTION! • What am I looking for? ▫ Spikes = High Bandwidth usage ▫ Peak hours = Concurrent usage ▫ Hick-ups = Re-occuring events
  32. 32. Questions?
  33. 33. Further reading • Perl / Python for log/text parsing • Unix command line skills (grep/ngrep) • SNMP Monitoring: MRTG / Cacti / Nagios / Zabbix …Google is your friend…
  34. 34. The end • Jacco van Buuren CISSP <jacco@bjvb.nl> • http://bjvb.nl/

×