The presentation addresses the most typical issues during network software development and testing, explains the causes and suggests solutions: - overlapping IP networks - invalid netmasks - incomplete routing configuration - incorrect local MAC addresses - unidirectional packet generator and unicast flood - disabled ethernet auto negotiation

1. Andriy Berestovskyy
2017
( ц ) А н д р
і й Б е р е с
т о в с ь к и
й
networking hour
TCP
UDP
NAT
IPsec
IPv4
IPv6
internet
protocolsAH
ESP
authentication
authorization
accounting
encapsulation
security
BGP
OSPF
ICMP
ACLSNAT
tunnel
PPPoE
GRE
ARP
discovery
NDP
OSI
broadcast
multicast
IGMP
PIM
MAC
DHCP
DNS
fragmentation
semihalf
berestovskyy
Why my network does not work?
Networking Quiz 2017

2. LAB
Quiz 1. Network Device Test
2
Lab Switch
Packet
Generator
DUT
LAN
Direct Link
How?

3. LAB
Network Device Test: Quiz!
3
Lab Switch
LAN
Direct Link
# ifconfig eth0 10.0.0.2/24
# ping 10.0.0.1
# ifconfig eth1 10.0.0.1/24
# ping 10.0.0.2
What’s
wrong?

4. LAB
Network Device Test: Symptoms
4
Lab Switch
LAN
Direct Link
# ifconfig eth0 10.0.0.2/24
# ping 10.0.0.1
# ifconfig eth1 10.0.0.1/24
# ping 10.0.0.2
What’s
wrong?
ssh: connection
timeout

5. LAB: 10.2/16
Network Device Test: Analysis
5
Lab Switch
LAN: 10.0/16
Direct Link: 10.0.0/24
# ifconfig eth0 10.0.0.2/24
# ping 10.0.0.1
# ifconfig eth1 10.0.0.1/24
# ping 10.0.0.2
T
G
I am
10.0.0.1!
I am
10.0.0.1!

6. Root cause — overlapping IP networks
6

7. Recap: IPv4 Private Address Space
7
10.0.0.0/8
172.16.0.0/12
192.168.0.0/16
Meaning?

8. Recap: IPv4 Subnet Mask
8
32-bit Number
0110 0100 0000 0011 0000 0010 0000 0101
100. 3. 2. 5 IPv4 Address in Dotted Decimal Notation
1111 1111 1111 1111 1111 1111 0000 0000
255. 255. 255. 0 Subnet Mask in Dotted Decimal Notation
&
0110 0100 0000 0011 0000 0010 0000 0000
100. 3. 2. 0 Subnet in Dotted Decimal Notation

9. Recap: IPv4 Subnet Mask Length
9
Subnet Mask Length = 24
1111 1111 1111 1111 1111 1111 0000 0000
255. 255. 255. 0 Subnet Mask in Dotted Decimal Notation
Dotted Decimal Notation:
IPv4 Address: 100.3.2.5
Subnet Mask: 255.255.255.0
CIDR* Notation:
IPv4 Prefix: 100.3.2.5/24
==
* Classless Inter-Domain Routing

10. Recap: IPv4 Private Address Space
10
10.0.0.0/8 = 10.0.0.0/24
10.0.1.0/24
[...]
10.255.255.0/24
172.16.0.0/12 = 172.16.0.0/24
172.16.1.0/24
[...]
172.31.255.0/24
192.168.0.0/16 = 192.168.0.0/24
192.168.1.0/24
[...]
192.168.255.0/24
224-8
= 216
= 65536 networks
224-12
= 212
= 4096 networks
224-16
= 28
= 256 networks

11. Network Device Test: Solutions
1. Immediate solution: disconnect direct link
2. Use 172.16.0.0/12 for network tests
11
172.16.0.0/12 = 172.16.0.0/24
172.16.1.0/24
[...]
172.31.255.0/24

12. Direct Link
LAB
Quiz 2. Network System Test
12
Lab Switch
Packet
Generator
DUT
LAN
Direct Link
DUT
How?
Packet
Generator

13. IP Subnetting (Example)
1. Numerate hosts: 1, 2, 3, 4
2. Numerate subnets: <prefix>.12 (connects host 1 and 2), ...
13
Host 1 Host 2 Host 3 Host 4
172.16.12 172.16.23 172.16.34
Host 1 Host 2 Host 3 Host 4

14. LAB
Network System Test: Quiz!
14
LAN
What’s
wrong?*
172.16.12 172.16.23 172.16.34
# ifconfig eth1 172.16.12.1
… eth1
172.16.12.2
… eth0 172.16.23.2 # ifconfig eth1 172.16.34.4
Host 1 Host 2 Host 3 Host 4
… eth0 172.16.23.3
… eth1
172.16.34.3
eth1 to eth1

15. LAB
Network System Test: Symptoms
15
LAN
What’s
wrong?*
172.16.12 172.16.23 172.16.34
# ifconfig eth1 172.16.12.1
… eth1
172.16.12.2
… eth0 172.16.23.2 # ifconfig eth1 172.16.34.4
Host 1 Host 2 Host 3 Host 4
… eth0 172.16.23.3
… eth1
172.16.34.3
ping 172.16.34.4
Host Unreachable

16. Network System Test: Cause 1
Default netmask:
ifconfig eth0 172.16.23.2 == ifconfig eth0 172.16.23.2/16
ifconfig eth1 172.16.12.2 == ifconfig eth1 172.16.12.2/16
eth0 subnet 172.16/16 == eth1 subnet 172.16/16
16
Why?

17. Classful IP Networks (Legacy)
17
Why?
A
Class Networks
27
0xxx xxxx
Leading Bits
C 221
110x xxxx
D —1110 xxxx
Hosts Start
232-8
232-24
—
0.0.0.0
192.0.0.0
224.0.0.0
B 214
10xx xxxx 232-16
128.0.0.0
Implicit
Masklen
/8
/24
—
/16
E —1111 xxxx — 240.0.0.0—
Source: https://en.wikipedia.org/wiki/Classful_network
End
127.255.255.255
223.255.255.255
239.255.255.255
191.255.255.255
255.255.255.255

18. LAB
System Test: Cause 1 Analysis
18
LAN
172.16.12 172.16.23 172.16.34
# ifconfig eth1 172.16.12.1
… eth1
172.16.12.2
… eth0 172.16.23.2 # ifconfig eth1 172.16.34.4
Host 1 Host 2 Host 3 Host 4
… eth0 172.16.23.3
… eth1
172.16.34.3
Who has
172.16.34.4?
Solutions?
Not
me!

19. System Test: Cause 1 Solution
Fix the subnet mask:
# ifconfig eth0 172.16.23.2/24
# ifconfig eth1 172.16.12.2/24
eth0 subnet 172.16.23/24 != eth1 subnet 172.16.12/24
19

20. LAB
Network System Test: Symptoms
20
LAN
What’s
wrong?*
172.16.12/24 172.16.23/24 172.16.34/24
# ifconfig eth1
172.16.12.1/24
… eth1
172.16.12.2/24
… eth0
172.16.23.2/24
# ifconfig eth1
172.16.34.4/24
Host 1 Host 2 Host 3 Host 4
… eth0
172.16.23.3/24
… eth1
172.16.34.3/24
ping 172.16.34.4
Packet Filtered

21. Network System Test: Cause 2
Host 1 Routing Table:
dhclient eth0 => 10.2/16 dev eth0
=> default via 10.2.0.254
ifconfig eth1 172.16.12.1/24 => 172.16.12/24 dev eth1
How to reach 172.16.23/24?
How to reach 172.16.34/24?
21
How?

22. LAB
System Test: Cause 2 Analysis
22
LAN Solutions?
172.16.12/24 172.16.23/24 172.16.34/24
# ifconfig eth1
172.16.12.1/24
… eth1
172.16.12.2/24
… eth0
172.16.23.2/24
# ifconfig eth1
172.16.34.4/24
Host 1 Host 2 Host 3 Host 4
… eth0
172.16.23.3/24
… eth1
172.16.34.3/24
Default
route
Filter the
packet!

23. System Test: Cause 2 Solution
Fix Host 1 Routing Table (Add Static Routing):
ifconfig eth1 172.16.12.1/24 => 172.16.12/24 dev eth1
ip route add 172.16.23/24 via 172.16.12.2 => 172.16.23/24 via 172.16.12.2
ip route add 172.16.34/24 via 172.16.12.2 => 172.16.34/24 via 172.16.12.2
23
172.16.12/24 172.16.23/24 172.16.34/24
172.16.23/24 via 172.16.12.2
172.16.34/24 via 172.16.12.2
… eth1
172.16.12.2/24
# ifconfig eth1
172.16.34.4/24
Host 1 Host 2 Host 3 Host 4

24. LAB
Network System Test: Symptoms
24
LAN
What’s
wrong?*
172.16.12/24 172.16.23/24 172.16.34/24
ifconfig + ip route
… eth1
172.16.12.2/24
# ifconfig eth1
172.16.34.4/24
Host 1 Host 2 Host 3 Host 4ping 172.16.34.4
Network Unreachable
… eth0
172.16.23.2/24

25. Network System Test: Cause 3
Host 2 Routing Table:
ifconfig eth0 127.16.23.2/24 => 172.16.23/16 dev eth0
ifconfig eth1 172.16.12.2/24 => 172.16.12/24 dev eth1
How to reach 10.2/16?
How to reach 172.16.34/24?
25
How?

26. LAB
System Test: Cause 3 Analysis
26
LAN
172.16.12/24 172.16.23/24 172.16.34/24
# ifconfig eth1
172.16.12.1/24
… eth1
172.16.12.2/24
… eth0
172.16.23.2/24
# ifconfig eth1
172.16.34.4/24
Host 1 Host 2 Host 3 Host 4
… eth0
172.16.23.3/24
… eth1
172.16.34.3/24
No route to
172.16.34/24
ping 172.16.34.4
Solutions?

27. System Test: Cause 3 Solution
Fix All Routing Tables!
27
172.16.12/24 172.16.23/24 172.16.34/24
172.16.23/24 via 172.16.12.2
172.16.34/24 via 172.16.12.2
Host 1 Host 2 Host 3 Host 4
172.16.12/24 via 172.16.34.3
172.16.23/24 via 172.16.34.3
172.16.12/24 via 172.16.23.2
default via 172.16.34.4
172.16.34/24 via 172.16.23.3
default via 172.16.12.1
4
1
1
4
Think both
directions!

28. Quiz 3. Local MAC Address
Why local MAC?
● Virtual Machine
● TUN/TAP
● NIC Virtual Function
28
More?

29. Local MAC Address: How?
29
1. ifconfig DEV hw ether MAC
2. ip link set dev DEV address MAC
3. /etc/network/interfaces (Ubuntu):
auto DEV
iface DEV inet dhcp
hwaddress ether MAC More?

30. Local MAC Address: Quiz!
30
Lab
Switch
Packet
Generator
DUT
# ifconfig eth0 172.16.0.1/24
# ping 172.16.0.2 # OK!
# dpdkgen 172.16.0.2@01:02:03:04:05:06
# ifconfig eth1 hw ether 01:02:03:04:05:06
# ifconfig eth1 172.16.0.2/24
What’s
wrong?
Ping is OK!

31. Local MAC Address: Symptoms
31
Lab
Switch
Packet
Generator
DUT
# ifconfig eth0 172.16.0.1/24
# dpdkgen 172.16.0.2@01:02:03:04:05:06
# ifconfig eth1 hw ether 01:02:03:04:05:06
# ifconfig eth1 172.16.0.2/24
What’s
wrong?LAN
@#*&!!
^*%$+!1!

32. 1. Receive frame, check Ethernet FCS
2. Add/update source MAC in MAC table
3. If multicast bit is set:
a. forward to all ports, but the source
4. If destination is in MAC table:
a. forward to the specific port
5. Else, forward to all ports
Recap: Per-VLAN Switch Logic
32
Any issues?

33. Recap: Ethernet MAC Address
33* OUI — Organizationally Unique Identifier
NIC IDOUI*2
24 bits 24 bits
b7
0: globally unique
1: locally managed
b6
0: unicast
1: multicast
b5 b4 b3 b2 b1 b0
MAC, 6 octets
First octets

34. Local MAC Address: Root Cause
34
0 0
1: multicast
0201
0 0 0 0 0 1
Local MAC03 04 05 06
First octets

35. Local MAC Address: Analysis
35
Lab
Switch
Packet
Generator
DUT
# ifconfig eth0 172.16.0.1/24
# dpdkgen 172.16.0.2@01:02:03:04:05:06
# ifconfig eth1 hw ether 01:02:03:04:05:06
# ifconfig eth1 172.16.0.2/24
LAN
M
G
M
GForwards to
all ports
Solutions?
M
G

36. Local MAC Address: Solutions
1. Immediate solution: disconnect traffic generator
2. Use local MAC starting with 02:
02:03:04:05:06:07
3. Connect DUT directly to packet generator
4. Use VLANs:
switch(config)# interface gigabitethernet 0/5 - 7
switch(config-if)# switchport access vlan VLAN
36
More?
Source: CCNP SWITCH, page 97

37. Local MAC Address: Admin Solution
37
Lab
Switch
Packet
Generator
DUT
LAN
M
G
lab-switch# configure terminal
lab-switch(config)# interface gigabitethernet 0/5
lab-switch(config-if)# description --- Packet Generator
lab-switch(config-if)# storm-control multicast level 1
Storm
control
Source: CCNP SWITCH, page 421
Storm
control

38. Quiz 4. Packet Generator
38
Lab
Switch
Packet
Generator
DUT
# ifconfig eth0 172.16.0.1/24
# dpdkgen 172.16.0.2@02:03:04:05:06:07
# ifconfig eth1 hw ether 02:03:04:05:06:07
# ifconfig eth1 172.16.0.2/24
What’s
wrong?

39. Packet Generator: Symptoms
39
Lab
Switch
Packet
Generator
DUT
# ifconfig eth0 172.16.0.1/24
# dpdkgen 172.16.0.2@02:03:04:05:06:07
# ifconfig eth1 hw ether 02:03:04:05:06:07
# ifconfig eth1 172.16.0.2/24
What’s
wrong?LAN
@#*&!!
^*%$+!1!

40. Recap: Per-VLAN Switch Logic
40
Host A
Host B
Host C
Host D
D
A
PortMAC
1
2
3
4
MAC Table
Learn!

41. 1A
Recap: Switch Logic
41
Host A
Host B
Host C
Host D
D
A
1
PortMAC
1
2
3
4
MAC Table
2
1. Check Ethernet FCS
2. Add A to MAC table
3. No D in MAC table:
○ what to do?

42. 1A
Recap: Switch Logic
42
Host A
Host B
Host C
Host D
D
A
1
PortMAC
1
2
3
4
MAC Table
2
D
A
D
A
3
3
“Unknown unicast”
D
A
3
1. Check Ethernet FCS
2. Add A to MAC table
3. No D in MAC table:
○ send to all ports
Problems?

43. 1A
Recap: Switch Logic
43
Host A
Host B
Host C
Host D
4
Port
D
MAC
1
2
3
4
MAC Table
2
A
D
1
1. Check Ethernet FCS
2. Add D to MAC table
3. A in MAC table:
○ send to port 1
A
D
3
Never
happens :(

44. 1. Receive frame, check Ethernet FCS
2. Add/update source MAC in MAC table
3. If multicast bit is set:
a. forward to all ports, but the source
4. If destination is in MAC table:
a. forward to the specific port
5. Else, forward to all ports
Recap: Per-VLAN Switch Logic
44

45. Root cause — unicast flood
due to unidirectional traffic
(packet generator —> device under test)
45
Unicast
flood?

46. Unicast flood — unintentional behavior of a switch
treating a unicast packet as a broadcast packet.
— Wikipedia
46

47. Traffic Generator: Analysis
47
Lab
Switch
Packet
Generator
DUT
# ifconfig eth0 172.16.0.1/24
# dpdkgen 172.16.0.2@02:03:04:05:06:07
# ifconfig eth1 hw ether 02:03:04:05:06:07
# ifconfig eth1 172.16.0.2/24
LAN
D
G
D
GForwards to
all ports
Solutions?
D
G

48. Packet Generator: Solutions
1. Immediate solution: disconnect traffic generator
2. Connect DUT directly to packet generator
3. Use VLANs:
switch(config)# interface gigabitethernet 0/5 - 7
switch(config-if)# switchport access vlan VLAN
4. Add static MAC entries:
switch(config)# mac address-table static MAC vlan VLAN interface gigabitethernet 0/5
ubuntu# bridge fdb add LLADDR dev DEV
48
More?
Source: CCNP SWITCH, pages 97, 40

49. Packet Generator: Admin Solution
49
Lab
Switch
Packet
Generator
DUT
LAN
D
G
lab-switch# configure terminal
lab-switch(config)# interface gigabitethernet 0/5
lab-switch(config-if)# description --- Packet Generator
lab-switch(config-if)# storm-control unicast level 1
Source: CCNP SWITCH, page 421
Storm
control
Storm
control

50. Why set speed on a port?
— Make sure it is the fastest possible option!
1000Mbit/s, Full Duplex
Quiz 5. Autonegotiation
50
More?

51. ubuntu# ethtool -s eth0 autoneg off speed 1000
arista(config-if-Et1)# speed forced 1000full
cisco(config-if)# speed 1000
Autonegotiation: How?
51

52. Autonegotiation: Quiz!
52
# ethtool -s eth0 autoneg off speed 1000
What’s
wrong?
Host
LAN

53. Autonegotiation: Symptoms
53
# ethtool -s eth0 autoneg off speed 1000
What’s
wrong?
Host
LAN
Slo-o-o-o-ow!!!
What’s
autonegotiation?

54. Autonegotiation — when two devices choose
common speed, duplex, and flow control
Note: if autonegotiation is disabled, the device that is capable of autonegotiation can
determine speed, but cannot determine duplex, so half duplex is always assumed
— Wikipedia
54Source: https://en.wikipedia.org/wiki/Autonegotiation

55. Root cause — duplex mismatch, i.e. two devices
operate in different duplex modes
— Wikipedia
55

56. Autonegotiation: Analysis
56
# ethtool -s eth0 autoneg off speed 1000
Host
LAN
Autonegotiation
is off
Autonegotiation
is on
Solutions?Implicit
full duplex
Fallbacks to half
duplex

57. Autonegotiation: Solutions
Enable autonegotiation!
Rule: both auto OR both manual
57

58. — So, why slow?
58

59. Collision Domain
Recap: Collision Domain
59
Host A
Host B
Host C
Host D
Repeater Hub
A
B
A
C
1
1
A
C
A
B
2
A
C
A
B
2
Solutions?

60. Recap: CSMA/CD
Carrier-Sense Multiple Access with Collision Detection
media access control method:
1. IF medium is not idle THEN goto 1
2. Start transmission + monitor for a collision
3. IF collision occurred THEN wait, goto 1
60
:(

61. Networking Debug Tools: tcpdump
Usage:
# tcpdump -envvi eth0
-e print ethernet header
-n numeric format
-v verbose
-v even more verbose (checksums)
or use WireShark
61

62. Networking Debug Tools: netstat
Usage:
# watch -n 1 “netstat -s | grep -vw 0”
-n update interval
-s display statistics
-v invert match
-w match whole words
62

63. Networking Debug Tools: counters
1. NIC counters:
# ip -s -s link
-s output more statistics
2. Switch counters:
switch> show interface ethernet 1
switch> show interface ethernet 1 status
63

64. Networking Quiz Checklist
1. Overlapping IP networks?
2. Rarely used private prefix?
3. Default netmask?
4. ifconfig vs routing?
5. Static routing?
6. Multicast MAC?
7. Unknown unicast?
8. Duplex mismatch?
9. Debug tools?
64

65. References
1. Kevin Wallace. CCNP Routing and Switching ROUTE 300-101 Official Cert Guide. Cisco Press, 2014
2. David Hucaby. CCNP Routing and Switching SWITCH 300-115 Official Cert Guide. Cisco Press, 2014
3. https://en.wikipedia.org/wiki/Autonegotiation
4. https://en.wikipedia.org/wiki/Duplex_mismatch
65