Slides for the June 2016 pfSense Hangout video

1. Connectivity Troubleshooting
June 2016 Hangout
Jim Pingle

2. Connectivity Troubleshooting
● Project News
● Diagnostic Tests
● Testing Methodology
● Disappearing Packets
● Subnets and Routes
● WAN Configuration
● LAN Configuration
● Firewall Rules
● Outbound NAT
● Firewall Connectivity
● DNS Issues
● Client Configuration
● Other Causes
● Q&A

3. Project News
● pfSense 2.3.1 update 5 is out (a.k.a. 2.3.1-RELEASE-p5 or 2.3.1_5)
– Maintenance release with security and stability fixes since 2.3.1 update 1
● Updates 2 through 4 were internal only, not released publicly
● XG-2758 case redesigned, can now accept an add-on board
– First offering is a 4x 1GB expansion card
● BSDCan was earlier this month
– netmap-fwd talk by Luiz:
● userland router application over netmap for FreeBSD, easy to use, tightly coupled with the OS and aimed at
10G networks
● Very, very fast
● IPv6 is working now, not in Github yet
● Being developed on our XG-2758 platform
● Link to code, summary and video at:
● https://www.bsdcan.org/2016/schedule/events/699.en.html
● Online training schedule for the rest of the year is up
– Spots are open in all announced times
– September is starting to fill up
– http://netgate.com/training/

4. Diagnostic Tests
● Before getting into possible causes, know the tools involved
● Ping – Basic connectivity test using ICMP echo requests
– Ping can succeed when TCP fails, however, so it is not a 100% reliable test
– Diagnostics > Ping
– Available on nearly every OS
● Traceroute – Trace a path to a destination
– Diagnostics > Traceroute
– Available on nearly every OS, command varies (traceroute vs tracert)
● Packet Captures – See what is on the wire
– Diagnostics > Packet Capture
– Varies by OS: tcpdump, wireshark, etc
● Netcat – Easy TCP port handshake test
– Diagnostics > Test Port, or from the shell, nc -vz x.x.x.x yyyy

5. Testing Methodology
● Always test from one end to the other, starting as close to the source of
the traffic as possible, stopping at each interface and hop along the way
● For inbound traffic:
– Remote system → Firewall WAN/VPN → Firewall LAN → Client LAN
● For outbound traffic:
– Client LAN → Firewall LAN → Firewall WAN/VPN → Remote system
● If available, also check/capture at hops between
● Check state table to see if the traffic is passing, firewall logs to see if it's
blocked
● Run a packet capture in each place along the path
– Confirm source and destination MACs match the client and firewall, firewall and
the upstream gateway, and so on, where relevant

6. Disappearing Packets
● There are a few ways that packets can “disappear” – they appear to enter the
firewall but do not appear in a capture on the expected exit interface
● Exiting an different interface
– Policy routing, static route, etc, sending out another path
● IPsec SPD Matches
– Can interfere with OpenVPN and other traffic
● Missing route (default or link route)
– If the firewall cannot determine where the packet should go, it is dropped and does not
exit
● Captive Portal on LAN blocking requests
● Dropped by pf
– pf state mismatches, other pf issues
– Less common, but some rare cases exist
– Reset states between tests

7. Subnets and Routes
● IP address and subnet mask together define the subnet which is local to a host
● Subnet mask is defined mathematically, use a calculator to find actual start of subnet
– Quite common for someone to mistakenly think 192.168.1.0/23 includes .1.x and .2.x, it does
not, it includes .0.x and .1.x
● Hosts find each other inside a subnet using ARP
● Hosts will have a link route for the subnet they are in, tying that subnet to a specific
interface
– On pfSense, this means only one interface can be in a specific subnet, otherwise the link route
can point to the wrong interface, and traffic will not flow (e.g. same subnet on WAN and LAN
w/o bridge)
● Destinations outside of directly connected subnets are routed via the host's static
routes or default gateway
– A missing or incorrect default gateway will prevent communication outside the subnet & static
routed networks
– When using static routes, typically both parties would need a static route, especially with site-
to-site

8. Subnets and Routes (cont'd)
● If a subnet mask is incorrect, several problems can occur:
– Subnet mask too small
● Could have problems reaching the gateway
●
Other hosts in what should be its subnet cannot be reached
● May use the wrong broadcast address
– Subnet mask too large
● Will attempt to ARP for destinations which are not local
● Static routes send specific destinations to an alternate gateway in the
current subnet
– An incorrect route can send traffic down an unexpected path
– Remote destinations not matching static routes are reached via the default
gateway
– Watch out for persistent static routes on clients when changing/renumbering
firewalls

9. WAN Configuration
● Have the IP address configuration info from the ISP on hand
● Check that the address being used is for the host (customer end) and not the
gateway (ISP end)
– In larger subnets, make sure the ISP is not also using additional addresses for
CARP/VRRP/HSRP
● Check that the subnet mask is correct and that it is neither too big (/1) or too small
(/32)
– A /1 subnet mask on WAN will cause approximately half of the Internet to be unreachable,
which may not be immediately obvious
● Ensure the correct gateway address is used, and that it is online and replies to ping
– If it does not respond to ping, check the ARP table, it may still be online but require an alternate
monitor IP address
● Check Diagnostics > Routes and ensure the default gateway is present and shown
as “default”
– If not, visit System > Routing, edit the gateway, check Default, Save, Apply

10. LAN Configuration
● Check that the firewall LAN IP address is correct
– It must not be in the same subnet as the WAN or any other directly connected interface
– It must not be the subnet ID nor the broadcast IP address of a subnet (e.g. .0 or .255 in a /
24)
– It must not conflict with anything else currently on the LAN segment (old routers, servers, etc)
● Check that the firewall LAN subnet mask is correct
– And incorrect mask can lead to communication problems with hosts in the LAN or with VPNs
and other remote networks
– A /32 mask would prevent it from communicating with any host in the LAN
● Check that the LAN interface does not have a gateway selected under Interfaces >
LAN
– This would cause the firewall to treat the interface as a WAN-type interface when configuring
automatic outbound NAT and firewall rules
● Check the other LAN interface settings to ensure things like “Block Private
Networks” are not set

11. Firewall Rules
● Check Status > System Logs, Firewall tab for blocked connections
– If traffic should be allowed, it is either hitting a block rule or failing to match a pass rule
● Look at rule hit counters on the tab where traffic originates
● Check state table contents if the logs are inconclusive
● Check the protocol on firewall rules to ensure the traffic you are trying to pass will
actually pass
– Example: Rule set for TCP only will not pass ping (ICMP) or most DNS (UDP)
● Check that LAN Rules pass to a destination of any and not “WAN net” – “WAN net”
is only the directly connected subnet including the WAN IP address, not “The
Internet”
● Check LAN rule gateways, ensure traffic is not being misdirected
● Check Floating rule tab and interface group rules, which may hold forgotten or
misconfigured rules
● Disable any installed firewall packages such as pfBlocker, at least until the
connectivity issue is resolved

12. Outbound NAT
● If traffic appears to be passing but cannot reach the Internet...
● Check the state table at Diagnostics > States, see what the WAN-
side state looks like. If the connection only shows the local private
IP address and the remote destination, NAT is not happening
● Firewall > NAT, Outbound tab
● Ensure NAT is set on auto or that there are appropriate rules for a
source of the LAN exiting WAN
● If on auto, make sure that Interfaces > WAN has a gateway
selected and that Interfaces > LAN does not
● Be sure to clear states between tests!
– Diagnostics > States, Reset States tab

13. Firewall Connectivity
● After checking all of the above, connectivity from the firewall should
be working
● Diagnostics > DNS, try a DNS lookup
– If it fails, check the DNS settings (next slide)
● Diagnostics > Ping, try a few Internet hosts by name and IP address
– Repeat the test using a source interface of the LAN to test NAT
● Diagnostics > Traceroute, try a few Internet hosts
● Diagnostics > Test Port, try to connect out, for example, to TCP/80
on www.google.com
● If anything fails go back and check the settings on WAN again, and
make sure there is no ISP issue

14. DNS Issues
● While not truly a connectivity issue in most cases, a lack of functioning DNS will
effectively prevent clients from reaching the Internet
● Test DNS from Diagnostics > DNS Lookup
● A few things additional issues can arise with DNS, which can affect client
connectivity:
● Upstream firewall or router might erroneously be blocking large DNS queries
● DNS Resolver
– Forwarding mode enabled
● Check DNS Servers under System > General Setup
● Disable DNSSEC it is not supported by the forwarders
– Forwarding mode disabled
● Ensure the default gateway is working and that the firewall has unfiltered outbound access to TCP/UDP
port 53 to any destination, as unbound must talk to the roots and other authoritative DNS servers
● DNS Forwarder
– Check DNS Servers under System > General Setup

15. Client Configuration
●
Check the client IP address and mask to ensure they are in the correct subnet
●
Check the client gateway to ensure it is pointed to the firewall IP address, not some other
gateway
– If the client must use another gateway, then it would require some other means of addressing traffic
to/from pfSense – static routes, outbound NAT on LAN of the firewall to mask the source, etc
●
Check the client DNS to ensure it is using valid DNS servers (e.g. firewall IP address)
●
Test if the client can ping…
– Its own IP address
– The LAN IP address of the firewall
● If it fails, check LAN rules, client IP/mask, pfSense LAN IP/mask
– The WAN IP address of the firewall
● If it fails, check the client IP/mask
– The default gateway on WAN
● If it fails, check outbound NAT, client IP/mask
– A host on the Internet by IP address
●
If it fails, check LAN rules, outbound NAT, client IP/mask, firewall WAN IP/mask, firewall gateway
– A host on the Internet by name
● If it fails, check client DNS, firewall DNS, and firewall rules to pass DNS

16. Other Causes
● Captive Portal would prevent a LAN host from getting out and
would also prevent a remote host (DMZ, WAN, or VPN) from
reaching a host on LAN unless the LAN host is logged in
– Add a bypass by MAC or IP address to allow connections without portal
authentication
● IPsec tunnel, configured and enabled but not connected, will stop
any traffic matching its P2s from flowing
● Misuse of static port on outbound NAT rules can cause what
appear to be intermittent failures. Some clients work, others do not
● A misconfigured or malfunctioning package such as Squid could
appear to make some traffic disappear or fail to function

17. Conclusion
● Questions?
● Ideas for hangout topics? Post on forum,
comment on the blog posts, Reddit, etc