Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Connectivity Troubleshooting - pfSense Hangout June 2016


Published on

Slides for the June 2016 pfSense Hangout video

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Connectivity Troubleshooting - pfSense Hangout June 2016

  1. 1. Connectivity Troubleshooting June 2016 Hangout Jim Pingle
  2. 2. Connectivity Troubleshooting ● Project News ● Diagnostic Tests ● Testing Methodology ● Disappearing Packets ● Subnets and Routes ● WAN Configuration ● LAN Configuration ● Firewall Rules ● Outbound NAT ● Firewall Connectivity ● DNS Issues ● Client Configuration ● Other Causes ● Q&A
  3. 3. Project News ● pfSense 2.3.1 update 5 is out (a.k.a. 2.3.1-RELEASE-p5 or 2.3.1_5) – Maintenance release with security and stability fixes since 2.3.1 update 1 ● Updates 2 through 4 were internal only, not released publicly ● XG-2758 case redesigned, can now accept an add-on board – First offering is a 4x 1GB expansion card ● BSDCan was earlier this month – netmap-fwd talk by Luiz: ● userland router application over netmap for FreeBSD, easy to use, tightly coupled with the OS and aimed at 10G networks ● Very, very fast ● IPv6 is working now, not in Github yet ● Being developed on our XG-2758 platform ● Link to code, summary and video at: ● ● Online training schedule for the rest of the year is up – Spots are open in all announced times – September is starting to fill up –
  4. 4. Diagnostic Tests ● Before getting into possible causes, know the tools involved ● Ping – Basic connectivity test using ICMP echo requests – Ping can succeed when TCP fails, however, so it is not a 100% reliable test – Diagnostics > Ping – Available on nearly every OS ● Traceroute – Trace a path to a destination – Diagnostics > Traceroute – Available on nearly every OS, command varies (traceroute vs tracert) ● Packet Captures – See what is on the wire – Diagnostics > Packet Capture – Varies by OS: tcpdump, wireshark, etc ● Netcat – Easy TCP port handshake test – Diagnostics > Test Port, or from the shell, nc -vz x.x.x.x yyyy
  5. 5. Testing Methodology ● Always test from one end to the other, starting as close to the source of the traffic as possible, stopping at each interface and hop along the way ● For inbound traffic: – Remote system → Firewall WAN/VPN → Firewall LAN → Client LAN ● For outbound traffic: – Client LAN → Firewall LAN → Firewall WAN/VPN → Remote system ● If available, also check/capture at hops between ● Check state table to see if the traffic is passing, firewall logs to see if it's blocked ● Run a packet capture in each place along the path – Confirm source and destination MACs match the client and firewall, firewall and the upstream gateway, and so on, where relevant
  6. 6. Disappearing Packets ● There are a few ways that packets can “disappear” – they appear to enter the firewall but do not appear in a capture on the expected exit interface ● Exiting an different interface – Policy routing, static route, etc, sending out another path ● IPsec SPD Matches – Can interfere with OpenVPN and other traffic ● Missing route (default or link route) – If the firewall cannot determine where the packet should go, it is dropped and does not exit ● Captive Portal on LAN blocking requests ● Dropped by pf – pf state mismatches, other pf issues – Less common, but some rare cases exist – Reset states between tests
  7. 7. Subnets and Routes ● IP address and subnet mask together define the subnet which is local to a host ● Subnet mask is defined mathematically, use a calculator to find actual start of subnet – Quite common for someone to mistakenly think includes .1.x and .2.x, it does not, it includes .0.x and .1.x ● Hosts find each other inside a subnet using ARP ● Hosts will have a link route for the subnet they are in, tying that subnet to a specific interface – On pfSense, this means only one interface can be in a specific subnet, otherwise the link route can point to the wrong interface, and traffic will not flow (e.g. same subnet on WAN and LAN w/o bridge) ● Destinations outside of directly connected subnets are routed via the host's static routes or default gateway – A missing or incorrect default gateway will prevent communication outside the subnet & static routed networks – When using static routes, typically both parties would need a static route, especially with site- to-site
  8. 8. Subnets and Routes (cont'd) ● If a subnet mask is incorrect, several problems can occur: – Subnet mask too small ● Could have problems reaching the gateway ● Other hosts in what should be its subnet cannot be reached ● May use the wrong broadcast address – Subnet mask too large ● Will attempt to ARP for destinations which are not local ● Static routes send specific destinations to an alternate gateway in the current subnet – An incorrect route can send traffic down an unexpected path – Remote destinations not matching static routes are reached via the default gateway – Watch out for persistent static routes on clients when changing/renumbering firewalls
  9. 9. WAN Configuration ● Have the IP address configuration info from the ISP on hand ● Check that the address being used is for the host (customer end) and not the gateway (ISP end) – In larger subnets, make sure the ISP is not also using additional addresses for CARP/VRRP/HSRP ● Check that the subnet mask is correct and that it is neither too big (/1) or too small (/32) – A /1 subnet mask on WAN will cause approximately half of the Internet to be unreachable, which may not be immediately obvious ● Ensure the correct gateway address is used, and that it is online and replies to ping – If it does not respond to ping, check the ARP table, it may still be online but require an alternate monitor IP address ● Check Diagnostics > Routes and ensure the default gateway is present and shown as “default” – If not, visit System > Routing, edit the gateway, check Default, Save, Apply
  10. 10. LAN Configuration ● Check that the firewall LAN IP address is correct – It must not be in the same subnet as the WAN or any other directly connected interface – It must not be the subnet ID nor the broadcast IP address of a subnet (e.g. .0 or .255 in a / 24) – It must not conflict with anything else currently on the LAN segment (old routers, servers, etc) ● Check that the firewall LAN subnet mask is correct – And incorrect mask can lead to communication problems with hosts in the LAN or with VPNs and other remote networks – A /32 mask would prevent it from communicating with any host in the LAN ● Check that the LAN interface does not have a gateway selected under Interfaces > LAN – This would cause the firewall to treat the interface as a WAN-type interface when configuring automatic outbound NAT and firewall rules ● Check the other LAN interface settings to ensure things like “Block Private Networks” are not set
  11. 11. Firewall Rules ● Check Status > System Logs, Firewall tab for blocked connections – If traffic should be allowed, it is either hitting a block rule or failing to match a pass rule ● Look at rule hit counters on the tab where traffic originates ● Check state table contents if the logs are inconclusive ● Check the protocol on firewall rules to ensure the traffic you are trying to pass will actually pass – Example: Rule set for TCP only will not pass ping (ICMP) or most DNS (UDP) ● Check that LAN Rules pass to a destination of any and not “WAN net” – “WAN net” is only the directly connected subnet including the WAN IP address, not “The Internet” ● Check LAN rule gateways, ensure traffic is not being misdirected ● Check Floating rule tab and interface group rules, which may hold forgotten or misconfigured rules ● Disable any installed firewall packages such as pfBlocker, at least until the connectivity issue is resolved
  12. 12. Outbound NAT ● If traffic appears to be passing but cannot reach the Internet... ● Check the state table at Diagnostics > States, see what the WAN- side state looks like. If the connection only shows the local private IP address and the remote destination, NAT is not happening ● Firewall > NAT, Outbound tab ● Ensure NAT is set on auto or that there are appropriate rules for a source of the LAN exiting WAN ● If on auto, make sure that Interfaces > WAN has a gateway selected and that Interfaces > LAN does not ● Be sure to clear states between tests! – Diagnostics > States, Reset States tab
  13. 13. Firewall Connectivity ● After checking all of the above, connectivity from the firewall should be working ● Diagnostics > DNS, try a DNS lookup – If it fails, check the DNS settings (next slide) ● Diagnostics > Ping, try a few Internet hosts by name and IP address – Repeat the test using a source interface of the LAN to test NAT ● Diagnostics > Traceroute, try a few Internet hosts ● Diagnostics > Test Port, try to connect out, for example, to TCP/80 on ● If anything fails go back and check the settings on WAN again, and make sure there is no ISP issue
  14. 14. DNS Issues ● While not truly a connectivity issue in most cases, a lack of functioning DNS will effectively prevent clients from reaching the Internet ● Test DNS from Diagnostics > DNS Lookup ● A few things additional issues can arise with DNS, which can affect client connectivity: ● Upstream firewall or router might erroneously be blocking large DNS queries ● DNS Resolver – Forwarding mode enabled ● Check DNS Servers under System > General Setup ● Disable DNSSEC it is not supported by the forwarders – Forwarding mode disabled ● Ensure the default gateway is working and that the firewall has unfiltered outbound access to TCP/UDP port 53 to any destination, as unbound must talk to the roots and other authoritative DNS servers ● DNS Forwarder – Check DNS Servers under System > General Setup
  15. 15. Client Configuration ● Check the client IP address and mask to ensure they are in the correct subnet ● Check the client gateway to ensure it is pointed to the firewall IP address, not some other gateway – If the client must use another gateway, then it would require some other means of addressing traffic to/from pfSense – static routes, outbound NAT on LAN of the firewall to mask the source, etc ● Check the client DNS to ensure it is using valid DNS servers (e.g. firewall IP address) ● Test if the client can ping… – Its own IP address – The LAN IP address of the firewall ● If it fails, check LAN rules, client IP/mask, pfSense LAN IP/mask – The WAN IP address of the firewall ● If it fails, check the client IP/mask – The default gateway on WAN ● If it fails, check outbound NAT, client IP/mask – A host on the Internet by IP address ● If it fails, check LAN rules, outbound NAT, client IP/mask, firewall WAN IP/mask, firewall gateway – A host on the Internet by name ● If it fails, check client DNS, firewall DNS, and firewall rules to pass DNS
  16. 16. Other Causes ● Captive Portal would prevent a LAN host from getting out and would also prevent a remote host (DMZ, WAN, or VPN) from reaching a host on LAN unless the LAN host is logged in – Add a bypass by MAC or IP address to allow connections without portal authentication ● IPsec tunnel, configured and enabled but not connected, will stop any traffic matching its P2s from flowing ● Misuse of static port on outbound NAT rules can cause what appear to be intermittent failures. Some clients work, others do not ● A misconfigured or malfunctioning package such as Squid could appear to make some traffic disappear or fail to function
  17. 17. Conclusion ● Questions? ● Ideas for hangout topics? Post on forum, comment on the blog posts, Reddit, etc