Slides for the March 2016 pfSense Hangout video

1. Multi-WAN on pfSense 2.3
March 2016 Hangout
Jim Pingle

2. Multi-WAN on pfSense 2.3
● Project News
● What is Multi-WAN?
● Why use Multi-WAN?
● Improvements in 2.3
● Failover or Load
Balancing?
● Choosing Service
Providers
● Example Setup
● Multi-WAN Tweaks
● Testing and
Troubleshooting
● Q&A

3. Project News
● 2.3-RC any moment now
– Release timing will roughly parallel FreeBSD 10.3-RELEASE
– Snapshots at https://snapshots.pfsense.org/
– New RRD Graph interface are in place (Status > Monitoring)
– Inline IPS mode (Netmap) with Suricata 3.0 now available
● XG-2758 units now shipping
● European pfSense Training Tour!
– April 7-8 in Bournemouth (UK, Amica Partner)
– April 12-13 in London (UK, Amica Partner)
– May 17-18 in Frankfurt (DE, Voletech Partner)
– http://netgate.com/training/ – All are 9am-6pm local time
● Good follow-up review of the SG-4860 from ServeTheHome:
http://www.servethehome.com/pfsense-sg-4860-6-month-review-great-fire
wall-router-combo/
● Keep an eye on the blog

4. 2.3 Multi-WAN Improvements
● New dpinger utility replaces apinger
– Watch for gateway settings changes on upgrade!
● Default gateway switching has been improved
– Now works properly with PPP-based WANs
● Lots of cleanup to gateway handling
● Weight limit for LB increased from 5 to 30
● RFC2136 Dynamic DNS now supports Multi-WAN

5. What is Multi-WAN?
● Multiple WAN connections on a single firewall
● Typically multiple service providers or at least paths to the
Internet
● Concept can apply to any interface with a path to the Internet,
even if it is not a direct ISP connection, so long as it has a
gateway.
– MPLS with remote site that has an Internet connection
● WANs can be any type (Static, DHCP, PPPoE, etc)
– Works without access to routing protocols
● BGP is not feasible for many, especially smaller companies, homes, etc
● Works fine with HA, but all nodes need access to all WANs!

6. Why use Multi-WAN?
● Redundancy (WAN failover)
– Outbound: Local devices can still reach the Internet
– Inbound: All WANs can accept inbound connections
for local hosted services on LAN/DMZ/etc
● reply-to sends packets back out the WAN they entered
– Some firewall services can switch WANs as well, so
VPNs can be made redundant for example

7. Why use Multi-WAN?
● Additional Bandwidth (Load Balancing)
– If one WAN does not provide enough throughput
– Some locations may not be able to get a single circuit with decent
speed
– True aggregation is not possible, however
● Except with MLPPP, if the service provider supports it
● Otherwise, LB is connection-based
● Can be weighted to utilize certain WANs more often
– Be wary of third-party devices that claim to aggregate bandwidth
across different circuits, they typically funnel all traffic through a
datacenter and add overhead, latency, security concerns, and more.

8. Why use Multi-WAN?
● Service or Bandwidth Segregation
– Priority services such as VoIP can have a dedicated
circuit
● Or one that is not shared unless another circuit fails
– Provides true isolation of bandwidth for high priority
traffic without relying solely on traffic shaping
– Can also provide a means to shunt lower-priority
traffic to a slower circuit, (e.g. Guest network)

9. Choosing Service Providers
● Try to choose different connection types
– Cable vs Fiber vs DSL vs Wireless vs …
● Different cable paths, if possible
– Reduces the chances that a cable cut, pole hit, etc will cause a complete outage
● Different ISPs, if available
– Reduces the chances that an upstream peering problem will cause a complete outage
– Be wary of resellers that are actually using the same infrastructure
● If the same ISP must be used, WANs REQUIRE different subnets and gateways
– Two WANs cannot share the same subnet or gateway, common on DHCP networks like
cable depending on the circuit type
● LACP (e.g. in a DC environment) will not yield greater bandwidth, but will provide
redundancy
– L2 hashing means a single MAC (pfSense) to a single MAC (gateway) will not utilize all
connections in a LAGG
● 3G/4G WANs as backup – costly, OK, but consumes bw for monitoring so disable

10. Failover or Load Balancing?
● Failover prefers one WAN, fails to another when the preferred WAN is
down
– When the WAN recovers, new connections will go back to the preferred WAN,
open connections will not be cut off
– Currently no way to force a fail-back
● Load Balancing performs connection-based balancing
– Browsers will open multiple connections, so effectively things will be balanced
– No single connection can fully utilize all circuits
– HTTPS can break in cases where the remote is strict about source IP
addresses, common with banks
● Use failover for HTTPS or try using Sticky connections
● Sticky builds a user-to-gateway pairing as long as connections remain open
– Users balance between WANs, but their connections individually use only a specific WAN
● Only MLPPP offers true aggregation and transparent failover

11. Example Setup Diagram
Internet Internet
LAN DMZ
LAN
10.3.0.1/24
DMZ
10.3.1.1/24
WAN
198.51.100.3
WAN2
203.0.113.3
ISP 1 Modem
198.51.100.1
ISP 2 Modem
203.0.113.1

12. Example Setup Walkthrough
● Assign & Configure the new Interface
● Configure Gateways
● Add Gateway Groups
● Configure DNS
● Use Gateway Groups in Rules

13. Example Setup – Interface
● Assign New Interface (if not already assigned)
– Interfaces > (assign)
– Pick new interface
– Click Add
– Note the name (e.g. OPT2)
● Configure the new Interface
– Interfaces > OPTx,
– Enable, Set type/IP address/add gateway if static
– Save/Apply

14. Example Setup – Gateway Settings
● System > Routing
● Edit the gateway(s) if needed
● Set monitor IP addresses
– Be wary of monitoring the gateway directly, may be local modem/CPE
– Use anycasted address like 8.8.8.8 / 8.8.4.4 to check not just the line but also
Internet connectivity
● Choose the default gateway (if desired)
– Only one WAN can be the default
– Traffic from the firewall itself will always leave the default unless a static route
changes the path
● UDP replies for daemons bound to any/all will also take default route!
– Default Gateway Switching (System > Advanced, Misc) will pick the next available
gateway should the chosen default fail
● Advanced Options: Set as needed, though most defaults are OK

15. Example Setup – Gateway Groups
● System > Routing, Gateway Groups tab
● Groups themselves do not affect behavior, they must be used somewhere
(rules, services, etc)
● Groups have Tiers to define behavior
– Tier 1 is highest priority, used first
– Multiple gateways on the same tier are load balanced
– If all gateways on a tier are down, the next tier is checked and used (if up)
● Typical setups have three groups to start with:
– A Load Balance group with both WANs on Tier 1
– A “PreferWAN1” group with WAN1 on Tier 1 and WAN2 on Tier 2
– A “PreferWAN2” group with WAN1 on Tier 2 and WAN2 on Tier 1
● More gateways can be used in multiple ways, failing over between various
scenarios with and without load balancing, many many different possibilities!

16. Example Setup – DNS
● DNS Resolver in Forwarding mode / DNS Forwarder
– System > General
– Set at least one DNS server per WAN
– If DNS servers were used as monitor IP addresses, ensure the same
WAN relationship is retained here
● e.g. 8.8.8.8 is WAN1 monitor and WAN1 DNS server
– Depending on upstream DNS servers, DNSSEC may not be available
– DNS Forwarder will query all servers at once, fast/reliable
● DNS Resolver – Non-forwarding mode
– Requires Default Gateway Switching since it needs direct contact to
roots and other authoritative DNS servers
– In non-forwarding mode, always utilizes the default gateway WAN

17. Example Setup – Using Groups
(Rules)
● Firewall > Rules
– LAN/DMZ/etc – Internal interfaces only!
– Edit pass rule, select gateway group (or gateway)
● Negate policy routing for local/VPN traffic
– RFC1918 alias is handy
– Pass to local/VPN destinations ABOVE other rules with a
gateway set
● Never use gateways on WAN rules!
● Cannot policy route traffic outbound from the firewall
itself

18. Example Setup – Using Groups
(Services)
● Firewall Services can use FAILOVER ONLY – Gateway groups must have each
gateway on a separate tier
● Dynamic DNS – Use gateway group for Interface
● OpenVPN
– Servers (See previous hangout for details!)
● Bind to localhost + port forward on each WAN
● Separate instance on each WAN with its own tunnel network
● Always-up tunnels using a routing protocol such as OSPF
● Use gateway group as interface
– Clients
● Use gateway group as interface
● Multiple remote lines to multiple far-side addresses, each with a static route
● IPsec
– Site-to-site: Gateway group for Interface + Dynamic DNS + far side uses hostname
– Mobile Clients: Gateway group for Interface + Dynamic DNS + Default Gateway switching

19. Multi-WAN NAT (Outbound)
● Firewall > NAT, Outbound tab
● NAT does not control where traffic goes, only how it is
handled when it leaves – Controlling traffic is up to
policy routing on rules and static routes
● On automatic, with proper gateways set on WANs,
nothing usually needs to be done
– If using Auto Outbound NAT and rules are missing, check
interface gateway setting (e.g. Interfaces > WAN2)
● For systems with manual outbound NAT, clone existing
rules and copy for WAN2, making other adjustments if
necessary

20. Multi-WAN NAT (Port Forwards/1:1)
● Port forwards and 1:1 NAT are only active on a
specific chosen WAN
● To have port forwards work on both WANs,
copy the rules and have one rule per WAN with
an appropriate destination address
● For 1:1 NAT on each WAN, would need
appropriate VIPs on each WAN

21. Multi-WAN Tweaks
● System > Advanced, Miscellaneous tab
● Use Sticky Connections / Source Tracking Timeout
– Builds a USER to GATEWAY relationship for all connections from that user
– Helps with HTTPS and other services that do not allow IP address changes mid-connection
– Relationship lasts as long as states from the source IP address are present. Adjust source tracking timeout
to keep the relationship alive longer
– Makes Load Balancing less effective but gains stability for troubled protocols
●
Enable Default Gateway Switching
– On 2.3 it works well with all known WAN types
– Gateway ordering is still tricky, can select internal non-Internet gateways so be careful if any are present!
●
State Killing on Gateway Failure
– When checked, if a gateway fails ALL states on ALL interfaces are flushed
– Helps some protocols such as VoIP that have issues failing over due to long-lived states
– Be wary of second gateway failing and resetting states even when preferred WAN is up!
●
Skip rules when Gateway is Down
– By default a rule with a gateway omits the gateway when the gateway is down
– When checked, the rule itself is completely omitted, allowing rules to lock users to a specific WAN or
otherwise more tightly control traffic

22. Testing & Troubleshooting
● Failover:
– Check gateway status and group status
– Verify the monitor IP addresses respond to ICMP echo requests
● Load Balancing:
– Always use a fresh browser, or even better, curl/wget/fetch
– Connection-based, so max bandwidth is the bandwidth of the WAN handling the connection,
unless the test uses multiple streams
– Use weights to adjust LB as needed
●
Try different WAN failure types
– Unplug interface, unplug upstream cable, cut power, etc
● Detecting WAN failure can take a minute or so, depending on gateway settings and
type of failure
● Detecting recovery takes some time as well because dpinger waits for the WAN to
become reliable again, rather than assuming it is up when the first ping returns.

23. Conclusion
● Questions?
● Ideas for hangout topics? Post on forum,
comment on the blog posts, Reddit, etc