1.
Server Load Balancing and Failover
January 2015 Hangout
Jim Pingle

2.
Project Notes
● pfSense 2.2-RELEASE! (yay)
– Read the blog post, release notes, and upgrade guide before updating!
Several known issues that could be harmful if not accounted for
– https://blog.pfsense.org/?p=1546
● pfSense University on-line training
– https://blog.pfsense.org/?p=1517
– Two-day course, Fundamentals and second day for Advanced
– Second day is optional but requires having attended a Fundamentals course
– Special introductory prices, see the blog post for details
– Several classes, some have filled up
● Wiki received many updates, edits, new articles and cleanup.

3.
About this Hangout
● We will cover Server Load Balancing and
Failover using relayd and HAProxy
● Advantages and disadvantages of each
● Assumes web servers are already in place,
including DNS entries
● Practical example, demonstrated

4.
About relayd and HAProxy
● relayd is built into the pfSense firmware
– Originally from OpenBSD
– Meant to work with pf directly
– Simple service for specific tasks
● HAProxy is an add-on package for pfSense
– Very powerful/flexible
– True proxy
– More capabilities, but uses more resources

5.
Comparison of relayd vs HAProxy
● How they Operate
– relayd works using NAT and pf, like a “super” port forward
– HAProxy is a true proxy, accepting client connections and making new connections
to servers
● Connection Handling
– relayd does not inspect the contents of packets, it only forwards based on the
packet headers
– HAProxy can look inside the request and can act on headers
● For example, HAProxy can route requests to specific servers based on requested hostname
● Service Types
– Both can handle arbitrary services
– The features in HAProxy are geared toward HTTP/HTTPS, but can work with other
protocols

6.
Comparison of relayd vs HAProxy
● Client Addresses
– relayd passes the actual client IP address on to the server, so the server sees the real
address in is logs
– By default, HAProxy connections appear to originate from the proxy itself
● HAProxy can add X-Forwarded-For which may be used by the web server
– Example: http://www.techstacks.com/howto/log-client-ip-and-xforwardedfor-ip-in-apache.html
● HAProxy also has a Transparent Client IP setting to pass through the actual client address, similar
to relayd
● Reporting
– relayd only reports up/down status and uptime percentages
– HAProxy is capable of giving very detailed statistics about usage and distribution of
traffic/requests
● Reliability
– For more than simple/basic tasks, relayd may be unreliable, especially with high loads
– HAProxy is more much robust and reliable, but does consume more resources in the
process

7.
Comparison of relayd vs HAProxy
● Daemon/Service Binding
– Because relayd works using NAT, relayd does NOT bind to a virtual server IP address
and port
– HAProxy must bind to the IP address and port specified for a Frontend virtual server
● If the GUI or GUI redirect is running on a port to be used by HAProxy, it must be moved
● Balancing Methods
– relayd on pfSense can only use round-robin style load balancing
– HAProxy supports several balancing algorithms
● Round robin, Static Round Robin, Least Connections, Source
● Client/Server Relationships
– relayd, even using Sticky, has issues maintaining client-server relationships
– HAProxy can maintain client-server relationships in several ways, such as by source
address or cookie values
● SSL Offloading
– HAProxy is capable of SSL Offloading, relayd is not

8.
Example Demonstration
Environment
● Test setup used for this demo of relayd and HAProxy includes:
– One firewall to act as a load balancer
– Three web servers
– Firewall WAN address of 192.168.20.74
– LAN servers are 10.42.42.11, 10.42.42.12, 10.42.42.13
– DNS entries set so www.example.com resolves to the WAN address,
along with other test hosts (devsite.example.com,
bloosh.example.com)
– Test client on WAN
● Virtual server for relayd will be on port 8080, HAProxy on 80, so
they can both be run at the same time for this demo

9.
Example relayd setup
● Full walk-through available on the Wiki:
https://doc.pfsense.org/index.php/Inbound_Loa
d_Balancing
● Before setting up servers, configure monitors as
needed
– This example uses basic HTTP for simplicity, but a
custom HTTP monitor can check a specific URI to
ensure the web server is operational
● Ensure the target servers are on-line and
operational

10.
Example relayd setup
● Setup Load Balancing Pool
– This defines the pool of actual web servers that exist on the internal network
– Services > Load Balancer, Pools tab
– Click +
– Name = MyServers, or something short (no spaces or special characters)
– Mode = Load Balance
– Port = 80
● This is the port that the ACTUAL web servers are listening on, NOT the public port!
– Retry = 5, how many times to test before declaring the server dead
– Monitor = HTTP, or whichever monitor is desired
– Enter one of the actual internal web server IP addresses, then click Add to Pool, repeat as
needed until all servers are present in the Enabled column.
– Click Save
● If there is a separate “maintenance” or static page server to use as a fall back if all of the
above servers are down, create a separate Pool and add that server

11.
Example relayd setup
● Setup a Virtual Servers
– Defines the public-facing address and port for the web site
– Services > Load Balancer, Virtual Server tab
– Click + to add a new Virtual Server
– Name = MyWebSite, or something similar (short, no special characters, etc)
– IP Address = The WAN address (typically public!) on which the site is to be hosted
● If this is not the WAN address, it may need a Virtual IP address defined
– Port = 80 (or 8080 for this example), the port on which clients will connect from the outside
● Typically 80 for HTTP and 443 for HTTPS
– Virtual Server Pool = The pool defined previously (e.g. MyServers)
– Fall Back Pool = none, or choose one if one was defined previously
– Relay Protocol = TCP
– Click Submit

12.
Example relayd setup
● Visit the Settings tab (optional)
– Timeout: Milliseconds before a health check is failed, defaults to 1000ms (1
second)
– Interval: Seconds between health checks. Default is 10 seconds.
– Prefork: Not used for TCP
● With the default 5 retries, 1000ms timeout, 10 seconds between
checks, it could be a full minute before a down server is detected.
Tuning is strongly advised!
● Add firewall rules to pass traffic to a destination of the Internal IP
addresses of the actual web servers and their ports. In this
example, pass to 10.42.42.11, 10.42.42.12, 10.42.42.13 on port 80
– Aliases are handy for this!

13.
Example relayd setup
● Clients on LAN cannot access servers on LAN without
manual outbound NAT rules to mask the traffic
– Interface = LAN, Source = LAN subnet, Destination = Web Servers,
Destination Port = Server Port, Translation = Interface Address
● Status is at Status > Load Balancer
– Uncheck a server and click Save to manually remove it from service
● Aim a web browser at it, e.g. http://www.example.com:8080
● When testing, use private browsing mode, shift+click reload
or ctrl+F5

14.
Example HAProxy setup
● To use port 80 on HAProxy with the GUI on another
port, disable redirect on System > Advanced
● If HAProxy will use the same port as the GUI, move
the GUI to another port on System > Advanced
● Install the HAProxy-devel package from System >
Packages, Available Packages tab
– Other variants may be available, but currently we
recommend HAProxy-devel. It's stable but still being
polished up and prepared to take over as the main
package

15.
Example HAProxy setup
● Add a Backend “pool”
– Services > HAProxy, Backend tab
– Name = MyWebServers (or similar)
– Servers = Add each of the actual internal web servers along with the port on which they are
listening internally
● Example: Active, serverX, Address+Port, 10.42.42.x, 80, unchecked, blank
– Balance = Round robin
– Transparent ClientIP = Your choice
– Health Check Method = HTTP
– (Optional): Stick tables
● Stick on Existing Cookie Value
● Cookie Name = PHPSESSID
● Length = 64
● Expire = 3h
● Size = 100k (max # of concurrent clients)
– Review other settings, set as needed
– Click Save

16.
Example HAProxy setup
● Add a Frontend (“Virtual Server”)
– Services > HAProxy, Frontend tab
– Name = MyWebSite
– Status = Active
– External Address
● Listen address = WAN address (IPv4)
– The PUBLIC facing IP address of the service
– Address to which the DNS entries resolve
– Could use a VIP here as well, of course
● Port = 80, the PUBLIC facing port for the service
– Backend Server Pool = MyWebServers
– Type = HTTP / HTTPS(offloading)
– Use 'forwardfor' option (optional) = checked, adds X-Forwarded-For header with true
client IP
– Use 'httpclose' option (optional) = httpclose, disables keep-alive, ensures X-
Forwarded-For is accurate
– Click Save

17.
Example HAProxy setup
● Setup HAProxy global settings
– Services > HAProxy, Settings tab
– Enable HAProxy = checked
– Maximum Connections = 1000 per backend (tune to suit available resources
and load!)
– Internal Stats Port = 2200
– Review other options
– Click Save
● Stats tab shows server status and stats, can manually disable servers
● Add firewall rules to pass traffic to the Frontend listen IP address and
port
● Aim a web browser at it, e.g. http://www.example.com
● When testing, use private browsing mode, shift+click reload or ctrl+F5

18.
Conclusion
● Questions?
● Ideas for hangout topics? Post on forum,
comment on the blog posts, Reddit, etc