This document summarizes the creation of a self-managed load balancer on AWS to distribute mail traffic across multiple mail gateway servers. It describes the existing mail system architecture, the need for a load balancer due to traffic volume limitations, and the technical implementation using Linux Virtual Server (LVS) and keepalived for load balancing and iptables for network address translation (SNAT) to support load balancing of SMTP traffic. The results were an increased ability to scale mail gateway servers elastically and observe traffic patterns from email services like Google Apps. A note of caution is provided about network bandwidth limitations based on the EC2 instance type used for the load balancer.

1. Self-Created Load Balancer
on AWS
Shinya Matsuura
Dep of Service Engeneering

2. What SEG is Doing?
Mail Filtering
(Mo)
Mail Archiving
(Nobita)
Anti-Spam
Anti-Virus(Mi)
Online Storage
(kumamsushi)
Access Conrroll
(shobosso)

3. Mail System Framework
Mail Filtering
Mail Archiving
Anti-Spam
Anti-Virus
Gateway
Server
Gateway
Server
Inbound Zone Outbound ZoneArchive Zone

4. Why We Need Load Balancer?
Gateway
Server
Limitation of
Instance Type
numerous
ammounts of mails
beyond expectation
Problem of
I/O

5. Extended Gateway Server
Gateway
Server
Gateway
Server
Gateway
Server
Gateway
Server
Load Balancer
・・・

6. Technology in Use
１） Linux Vritual Server (LVS)
+ keepalived
２） SNAT in iptables

7. 1) LVS + keepalived
- Install ipvsadm
- Enable packet forwarding between interfaces
# /etc/sysctl.conf
===
net.ipv4.ip_forward = 1
===
# /sbin/sysctl -p

8. 1) LVS + keepalived
- Install keepalived and configure keepalive.conf
# /etc/keepalived/keepalived.conf
===
! Configuration File for keepalived
global_defs {
router_id lbx.oz.hdemail.jp
}
virtual_server 10.x.x.x 25 {
delay_loop 30
lb_algo lc
lb_kind NAT
protocol TCP
real_server 10.x.x.x 25 {
weight 1
inhibit_on_failure
SMTP_CHECK {
connect_port 25
connect_timeout 10
retry 2
delay_before_retry 5
helo_name lbx.oz.hdemail.jp
}
}
Global Deifinition
Child Server
Virtual Server(LB)

9. 1) LVS + keepalived
- Command
・Join a child server
$ ipvsadm -a -t [LB IP]:25 -r [Child Server IP]:25 -m
・Drop a child server
$ ipvsadm -d -t [LB IP]:25 -r [LB IP]:25 -m
・Confirm child servers
$ ipvsadm -Ln
===
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn
InActConn
TCP 10.x.x.x:25 lc
-> 10.x.x.x:25 Masq 1 59 17
-> 10.x.x.x:25 Masq 1 55 3

10. 2) SNAT in iptables
- Problem
When we use EC2 instance for Load Balancer
we cannot create a SMTP connection,
since SMTP trafic returns towerd
EC2 default gateway not toword Load Balancer.

11. 2) SNAT in iptables
# iptables -t nat -A POSTROUTING -m ipvs --vaddr 10.X.X.X --
vport 25 -j SNAT --to-source 10.X.X.X
- Solution
We use an SNAT function in iptables to overwrite a sender IP
address. For example, a sender IP address of packet which is sent
to Port 25 in Load Balancer can be overritten by the local IP
address of LB.

12. Tools
・Fabric task to join/drop child server and update keepalived
$ fab join_ozmax_gw:10.x.x.x
$ fab drop_ozmax_gw:10.x.x.x
$ fab update_ozmax_lb_healthcheck
・Scrpt to automatically generate keepalived.conf in accordance
with MX records
・Script to update iptables when LB instance starts
・Script to observe number of connections, network bandwidth
etc.
What we needed

13. Results
・We can increase a child server as much as
and whenever we want !
・We can observe how Google Apps
changes their way to send mails !
・We can also use a Self-Created LB as an
internl LB in EC2!

14. A Note of Causion
There is a limitation of network bandwidth
dependent on instance type.
m1.small -> m1.medium -> c3.xlarge

15. Extended Gateway Server Now
Gateway
Server
Gateway
Server
Gateway
Server
Gateway
Server
Load Balancer
× n

16. FIN