Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Self Created Load Balancer for MTA on AWS


Published on

Published in: Technology
  • Be the first to comment

Self Created Load Balancer for MTA on AWS

  1. 1. Self-Created Load Balancer on AWS Shinya Matsuura Dep of Service Engeneering
  2. 2. What SEG is Doing? Mail Filtering (Mo) Mail Archiving (Nobita) Anti-Spam Anti-Virus(Mi) Online Storage (kumamsushi) Access Conrroll (shobosso)
  3. 3. Mail System Framework Mail Filtering Mail Archiving Anti-Spam Anti-Virus Gateway Server Gateway Server Inbound Zone Outbound ZoneArchive Zone
  4. 4. Why We Need Load Balancer? Gateway Server Limitation of Instance Type numerous ammounts of mails beyond expectation Problem of I/O
  5. 5. Extended Gateway Server Gateway Server Gateway Server Gateway Server Gateway Server Load Balancer ・・・
  6. 6. Technology in Use 1) Linux Vritual Server (LVS) + keepalived 2) SNAT in iptables
  7. 7. 1) LVS + keepalived - Install ipvsadm - Enable packet forwarding between interfaces # /etc/sysctl.conf === net.ipv4.ip_forward = 1 === # /sbin/sysctl -p
  8. 8. 1) LVS + keepalived - Install keepalived and configure keepalive.conf # /etc/keepalived/keepalived.conf === ! Configuration File for keepalived global_defs { router_id } virtual_server 10.x.x.x 25 { delay_loop 30 lb_algo lc lb_kind NAT protocol TCP real_server 10.x.x.x 25 { weight 1 inhibit_on_failure SMTP_CHECK { connect_port 25 connect_timeout 10 retry 2 delay_before_retry 5 helo_name } } Global Deifinition Child Server Virtual Server(LB)
  9. 9. 1) LVS + keepalived - Command ・Join a child server $ ipvsadm -a -t [LB IP]:25 -r [Child Server IP]:25 -m ・Drop a child server $ ipvsadm -d -t [LB IP]:25 -r [LB IP]:25 -m ・Confirm child servers $ ipvsadm -Ln === IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.x.x.x:25 lc -> 10.x.x.x:25 Masq 1 59 17 -> 10.x.x.x:25 Masq 1 55 3
  10. 10. 2) SNAT in iptables - Problem When we use EC2 instance for Load Balancer we cannot create a SMTP connection, since SMTP trafic returns towerd EC2 default gateway not toword Load Balancer.
  11. 11. 2) SNAT in iptables # iptables -t nat -A POSTROUTING -m ipvs --vaddr 10.X.X.X -- vport 25 -j SNAT --to-source 10.X.X.X - Solution We use an SNAT function in iptables to overwrite a sender IP address. For example, a sender IP address of packet which is sent to Port 25 in Load Balancer can be overritten by the local IP address of LB.
  12. 12. Tools ・Fabric task to join/drop child server and update keepalived  $ fab join_ozmax_gw:10.x.x.x  $ fab drop_ozmax_gw:10.x.x.x  $ fab update_ozmax_lb_healthcheck ・Scrpt to automatically generate keepalived.conf in accordance with MX records ・Script to update iptables when LB instance starts ・Script to observe number of connections, network bandwidth etc. What we needed
  13. 13. Results ・We can increase a child server as much as and whenever we want ! ・We can observe how Google Apps changes their way to send mails ! ・We can also use a Self-Created LB as an internl LB in EC2!
  14. 14. A Note of Causion There is a limitation of network bandwidth dependent on instance type. m1.small -> m1.medium -> c3.xlarge
  15. 15. Extended Gateway Server Now Gateway Server Gateway Server Gateway Server Gateway Server Load Balancer × n
  16. 16. FIN