Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Powering Your Uptime 
HAProxy Technologies 
HAProxy and Mysql 
EMEA Headquarters 
3, rue du petit Robinson 
ZAC des Metz 
...
Performance Tuning of HAProxy for Database 
Load Balancing 
HAProxy Technologies - HAProxy and Mysql - http://www.haproxy....
Agenda 
• Introduction of HAProxy Technologies 
• Inside HAProxy 
• New features in HAProxy 1.5 + focus on SSL 
• HAProxy ...
HAProxy Technologies, who are we??? 
• 8 years old company, used to be named exceliance 
• HAProxy task force: main develo...
Inside HAProxy 
• How is HAProxy implemented: 
• userland software 
• single thread / process, 100% event driven, non bloc...
Inside HAProxy - the dying process 
• This is the previous process remaining after a configuration reload. 
• A few things...
HAProxy 1.5 new features 
• SSL: one of the most complete stack out there (https://istlsfastyet.com/) 
• full IPv6: both c...
HAProxy 1.5: A focus on SSL 
• requires OpenSSL 
• many advanced features: 
• TLS/SSL version and cipher choice 
• cipher ...
Any good reason to switch to 1.5 for MySQL ?? 
Don't fix it if it works! 
• No, since 1.4 has already the most interesting...
HAProxy multi-process: advantages 
• ability to dedicate a process to a task (or application, or protocol) 
In example: 1 ...
HAProxy multi-process: limitations 
Each process has its own memory area, which means: 
• debug mode cancels multi-process...
HAProxy multi-process: configuration example 
1 # **DON'T RUN IN PRODUCTION, THERE ARE NO TIMEOUTS** 
2 global 
3 nbproc 2...
HAProxy dynamic configuration 
• As seen before, HAProxy reads and loads its configuration at startup from a raw text file...
HAProxy dynamic configuration 
• the following parameters can be updated into HAProxy's running configuration: 
• ACL cont...
HAProxy dynamic configuration 
• Examples: 
• Drain traffic to a server (only connection with persistence to the server wi...
HAProxy reports 
HAProxy embeds three main reporting features: 
1. statistic page 
In a backend or a listen section: 
stat...
HAProxy reports - Statistic page 
Useful to get a good overview of current health of the architecture. 
• this page is gen...
HAProxy reports - Statistic page 
Example of a statistic page generated by HAProxy: 
HAProxy Technologies - HAProxy and My...
HAProxy reports - Error logs 
• Error logs 
Provides an accurate message when an error occurred during the connection phas...
HAProxy reports - TCP logs 
• TCP log 
Information provided for a frontend / backend in TCP mode 
Aug 15 19:25:13 localhos...
HAProxy reports - HTTP logs 
• HTTP log 
Information provided for a frontend / backend in HTTP mode 
Aug 15 19:15:49 local...
HAProxy reports - Custom logs 
• Custom logs 
• It is possible to declare your own log format using the log-format directi...
HAProxy reports - logs: temination state 
• First character: code reporting the event which caused the end of the session....
HAProxy reports - logs: temination state 
examples 
• Combination of termination states can be used to trigger some cases ...
Weakness in MySQL client library 
The information below is independent from HAProxy. It is purely related to MySQL. 
• MyS...
Weakness in MySQL client library 
• The source ports exhaustion leads HAProxy to generate the no free ports error log 
mes...
HAProxy and MySql overview 
• HAProxy just forward TCP connections from MySQL client to MySQL server 
• HAProxy is not awa...
Hints for short live connections - Sysctls 
sysctls to take care of for performance purpose: 
• net.ipv4.ip_local_port_ran...
Hints for short live connections - CPU 
• Remember HAProxy's design: single process, single thread, 100% event driven => t...
Hints for short live connections - HAProxy 
• What's are the impact of reloading HAProxy's configuration? 
Since connectio...
Hints for persistent connections - Sysctl 
sysctls to take care of for performance purpose: 
• net.ipv4.ip_local_port_rang...
Hints for persistent connections - HAProxy 
• What are the impact of reloading HAProxy's configuration? 
Since connections...
Hints for persistent connections - HAProxy 
• Don't hesitate to calculate HAProxy's memory footprint required for the numb...
Active/active failover scenarios 
• You are responsible to failover the traffic from one HAProxy to the other one 
• Diffe...
Active/active failover scenarios - VRRP 
• Active/active scenario with VRRP: 
• if one HAProxy node fails, both VIPs will ...
• requires a layer 2 link for VRRP to work, not compatible with clouds 
• VLANs must be shared by both datacneter 
• can h...
Active/active failover scenarios - DNS 
• Active/active scenario with DNS: 
HAProxy Technologies - HAProxy and Mysql - htt...
• if one HAProxy node fails, DNS must be updated accordingly and quickly 
• HAProxy's configuration can help: 
• provide a...
Active/active failover scenarios 
• Confguration example to detect application availability per datacenter and failover 
•...
HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
Security considerations 
• filter access to your MySQL frontends: 
frontend ft_dc1 
bind 10.0.0.1:3306 
tcp-request conten...
Upcoming SlideShare
Loading in …5
×

Performance Tuning of HAProxy for Database Load Balancing - Slides

52,144 views

Published on

As a follow-up to our previous webinar on MySQL Load Balancing and HAProxy, we present this webinar on Performance Tuning of HAProxy. Tuning your HAProxy instances can significantly increase the performance of your application and decrease response times.

This webinar covers the performance tuning basics for HAProxy and explain how to take advantage of some of the new features in 1.5, released in June 2014 after 4 years of development work. We’re delighted to be joined by a well known HAProxy expert, Baptiste Assmann.

Agenda:

* New features in HAProxy 1.5
* What can HAProxy tell you about your application and your database
* OS/Network performance tuning for databases with short living connections
* Handling persistent connections
* Dynamic re-configuration
* HAProxy multi-process: advantages and limitations
* HAProxy active/active failover setups
* Security considerations

Published in: Technology
  • Be the first to comment

Performance Tuning of HAProxy for Database Load Balancing - Slides

  1. 1. Powering Your Uptime HAProxy Technologies HAProxy and Mysql EMEA Headquarters 3, rue du petit Robinson ZAC des Metz 78350 Jouy en Josas France http://www.haproxy.com HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  2. 2. Performance Tuning of HAProxy for Database Load Balancing HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  3. 3. Agenda • Introduction of HAProxy Technologies • Inside HAProxy • New features in HAProxy 1.5 + focus on SSL • HAProxy multi-process: advantages, limitations, configuration example • Dynamic re-configuration • What can HAProxy tell you about your application and your database • Weakness in MySQL client library • Hints for short live connections • Hints for persistent connections • HAProxy active/active failover setups • Security considerations HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  4. 4. HAProxy Technologies, who are we??? • 8 years old company, used to be named exceliance • HAProxy task force: main developers, main contributors • HAProxy pure player: • ALOHA Load-Balancer: the HAProxy appliance • HAPee: improved and supported HAProxy package • Prof services: expertise on your HAProxy deployments • More to come, so stay tuned!!!!! • serve customers all over the world • Website: http://www.haproxy.com/ • Twitter: https://twitter.com/haproxy_tech HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  5. 5. Inside HAProxy • How is HAProxy implemented: • userland software • single thread / process, 100% event driven, non blocking, priority based internal scheduler • HTTP / TCP reverse proxy, relies on underlying kernel to manage TCP connections • forward streams between two connections. Can't forward datagram (UDP) • get client traffic in a frontend and forward it to servers through a backend • Startup / reload procedure: 1. load, parse, and validate all configuration files 2. signal previous instance (if any) that it must release listening ports 3. bind new listening ports to accept incoming traffic 4. upon success, the reload command decides whether previous instance continues to manage existing connections or immediately quits 5. upon failure, new instance stops and previous one continues its job • Note: the previous instance will be called the dying process in the next slides HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  6. 6. Inside HAProxy - the dying process • This is the previous process remaining after a configuration reload. • A few things to know about the dying process: • it handles established connections • it runs the old configuration • can't be managed by the stats socket • does not perform any health check HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  7. 7. HAProxy 1.5 new features • SSL: one of the most complete stack out there (https://istlsfastyet.com/) • full IPv6: both client and server side, internal features • support of UNIX sockets • HTTP compression • HTTP keep alive (!!) • maps • agent check • raw TCP check • improved stick tables: behavior analysis • many new ACLs • better management of multi process • custom log format • etc... HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  8. 8. HAProxy 1.5: A focus on SSL • requires OpenSSL • many advanced features: • TLS/SSL version and cipher choice • cipher traffic on client or server side • client certificate management • SNI: SNI based routing or virtual hosting • NPN/ALPN • SSL information in HTTP headers • AES-NI CPU instructions (with compatible library) • multi-process session cache • haproxy -vv output when built with SSL support: Built with OpenSSL version : OpenSSL 1.0.1i 6 Aug 2014 Running on OpenSSL version : OpenSSL 1.0.1i 6 Aug 2014 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Special thanks to @emericBr HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  9. 9. Any good reason to switch to 1.5 for MySQL ?? Don't fix it if it works! • No, since 1.4 has already the most interesting basic features: • TCP splicing for long live session • mysql health checks • connection queueing for short live sessions • on-marked-down shutdown-sessions • A few advantages for 1.5: • improve multiprocess mode (next release will carry on improving this point) • raw tcp-check to execute arbitrary send/expect binary checks (not tested with MySQL) • agent check: an agent installed on the server can update server's weight in HAProxy • on-marked-up shutdown-backup-sessions • painless upgrade, but test in lab first... • Note: SSL offloading can't work with MySQL. MySQL SSL is a bit like IMAP's STARTTLS HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  10. 10. HAProxy multi-process: advantages • ability to dedicate a process to a task (or application, or protocol) In example: 1 process for HTTP and 1 process for MySQL • scale up: same hardware, more processing capacity by binding processes to different CPU cores • useful when massive SSL offloading processing is required key generation scales almost linearly with number of processes, but session resumption gets little gain over 3 processes HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  11. 11. HAProxy multi-process: limitations Each process has its own memory area, which means: • debug mode cancels multi-process (a single process is started) • frontend(s) and associated backend(s) must run on the same process • not compatible with peers section (stick table synchronization) • information is stored locally in each process memory area and can't be shared: • stick table + tracked counters • statistics • maxconn (queue management) • connection rate • Each HAProxy process performs its health check: • a service is probed by each process • a service can temporarly have different status in each process • managing a configuration which starts up multiple processes can be more complicated HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  12. 12. HAProxy multi-process: configuration example 1 # **DON'T RUN IN PRODUCTION, THERE ARE NO TIMEOUTS** 2 global 3 nbproc 2 4 cpu-map 1 1 5 cpu-map 2 2 6 stats socket /var/run/haproxy/socket_web process 1 7 stats socket /var/run/haproxy/socket_mysql process 2 8 9 defaults HTTP 10 bind-process 1 11 mode http 12 13 frontend f_web 14 bind 127.0.0.1:9000 15 default_backend b_web 16 17 backend b_web 18 server s 127.0.0.1:8000 19 20 defaults MYSQL 21 bind-process 2 22 mode tcp 23 24 frontend f_mysql 25 bind 127.0.0.1:3306 26 default_backend b_mysql 27 28 backend b_mysql 29 server s 192.168.10.11:3306 HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  13. 13. HAProxy dynamic configuration • As seen before, HAProxy reads and loads its configuration at startup from a raw text file • It is possible to update some settings from HAProxy's in memory configuration • don't forget to update to update the configuration text file accordingly • first, enable HAProxy stats socket in your global section (which is more a management socket nowadays) stats socket /var/run/haproxy/socket level admin • the stats socket follows the bind configuration, so it can listen on a TCP port and traffic can even be ciphered (remote management) • then (apt|yum) install socat • finaly, run: socat /var/run/haproxy/socket - <<< "help" HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  14. 14. HAProxy dynamic configuration • the following parameters can be updated into HAProxy's running configuration: • ACL content modification: add / del • map modification: set / add / del • server: weight / disable / enable • table: clear / set • frontend: maxconn global • global: maxconn / rate-limit HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  15. 15. HAProxy dynamic configuration • Examples: • Drain traffic to a server (only connection with persistence to the server will be allowed to reach it): socat /var/run/haproxy/socket_w - <<< "set weight b_web/s 0" • Disable a server (set it up for maintenance): socat /var/run/haproxy/socket_w - <<< "disable server b_web/s" • Change a frontend's maxconn value: socat /var/run/haproxy/socket_w - <<< "set maxconn frontend f_web 4000" HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  16. 16. HAProxy reports HAProxy embeds three main reporting features: 1. statistic page In a backend or a listen section: stats enable stats uri /haproxy-stats 2. logs global log 127.0.0.1:514 local1 log 127.0.0.1:514 local2 notice defaults log global option httplog # or tcplog or log-format 3. Management socket global stats socket /var/run/haproxy/socket HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  17. 17. HAProxy reports - Statistic page Useful to get a good overview of current health of the architecture. • this page is generated on the fly and is not stored on the fileystem • traffic statistic, at the time the statistic page was generated • session rates during the last second • currently establised sessions • errors / Warnings (retries and redispatches) • denied sessions (requests or responses) • frontend/backend/server health (current and past) • frontend status (OPEN or FULL (maxconn reached)) • backend status • server status and history • time since last status change / number of changes • time elapsed in DOWN status • time since last processed session HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  18. 18. HAProxy reports - Statistic page Example of a statistic page generated by HAProxy: HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  19. 19. HAProxy reports - Error logs • Error logs Provides an accurate message when an error occurred during the connection phase. Dec 3 18:27:14 localhost haproxy[6103]: 127.0.0.1:56059 [03/Dec/2012:17:35:10.380] frt/f1: Connection error during SSL handshake HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  20. 20. HAProxy reports - TCP logs • TCP log Information provided for a frontend / backend in TCP mode Aug 15 19:25:13 localhost haproxy[12002]: 192.168.10.11:42991 [15/Aug/2014:19:25:13.960] f_my b_my/m 1/0/5 58 -- 0/0/0/0/0 0/0 • It provides the following information: • client ip and port • date when the session started (milisecond) • path followed by the session inside HAProxy: frontend f_my, backend b_my, server m • timers: time spent in queues, connection time to the server, total session duration • termination state: tells why, when and source of a session error • number of sessions on HAProxy, the frontend, backend and server in use and the number of retries • number of session in queue before this one could be processed HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  21. 21. HAProxy reports - HTTP logs • HTTP log Information provided for a frontend / backend in HTTP mode Aug 15 19:15:49 localhost haproxy[10206]: 127.0.0.1:3168 [15/Aug/2014:19:15:49.413] f_web b_web/s 4/0/0/1/5 200 11986 - - ---- 1/1/0/1/0 0/0 "GET / HTTP/1.1" • It provides the same information as TCP plus the following ones: • timers: time for the client to send the whole request, server process time • HTTP status code, bytes from server to client • captured cookies (if any) • persistence flags • URL (of course!!!!) HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  22. 22. HAProxy reports - Custom logs • Custom logs • It is possible to declare your own log format using the log-format directive. • Defined as a string in which we can integrate variables • it is also possible to fetch sample from the request or response • In example, option httplog equivalent with the log-format directive: log-format %ci:%cp [%t] %ft %b/%s %Tq/%Tw/%Tc/%Tr/%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r • Same as above, but also constructs the whole URL, including the Host header field: capture request header Host len 32 log-format %ci:%cp [%t] %ft %b/%s %Tq/%Tw/%Tc/%Tr/%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs "%[capture.req.method] %[capture.req.hdr(0)]%[capture.req.uri] %[capture.req.ver]" HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  23. 23. HAProxy reports - logs: temination state • First character: code reporting the event which caused the end of the session. Main ones are: • -: no problem • C: session aborted by the client • S: session aborted by the server • c: client side timeout expired • s: server side timeout expired • P: session prematurely aborted by HAProxy • L: session locally processed by HAProxy • Second character: TCP or HTTP session state when closing occurred. Main ones are: • -: session terminated normally • R: (HTTP mode only) HAProxy was waiting for a complete valid HTTP request from the client • C: HAProxy was waiting for a connection to be established on the server • H: (HTTP mode only) HAProxy was waiting for a complete valid response headers from the server • D: the session was in the DATA phase HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  24. 24. HAProxy reports - logs: temination state examples • Combination of termination states can be used to trigger some cases like: • client or server side network outage • application or server slowness or issues • Most common termination states are: • CD: the client aborted the session when HAProxy was sending it the response • cD: HAProxy's client timeout triggered because not data has been acknowledged by the client (device turned off, network outage, etc...) • SD: the server aborted the session while sending the response. In HTTP mode, a 502 is generated • sD: HAProxy's timeout server triggered because either the server was too slow to generate a response (HTTP mode only, HAProxy returns a 504) or too slow to acknowledge data. HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  25. 25. Weakness in MySQL client library The information below is independent from HAProxy. It is purely related to MySQL. • MySQL clients don't close connections properly: • They send the layer 7 "QUIT" and shut the connection without waiting for the server to confirm the sequence • the connection is shut a TCP FIN • This is due to the way the MySQL client library manages the close Mysql Client ==> "QUIT" sequence ==> Mysql Server Mysql Client ==> FIN ==> MySQL Server Mysql Client <== FIN ACK <== MySQL Server Mysql Client ==> ACK ==> MySQL Server • A TCP FIN means the system must wait for the tcp TIME_WAIT before re-using the source port to establish a new connection to the couple [dest IP]:[dest port] • This becomes a problem when the connection rate is high (several hundreds per second), since it can lead to a tcp source port exhaustion. • by default, TIME_WAIT is 2 times MSL (Maximum Segment Lifetime), so 240s, but force at 60s in Linux kernel • TIME_WAIT main purpose is to prevent potential overlap with new connections HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  26. 26. Weakness in MySQL client library • The source ports exhaustion leads HAProxy to generate the no free ports error log message • Rule to compute the maximum connection rate to a MySQL server: Max conn/s = (ip_local_port_range[max] - ip_local_port_range[min]) / TIME_WAIT • For default values: Max conn/s = (61000 - 32768) / 60 = 470 • With recommended values: Max conn/s = (65024 - 1024) / 60 = 1066 • It is recommended to let HAProxy to manage TCP source ports. It is much more efficient than the kernel to do it. • Use multiple IPs configured on HAProxy server to increase the Max conn/s rate and let HAProxy to manage available TCP source port: server mysql1 10.0.0.1:3306 check source 10.0.0.100:1024-65024 server mysql1_bis 10.0.0.1:3306 check source 10.0.0.101:1024-65024 HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  27. 27. HAProxy and MySql overview • HAProxy just forward TCP connections from MySQL client to MySQL server • HAProxy is not aware of request nor response content: MySQL is only treated as payload • MySQL clients have two main ways of working: • short live on-demand connections established, used then closed (PHP way) • persistent connections established and maintained by the client and used as a pool of resources (the JAVA way) • more information from severalnines.com: http://www.severalnines.com/resources/clustercontrol-mysql-haproxy-load-balancing-tutorial HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  28. 28. Hints for short live connections - Sysctls sysctls to take care of for performance purpose: • net.ipv4.ip_local_port_range: local port range allowed to reach a server IP address. The wider, the best. default: "32768 61000", recommended: "1024 65024" • net.ipv4.tcp_max_syn_backlog: required to support high connection rate (and better support SYN floods attacks) default: 1024, recommended: 60000 • net.ipv4.tcp_tw_reuse: Allow early reuse of a same source port for outgoing connections. default: 0, recommended: 1 • net.core.somaxconn: per socket outstanding connect requests that have not been accepted/serviced yet by HAProxy default: 128, recommended: 1024 HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  29. 29. Hints for short live connections - CPU • Remember HAProxy's design: single process, single thread, 100% event driven => this means HAProxy performs very well with a single CPU core !!!! We can easily reach 50K HTTP connections per second with a single CPU core! • CPU is very important. Prefer the frequency over the number of cores. The bigger the CPU cache, the best too • Whenever possible, bind network interrupts to a CPU core and HAProxy to one close to it • Use cpu-map HAProxy's global directive to let it bind itself to the core you want # process-id cpu-id cpu-map 1 1 • DISABLE and uninstall IRQ Balance • avoid VMs when high connections rate is required (high means >5K/s) HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  30. 30. Hints for short live connections - HAProxy • What's are the impact of reloading HAProxy's configuration? Since connections last a very short time (less than a second), all the clients will switch to the new process quickly, so no impact when reloading the configuration. • What are the impact of running HAProxy in multiprocess mode? No. Since server's maxconn (queueing management) and health checks will be performed per process No. Since it is impossible to synchronise stick tables (some people use stick table to store the MySQL master server id) HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  31. 31. Hints for persistent connections - Sysctl sysctls to take care of for performance purpose: • net.ipv4.ip_local_port_range: local port range allowed to reach a server IP address. The wider, the best. default: "32768 61000", recommended: "1024 65534" • net.ipv4.tcp_rmem: minimum, default and maximum read buffer size (in bytes) allocated per socket • net.ipv4.tcp_wmem: minimum, default and maximum write buffer size (in bytes) allocated per socket HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  32. 32. Hints for persistent connections - HAProxy • What are the impact of reloading HAProxy's configuration? Since connections are established for a very long time, they'll be managed by the old process No connections will be killed. That said, it is currently not possible to monitor neither manage connections established on the old process • What are the impact of running HAProxy in multiprocess mode? server's maxconn (queueing management) and health checks will be performed per process it is impossible to synchronise stick tables (some people use stick table to store the MySQL master server id) • Don't use HAProxy's queueing mechanism since your connections are established for a long time, new ones may be queued for a long time as well. HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  33. 33. Hints for persistent connections - HAProxy • Don't hesitate to calculate HAProxy's memory footprint required for the number of connections you need. For each MySQL connection passing through HAProxy: • 1 read and 1 write kernel buffer for both client and server connection (sysctls tcp_rmem and tcp_wmem) •2 HAProxy buffers (tune.bufsize) memory footprint = 2 * tcp_rmem[min] + 2 * tcp_wmem[min] + 2 * tune.bufsize = 2 * 8K + 2 * 8K + 2 * 16K = 48KBytes If HAProxy has to maintain 1000 connections to a MySQL cluster, it needs around 50 MBytes of memory • The rule above is a bit simplified, add 10% of margin • if SSL offloading is enable, also add 2 SSL buffers of 64KB each... HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  34. 34. Active/active failover scenarios • You are responsible to failover the traffic from one HAProxy to the other one • Different ways exist to ensure high availability of services hosted by HAProxy • VRRP: use keepalived crossed VIPs • DNS: use at your own risks • RHI: use a routing protocols (iBGP, OSPF) to anounce availability of the server (bird, exabgp) • The choice of high availability scenario is driven by: • the main purpose to achieve • network architecture topology and limitations • network latency between locations • compatibility of the application being load-balanced • Whatever the chosen scenario, HAProxy configuration can help: • use local server first • if local servers are unavailable, then forward the traffic to the other datacenter, if available • generates HTTP redirects HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  35. 35. Active/active failover scenarios - VRRP • Active/active scenario with VRRP: • if one HAProxy node fails, both VIPs will be hosted by the remaining one HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  36. 36. • requires a layer 2 link for VRRP to work, not compatible with clouds • VLANs must be shared by both datacneter • can hardly scale over 2 DC • failover takes up to 3s HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  37. 37. Active/active failover scenarios - DNS • Active/active scenario with DNS: HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  38. 38. • if one HAProxy node fails, DNS must be updated accordingly and quickly • HAProxy's configuration can help: • provide a URL to be monitored by DNS servers • can route requests to the remaining node during DNS propagation • no links between the datacenter or cloud provider is required • scaling is linked to application capabilities and to sensitivity to latency • failover is unpredictible. 90% of users fail over quickly, no idea for the last 10% • some ISPs rewrite DNS response TTL to a long value (20m for mine) • IP1 and IP2 could be hosted by VRRP in each datacenter HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  39. 39. Active/active failover scenarios • Confguration example to detect application availability per datacenter and failover • the IP address (or DNS resolution) should also failover to the remaining datacenter frontend ft_dc1 bind 10.0.0.1:3306 acl bk_dc1_DOWN nb_srv(bk_dc1) eq 0 acl bk_dc2_UP nb_srv(bk_dc2) ge 1 # fail over to DC2 if: # - no more servers in bk_dc1 # - still some servers in bk_dc2 use_backend bk_dc2 if bk_dc1_DOWN bk_dc2_UP # default rule default_backend bk_dc1 frontend ft_dc2 bind 10.0.0.2:3306 acl bk_dc2_DOWN nb_srv(bk_dc2) eq 0 acl bk_dc1_UP nb_srv(bk_dc1) ge 1 # fail over to DC1 if: # - no more servers in bk_dc2 # - still some servers in bk_dc1 use_backend bk_dc1 if bk_dc2_DOWN bk_dc1_UP # default rule default_backend bk_dc2 backend bk_dc1 server mysql1 10.0.0.11:3306 name mysql maxconn 1000 check backend bk_dc2 server mysql2 10.0.0.12:3306 name mysql maxconn 1000 check HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  40. 40. HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document
  41. 41. Security considerations • filter access to your MySQL frontends: frontend ft_dc1 bind 10.0.0.1:3306 tcp-request content reject unless { src 10.0.0.0/24 } • limit the number of active connections per source IP: # Table definition stick-table type ip size 100k expire 30s store conn_cur tcp-request connection track-sc1 src # Allow clean known IPs to bypass the filter tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst } # Shut the new connection as long as the client has already 10 opened tcp-request connection reject if { sc1_conn_cur ge 10 } • slowloris protection: (HTTP) timeout http-request 10s HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document

×