Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Confidential
Load Balancing with HAProxy and MySQL
!
!
July 9, 2014
Alex Yu

alex@severalnines.com
Copyright Severalnines AB
Webinar Housekeeping
!This webinar is being recorded
!A link to the recording & slides will be p...
Copyright Severalnines AB
Agenda
!What Is HAProxy?
!SQL Load Balancing with HAProxy
! Failure detection with MySQL
!Fault ...
Copyright Severalnines AB
Why Load Balance?
!Optimising server utilisation
! Queuing and throttling
!Maximise availability...
!Reliable High Performance TCP/HTTP Load Balancer
! v1.5 stable released June 2014
! Native SSL support, IPv6, full HTTP k...
Copyright Severalnines AB
SQL Load Balancing for MySQL
!Configuration file
! Sections; global, defaults, frontend, backend...
Copyright Severalnines AB
SQL Load Balancing for MySQL (cont.)
!defaults
7
MySQL
[WSREP]
Galera Replication (Synchronous)
...
Copyright Severalnines AB
SQL Load Balancing for MySQL (cont.)
!listen (frontend + backend)
8
listen s9s1_33306_defaul_LB
...
Copyright Severalnines AB
SQL Load Balancing for MySQL (cont.)
!listen (stats ui)
9
userlist STATSUSERS
group admin users ...
Copyright Severalnines AB
10
!Session rate
! number of new connections per second
!Sessions
! current number of sessions/c...
Copyright Severalnines AB
MySQL Health Checks
11
!Built-in MySQL health check
! option mysql-check user <username>
! Sends...
Copyright Severalnines AB
MySQL Health Checks (cont.)
12
!Galera Node
! Check 1: Get node state
! SYNCED or DONOR
! Check ...
Copyright Severalnines AB
MySQL Health Checks (cont.)
13
!xinetd service
! mysqlchk.sh (shell script)
! spams syslog by de...
Copyright Severalnines AB
Network Tuning
!TCP tuning
14
#/etc/sysctl.conf
net.ipv4.ip_nonlocal_bind=1
net.ipv4.tcp_tw_reus...
Copyright Severalnines AB
Fault Tolerance for HAProxy
15
Client
HAProxy 1
MySQL
 MySQL
 MySQL

IP: 10.10.10.20
Copyright Severalnines AB
Fault Tolerance for HAProxy (cont.)
!Active/Passive IP failover
! keepalived
! VRRP protocol pro...
Copyright Severalnines AB
Fault Tolerance for HAProxy (cont.)
!keepalived configuration file
17
#/etc/keepalived/keepalive...
Copyright Severalnines AB
ClusterControl HAProxy + Keepalived
!Install HAProxy with our admin scripts
! http://www.several...
Copyright Severalnines AB
ClusterControl HAProxy + Keepalived (cont.)
!Deployment and configuration automated from Cluster...
Copyright Severalnines AB
ClusterControl HAProxy + Keepalived (cont.)
20
Copyright Severalnines AB
Deployment Scenarios
21
!MySQL Cluster / NDB Engine
! Active/Active, Shared nothing architecture...
Copyright Severalnines AB
Deployment Scenarios (cont.)
22
NDB Data
Node
NDB Data
Node
!MySQL Cluster / NDB Engine
! Active...
Copyright Severalnines AB
Deployment Scenarios (cont.)
23
NDB Data
Node
NDB Data
Node
!MySQL Cluster / NDB Engine
! Active...
Copyright Severalnines AB
Deployment Scenarios (cont.)
24
MySQL Slave
MySQL
Master
!MySQL Replication Master/Slave
! 1 Mas...
Copyright Severalnines AB
Deployment Scenarios (cont.)
25
!Galera Cluster
! Synchronous Replication, Read & Write to any N...
Copyright Severalnines AB
Deployment Scenarios (cont.)
26
!Galera Cluster
! Synchronous Replication, Read & Write to all N...
Copyright Severalnines AB
Deployment Scenarios (cont.)
27
MySQL
[WSREP]
Galera Replication (Synchronous)
LB
MySQL
[WSREP]
...
Copyright Severalnines AB
Alternatives
!MySQL’s Connector/J JDBC driver
!jdbc:mysql:loadbalance://host-1,host-2,…
!PHP mys...
Copyright Severalnines AB
Thank You!
!Tutorial – MySQL Load Balancing with HAProxy
! http://www.severalnines.com/resources...
Upcoming SlideShare
Loading in …5
×

How To Set Up SQL Load Balancing with HAProxy - Slides

17,468 views

Published on



We continuously see great interest in MySQL load balancing and HAProxy, so we thought it was about time we organised a live webinar on the topic! Here is the replay of that webinar!

As most of you will know, database clusters and load balancing go hand in hand.

Once your data is distributed and replicated across multiple database nodes, a load balancing mechanism helps distribute database requests, and gives applications a single database endpoint to connect to.

Instance failures or maintenance operations like node additions/removals, reconfigurations or version upgrades can be masked behind a load balancer. This provides an efficient way of isolating changes in the database layer from the rest of the infrastructure.

In this webinar, we cover the concepts around the popular open-source HAProxy load balancer, and show you how to use it with your SQL-based database clusters. We also discuss HA strategies for HAProxy with Keepalived and Virtual IP.

Agenda:

* What is HAProxy?
* SQL Load balancing for MySQL
* Failure detection using MySQL health checks
* High Availability with Keepalived and Virtual IP
* Use cases: MySQL Cluster, Galera Cluster and MySQL Replication
* Alternative methods: Database drivers with inbuilt cluster support, MySQL proxy, MaxScale, ProxySQL

Published in: Technology
  • Be the first to comment

How To Set Up SQL Load Balancing with HAProxy - Slides

  1. 1. Confidential Load Balancing with HAProxy and MySQL ! ! July 9, 2014 Alex Yu
 alex@severalnines.com
  2. 2. Copyright Severalnines AB Webinar Housekeeping !This webinar is being recorded !A link to the recording & slides will be posted on severalnines.com !We welcome questions: enter questions into the chat box and we will respond at the end of the presentation !Think of something later? ! Email Severalnines at info@severalnines.com 2
  3. 3. Copyright Severalnines AB Agenda !What Is HAProxy? !SQL Load Balancing with HAProxy ! Failure detection with MySQL !Fault Tolerance with HAProxy ! Active/Passive setup !Deployment Scenarios ! MySQL Cluster, Galera Cluster and MySQL Replication !Load Balancing Alternatives 3
  4. 4. Copyright Severalnines AB Why Load Balance? !Optimising server utilisation ! Queuing and throttling !Maximise availability ! No SPOF !Maximise throughput ! Distribute workload across a set of servers !Scalability ! Site traffic grows ! Add DB servers to provide more throughput 4 LB DB DB DB …
  5. 5. !Reliable High Performance TCP/HTTP Load Balancer ! v1.5 stable released June 2014 ! Native SSL support, IPv6, full HTTP keepalive, HTTP Compression … ! v1.4 most deployed and stable version, released Feb 2010 ! client-side keep alive, TCP speedups, source base stickiness … ! Single Process Event-Driven. Can easily saturate 10Gb link with a single instance ! 108 000 session/s, http://www.haproxy.org/10g.html (2009) !Reverse Proxy ! load balancing, scaling out, failover (health checks) !High traffic sites like ! reddit, DISQUS, GitHub, Imgur, StackOverflow/Server Fault, Instagram … Copyright Severalnines AB HAProxy? 5
  6. 6. Copyright Severalnines AB SQL Load Balancing for MySQL !Configuration file ! Sections; global, defaults, frontend, backend, listen ! /etc/haproxy/haproxy.cfg !Global (process wide) 6 MySQL [WSREP] Galera Replication (Synchronous) LB MySQL [WSREP] MySQL [WSREP] R/W R/W R/W Client Client Client global
 log 127.0.0.1 local0 pidfile /var/run/haproxy.pid daemon user nobody group nobody stats socket /tmp/haproxy.socket … node HAProxy_1 description HAProxy 1 maxconn 40000 spread-checks 3 quiet
  7. 7. Copyright Severalnines AB SQL Load Balancing for MySQL (cont.) !defaults 7 MySQL [WSREP] Galera Replication (Synchronous) LB MySQL [WSREP] MySQL [WSREP] R/W R/W R/W Client Client Client defaults log global mode tcp maxconn 40000 option dontlognull option tcp-smart-accept option tcp-smart-connect option redispatch retries 3 timeout queue 3500ms timeout connect 3500ms timeout client 120s timeout server 120s
  8. 8. Copyright Severalnines AB SQL Load Balancing for MySQL (cont.) !listen (frontend + backend) 8 listen s9s1_33306_defaul_LB mode tcp
 bind *:33306 timeout client 60000ms timeout server 60000ms balance leastconn option httpchk #option allbackups default-server port 9200 inter 2s downinter 5s rise 3 fall 2 slowstart 60s maxconn 256 maxqueue 128 weight 100 server db1 10.0.3.70:3306 check server db2 10.0.3.60:3306 check server db3 10.0.3.50:3306 check
  9. 9. Copyright Severalnines AB SQL Load Balancing for MySQL (cont.) !listen (stats ui) 9 userlist STATSUSERS group admin users admin user admin insecure-password admin ! listen admin_page 0.0.0.0:9600 mode http stats enable stats refresh 60s stats uri / acl AuthOkay_ReadOnly http_auth(STATSUSERS) acl AuthOkay_Admin http_auth_group(STATSUSERS) admin stats http-request auth realm admin_page unless AuthOkay_ReadOnly
  10. 10. Copyright Severalnines AB 10 !Session rate ! number of new connections per second !Sessions ! current number of sessions/connections !LastChk ! health check !Wght ! server weight
  11. 11. Copyright Severalnines AB MySQL Health Checks 11 !Built-in MySQL health check ! option mysql-check user <username> ! Sends two MySQL packages ! Authentication and Quit ! No errors unless MySQL server is down !Complete HTTP Request ! option httpchk ! Response: HTTP/1.1 200 OK ! Check Host and/or MySQL server state ! Galera Node State MySQL [WSREP] Galera Replication (Synchronous) LB MySQL [WSREP] MySQL [WSREP] R/W R/W R/W Client Client Client
  12. 12. Copyright Severalnines AB MySQL Health Checks (cont.) 12 !Galera Node ! Check 1: Get node state ! SYNCED or DONOR ! Check 2: If Node is a “DONOR” ! Get the SST method ! xtrabackup ! Return “200 OK” ! rsync or mysqldump ! Return “503 Service Unavailable” Galera Node State
  13. 13. Copyright Severalnines AB MySQL Health Checks (cont.) 13 !xinetd service ! mysqlchk.sh (shell script) ! spams syslog by default ! log_on_success = #/etc/xinetd.d/mysqlchk # default: on # description: mysqlchk service mysqlchk { flags = REUSE socket_type = stream port = 9200 wait = no user = nobody server = /usr/local/bin/mysqlchk.sh log_on_failure += USERID disable = no #only_from = 0.0.0.0/0 per_source = UNLIMITED }
  14. 14. Copyright Severalnines AB Network Tuning !TCP tuning 14 #/etc/sysctl.conf net.ipv4.ip_nonlocal_bind=1 net.ipv4.tcp_tw_reuse=1 # reuse TIME-WAIT sockets net.ipv4.ip_local_port_range=1024 65023 # increase max num of ports net.ipv4.tcp_max_syn_backlog=40000 #Increase the number of outstanding syn requests net.ipv4.tcp_max_tw_buckets=400000 # Maximal number of timewait sockets net.ipv4.tcp_max_orphans=60000 net.ipv4.tcp_max_syn_backlog=40000 # TCP SYN Flood Protection net.ipv4.tcp_synack_retries=3 # TCP SYN Flood Protection net.core.somaxconn=40000 # burst connection rate net.ipv4.tcp_fin_timeout=5 # how long to keep sockets in FIN-WAIT-2
  15. 15. Copyright Severalnines AB Fault Tolerance for HAProxy 15 Client HAProxy 1 MySQL
 MySQL
 MySQL
 IP: 10.10.10.20
  16. 16. Copyright Severalnines AB Fault Tolerance for HAProxy (cont.) !Active/Passive IP failover ! keepalived ! VRRP protocol provides router/IP failover !VIP, “floating IP”: 10.10.10.10 ! net.ipv4.ip_nonlocal_bind=1 !Master - HAProxy 1 ! IP: 10.10.10.20 !Backup - HAProxy 2 ! IP: 10.10.10.30 16
  17. 17. Copyright Severalnines AB Fault Tolerance for HAProxy (cont.) !keepalived configuration file 17 #/etc/keepalived/keepalived.conf vrrp_script chk_haproxy { script "killall -0 haproxy" # verify the pid existence interval 2 # check every 2 seconds weight 2 # adjust priority by this weight } vrrp_instance VI_1 { interface eth0 state MASTER virtual_router_id 51 # ID for this route priority 101 # 101 on master
 # 100 on backup virtual_ipaddress { 10.10.10.10 # the virtual IP } track_script { chk_haproxy } }
  18. 18. Copyright Severalnines AB ClusterControl HAProxy + Keepalived !Install HAProxy with our admin scripts ! http://www.severalnines.com/blog/using-haproxy-database-clusters 18 $ git clone https://github.com/severalnines/s9s-admin.git $ cd s9s-admin/cluster/ # install haproxy1 on 10.10.10.20 listening on 33306 $ ./s9s_haproxy --install -i 1 -n <ip db1>,<ip db2>,<ip db3> -h10.10.10.20 # install haproxy2 on 10.10.10.30 listening on 33306 $ ./s9s_haproxy --install -i 1 -n <ip db1>,<ip db2>,<ip db3> -h10.10.10.30 # scale out db nodes on haproxy1 $ ./s9s_haproxy --add-node -i 1 -h 10.10.10.20 -n <ip db4> $ ./s9s_haproxy --add-node -i 1 -h 10.10.10.20 -n <ip db5> # scale out db nodes on haproxy2 $ ./s9s_haproxy --add-node -i 1 -h 10.10.10.30 -n <ip db4> $ ./s9s_haproxy --add-node -i 1 -h 10.10.10.30 -n <ip db5>
  19. 19. Copyright Severalnines AB ClusterControl HAProxy + Keepalived (cont.) !Deployment and configuration automated from ClusterControl 19
  20. 20. Copyright Severalnines AB ClusterControl HAProxy + Keepalived (cont.) 20
  21. 21. Copyright Severalnines AB Deployment Scenarios 21 !MySQL Cluster / NDB Engine ! Active/Active, Shared nothing architecture ! SQL, Data and MGMT Nodes ! HAProxy co-located with MGMT Nodes ! MGMT is a light weight process ! Re-use “idle” servers ! Connections originate from HAProxy server ! Additional latency ! App Server <-> HAProxy <-> SQL Node <-> Data Node NDB Data Node NDB Data Node (MGMT) HAProxy (MGMT) HAProxy SQL Node
 MySQL Server SQL Node
 MySQL Server
  22. 22. Copyright Severalnines AB Deployment Scenarios (cont.) 22 NDB Data Node NDB Data Node !MySQL Cluster / NDB Engine ! Active/Active, Shared nothing architecture ! SQL, Data and MGMT Nodes ! HAProxy co-located with App servers ! No extra latency ! Extra load on app servers ! Health check loads on SQL nodes from all LBs ! Alt: MySQL Connector/J’s JDBC driver ! jdbc:mysql:loadbalance://host-1,host-2,…host-n/database? loadBalanceBlacklistTimeout=5000 ! Config changes propagate to all app servers (App) HAProxy (App) HAProxy SQL Node
 MySQL Server SQL Node
 MySQL Server
  23. 23. Copyright Severalnines AB Deployment Scenarios (cont.) 23 NDB Data Node NDB Data Node !MySQL Cluster / NDB Engine ! Active/Active, Shared nothing architecture ! SQL, Data and MGMT Nodes ! HAProxy co-located with SQL nodes ! No extra latency ! Extra load on SQL nodes ! Do not use JDBC load balancing ! VIP (App) (App) SQL Node
 MySQL Server SQL Node
 MySQL Server HAProxy HAProxy …
  24. 24. Copyright Severalnines AB Deployment Scenarios (cont.) 24 MySQL Slave MySQL Master !MySQL Replication Master/Slave ! 1 Master for Writes ! N Slaves for Reads ! Separate front and backends configs for Master and Slaves !ex: backend configuration for Slaves HAProxy MySQL Slave MySQL Slave Writes Reads frontend front_reads mode tcp bind 192.168.100.110:3316 default_backend cluster_reads 
 backend cluster_reads mode tcp balance roundrobin server Slave1 10.10.10.30:3306 check server Slave2 10.10.10.40:3306 check
  25. 25. Copyright Severalnines AB Deployment Scenarios (cont.) 25 !Galera Cluster ! Synchronous Replication, Read & Write to any Node ! Cluster wide optimistic locking ! Higher probability for “deadlocks” ! Pessimistic locking on the Node MySQL [WSREP] Galera Replication (Synchronous) LB MySQL [WSREP] MySQL [WSREP] R/W R/W R/W Client Client Client
  26. 26. Copyright Severalnines AB Deployment Scenarios (cont.) 26 !Galera Cluster ! Synchronous Replication, Read & Write to all Nodes ! Cluster wide optimistic locking ! Higher probability for “deadlocks” ! Pessimistic locking on the Node MySQL [WSREP] Galera Replication (Synchronous) LB MySQL [WSREP] MySQL [WSREP] R/W R/W R/W Client Client Client
  27. 27. Copyright Severalnines AB Deployment Scenarios (cont.) 27 MySQL [WSREP] Galera Replication (Synchronous) LB MySQL [WSREP] MySQL [WSREP] R/W R/W R/W Client Client Client listen s9s1_33307_LB1 bind *:33307 mode tcp timeout client 60000ms timeout server 60000ms balance leastconn option httpchk default-server port 9200 inter 2s <…> server db1 10.0.3.70:3306 check server db2 10.0.3.60:3306 check backup server db3 10.0.3.50:3306 check backup !Handle “dead locks” ! Write to only one Node - Use HAProxy ! Expect deadlocks and make sure applications handles it properly. Retry the transaction. ! wsrep_retry_autocommit=1 ! Changes to the App code to minimise data hotspots.
  28. 28. Copyright Severalnines AB Alternatives !MySQL’s Connector/J JDBC driver !jdbc:mysql:loadbalance://host-1,host-2,… !PHP mysqlnd ! http://www.php.net/manual/en/intro.mysqlnd-ms.php !Oracle’s MySQL Proxy 0.84 ! http://dev.mysql.com/downloads/mysql-proxy/ !SkySQL’s MaxScale 1.0.0. beta ! https://github.com/skysql/MaxScale !ProxySQL ! https://github.com/renecannao/proxysql !Pen 0.24.0 ! http://morestuff.siag.nu/category/pen/ 28
  29. 29. Copyright Severalnines AB Thank You! !Tutorial – MySQL Load Balancing with HAProxy ! http://www.severalnines.com/resources/clustercontrol-mysql-haproxy-load- balancing-tutorial !Severalnines Blog ! www.severalnines.com/blog !s9s-admin tools ! https://github.com/severalnines/s9s-admin !More Questions? Contact us at: ! info@severalnines.com 29

×