• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
High Availability With DRBD & Heartbeat
 

High Availability With DRBD & Heartbeat

on

  • 29,970 views

This was my presentation for the June 2008 Twin Cities MySQL & PHP User Group.

This was my presentation for the June 2008 Twin Cities MySQL & PHP User Group.

Statistics

Views

Total Views
29,970
Views on SlideShare
26,536
Embed Views
3,434

Actions

Likes
14
Downloads
562
Comments
0

29 Embeds 3,434

http://www.cb1inc.com 2447
http://swik.net 425
http://www.sofee.cn 303
http://www.slideshare.net 120
http://cb1inc.com 79
http://www.planetmysql.org 8
http://static.slideshare.net 7
http://translate.googleusercontent.com 6
http://66.102.9.104 5
http://webcache.googleusercontent.com 4
http://209.85.229.132 4
http://www.e-presentations.us 3
http://planet.mysql.com 3
http://74.125.153.132 2
http://www.netvibes.com 2
http://www.xianguo.com 2
http://ambientreality.com 2
http://209.85.135.132 1
http://72.14.235.132 1
http://72.14.203.101 1
http://dojotoolkit.org 1
http://lj-toys.com 1
http://74.125.39.104 1
http://216.239.59.104 1
http://209.85.173.104 1
http://209.85.129.132 1
http://feeds.feedburner.com 1
http://localhost 1
http://web.archive.org 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-ShareAlike LicenseCC Attribution-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

High Availability With DRBD & Heartbeat High Availability With DRBD & Heartbeat Presentation Transcript

  • High Availability with DRBD & Heartbeat Chris Barber http://www.cb1inc.com
  • What is DRBD? ● Distributed Replicated Block Device ● RAID 1 mirror across network ● Realtime replications ● Linux-only kernel module ● 2 nodes – Primary/Secondary ● Single/Dual primary mode ● Open source, free
  • Visual Overview Primary Server Secondary Server Service (MySQL, Apache, etc) Service (MySQL, Apache, etc) File system File system OS OS Buffer cache Buffer cache DRBD DRBD Disk TCP/IP TCP/IP Disk Scheduler Scheduler Disk Driver NIC Driver NIC Driver Disk Driver Network Card Network Card Disk Disk Network Card Network Card Original chart: http://www.drbd.org/users-guide/ch-fundamentals.html
  • Uses ● Database Server( MySQL) ● Web Server (Apache) ● File Server (Samba) ● FTP ● DNS ● Email (Postfix) ● Source Repository (Subversion)
  • My Setup ● App 1 ● App 2 ● AMD Opteron 180 ● AMD Sempron 2800+ 2.4GHz dual-core ● 3GB RAM ● 4GB RAM ● 1 x 250 IDE Hard ● 2 x 250GB SATA Drive Software RAID 1 ● 2 x Gigabit NICs ● 2 x Gigabit NICs
  • My Setup App 1 App 2 File Server
  • My Setup Gigabit Crossover Cable
  • Partitions ● 250GB total ● 20GB Operating system (/) ● 4GB swap ● 226GB DRBD (/mnt/data)
  • /mnt/data ● /mnt/data/apache2 – Apache conf, SSL certs, htpasswd ● /mnt/data/mail – Mail directory (Postfix/CourierIMAP) ● /mnt/data/mysql – MySQL conf, database files, binlog ● /mnt/data/svn – Subversion repository ● /mnt/data/www - Websites
  • DRBD Install Overview ● Download source ● Compile driver ● modprobe drbd ● Create /etc/drbd.conf ● Create meta disk - drbdadm create-md r0 ● Bring up device - drbdadm up r0 ● Become primary – drbdadm -- --overwrite-data-of-peer primary r0 ● mkfs.ext3 /dev/drbd0 ● mount /dev/drbd0 /mnt/data
  • /etc/drbd.conf net { sndbuf-size 512k; timeout 60; # 6 seconds (unit = 0.1 seconds) connect-int 10; # 10 seconds (unit = 1 second) global { ping-int 10; # 10 seconds (unit = 1 second) usage-count no; ping-timeout 5; # 500 ms (unit = 0.1 seconds) } max-buffers 8000; max-epoch-size 8000; common { cram-hmac-alg quot;sha1quot;; protocol C; shared-secret quot;secretquot;; } syncer { } rate 30M; al-extents 1801; resource r0 { } on app1 { disk /dev/md2; startup { address 10.10.10.21:7788; wfc-timeout 0; device /dev/drbd0; degr-wfc-timeout 15; meta-disk internal; } } disk { on app2 { on-io-error detach; disk /dev/sda3; # fencing resource-and-stonith; address 10.10.10.31:7788; } device /dev/drbd0; meta-disk internal; } }
  • Checking DRBD ● cat /proc/drbd root@app1:/mnt/data# cat /proc/drbd version: 8.0.11 (api:86/proto:86) GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12 11:56:43 0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r--- ns:237740648 nr:7160 dw:19725440 dr:242166751 al:2274 bm:17504 lo:0 pe:0 ua:0 ap:0 resync: used:0/31 hits:13841287 misses:13628 starving:0 dirty:0 changed:13628 act_log: used:0/1801 hits:4927296 misses:2427 starving:0 dirty:153 changed:2274
  • Heartbeat ● Exchange heartbeats ● When the heartbeat fails, other begins takeover ● Use 2 NICs (try to prevent split brain) ● Uses LSB init scripts ● Version 1 – Simple as cake ● Version 2 – GUI, allows more complex situations
  • Heartbeat Installation Overview ● Install heartbeat (apt-get install heartbeat) ● 3 conf files /etc/ha.d/ha.cf keepalive 2 /etc/ha.d/haresources deadtime 15 app1 warntime 5 IPaddr::192.168.1.211/24/eth0 initdead 30 IPaddr::192.168.1.212/24/eth0 autojoin none drbddisk::r0 bcast eth1 Filesystem::/dev/drbd0::/mnt/data::ext3 mcast eth0 239.0.0.42 694 1 0 bind9 auto_failback off postfix node app1 app2 courier-imap logfacility local0 courier-imap-ssl mysql apache2 /etc/ha.d/authkeys samba auth 1 1 sha1 secret
  • Finalizing Heartbeat Install ● Stop services – /etc/init.d/apache2 stop ● Disable services (Ubuntu/Debian) – update-rc.d -f apache2 remove ● Disable mounting of DRBD drive (/etc/fstab) – /dev/drbd0 /mnt/data ext3 noauto 0 0
  • Running Heartbeat ● /etc/init.d/heartbeat start ● /usr/lib/heartbeat/hb_takeover ● /usr/lib/heartbeat/hb_standby
  • Samba [global] log file = /var/log/samba/log.%m passwd chat = *EntersnewsUNIXspassword:* %nn *RetypesnewsUNIXspassword:* %nn [data] *passwd:*passwordsupdatedssuccessfully* . writeable = yes socket options = TCP_NODELAY path = /mnt/data obey pam restrictions = yes force group = root encrypt passwords = true force user = root passwd program = /usr/bin/passwd %u create mode = 664 passdb backend = tdbsam directory mode = 775 dns proxy = no server string = %h server (Samba, Ubuntu) [www] invalid users = root force user = www-data workgroup = WORKGROUP writeable = yes os level = 20 create mode = 664 auto services = data path = /mnt/data/www security = user directory mode = 775 syslog = 0 force group = www-data panic action = /usr/share/samba/panic-action %d max log size = 1000 # guest account = nobody
  • Split Brain ● Both servers are primary ● Which one is up-to-date? ● How do you prevent split brain?
  • Stonith apt-get install stonith Shoot The Other Node In The Head ● Home made stonith device ● Connects serial port to other server's reset switch ● ~$40 in Radio Shack parts to build 2 of them ● http://www.scl.co.uk/rcd_serial/README.rcd_serial
  • MySQL ● For maximum performance, use RAID 10 with battery backed cache RAID controller ● Write transactions to disk so they get replicated – innodb_flush_log_at_trx_commit=1 – sync_binlog=1 ● More info: – http://www.mysqlperformanceblog.com/2008/06/02/how-much-overhead-drdb-could-cause/
  • Demo ● Reboot app1 ● App2 become primary ● App1 comes online, re-syncs ● Turn off app1's network ● DRBD on app2 loses connection with app1 ● Run home and enable app1's networking :)
  • More Info ● http://www.drbd.org/ ● http://www.cb1inc.com/taxonomy/term/70
  • Questions? Thank you!