MOW2010: 1TB MySQL Database Migration and HA Infrastructure by Alex Gorbachev, Pythian

  • 2,986 views
Uploaded on

Slides from MOW2010 presentation. …

Slides from MOW2010 presentation.
On the example of this real-life project, we will demonstrate how mature MySQL database software is and what an experienced Oracle DBA can expect in MySQL land. The project included setting up highly available clustered infrastructure and disaster recovery site for MySQL.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
2,986
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
0
Comments
0
Likes
12

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide


  • - Successful growing business for more than 10 years
    - Served many customers with complex requirements/infrastructure just like yours.
    - Operate globally for 24 x 7 “always awake” services






















Transcript

  • 1. 1 TB MySQL Database Migration and HA Infrastructure Alex Gorbachev Miracle OpenWorld 2010 16 April 2010
  • 2. Alex Gorbachev • CTO, The Pythian Group • Blogger • OakTable Network member • Oracle ACE • BattleAgainstAnyGuess.com • Vice-president, Oracle RAC SIG 2 © 2009/2010 Pythian
  • 3. Why Companies Trust Pythian • Recognized Leader: • Global industry-leader in remote database administration services and consulting for Oracle, Oracle Applications, MySQL and SQL Server • Work with over 150 multinational companies such as Forbes.com, Fox Interactive media, and MDS Inc. to help manage their complex IT deployments • Expertise: • One of the world’s largest concentrations of dedicated, full-time DBA expertise. • Global Reach & Scalability: • 24/7/365 global remote support for DBA and consulting, systems administration, special projects or emergency response 3 © 2009/2010 Pythian
  • 4. Agenda • Migration • Schema, data, application code • HA infrastructure • Options available • Implemented - Heartbeat cold failover cluster • Acceptance testing • How we simulated failures • DR setup & backups • Replication between two data-centers • 1 TB on MySQL - that’s not a simple e-commerce web-site 4 © 2009/2010 Pythian
  • 5. Project profile • Document management solution • Archival & retrieve • Web front-end • Critical availability requirements • 1 TB now, growth to 2-3 TB in couple years 5 © 2009/2010 Pythian
  • 6. Migration from Oracle RDB • MySQL Migration Toolkit • RDB has a package to connect via Oracle TNS • Java • Create and review schema • Pump the data (1TB) 6 © 2009/2010 Pythian
  • 7. Schema conversion • Integer sizes mismatch - smallint, mediumint, decimal(10.2), etc... • DATE VMS => DATE or DATETIME • MEDIUMBLOB + LONGBLOB • no DEFERRABLE constraints in MySQL • character set / VARCHAR behavior (trailing space) • Sequences => AUTO INCREMENT • InnoDB storage => file per table • want Oracle tablespaces there! • page size 16 KB • No stored procedures and modules conversion 7 © 2009/2010 Pythian
  • 8. 1 TB data move • ARCHIVE part • Separate and load in advance - 800 GB • LIVE part • 200 GB - 30 hours • MySQL migration toolkit • agent mode to speed up data transfer • Speeding up • Disable binlogs • Build indexes and constraints later 8 © 2009/2010 Pythian
  • 9. Hardware • Primary data-center • 2 x IBM x3850 Servers • Each in different chassis • 4 quad core Intel XEON E7330, 2.4 GHz • 16 GB RAM • Storage IBM DS4700 Express Model 72 • Fiber-channel • RAID5 with 6 300GB disks +spare = 1.5 TB • DR data-center • 1 x IBM x3850 Servers • Same storage 9 © 2009/2010 Pythian
  • 10. Primary DC HA: Options • MySQL replication • - Can loose some data (seconds), not reliable • - Double storage requirements • + potential to scale out • DRBD replication • - Performance impact in SYNC mode • - Double storage requirements • - no scale out (primary + mirror only) • + reliable • Third-party replication 10 © 2009/2010 Pythian
  • 11. Primary DC HA: cold failover cluster • Heartbeat controls resources • Shared storage • LUN’s accessible from two servers • ext3 - mounted on active node *only* • no LVM - LVM is not clustered • Virtual IP / VIP • Up only on one node • MySQL 5.0.67 instance is running on active node • read-write data - must be InnoDB • read-only data - can be MyISAM 11 © 2009/2010 Pythian
  • 12. Heartbeat - simple clustering solution • Linux-HA.org 12 © 2009/2010 Pythian
  • 13. Heartbeat and network infrastructure Chassis 1 Chassis 2 (>?#2@'F (>?#2@'7 (?;<A&'BCD'4%&E :";"<&=&;#'(>?#2@ (?;<A&'BCD'4%&E !"#" !"#" ,(- ,(- ./)# ./)# :";"<&=&;#' ./)# :";"<&=&;# ./)# D)/%%/*&)'!1I' J&="A& 0-'1"2345'6 ,(789 0-'G D-H9 !"#"$"%&'(&)*&)'+ (?;<A&'BCD' 2)/%%/*&)' !"#"$"%&'(&)*&)'K 13 © 2009/2010 Pythian
  • 14. Heartbeat and network infrastructure • Private heartbeat network • Cross-over ethernet patch-cord • ++ Simple $100 switch - works great • --- Expensive switch and VLAN - no good • Serial link heartbeat • Redundant to ethernet • Access to RSA2 cards • Remote reset and remote power off / lights-out • Dedicated management network and management switches 14 © 2009/2010 Pythian
  • 15. Shared storage setup • Linux multipathing MPIO • 2 HBA’s per server • 2 controllers on SAN box • Added the 2nd SAN box (cheap SATA disks) • errors=panic in mount options • default is make it read-only • SANLUN’s visible from both nodes • NEVER MOUNT FILESYSTEM ON BOTH NODES!!! • ext3 is not clustered 15 © 2009/2010 Pythian
  • 16. Heartbeat and monitoring • Heartbeat 1.0 • Starts and stops resources in sequence • Failure detected during start • No resources monitoring - required Heartbeat 2.0 • Not sure if 2.0 is stable enough • mon 1.2.0 Service Monitoring Daemon • mon.wiki.kernel.org • Stable • Has number of “monitors” out-of-the-box • Can write custom monitors 16 © 2009/2010 Pythian
  • 17. Heartbeat resources Start sequence (stop is reverse) 1. Virtual / floating IP 2. SAN mount points 3. MySQL daemon / instance 4. mon 5. mon-shadow mon monitors all resource and initiates a failover mon-shadow monitors and restarts mon only mon monitors and restarts mon-shadow 17 © 2009/2010 Pythian
  • 18. “mon” monitors • msql-mysql.monitor • fping.monitor • freespace.monitor custom mount point monitor • mon.monitor On resource failure - goes to standby role. Other potential options - stop heartbeat or reboot or reset. 18 © 2009/2010 Pythian
  • 19. Improving failover • innodb_max_dirty_pages_pct=5 in my.cnf • service_startup_timeout=60 in /etc/init.d/mysql • Heartbeat resource manager retries offline 10 times • /usr/lib64/heartbeat/ResourceManager => ${HA_STOPRETRYMAX=10} • Changed to one • mysql.pid- don’t place it on shared storage • mon didn’t have timeout functionality • Hacked the perl script and added timeout 19 © 2009/2010 Pythian
  • 20. Other gotchas • Standard MySQL monitor improvement • Added insert/delete from a dummy table • Standard /etc/init.d/mysql is not POSIX compliant • mysql start returns error when MySQL is already up • mysql stop returns error when MySQL is already down • SELINUX=disabled • innodb-flush-method= O_DSYNC or O_DIRECT • ibmrsa-telnet STONITH plug-in has a bug • http://lists.community.tummy.com/pipermail/linux-ha/2008-June/ 033279.html • Heartbeat’s test suite - BasicSanityCheck 20 © 2009/2010 Pythian
  • 21. Acceptance testing - 42 individual tests (1) • Node down • power-off, halt command, cpu overload • Network tests • (ifconfig) -Heartbeat NIC down, app NIC down, management NIC down • spam serial link - cat /dev/zero >/dev/ttyS0 • pulling heartbeat cables - one at a time and together • Storage tests • freeze IO - dmsetup suspend --noflush lunmultipathproddb-01 • pull cables (one HBA and both HBA ports) • mess up mount points between two servers 21 © 2009/2010 Pythian
  • 22. Acceptance testing - 42 individual tests (2) • MySQL daemon test • MySQL dies - kill -9 {mysqld_pid} {mysql_safe_pid} • MySQL hangs - kill -STOP {mysqld_pid} • MySQL can’t connect (max connections) • “mon” tests • kill -9, kill -STOP, manual start on wrong node (including shadow) • Heartbeat • kill -9, kill -STOP • Stopping and starting • Graceful switchover between the nodes 22 © 2009/2010 Pythian
  • 23. • Split into LIVE and ARCHIVE Backup infrastructure • LIVE - InnoDB 200-500GB • ARCHIVE - MyISAM 2 TB • ARCHIVE backup - production • can lock + rsync • no LVM => no snapshot • storage snapshot is expensive • LIVE backup - on slave • FLUSH ... WITH READ LOCK • Stop slave SQL thread • LVM snapshot or RSYNC • Restore • LIVE first as a whole instance • ARCHIVE later - it’s MyISAM 23 © 2009/2010 Pythian
  • 24. Disaster recovery infrastructure 24 © 2009/2010 Pythian
  • 25. Q&A Thank you! • gorbachev@pythian.com • http://pythian.com/news/author/alex • Twitter @alexgorbachev 25 © 2009/2010 Pythian