State of the art of MySQL replication and clustering

1,706 views

Published on

State of the art of MySQL replication and clustering

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,706
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
49
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

State of the art of MySQL replication and clustering

  1. 1. State of the Art in MySQL HA and Replication Robert Hodges, CEO Giuseppe Maxia, Director of QA ©Continuent 2012.Tuesday, November 13, 12 1
  2. 2. Introducing Continuent • The leading provider of clustering and replication for open source DBMS • Our Product: Continuent Tungsten • Clustering - Commercial-grade HA, performance scaling and data management for MySQL • Replication - Flexible, high-performance data movement ©Continuent 2012 2Tuesday, November 13, 12 2
  3. 3. Quick Continuent Facts • Largest Tungsten clustering installation processes about 500M transactions each day • Tungsten Replicator was application of the year at the 2011 MySQL User Conference • Continuent is well known in the MySQL community for its clustering and replication • Over 90 customers currently deployed ©Continuent 2012 3Tuesday, November 13, 12 3
  4. 4. A Few Continuent Customers ©Continuent 2012 4Tuesday, November 13, 12 4
  5. 5. Future-Proof HA and Replication for MySQL ©Continuent 2012 5Tuesday, November 13, 12 5
  6. 6. What is Future-Proo!ng? • Future-proo!ng = creating systems that last while parts change and improve • MySQL is not losing out to other solutions for data management • The real problem is using MySQL as a building block with other technologies that are changing rapidly ©Continuent 2012 6Tuesday, November 13, 12 6
  7. 7. But Wait... Isn’t MySQL Dead?? MySQL “You were so open” Born May 25, 1995 Died 2008, 2009, 2010, 2011, 2012, ... ©Continuent 2012 7Tuesday, November 13, 12 7
  8. 8. Oracle/MySQL 5.6 Replication (On the Way) • Global transaction IDs • Parallel slave threads • Crash safe slaves • Optimized row updates • Time delay replication • Informational log events ©Continuent 2012 8Tuesday, November 13, 12 8
  9. 9. Galera Synchronous Clusters (New Arrival in 2012) • Multi-master for InnoDB • Synchronous replication avoids data loss • Automatic cluster membership management • Simple node provisioning model • Based on innovative state-machine work from Fernando Pedone ©Continuent 2012 9Tuesday, November 13, 12 9
  10. 10. Tungsten Async Replication (2011 Belle of the Ball) • All 5.6 features including parallel replication for MySQL 5.0 onwards • Optimized support for failover • Programmable transaction !ltering • Backup/restore integration • Multi-master, fan-in, and star replication • Replication to/from Oracle and batch loading of data warehouses • Replication to MongoDB ©Continuent 2012 10Tuesday, November 13, 12 10
  11. 11. Not Bad for a Dying Market THE$TOTAL$MARKET$FOR$MYSQL,$NOSQL$$ My/NewSQL, AND$NEWSQL$$ 5% NewSQL,2% 1000" 900" NoSQL,5% 800" 700" 600" Text 500" MySQL,88% 400" 300" 200" 100" 0" (451Group) 2011" 2012" 2013" 2014" 2015" ©Continuent 2012 11Tuesday, November 13, 12 11
  12. 12. MySQL Is Part of Bigger Trends • Cloud and Big Data investment dwarf the MySQL marketplace ©Continuent 2012 12Tuesday, November 13, 12 12
  13. 13. 21st Century CRM/Call Center Fat Client CRM Web CRM Web Accounting Application Application Call Session Data Online Legacy Transaction Hadoop Oracle Processing Analytics On-Premise Amazon Web Services Telephony Telephony Data Streams Data Streams ©Continuent 2012 13Tuesday, November 13, 12 13
  14. 14. Revolutionary Challenges for MySQL • Failures in unstable cloud environments • Zero-downtime maintenance • Rapidly growing data volumes, esp. in cloud • Distributing data to geographical regions • Integration between MySQL, NoSQL, commercial RDBMS • Supplying real-time analytics • Technology upgrade/replacement ©Continuent 2012 14Tuesday, November 13, 12 14
  15. 15. Future-Proofed HA and Replication Data Fabric Encapsulated, fault-tolerant, horizontally scalable, globally accessible, integrated data ©Continuent 2012 15Tuesday, November 13, 12 15
  16. 16. Data Fabric Design Patterns • Fabric Connector • Fault-Tolerant Data Service • Sharded Data Service • Multi-Site Data Service • Real-Time Data Bridge ©Continuent 2012 16Tuesday, November 13, 12 16
  17. 17. 1. Fabric Connector • Single point of entry for applications • Service catalog locates data within fabric • Transparent connectivity, multiple protocols • Security, auditing, performance management Fabric Connector Application Application Stack Stack Fabric Connector ©Continuent 2012 17Tuesday, November 13, 12 17
  18. 18. 2. Fault-Tolerant Data Service • Encapsulate redundant database copies • Sync and async clustering models • Protect against local DBMS failure • Rolling maintenance of replicas Master/ Multi- Slave Master ©Continuent 2012 18Tuesday, November 13, 12 18
  19. 19. 3. Sharded Data Service • Partition datasets based on RAM / storage speed / resident set size • Multiple buckets per server • Look-up methods for locating data in buckets • Re-sharding / migration for high growth ©Continuent 2012 19Tuesday, November 13, 12 19
  20. 20. 4. Multi-Site Data Service • Robust protection against region/site failures • Geographic distribution of data • Primary/DR vs. multi-master • Eventually consistent replication for SQL Amazon East EU West US East APAC Tokyo Rackspace DFW ©Continuent 2012 20Tuesday, November 13, 12 20
  21. 21. 5. Real-Time Data Bridge • Replicate from logs in real-time • Enable apps to get data from one data source • Heterogeneous transfer and transformation • High performance, low application impact Bridge Bridge Bridge ©Continuent 2012 21Tuesday, November 13, 12 21
  22. 22. Future-Proofed CRM/Call Center Data Fat Client CRM Web CRM Web Telephony Telephony Accounting Application Application Data Streams Data Streams Fabric Fabric Fabric Fabric Fabric Connector Connector Connector Connector Connector Bridge Bridge Call Session Bridge Bridge Data Legacy Oracle OLTP Primary Bridge Hadoop Analytics OLTP DR Data Marts ©Continuent 2012 22Tuesday, November 13, 12 22
  23. 23. Future-Proofed CRM/Call Center Data Fat Client CRM Web CRM Web Telephony Telephony Accounting Application Application Data Streams Data Streams Fabric Fabric Fabric Fabric Fabric Connector Connector Connector Connector Connector Bridge Bridge Call Session Bridge Bridge Data Legacy Oracle OLTP Primary Bridge Hadoop Analytics OLTP DR Data Marts ©Continuent 2012 23Tuesday, November 13, 12 23
  24. 24. Enabling Data Fabric With Tungsten ©Continuent 2012 24Tuesday, November 13, 12 24
  25. 25. Tungsten Fault Tolerant Data Service GonzoPortal.com Tungsten clusters combine o"- the-shelf open source DBMS servers into data services with: apache /php • 24x7 data access • Scaling of load on replicas Connector Connector • Simple management commands ...without app changes or data migration Amazon US West ©Continuent 2012 25Tuesday, November 13, 12 25
  26. 26. Doesn’t Replication Do This Already? Application Application Libmysqlclient.a Libmysqlclient.a Db2 Db1 Db3 Replicator Replicator Replicator Slave Master Slave ©Continuent 2012 26Tuesday, November 13, 12 26
  27. 27. Replication Limitations Replication alone cannot enable constant access to data... • What happens if the master fails? • What happens if a slave is lagging too far behind to be usable? • How do I maintain the DBMS without stopping apps? ©Continuent 2012 27Tuesday, November 13, 12 27
  28. 28. From Replicating to a Data Service Application Application Tungsten Connector Tungsten Connector Monitoring and Control Monitoring and Control Db2 Db1 Db3 Manager Manager Manager Replicator Replicator Replicator Slave Master Slave Data Service: sjc ©Continuent 2012 28Tuesday, November 13, 12 28
  29. 29. Basic Operations • Switch -- Promote a slave node to master • Shun -- Kick node out of cluster • Welcome -- Return node to cluster • Backup -- Run a backup • Restore -- [Re-]provision a node • Recover -- Return a broken node to slave • Heartbeat -- Check cluster liveness ©Continuent 2012 29Tuesday, November 13, 12 29
  30. 30. Automatic Failover Application Tungsten Connector ©Continuent 2012 30Tuesday, November 13, 12 30
  31. 31. Automatic Failover Application Tungsten Connector 2. Halt master connections 1. Detect non- responsive master ©Continuent 2012 31Tuesday, November 13, 12 31
  32. 32. Automatic Failover Application Tungsten Connector 3. Select most advanced slave and wait for pending transactions to apply ©Continuent 2012 32Tuesday, November 13, 12 32
  33. 33. Automatic Failover Application Tungsten Connector 4. Promote to master , reconfigure slave(s), and re-enable connections ©Continuent 2012 33Tuesday, November 13, 12 33
  34. 34. Automatic Failover Application Tungsten Connector 5. Recover failed master ©Continuent 2012 34Tuesday, November 13, 12 34
  35. 35. Rolling SQL Schema Upgrade Slave Upgrade Slave Upgrade Switch Final Upgrade • Shun slave 1, • Repeat for • Switch • Upgrade old run upgrade, additional master to master. and return to slave(s) promote an Cluster cluster. upgraded schema is now Discard and slave fully upgraded recover on failure ©Continuent 2012 35Tuesday, November 13, 12 35
  36. 36. Backup, Recovery, and Restore • Backups /cluster > datasource DB1 backup • Recover /cluster > datasource DB2 recover • Restore /cluster > datasource DB2 restore • Out-of-box support for mysqldump/ XtraBackup; simple integration for others ©Continuent 2012 36Tuesday, November 13, 12 36
  37. 37. Data Fabric Connectivity ©Continuent 2012 37Tuesday, November 13, 12 37
  38. 38. Tungsten Connector Basics • Any MySQL/PostgreSQL client can connect • Initiates connections using JDBC App Native protocol Connector COM_QUERY COM_INIT_DB COM_DROP_DB ©Continuent 2012 38Tuesday, November 13, 12 38
  39. 39. Tungsten Connector Basics • Pass-through operation after connection • Full transparency & low overhead for clients Protocol packet COM_QUERY select * from t App Connector Protocol packet OK ResultSet Rows:1 ©Continuent 2012 39Tuesday, November 13, 12 39
  40. 40. Simple Read/Write Splitting • Very fast, easy-to-understand scaling model • Transactions go to master • Auto-commit SELECTs go to slaves, if available, but... • Reads potentially outdated data • Ephemeral SQL objects won’t be available • May miss: select update_function() ©Continuent 2012 40Tuesday, November 13, 12 40
  41. 41. SmartScale Session Load Balancing Master Transaction DBMS History Log Connect/Write Session x App Session y Connector Slave replay position Slave DBMS ©Continuent 2012 41Tuesday, November 13, 12 41
  42. 42. SmartScale Session Load Balancing Master Transaction DBMS History Log Read Session x App Session y Connector Slave replay position Slave DBMS ©Continuent 2012 42Tuesday, November 13, 12 42
  43. 43. SmartScale Session Load Balancing Master Transaction DBMS History Log Read Session x App Connector Slave replay position Slave DBMS ©Continuent 2012 43Tuesday, November 13, 12 43
  44. 44. Multi-Site Data Service ©Continuent 2012 44Tuesday, November 13, 12 44
  45. 45. Multi-Master Pattern • Updates on 2+ sites (active-active mode) • Enables geographic distribution of data • No failover necessary if network fails or site becomes unavailable • Not all applications can handle multi-master • Applications must avoid con!icts • Careful testing required • Restoration of broken systems may not be easy ©Continuent 2012 45Tuesday, November 13, 12 45
  46. 46. Simple Multi-Master Con!guration NYC Replicator Frankfurt Replicator fra (slave) fra (master) nyc (master) nyc (slave) Database-to-Database ©Continuent 2012 46Tuesday, November 13, 12 46
  47. 47. Clustered Multi-Master Con!guration NYC Frankfurt master master fra nyc slave fra nyc slave ©Continuent 2012 47Tuesday, November 13, 12 47
  48. 48. Clustered Multi-Master Con!guration NYC Frankfurt slave master fra nyc master fra nyc slave NYC Master Offline ©Continuent 2012 48Tuesday, November 13, 12 48
  49. 49. Clustered Multi-Master Con!guration NYC Frankfurt master master fra nyc slave fra nyc slave NYC Replicator Offline ©Continuent 2012 49Tuesday, November 13, 12 49
  50. 50. Complex Multi-Master Topologies All Masters Star Replication ©Continuent 2012 50Tuesday, November 13, 12 50
  51. 51. Primary/DR Pattern • Updates go to primary site master • DR site receives updates in real time • Simple command promotion • Tungsten connectors can route to primary and DR sites • No application changes required • Simplest way to protect against region/site failures ©Continuent 2012 51Tuesday, November 13, 12 51
  52. 52. Tungsten Composite Data Services App Server App Server + Connector + Connector San Jose New York sj_nyc Master Slave ©Continuent 2012 52Tuesday, November 13, 12 52
  53. 53. Real-Time Data Bridge ©Continuent 2012 53Tuesday, November 13, 12 53
  54. 54. Problem: Real-Time Order Ful!llment Web-Facing Sales Purchase Ordering PHP/ Java/ Perl JBoss 1-2M transactions daily 3M Order Items Real-Time Replication MySQL Oracle ©Continuent 2012 54Tuesday, November 13, 12 54
  55. 55. How Does Tungsten Do That? MySQL Tungsten Master Tungsten Slave Oracle Replicator Replicator Service oracle Service oracle MySQLExtractor Special Filters Special Filters * Ignore extra tables * Transform enum to string * Map names to upper case * Optimize updates to MySQL remove unchanged columns Binlog binlog_format=row ©Continuent 2012 55Tuesday, November 13, 12 55
  56. 56. Batch Loading to Data Warehouses MySQL Vertica Master Data Replicator Warehouse Service batch London Transaction Csv Csv Csv History Log THL File File ... File ©Continuent 2012 56Tuesday, November 13, 12 56
  57. 57. Possibilities for Analytics... Complex, near real-time MySQL Master reporting Light-weight, real-time operational status OLTP Data Web-facing mini- data marts for SaaS users ©Continuent 2012 57Tuesday, November 13, 12 57
  58. 58. Getting Started ©Continuent 2012 58Tuesday, November 13, 12 58
  59. 59. How Do I Obtain Tungsten? • Register at Continuent to get downloads for Oracle replication: http://www.continuent.com/downloads/software • Documentation is available on Continuent website • MySQL replication is available from Tungsten Replicator project on code.google.com http://code.google.com/p/tungsten-replicator/ ©Continuent 2012 59Tuesday, November 13, 12 59
  60. 60. 560 S. Winchester Blvd., Suite 500 Our Blogs: San Jose, CA 95128 http://scale-out-blog.blogspot.com Tel +1 (866) 998-3642 http://datacharmer.org/blog Fax +1 (408) 668-1009 http://www.continuent.com/news/blogs e-mail: sales@continuent.com Continuent Web Page: http://www.continuent.com Tungsten Replicator 2.0: http://code.google.com/p/tungsten-replicator ©Continuent 2012.Tuesday, November 13, 12 60

×