OSSCube MySQL Cluster Tutorial By Sonali At Osspac 09

3,538 views

Published on

Sonali from OSSCube presents on MySQL Cluster Tutorial at OSSPAC 2009

OSSCube-Leading OpenSource Evangelist Company.

To know how we can help your business grow, contact:

India: +91 995 809 0987
USA: +1 919 791 5472
WEB: www.osscube.com
Mail: sales@osscube.com

Published in: Technology, Health & Medicine

OSSCube MySQL Cluster Tutorial By Sonali At Osspac 09

  1. 1. MySQL Cluster Tutorial Presented by – Sonali Minocha
  2. 2. Who Am I?MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  3. 3. Architecture and Organization
  4. 4. What is a Cluster?MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  5. 5. Single MySQL Server Architecture SQL Request MySQL Client MySQL Server Response
  6. 6. MySQL Cluster Architecture NDB API Call SQL RequestMySQL Client Partitioning Response Replication Message Response Data Nodes SQL Node (MySQL server)
  7. 7. Components of MySQL ClusterMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  8. 8. Various Nodes API Node Data NodeApplication Management Node MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  9. 9. NdbCluster NdbCluster NdbClusterEngine Engine Engine (Data Nodes) MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  10. 10. OrganizationMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  11. 11. PartitionMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  12. 12. PartitioningMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  13. 13. FragmentsMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  14. 14. Node GroupsMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  15. 15. Split BrainI have full Servers can I have fullresponsibility not see responsibilityof the cluster each other of the cluster MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  16. 16. Arbitration NG 1 Network SplitFirst Node to ask will continue while the other will be shutdown MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  17. 17. More Data Nodes NG Network 1 Split/ Shutdown N G 2MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  18. 18. Arbitrator IssuesMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  19. 19. Arbitrator IssuesMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  20. 20. Normal Operation I a ive m AlAl m Ia e iv e liv A m Al m Ia Ia e MySQL Cluster Tutorial, OSSPAC 09 iv Singapore, © OSSCube
  21. 21. Missed Heartbeats I a ive m Al Alive I am Al m Ia eMySQL Cluster Tutorial, OSSPAC 09 iv Singapore, © OSSCube
  22. 22. SynchronousMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  23. 23. Phase One: Commit- RequestCommit CommitMessage MessageResponse ResponseMessage Message MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  24. 24. Phase Two: Successful CommitTransaction TransactionSuccessful Successful Commit Commit Transaction Transaction MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  25. 25. Phase Two: Failure (Abort Commit)Transaction Transaction Failed Successful Rollback Rollback Transaction Transaction MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  26. 26. INSTALL MySQL CLUSTER
  27. 27. Initiate MySQL Cluster• # file "config.ini" - 2 data nodes and 2 SQL nodes# This file is placed in the startup directory of ndb_mgmd (the# management server)• # The first MySQL Server can be started from any host. The second# can be started only on the host MySQLd_5.MySQL.com• [NDBD DEFAULT] NoOfReplicas= 2 DataDir= /var/lib/MySQL-cluster• [NDB_MGMD] Hostname= ndb_mgmd.MySQL.com DataDir= /var/lib/MySQL-cluster• [NDBD] HostName= ndbd_2.MySQL.com• [NDBD] HostName=ndbd_3.MySQL.com• [MySQLD]• [MySQLD] HostName= MySQLd_5.MySQL.com MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  28. 28. Management NodeMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  29. 29. Data Nodes• my.cnf# example additions to my.cnf for MySQL Cluster• [ndbd] connect- string=ndb_mgmd.MySQL.com MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  30. 30. Data Node Configurations OptionsMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  31. 31. SQL Nodes• my.cnf• # example additions to my.cnf for MySQL Cluster # (will work on all versions)• # enable ndbcluster storage engine, and provide connectstring for management• # server host to the default port 1186• [MySQLd] Ndbcluster ndbconnectstring=ndb_mgmd.MySQL.com MySQL Cluster Tutorial, OSSPAC 09 :1186 Singapore, © OSSCube
  32. 32. Config.ini ParametersGeneral Parameters – Id – Hostname File and Directory Location Parameter – BackupDataDir – DataDir – FileSystemPath MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  33. 33. •Transaction Handling Parameters •TransactionInactiveTimeout •TransactionDeadlockDetectionTimeou t•Memory Usage Parameters •LockPagesInMainMemory•Utility Programs •ndb_size.pl
  34. 34. Monitoring the Startup PhasesMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube. http://osscube.com/technology/mysql
  35. 35. MySQL Cluster Storage Engine --NDB MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  36. 36. NDB Storage EngineMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  37. 37. NDB’s Storage Method• In-Memory Storage• Disk-Based Storage – New in MySQL 5.1 – Undo Logging – Fixed-Width Columns MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  38. 38. Tablespace Tablespace Datafile Datafile DatafileDatafile Log File group Redo log Undo Log
  39. 39. IndexesMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  40. 40. Creating Indexes• Each table in the cluster always has a hash index for a Primary Key• Ordered indexes can be created with ALTER TABLE ADD INDEX .... MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  41. 41. First Node/First Fragment• Hidden field/table created that will handle the management of the Unique Index on Name• Ordered Index (T-Tree) created for Continent field Code_Hash Code Name Continent 1351 CHE Switzerland Europe 1785 CHN China Asia Hidden 1943 FIN Finland Europe Field 2031 GOR United Kingdom Europe 2345 SKO South Korea Asia Ordered Index Hidden Table MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  42. 42. Second Node/Second Fragment• Same as First Node/Fragment setup• Contains the remaining data Code Hash Code Name Continent 1231 BRA Brazil South America 1853 ESP Spain EuropeHidden Field 2145 IND India Asia 2197 MOR Morocco Africa Ordered IndexHiddenTable MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  43. 43. UniqueKey Lookup network hop Primary Key Lookup may Unique Key Lookup require another Function() Look up rowPK values Hash Data node Look up row Data node MySQL Server MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  44. 44. Parallel Ordered Index Scan T- Data node Tree Index rowsMySQLServer Data node T- Tree Index MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  45. 45. Parallel full table scan Data node rowsMySQLServer Data node MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  46. 46. Parallel full table scan ( WHERE condition processed in SQL node) SQL Query Data nodeFew Rows rows Rows Many MySQL Where Server Conditi on Data node MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  47. 47. SQL Query Data node WhereFew Rows Conditi Fewrows on Rows MySQL Server Where Data node Conditi on MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  48. 48. Storage of IndexesMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  49. 49. Characteristics of NDB storage EngineMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  50. 50. DurabilityMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  51. 51. Global Check Point• A GCP occurs every few seconds, when transactions for all nodes are synchronized and the REDO log is flushed to disk MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  52. 52. Local Check PointMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  53. 53. Memory - RAM• Memory Allocation Total Memory = N(data nodes)* Local Memory / NoOfReplicas -- or -- Local Memory = Total Memory * NoOfReplicas / N(data nodes)• Sizing up the Memory – Fixed size in memory – Variable sized in memory – Fixed size on disk MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  54. 54. Memory PagesMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  55. 55. MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  56. 56. HANDLING DATA IN MySQLCLUSTER
  57. 57. Disk Data ObjectsMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  58. 58. Creating the Log File Group• Create Log File Group CREATE LOGFILE GROUP lg_1 ADD UNDOFILE undo_1.dat INITIAL_SIZE 16M UNDO_BUFFER_SIZE 2M ENGINE NDB;• Adding Undo Files to Log File Group ALTER LOGFILE GROUP lg_1 ADD UNDOFILE undo_2.dat INITIAL_SIZE 12M ENGINE NDB; MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  59. 59. Verifying UNDO Files are Created• INFORMATION_SCHEMA Tables SELECT LOGFILE_GROUP_NAME, LOGFILE_GROUP_NUMBER, EXTRA FROM INFORMATION_SCHEMA.FILES WHERE FILE_NAME = undo_1.dat; SELECT LOGFILE_GROUP_NAME, LOGFILE_GROUP_NUMBER, EXTRA FROM INFORMATION_SCHEMA.FILES WHERE FILE_NAME = undo_2.dat; MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  60. 60. Creating the Tablespace• Create a Tablespace CREATE TABLESPACE ts_1 ADD DATAFILE data_1.dat USE LOGFILE GROUP lg_1 INITIAL_SIZE 32M ENGINE NDB;• Add an Additional Data File ALTER TABLESPACE ts_1 ADD DATAFILE data_2.dat INITIAL_SIZE 48M ENGINE NDB;• Verifying Data Files are Created SELECT FILE_NAME, LOGFILE_GROUP_NAME, EXTRA FROM INFORMATION_SCHEMA.FILES WHERE TABLESPACE_NAME = ts_1 AND FILE_TYPE = DATAFILE; MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  61. 61. Create a Disk Data Table• Non-Indexed Columns Stored on Disk in Tablespace CREATE TABLE dt_1 ( memberId INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY, lName VARCHAR(50) NOT NULL, fName VARCHAR(50) NOT NULL, dob DATE NOT NULL, joined DATE NOT NULL, INDEX(lName, fName) ) TABLESPACE ts_1 STORAGE DISK ENGINE NDB;• Alter Existing Non-Cluster Tables ALTER TABLE city TABLESPACE ts1 STORAGE DISK ENGINE=NDB;• View Table Status SHOW TABLE STATUS LIKE cityG MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  62. 62. Monitoring Free Space MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  63. 63. Dropping Cluster Tables MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  64. 64. MySQL CLUSTERMANAGEMENT
  65. 65. MySQL Cluster’s Native Backup Tool Backup-1.2.ctl• Backup Data Data Data Backup-1.2.log Backup-1.0.2.Data – .ctl Node Node Backup-1.1.2.Data – .log – .data• Abort Backup Backup-1.3.ctl Data Data Backup-1.3.log ndb_mgm> ABORT BACKUP 6 Node Node Backup-1.0.3.Data Backup-1.1.3.Data Abort of backup 6 ordered Node 2: Backup 10 started from 1 has been aborted. Error: 1321 MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  66. 66. MySQLdumpMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  67. 67. Restoring Backups• Recovering MySQL Clusters Native Backup Files (1/3) – Verify Management Node is Running shell> ndb_mgm -e "SHOW" – Start and Empty the Data Nodes • Start each data node with the --initial option. MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  68. 68. Restoring Backups MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  69. 69. Restoring Backups MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  70. 70. Restoring MySQLdump Files MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  71. 71. Optimization
  72. 72. EXPLAINMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  73. 73. Full Table Scan• Without Condition Pushdown• With Condition Pushdown MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  74. 74. Parallel full table scan ( WHERE condition processed in SQL node) SQL Query Data node WhereFew Rows Condition rows Rows Many MySQL Server Data node MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  75. 75. Parallel full table scan ( WHERE condition processed in data node) Where SQL Query Condition Data nodeFew Rows Few Rows rows MySQL Where Server Condition node Data MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  76. 76. Index Statistics MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  77. 77. MySQL Cluster Query Cache MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  78. 78. Data Size/Usage Issues MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  79. 79. Query DesignMySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  80. 80. MySQL ClusterReplication
  81. 81. MySQL Replication MySQL Server Application W SlaveMySQL Server RelaylogMaster IO thread SQL thread MySQL Server Binlog Slave Relaylog Binlog
  82. 82. MySQLd MySQLd Update Data node Update Data node Cluster Data node Data node Update UpdateNdb API Ndb API
  83. 83. NDB Injector Thread•A thread inside MySQL server•Responsible for injecting rows inbinlog and produces “Single connicalbinlog for cluster” • not just one MySQL Server •It contains everything written on all ndbApi program including MySQLd connected to cluster
  84. 84. MySQL Cluster Replication MySQLd(slave) Binlog MySQLd(Master) Replication NdbClsuter HandlerNdbClsuter Handler Relaylog IO thread Binlog Apply thread Data node Data nodeData node Data node Cluster Cluster Data node Data nodeData node Data node
  85. 85. Thank you for your time and attention www.osscube.comFor more information, please feel free to drop in a line to sonali@osscube.com or visit http://www.osscube.com MySQL Cluster Tutorial, OSSPAC 09 Singapore, © OSSCube
  86. 86. QnA

×