Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Severalnines Self-Training: MySQL® Cluster - Part I

If you are planning to deploy MySQL® Cluster, then you should consider our MySQL® Cluster training. Our MySQL® Cluster experts, all former MySQL AB staff, can help you and your business get the most out of your MySQL® Cluster solution. As a first step, you can now read through our free online training. We will be releasing slides of this training on MySQL® Cluster in sections over the coming weeks. Make sure to tune in here regularly to catch all of the sections of this expert training on MySQL® Cluster.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

Severalnines Self-Training: MySQL® Cluster - Part I

  1. 1. MySQL Cluster Trainingpresented by severalnines.com<br />Address:<br />c/o Severalnines AB<br />SICS, Box 1263<br />Isafjordsgatan22<br />SE-164-29 Kista<br />Contact:<br />Jean-Jérôme Schmidt<br />Email: services@severalnines.com<br />
  2. 2. Introduction<br />At Severalnines, we believe in sharing information and knowledge; we all come from an open source background<br />We know a lot of things about MySQL Cluster and think that MySQL Cluster is a great technology<br />These free MySQL Cluster Training slides are a contribution of ours to the knowledge and information sharing that’s common practice in the open source community<br />If you have any questions on these slides or would like to book an actual training class, please contact us at: services@severalnines.com<br />
  3. 3. Training Slides - Concept<br />Over the coming weeks we will be chronologically releasing slides for the different sections of our MySQL Cluster Training program on our website.<br />The full agenda of the training with all of its modules is outlined in the next slides so that you can see what topics will be covered over the coming weeks.<br />Particularly specialised topics such as Cluster/J or NDB API are not fully covered in the slides. We recommend our instructor-led training classes for such topics. <br />Please contact us for more details: services@severalnines.com<br />
  4. 4. Full Training Agenda (1/4)<br />MySQL Cluster Introduction<br />MySQL eco system<br />Scale up, scale out, and sharding<br />MySQL Cluster Architecture<br />Use cases<br />Features<br />Node types and Roles<br />Detailed Concepts<br />Data Distribution<br />Verifying data distribution<br />Access Methods<br />Partitioning<br />Node failures and failure detection<br />Network Partitioning<br />Transactions and Locking<br />Consistency Model<br />Redo logging and Checkpointing<br />
  5. 5. Agenda (2/4)<br />Installing MySQL Cluster<br />Setting up MySQL Cluster<br />Starting/stopping nodes<br />Recovery and restarts<br />Upgrading configuration<br />Upgrading Cluster<br />Performance Tuning (instructor-led only; contact us at services@severalnines.com)<br />Differences compared to Innodb/MyISAM<br />Designing efficient and fast applications<br />Identifying bottlenecks<br />Tweaking configuration (OS and MySQL Cluster)<br />Query Tuning<br />Schema Design<br />Index Tuning<br />
  6. 6. Agenda (3/4)<br />Management and Administration<br />Backup and Restore<br />Geographical Replication<br />Online and offline operations<br />Ndbinfo tables<br />Reporting<br />Single user mode<br />Scaling Cluster<br />Disk Data<br />Use cases<br />Limitations<br />Best practice configuration<br />Designing a Cluster<br />Capacity Planning and Dimensioning<br />Hardware recommendations<br />Best practice Configuration<br />Storage calculations<br />
  7. 7. Agenda (4/4)<br />Resolving Issues<br />Common problems<br />Error logs and Tracefiles<br />Recovery and Escalation procedures<br />Connectivity Overview<br />NDBAPI<br />Cluster/J<br />LDAP<br />Severalnines Tools<br />Monitoring and Management<br />Benchmarking<br />Sandboxes<br />Configuration and capacity planning<br />Conclusion<br />
  8. 8. Agenda: Lab Exercises(only applicable to instructor-led training classes)<br />Lab Exercises<br />Installing and Loading data into MySQL Cluster<br />Starting/stopping nodes, recovery<br />Query tuning<br />Backup and Restore<br />Configuration Upgrade<br />Would you like to try something particular?<br />This is possible too, speak with your instructor<br />
  9. 9. Prerequisites<br />Readers / Participants have understanding of SQL and basic database concepts. <br />Laptops/PCs for hands-on exercises<br />Linux: 1GB RAM<br />Windows: 2GB RAM<br />Approx. 20GB disk space and Virtualbox installed. <br />Virtualbox can be downloaded for free at http://www.virtualbox.org/wiki/Downloads<br />MySQL Cluster version 7.1 or later<br />
  10. 10. Week 1 – Training SlidesSection 1: MySQL Cluster Introduction<br />
  11. 11. Topics<br />Origins of MySQL Cluster <br />Scale up, scale out, and sharding<br />MySQL Cluster Architecture<br />Use cases<br />Features<br />Node types and Roles<br />
  12. 12. Origins of MySQL Cluster<br />Ericsson started development of NDB Cluster in 1996<br />Focus on HA/performance requirements of Telecom<br />Offered as a product in 2001 by Alzato, fully owned by Ericsson<br />Acquisition by MySQL in September 2003<br />Core team is transferred <br />Intellectual Property transferred (9 patents)<br />New product called MySQL Cluster <br />Original NDB Cluster + SQL engine + MySQL connectors <br />Major investment in usability<br />Co-developed with some telecom vendors<br />
  13. 13. Telco-grade Data Management<br />High Availability<br />99.999% (< 5 mins/yr) or 99.9999% (< 30 secs/yr)<br />Scalability<br />Throughput in the range of 1000s or 10,000s per second<br />Predictable Real-time Performance<br />Typically 5 to 10 milliseconds<br />In-memory performance<br />Cost effective <br />ScaleOut vs. ScaleUp<br />COTS hardware <br />
  14. 14. Scale up, scale out, sharding<br />Scale up means having one beefy machine handling load.<br />At some point it won’t scale<br />Depending on implementation, mutexes and semaphores these systems scales up to about 16 processors/cores.<br />With the MySQL Server you will be limited by the number of incoming connections at some stage.<br />Hard to increase the capacity<br />Reconfigure with more cores<br />Examples<br />Sybase. Customer has a Sun E10000, next step is to get a Sun E15000, and then license cost for additional processors<br />Gets expensive.<br />Scale up is old school.<br />
  15. 15. Scale up, scale out, sharding<br />Scale-out means that instead of using one beefy machine you use many.<br />This can be accomplished with<br />MySQL Replication<br />Master -> slave replication<br />Pros: easy to add slaves<br />Cons: Write scaling is hard, one single master, read/write split<br />MySQL Cluster<br />Will be addressed in this training.<br />Read<br />Read/writes<br />Asynch. replication<br />Read<br />
  16. 16. Scale up, scale out, sharding<br />Shardingmeans that instead of using one beefy machine you use many and partitions the data on the machines.<br />Problems:<br />Hard do find a good key to partition by<br />Adding nodes are hard – how to redistribute the data?<br />MySQL Cluster addresses a lot of these problems<br />Sharding Aware Layer in Application Code<br />Users1-1000<br />Users1001-2000<br />Users2001-3000<br />
  17. 17. Application R/I/U/D<br />Application R/I/U/D<br />Application R/I/U/D<br />Java, C++, LDAP<br />Application R/I/U/D<br />MySQL Cluster Architecture<br />Native API<br />DATANODE<br />DATANODE<br />Management Server<br />Management Server<br />DATANODE<br />DATANODE<br />
  18. 18. <ul><li>Parallel Database Server
  19. 19. Shared-nothing architecture
  20. 20. Data partitioned across nodes
  21. 21. In a RAID10 fashion
  22. 22. Redundancy
  23. 23. Synchronous replication
  24. 24. Automatic failover and repair
  25. 25. Performance
  26. 26. Ability to mix and match disk and memory tables
  27. 27. Different access methods (SQL, C++, Java,LDAP) </li></ul>MySQL Cluster basics<br />
  28. 28. <ul><li>No single point of failure
  29. 29. Any node can fail without affecting the system
  30. 30. Configurable number of data copies
  31. 31. Up to 4 copies
  32. 32. Synchronous replication
  33. 33. Automatic failure detection and failover
  34. 34. Detection time determined by heartbeat
  35. 35. Failover is instantaneous
  36. 36. Automatic resynchronization
  37. 37. Delta or Full synchronization</li></ul>Availability Features (1/2)<br />
  38. 38. <ul><li>Handling of network partitioning
  39. 39. Uses an Arbitrator
  40. 40. Online Software Upgrade
  41. 41. Supports upgrades and downgrade of system without outages
  42. 42. Disaster Recovery
  43. 43. Geographical Replication using Asynchronous protocol
  44. 44. Online Backup </li></ul>Availability Features (2/2)<br />
  45. 45. Use Cases<br />Good fit<br />OLTP apps with short running queries<br />Application with real-time characteristics and requirements<br />A lot of concurrent requests<br />Write intensive applications<br />Typically the following are a poor fit:<br />Heavy reporting type (OLAP)<br />Data Warehouse<br />Complex JOINs (much improved in 7.2) scale badly.<br />A couple of tables and with about 1000 records meeting the JOIN criteria is just fine)<br />Complex and long running transactions in a realtime system is just wrong.<br />However, replicate from MySQL Cluster to regular MySQL (innodb) which runs the reporting.<br />
  46. 46. Application R/I/U/D<br />Application R/I/U/D<br />Application R/I/U/D<br />Java, C++, LDAP<br />Application R/I/U/D<br />MySQL Cluster Architecture<br />Native API<br />DATANODE<br />DATANODE<br />Management Server<br />Management Server<br />DATANODE<br />DATANODE<br />
  47. 47. Node Types and Roles<br />MySQL Cluster is comprised of the following node types:<br />Management Nodes<br />Maintains the cluster configuration<br />Involved in handling network partitioning<br />Writes a cluster log containing events occurring in Cluster.<br />Typically two are deployed for redundancy <br />Data Nodes<br />Transaction, query handling and data storage <br />Atleast two for redundancy<br />Application Nodes<br />Mysql server, Cluster/J<br />Requests are ultimately converted to NDBAPI calls <br />
  48. 48. Starting a MySQL Cluster<br />Pretend that we have four hosts:<br />app01<br />app02<br />storage01<br />storage02<br />We will start a HA cluster consisting of four nodes:<br />2 data nodes, one each on storage01 and storage02<br />2 mysql servers to access the data nodes, one on each of app01 and app02<br />2 management servers to deal with configuration and network partitioning, one of each on app01 and app02.<br />
  49. 49. Management Nodes/Servers<br />Owns the Cluster configuration and loads a file called config.ini.<br />Config.ini describes the Cluster<br />Number of management nodes, data nodes access (mysql/ndbapi) nodes<br />Ports, Resources, Data Directories<br />The management node is started with<br />ndb_mgmd –f /etc/mysql/config.ini --configdir=/etc/mysql<br />The management server generates a binary versioned configuration file that is stored in configdir.<br />All nodes must first connect to the management nodes to fetch the configuration<br />Listens by default on port 1186<br />Writes a cluster log in Datadir (e.gDatadir=/data/)<br />
  50. 50. MySQLClusterTwo management servers are started<br />The management servers loads in the config.ini file.<br />Compare with each other they have the same version. If not, they won’t start.<br />The management servers start writing a cluster log<br />Ndb_X_cluster.log<br />Located in the datadir.<br />Hostname=app01datadir=/data/Listens on *:1186<br />Hostname=app02datadir=/data/Listens on *:1186<br />Management Server1<br />Management Server2<br />Compareconfig<br />app01> ndb_mgmd –f /etc/mysql/config.ini –configdir=/etc/mysql --reload<br />app02> ndb_mgmd –f /etc/mysql/config.ini –configdir=/etc/mysql --reload<br />
  51. 51. Data Nodes<br />Connects to the management nodes and fetch the configuration<br />Needs a connectstring pointing to the management nodes<br />The data nodes are the processes<br />Storing data<br />Managing transaction, indexing, data distribution etc etc.<br />When the data node gets the configuration it will try and setup connections to other nodes.<br />Started with<br />ndbd –c “hostname1:1186,hostname2:1186”<br />ndbmtd –c “hostname1:1186,hostname2:1186”<br />Ndbmtd is the multithreaded daemon and can use up to 8 cores. Ndbd is for single/dual core machines.<br />
  52. 52. MySQL ClusterData Node 1 Joins<br />Hostname=app01datadir=/data/Listens on *:1186<br />Hostname=app02datadir=/data/Listens on *:1186<br />Hostname=storage01datadir=/data/<br />Management Server1<br />Management Server2<br />DATANODE1<br />Get config<br />Starts listening for other data node<br />storage01> ndbd –c “app01:1186,app02:1186”<br />
  53. 53. MySQL ClusterData Node 2 Joins<br />Hostname=app01datadir=/data/Listens on *:1186<br />Hostname=app02datadir=/data/Listens on *:1186<br />Hostname=storage01datadir=/data/<br />Management Server1<br />Management Server2<br />DATANODE1<br />Get config<br />Hostname=storage01datadir=/data/<br />DATANODE1<br />storage02> ndbd –c “app01:1186,app02:1186”<br />
  54. 54. API Nodes<br />NDBAPI programs<br />Direct API (no SQL)<br />C++ programs, uses libndbclient<br />Cluster/J<br />Java, has a JNI interface that uses libndbclient<br />MySQL nodes/server<br />Convert SQL into NDBAPI code<br />Uses libndbclient<br />Stores data/logs in a data directory<br />Configuration for the server in a my.cnf file<br />In any case<br />A connectstring is a always needed pointing to the management nodes.<br />
  55. 55. MySQL ClusterA MySQL Server Joins<br />Get config<br />Hostname=app01datadir=/data/Listens on *:1186<br />Hostname=app02datadir=/data/Listens on *:1186<br />Hostname=storage01datadir=/data/<br />Hostname=app03datadir=/data/<br />Management Server1<br />Management Server2<br />DATANODE1<br />Hostname=storage01datadir=/data/<br />DATANODE1<br />app03> mysqld &<br />
  56. 56. MySQL ClusterA MySQL Server Joins<br />Hostname=app01datadir=/data/Listens on *:1186<br />Hostname=app02datadir=/data/Listens on *:1186<br />Hostname=storage01datadir=/data/<br />Hostname=app03datadir=/data/<br />Management Server1<br />Management Server2<br />DATANODE1<br />Hostname=storage01datadir=/data/<br />DATANODE1<br />app03> mysqld &<br />
  57. 57. MySQL ClusterFullystarted<br />Hostname=app01datadir=/data/mysqlclusterListens on *:1186<br />Hostname=app02datadir=/data/mysqlclusterListens on *:1186<br />Hostname=storage01datadir=/data/mysqlcluster<br />Hostname=app01datadir=/data/mysql<br />Management Server1<br />Management Server2<br />DATANODE1<br />Hostname=storage01datadir=/data/mysqlcluster<br />Hostname=app02datadir=/data/mysql<br />DATANODE1<br />
  58. 58. Fully Connected<br />All data nodes are connected to every other node<br />Epoll is used<br />Max 255 Nodes in the Cluster<br />Max 48 Data Nodes<br />Biggest Deployed Cluster<br />48 data nodes<br />
  59. 59. Coming next in week 2:Section 2: Detailed Concepts<br />
  60. 60. Topics covered in Week 2<br />Data Distribution<br />Verifying data distribution<br />Access Methods<br />Partitioning<br />Node failures and failure detection<br />Network Partitioning<br />Transactions and Locking<br />Consistency Model<br />Redo logging and Checkpointing<br />
  61. 61. We hope these training slides are useful to you!Please visit our website to view the next section of this training.For any questions, comments, feedback or to book a training class, please contact us at: services@severalnines.comThank you!<br />
  62. 62. Disclaimer<br />© Copyright 2011 Severalnines AB. All rights reserved.<br />Severalnines and the Severalnineslogo(s) are<br />trademarks of Severalnines AB. <br />MySQL is a registered trademark of Oracle and/or its<br />affiliates. <br />Other names may be trademarks of their<br />respective owners. <br />

×