Installation of MySQL 5.1 Cluster Software on the Solaris 10 ...
Upcoming SlideShare
Loading in...5
×
 

Installation of MySQL 5.1 Cluster Software on the Solaris 10 ...

on

  • 6,607 views

 

Statistics

Views

Total Views
6,607
Views on SlideShare
6,607
Embed Views
0

Actions

Likes
0
Downloads
85
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Installation of MySQL 5.1 Cluster Software on the Solaris 10 ... Installation of MySQL 5.1 Cluster Software on the Solaris 10 ... Document Transcript

  • Installation of MySQLTM 5.1 Cluster Software on the SolarisTM 10 OS for x64 Platforms Rainer Walter Sun Microsystems September 2008
  • Sun Microsystems, Inc. Copyright © 2008 Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, California 95054, U.S.A. All rights reserved. U.S. Government Rights - Commercial software. Government users are subject to the Sun Microsystems, Inc. standard license agreement and applicable provisions of the FAR and its supplements. Use is subject to license terms. This distribution may include materials developed by third parties. Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX is a registered trademark in the U.S. and in other countries, exclusively licensed through X/Open Company, Ltd. X/Open is a registered trademark of X/Open Company, Ltd. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the United States and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc. AMD, Opteron, the AMD logo, the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. Sun, Sun Microsystems, the Sun logo, MySQL, Solaris, Sun BluePrints, Sun Fire, and SunSolve are trademarks or registered trademarks of Sun Microsystems, Inc. or its subsidiaries in the United States and other countries. This product is covered and controlled by U.S. Export Control laws and may be subject to the export or import laws in other countries. Nuclear, missile, chemical biological weapons or nuclear maritime end uses or end users, whether direct or indirect, are strictly prohibited. Export or reexport to countries subject to U.S. embargo or to entities identified on U.S. export exclusion lists, including, but not limited to, the denied persons and specially designated nationals lists is strictly prohibited. DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.
  • Sun Microsystems, Inc. Table of Contents Summary...............................................................................................................................................................1 Introduction..........................................................................................................................................................2 The MySQL Software....................................................................................................................................2 MySQL Cluster Software...............................................................................................................................2 Node Types..................................................................................................................................................2 High Availability Concept..............................................................................................................................3 Partitioning.................................................................................................................................................3 Availability of the MySQL Cluster Software.....................................................................................................3 Scope and Limitations............................................................................................................................................4 Audience...............................................................................................................................................................4 Installation............................................................................................................................................................5 Prerequisites................................................................................................................................................5 Compilation of the Source............................................................................................................................5 Installation of the MySQL Server Software.....................................................................................................6 Configuration of a MySQL Cluster............................................................................................................................7 Overview.....................................................................................................................................................7 MySQL Cluster Configuration Files.................................................................................................................8 Starting and Stopping Components of a MySQL Cluster.................................................................................14 Cluster Messages and Error Log Files............................................................................................................16 User Accounts and File Ownership...............................................................................................................16 Storage Management in a MySQL Cluster.....................................................................................................16 Glossary...............................................................................................................................................................19 For More Information...........................................................................................................................................19 Acknowledgements..............................................................................................................................................21 Revision History...................................................................................................................................................21
  • 1 Summary Sun Microsystems, Inc. Summary This document describes the installation of MySQLTM cluster software on three Sun FireTM X4100 M2 servers running the SolarisTM 10 Operating System for x64 platforms. For two scenarios where data is loaded and manipulated, the adjustment of the MySQL configuration necessary to execute these tasks is described. (Note: The installation was performed with the Release Candidate Development Release of the MySQL 5.1 Community Edition.) Introduction The MySQL Software MySQL software is the most popular relational database management system (RDBMS) software in the open source space and part of the LAMP stack, which is a collection of software for dynamic web pages. Among the features that differentiate MySQL software from other well-known RDBMS software are: ● Support for a broad variety of operating system platforms ● Choice of storage engines with different locking, indexing, and logging capabilities, enabling an optimal tailoring of the database with regards to hardware resources and requirements of the application on top MySQL Cluster Software MySQL cluster software enables users to implement a MySQL database on commodity hardware using an approach for fault tolerance, which ensures that data remains available even in the event of a failure of a database server or storage component. The manner of clustering MySQL software described in this paper is accomplished on the application level with no resources being shared among the cluster members at all, in contrast to cluster concepts that operate on the operating system level, such as SunTM Cluster software, where the cluster members share one or more storage resources. Implementing high availability for MySQL software using Sun Cluster software is described elsewhere1. While a detailed technical description of the design principles of MySQL cluster software can be found on the MySQL web site2, an exhaustive discussion about the benefits and caveats of such a high availability solution is beyond the scope of this paper. The following section provides a brief terminology of a MySQL cluster environment. It should be noted that, while the availability and integrity of data in the MySQL database can be shielded against hardware failures, existing client sessions, client connections, and active transactions are not protected by this MySQL clustering concept and might suffer from such failures. 1 Sun Cluster Data Service for MySQL Guide for Solaris OS, August 2005, Part No: 819-1088-11 and http://blogs.sun.com/luojiach/entry/installation_and_configuration_sun_cluster1 2 M. Ronström, L. Thalmann: MySQL Cluster Architecture Overview, a MySQL technical white paper, 2004; New Features in MySQL Cluster 5.1, a MySQL technical white paper, August 2006. Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms
  • 2 Introduction Sun Microsystems, Inc. Node Types There are three different functions in a MySQL cluster, which are associated with respective daemons: ● Data (storage) node with daemon ndbd ● SQL node with daemon mysqld ● Management node with daemon ndb_mgmd A server can host more than one function of a MySQL cluster. High Availability Concept High availability is achieved on the application level by providing redundant services and storage resources on independent servers in a shared-nothing architecture. Network Database (NDB) is the storage engine required for this architecture. The data portion of the database that one data (storage) node holds is called the cluster partition. Cluster partitions are managed as replicas. A primary replica is the data portion, which is assigned to a data node in the cluster; a backup replica is a copy of a partition that is assigned to another node. Data manipulation statements (update , insert, delete ) and data definition statements (for example, create table, create tablespace , and so on) are performed through a two-phase commit protocol, ensuring that the database, being spread across multiple data nodes, remains consistent and that the content of backup replicas is identical. The distribution of data in one replica across different storage nodes is accomplished by horizontal partitioning. Partitioning Data in the database is evenly split into as many partitions as data nodes exist. Copies of partitions (replicas) can be kept on data nodes other than the node that holds the original partition for redundancy. While the number of partitions is always equal to the number of storage nodes, the number of replicas can be one or two, where a setup with two replicas provides redundancy and is, therefore, recommended. The number of replicas controls how many node groups are formed in a MySQL cluster. A node group is a subset of data nodes, which holds replicas of the same partition; if all data nodes of a MySQL cluster that form one node group are missing, one partition is not available and so the database cannot be brought online. Availability of the MySQL Cluster Software Until MySQL version 5.1 software, MySQL cluster software came in two flavors: ● As part of the "ordinary" MySQL software distribution3. ● As MySQL Cluster Carrier Grade Edition, a special version for network service providers and telecom equipment manufacturers4, which offers some extra features not available in other distributions. The versioning of this edition is different from the "ordinary" software distribution. 3 For MySQL software version 5.1: http://dev.mysql.com/downloads/mysql/5.1.html 4 More details on http://www.mysql.com/why-mysql/communications/ MySQL Cluster Carrier Grade Edition download: http://dev.mysql.com/downloads/cluster/index.html Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms
  • 3 Introduction Sun Microsystems, Inc. The installation of the MySQL software was performed with the Release Candidate Development Release version 5.1.24 of the MySQL Community Edition software distribution. While cluster support is included in the package of MySQL version 5.1.24 for the Solaris OS for SPARC® platforms, the official MySQL download site did not offer an analogous package with cluster support for the Solaris OS for x64 platforms as of the writing this paper. Instead, the source of MySQL version 5.1.24 was downloaded and was compiled prior to installation. Planned future distributions of MySQL 5.1 binaries would no longer contain the MySQL cluster functionality; instead, the official successor version of MySQL Cluster 5.0 software would be MySQL Cluster Carrier Grade Edition 6.2. Scope and Limitations This paper describes the installation of the MySQL software and the steps that are necessary to set up a shared- nothing MySQL cluster in a minimal setup of three cluster nodes, which does not provide universal fault tolerance and is not, therefore, recommended for production purposes. The settings that need to be specified for loading and manipulating data in two different scenarios are discussed. An investigation about performance optimization and about security aspects of a MySQL cluster environment is beyond the scope of this paper. The default paths to executables and configuration files might differ between MySQL installations performed from binaries that are in Solaris package format and binaries built from source. For instance, the binaries of MySQL daemons, such as ndbd , mysqld , ndb_mgmd , are located in the /bin folder of the MySQL installation directory built from a Solaris package for SPARC platforms. However, installation on the Solaris OS for x64 platforms, which is based on compiled binaries from source, keeps these daemon executables in the /libexec subfolder of the installation directory, separate from other executables in /bin. Audience Readers of this paper should be familiar with concepts of relational databases and have basic knowledge of the SQL programming language. Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms
  • 4 Installation Sun Microsystems, Inc. Installation The installation of MySQL 5.1.24 software that is described herein was performed on three Sun Fire X4100 M2 servers with the following specifications: ● 2x dual-core AMD OpteronTM CPUs at 2.4 GHz ● 4 Gbytes of RAM ● Storage: Two 68-Gbyte internal hard disks ● Operating System: Solaris 10 5/08 OS for x64 platforms (s10x_u5wos09 X86) The installation of the MySQL software from the source comprises three steps: ● Run the configure script in the MySQL source directory to produce makefile scripts. ● Use gmake to produce the appropriate binaries. ● Run gmake install to set up the directory structure and to copy binaries and configuration files. Prerequisites The MySQL source can be obtained in compressed tar format from the MySQL web site3. For an installation of MySQL software from binaries that have been compiled from source, it is recommended that a user account with the user name mysql exists and a group names mysql exists, which the mysql user is member of. There is no need to set up a home directory for this user, nor to define a password. For the compilation of the MySQL 5.1.24 source on the Solaris 10 OS for x64 platforms, the executables gcc (version 3.4.3) and gmake (version 3.80) in /usr/sfw/bin are used. Additionally, the ar executable in /usr/ccs/bin is needed. The PATH variable must, therefore, be adjusted: PATH=$PATH:/usr/sfw/bin:/usr/ccs/bin If 64-bit binaries are compiled, a symbolic link is needed in the directory /usr/sfw/lib/amd64 : ln -s libgcc_s.so.1 libgcc_s.so Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms
  • 5 Installation Sun Microsystems, Inc. Compilation of the Source The MySQL source can be used to produce both 32-bit and 64-bit binaries. The procedure for both differs in the environment variables that are supplied when executing the configure script, while two options need to be provided in both cases when calling the configure script: ● The option --prefix specifies the installation directory of the MySQL binaries, which has been set to /opt/mysql . If omitted, the binaries are installed in /usr/local/mysql . ● The option --with-ndbcluster instructs the configuration process to include the binaries of the NDB engine, which is needed for storage nodes and the management node (or nodes) of a MySQL cluster, and which are not included by default. Building a 32-bit version of MySQL software: After extracting the tar archive of the MySQL source, the script configure is called with the following parameters from the directory of the extracted source files (mysql- 5.1.24-rc): CC=gcc CXX=gcc ./configure --prefix=/opt/mysql --with-ndbcluster Building a 64-bit version of MySQL software: After extracting the tar archive of the MySQL source, the script configure is called with the following parameters from the directory of the extracted source files (mysql- 5.1.24-rc): CC='gcc -m64' CXX='g++ -m64' LDFLAGS="-R/usr/lib/64 -R/lib/64 -R /usr/sfw/lib/64 -L/usr/sfw/lib/64 -L/usr/lib/64 -L/lib/64 -lrt" ./configure --prefix=/opt/mysql --with-ndbcluster The binaries are created by the following command, which is executed in the directory where the source files are located: gmake Installation of the MySQL Server Software The installation of the freshly compiled binaries is launched from the folder of the MySQL source files by issuing the following command: gmake install After the installation of the MySQL software, a script must be run, which initializes the grant tables of the MySQL server. This script is called mysql_install_db and it is located in the /bin folder of the MySQL installation directory (/opt/mysql/bin , in this case). This script creates a directory structure for two databases, mysql and test . Since these database directories must be readable and writable by the MySQL server and, therefore, must be owned by the user mysql, the command mysql_install_db , when executed in the context of the root user, should be run with the option -- user=mysql , which ensures that the necessary privileges and ownership flags are set. Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms
  • 6 Installation Sun Microsystems, Inc. By default, the database directories are set up in the MySQL installation directory under the subfolder data. If these database files and directories need to located elsewhere, two alternatives can be followed: ● Run the mysql_install_db command with the option --datadir=<path for parent directory of MySQL database files> ● Add an entry for datadir in the configuration file of the MySQL server, my.cnf , before running mysql_install_db The initial size of the folder containing the data files of the mysql database is 1.5 Mbytes. The size of the test database is initially 1 Kbyte. Configuration of a MySQL Cluster Overview A MySQL cluster environment comprises three different functions. Each function is characterized by specific sections in the my.cnf file. ● A MySQL cluster management node provides the configuration data of the cluster to storage nodes. It stops single storage nodes or shuts down the entire cluster. The MySQL cluster management node is based on the ndb_mgmd process, which reads the cluster configuration from the config.ini file. Cluster management is performed using the management console <MySQL_Install_Directory>/bin/ndb_mgm . ● A MySQL storage (data) node provides the storage capacity for data in a MySQL cluster database. It uses the NDB storage engine, which is based on the ndbd process. The ndbd process of a storage node obtains its configuration upon startup from the management node. ● A MySQL SQL node provides access to the data in a database of a MySQL cluster. It is based on a running mysqld process. This daemon accepts connections from client programs, such as MySQL monitor, <MySQL_Installation_Directory>/bin/mysql , which provides an interface for executing SQL code. The environment used for experiments described here is depicted in Figure 1. Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms
  • 7 Configuration of a MySQL Cluster Sun Microsystems, Inc. Figure 1. MySQL Cluster Topology MySQL Cluster Configuration Files MySQL Server Configuration File: my.cnf The MySQL server configuration file must exist on each member of a MySQL cluster and defines the roles in the cluster that each member has. Multiple my.cnf files can exist on one system, which are processed in a hierarchical manner: ● A my.cnf file with global settings must be located in /etc or in /etc/mysql 5. ● Server-specific options can be supplied in a my.cnf that resides in the installation path of the MySQL software or in its data directory. ● User-specific options are interpreted from a my.cnf file in the user's home directory. In the setup described in this guide, one single my.cnf file in /etc/mysql was used. my.cnf for the Management Node The following entries were made in a my.cnf file that defines the management node of a MySQL cluster. While the [mysqld] section exists in the template, the sections [ndb_mgm] and [ndb_mgmd] need to be added, which both define the role as cluster management node of this server in the cluster. [mysqld] user = mysql ndbcluster 5 MySQL 5.1 Reference Manual, revision: 10113, 2008-03-04. See also http://dev.mysql.com/doc/refman/5.1/en/ for the most current, online version. Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms
  • 8 Configuration of a MySQL Cluster Sun Microsystems, Inc. ndb-connectstring = 10.9.166.30 datadir = /export/mysql/MySQL_Data [ndb_mgm] connect-string = 10.9.166.30 [ndb_mgmd] config-file = /etc/mysql/config.ini The entries in [mysqld] are: ● user = mysql – This setting lets the mysqld daemon run in the context of the user mysql. ● ndbcluster – This server can act as an SQL node in a MySQL cluster. DDL statements can use the NDB storage engine. Although the binaries of the installation are compiled with the ndbcluster option, the NDB storage engine is not automatically available when the MySQL server is started. The NDB storage engine is activated by this entry in the my.cnf , or by specifying the --ndbcluster argument in the start command of the MySQL server. To verify that the NDB storage engine is present and usable, the command show engines can be executed using MySQL monitor, as shown in Figure 2. Figure 2. Output of a show engines Query ● ndb-connectstring – This SQL node tries to contact the management node running on a system with the IP address specified. A connection to such a management node is established if (1) this SQL node is listed and specified in the management server's config.ini or (2) the management server has "blank" entries for unspecified SQL nodes. No user authentication is required. ● datadir – This is the path to the files and directories of a MySQL server that the script mysql_install_db creates. The entry connect-string in the section [ndb_mgm] specifies the IP address of the server hosting the NDB management daemon ndb_mgmd that the cluster management console is supposed to talk to. The section [ndb_mgmd] indicates that the host acts as cluster management node. The value for config-file points to the location of the MySQL cluster configuration file config.ini . my.cnf for the Storage Nodes The section [mysql_cluster] signals that the host acts as storage node in the cluster. The ndb-connectstring value points to the management node from which a cluster storage node shall obtain its cluster-specific configuration. Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms
  • 9 Configuration of a MySQL Cluster Sun Microsystems, Inc. [mysqld] user = mysql ndbcluster ndb-connectstring = 10.9.166.30 datadir = /export/mysql/MySQL_Data [mysql_cluster] ndb-connectstring = 10.9.166.30 MySQL Cluster Configuration File: config.ini The topology and the configuration of a MySQL cluster is defined and maintained in one single file on the management node: config.ini . The location of this MySQL cluster configuration file is specified in the my.cnf of the cluster management node. The cluster storage nodes receive their individual configuration from their management node upon startup. Any modification, therefore, requires a restart of the affected nodes. In special cases, for example, when settings of REDO logs shall be changed, the affected data node must be restarted with the option --initial , which erases the contents of the ndb_<NodeID>_fs folder, so that all data of databases using the NDB engine on this node are lost unless an active replica node is available. Therefore, care should be taken when issuing the command: <MySQL_Installation_Directory>/libexec/ndbd --initial A template for a two-data-node cluster config.ini file (ndb-config-2-node.ini in <MySQL_Install_Directory>/share/mysql ), which comes with the installation of the MySQL software, was used to produce the operative version for the tests described herein. The config.ini used for the test environment covers five aspects: ● General properties of the cluster ● Management node ● Storage nodes ● SQL nodes ● TCP/IP settings General properties of the cluster are listed in the [ndbd default] section. This section specifies: ● How many replicas shall be produced and maintained: NoOfReplicas . Possible values are 1 (no redundancy) and 2. ● How much physical memory shall be assigned to MySQL: DataMemory , IndexMemory . The values for these parameters depend on the total amount of RAM available on the host. ● The location of cluster logs and other files of the cluster: DataDir . Note that my.cnf uses a parameter with the same name for specifying the location for files and directories that a mysqld daemon relies on. In the setup used for tests described in this paper, different locations were supplied for the DataDir of mysqld (in my.cnf) and ndbd (in config.ini ). Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms
  • 10 Configuration of a MySQL Cluster Sun Microsystems, Inc. Settings for storage nodes can be defined globally, that is, for all storage nodes, in the [ndbd default] section, or individually for a single storage node in separate [ndbd] sections. For each storage node, an extra [ndbd] section must exist, which at least contains a HostName entry. A SQL node is specified with its host name in a separate [mysqld] section. If a [mysqld] section does not contain a HostName entry, any MySQL server can connect to the cluster if its my.cnf file contains a ndb- connectstring entry with the valid IP address of the cluster management node. It is recommended to define one [mysqld] object in config.ini that does not have a specified host entry, which is helpful, for example, for ndb_restore operations. This is the initial version of config.ini : [ndbd default] NoOfReplicas= 2 MaxNoOfConcurrentOperations= 10000 DataMemory= 2048M IndexMemory= 512M TimeBetweenWatchDogCheck= 30000 DataDir= /export/mysql/mysql-cluster MaxNoOfOrderedIndexes= 512 [ndb_mgmd default] DataDir= /export/mysql/mysql-cluster [ndb_mgmd] Id=1 HostName= 10.9.166.30 [ndbd] Id= 2 HostName= 10.9.166.28 [ndbd] Id= 3 HostName= 10.9.166.29 [mysqld] #Id= 4 HostName= 10.9.166.28 [mysqld] #Id= 5 HostName= 10.9.166.29 [mysqld] HostName= 10.9.166.30 [mysqld] # One empty entry recommended [tcp default] PortNumber= 63132 Depending on the database load characteristics, default settings must be changed by modifying existing entries or adding new entries in the appropriate sections to override default values. In the following, some workloads have been executed on the MySQL cluster, and the adjustments needed in the config.ini file for a smooth operation are described. All tests were run with tables using disk-based storage. No attempts were made to optimize performance during these tests. Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms
  • 11 Configuration of a MySQL Cluster Sun Microsystems, Inc. Scenario 1: Table With 3 Columns, Simple Records The table used for the tests with insert, update, and index creation is described in Figure 3. Figure 3. Schema of the Test Table Test #1: Inserting 10 Million Records An SQL procedure with a loop was used to create 10 million records. Result: No modifications in config.ini are necessary. Test #2: Update 10 Million Records An SQL statement updates the Date field of all records. This attempt produces errors of the following kind: ● Got temporary error 233 'Out of operation records in transaction coordinator' from NDBCLUSTER ● Got temporary error 1217 'Out of operation records in local data manager' from NDBCLUSTER Result: Two parameters for the configuration of data nodes have default values that are not sufficient to run an update statement for millions of records: ● MaxNoOfConcurrentOperations (Default: 32768). This is the number of records that is updated simultaneously in a transaction. ● MaxNoOfLocalOperations (Default: 1.1 * MaxNoOfConcurrentOperations ) By default, MaxNoOfLocalOperations is a function of MaxNoOfConcurrentOperations , so only the latter needs to be changed in config.ini . When the value for MaxNoOfConcurrentOperations was increased to 10,000,000 (the number of records that shall be updated in one transaction), the storage nodes could not be started due to a lack of memory (error 2327). This situation might depend on the amount of RAM assigned to the MySQL software and the capacity in RAM allocated by table data. The update statement must, therefore, have a where clause that the number of records to be updated keeps within the limit of a tolerable value for MaxNoOfConcurrentOperations . In the test environment, updating 500,000 rows in a single transaction was possible using the following setting: ● MaxNoOfConcurrentOperations = 1000000 Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms
  • 12 Configuration of a MySQL Cluster Sun Microsystems, Inc. Test #3: Inserting 2x 10 Million Records Into a Table From Two Client Connections in Parallel Result: One parameter in section [ndbd default] needs to be increased to avoid error 1205 (Lock wait timeout exceeded ): ● TransactionDeadlockDetectionTimeout (default: 1200 ms). The value for TransactionDeadlockDetectionTimeout was increased to 10000, and the parallel execution of the insert statement went smoothly. Test #4: Inserting 3x 10 Million Records Into a Table From Three Client Connections in Parallel Result: No changes need to be made to config.ini to have the three insert statements complete successfully. Test #5: Table With 10 Million Rows: Creating an Index An additional index was created on the Firstname field using an ALTER TABLE statement. Result: No modification in config.ini is necessary to run the statement. Scenario 2: Table With BLOB Field The table used for tests with BLOB data in the MySQL software has the schema shown in Figure 4. Figure 4. Schema of the Table With BLOB Field Test #1: Insert 5000 Records With a Binary File Five thousand individual binary files6 with file sizes ranging from 512 Kbytes to 3012 Kbytes were inserted using the LOAD_FILE option in the INSERT INTO clause using a script with individual INSERT INTO statements. Result: A data node was forced to shut down because the default value for SendBufferMemory was too low. The largest file that could be inserted was 1624 Kbytes in size. 6 Binary files created with the command dd if=/dev/urandom of=<Path_of_File> bs=512 count=<N> Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms
  • 13 Configuration of a MySQL Cluster Sun Microsystems, Inc. The following parameter was added in the [tcp default] section of the cluster configuration file: ● SendBufferMenory = 10M Error 1220 was thrown while loading data into the table: Got temporary error 1220 'REDO log files overloaded, consult online manual (increase FragmentLogFileSize)' from NDBCLUSTER . The following parameter was added to the [ndbd default] section of config.ini : ● FragmentLogFileSize (default: 16 Mbytes) The value for FragmentLogFileSize was set to 64 Mbytes. Error 1221 was thrown while loading data into the table: Temporary error: 1221 'REDO buffers overloaded, consult online manual (increase RedoBuffer)' . The following parameter was added to the [ndbd default] section of config.ini : ● RedoBuffer (default: 8 Mbytes) The value for RedoBuffer was increased to 64 Mbytes. Starting and Stopping Components of a MySQL Cluster It is mandatory to start the cluster management daemon (ndb_mgmd ) first because the storage nodes get their cluster configuration from the config.ini file on the management node. This daemon is launched by executing the following command: <MySQL_Installation_Directory>/libexec/ndb_mgmd The management console is brought up with the following command: <MySQL_Installation_Directory>/bin/ndb_mgm The initial state of the cluster environment described herein can be displayed by the command show, as shown in Figure 5. Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms
  • 14 Configuration of a MySQL Cluster Sun Microsystems, Inc. Figure 5. MySQL Cluster Management Console The cluster consists of two data nodes [ndbd(NDB)] and three SQL nodes [mysqld(API)] . All data nodes and two SQL nodes are bound to specified IP addresses (that is, connections from other MySQL servers, even when having suitable entries in their my.cnf , will be rejected). One SQL node connection is defined in the config.ini file without an IP address, so that any host with a mysqld process running and with an appropriate ndb-connectstring in its my.cnf can establish a connection with the management daemon of this MySQL cluster and act as an SQL node. When the cluster management daemon is running, the storage node daemon (ndbd) can be started. If the ndbd process cannot contact the management node specified in the ndb-connectstring of the [mysql-cluster] section of its my.cnf, an error message occurs: Unable to connect with connect string: nodeid=0, <IP address>:1186. Retrying every 5 seconds . The execution of <MySQL_Installation_Directory>/libexec/ndbd launches two processes called ndbd: The first ndbd process (angel process) watches its child ndbd process and restarts it when it detects that the child process is no longer active. After starting the ndbd daemon on the storage nodes, the show command in the cluster management console displays the status shown in Figure 6. Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms
  • 15 Configuration of a MySQL Cluster Sun Microsystems, Inc. Figure 6. MySQL Cluster Running The MySQL cluster management console is the preferred tool for stopping single data nodes or for shutting down the whole cluster. A single data node can be stopped by issuing the command <NodeID> stop ; the whole cluster is stopped with the command shutdown , as shown in Figure 7. Figure 7. Shutting Down MySQL Cluster The MySQL cluster can be run independently from any MySQL server process. Such a process is needed only to access databases that use the NDB storage engine and retrieve or modify records and database objects. It is recommended to start an SQL node using the following script: <MySQL_Installation_Directory>/bin/mysqld_safe & It is also recommended that the mysqld process runs in the context of a user other than root, for example, in the context of the user mysql (see the section about file ownership on page 16). If the my.cnf file does not contain an entry like user=mysql , the command shown above can be executed with the option --user=mysql . Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms
  • 16 Configuration of a MySQL Cluster Sun Microsystems, Inc. Alternatively, the shell script mysql.server in <MySQL_Installation_Directory>/share/mysql can be copied to /etc/init.d so that the MySQL server process is automatically started when the server is restarted. On the server running the Solaris 10 OS for x64 platforms, the following adjustments were made in order to make the MySQL server start script executable: ● Replaced the first line by the expression #!/bin/bash ● Specified the MySQL installation directory in the variable basedir ● Specified the path to the database files created by mysql_install_db in the variable datadir ● Specified the path of the my.cnf file in the variable conf The mysqld process on an SQL node can be stopped using the following command: <MySQL_Installation_Directory>/bin/mysqladmin -u root shutdown Since the storage node process requires that the cluster management node daemon is up and running, it does not appear reasonable to provide an analogous start script for the ndbd daemon. Cluster Messages and Error Log Files The error log files of the MySQL cluster are located in the directory specified in the DataDir parameters for data and management nodes of the config.ini file. The management node maintains the ndb_1_cluster.log , while each data node has an ndb_<NodeID>_error.log and trace files, if errors have occurred. The value for NodeID refers to the numbering provided in the management console view (see Figure 6). The cluster log and the error logs are rolled over when their size reaches a threshold value. User Accounts and File Ownership The context of the user, under which a mysqld or ndbd daemon is started, is critical. SQL node: After the installation of the MySQL software, it is recommended to run the script mysql_install_db as user mysql and not as user root . This user becomes the owner of the database directories and files, which are created by the script. If the MySQL server is started with a different user ID, the MySQL server cannot access databases and tables and produces errors upon startup. Storage node: All content in the Datadir of the ndbd daemon are owned by the user that launched the ndbd daemon. There is no recommendation regarding a preferred user for running ndbd as long as the same user is always starting this daemon. In the tests with the MySQL cluster described herein, the ndbd process was launched as user root. Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms
  • 17 Configuration of a MySQL Cluster Sun Microsystems, Inc. Storage Management in a MySQL Cluster Prior to MySQL software version 5.1, all database data using the NDB storage engine was kept in the physical memory of the storage nodes assigned to the ndbd daemon. With version 5.1, it is possible to store table content in columns that are not indexed in disk-based tablespaces, while the indexed columns of a table reside in memory. The keyword STORAGE DISK in a CREATE TABLE statement in conjunction with the ENGINE=NDB clause indicates that columns without index shall allocate pages on disk rather than in memory. The amount of RAM that is at disposal for a MySQL cluster is specified by the parameters DataMemory and IndexMemory in the [ndbd] section of the config.ini . The utilization of physical memory by data and indexes can be determined by executing the command all dump 1000 on the MySQL cluster management console7 (see Figure 8). Figure 8. MySQL Cluster Management: All Dump 1000 The output of an all dump 1000 command is written to the MySQL cluster log, and shows the utilization of the assigned memory for data and indexes as a percentage and as the number of pages, respectively, and might look like the following: 2008-05-21 03:39:25 [MgmSrvr] INFO -- Node 2: Data usage is 42(27885 32K pages of total 65536) 2008-05-21 03:39:25 [MgmSrvr] INFO -- Node 2: Index usage is 47(30959 8K pages of total 65568) 2008-05-21 03:39:25 [MgmSrvr] INFO -- Node 2: Resource 0 min: 0 max: 639 curr: 269 2008-05-21 03:39:25 [MgmSrvr] INFO -- Node 2: Resource 1 min: 0 max: 0 curr: 259 2008-05-21 03:39:25 [MgmSrvr] INFO -- Node 2: Resource 2 min: 0 max: 0 curr: 10 2008-05-21 03:39:25 [MgmSrvr] INFO -- Node 3: Data usage is 42(27883 32K pages of total 65536) 2008-05-21 03:39:25 [MgmSrvr] INFO -- Node 3: Index usage is 47(30958 8K pages of total 65568) 2008-05-21 03:39:25 [MgmSrvr] INFO -- Node 3: Resource 0 min: 0 max: 639 curr: 269 2008-05-21 03:39:25 [MgmSrvr] INFO -- Node 3: Resource 1 min: 0 max: 0 curr: 259 2008-05-21 03:39:25 [MgmSrvr] INFO -- Node 3: Resource 2 min: 0 max: 0 curr: 10 Information about the utilization of disk space allocated in data files of tablespaces and undo files of log file groups can be retrieved from the table INFORMATION_SCHEMA.Files (see Figure 9). 7 In planned future versions, the command ALL REPORT MEMORYUSAGE would do the same. Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms
  • 18 Configuration of a MySQL Cluster Sun Microsystems, Inc. Figure 9. Data Files and Undo Files: Utilization of Disk-Based Storage Tablespaces of the NDB engine do not have an autoextend feature. When creating a data file of a tablespace using the CREATE TABLESPACE or the ALTER TABLESPACE statement, either a file name or the full path of the data file must be supplied. If only a file name is supplied, the file is created in the default path <DataDir>/ndb_<NodeID_fs> . The same applies to undo files that are created for a log file group. If undo files shall be physically separated from data files to minimize concurrent I/O during insert and update activities, the full path of the undo/data file must be supplied in the CREATE LOGFILE GROUP and CREATE TABLESPACE statements, respectively. In a MySQL 5.1 cluster database with tables using disk-based storage, three types of logging can occur: 1. REDO logging. The data nodes of a MySQL cluster use, in the default configuration, 16 sets of 4x16 Mbyte files for circular logging, that is, old log entries can be overwritten if three local checkpoints have been completed. The default size of 1 Gbyte in total for REDO logs can be adjusted by supplying custom [ndbd] parameters. With MySQL REDO logs, there is no danger of a continuous growth of disk space consumption. 2. UNDO logging. If data nodes of a MySQL cluster use disk data objects, that is, log file groups and tablespaces in conjunction with a STORAGE DISK clause, disk space is allocated in undo files. Because these undo files cannot be automatically extended, there is no immediate risk of filling up the file system over time. 3. Binary logging. The setting for log-bin in the my.cnf file ensures that all statements that manipulate data and database objects are logged. These log files are written to the Datadir target of the mysqld daemon on SQL nodes of a MySQL cluster. This binary log is needed for recovery after restoring the database from a backup and for replication. In contrast to the logging mechanisms mentioned before, if not maintained, these binary logs grow, accumulate, and might consume a significant amount of disk space on SQL nodes. Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms
  • 19 Glossary Sun Microsystems, Inc. Glossary The following table contains common abbreviations and acronyms that are used in this paper. Abbreviation Explanation BLOB Binary Large Object DDL Data Definition Language LAMP Linux, Apache, MySQL, Perl|Python|PHP NDB Network Database RDBMS Relational Database Management System SQL Structured Query Language For More Information Here are additional resources: ● Documentation: ● MySQL documentation site: http://dev.mysql.com/doc/ ● MySQL Cluster 5.1 - New Features white paper: http://www.mysql.com/why-mysql/white-papers/mysql_wp_cluster51.php ● MySQL Cluster Architecture Overview white paper: http://www.mysql.com/why-mysql/white-papers/cluster-technical.php ● MySQL training courses: http://www.sun.com/software/products/mysql/training.jsp ● MySQL download sites: ● http://www.sun.com/software/products/mysql/getit.jsp ● http://dev.mysql.com/downloads/mysql/5.1.html ● MySQL forums: http://forums.mysql.com/ ● Other web sites and resources: ● MySQL site on sun.com: http://www.sun.com/software/products/mysql/index.jsp ● mysql.com site: http://www.mysql.com/ ● Features & Benefits - MySQL Cluster: http://www.sun.com/software/products/mysql/cluster.jsp ● MySQL and MySQL Cluster FAQs: http://www.sun.com/software/products/mysql/faqs.jsp ● Sun BluePrintsTM article: Optimize MySQL Server on Sun x64 Servers and Storage http://wikis.sun.com/display/BluePrints/Optimize+MySQL+Server+on+Sun+x64+Serv ers+and+Storage Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms
  • 20 For More Information Sun Microsystems, Inc. ● Resources on BigAdmin: ● Setting Up MySQL Cluster Software Using Solaris Zones Partitioning Technology: http://www.sun.com/bigadmin/features/articles/mysql_cluster_zones.jsp ● MySQL Resources for System Administrators: http://www.sun.com/bigadmin/topics/mysql/ ● Creating a Script for Scheduled MySQL Backups (community submission): http://www.sun.com/bigadmin/content/submitted/sched_mysql_backup.jsp ● Database Resource Collection: http://www.sun.com/bigadmin/collections/database.html ● Resources on Sun Developer Network (SDN): ● Using MySQL and Memcached on the GlassFish Application Server: http://developers.sun.com/appserver/reference/techart/mysql_gf/ ● Sun and MySQL: How It Stacks Up for Developers: http://java.sun.com/developer/technicalArticles/mysql_acq/index.html ● Support: ● Register your Sun gear: https://inventory.sun.com/inventory/ ● Services: http://www.sun.com/service/ ● SunSolveSM Online: http://sunsolve.sun.com ● Events of interest to users of Sun products: ● Sun and MySQL Around the World--Meetup Mashup: http://www.sun.com/events/mysqltour/index.jsp ● Find a MySQL Meetup Group near you: http://mysql.meetup.com/ ● Worldwide developer events and Sun Tech Days: http://developers.sun.com/events/ ● Current events: http://www.sun.com/events/index.jsp Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms
  • 21 Acknowledgements Sun Microsystems, Inc. Acknowledgements We want to thank the following colleagues for their kind help, assistance, and support during the work on this paper: Ryan Arneson, Sun Microsystems in Broomfield, CO, for preparing the Sun Fire X4100 M2 servers, and for helping immediately when a hardware piece caused trouble. Matthias Schmidt, Sun Microsystems in Walldorf, for his help in preparing the shell scripts for data generation. Jürgen Zimmermann, Sun Microsystems in Walldorf, for his support during the attempts to compile the MySQL source code. Ritu Kamboj, Sun Microsystems in Menlo Park, for helping with the parameters for the 64-bit compilation. Hartmut Holzgraefe, Sun Microsystems, for reviewing the paper and for his valuable comments. Revision History Revision Notes Date 0.1 Draft for Revision 29.05.2008 1.0 First Version 16.06.2008 Licensing Information Unless otherwise specified, the use of this software is authorized pursuant to the terms of the license found at http://www.sun.com/bigadmin/common/berkeley_license.html. Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms