MySQL High Availability with Group ReplicationNuno Carvalho
MySQL Group Replication is a multi-master update everywhere replication plugin that provides high availability. It removes the need for handling server failover, provides fault tolerance, and automates group reconfiguration. Transactions are replicated to all group members, with conflicts detected and resolved using a first committer wins rule. Failed members automatically rejoin the group and synchronize with the others transparently. Group Replication uses the standard MySQL and InnoDB architecture, so existing users will feel familiar. It also supports features like auto-increment handling, GTIDs, secure connections, and a new single primary mode.
Mix ‘n’ Match Async and Group Replication for Advanced Replication SetupsPedro Gomes
The document discusses mixing asynchronous replication and group replication for advanced MySQL replication setups. It provides an overview of asynchronous replication, semi-synchronous replication, multi-source replication, and group replication. It then discusses some basic scenarios for mixing these technologies, such as using asynchronous replication for read scaling beyond the group's 9 member limit or aggregating data from multiple groups. The document also covers migrating from asynchronous to group replication and migrating disjoint servers into a group.
What you wanted to know about MySQL, but could not find using inernal instrum...Sveta Smirnova
This document discusses various ways that MySQL could provide more detailed instrumentation and troubleshooting information. It begins by noting that while modern versions have online documentation, errors are often discovered from log files after context is lost. It then provides examples of specific bugs filed requesting additional metrics around areas like connections, replication, tracing, and temporary tables to help troubleshoot issues. The document argues that more visibility into internal operations could help identify causes of problems.
MySQL High Availability with Group ReplicationNuno Carvalho
MySQL Group Replication is a multi-master update everywhere replication plugin that provides high availability. It removes the need for handling server failover, provides fault tolerance, and automates group reconfiguration. Transactions are replicated to all group members, with conflicts detected and resolved using a first committer wins rule. Failed members automatically rejoin the group and synchronize with the others transparently. Group Replication uses the standard MySQL and InnoDB architecture, so existing users will feel familiar. It also supports features like auto-increment handling, GTIDs, secure connections, and a new single primary mode.
Mix ‘n’ Match Async and Group Replication for Advanced Replication SetupsPedro Gomes
The document discusses mixing asynchronous replication and group replication for advanced MySQL replication setups. It provides an overview of asynchronous replication, semi-synchronous replication, multi-source replication, and group replication. It then discusses some basic scenarios for mixing these technologies, such as using asynchronous replication for read scaling beyond the group's 9 member limit or aggregating data from multiple groups. The document also covers migrating from asynchronous to group replication and migrating disjoint servers into a group.
What you wanted to know about MySQL, but could not find using inernal instrum...Sveta Smirnova
This document discusses various ways that MySQL could provide more detailed instrumentation and troubleshooting information. It begins by noting that while modern versions have online documentation, errors are often discovered from log files after context is lost. It then provides examples of specific bugs filed requesting additional metrics around areas like connections, replication, tracing, and temporary tables to help troubleshoot issues. The document argues that more visibility into internal operations could help identify causes of problems.
MySQL Replication Performance Tuning for Fun and Profit!Vitor Oliveira
MySQL Replication, in addition to bringing high-availability, is the foundation to build high-performance MySQL database systems. Using read scale-out and sharding one can design systems that go from the capacity of a single server to supporting the largest internet sites. But to design and operate high-performance, efficient, manageable and reliable deployments requires knowing the intricacies of the underlying technologies.
This session will provide insights on the main factors that affect the performance of Asynchronous Replication and Group Replication, and how to configure them to make the most out of the underlying computing system. It will also show the latest developments in MySQL 5.7 and 8.0, in areas spanning from group communication to the multi-threaded slave applier, and how effective they are in helping meet the performance requirements in terms of throughput, latency and durability to support the most demanding workload types.
This document provides an overview of techniques for capturing and analyzing SQL queries in MySQL databases. It discusses built-in MySQL options like the slow query log, general query log, and binary log. It also covers other techniques like using MySQL Proxy, TCP/IP capture, Dtrace/SystemTap, and application management. Specific examples are provided for slow query log output, the general query log from WordPress, and the binary log and processlist.
Inno db internals innodb file formats and source code structurezhaolinjnu
This document discusses the goals, architecture, and on-disk format of the InnoDB storage engine for MySQL. InnoDB aims to provide transaction support, reliability, and scalability. Its architecture includes buffering, locking, recovery, and efficient I/O mechanisms. The on-disk format is designed for durability, performance through techniques like compression, and compatibility through file format management. Source code is organized into subdirectories corresponding to major subsystems.
MySQL High Availability and Disaster Recovery with Continuent, a VMware companyContinuent
Users seeking high availability, disaster recovery and zero downtime maintenance operation for business-critical MySQL applications face confusing choices. Is multi-master or master/slave clustering better? What about synchronous versus asynchronous replication? Using a plain vanilla, stock MySQL or a modified version of it? Which of these choices are right for data-driven businesses that depend on fast, reliable data access?
This no-BS webinar cuts through the FUD to explore the real trade-offs between the different clustering and replication methods, thens show you how Continuent's asynchronous master/slave clusters support these important capabilities for business-critical applications:
- High application write rates Master/slave clustering with Continuent
- Mixed workloads consisting of large and small transactions
- Data across multiple geographically distributed locations
- Failures and more importantly recovery from them
- Zero downtime maintenance and software upgrades
- Use of off-the-shelf MySQL/MariaDB to avoid application changes and allow clusters to improve as MySQL itself does.
We illustrate key points with demonstrations and case studies from deployed systems.
Designing an extensible, flexible schema that supports user customization is a common requirement, but it's easy to paint yourself into a corner.
Examples of extensible database requirements:
- A database that allows users to declare new fields on demand.
- Or an e-commerce catalog with many products, each with distinct attributes.
- Or a content management platform that supports extensions for custom data.
The solutions we use to meet these requirements is overly complex and the performance is terrible. How should we find the right balance between schema and schemaless database design?
I'll briefly cover the disadvantages of Entity-Attribute-Value (EAV), a problematic design that's an example of the antipattern called the Inner-Platform Effect, That is, modeling an attribute-management system on top of the RDBMS architecture, which already provides attributes through columns, data types, and constraints.
Then we'll discuss the pros and cons of alternative data modeling patterns, with respect to developer productivity, data integrity, storage efficiency and query performance, and ease of extensibility.
- Class Table Inheritance
- Serialized BLOB
- Inverted Indexing
Finally we'll show tools like pt-online-schema-change and new features of MySQL 5.6 that take the pain out of schema modifications.
The document discusses high availability and scalability in MySQL. It describes various techniques for achieving high availability including replication, clustering, and shared storage solutions. It also discusses different approaches for scaling MySQL including replication, sharding, and clustering. MySQL replication is described as asynchronous with a single master and multiple read-only slaves. MySQL Cluster provides synchronous replication across nodes and automatic failover for high availability.
We all have tasks from time to time for bulk-loading external data into MySQL. What's the best way of doing this? That's the task I faced recently when I was asked to help benchmark a multi-terrabyte database. We had to find the most efficient method to reload test data repeatedly without taking days to do it each time. In my presentation, I'll show you several alternative methods for bulk data loading, and describe the practical steps to use them efficiently. I'll cover SQL scripts, the mysqlimport tool, MySQL Workbench import, the CSV storage engine, and the Memcached API. I'll also give MySQL tuning tips for data loading, and how to use multi-threaded clients.
Group Replication: A Journey to the Group Communication CoreAlfranio Júnior
This document discusses the journey of MySQL Group Replication from using a third-party group communication system (Corosync) to its own built-in group communication engine (XCOM) based on Paxos algorithms. It describes the key components of MySQL Group Replication including the group communication interface, XCOM engine, and optimizations in XCOM like pipelining and batching. It also provides performance results of MySQL Group Replication with multiple writers.
This document provides an overview and agenda for a presentation on MySQL best practices for DBAs and developers. The presentation covers essential MySQL configuration practices like server SQL mode and storage engines. It also discusses improving SQL, user security, schema optimizations, instrumentation, and monitoring. Specific topics include comment SQL, formatting SQL, future proofing SQL, and analyzing SQL. The document provides examples and recommendations for each topic.
A presentation about how to make MySQL highly available, presented at the San Francisco MySQL Meetup (http://www.sfmysql.org/events/15760472/) on January 26th, 2011.
A video recording of this presentation is available from Ustream: http://ustre.am/fyLk
MySQL Group Replication - HandsOn TutorialKenny Gryp
Group Replication is a plugin for MySQL that provides multi-master replication. It works by having each node send write transactions to other nodes through a group communication system. The writes are certified locally in an asynchronous manner to ensure total order of transactions across all nodes. Group Replication uses optimistic locking where local locks are released right after commit, and conflict detection happens during certification rather than at the start of transactions.
Using Apache Spark and MySQL for Data AnalysisSveta Smirnova
The document discusses using Apache Spark and MySQL for data analysis. It provides examples of loading Wikipedia usage statistics (Wikistats) data into both MySQL and Spark for analysis. Loading the full 10+ TB of Wikistats data into MySQL took over a month, while Spark was able to scan and analyze the entire dataset in under an hour by leveraging its ability to perform distributed, parallel processing across multiple nodes. The document compares key differences between Spark and MySQL for big data processing, such as Spark's lack of indexes but ability to perform full scans in parallel across nodes.
This document discusses troubleshooting an error of "Too many open files" when running MySQL. It describes how the open_files_limit parameter was not taking effect due to MySQL setting the file descriptor limit internally based on other factors like max_connections. Through debugging with GDB and tracing the code, it was found that MySQL was calling setrlimit() to set the limit based on the lower of the configured open_files_limit or the OS file limit. The solution involved configuring the ulimit directly via mysqld_safe to set the limit before MySQL started.
Multi Source Replication With MySQL 5.7 @ VerisureKenny Gryp
Verisure migrated their data warehouse from using Tungsten Replicator to native multi-source replication in MySQL 5.7 to simplify operations. They loaded data from production shards into the new data warehouse setup using XtraBackup backups and improved replication capacity with MySQL's parallel replication features. Some issues were encountered with replication lag reporting and crashes during the upgrade but most were resolved. Monitoring and management tools also required updates to support the new multi-source replication configuration.
17. show table status Table 的各种信息,都在 tables_fields_info[] 数组里定义义 ,Name,Engine,Version,Row_format,Rows,Avg_row_lenght,Data_length 等等
18. show table status 再看 ST_SCHEMA_TABLE schema_tables[] 关于 tables 的定义 表的名字: TABLES 表结构的定义: tables_fields_info 表的创建函数: create_schema_table 填冲数据函数: by function get_all_tables and get_schema_tables_record
Mysql 的作者遍布全世界 root@(none) 01:45:16>SHOW AUTHORS; +--------------------------------+---------------------------------------+----------------------------------------------------------------------+ | Name | Location | Comment | +--------------------------------+---------------------------------------+----------------------------------------------------------------------+ | Brian (Krow) Aker | Seattle, WA, USA | Architecture, archive, federated, bunch of little stuff :) | | Venu Anuganti | | Client/server protocol (4.1) | | David Axmark | Uppsala, Sweden | Small stuff long time ago, Monty ripped it out! | | Alexander (Bar) Barkov | Izhevsk, Russia | Unicode and character sets (4.1) | | Omer BarNir | Sunnyvale, CA, USA | Testing (sometimes) and general QA stuff | | Guilhem Bichot | Bordeaux, France | Replication (since 4.0) | | John Birrell | | Emulation of pthread_mutex() for OS/2 | | Andreas F. Bobak | | AGGREGATE extension to user-defined functions | | Alexey Botchkov (Holyfoot) | Izhevsk, Russia | GIS extensions (4.1), embedded server (4.1), precision math (5.0) | | Reggie Burnett | Nashville, TN, USA | Windows development, Connectors | | Oleksandr Byelkin | Lugansk, Ukraine | Query Cache (4.0), Subqueries (4.1), Views (5.0) | | Kent Boortz | Orebro, Sweden | Test platform, and general build stuff | | Tim Bunce | | mysqlhotcopy | | Yves Carlier | | mysqlaccess | | Joshua Chamas | Cupertino, CA, USA | Concurrent insert, extended date syntax | | Petr Chardin | Moscow, Russia | Instance Manager (5.0), Server log tables (5.1) | | Wei-Jou Chen | | Chinese (Big5) character set | | Albert Chin-A-Young | | Tru64 port, large file support, better TCP wrappers support | | Jorge del Conde | Mexico City, Mexico | Windows development | | Antony T. Curtis | Norwalk, CA, USA | Parser, port to OS/2, storage engines and some random stuff | | Yuri Dario | | OS/2 port | | Andrei Elkin | Espoo, Finland | Replication | | Patrick Galbraith | Sharon, NH | Federated Engine, mysqlslap | | Sergei Golubchik | Kerpen, Germany | Full-text search, precision math | | Lenz Grimmer | Hamburg, Germany | Production (build and release) engineering | | Nikolay Grishakin | Austin, TX, USA | Testing - Server | | Wei He | | Chinese (GBK) character set | | Eric Herman | Amsterdam, Netherlands | Bug fixing - federated | | Andrey Hristov | Walldorf, Germany | Event scheduler (5.1) | | Alexander (Alexi) Ivanov | St. Petersburg, Russia | Replication | | Alexander (Salle) Keremidarski | Sofia, Bulgaria | Bug fixing | | Mats Kindahl | Storvreta, Sweden | Replication | | Serge Kozlov | Velikie Luki, Russia | Testing - Cluster | | Hakan Kü?üky?lmaz | Walldorf, Germany | Testing - Server | | Greg (Groggy) Lehey | Uchunga, SA, Australia | Backup | | Matthias Leich | Berlin, Germany | Testing - Server | | Dmitri Lenev | Moscow, Russia | Time zones support (4.1), Triggers (5.0) | | Arjen Lentz | Brisbane, Australia | Documentation (2001-2004), Dutch error messages, LOG2() | | Marc Liyanage | | Created Mac OS X packages | | Zarko Mocnik | | Sorting for Slovenian language | | Per-Erik Martin | Uppsala, Sweden | Stored Procedures (5.0) | | Alexis Mikhailov | | User-defined functions | | Sinisa Milivojevic | Larnaca, Cyprus | UNION (4.0), Subqueries in FROM clause (4.1), many other features | | Jonathan (Jeb) Miller | Kyle, TX, USA | Testing - Cluster, Replication | | Elliot Murphy | Cocoa, FL, USA | Replication and backup | | Kristian Nielsen | Copenhagen, Denmark | General build stuff | | Pekka Nouisiainen | Stockholm, Sweden | NDB Cluster: BLOB support, character set support, ordered indexes | | Alexander Nozdrin | Moscow, Russia | Bug fixing (Stored Procedures, 5.0) | | Per Eric Olsson | | Testing of dynamic record format | | Jonas Oreland | Stockholm, Sweden | NDB Cluster, Online Backup, lots of other things | | Konstantin Osipov | Moscow, Russia | Prepared statements (4.1), Cursors (5.0) | | Alexander (Sasha) Pachev | Provo, UT, USA | Statement-based replication, SHOW CREATE TABLE, mysql-bench | | Irena Pancirov | | Port to Windows with Borland compiler | | Jan Pazdziora | | Czech sorting order | | Benjamin Pflugmann | | Extended MERGE storage engine to handle INSERT | | Igor Romanenko | | mysqldump | | Mikael Ronstr?m | Stockholm, Sweden | NDB Cluster, Partitioning (5.1), Optimizations | | T?nu Samuel | | VIO interface, other miscellaneous features | | Carsten Segieth (Pino) | Fredersdorf, Germany | Testing - Server | | Martin Sk?ld | Stockholm, Sweden | NDB Cluster: Unique indexes, integration into MySQL | | Timothy Smith | Auckland, New Zealand | Dynamic character sets, parts of the build system, libmysqld | | Miguel Solorzano | Florianopolis, Santa Catarina, Brazil | Windows development, Windows NT service | | Punita Srivastava | Austin, TX, USA | Testing - Merlin | | Alexey Stroganov (Ranger) | Lugansk, Ukraine | Testing - Benchmarks | | Ingo Strüwing | Berlin, Germany | Bug fixing | | Magnus Svensson | ?regrund, Sweden | NDB Cluster: Integration into MySQL, test framework | | Zeev Suraski | | FROM_UNIXTIME(), ENCRYPT() | | TAMITO | | The _MB character set macros and UJIS and SJIS character sets | | Jani Tolonen | Helsinki, Finland | mysqlimport, extensions to command-line clients, PROCEDURE ANALYSE() | | Lars Thalmann | Stockholm, Sweden | Replication and cluster development | | Tomas Ulin | Stockholm, Sweden | NDB Cluster: Configuration, installation | | Gianmassimo Vigazzola | | Initial Windows port | | Sergey Vojtovich | Izhevsk, Russia | Plugins infrastructure (5.1) | | Matt Wagner | Northfield, MN, USA | Bug fixing | | Jim Winstead Jr. | Los Angeles, CA, USA | Bug fixing | | Michael (Monty) Widenius | Tusby, Finland | Lead developer and main author | | Peter Zaitsev | Tacoma, WA, USA | SHA1(), AES_ENCRYPT(), AES_DECRYPT(), bug fixing | +--------------------------------+---------------------------------------+----------------------------------------------------------------------+
root@(none) 01:45:40>show contributors; +-------------------+---------------------+-------------------------------------+ | Name | Location | Comment | +-------------------+---------------------+-------------------------------------+ | Ronald Bradford | Brisbane, Australia | EFF contribution for UC2006 Auction | | Sheeri Kritzer | Boston, Mass. USA | EFF contribution for UC2006 Auction | | Mark Shuttleworth | London, UK. | EFF contribution for UC2006 Auction | +-------------------+---------------------+-------------------------------------+ 3 rows in set (0.01 sec)
root@test 03:11:33>show create database test; +----------+--------------------------------------------------------------+ | Database | Create Database | +----------+--------------------------------------------------------------+ | test | CREATE DATABASE `test` /*!40100 DEFAULT CHARACTER SET gbk */ | +----------+--------------------------------------------------------------+ 1 row in set (0.00 sec)
root@test 03:26:59>show privileges; +-------------------------+---------------------------------------+-------------------------------------------------------+ | Privilege | Context | Comment | +-------------------------+---------------------------------------+-------------------------------------------------------+ | Alter | Tables | To alter the table | | Alter routine | Functions,Procedures | To alter or drop stored functions/procedures | | Create | Databases,Tables,Indexes | To create new databases and tables | | Create routine | Databases | To use CREATE FUNCTION/PROCEDURE | | Create temporary tables | Databases | To use CREATE TEMPORARY TABLE | | Create view | Tables | To create new views | | Create user | Server Admin | To create new users | | Delete | Tables | To delete existing rows | | Drop | Databases,Tables | To drop databases, tables, and views | | Event | Server Admin | To create, alter, drop and execute events | | Execute | Functions,Procedures | To execute stored routines | | File | File access on server | To read and write files on the server | | Grant option | Databases,Tables,Functions,Procedures | To give to other users those privileges you possess | | Index | Tables | To create or drop indexes | | Insert | Tables | To insert data into tables | | Lock tables | Databases | To use LOCK TABLES (together with SELECT privilege) | | Process | Server Admin | To view the plain text of currently executing queries | | References | Databases,Tables | To have references on tables | | Reload | Server Admin | To reload or refresh tables, logs and privileges | | Replication client | Server Admin | To ask where the slave or master servers are | | Replication slave | Server Admin | To read binary log events from the master | | Select | Tables | To retrieve rows from table | | Show databases | Server Admin | To see all databases with SHOW DATABASES | | Show view | Tables | To see views with SHOW CREATE VIEW | | Shutdown | Server Admin | To shut down the server | | Super | Server Admin | To use KILL thread, SET GLOBAL, CHANGE MASTER, etc. | | Trigger | Tables | To use triggers | | Update | Tables | To update existing rows | | Usage | Server Admin | No privileges - allow connect only | +-------------------------+---------------------------------------+-------------------------------------------------------+
sql/Table.h /* Make sure that the order of schema_tables and enum_schema_tables are the same. */ enum enum_schema_tables { SCH_CHARSETS= 0, SCH_COLLATIONS, SCH_COLLATION_CHARACTER_SET_APPLICABILITY, SCH_COLUMNS, SCH_COLUMN_PRIVILEGES, SCH_ENGINES, SCH_EVENTS, SCH_FILES, SCH_GLOBAL_STATUS, SCH_GLOBAL_VARIABLES, SCH_KEY_COLUMN_USAGE, SCH_OPEN_TABLES, SCH_PARTITIONS, SCH_PLUGINS, SCH_PROCESSLIST, SCH_PROFILES, SCH_REFERENTIAL_CONSTRAINTS, SCH_PROCEDURES, SCH_SCHEMATA, SCH_SCHEMA_PRIVILEGES, SCH_SESSION_STATUS, SCH_SESSION_VARIABLES, SCH_STATISTICS, SCH_STATUS, SCH_TABLES, SCH_TABLE_CONSTRAINTS, SCH_TABLE_NAMES, SCH_TABLE_PRIVILEGES, SCH_TRIGGERS, SCH_USER_PRIVILEGES, SCH_VARIABLES, SCH_VIEWS }; 参考: http://forge.mysql.com/wiki/INFORMATION_SCHEMA_storage_engine
如何把一台 mysql 做成多台 master 的 slave?
各个版本的 mysql 所展示的 show innodb status 会有差异 5.1.45-log Source distribution Innodb plugin
/*********************************************************************//** Calculates new estimates for table and index statistics. The statistics are used in query optimization. */ UNIV_INTERN void dict_update_statistics_low( /*=======================*/ dict_table_t* table, /*!< in/out: table */ ibool has_dict_mutex __attribute__((unused))) /*!< in: TRUE if the caller has the dictionary mutex */ { dict_index_t* index; ulint size; ulint sum_of_index_sizes = 0; if (table->ibd_file_missing) { ut_print_timestamp(stderr); fprintf(stderr, &quot; InnoDB: cannot calculate statistics for table %s
&quot; &quot;InnoDB: because the .ibd file is missing. For help,&quot; &quot; please refer to
&quot; &quot;InnoDB: &quot; REFMAN &quot;innodb-troubleshooting.html
&quot;, table->name); return; } /* If we have set a high innodb_force_recovery level, do not calculate statistics, as a badly corrupted index can cause a crash in it. */ if (srv_force_recovery >= SRV_FORCE_NO_IBUF_MERGE) { return; } /* Find out the sizes of the indexes and how many different values for the key they approximately have */ // 先查找第一个索引,至少有一个索引 index = dict_table_get_first_index(table); if (index == NULL) { /* Table definition is corrupt */ return; } while (index) { // 取得当前索引大小精确值 size = btr_get_size(index, BTR_TOTAL_SIZE); index->stat_index_size = size; sum_of_index_sizes += size; // 取得当前索引叶子结点页数 size = btr_get_size(index, BTR_N_LEAF_PAGES); if (size == 0) { /* The root node of the tree is a leaf */ size = 1; } index->stat_n_leaf_pages = size; // 预测不同的索引键值个数 btr_estimate_number_of_different_key_vals(index); // 查找下一个索引 index = dict_table_get_next_index(index); } // 第一个索引,总是 cluster_index, 怪不得其它索引称为第二索引 index = dict_table_get_first_index(table); dict_index_stat_mutex_enter(index); // 表的行数,从索引的统计数据里获取 table->stat_n_rows = index->stat_n_diff_key_vals[ dict_index_get_n_unique(index)]; dict_index_stat_mutex_exit(index); // 主索引大小 cluster index table->stat_clustered_index_size = index->stat_index_size; // 其它索引总共大小 = 索引总大小 -cluster_index_size table->stat_sum_of_other_index_sizes = sum_of_index_sizes - index->stat_index_size; table->stat_initialized = TRUE; table->stat_modified_counter = 0; }