- The document discusses various parallel replication technologies in MySQL/MariaDB including schema-based parallel replication in MySQL 5.6, group commit-based approaches in MariaDB 10.0 and MySQL 5.7, and optimistic parallel replication in MariaDB 10.1.
- It provides an overview of how each approach tags and dispatches transactions to worker threads on slaves and their limitations regarding transaction ordering and gaps.
- Examples from Booking.com show how parallel replication can scale to thousands of servers but also hit issues like long transactions blocking progress.
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...Jean-François Gagné
Since 5.7.2, MySQL implements parallel replication in the same schema, also known as LOGICAL_CLOCK (DATABASE based parallel replication is also implemented in 5.6 but this is not covered in this talk). In early 5.7 versions, parallel replication was based on group commit (like MariaDB) and 5.7.6 changed that to intervals.
Intervals are more complicated but they are also more powerful. In this talk, I will explain in detail how they work and why intervals are better than group commit. I will also cover how to optimize parallel replication in MySQL 5.7 and what improvements are coming in MySQL 8.0. I will also explain why Group Replication is replicating faster than standard asynchronous replication.
Come to this talk to get all the details about MySQL 5.7 Parallel Replication.
MySQL Parallel Replication: inventory, use-case and limitationsJean-François Gagné
In the last 24 months, MySQL/MariaDB replication speed has improved a lot thanks to parallel replication. MySQL and MariaDB have different types of parallel replication; in this talk, I present the different implementations, with their limitations and the corresponding tuning parameters. I cover what to do to make parallel replication faster and what to avoid for maximizing parallel replication benefits. I also present benchmark results from real Booking.com workloads. Finally, I discuss some deployments at Booking.com that take advantage of parallel replication speed improvements.
MySQL Parallel Replication: inventory, use-case and limitationsJean-François Gagné
In the last 24 months, MySQL replication speed has improved a lot thanks to implementing parallel replication. MySQL and MariaDB have different types of parallel replication; in this talk, I present in details the different implementations, with their limitations and the corresponding tuning parameters. I also present benchmark results from real Booking.com workloads. Finally, I discuss some deployments at Booking.com that benefits from parallel replication speed improvements.
MySQL Parallel Replication: inventory, use-cases and limitationsJean-François Gagné
In the last 24 months, MySQL replication speed has improved a lot thanks to implementing parallel replication. MySQL and MariaDB have different types of parallel replication; in this talk, I present in detail the different implementations, with their limitations and the corresponding tuning parameters (covering MySQL 5.6, MariaDB 10.0, MariaDB 10.1 and MySQL 5.7). I also present benchmark results from real Booking.com workloads. Finally, I discuss some deployments at Booking.com that benefits from parallel replication speed improvements.
Riding the Binlog: an in Deep Dissection of the Replication StreamJean-François Gagné
Binary Logs are the cornerstone of MySQL Replication, but is it fully understood ? To start apprehending this, we can think of the binary logs as a transport for a Stream of Transactions. Traveling from master to slave, sometimes via Intermediate Masters, this stream evolves: it can shrink by the application of filters, can grow by the addition of slave-local transactions, and two streams can merge by the usage of multi-source replication. After presenting the binary logs Stream Model, the different MySQL use-cases will be mapped to the model, which can serve as a validation of the model. After this validation, the model will be used to make prediction on new use-cases/features that could emerge in the future.
Since 5.7.2, MySQL implements parallel replication in the same schema, also known as LOGICAL_CLOCK (DATABASE based parallel replication is also implemented in 5.6 but this is not covered in this talk). In early 5.7 versions, parallel replication was based on group commit (like MariaDB) and 5.7.6 changed that to intervals.
Intervals are more complicated but they are also more powerful. In this talk, I will explain in detail how they work and why intervals are better than group commit. I will also cover how to optimize parallel replication in MySQL 5.7 and what improvements are coming in MySQL 8.0.
MySQL/MariaDB replication is asynchronous. You can make replication faster by using better hardware (faster CPU, more RAM, or quicker disks), or you can use parallel replication to remove it single-threaded limitation; but lag can still happen. This talk is not about making replication faster, it is how to deal with its asynchronous nature, including the (in-)famous lag.
We will start by explaining the consequences of asynchronous replication and how/when lag can happen. Then, we will present the solution used at Booking.com to avoid both creating lag and minimize the consequence of stale reads on slaves (hint: this solution does not mean reading from the master because this does not scale).
Once all above is well understood, we will discuss how Booking.com’s solution can be improved: this solution was designed years ago and we would do this differently if starting from scratch today. Finally, I will present an innovative way to avoid lag: the no-slave-left-behind MariaDB patch.
This tutorial covers all parallel replication implementation in MariaDB 10.0 and 10.1 and MySQL 5.6, 5.7 and 8.0 (including how it works in Group Replication).
MySQL and MariaDB have different types of parallel replication. In this tutorial, we present the different implementations that allow us to understand their limitations and tuning parameters. We cover how to make parallel replication faster and what to avoid for maximizing its benefits. We also present tests from Booking.com workloads.
Some of the subjects that are covered are group commit and optimistic parallel replication in MariaDB, the parallelism interval of MySQL and its Write Set optimization, and the ?slowing down the master to speed up the slave? optimization.
After this tutorial, you will know everything you need to implement and tune parallel replication in your environment. But more importantly, we will show how you can test parallel replication benefit in a non-disruptive way before deployment.
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...Jean-François Gagné
Since 5.7.2, MySQL implements parallel replication in the same schema, also known as LOGICAL_CLOCK (DATABASE based parallel replication is also implemented in 5.6 but this is not covered in this talk). In early 5.7 versions, parallel replication was based on group commit (like MariaDB) and 5.7.6 changed that to intervals.
Intervals are more complicated but they are also more powerful. In this talk, I will explain in detail how they work and why intervals are better than group commit. I will also cover how to optimize parallel replication in MySQL 5.7 and what improvements are coming in MySQL 8.0. I will also explain why Group Replication is replicating faster than standard asynchronous replication.
Come to this talk to get all the details about MySQL 5.7 Parallel Replication.
MySQL Parallel Replication: inventory, use-case and limitationsJean-François Gagné
In the last 24 months, MySQL/MariaDB replication speed has improved a lot thanks to parallel replication. MySQL and MariaDB have different types of parallel replication; in this talk, I present the different implementations, with their limitations and the corresponding tuning parameters. I cover what to do to make parallel replication faster and what to avoid for maximizing parallel replication benefits. I also present benchmark results from real Booking.com workloads. Finally, I discuss some deployments at Booking.com that take advantage of parallel replication speed improvements.
MySQL Parallel Replication: inventory, use-case and limitationsJean-François Gagné
In the last 24 months, MySQL replication speed has improved a lot thanks to implementing parallel replication. MySQL and MariaDB have different types of parallel replication; in this talk, I present in details the different implementations, with their limitations and the corresponding tuning parameters. I also present benchmark results from real Booking.com workloads. Finally, I discuss some deployments at Booking.com that benefits from parallel replication speed improvements.
MySQL Parallel Replication: inventory, use-cases and limitationsJean-François Gagné
In the last 24 months, MySQL replication speed has improved a lot thanks to implementing parallel replication. MySQL and MariaDB have different types of parallel replication; in this talk, I present in detail the different implementations, with their limitations and the corresponding tuning parameters (covering MySQL 5.6, MariaDB 10.0, MariaDB 10.1 and MySQL 5.7). I also present benchmark results from real Booking.com workloads. Finally, I discuss some deployments at Booking.com that benefits from parallel replication speed improvements.
Riding the Binlog: an in Deep Dissection of the Replication StreamJean-François Gagné
Binary Logs are the cornerstone of MySQL Replication, but is it fully understood ? To start apprehending this, we can think of the binary logs as a transport for a Stream of Transactions. Traveling from master to slave, sometimes via Intermediate Masters, this stream evolves: it can shrink by the application of filters, can grow by the addition of slave-local transactions, and two streams can merge by the usage of multi-source replication. After presenting the binary logs Stream Model, the different MySQL use-cases will be mapped to the model, which can serve as a validation of the model. After this validation, the model will be used to make prediction on new use-cases/features that could emerge in the future.
Since 5.7.2, MySQL implements parallel replication in the same schema, also known as LOGICAL_CLOCK (DATABASE based parallel replication is also implemented in 5.6 but this is not covered in this talk). In early 5.7 versions, parallel replication was based on group commit (like MariaDB) and 5.7.6 changed that to intervals.
Intervals are more complicated but they are also more powerful. In this talk, I will explain in detail how they work and why intervals are better than group commit. I will also cover how to optimize parallel replication in MySQL 5.7 and what improvements are coming in MySQL 8.0.
MySQL/MariaDB replication is asynchronous. You can make replication faster by using better hardware (faster CPU, more RAM, or quicker disks), or you can use parallel replication to remove it single-threaded limitation; but lag can still happen. This talk is not about making replication faster, it is how to deal with its asynchronous nature, including the (in-)famous lag.
We will start by explaining the consequences of asynchronous replication and how/when lag can happen. Then, we will present the solution used at Booking.com to avoid both creating lag and minimize the consequence of stale reads on slaves (hint: this solution does not mean reading from the master because this does not scale).
Once all above is well understood, we will discuss how Booking.com’s solution can be improved: this solution was designed years ago and we would do this differently if starting from scratch today. Finally, I will present an innovative way to avoid lag: the no-slave-left-behind MariaDB patch.
This tutorial covers all parallel replication implementation in MariaDB 10.0 and 10.1 and MySQL 5.6, 5.7 and 8.0 (including how it works in Group Replication).
MySQL and MariaDB have different types of parallel replication. In this tutorial, we present the different implementations that allow us to understand their limitations and tuning parameters. We cover how to make parallel replication faster and what to avoid for maximizing its benefits. We also present tests from Booking.com workloads.
Some of the subjects that are covered are group commit and optimistic parallel replication in MariaDB, the parallelism interval of MySQL and its Write Set optimization, and the ?slowing down the master to speed up the slave? optimization.
After this tutorial, you will know everything you need to implement and tune parallel replication in your environment. But more importantly, we will show how you can test parallel replication benefit in a non-disruptive way before deployment.
Almost Perfect Service Discovery and Failover with ProxySQL and OrchestratorJean-François Gagné
Of course there is no such thing as perfect service discovery, and we will see why in the talk. However, the way ProxySQL is deployed in this case minimizes the risk for split-brains, and this is why I qualify it as almost perfect. But let’s step back a little...
MySQL alone is not a high availability solution. To provide resilience to primary failure, other components need to be integrated with MySQL. At MessageBird, these additional components are ProxySQL and Orchestrator. In this talk, we describe how ProxySQL is architectured to provide close to perfect Service Discovery and how this, combined with Orchestrator, allows for automatic failover. The talk presents the details of the integration of MySQL, ProxySQL and Orchestrator in Google Cloud (and it would be easy to re-implement a similar architecture at other cloud vendors or on-premises). We will also cover lessons learned for the 2 years this architecture has been in production. Come to this talk to learn more about MySQL high availability, ProxySQL and Orchestrator.
Have you ever needed to get some additional write throughput from MySQL ? If yes, you probably found that setting sync_binlog to 0 (and trx_commit to 2) gives you an extra performance boost. As all such easy optimisation, it comes at a cost. This talk explains how this tuning works, presents its consequences and makes recommendations to avoid them. This will bring us to the details of how MySQL commits transactions and how those are replicated to slaves. Come to this talk to learn how to get the benefit of this tuning the right way and to learn some replication internals.
Up to MySQL 5.5, replication was not crash safe: after an unclean shutdown, it would fail with “duplicate key” or “row not found” error, or might generate silent data corruption. It looks like 5.6 is much better, right ? The short answer is maybe: in the simplest case, it is possible to achieve replication crash safety, but it is not the default setting. MySQL 5.7 is not much better, 8.0 has better defaults, but it is still not replication crash-safe by default, and it is still easy to get things wrong.
Crash safety is impacted by replication positioning (File+Position or GTID), type (single-threaded or MTS), MTS settings (Database or Logical Clock, and with or without slave preserve commit order), the sync-ing of relay logs, the presence of binary logs, log-slave-updates and the sync-ing of binary logs. This is very complicated stuff and even the manual is sometimes confused about it.
In this talk, I will explain the impact of the above and help you find the path to crash safety nirvana. I will also give details about replication internals, so you might learn a thing or two.
MySQL Scalability and Reliability for Replicated EnvironmentJean-François Gagné
You have a working application that is using MySQL: great! At the beginning, you are probably using a single database instance, and maybe – but not necessarily – you have replication for backups, but you are not reading from slaves yet. Scalability and reliability were not the main focus in the past, but they are starting to be a concern. Soon, you will have many databases and you will have to deal with replication lag. This talk will present how to tackle the transition.
We mostly cover standard/asynchronous replication, but we will also touch on Galera and Group Replication. We present how to adapt the application to become replication-friendly, which facilitate reading from and failing over to slaves. We also present solutions for managing read views at scale and enabling read-your-own-writes on slaves. We also touch on vertical and horizontal sharding for when deploying bigger servers is not possible anymore.
Are UNIQUE and FOREIGN KEYs still possible at scale, what are the downsides of AUTO_INCREMENTs, how to avoid overloading replication, what are the limits of archiving, … Come to this talk to get answers and to leave with tools for tackling the challenges of the future.
Up to MySQL 5.5, replication was not crash safe: it would fail with “dup.key” or “not found” error (or data corruption). So 5.6 is better, right? Maybe: it is possible, but not the default. MySQL 5.7 is not much better, 8.0 has safer defaults but it is still easy to get things wrong.
Crash safety is impacted by positioning (File+Pos or GTID), type (single/multi-threaded), MTS settings (Db/Logical Clock, and preserve commit order), the sync-ing of relay logs, the presence of binlogs, log-slave-updates and their sync-ing. This is complicated and even the manual is confused about it.
In this talk, I will explain above with details on replication internals, so you might learn a thing or two.
MySQL has multiple timeouts variables to control its operations. This presentation focus on the purpose of each timeout variables and how it can be used.
Yahoo: Experiences with MySQL GTID and Multi Threaded ReplicationYashada Jadhav
Our experiences running GTID and parallel replication in production, advantages, limitations, lessons learnt and tips and tricks we discovered along the way at Oracle Openworld 2015
Replication Troubleshooting in Classic VS GTIDMydbops
This presentation talk will assist you in troubleshooting MySQL replication for the most common issues we might face with a simple comparison of how can we get them solved in the different replication methods (Classic VS GTID).
Up to MySQL 5.5, replication was not crash safe: after a crash, it would fail with "duplicate key" or "row not found" error, or might generate silent data corruption. It looks like 5.6 is much better, right? The short answer is maybe: in the simplest case, it is possible to achieve replication crash safety but it is not the default setting. MySQL 5.7 is not much better, 8.0 has safer defaults but it is still easy to get things wrong.
Crash safety is impacted by replication positioning (File+Pos or GTID), type (single-threaded or MTS), MTS settings (Database or Logical Clock, and with or without slave preserve commit order), the sync-ing of relay logs, the presence of binary logs, log-slave-updates and their sync-ing. This is very complicated stuff and even the manual is confused about it.
In this talk, I will explain the impact of above and help you finding the path to crash safety nirvana. I will also give details about replication internals, so you might learn a thing or two.
MySQL Scalability and Reliability for Replicated EnvironmentJean-François Gagné
You have a working application that is using MySQL: great! At the beginning, you are probably using a single database instance, and maybe – but not necessarily – you have replication for backups, but you are not reading from slaves yet. Scalability and reliability were not the main focus in the past, but they are starting to be a concern. Soon, you will have many databases and you will have to deal with replication lag. This talk will present how to tackle the transition.
We mostly cover standard/asynchronous replication, but we will also touch on Galera and Group Replication. We present how to adapt the application to become replication-friendly, which facilitate reading from and failing over to slaves. We also present solutions for managing read views at scale and enabling read-your-own-writes on slaves. We also touch on vertical and horizontal sharding for when deploying bigger servers is not possible anymore.
Are UNIQUE and FOREIGN KEYs still possible at scale, what are the downsides of AUTO_INCREMENTs, how to avoid overloading replication, what are the limits of archiving, … Come to this talk to get answers and to leave with tools for tackling the challenges of the future.
MariaDB Server on macOS - FOSDEM 2022 MariaDB DevroomValeriy Kravchuk
Current MariaDB Server GA versions are formally not supported (and probably not even regularly built or tested) on macOS 10.x and 11.y. But it's relatively easy to set up the environment and build MariaDB Server from current 10.2 - 10.8 GitHub sources, with few minor issues to resolve in the process, depending on macOS and major server version used.
This talk is a summary of my related experience on 10.13 High Sierra that I had a chance to work on recently, with additional quick review of related fixed and open bugs, as well as some unique features like DTrace support that one may benefit from on macOS. Actually, studying DTrace in context of MariaDB Server troubleshooting and performance tuning was one of the goals why I started to use macOS again.
MySQL replication has evolved a lot in 5.6 ,5.7 and 8.0. This presentation focus on the changes made in parallel replication. It covers MySQL 8.0. It was presented at Mydbops database meetup on 04-08-2016 in Bangalore.
This session is about using GNU debugger (gdb) as a tool to study MySQL internals (namely, InnoDB locks and metadata locks) and as a last resort in cases when server hangs or has to be restarted for other reason. It never hurts to try a trick or two before giving up and restarting.
Sometimes MySQL DBAs have to work with stalled/hanged/unresponsive MySQL instance, where their usual SQL-based tricks do not work any more. Sometimes they can not even connect to check what's going on inside server.
In other cases they know what to do and everything still works, but they have to implement changes to read-only server variables. Server restart is often not an option in production, as it means some downtime and may cause negative performance impact.
In these cases one could do something given read and write access to server memory/internals. Here comes gdb, that, alone with careful reading of the source code helps to often resolve the problems described above. During this session I'll show what can be done with gdb when server already is in
troubles, and how to use gdb to "see" and understand MySQL internals (like InnoDB locks or metadata locks) better.
Almost Perfect Service Discovery and Failover with ProxySQL and OrchestratorJean-François Gagné
Of course there is no such thing as perfect service discovery, and we will see why in the talk. However, the way ProxySQL is deployed in this case minimizes the risk for split-brains, and this is why I qualify it as almost perfect. But let’s step back a little...
MySQL alone is not a high availability solution. To provide resilience to primary failure, other components need to be integrated with MySQL. At MessageBird, these additional components are ProxySQL and Orchestrator. In this talk, we describe how ProxySQL is architectured to provide close to perfect Service Discovery and how this, combined with Orchestrator, allows for automatic failover. The talk presents the details of the integration of MySQL, ProxySQL and Orchestrator in Google Cloud (and it would be easy to re-implement a similar architecture at other cloud vendors or on-premises). We will also cover lessons learned for the 2 years this architecture has been in production. Come to this talk to learn more about MySQL high availability, ProxySQL and Orchestrator.
Have you ever needed to get some additional write throughput from MySQL ? If yes, you probably found that setting sync_binlog to 0 (and trx_commit to 2) gives you an extra performance boost. As all such easy optimisation, it comes at a cost. This talk explains how this tuning works, presents its consequences and makes recommendations to avoid them. This will bring us to the details of how MySQL commits transactions and how those are replicated to slaves. Come to this talk to learn how to get the benefit of this tuning the right way and to learn some replication internals.
Up to MySQL 5.5, replication was not crash safe: after an unclean shutdown, it would fail with “duplicate key” or “row not found” error, or might generate silent data corruption. It looks like 5.6 is much better, right ? The short answer is maybe: in the simplest case, it is possible to achieve replication crash safety, but it is not the default setting. MySQL 5.7 is not much better, 8.0 has better defaults, but it is still not replication crash-safe by default, and it is still easy to get things wrong.
Crash safety is impacted by replication positioning (File+Position or GTID), type (single-threaded or MTS), MTS settings (Database or Logical Clock, and with or without slave preserve commit order), the sync-ing of relay logs, the presence of binary logs, log-slave-updates and the sync-ing of binary logs. This is very complicated stuff and even the manual is sometimes confused about it.
In this talk, I will explain the impact of the above and help you find the path to crash safety nirvana. I will also give details about replication internals, so you might learn a thing or two.
MySQL Scalability and Reliability for Replicated EnvironmentJean-François Gagné
You have a working application that is using MySQL: great! At the beginning, you are probably using a single database instance, and maybe – but not necessarily – you have replication for backups, but you are not reading from slaves yet. Scalability and reliability were not the main focus in the past, but they are starting to be a concern. Soon, you will have many databases and you will have to deal with replication lag. This talk will present how to tackle the transition.
We mostly cover standard/asynchronous replication, but we will also touch on Galera and Group Replication. We present how to adapt the application to become replication-friendly, which facilitate reading from and failing over to slaves. We also present solutions for managing read views at scale and enabling read-your-own-writes on slaves. We also touch on vertical and horizontal sharding for when deploying bigger servers is not possible anymore.
Are UNIQUE and FOREIGN KEYs still possible at scale, what are the downsides of AUTO_INCREMENTs, how to avoid overloading replication, what are the limits of archiving, … Come to this talk to get answers and to leave with tools for tackling the challenges of the future.
Up to MySQL 5.5, replication was not crash safe: it would fail with “dup.key” or “not found” error (or data corruption). So 5.6 is better, right? Maybe: it is possible, but not the default. MySQL 5.7 is not much better, 8.0 has safer defaults but it is still easy to get things wrong.
Crash safety is impacted by positioning (File+Pos or GTID), type (single/multi-threaded), MTS settings (Db/Logical Clock, and preserve commit order), the sync-ing of relay logs, the presence of binlogs, log-slave-updates and their sync-ing. This is complicated and even the manual is confused about it.
In this talk, I will explain above with details on replication internals, so you might learn a thing or two.
MySQL has multiple timeouts variables to control its operations. This presentation focus on the purpose of each timeout variables and how it can be used.
Yahoo: Experiences with MySQL GTID and Multi Threaded ReplicationYashada Jadhav
Our experiences running GTID and parallel replication in production, advantages, limitations, lessons learnt and tips and tricks we discovered along the way at Oracle Openworld 2015
Replication Troubleshooting in Classic VS GTIDMydbops
This presentation talk will assist you in troubleshooting MySQL replication for the most common issues we might face with a simple comparison of how can we get them solved in the different replication methods (Classic VS GTID).
Up to MySQL 5.5, replication was not crash safe: after a crash, it would fail with "duplicate key" or "row not found" error, or might generate silent data corruption. It looks like 5.6 is much better, right? The short answer is maybe: in the simplest case, it is possible to achieve replication crash safety but it is not the default setting. MySQL 5.7 is not much better, 8.0 has safer defaults but it is still easy to get things wrong.
Crash safety is impacted by replication positioning (File+Pos or GTID), type (single-threaded or MTS), MTS settings (Database or Logical Clock, and with or without slave preserve commit order), the sync-ing of relay logs, the presence of binary logs, log-slave-updates and their sync-ing. This is very complicated stuff and even the manual is confused about it.
In this talk, I will explain the impact of above and help you finding the path to crash safety nirvana. I will also give details about replication internals, so you might learn a thing or two.
MySQL Scalability and Reliability for Replicated EnvironmentJean-François Gagné
You have a working application that is using MySQL: great! At the beginning, you are probably using a single database instance, and maybe – but not necessarily – you have replication for backups, but you are not reading from slaves yet. Scalability and reliability were not the main focus in the past, but they are starting to be a concern. Soon, you will have many databases and you will have to deal with replication lag. This talk will present how to tackle the transition.
We mostly cover standard/asynchronous replication, but we will also touch on Galera and Group Replication. We present how to adapt the application to become replication-friendly, which facilitate reading from and failing over to slaves. We also present solutions for managing read views at scale and enabling read-your-own-writes on slaves. We also touch on vertical and horizontal sharding for when deploying bigger servers is not possible anymore.
Are UNIQUE and FOREIGN KEYs still possible at scale, what are the downsides of AUTO_INCREMENTs, how to avoid overloading replication, what are the limits of archiving, … Come to this talk to get answers and to leave with tools for tackling the challenges of the future.
MariaDB Server on macOS - FOSDEM 2022 MariaDB DevroomValeriy Kravchuk
Current MariaDB Server GA versions are formally not supported (and probably not even regularly built or tested) on macOS 10.x and 11.y. But it's relatively easy to set up the environment and build MariaDB Server from current 10.2 - 10.8 GitHub sources, with few minor issues to resolve in the process, depending on macOS and major server version used.
This talk is a summary of my related experience on 10.13 High Sierra that I had a chance to work on recently, with additional quick review of related fixed and open bugs, as well as some unique features like DTrace support that one may benefit from on macOS. Actually, studying DTrace in context of MariaDB Server troubleshooting and performance tuning was one of the goals why I started to use macOS again.
MySQL replication has evolved a lot in 5.6 ,5.7 and 8.0. This presentation focus on the changes made in parallel replication. It covers MySQL 8.0. It was presented at Mydbops database meetup on 04-08-2016 in Bangalore.
This session is about using GNU debugger (gdb) as a tool to study MySQL internals (namely, InnoDB locks and metadata locks) and as a last resort in cases when server hangs or has to be restarted for other reason. It never hurts to try a trick or two before giving up and restarting.
Sometimes MySQL DBAs have to work with stalled/hanged/unresponsive MySQL instance, where their usual SQL-based tricks do not work any more. Sometimes they can not even connect to check what's going on inside server.
In other cases they know what to do and everything still works, but they have to implement changes to read-only server variables. Server restart is often not an option in production, as it means some downtime and may cause negative performance impact.
In these cases one could do something given read and write access to server memory/internals. Here comes gdb, that, alone with careful reading of the source code helps to often resolve the problems described above. During this session I'll show what can be done with gdb when server already is in
troubles, and how to use gdb to "see" and understand MySQL internals (like InnoDB locks or metadata locks) better.
Pseudo GTID and Easy MySQL Replication Topology ManagementShlomi Noach
Use Pseudo GTID to refactor your MySQL replication topology even in face of master/intermediate master craches; use open source orchestrator to automate, visualize and persist your topologies
Managing and Visualizing your Replication Topologies with OrchestratorShlomi Noach
Introducing Orchestrator: a MySQL replication topology management service, that greatly simplifies DBA's tasks and enhances visibility on your topologies.
In this presentation we discuss the New Features of MariaDB 10.4. First we give a short overview of the MariaDB Branches and Forks. Then we talk about the announced IPO. Technically we cover topics like Authentication, Accounts, InnoDB, Optimizer improvements, Application-Time Period Tables the new Backup Stage Galera 4 and other changes...
How to migrate from Oracle Database with easeMariaDB plc
MariaDB introduced Oracle Database compatibility last May with support for Oracle Database data types, sequences, stored procedures (PL/SQL) and more, making it easier than ever to migrate to MariaDB. In this session, MariaDB's Alexander Bienemann and Wagner Bianchi share best practices and lessons learned from our experiences helping customers migrate from Oracle Database. They explain how MariaDB approaches migrations, what’s needed to complete a successful migration and the tools used to determine the level of effort required.
Faster, better, stronger: The new InnoDBMariaDB plc
For MariaDB Enterprise Server 10.5, the default transactional storage engine, InnoDB, has been significantly rewritten to improve the performance of writes and backups. Next, we removed a number of parameters to reduce unnecessary complexity, not only in terms of configuration but of the code itself. And finally, we improved crash recovery thanks to better consistency checks and we reduced memory consumption and file I/O thanks to an all new log record format.
In this session, we’ll walk through all of the improvements to InnoDB, and dive deep into the implementation to explain how these improvements help everything from configuration and performance to reliability and recovery.
Migrations from PLSQL and Transact-SQL - m18Wagner Bianchi
Check out the status of the current features supported by MariaDB Server 10.3 and the best practices brought to the marketplace by MariaDB Practitioners for migrating Oracle and SQL Server to MariaDB Server. Also, some of the many features related to the PL/SQL is exhibited in this presentation. You can watch the presentation's video at https://www.linkedin.com/feed/update/urn:li:activity:6381185127497625601
Troubleshooting MySQL from a MySQL Developer PerspectiveMarcelo Altmann
Working as a MySQL Developer as part of the Bugs committee exposes you to a variety of bugs, such as server crashes, memory leaks, wrong query results, internal thread deadlocks, and others. In this talk, I will cover some of the technics we utilize to troubleshoot MySQL when things are not working as expected.
Some of the topics covered include:
Reproducible test cases
Git Bisect
Stack Traces
GDB
Record and Replay
By the end of this session, attendees will grasp how to tackle analyses of when software is not working as expected.
Webinar slides: Migrating to Galera Cluster for MySQL and MariaDBSeveralnines
Galera Cluster is a mainstream option for high availability MySQL and MariaDB. And though it has established itself as a credible replacement for traditional MySQL master-slave architectures, it is not a drop-in replacement.
While Galera Cluster has some characteristics that make it unsuitable for certain use cases, most applications can still be adapted to run on it.
The benefits are clear: multi-master InnoDB setup with built-in failover and read scalability.
But how do you migrate? Does the schema or application change? What are the limitations? Can a migration be done online, without service interruption? What are the potential risks?
In this webinar, Severalnines Support Engineer Bart Oles walks you through what you need to know in order to migrate from standalone or a master-slave MySQL/MariaDB setup to Galera Cluster.
AGENDA
- Application use cases for Galera
- Schema design
- Events and Triggers
- Query design
- Migrating the schema
- Load balancer and VIP
- Loading initial data into the cluster
- Limitations:
- Cluster technology
- Application vendor support
- Performing Online Migration to Galera
- Operational management checklist
- Belts and suspenders: Plan B
- Demo
SPEAKER
Bartlomiej Oles is a MySQL and Oracle DBA, with over 15 years experience in managing highly available production systems at IBM, Nordea Bank, Acxiom, Lufthansa, and other Fortune 500 companies. In the past five years, his focus has been on building and applying automation tools to manage multi-datacenter database environments.
GTIDs were introduced to solve replication problems and improve database consistency in MySQL database replication.
When, accidentally, transactions occur on a replica, this introduces GTIDs on that replica that don't exist on the master. When, on a master failover, this replica becomes the new master, and the corresponding binlogs of the errant GTIDs are already purged, replication breaks on the replicas of this new master, because those missing GTIDs can't be retrieved from the binlogs of this new master.
This presentation will talk about GTIDs and how to detect errant GTIDs on a replica (before the corresponding binlogs are purged) and how to look at the corresponding transactions in the binlogs. I'll give some examples of transactions that could happen on a replica that didn't originate from a primary node, explain how this is possible and share some tips on how to avoid this.
Basic understanding of MySQL database replication is assumed.
This presentation was at Percona Live 2019 in Austin, Texas.
https://www.percona.com/live/19/sessions/errant-gtids-breaking-replication-how-to-detect-and-avoid-them
MariaDB 10.4 became General Available (GA = ready for production) this summer. So it is time to look at the new Features in MariaDB 10.4. After a short intro about history we look for the reason of broad usage of MariaDB nowadays. Most important improvements where in User Authentication, InnoDB improvements, and Optimizer enhancements. A completely New Feature is Application-Time Period Tables. Backup got a new Locking behaviour so LVM snapshots are possible and officially supported now. And last but not least MariaDB 10.4 comes with Galera 4.
How to Get CNIC Information System with Paksim Ga.pptxdanishmna97
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
MySQL/MariaDB Parallel Replication: inventory, use-case and limitations
1. MySQL/MariaDB Parallel Replication:
inventory, use-cases and limitations
Jean-François Gagné (System Engineer)
jeanfrancois DOT gagne AT booking.com
April 26, 2017 – Percona Live Santa Clara 2017
3. Booking.com
● Based in Amsterdam since 1996
● Online Hotel and Accommodation (Travel) Agent (OTA):
● +1.220.000 properties in 227 countries
● +1.200.000 room nights reserved daily
● +40 languages (website and customer service)
● +13.000 people working in 187 offices worldwide
● Part of the Priceline Group
● And we use MySQL:
● Thousands (1000s) of servers, ~90% replicating
● >150 masters: ~30 >50 slaves & ~10 >100 slaves
3
4. Session Summary
1. Introducing Parallel Replication (// Replication)
2. MySQL 5.6: schema based
MariaDB 10.0: Out-of-Order and In-Order
MariaDB 10.1: MariaDB 10.0 + Optimistic
MySQL 5.7: MySQL 5.6 + Logical Clock
MySQL 8.0 & Group Rpl: MySQL 5.7 + Write Set
3. Benchmark Results from Booking.com
4
5. // Replication
● Relatively new because it is hard
● It is hard because of data consistency
● Running trx in // must give the same result on all slaves (= the master)
● Why is it important ?
● Computers have many Cores, using a single one for writes is a waste
● Some computer resources can give more throughput when used in parallel
(RAID1 has 2 disks we can do 2 Read IOs in parallel)
(SSDs can serve many Read and/or Write IOs in parallel)
5
6. // Replication: MySQL 5.6
● Concept: if transactions are “schema-local”,
two transactions in different schema can be run in parallel on slaves
● Implementation:
● the master tags transactions with their schema in the binary logs
● the SQL thread dispatches work to worker threads according to the schema from the binlog
● Deployment:
● On the master: nothing to do (except having multiple independent schemas)
● On the slave: “SET GLOBAL slave_parallel_workers = N;” (with N > 1)
● MySQL 5.7 has the same feature (default for slave-parallel-type = DATABASE)
● MySQL 8.0 defaults might be different:
● Would need to “SET GLOBAL slave-parallel-type = DATABASE;”
http://mysqlhighavailability.com/mysql-replication-defaults-after-5-7/
6
7. // Replication: MySQL 5.6’
● Implication: transactions on slaves can be committed in a different order
than the order they appear in the binary logs of the master
● On the master, some transactions in schema A and B:
● Order in the binary logs of the master: A1, A2, B1, B2, A3, B3
● On the slave, transactions in different schema are run in parallel:
● “A1, A2, A3” run in parallel with “B1, B2, B3”
● One possible commit order: A1, B1, A2, B2, A3, B3
● Another if B1 is long to execute : A1, A2, A3, B1, B2, B3
● Many other possible orders…
● Out-of-order commit on slave has many impacts…
7
8. // Replication: MySQL 5.6’’
● Impacts on the binary log content on slaves:
● 2 slaves can have different binlogs (also different from the master binlogs)
● Impacts on “SHOW SLAVE STATUS”:
● All transactions before the reported SQL thread file and position are committed
● This “all committed before” position is called a checkpoint
● Some trx might be committed after the checkpoint and some might still be running gaps
● Impacts on replication crash recovery (because gaps)
● Impacts on GTIDs: temporary holes in @@global.gtid_executed (because gaps)
● More impacts: skipping transactions, backups, heartbeat, …
8
9. // Replication: MySQL 5.6’’’
● Removing gaps in transaction execution:
● “STOP SLAVE; START SLAVE UNTIL SQL_AFTER_MTS_GAPS;”
● MySQL is not parallel replication crash safe without GTIDs (this is a bug):
● http://jfg-mysql.blogspot.com/2016/01/replication-crash-safety-with-mts.html
● For skipping transactions (with sql_slave_skip_counter): first remove gaps
● For backups: make sure your tool is parallel replication aware
● Worker states stored in mysql.slave_worker_info:
● https://dev.mysql.com/worklog/task/?id=5599 (not an easy read)
● Tuning parameters:
● slave-pending-jobs-size-max: RAM for unprocessed events (default 16M)
● slave_checkpoint_group: see next slide (default 512)
● slave_checkpoint_period: see next slide (default 300 ms)
10. // Replication: MySQL 5.6’’’ ’
● MTS checkpoint:
● After making sure gaps are filled, checkpointing advances the position of “SHOW SLAVE STATUS”
● Checkpointing is tried every slave_checkpoint_period (300 ms by default)
● A checkpoint attempt might fail if a worker is still working on the next needed
transaction long transaction might block checkpointing:
● Binlog content: A1,A2,B1,B2,B3,B4,B5…B500,B501,…B600
● If A2 is very long (ALTER TABLE), it will block checkpointing
● This will block the slave execution at ~B511
● If this happens, workers will not be able to go beyond the group size
● Solution: increase slave_checkpoint_group (512 by default)
● Similar problems happen if transactions are big (in the binlogs)
● Solution: increase slave-pending-jobs-size-max (16M by default)
● But try keeping your trx small (avoid LOAD DATA INFILE and others…)
11. // Replication: MariaDB 10.0 (out-of-order)
● Concept: manually tags independent transactions in “write domains”
● Implementation:
● MariaDB GTIDs: <domain ID>-<server ID>-<Sequence Number> (0-1-10)
● the SQL thread becomes a coordinator that dispatches work
● Deployment:
● On the master and for each trx: “SET SESSION gtid_domain_id = D;”
● On the slave: “SET GLOBAL slave_parallel_threads = N;” (with N > 1)
● But advertise the Write Domain right !
● MySQL protects you from multi-schema trx, MariaDB cannot do the same for write domains
● Also out-of-order commits of transactions on slaves:
● There will be gaps, those gaps are managed by MariaDB GTIDs,
● Impact on binary logs, SHOW SLAVE STATUS, skipping transactions, backups, heartbeat, …
11
12. // Replication: MariaDB 10.0 (out-of-order)’
● Difference with MySQL 5.6:
● “SHOW SLAVE STATUS”: position of the latest committed trx (there might be gaps before…)
● If the SQL thread stops (or is stopped), its position will “rewind” to a “safe” position
https://jira.mariadb.org/browse/MDEV-6589 & MDEV-9138 & MDEV-10863 (all fixed bugs)
● Removing gaps: STOP SLAVE; SET GLOBAL slave_parallel_threads = 0; START SLAVE;
● To avoid re-downloading relay logs, use below but see two MDEVs above:
STOP SLAVE SQL_THREAD; SET GLOBAL slave_parallel_threads = 0; START SLAVE;
http://jfg-mysql.blogspot.com/2015/10/bad-commands-with-mariadb-gtids-2.html
● Skipping transactions:
● Go back to single threaded replication, START SLAVE break again, then skip
● Like above, restart the IO thread if you want to avoid problems
12
13. // Replication: MariaDB 10.0 (in-order)
● Concept:
trx committing together on the master can be executed in parallel on slaves
● Implementation:
● Build on top of the binary log Group Commit optimization:
the master tags transactions in the binary logs with their Commit ID (cid)
● As the name implies, trx are committed in the same order as in the binlogs of the master
● Deployment:
● Needs a MariaDB 10.0 master
● On slaves: “SET GLOBAL slave_parallel_threads = N;” (with N > 0)
13
15. // Replication: MariaDB 10.0 (in-order)’’
● Good (large groups) or bad (small groups) parallelism from the master:
● When sync_binlog = 1, instead of syncing the binlog after each transaction,
MariaDB buffers trx during previous sync before writing all of them as a group and then syncing
● Setting sync_binlog = 0 or > 1 might lead to smaller groups (bad for parallel replication)
● When there is not enough parallelism, or if sync are very fast, grouping might also be suboptimal
● Global Statuses can be used to monitor grouping on the master:
● BINLOG_COMMITS: number of commits in the binary logs
● BINLOG_GROUP_COMMITS: number of group commits in the binary logs (lower is better)
● The first divided by the second gives the group size (larger is better)
● Grouping optimization (slowing down the master to speedup slaves):
● BINLOG_COMMIT_WAIT_USEC (BCWU): timeout for waiting more transactions joining the group
● BINLOG_COMMIT_WAIT_COUNT (BCWC): number of transactions that short-circuit waiting
15
17. // Replication: MariaDB 10.0 (in-order)’’’ ’
● Long transactions can block the parallel execution pipeline
● On the master: ---------------- Time --------------->
T1: B-------------------------C
T2: B--C
T3: B--C
● On the slaves: T1: B-------------------------C
T2: B-- . . . . . . . . . . . C
T3: B-- . . . . . . . . . . . C
Try reducing as much as possible the number of big transactions:
• Easier said than done: 10 ms is big compared to 1 ms
Avoid monster transactions (LOAD DATA, unbounded UPDATE or DELETE, …)
17
18. // Replication: MariaDB 10.0 (in-order)’’’ ’’
● Replicating through intermediate masters (IM) loses grouping
● Four transactions on X, Y and Z:
+---+
| X |
+---+
|
V
+---+
| Y |
+---+
|
V
+---+
| Z |
+---+
● To get maximum replication speed, replace intermediate master by Binlog Servers
● More details at http://blog.booking.com/better_parallel_replication_for_mysql.html
18
On Y:
----Time---->
B---C
B---C
B-------C
B-------C
On Z:
----Time--------->
B---C
B---C
B-------C
B-------C
On X:
----Time---->
T1 B---C
T2 B---C
T3 B-------C
T4 B-------C
19. // Replication: Slave Group Commit
● On a single-threaded slave, transactions are run sequentially:
-------- Time ------->
T1: B----C
T2: B----C
● If T1 and T2 are in different cid, they cannot be run in parallel
● But if they do not conflict, delaying committing of T1 might allow to completely
run T2 in another thread, achieving group commit:
T1: B---- . . C (in thread #1)
T2: B----C (in thread #2)
● Above has identified that T1 and T2 can be run in parallel (and saved a fsync)
19
20. // Replication: Slave Group Commit’
● MariaDB 10.0 implements Slave Group Commit when
1. the master is running MariaDB 10.0,
2. slave_parallel_threads (SPT) > 1 & BCWC > 1 & BCWU > 0
● Waiting is short-circuited when a transaction Tn blocks on Tn-i
so this should not happen:
T1: B---- . . . . C
T2: B--- . . . --C
No penalty for using big value of BCWU on slaves
● This mitigates the problem with intermediate masters
● Except for DDL where short-circuit is not implemented
20
21. // Replication: MariaDB 10.1 (in-order)
● MariaDB 10.1 has five different slave parallel modes:
1. none: classic single-threaded slave (same as slave_parallel_threads = 0)
2. minimal: in different threads, serialized execution of transaction
(this is for slave group commit: needs BCWC > 1 and BCWU > 0)
(and out-of-order parallel replication disabled in this mode)
3. conservative: parallel execution based on group commit (= MariaD 10.0)
4. optimistic: a new type of parallel execution
5. aggressive: a more aggressive optimistic mode
21
22. // Replication: Back to MariaDB 10.0
● With MariaDB 10.0, naïve parallel replication could deadlock
● On the master, T1 and T2 commit together:
T1: B-------C
T2: B--C
● On the slaves, T2 (ready to commit) blocks T1 (because index update, …),
but T1 must commit before T2 deadlock
T1: B---- . . . . . . . . . . . . .
T2: B-- . . . . . . . . . . . . . .
● To solve this deadlock, MariaDB kills T2, which unblocks T1
● Corresponding global status: slave_retried_transactions
22
23. // Replication: Back to MariaDB 10.0’
● Number of retried transactions catching up many hours of replication
delay (~2.5K transactions per second):
Retry happen 3 times for 600,000 transactions not often at all
23
24. // Replication: MariaDB 10.1 (optimistic)
● Concept: run all transactions in parallel, if they conflict (replication blocked
because in-order commit), deadlock detection unblocks the slave
● Deployment:
● Needs a MariaDB 10.1 master
● Assume same table transactional property on master and slave
(could produce corrupted results if the master is InnoDB and slave MyISAM)
● SET GLOBAL slave_parallel_thread = N; (with N > 1)
● SET GLOBAL slave_parallel_mode = {optimistic | aggressive};
Optimistic will try to reduce the number of deadlocks (and rollbacks) using information put in the
binary logs from the master, aggressive will run as many transactions in parallel as possible
(bounded by the number of threads)
● DDLs cannot be rollbacks they cannot be replicated optimistically:
DDL blocks the parallel replication pipeline (and same for other non-transactional operations)
24
25. // Replication: MySQL 5.7
● MySQL 5.7 has two slave_parallel_type:
● both need “SET GLOBAL slave_parallel_workers = N;” (with N > 1)
● DATABASE: the schema based parallel replication from MySQL 5.6
● LOGICAL_CLOCK: “Transactions that are part of the same binary log group commit on a
master are applied in parallel on a slave.” (from the doc. but not exact: Bug#85977)
● the LOGICAL_CLOCK type is implemented by putting interval information in the binary logs
● Slowing down the master to speedup the slave:
● binlog_group_commit_sync_delay
● binlog_group_commit_sync_no_delay_count
● We can expect the same problems as with MariaDB 10.0:
● Problems with long/big transactions
● Problems with intermediate masters
25
26. MySQL 5.7: LOGICAL CLOCK
● By default, MySQL 5.7 in logical clock does out-of-order commit:
There will be gaps (“START SLAVE UNTIL SQL_AFTER_MTS_GAPS;”)
● Not replication crash safe without GTIDs
http://jfg-mysql.blogspot.com/2016/01/replication-crash-safety-with-mts.html
● And also everything else:
binary logs content, SHOW SLAVE STATUS, skipping transactions, backups, …
● Using slave_preserve_commit_order = 1 does what you expect:
● This configuration does not generate gap
● But it needs log-slave-updates, there is a feature request to remove this limitation: Bug#75396
● And it is still not replication crash safe (surprising because no gap): Bug#80103 & Bug#81840
26
27. // Replication: MySQL 8.0.1 and GR
● Group Replication (GR) and MySQL 8.0.1 adds Write Set
● Write Set are a new way to identify parallelism on masters or even on slaves
● allows a node in Group Replication to replicate faster than a “standard” slave
● not limited by “Group Commit Size” (no need for transaction to “commit together”)
● works on a single-threaded workload
● Write Set can be used to identify parallelism on a single-threaded intermediate
master, which allows to do test without upgrading the master
I cannot explain how this works in this talk (not enough time)
come to my talk tomorrow for all the details
MySQL Parallel Replication: all the 5.7 (and some of the 8.0) details
27
28. // Replication: results from B.com
● MariaDB 10.0 tests:
● On 4 environments, from a MySQL 5.6 masters, thanks to Slave Group Commit
● MariaDB 10.1 tests: conservative vs optimistic (aggressive)
● Same environments and trx: how much better (or worse) will aggressive be ?
● MySQL 5.6 real production deployment (schema based)
● MariaDB 10.0 and 10.1 real production deployment
● No results from MySQL 5.7 in this talk (more in another talk)
28
29. // Replication: MariaDB 10.0
29
● Four environments (E1, E2, E3 and E4):
● A is a MySQL 5.6 master
● B is a MariaDB 10.0 intermediate master
● C is a MariaDB 10.0 intermediate master doing slave group commit
● D is using the group commit information from C to run transaction in parallel
+---+ +---+ +---+ +---+
| A | --> | B | --> | C | --> | D |
+---+ +---+ +---+ +---+
● Note that slave group commit generates smaller group
than a group committing master, more information in:
http://blog.booking.com/evaluating_mysql_parallel_replication_3-under_the_hood.html#group_commit_slave_vs_master
30. // Replication: MariaDB 10.0 tests
30
● Parallel replication with MariaDB 10.0 (or with 10.1 conservative):
● Catching up 24 hours of replication delay with 0, 5, 10, 20 and 40 threads
32. // Replication: MariaDB 10.0 tests’’
32
HD Single-Threaded: 3h09.34
ND Single-Threaded: 1h24.09
Slave with binlogs (SB) but without log-slave-updates
High Durability (HD): “sync_binlog = 1” + “trx_commit = 1”
No Durability (ND): “sync_binlog = 0” + “trx_commit = 2”
1
1,2
1,4
1,6
1,8
2
1 2 3 4
E1 SB-HD&ND
HD ND
33. // Replication: MariaDB 10.0 tests’’’
33
1
1,5
2
2,5
1 2 3 4
E1 SB-HD&ND
HD ND ND vs HD Single-Threaded ND vs HD
1
1,5
2
2,5
3
1 2 3 4
E2 SB-HD&ND
HD ND ND vs HD Single-Threaded ND vs HD
34. // Replication: MariaDB 10.0 tests’’’ ’
34
1
1,1
1,2
1,3
1,4
1 2 3 4
E3 SB-HD&ND
HD ND ND vs HD Single-Threaded ND vs HD
1
1,2
1,4
1,6
1 2 3 4
E4 SB-HD&ND
HD ND ND vs HD Single-Threaded ND vs HD
35. // Replication: MariaDB 10.1 tests
35
● Four same environments, D now runs MariaDB 10.1, and to take advantage
of optimistic parallel replication, we need a 10.1 master add C2
+---+ +---+ +---+ +---+
| A | --> | B | --> | C | --> | D |
+---+ +---+ +---+ +---+
|
| +---+ +---+
+-----> | C2| --> | D2|
+---+ +---+
● D and D2 are the same hardware
● D runs with SPT = conservative (using the “slave Group Commit” binary logs)
● D2 runs with SPT = aggressive (needs a 10.1 master to work)
40. // Replication: MySQL 5.6 real
● Booking.com session store is sharded with many schema per database:
● Running MySQL 5.6, >1 TB per node, 20 schema per node, magnetic disks
● PLAMS 2015: Combining Redis and MySQL to store HTTP cookie data
https://www.percona.com/live/europe-amsterdam-2015/sessions/combining-redis-and-mysql-store-
http-cookie-data
40
1
2
3
4
1 2 3 4 5 6
Sess SB-HD&ND
HD ND ND vs HD Single-Threaded ND vs HD
41. // Replication: MariaDB 10.0 real
● Booking.com is also running MariaDB 10.0:
● Test is run on a multi-tenths of TB database
● Hosted on a disk storage array: low write latency (cache) and many reads in parallel
● We optimized group commit and enabled parallel replication:
set global BINLOG_COMMIT_WAIT_COUNT = 75;
set global BINLOG_COMMIT_WAIT_USEC = 300000; (300 milliseconds)
set global SLAVE_PARALLEL_THREADS = 80;
● And we tuned the application:
● Break big transaction in many small transactions
● Increase the number of concurrent sessions to the database
41
42. // Replication: MariaDB 10.0 real’
42
● Commit rate when disabling/enabling parallel replication:
● Corresponding group commit size:
(Time is not the same on X-Axis as the slave was delayed on purpose)
43. // Replication: MariaDB 10.1 real
43
● Single-threaded, conservative and aggressive commit rate:
44. Kudos to MariaDB Engineers
● Parallel Replication allows to identify old bugs:
● Queries at the same time on the same table is unlikely IRL but common with // replication
● An InnoDB race condition caused a query by Primary Key to do a full table scan:
● Hard to notice for a “1-in-a-million” SELECT (or UPDATE/DELETE on master)
● But in replication, this is obvious: one of many slaves blocked for minutes on an UPDATE by PK
● Kudos to MariaDB Engineers for finding and fixing this very hard bug:
● Valerii Kravchuk spotted a strange behavior in SHOW ENGINE INNODB STATUS output
and narrowed it down by rightfully asking for a SHOW EXPLAIN FOR on the “slow” UPDATE
● Michael Widenius (Monty) provided me with many patches to identify the problem (debug in prod.)
● Sergey Petrunya fixed the bug in InnoDB and reported it upstream
(MDEV-10649 fixed in MariaDB 10.0.28 and 10.1.18)
(Bug#82968 fixed in 5.7.18 and Bug#82969 still open)
(more context in https://www.facebook.com/valerii.kravchuk/posts/1073608056064467)
44
45. // Replication: Summary
● Parallel replication is not simple
● MariaDB 10.0 in-order (and probably MySQL 5.7 logical clock) has limitations:
● Long transactions block the parallel replication pipeline
● Intermediate master loses parallelism and reduce replication speed on slaves
● MySQL 5.6 and 5.7 are not fully MTS crash-safe (without GTIDs)
● MariaDB out-of-order needs careful and precise developer involvement
● MySQL schema-based solution looks safer and simpler to use
than MariaDB out-of-order which is more flexible but more complex
● MariaDB 10.1 aggressive mode much better than conservative
● Try very high number of threads
● In all cases, avoid big transactions in the binary logs
47. // Replication: Links
● Evaluating MySQL Parallel Replication Part 4: More Benchmarks in Production:
http://blog.booking.com/evaluating_mysql_parallel_replication_4-more_benchmarks_in_production.html
(see also Part 3, 2 and 1 that are linked in the post)
● Replication crash safety with MTS in MySQL 5.6 and 5.7: reality or illusion?
https://jfg-mysql.blogspot.com/2016/01/replication-crash-safety-with-mts.html
● Binlog Servers:
● http://blog.booking.com/mysql_slave_scaling_and_more.html
● Better Parallel Replication for MySQL: http://blog.booking.com/better_parallel_replication_for_mysql.html
● http://blog.booking.com/abstracting_binlog_servers_and_mysql_master_promotion_wo_reconfiguring_slaves.html
● Others:
● https://mariadb.com/blog/how-get-mysql-56-parallel-replication-and-percona-xtrabackup-play-nice-together
● https://www.percona.com/blog/2015/01/29/multi-threaded-replication-with-mysql-5-6-use-gtids/
47
48. // Replication: Links’
● Bugs/feature requests:
● The doc. of slave-parallel-type=LOGICAL_CLOCK wrongly reference Group Commit: Bug#85977
● Allow slave_preserve_commit_order without log-slave-updates: Bug#75396
● Automatic Replication Recovery Does Not Handle Lost Relay Log Events: Bug#81840
● FTWRL deadlock: MDEV-10644 (workaround: kill the FTWRL session, maybe needing gdb magic)
● (More MySQL 5.7 and 8.0 bugs in the other talk)
● Fixed bugs
(make sure to use MariaDB 10.0.29+ or 10.1.20+ to avoid fixed bugs):
● Replication position lost after crash on MTS configured slave: Bug#77496
● MariaDB deadlocks and replication breakages: MDEV-7326, MDEV-7458 and MDEV-10863
● Full table scan bug in InnoDB: MDEV-10649, Bug#82968 and Bug#82969
48