The document discusses the write path for transactions in InnoDB from the client connection to physical storage. It compares InnoDB's transaction and storage layers to the OSI model. Key aspects covered include how SQL statements are executed, how rows are locked, written to indexes and undo logs, and how transactions are committed or rolled back. Mini-transactions provide atomic durable changes to multiple pages using write-ahead logging to the redo log.
This tutorial covers all parallel replication implementation in MariaDB 10.0 and 10.1 and MySQL 5.6, 5.7 and 8.0 (including how it works in Group Replication).
MySQL and MariaDB have different types of parallel replication. In this tutorial, we present the different implementations that allow us to understand their limitations and tuning parameters. We cover how to make parallel replication faster and what to avoid for maximizing its benefits. We also present tests from Booking.com workloads.
Some of the subjects that are covered are group commit and optimistic parallel replication in MariaDB, the parallelism interval of MySQL and its Write Set optimization, and the ?slowing down the master to speed up the slave? optimization.
After this tutorial, you will know everything you need to implement and tune parallel replication in your environment. But more importantly, we will show how you can test parallel replication benefit in a non-disruptive way before deployment.
The presentation covers improvements made to the redo logs in MySQL 8.0 and their impact on the MySQL performance and Operations. This covers the MySQL version still MySQL 8.0.30.
Abstract: A histogram represents the frequency of data distribution. This histogram can help in predicting a better execution plan. MariaDB and MySQL support HIstogram. Though they serve a common purpose they are implemented differently in MariaDB and MySQL, they have their control knobs. Our Consultants ( Monu Mahto and Madhavan ) articulate how histograms can be used in your production MySQL and MariaDB databases and how it helps in bringing down the execution time.
MySQL Database Monitoring: Must, Good and Nice to HaveSveta Smirnova
It is very easy to find if a database installation is having issues. You only need to enable Operating System monitoring. A disk, memory, or CPU usage change will alert you about the problems. But they would not show *why* the trouble happens. You need the help of database-specific monitoring tools.
As a Support Engineer, I am always very upset when handling complaints about the database behavior lacking specific database monitoring data because I cannot help!
There are two reasons database and system administrators do not enable necessary instrumentation. The first is a natural or expected performance impact. Second is the lack of knowledge on what needs to be on to resolve a particular issue.
In this talk, I will cover both concerns.
I will show which monitoring instruments will give information on what causes disk, memory, or CPU problems.
I will teach you how to use them.
I will uncover which performance impact these instruments have.
I will use both MySQL command-line client and open-source graphical instrument Percona Monitoring and Management (PMM) for the examples.
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookThe Hive
This presentation describes the reasons why Facebook decided to build yet another key-value store, the vision and architecture of RocksDB and how it differs from other open source key-value stores. Dhruba describes some of the salient features in RocksDB that are needed for supporting embedded-storage deployments. He explains typical workloads that could be the primary use-cases for RocksDB. He also lays out the roadmap to make RocksDB the key-value store of choice for highly-multi-core processors and RAM-speed storage devices.
This tutorial covers all parallel replication implementation in MariaDB 10.0 and 10.1 and MySQL 5.6, 5.7 and 8.0 (including how it works in Group Replication).
MySQL and MariaDB have different types of parallel replication. In this tutorial, we present the different implementations that allow us to understand their limitations and tuning parameters. We cover how to make parallel replication faster and what to avoid for maximizing its benefits. We also present tests from Booking.com workloads.
Some of the subjects that are covered are group commit and optimistic parallel replication in MariaDB, the parallelism interval of MySQL and its Write Set optimization, and the ?slowing down the master to speed up the slave? optimization.
After this tutorial, you will know everything you need to implement and tune parallel replication in your environment. But more importantly, we will show how you can test parallel replication benefit in a non-disruptive way before deployment.
The presentation covers improvements made to the redo logs in MySQL 8.0 and their impact on the MySQL performance and Operations. This covers the MySQL version still MySQL 8.0.30.
Abstract: A histogram represents the frequency of data distribution. This histogram can help in predicting a better execution plan. MariaDB and MySQL support HIstogram. Though they serve a common purpose they are implemented differently in MariaDB and MySQL, they have their control knobs. Our Consultants ( Monu Mahto and Madhavan ) articulate how histograms can be used in your production MySQL and MariaDB databases and how it helps in bringing down the execution time.
MySQL Database Monitoring: Must, Good and Nice to HaveSveta Smirnova
It is very easy to find if a database installation is having issues. You only need to enable Operating System monitoring. A disk, memory, or CPU usage change will alert you about the problems. But they would not show *why* the trouble happens. You need the help of database-specific monitoring tools.
As a Support Engineer, I am always very upset when handling complaints about the database behavior lacking specific database monitoring data because I cannot help!
There are two reasons database and system administrators do not enable necessary instrumentation. The first is a natural or expected performance impact. Second is the lack of knowledge on what needs to be on to resolve a particular issue.
In this talk, I will cover both concerns.
I will show which monitoring instruments will give information on what causes disk, memory, or CPU problems.
I will teach you how to use them.
I will uncover which performance impact these instruments have.
I will use both MySQL command-line client and open-source graphical instrument Percona Monitoring and Management (PMM) for the examples.
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookThe Hive
This presentation describes the reasons why Facebook decided to build yet another key-value store, the vision and architecture of RocksDB and how it differs from other open source key-value stores. Dhruba describes some of the salient features in RocksDB that are needed for supporting embedded-storage deployments. He explains typical workloads that could be the primary use-cases for RocksDB. He also lays out the roadmap to make RocksDB the key-value store of choice for highly-multi-core processors and RAM-speed storage devices.
MySQL replication has evolved a lot in 5.6 ,5.7 and 8.0. This presentation focus on the changes made in parallel replication. It covers MySQL 8.0. It was presented at Mydbops database meetup on 04-08-2016 in Bangalore.
When does InnoDB lock a row? Multiple rows? Why would it lock a gap? How do transactions affect these scenarios? Locking is one of the more opaque features of MySQL, but it’s very important for both developers and DBA’s to understand if they want their applications to work with high performance and concurrency. This is a creative presentation to illustrate the scenarios for locking in InnoDB and make these scenarios easier to visualize. I'll cover: key locks, table locks, gap locks, shared locks, exclusive locks, intention locks, insert locks, auto-inc locks, and also conditions for deadlocks.
MySQL Parallel Replication: inventory, use-case and limitationsJean-François Gagné
In the last 24 months, MySQL/MariaDB replication speed has improved a lot thanks to parallel replication. MySQL and MariaDB have different types of parallel replication; in this talk, I present the different implementations, with their limitations and the corresponding tuning parameters. I cover what to do to make parallel replication faster and what to avoid for maximizing parallel replication benefits. I also present benchmark results from real Booking.com workloads. Finally, I discuss some deployments at Booking.com that take advantage of parallel replication speed improvements.
Up to MySQL 5.5, replication was not crash safe: after an unclean shutdown, it would fail with “duplicate key” or “row not found” error, or might generate silent data corruption. It looks like 5.6 is much better, right ? The short answer is maybe: in the simplest case, it is possible to achieve replication crash safety, but it is not the default setting. MySQL 5.7 is not much better, 8.0 has better defaults, but it is still not replication crash-safe by default, and it is still easy to get things wrong.
Crash safety is impacted by replication positioning (File+Position or GTID), type (single-threaded or MTS), MTS settings (Database or Logical Clock, and with or without slave preserve commit order), the sync-ing of relay logs, the presence of binary logs, log-slave-updates and the sync-ing of binary logs. This is very complicated stuff and even the manual is sometimes confused about it.
In this talk, I will explain the impact of the above and help you find the path to crash safety nirvana. I will also give details about replication internals, so you might learn a thing or two.
How to Analyze and Tune MySQL Queries for Better Performanceoysteing
Tutorial at Oracle Open World 2015:
Performance of SQL queries plays a big role in application performance. If some queries execute slowly, these queries or the database schema may need tuning. This tutorial covers query processing, optimization methods, and how the MySQL optimizer chooses a specific plan to execute SQL. See demonstrations on how to use tools such as EXPLAIN (including the JSON-based variant), optimizer trace, and performance schema to analyze query plans. See how the Visual Explain functionality in MySQL Workbench helps you to visualize these plans. Based on the analysis, the tutorial covers how to take the next steps for performance tuning. It might mean forcing a particular index, changing the schema, or modifying configuration parameters.
Transparent sharding with Spider: what's new and getting startedMariaDB plc
OpenWorks 2019 Session
MariaDB Server 10.3 introduced transparent, built-in sharding with the Spider storage engine to scale out reads, writes and storage. MariaDB Server 10.4 will include a number of improvements, including DDL pushdown. In this session, Ralf Gebhardt and Kentoku Shiba of MariaDB show how to set up a sharded MariaDB cluster and scale out on demand, as well explore as best practices for high availability and consistency in a sharded deployment.
This presentation covers MySQL data encryption at disk. How to encrypt all tablespaces and MySQL related files for the compliances ? The new releases in MySQL 8.0 take care of the encryption of the system tablespace and supporting tables unlike MySQL 5.7.
MySQL Performance Schema in Action: the Complete TutorialSveta Smirnova
Performance Schema is powerful diagnostic instrument for:
- Query performance
- Complicated locking issues
- Memory leaks
- Resource usage
- Problematic behavior, caused by inappropriate settings
- More
It comes with hundreds of options which allow precisely tune what to instrument. More than 100 consumers store collected data.
In this tutorial we will try all important instruments out. We will provide test environment and few typical problems which could be hardly solved without Performance Schema. You will not only learn how to collect and use this information, but have experience with it.
Made it on PerconaLive Frankfurt, 2018: https://www.percona.com/live/e18/sessions/mysql-performance-schema-in-action-the-complete-tutorial
MySQL has multiple timeouts variables to control its operations. This presentation focus on the purpose of each timeout variables and how it can be used.
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)Jean-François Gagné
To get better replication speed and less lag, MySQL implements parallel replication in the same schema, also known as LOGICAL_CLOCK. But fully benefiting from this feature is not as simple as just enabling it.
In this talk, I explain in detail how this feature works. I also cover how to optimize parallel replication and the improvements made in MySQL 8.0 and back-ported in 5.7 (Write Sets), greatly improving the potential for parallel execution on replicas (but needing RBR).
Come to this talk to get all the details about MySQL 5.7 and 8.0 Parallel Replication.
MySQL Administrator
Basic course
- MySQL 개요
- MySQL 설치 / 설정
- MySQL 아키텍처 - MySQL 스토리지 엔진
- MySQL 관리
- MySQL 백업 / 복구
- MySQL 모니터링
Advanced course
- MySQL Optimization
- MariaDB / Percona
- MySQL HA (High Availability)
- MySQL troubleshooting
네오클로바
http://neoclova.co.kr/
Performance Schema is a powerful diagnostic instrument for:
- Query performance
- Complicated locking issues
- Memory leaks
- Resource usage
- Problematic behavior, caused by inappropriate settings
- More
It comes with hundreds of options which allow precisely tuning what to instrument. More than 100 consumers store collected data.
In this tutorial, we will try all the important instruments out. We will provide a test environment and a few typical problems which could be hardly solved without Performance Schema. You will not only learn how to collect and use this information but have experience with it.
Tutorial at Percona Live Austin 2019
This ppt was used by Devrim at pgDay Asia 2017. He talked about some important facts about WAL - Transaction Logs or xlogs in PostgreSQL. Some of these can really come handy on a bad day
How to Take Advantage of Optimizer Improvements in MySQL 8.0Norvald Ryeng
MySQL 8.0 introduces several improvements to the query optimizer that may give improved performance for your queries. This presentation looks at what kind of queries the different improvements apply to, and the focus is on what you can do to get the most out of the optimizer improvements. The main topics are changes to the optimizer cost model, histograms, and new optimizer hints, but other improvements to how MySQL executes queries are also covered. The presentation includes many practical examples of how you can get a significant speedup for your MySQL queries.
Faster, better, stronger: The new InnoDBMariaDB plc
For MariaDB Enterprise Server 10.5, the default transactional storage engine, InnoDB, has been significantly rewritten to improve the performance of writes and backups. Next, we removed a number of parameters to reduce unnecessary complexity, not only in terms of configuration but of the code itself. And finally, we improved crash recovery thanks to better consistency checks and we reduced memory consumption and file I/O thanks to an all new log record format.
In this session, we’ll walk through all of the improvements to InnoDB, and dive deep into the implementation to explain how these improvements help everything from configuration and performance to reliability and recovery.
MySQL replication has evolved a lot in 5.6 ,5.7 and 8.0. This presentation focus on the changes made in parallel replication. It covers MySQL 8.0. It was presented at Mydbops database meetup on 04-08-2016 in Bangalore.
When does InnoDB lock a row? Multiple rows? Why would it lock a gap? How do transactions affect these scenarios? Locking is one of the more opaque features of MySQL, but it’s very important for both developers and DBA’s to understand if they want their applications to work with high performance and concurrency. This is a creative presentation to illustrate the scenarios for locking in InnoDB and make these scenarios easier to visualize. I'll cover: key locks, table locks, gap locks, shared locks, exclusive locks, intention locks, insert locks, auto-inc locks, and also conditions for deadlocks.
MySQL Parallel Replication: inventory, use-case and limitationsJean-François Gagné
In the last 24 months, MySQL/MariaDB replication speed has improved a lot thanks to parallel replication. MySQL and MariaDB have different types of parallel replication; in this talk, I present the different implementations, with their limitations and the corresponding tuning parameters. I cover what to do to make parallel replication faster and what to avoid for maximizing parallel replication benefits. I also present benchmark results from real Booking.com workloads. Finally, I discuss some deployments at Booking.com that take advantage of parallel replication speed improvements.
Up to MySQL 5.5, replication was not crash safe: after an unclean shutdown, it would fail with “duplicate key” or “row not found” error, or might generate silent data corruption. It looks like 5.6 is much better, right ? The short answer is maybe: in the simplest case, it is possible to achieve replication crash safety, but it is not the default setting. MySQL 5.7 is not much better, 8.0 has better defaults, but it is still not replication crash-safe by default, and it is still easy to get things wrong.
Crash safety is impacted by replication positioning (File+Position or GTID), type (single-threaded or MTS), MTS settings (Database or Logical Clock, and with or without slave preserve commit order), the sync-ing of relay logs, the presence of binary logs, log-slave-updates and the sync-ing of binary logs. This is very complicated stuff and even the manual is sometimes confused about it.
In this talk, I will explain the impact of the above and help you find the path to crash safety nirvana. I will also give details about replication internals, so you might learn a thing or two.
How to Analyze and Tune MySQL Queries for Better Performanceoysteing
Tutorial at Oracle Open World 2015:
Performance of SQL queries plays a big role in application performance. If some queries execute slowly, these queries or the database schema may need tuning. This tutorial covers query processing, optimization methods, and how the MySQL optimizer chooses a specific plan to execute SQL. See demonstrations on how to use tools such as EXPLAIN (including the JSON-based variant), optimizer trace, and performance schema to analyze query plans. See how the Visual Explain functionality in MySQL Workbench helps you to visualize these plans. Based on the analysis, the tutorial covers how to take the next steps for performance tuning. It might mean forcing a particular index, changing the schema, or modifying configuration parameters.
Transparent sharding with Spider: what's new and getting startedMariaDB plc
OpenWorks 2019 Session
MariaDB Server 10.3 introduced transparent, built-in sharding with the Spider storage engine to scale out reads, writes and storage. MariaDB Server 10.4 will include a number of improvements, including DDL pushdown. In this session, Ralf Gebhardt and Kentoku Shiba of MariaDB show how to set up a sharded MariaDB cluster and scale out on demand, as well explore as best practices for high availability and consistency in a sharded deployment.
This presentation covers MySQL data encryption at disk. How to encrypt all tablespaces and MySQL related files for the compliances ? The new releases in MySQL 8.0 take care of the encryption of the system tablespace and supporting tables unlike MySQL 5.7.
MySQL Performance Schema in Action: the Complete TutorialSveta Smirnova
Performance Schema is powerful diagnostic instrument for:
- Query performance
- Complicated locking issues
- Memory leaks
- Resource usage
- Problematic behavior, caused by inappropriate settings
- More
It comes with hundreds of options which allow precisely tune what to instrument. More than 100 consumers store collected data.
In this tutorial we will try all important instruments out. We will provide test environment and few typical problems which could be hardly solved without Performance Schema. You will not only learn how to collect and use this information, but have experience with it.
Made it on PerconaLive Frankfurt, 2018: https://www.percona.com/live/e18/sessions/mysql-performance-schema-in-action-the-complete-tutorial
MySQL has multiple timeouts variables to control its operations. This presentation focus on the purpose of each timeout variables and how it can be used.
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)Jean-François Gagné
To get better replication speed and less lag, MySQL implements parallel replication in the same schema, also known as LOGICAL_CLOCK. But fully benefiting from this feature is not as simple as just enabling it.
In this talk, I explain in detail how this feature works. I also cover how to optimize parallel replication and the improvements made in MySQL 8.0 and back-ported in 5.7 (Write Sets), greatly improving the potential for parallel execution on replicas (but needing RBR).
Come to this talk to get all the details about MySQL 5.7 and 8.0 Parallel Replication.
MySQL Administrator
Basic course
- MySQL 개요
- MySQL 설치 / 설정
- MySQL 아키텍처 - MySQL 스토리지 엔진
- MySQL 관리
- MySQL 백업 / 복구
- MySQL 모니터링
Advanced course
- MySQL Optimization
- MariaDB / Percona
- MySQL HA (High Availability)
- MySQL troubleshooting
네오클로바
http://neoclova.co.kr/
Performance Schema is a powerful diagnostic instrument for:
- Query performance
- Complicated locking issues
- Memory leaks
- Resource usage
- Problematic behavior, caused by inappropriate settings
- More
It comes with hundreds of options which allow precisely tuning what to instrument. More than 100 consumers store collected data.
In this tutorial, we will try all the important instruments out. We will provide a test environment and a few typical problems which could be hardly solved without Performance Schema. You will not only learn how to collect and use this information but have experience with it.
Tutorial at Percona Live Austin 2019
This ppt was used by Devrim at pgDay Asia 2017. He talked about some important facts about WAL - Transaction Logs or xlogs in PostgreSQL. Some of these can really come handy on a bad day
How to Take Advantage of Optimizer Improvements in MySQL 8.0Norvald Ryeng
MySQL 8.0 introduces several improvements to the query optimizer that may give improved performance for your queries. This presentation looks at what kind of queries the different improvements apply to, and the focus is on what you can do to get the most out of the optimizer improvements. The main topics are changes to the optimizer cost model, histograms, and new optimizer hints, but other improvements to how MySQL executes queries are also covered. The presentation includes many practical examples of how you can get a significant speedup for your MySQL queries.
Faster, better, stronger: The new InnoDBMariaDB plc
For MariaDB Enterprise Server 10.5, the default transactional storage engine, InnoDB, has been significantly rewritten to improve the performance of writes and backups. Next, we removed a number of parameters to reduce unnecessary complexity, not only in terms of configuration but of the code itself. And finally, we improved crash recovery thanks to better consistency checks and we reduced memory consumption and file I/O thanks to an all new log record format.
In this session, we’ll walk through all of the improvements to InnoDB, and dive deep into the implementation to explain how these improvements help everything from configuration and performance to reliability and recovery.
MySQL Parallel Replication: inventory, use-case and limitationsJean-François Gagné
In the last 24 months, MySQL replication speed has improved a lot thanks to implementing parallel replication. MySQL and MariaDB have different types of parallel replication; in this talk, I present in details the different implementations, with their limitations and the corresponding tuning parameters. I also present benchmark results from real Booking.com workloads. Finally, I discuss some deployments at Booking.com that benefits from parallel replication speed improvements.
MySQL backups overview. Characteristics of every backup type, including dumps, Xtrabackup and snapshots. Planning proper backup strategies. Why and how to test backups.
MySQL Parallel Replication: inventory, use-cases and limitationsJean-François Gagné
In the last 24 months, MySQL replication speed has improved a lot thanks to implementing parallel replication. MySQL and MariaDB have different types of parallel replication; in this talk, I present in detail the different implementations, with their limitations and the corresponding tuning parameters (covering MySQL 5.6, MariaDB 10.0, MariaDB 10.1 and MySQL 5.7). I also present benchmark results from real Booking.com workloads. Finally, I discuss some deployments at Booking.com that benefits from parallel replication speed improvements.
Ramp-Tutorial for MYSQL Cluster - Scaling with Continuous AvailabilityPythian
Rene Cannao's Ramp-Tutorial for MYSQL Cluster - Scaling with Continuous Availability. Rene, a Senior Operational DBA at PalominoDB.com, will guide attendees through a hands-on experience in the installation, configuration management and tuning of MySQL Cluster.
Agenda:
- MySQL Cluster Concepts and Architecture: we will review the principle of a fault-tolerant shared nothing architecture, and how this is implemented into NDB;
- MySQL Cluster processes : attendees will understand the various roles and interactions between Data Nodes, API Nodes and Management Nodes;
- Installation : we will install a minimal HA solution with MySQL Cluster on 3 virtual machines;
- Configuration of a basic system : upon describing the most important configuration parameters, Data/API/Management nodes will be configured and the Cluster launched;
- Loading data: the "world" schema will be imported into NDB using "in memory" and "disk based" storages; the attendees will experience how data changes are visible across API Nodes;
- Understand the NDB Storage Engine : internal implementation details will be explained, like synchronous replication, transaction coordinator, heartbeat, communication, failure detection and handling, checkpoint, etc;
- Query and schema design : attendees will understand the execution plan of queries with NDB, how SQL and Data Nodes communicate, how indexes and partitions are implemented, condition pushdown, join pushdown, query cache;
- Management and Administration: the attendees will test High Availability of NDB when a node become unavailable will learn how to read log file, how to stop/start any component of the Cluster to perform a rolling restart with no downtime, and how to handle a degraded setup;
- Backup and Recovery: attendees will be driven through the procedure of using NDB-native online backup and restore, and how this differs from mysqldump;
- Monitor and improve performance: attendee will learn how to boost performance tweaking variables according to hardware configuration and application workload
MySQL/MariaDB Parallel Replication: inventory, use-case and limitationsJean-François Gagné
In the last 24 months, MySQL/MariaDB replication speed has improved a lot, thanks to parallel replication. MySQL and MariaDB Server have different types of parallel replication; in this talk, I present the different implementations which will allow us to understand their limitations and tuning parameters. I am covering how to make parallel replication faster and what to avoid for maximizing its benefits. I also present benchmark results from Booking.com workloads. Finally, I discuss some deployments at Booking.com that take advantage of parallel replication speed improvements.
Percona XtraBackup - New Features and ImprovementsMarcelo Altmann
Percona XtraBackup is an open-source hot backup utility for MySQL - based servers that doesn't lock your database during the backup. In this talk, we will cover the latest development and new features introduced on Xtrabackup and its auxiliary tools: - Page Tracking - Azure Blob Storage Support - Exponential Backoff - Keyring Components - and more.
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, ConfluentHostedbyConfluent
Currently, Apache Kafka® uses Apache ZooKeeper™ to store its metadata. Data such as the location of partitions and the configuration of topics are stored outside of Kafka itself, in a separate ZooKeeper cluster. In 2019, we outlined a plan to break this dependency and bring metadata management into Kafka itself through a dynamic service that runs inside the Kafka Cluster. We call this the Quorum Controller.
In this talk, we’ll look at how the Quorum Controller works and how it integrates with other parts of the next-generation Kafka architecture, such as the Raft quorum and snapshotting mechanism. We’ll also explain how the Quorum Controller will simplify operations, improve security, and enhance scalability and performance.
Finally, we’ll look at some of the practicalities, such as how to monitor and run the Quorum Controller yourself. We’ll talk about some of the performance gains we’ve seen, and our plans for the future.
MySQL Cluster Asynchronous replication (2014) Frazer Clement
Slides from 2014 describing basic features and implementation of MySQL Cluster asynchronous (binlog) replication, with some monitoring and tuning guidance.
Similar to M|18 Deep Dive: InnoDB Transactions and Write Paths (20)
SkySQL is the first and only database-as-a-service (DBaaS) to perform workload analysis with advanced deep learning models, identifying and classifying discrete workload patterns so DBAs can better understand database workloads, identify anomalies and predict changes.
In this session, we’ll explain the concepts behind workload analysis and show how it can be used in the real world (and with sample real-world data) to improve database performance and efficiency by identifying key metrics and changes to cyclical patterns.
SkySQL uses best-of-breed software, and when it comes to metrics and monitoring that means Prometheus and Grafana. SkySQL Monitor is built on both, and provides customers with interactive dashboards for both real-time and historic metrics monitoring. In addition, it meets the same high availability and security requirements as other SkySQL components, ensuring metrics are always available and always secure.
In this session, we’ll explain how SkySQL Monitor works, walk through its dashboards and show how to monitor key metrics for performance and replication.
Introducing the R2DBC async Java connectorMariaDB plc
Not too long ago, a reactive variant of the JDBC driver was released, known as Reactive Relational Database Connectivity (R2DBC for short). While R2DBC started as an experiment to enable integration of SQL databases into systems that use reactive programming models, it now specifies a full-fledged service-provider interface that can be used to retrieve data from a target data source.
In this session, we’ll take a look at the new MariaDB R2DBC connector and examine the advantages of fully reactive, non-blocking development with MariaDB. And, of course, we’ll dive in and get a first-hand look at what it’s like to use the new connector with some live coding!
The capabilities and features of MariaDB Platform continue to expand, resulting in larger and more sophisticated production deployments – and the need for better tools. To provide DBAs with comprehensive, consolidating tooling, we created MariaDB Enterprise Tools: an easy-to-use, modular command-line interface for interacting with any part of MariaDB Platform.
In this session, we will provide a preview of the MariaDB Enterprise Client, walk through current and planned modules and discuss future plans for MariaDB Enterprise Tools – including SkySQL modules and the ability to create custom modules.
SkySQL implements a groundbreaking, state-of-the-art architecture based on Kubernetes and ServiceNow, and with a strong emphasis on cloud security – using compartmentalization and indirect access to secure and protect customer databases.
In this session, we’ll walk through the architecture of SkySQL and discuss how MariaDB leverages an advanced Kubernetes operator and powerful ServiceNow configuration/workflow management to deploy and manage databases on cloud infrastructure.
What to expect from MariaDB Platform X5, part 1MariaDB plc
MariaDB Platform X5 will be based on MariaDB Enterprise Server 10.5. This release includes Xpand, a fully distributed storage engine for scaling out, as well as many new features and improvements for DBAs and developers alike, including enhancements to temporal tables, additional JSON functions, a new performance schema, non-blocking schema changes with clustering and a Hashicorp Vault plugin for key management.
In this session, we’ll walk through all of the new features and enhancements available in MariaDB Enterprise Server 10.5. In addition, we will highlight those being backported to maintenance releases of MariaDB Enterprise Server 10.2, 10.3 and 10.4.
Explore our comprehensive data analysis project presentation on predicting product ad campaign performance. Learn how data-driven insights can optimize your marketing strategies and enhance campaign effectiveness. Perfect for professionals and students looking to understand the power of data analysis in advertising. for more details visit: https://bostoninstituteofanalytics.org/data-science-and-artificial-intelligence/
Techniques to optimize the pagerank algorithm usually fall in two categories. One is to try reducing the work per iteration, and the other is to try reducing the number of iterations. These goals are often at odds with one another. Skipping computation on vertices which have already converged has the potential to save iteration time. Skipping in-identical vertices, with the same in-links, helps reduce duplicate computations and thus could help reduce iteration time. Road networks often have chains which can be short-circuited before pagerank computation to improve performance. Final ranks of chain nodes can be easily calculated. This could reduce both the iteration time, and the number of iterations. If a graph has no dangling nodes, pagerank of each strongly connected component can be computed in topological order. This could help reduce the iteration time, no. of iterations, and also enable multi-iteration concurrency in pagerank computation. The combination of all of the above methods is the STICD algorithm. [sticd] For dynamic graphs, unchanged components whose ranks are unaffected can be skipped altogether.
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...John Andrews
SlideShare Description for "Chatty Kathy - UNC Bootcamp Final Project Presentation"
Title: Chatty Kathy: Enhancing Physical Activity Among Older Adults
Description:
Discover how Chatty Kathy, an innovative project developed at the UNC Bootcamp, aims to tackle the challenge of low physical activity among older adults. Our AI-driven solution uses peer interaction to boost and sustain exercise levels, significantly improving health outcomes. This presentation covers our problem statement, the rationale behind Chatty Kathy, synthetic data and persona creation, model performance metrics, a visual demonstration of the project, and potential future developments. Join us for an insightful Q&A session to explore the potential of this groundbreaking project.
Project Team: Jay Requarth, Jana Avery, John Andrews, Dr. Dick Davis II, Nee Buntoum, Nam Yeongjin & Mat Nicholas
M|18 Deep Dive: InnoDB Transactions and Write Paths
1. Deep Dive: InnoDB
Transactions and Write Paths
From the client connection to physical storage
Marko Mäkelä, Lead Developer InnoDB
Michaël de Groot, MariaDB Consultant
2. InnoDB
Concepts
A mini-transaction is an atomic
set of page reads or writes,
with write-ahead redo log.
A transaction writes undo log
before modifying indexes.
The read view of a transaction
may access the undo logs of
newer transactions to retrieve
old versions.
Purge may remove old undo logs
and delete-marked records
once no read view needs them.
Some terms that an Advanced
DBA should be familiar with
7. Step 1: SQL Layer
UPDATE talk SET attendees = 25 WHERE conference="M|18"
AND name="Deep Dive";
● Constructs a parse tree.
● Checks for permissions.
● Acquires metadata lock on the table name (prevent DDL) and opens the table.
● Retrieves index cardinality statistics from the storage engine(s).
● Constructs a query execution plan.
8. Step 2a: Read via the Storage Engine Interface
UPDATE talk SET attendees = 25 WHERE conference="M|18"
AND name="Deep Dive";
● Find the matching record(s) via the chosen index (secondary index or primary
key index), either via lookup or index scan.
● On the very first read, InnoDB will lazily start a transaction:
○ Assign a new DB_TRX_ID (incrementing number global in InnoDB)
○ No read view is needed, because this is a locking operation.
9. Step 2b: Filtering the Rows
UPDATE talk SET attendees = 25 WHERE conference="M|18"
AND name="Deep Dive";
● If the WHERE condition can only be satisfied by range scan or table scan, we
will have to filter out non-matching rows.
○ If Index Condition Pushdown is used, InnoDB evaluates the predicate and filters rows.
○ Else, InnoDB will return every record one by one, and the SQL layer will decide what to do.
● After returning a single record, a new mini-transaction has to be started
○ Repositioning the cursor (search for a key) is moderately expensive
○ Optimization: After 4 separate reads, InnoDB will prefetch 8 rows (could be improved)
10. Step 2c: Locking the rows
● InnoDB will write-lock each index leaf read
○ Note: InnoDB has no “table row locks”, but “record locks” in each index separately.
○ If using index condition pushdown, non-matching records are not locked.
● All records that may be changed are locked exclusively
○ When using a secondary index, all possibly matching records stay locked
○ Depending on the isolation level, InnoDB may unlock non-matching rows in primary key scan
(MySQL Bug #3300, “semi-consistent read”)
○ Updates with a WHERE condition on PRIMARY KEY are faster!
12. InnoDB Transaction Layer
The InnoDB transaction layer relies on atomically updated data pages forming
persistent data structures:
● Page allocation bitmap, index trees, undo log directories, undo logs
● Undo logs are the glue that make the indexes of a table look consistent.
● On startup, any pending transactions (with implicit locks) will be recovered
from undo logs.
○ Locks will be kept until incomplete transactions are rolled back, or
○ When found in binlog after InnoDB recovery completes, or
○ until explicit XA COMMIT or XA ROLLBACK.
● InnoDB supports non-locking reads (multi-versioning concurrency control)
by read view ‘snapshots’ that are based on undo logs.
13. 128
B-tree root page B-tree leaf page
DB_ROLL_PTR
A Page View of the InnoDB Transaction Layer
TRX_SYS
Page
(ibdata1)
Rollback segment header page
Table
.ibd
Index
History List (to-purge committed)
Uncommitted transactions
(pkN,child page)
(pkM,child page)
B-tree internal page
(pkN1,child page)
(pkN2,child page) (pk2,DB_TRX_ID,DB_ROLL_PTR,cols)
(pk1,DB_TRX_ID,DB_ROLL_PTR,cols)
Undo Log
(linked list of pages)
Undo Log
(linked list of pages)
14. DB_TRX_ID and DB_ROLL_PTR
● DB_TRX_ID is an increasing sequence number, global in InnoDB
○ Each user transaction (except read-only non-locking) gets a new one, upon transaction
create
○ While other transaction ids (GTID, binlog position) are in commit order, this transaction id is in
create order
○ Each record has metadata with its current DB_TRX_ID
○ If the DB_TRX_ID is in a list of uncommitted transactions, there is an implicit lock
● DB_ROLL_PTR is a pointer to the undo log record of the previous version
○ Stored in the record's metadata (hidden column in the clustered index)
○ Or a flag in it is set, meaning that there is no previous version (the record was inserted)
15. Step 3a: Creating undo log records
● On the first write, the transaction is created in the buffer pool:
○ Create or reuse a page (same size as other pages)
○ Create undo log header, update undo log directory (also known as “rollback segment page”).
○ Done in separate mini-transaction from undo log record write; optimized away in MariaDB 10.3
● Undo log pages are normal data pages: They can fill up and get persisted
● All changes of the same transaction are added to the same undo log page(s)
● Writing each undo log record means (in a mini-transaction):
○ Append to last undo page, or allocate a new undo page and write the record there.
○ InnoDB writes each undo log in advance, before updating each index affected by the row
operation.
● Undo log records need to be purged when they are no longer needed
16. UPDATE talk SET attendees = 25 WHERE conference="M|18"
AND name="Deep Dive";
● update_row(old_row,new_row) is invoked for each matching row
● InnoDB calculates an “update vector” and applies to the current row.
● The primary key (clustered index) record will be write-locked.
○ It may already have been locked in the “Read” step.
● An undo log record will be written in its own mini-transaction.
● The primary key index will be updated in its own mini-transaction.
● For each secondary index on updated columns, use 2 mini-transactions:
(1) search, lock, delete-mark old record, (2) insert new record.
Step 3b: Updating the Matching Rows
17. UPDATE talk SET attendees = 25 WHERE conference="M|18"
AND name="Deep Dive";
● Because autocommit was enabled and there was no BEGIN, the SQL layer will
automatically commit at the end of each statement.
● In InnoDB, commit updates the undo log information in a mini-transaction:
1. Update undo log directory (rollback segments) and undo log header pages.
2. Commit the mini-transaction (write to the redo log buffer).
3. Obtain the last written LSN (redo log sequence number) in the buffer.
4. Optionally, ensure that all redo log buffer is written at least up to the LSN.
Step 4a: Commit
18. Alternate step 4a: Rollbacks
● Rollback is expensive because
○ All changes are done in reverse (jumping read pointers)
○ Reading undo log or index pages that were evicted from the buffer pool to disk
○ Secondary index leaf records might need removal (must look up primary key records)
● For each undo log record from the end to start (or savepoint), the log is
applied in reverse:
○ For UPDATE: the old version of the data is read and an update vector calculated
● The same process as the update step is followed but reversed
○ 1 mini-transaction per clustered index
○ 2 mini-transactions per secondary index
● The undo log pages are freed in a maintenance operation called Truncate
undo log tail.
19. Step 4b: Cleaning up
After User COMMIT or ROLLBACK is done, there are some clean-up steps:
● Transactional row locks can be released and waiting threads woken up.
● InnoDB will also wake up any other transactions that waited for the locks.
● If the binlog is enabled, a commit event will be synchronously written to it.
○ An internal XA PREPARE transition must have been persisted in the InnoDB redo log earlier.
○ Only the internal XA PREPARE but not the XA COMMIT needs to be flushed to redo log.
○ Listen to the replication talk by Andrei Elkin (next talk in the same room)
● Finally, the SQL layer will release metadata locks on the table name - the
table can now be ALTERed again
21. The InnoDB Mini-Transaction Layer
Each layer provides service to the upper layers, relying on the service provided
by the lower layer, adding some encapsulation or refinement:
● File system (or the layers below): Block address translation
○ Wear leveling, bad block avoidance
○ Deduplication
○ Redundancy or replication for fault tolerance or HA
● The InnoDB mini-transaction layer depends on a write-ahead redo log.
○ Provides an AcID service of modifying a number of persistent pages.
■ Example: Inserting a record into a B-tree, with a series of page splits.
○ An atomic mini-transaction becomes durable when its log is fully written to the redo log.
○ Recovery: Any redo log records after the latest checkpoint will be parsed and applied to the
buffer pool copies of referenced persistent data file pages.
22. Mini-Transactions and Page Locking
A mini-transaction is not a user transaction. It is a short operation comprising:
● A list of index, tablespace or page locks that have been acquired.
● A list of modifications to pages.
There is no rollback for mini-transactions.
Locks of unmodified pages can be released any time. Commit will:
1. Append the log (if anything was modified) to the redo log buffer
2. Release all locks
Index page lock acquisition must avoid deadlocks (server hangs):
● Backward scans are not possible. Only one index update per
mini-transaction.
● Forward scan: acquire next page lock, release previous page (if not changed)
23. Mini-Transaction
Mini-Transactions: RW-Locks and Redo Logs
Memo:
Locks or
Buffer-Fixes
Index tree latch
(dict_index_t::lock):
covers internal pages
Tablespace latch
(fil_space_t::lock):
allocating/freeing pages
Log:
Page
Changes Data Files
FIL_PAGE_LSN
Flush (after log written)
Redo Log Files
(ib_logfile*)
Redo Log Buffer
(log_sys_t::buf)
Write ahead (of page flush) to log (make durable)
Buffer pool page
buf_page_t::oldest_
modification
commit
A mini-transaction commit
stores the log position (LSN) to
each changed page.
Recovery will redo changes:
Apply log if the page LSN is
older than the log record LSN.
Log position
(LSN)
24. Mini-transactions and Durability
Depending on when the redo log buffer was written to the redo log files, recovery
may miss some latest mini-transaction commits.
When innodb_flush_log_at_trx_commit=1, the only mini-transaction
persisted to disk is that of a User transaction commit. All earlier mini-transactions
are also persisted.
25. Mini-transaction Write operations on Index Trees
In InnoDB, a table is a collection of indexes: PRIMARY KEY (clustered index,
storing all materialized columns), and optional secondary indexes.
A mini-transaction can only modify one index at a time.
● Metadata change: Delete-mark (or unmark) a record
● Insert: optimistic (fits in leaf page) or pessimistic (allocate page+split+write)
○ Allocate operation modifies the page allocation bitmap page belonging to that page number
○ Tablespace extension occurs if needed (file extended, tablespace metadata change)
● Delete (purge,rollback): optimistic or pessimistic (merge+free page)
● Index lookup: Acquire index lock, dive from the root to the leaf page to edit
26. Read Mini-Transactions in Transactions
In InnoDB, a table is a collection of indexes: PRIMARY KEY (clustered index,
storing all materialized columns), and optional secondary indexes.
A mini-transaction can read only 1 secondary index, the clustered index and its
undo log records
● Index lookup: Look up a single record, dive from the root to the leaf page.
○ If this version is not to be seen, InnoDB will read the older version using DB_ROLL_PTR
● Index range scan: Index lookup followed by:
○ release locks except the leaf page
○ acquire next leaf page lock, release previous page lock, …
○ Typical use case: Filtering records for the read view, or for pushed-down WHERE clause
27. ● Encryption key rotation: Write a dummy change to an encrypted page, to cause
the page to be encrypted with a new key.
28. Flushing dirty pages
● This process runs asynchronously from the transactions
● Every second up to innodb_io_capacity iops will be used to write
changes to data files
● Includes (potentially no longer needed) undo log pages and the contents of
freed pages. To be improved: Collect garbage, do not write it!
● Data-at-rest encryption and page compression is done at this point
● Pages with oldest changes are written first
○ InnoDB keeps a list of dirty pages, sorted by the mini-transaction end LSN
29. On startup, any redo log after the latest checkpoint will be read and applied.
● A redo log checkpoint logically truncates the start of the redo log, up to the
LSN of the oldest dirty page.
● A clean shutdown ends with a log checkpoint.
● The InnoDB master thread periodically makes log checkpoints.
● When the redo log is full, a checkpoint will be initiated. Aggressive flushing
can occur on any mini-transaction write.
○ Until enough space is freed up, the server blocks any other write
○ This is very bad for performance, a bigger innodb_log_file_size would help
Redo Log Checkpoint (Truncating Redo Log)
30. What if the server was killed in the middle of a page flush?
● On Linux, killing a process may result in a partial write (usually 4096n bytes)
● Power outage? Disconnected cable? Cheating fsync()?
● On Windows with innodb_page_size=4k and matching sector size, no
problem.
Page writes first go to the doublewrite buffer (128 pages in ibdata1).
● If startup finds a corrupted page, it will try doublewrite buffer recovery.
● If it fails, then we are out of luck, because redo log records (usually) do not
contain the full page image.
Bugs and improvement potential: MDEV-11799, MDEV-12699, MDEV-12905
One More Layer: the Doublewrite Buffer
31. On startup, InnoDB performs some recovery steps:
● Read and apply all redo log since the latest redo log checkpoint.
○ While doing this, restore any half-written pages from the doublewrite buffer.
○ A clean shutdown ends with a log checkpoint, so the log would be empty.
● Resurrect non-committed transactions from the undo logs.
○ Tables referred to by the undo log records are opened.
○ This also resurrects implicit locks on all modified records, and table locks to block DDL.
○ XA PREPARE transactions will remain until explicit XA COMMIT or XA ROLLBACK.
● Initiate background ROLLBACK of active transactions.
○ Until this is completed, the locks may block some operations from new operations.
○ DBA hack: If you intend to DROP TABLE right after restart, delete the .ibd file upfront
Crash Recovery
33. Read Views and Isolation Levels
The lowest transaction isolation level is READ UNCOMMITTED.
● Returns the newest data from the indexes, “raw” mini-transaction view.
● Indexes may appear inconsistent both with each other and internally: an
UPDATE of a key may be observed as a DELETE, with no INSERT observed.
REPEATABLE READ uses non-locking read views:
● Created on the first read of a record from an InnoDB table, or
● Explicitly by START TRANSACTION WITH CONSISTENT SNAPSHOT
READ COMMITTED is similar to REPEATABLE READ, but the read view is
● Created at the start of each statement, on the first read of an InnoDB record
34. Locks and Isolation Levels
The highest transaction isolation level SERIALIZABLE implies
SELECT…LOCK IN SHARE MODE
● Only a committed record can be locked. It is always the latest version.
● DELETE and UPDATE will internally do SELECT…FOR UPDATE, acquiring
explicit exclusive locks.
● INSERT initially acquires an implicit lock, identified by DB_TRX_ID pointing
to a non-committed transaction.
35. Read Views, Implicit Locks, and the Undo Log
InnoDB may have to read undo logs for the purposes of:
● Multi-versioned (non-locking) reads based on a read view:
○ Get the appropriate version of a record, or
○ “Unsee” a record that was inserted in the future or whose deletion is visible.
● Determining if a secondary index record is locked by an active transaction.
○ The PRIMARY KEY record is implicitly locked if DB_TRX_ID refers to an active transaction.
○ On conflict, the lock waiter will convert the implicit exclusive lock into an explicit one.
● Transaction rollback
○ Rollback to the start of statement, for example on duplicate key error
○ ROLLBACK [TO SAVEPOINT]
○ Explicit locks will be retained until the transaction is completed
○ Implicit locks are released one by one, for each rolled-back row change.
36. MVCC Read Views and the Undo Log
Multi-versioning concurrency control: provide non-locking reads from a virtual
‘snapshot’ that corresponds to a read view: The current DB_TRX_ID and a list
of DB_TRX_IDs that are not committed at that time.
The read view contains:
● Any changes made by the current transaction.
● Any records that were committed when the read view was created.
● No changes of transactions that were started or committed after the read
view was created.
“Too new” INSERT can be simply ignored. DELETE and UPDATE make the
previous version available by pointing to the undo log record.
37. MVCC Read Views in the PRIMARY Index
The hidden PRIMARY index fields DB_TRX_ID, DB_ROLL_PTR and undo log
records constitute a singly-linked list, from the newest version to the oldest.
A non-locking read iterates this list in a loop, starting from the newest version:
● If DB_TRX_ID is in the read view:
Return the record, or skip it if delete-marked.
● If DB_TRX_ID is in the future and DB_ROLL_PTR carries the “insert” flag:
Skip the record (the oldest version was inserted in the future).
● Else: Find the undo log record by DB_ROLL_PTR , construct the previous
version, go to next iteration.
38. MVCC Read Views and Secondary Indexes
Secondary indexes only contain a PAGE_MAX_TRX_ID.
● Secondary indexes may contain multiple (indexed_col, pk) records pointing to
a single pk, one for each value of indexed_col.
● All versions except at most one must be delete-marked.
● If a secondary index record is delete-marked, MVCC must look up pk in the
PRIMARY index and attempt to construct a version that matches indexed_col,
to determine if (indexed_col, pk) exists in the read view.
● If the PAGE_MAX_TRX_ID is too new, for each record in the secondary index
leaf page, we must look up pk.
● For old read views, secondary index scan can be very expensive!
40. UPDATE talk SET attendees = 25 WHERE conference="M|18"
AND name="Deep Dive";
● If there were any secondary indexes that depend on the updated columns, we
delete-marked the records that contained the old values, before inserting
attendees=25. Those records should be removed eventually.
● In the primary key index, in MariaDB 10.3 we would set DB_TRX_ID=0 after
the history is no longer needed, to speed up future MVCC and locking checks.
● Also, the undo log pages are eventually freed for reuse, in a mini-transaction
Truncate undo log head.
Purging Old History
41. The purge threads can start removing history as soon as it is no longer visible
to any active read view.
● READ COMMITTED switches read views at the start of each statement.
● REPEATABLE READ keeps the read view until commit (or full rollback).
Undo log is by default stored in the InnoDB system tablespace (ibdata1).
● Because of purge lag, this file could grow a lot. InnoDB files never shrink!
● Open read views prevent purge from running.
● Purge lag (long History list length in SHOW ENGINE INNODB
STATUS) causes secondary indexes and undo logs to grow, and slows down
operations.
Purge Lag and Long-Running Transactions
42. Secondary Indexes and the Purge Lag
In the worst case, MVCC and implicit lock checks will require a clustered index
lookup for each secondary index record, followed by undo log lookups.
● Updates of indexed columns make this very expensive: O(versions·rows²).
● Ensure that purge can remove old history: Avoid long-lived read views.
● Avoid secondary index scans in long-lived read views.
● Watch the History list length in SHOW ENGINE INNODB STATUS!
43. Thank you!
To be continued in the other Deep Dive:
InnoDB Transactions and Replication