This document provides an overview and agenda for a presentation on MySQL 5.6 performance tuning and best practices. The presentation covers analyzing MySQL workload and internals, performance improvements in MySQL 5.6 and 5.7, benchmark results, and pending issues. It emphasizes the importance of monitoring systems to understand performance bottlenecks and the need for an iterative process of monitoring, tuning, optimizing, and improving database performance over time.
MySQL Scalability and Reliability for Replicated EnvironmentJean-François Gagné
You have a working application that is using MySQL: great! At the beginning, you are probably using a single database instance, and maybe – but not necessarily – you have replication for backups, but you are not reading from slaves yet. Scalability and reliability were not the main focus in the past, but they are starting to be a concern. Soon, you will have many databases and you will have to deal with replication lag. This talk will present how to tackle the transition.
We mostly cover standard/asynchronous replication, but we will also touch on Galera and Group Replication. We present how to adapt the application to become replication-friendly, which facilitate reading from and failing over to slaves. We also present solutions for managing read views at scale and enabling read-your-own-writes on slaves. We also touch on vertical and horizontal sharding for when deploying bigger servers is not possible anymore.
Are UNIQUE and FOREIGN KEYs still possible at scale, what are the downsides of AUTO_INCREMENTs, how to avoid overloading replication, what are the limits of archiving, … Come to this talk to get answers and to leave with tools for tackling the challenges of the future.
This tutorial covers all parallel replication implementation in MariaDB 10.0 and 10.1 and MySQL 5.6, 5.7 and 8.0 (including how it works in Group Replication).
MySQL and MariaDB have different types of parallel replication. In this tutorial, we present the different implementations that allow us to understand their limitations and tuning parameters. We cover how to make parallel replication faster and what to avoid for maximizing its benefits. We also present tests from Booking.com workloads.
Some of the subjects that are covered are group commit and optimistic parallel replication in MariaDB, the parallelism interval of MySQL and its Write Set optimization, and the ?slowing down the master to speed up the slave? optimization.
After this tutorial, you will know everything you need to implement and tune parallel replication in your environment. But more importantly, we will show how you can test parallel replication benefit in a non-disruptive way before deployment.
Using all of the high availability options in MariaDBMariaDB plc
MariaDB provides a number of high availability options, including replication with automatic failover and multi-master clustering. In this session Wagner Bianchi, Principal Remote DBA, provides a comprehensive overview of the high availability features in MariaDB, highlights their impact on consistency and performance, discusses advanced failover strategies and introduces new features such as casual reads and transparent connection failover.
MySQL and MariaDB though they share the same roots for replication .They support parallel replication , but they diverge the way the parallel replication is implemented.
This talk will tell the story of an analytics use case database from a non-OLAP and ACID-compliant RDBMS (MySQL) perspective.
I will cover the basics of the Clickhouse database Sample Clickhouse installation in a lab environment.
We are configuring Clickhouse for essential operations.
We will load the sample data set and monitor it.
We will query and visualize the results.
This talk will also base on how Kubernetes can help Clickhouse implementation via an operator.
Conclusions will include Do's and Don't of this emerging technology. Best practices and some advice around ingesting and analyzing terabytes of data efficiently.
MySQL Administrator
Basic course
- MySQL 개요
- MySQL 설치 / 설정
- MySQL 아키텍처 - MySQL 스토리지 엔진
- MySQL 관리
- MySQL 백업 / 복구
- MySQL 모니터링
Advanced course
- MySQL Optimization
- MariaDB / Percona
- MySQL HA (High Availability)
- MySQL troubleshooting
네오클로바
http://neoclova.co.kr/
MySQL Scalability and Reliability for Replicated EnvironmentJean-François Gagné
You have a working application that is using MySQL: great! At the beginning, you are probably using a single database instance, and maybe – but not necessarily – you have replication for backups, but you are not reading from slaves yet. Scalability and reliability were not the main focus in the past, but they are starting to be a concern. Soon, you will have many databases and you will have to deal with replication lag. This talk will present how to tackle the transition.
We mostly cover standard/asynchronous replication, but we will also touch on Galera and Group Replication. We present how to adapt the application to become replication-friendly, which facilitate reading from and failing over to slaves. We also present solutions for managing read views at scale and enabling read-your-own-writes on slaves. We also touch on vertical and horizontal sharding for when deploying bigger servers is not possible anymore.
Are UNIQUE and FOREIGN KEYs still possible at scale, what are the downsides of AUTO_INCREMENTs, how to avoid overloading replication, what are the limits of archiving, … Come to this talk to get answers and to leave with tools for tackling the challenges of the future.
This tutorial covers all parallel replication implementation in MariaDB 10.0 and 10.1 and MySQL 5.6, 5.7 and 8.0 (including how it works in Group Replication).
MySQL and MariaDB have different types of parallel replication. In this tutorial, we present the different implementations that allow us to understand their limitations and tuning parameters. We cover how to make parallel replication faster and what to avoid for maximizing its benefits. We also present tests from Booking.com workloads.
Some of the subjects that are covered are group commit and optimistic parallel replication in MariaDB, the parallelism interval of MySQL and its Write Set optimization, and the ?slowing down the master to speed up the slave? optimization.
After this tutorial, you will know everything you need to implement and tune parallel replication in your environment. But more importantly, we will show how you can test parallel replication benefit in a non-disruptive way before deployment.
Using all of the high availability options in MariaDBMariaDB plc
MariaDB provides a number of high availability options, including replication with automatic failover and multi-master clustering. In this session Wagner Bianchi, Principal Remote DBA, provides a comprehensive overview of the high availability features in MariaDB, highlights their impact on consistency and performance, discusses advanced failover strategies and introduces new features such as casual reads and transparent connection failover.
MySQL and MariaDB though they share the same roots for replication .They support parallel replication , but they diverge the way the parallel replication is implemented.
This talk will tell the story of an analytics use case database from a non-OLAP and ACID-compliant RDBMS (MySQL) perspective.
I will cover the basics of the Clickhouse database Sample Clickhouse installation in a lab environment.
We are configuring Clickhouse for essential operations.
We will load the sample data set and monitor it.
We will query and visualize the results.
This talk will also base on how Kubernetes can help Clickhouse implementation via an operator.
Conclusions will include Do's and Don't of this emerging technology. Best practices and some advice around ingesting and analyzing terabytes of data efficiently.
MySQL Administrator
Basic course
- MySQL 개요
- MySQL 설치 / 설정
- MySQL 아키텍처 - MySQL 스토리지 엔진
- MySQL 관리
- MySQL 백업 / 복구
- MySQL 모니터링
Advanced course
- MySQL Optimization
- MariaDB / Percona
- MySQL HA (High Availability)
- MySQL troubleshooting
네오클로바
http://neoclova.co.kr/
Boosting I/O Performance with KVM io_uringShapeBlue
Storage performance is becoming much more important. KVM io_uring attempts to bring the I/O performance of a virtual machine on almost the same level of bare metal. Apache CloudStack has support for io_uring since version 4.16. Wido will show the difference in performance io_uring brings to the table.
Wido den Hollander is the CTO of CLouDinfra, an infrastructure company offering total Webhosting solutions. CLDIN provides datacenter, IP and virtualization services for the companies within TWS. Wido den Hollander is a PMC member of the Apache CloudStack Project and a Ceph expert. He started with CloudStack 9 years ago. What attracted his attention is the simplicity of CloudStack and the fact that it is an open-source solution. During the years Wido became a contributor, a PMC member and he was a VP of the project for a year. He is one of our most active members, who puts a lot of efforts to keep the project active and transform it into a turnkey solution for cloud builders.
-----------------------------------------
The CloudStack European User Group 2022 took place on 7th April. The day saw a virtual get together for the European CloudStack Community, hosting 265 attendees from 25 countries. The event hosted 10 sessions with from leading CloudStack experts, users and skilful engineers from the open-source world, which included: technical talks, user stories, new features and integrations presentations and more.
------------------------------------------
About CloudStack: https://cloudstack.apache.org/
Oracle Real Application Clusters 19c- Best Practices and Internals- EMEA Tour...Sandesh Rao
In this session, I will cover under-the-hood features that power Oracle Real Application Clusters (Oracle RAC) 19c specifically around Cache Fusion and Service management. Improvements in Oracle RAC helps in integration with features such as Multitenant and Data Guard. In fact, these features benefit immensely when used with Oracle RAC. Finally we will talk about changes to the broader Oracle RAC Family of Products stack and the algorithmic changes that helps quickly detect sick/dead nodes/instances and the reconfiguration improvements to ensure that the Oracle RAC Databases continue to function without any disruption
Apache Ignite vs Alluxio: Memory Speed Big Data AnalyticsDataWorks Summit
Apache Ignite vs Alluxio: Memory Speed Big Data Analytics - Apache Spark’s in memory capabilities catapulted it as the premier processing framework for Hadoop. Apache Ignite and Alluxio, both high-performance, integrated and distributed in-memory platform, takes Apache Spark to the next level by providing an even more powerful, faster and scalable platform to the most demanding data processing and analytic environments.
Speaker
Irfan Elahi, Consultant, Deloitte
Advanced Percona XtraDB Cluster in a nutshell... la suiteKenny Gryp
Percona XtraDB Cluster is a high availability and high scalability solution for MySQL clustering. Percona XtraDB Cluster integrates Percona Server with the Galera synchronous replication library in a single product package which enables you to create a cost-effective MySQL cluster.
Since three years during Percona Live we initiate people to this technology... but what's next ? This tutorial is the continuation. It targets users that already have experience with PXC and want to go further.
This tutorial will cover the following topics:
- monitoring and trending
- problem solving
- limitations, when not to choose for PXC
- how to test ? (benchmark)
- schema changes
- backups
- multi datacenter
- advanced load balancing with HA Pproxy and Maxscale
- fine tune some important variables like galera cache, flow control limit, ...
In this session, we looked at five things you might not know about the Oracle Database or might have forgotten. For each topic, I explained the functionality and demonstrated the benefits using real-world examples. The topics covered apply to anyone running Oracle Database 11g and up, including Standard Edition, with only a few minor exceptions.
Oracle 21c: New Features and Enhancements of Data Pump & TTSChristian Gohmann
At the end of the year 2020, Oracle released 21c on its Cloud infrastructure. The on-premises version will follow later this year. As with every new Oracle version, the Data Pump utility gets new features or enhancements for existing features.
This presentation gives an overview of the enhancements of Data Pump and Transportable Tablespaces. The following list is an excerpt of the points I will talk about
- Simultaneous use of EXCLUDE and INCLUDE
- Parallelized import of metadata during a TTS import operation
- Checksum support for dump files
- Direct access to Oracle Cloud Object Store for exports and imports
ProxySQL is a popular database proxy for MySQL/MariaDB servers. This focuses on the possible High availability options for ProxySQL and operations of inbuilt clustering feature in ProxySQL. This tech talk was presented at Mydbops Database Meetup on 27-04-2019 by Aakash M, Database Administrator with Mydbops and Vignesh Prabhu, Database Administrator with Mydbops.
We will show the advantages of having a geo-distributed database cluster and how to create one using Galera Cluster for MySQL. We will also discuss the configuration and status variables that are involved and how to deal with typical situations on the WAN such as slow, untrusted or unreliable links, latency and packet loss. We will demonstrate a multi-region cluster on Amazon EC2 and perform some throughput and latency measurements in real-time (video http://galeracluster.com/videos/using-galera-replication-to-create-geo-distributed-clusters-on-the-wan-webinar-video-3/)
Protecting the Galaxy - Multi-Region Disaster Recovery with OpenStack and CephSean Cohen
IT organizations require a disaster recovery strategy addressing outages with loss of storage, or extended loss of availability at the primary site. Applications need to rapidly migrate to the secondary site and transition with little or no impact to their availability.This talk will cover the various architectural options and levels of maturity in OpenStack services for building multi-site configurations using the Mitaka release. We’ll present the latest capabilities for Volume, Image and Object Storage with Ceph as the backend storage solution, and look at the future developments the OpenStack and Ceph communities are driving to improve and simplify the relevant use cases.
Slides from OpenStack Austin Summit 2016 session: http://alturl.com/hpesz
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...ScaleGrid.io
Compare top PostgreSQL high availability frameworks - PostgreSQL Automatic Failover (PAF), Replication Manager (repmgr) and Patroni to improve your app uptime. ScaleGrid blog - https://scalegrid.io/blog/whats-the-best-postgresql-high-availability-framework-paf-vs-repmgr-vs-patroni-infographic/
4.17.0 is the latest Apache CloudStack major release. In this talk, Nicolas goes through the new features introduced in this version from an administrator/user perspective, explaining their benefits and the problems those features resolve. He also ran a live demo to see the new features in action.
Nicolas Vazquez is a Senior Software Engineer at ShapeBlue and is a PMC member of the Apache CloudStack project. He spends his time designing and implementing features in Apache CloudStack and can be seen acting as a release manager also. Nicolas is based in Uruguay and is a father of a young girl. He is a fan of sports, enjoys playing tennis and football. In his free time, he also enjoys reading and listening to economic and political materials.
-----------------------------------------
CloudStack Collaboration Conference 2022 took place on 14th-16th November in Sofia, Bulgaria and virtually. The day saw a hybrid get-together of the global CloudStack community hosting 370 attendees. The event hosted 43 sessions from leading CloudStack experts, users and skilful engineers from the open-source world, which included: technical talks, user stories, new features and integrations presentations and more.
Built-in MySQL Replication is known for its capability to enable to scale reads easily. However, there are some limitations and known issues with this solution because of the asynchronous nature of this replication. This talk will describe another way of doing MySQL replication, by using synchronous replication, available in Percona XtraDB Cluster. The open source solution will be explained and compared to traditional asynchronous MySQL replication, as well as some known use cases will be described. Percona XtraDB Cluster is an, open source, high availability and high scalability solution for MySQL clustering. Features include: Synchronous replication, Multi-master replication support, Parallel replication, Automatic node provisioning.
Modeling Data and Queries for Wide Column NoSQLScyllaDB
Discover how to model data for wide column databases such as ScyllaDB and Apache Cassandra. Contrast the differerence from traditional RDBMS data modeling, going from a normalized “schema first” design to a denormalized “query first” design. Plus how to use advanced features like secondary indexes and materialized views to use the same base table to get the answers you need.
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)Aurimas Mikalauskas
Is my MySQL server configured properly? Should I run Community MySQL, MariaDB, Percona or WebScaleSQL? How many innodb buffer pool instances should I run? Why should I NOT use the query cache? How do I size the innodb log file size and what IS that innodb log anyway? All answers are inside.
Aurimas Mikalauskas is a former Percona performance consultant and architect currently writing and teaching at speedemy.com. He's been involved with MySQL since 1999, scaling and optimizing MySQL backed systems since 2004 for companies such as BBC, EngineYard, famous social networks and small shops like EstanteVirtual, Pine Cove and hundreds of others.
Additional content mentioned in the presentation can be found here: http://speedemy.com/17
Boosting I/O Performance with KVM io_uringShapeBlue
Storage performance is becoming much more important. KVM io_uring attempts to bring the I/O performance of a virtual machine on almost the same level of bare metal. Apache CloudStack has support for io_uring since version 4.16. Wido will show the difference in performance io_uring brings to the table.
Wido den Hollander is the CTO of CLouDinfra, an infrastructure company offering total Webhosting solutions. CLDIN provides datacenter, IP and virtualization services for the companies within TWS. Wido den Hollander is a PMC member of the Apache CloudStack Project and a Ceph expert. He started with CloudStack 9 years ago. What attracted his attention is the simplicity of CloudStack and the fact that it is an open-source solution. During the years Wido became a contributor, a PMC member and he was a VP of the project for a year. He is one of our most active members, who puts a lot of efforts to keep the project active and transform it into a turnkey solution for cloud builders.
-----------------------------------------
The CloudStack European User Group 2022 took place on 7th April. The day saw a virtual get together for the European CloudStack Community, hosting 265 attendees from 25 countries. The event hosted 10 sessions with from leading CloudStack experts, users and skilful engineers from the open-source world, which included: technical talks, user stories, new features and integrations presentations and more.
------------------------------------------
About CloudStack: https://cloudstack.apache.org/
Oracle Real Application Clusters 19c- Best Practices and Internals- EMEA Tour...Sandesh Rao
In this session, I will cover under-the-hood features that power Oracle Real Application Clusters (Oracle RAC) 19c specifically around Cache Fusion and Service management. Improvements in Oracle RAC helps in integration with features such as Multitenant and Data Guard. In fact, these features benefit immensely when used with Oracle RAC. Finally we will talk about changes to the broader Oracle RAC Family of Products stack and the algorithmic changes that helps quickly detect sick/dead nodes/instances and the reconfiguration improvements to ensure that the Oracle RAC Databases continue to function without any disruption
Apache Ignite vs Alluxio: Memory Speed Big Data AnalyticsDataWorks Summit
Apache Ignite vs Alluxio: Memory Speed Big Data Analytics - Apache Spark’s in memory capabilities catapulted it as the premier processing framework for Hadoop. Apache Ignite and Alluxio, both high-performance, integrated and distributed in-memory platform, takes Apache Spark to the next level by providing an even more powerful, faster and scalable platform to the most demanding data processing and analytic environments.
Speaker
Irfan Elahi, Consultant, Deloitte
Advanced Percona XtraDB Cluster in a nutshell... la suiteKenny Gryp
Percona XtraDB Cluster is a high availability and high scalability solution for MySQL clustering. Percona XtraDB Cluster integrates Percona Server with the Galera synchronous replication library in a single product package which enables you to create a cost-effective MySQL cluster.
Since three years during Percona Live we initiate people to this technology... but what's next ? This tutorial is the continuation. It targets users that already have experience with PXC and want to go further.
This tutorial will cover the following topics:
- monitoring and trending
- problem solving
- limitations, when not to choose for PXC
- how to test ? (benchmark)
- schema changes
- backups
- multi datacenter
- advanced load balancing with HA Pproxy and Maxscale
- fine tune some important variables like galera cache, flow control limit, ...
In this session, we looked at five things you might not know about the Oracle Database or might have forgotten. For each topic, I explained the functionality and demonstrated the benefits using real-world examples. The topics covered apply to anyone running Oracle Database 11g and up, including Standard Edition, with only a few minor exceptions.
Oracle 21c: New Features and Enhancements of Data Pump & TTSChristian Gohmann
At the end of the year 2020, Oracle released 21c on its Cloud infrastructure. The on-premises version will follow later this year. As with every new Oracle version, the Data Pump utility gets new features or enhancements for existing features.
This presentation gives an overview of the enhancements of Data Pump and Transportable Tablespaces. The following list is an excerpt of the points I will talk about
- Simultaneous use of EXCLUDE and INCLUDE
- Parallelized import of metadata during a TTS import operation
- Checksum support for dump files
- Direct access to Oracle Cloud Object Store for exports and imports
ProxySQL is a popular database proxy for MySQL/MariaDB servers. This focuses on the possible High availability options for ProxySQL and operations of inbuilt clustering feature in ProxySQL. This tech talk was presented at Mydbops Database Meetup on 27-04-2019 by Aakash M, Database Administrator with Mydbops and Vignesh Prabhu, Database Administrator with Mydbops.
We will show the advantages of having a geo-distributed database cluster and how to create one using Galera Cluster for MySQL. We will also discuss the configuration and status variables that are involved and how to deal with typical situations on the WAN such as slow, untrusted or unreliable links, latency and packet loss. We will demonstrate a multi-region cluster on Amazon EC2 and perform some throughput and latency measurements in real-time (video http://galeracluster.com/videos/using-galera-replication-to-create-geo-distributed-clusters-on-the-wan-webinar-video-3/)
Protecting the Galaxy - Multi-Region Disaster Recovery with OpenStack and CephSean Cohen
IT organizations require a disaster recovery strategy addressing outages with loss of storage, or extended loss of availability at the primary site. Applications need to rapidly migrate to the secondary site and transition with little or no impact to their availability.This talk will cover the various architectural options and levels of maturity in OpenStack services for building multi-site configurations using the Mitaka release. We’ll present the latest capabilities for Volume, Image and Object Storage with Ceph as the backend storage solution, and look at the future developments the OpenStack and Ceph communities are driving to improve and simplify the relevant use cases.
Slides from OpenStack Austin Summit 2016 session: http://alturl.com/hpesz
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...ScaleGrid.io
Compare top PostgreSQL high availability frameworks - PostgreSQL Automatic Failover (PAF), Replication Manager (repmgr) and Patroni to improve your app uptime. ScaleGrid blog - https://scalegrid.io/blog/whats-the-best-postgresql-high-availability-framework-paf-vs-repmgr-vs-patroni-infographic/
4.17.0 is the latest Apache CloudStack major release. In this talk, Nicolas goes through the new features introduced in this version from an administrator/user perspective, explaining their benefits and the problems those features resolve. He also ran a live demo to see the new features in action.
Nicolas Vazquez is a Senior Software Engineer at ShapeBlue and is a PMC member of the Apache CloudStack project. He spends his time designing and implementing features in Apache CloudStack and can be seen acting as a release manager also. Nicolas is based in Uruguay and is a father of a young girl. He is a fan of sports, enjoys playing tennis and football. In his free time, he also enjoys reading and listening to economic and political materials.
-----------------------------------------
CloudStack Collaboration Conference 2022 took place on 14th-16th November in Sofia, Bulgaria and virtually. The day saw a hybrid get-together of the global CloudStack community hosting 370 attendees. The event hosted 43 sessions from leading CloudStack experts, users and skilful engineers from the open-source world, which included: technical talks, user stories, new features and integrations presentations and more.
Built-in MySQL Replication is known for its capability to enable to scale reads easily. However, there are some limitations and known issues with this solution because of the asynchronous nature of this replication. This talk will describe another way of doing MySQL replication, by using synchronous replication, available in Percona XtraDB Cluster. The open source solution will be explained and compared to traditional asynchronous MySQL replication, as well as some known use cases will be described. Percona XtraDB Cluster is an, open source, high availability and high scalability solution for MySQL clustering. Features include: Synchronous replication, Multi-master replication support, Parallel replication, Automatic node provisioning.
Modeling Data and Queries for Wide Column NoSQLScyllaDB
Discover how to model data for wide column databases such as ScyllaDB and Apache Cassandra. Contrast the differerence from traditional RDBMS data modeling, going from a normalized “schema first” design to a denormalized “query first” design. Plus how to use advanced features like secondary indexes and materialized views to use the same base table to get the answers you need.
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)Aurimas Mikalauskas
Is my MySQL server configured properly? Should I run Community MySQL, MariaDB, Percona or WebScaleSQL? How many innodb buffer pool instances should I run? Why should I NOT use the query cache? How do I size the innodb log file size and what IS that innodb log anyway? All answers are inside.
Aurimas Mikalauskas is a former Percona performance consultant and architect currently writing and teaching at speedemy.com. He's been involved with MySQL since 1999, scaling and optimizing MySQL backed systems since 2004 for companies such as BBC, EngineYard, famous social networks and small shops like EstanteVirtual, Pine Cove and hundreds of others.
Additional content mentioned in the presentation can be found here: http://speedemy.com/17
For years, the common industry perception has been that MySQL is faster and easier to use than PostgreSQL. PostgreSQL is perceived as more powerful, more focused on data integrity, and stricter at complying with SQL specifications, but correspondingly slower and more complicated to use.
Like many perceptions formed in the past, these things aren\'t as true with the current generation of releases as they used to be.
Getting to Know MySQL Enterprise MonitorMark Leith
MySQL Enterprise Monitor is the monitoring and management solution for DBAs and developers delivered as part of MySQL Enterprise Edition. It provides background monitoring, alerting, trending, and analysis of the MySQL database and the statement traffic that is running within it.
View this session to learn how to install/configure, customize, and use MySQL Enterprise Monitor to suit your environment. Whether you use a single server or have hundreds of instances, MySQL Enterprise Monitor can provide great insights into how your environment is performing.
The MySQL sys schema was integrated fully into MySQL Server from version 5.7.7 and has been improved in MySQL 8.0. Whether you are a DBA trying to determine where the resources are being used on your database instance and by whom, or a developer trying to figure out why your MySQL statements are running too slowly, the MySQL sys schema can help. Join this session to learn how to better use the MySQL sys schema to answer your day-to-day questions—from the original developer of the MySQL sys schema.
The Proper Care and Feeding of a MySQL Database for Busy Linux Admins -- SCaL...Dave Stokes
If you are a Linux administrator and ALSO have to take care of a MySQL databases, this presentation if for you, While it will not turn you instantly into a DBA it will help you understand how to properly care and feed your instances
Proper Care and Feeding of a MySQL Database for Busy Linux AdministratorsDave Stokes
Do you 'also' have MySQL DBA responsibilities as part of your job but have no DBA training? This presentation covers a lot of the DBA level info for those who have a Linux admin background but are not DBAs
MariaDB / MySQL tripping hazard and how to get out again?FromDual GmbH
MySQL is NOT MariaDB, KISS, MyISAM Table Locking, big ibdata1, Noisy Neighbours, No BLOBs in a RDBMS, Disk full!, Locking, Crash, Out of Memory, Oom, Restart, Crash with Stacktrace, Tables without a Primary Key, Server has gone away! NUMA, Slave Lag, Sending data, Too many connections, Simple Query Tuning, Galera Cluster, Progress bar
The Peoper Care and Feeding of a MySQL Server for Busy Linux AdminDave Stokes
Webcast 16 September 2015 with a big thanks to SolarWinds. This is a collection of best practices for Linux Admin who 'also have' database responsibility but are not DBAs
Linuxfest Northwest Proper Care and Feeding Of a MySQL for Busy Linux AdminsDave Stokes
Did they also hand you the responsibility for the database with all the other Linux admin duties but you have no DBA training? This is for you. Learn the tricks they don't mention to keep you data safe
XPages Blast - Ideas, Tips and More.
This session will take you on a roller-coaster ride through the "best of the best" ideas and time-saving techniques for creating world-class XPages applications.
Thirty all new top tips - this is going to be fast-paced and packed with loads of information you will refer to time and time again! Everything from introductory tips on getting started with XPages, to complex tips - such as making use of Java. Also, the support app showing all of the tips and tricks.
Not a DBA but your work thinks you are? This is the session for you. We will give you a crash course into the most valuable variables, places to look, and gotcha's. This is not designed to replace traditional training but rather to ensure you at least have the most basic skills to ensure your foray into the world of MySQL DBA's gets off on a good foot.
Ao contrário do que todo mundo pensa, o Doctrine não é somente um Mapeador de objeto relacional. É um projeto focado em desenvolver soluções para persistência de dados e tecnologias relacionadas. Nessa palestra você verá o uso de várias ferramentas que fazem o uso de pacotes do projeto que serão úteis no seu ambiente desenvolvimento desde a implementação ao deploy.
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...Valeriy Kravchuk
Linux with kernels 2.6+. provides different ways to add user probes to almost every other line of code dynamically, and collect the resulting trace and profiling data in a safe and efficient way. This session discusses basic use of ftrace, perf, bcc tools and bpftrace utility, highlights typical problems MariaDB DBAs and developers may hit while trying to apply them, as well as solutions to some of them.
Details:
• DevOps and Business Intelligence?
• CI/CD Pipelines: What are they?
• Database Deployments: State based vs Migration based
• Snowflake features for CI/CD
• Azure DevOps: Build and Release Pipelines
• Putting it all together: End to End solution
• Demo
Linux /proc filesystem for MySQL DBAs - FOSDEM 2021Valeriy Kravchuk
Tools and approaches based on /proc sampling (like 0x.tools by Tanel Poder or ad hoc scripts) allow to measure individual thread level activity in MySQL server on Linux, like thread sleep states, currently executing system calls and kernel wait locations. If needed you can drill down into CPU usage of any thread or the system as a whole. Historical data can be captured for post factum analysis, without much impact on the system and no need to install or change anything in its configuration. In this presentation I am going to summarize what's possible with /proc and show useful examples for MySQL DBAs.
Hadoop Operations: Keeping the Elephant Running SmoothlyMichael Arnold
Pune Hadoop Admins Meetup
From its beginnings years ago at large Internet sites, Hadoop is spreading everywhere. There are multitudes of cool and interesting things that Hadoop allows your organization to do, but running the actual infrastructure may not be as sexy as the application(s) running on top. Operations can be pure grunt-work, exacerbated by the fact that there is usually one machine out of dozens (or more) that is throwing a wrench in the works. In this talk, I will cover my experiences of running Hadoop, provide some recommended practices to simplify your days and nights in the trenches, and highlight some of the lessons learned along the way.
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
3. The following is intended to outline our general product
direction. It is intended for information purposes only, and
may not be incorporated into any contract.
It is not a commitment to deliver any material, code, or
functionality, and should not be relied upon in making
purchasing decisions. The development, release, and timing
of any features or functionality described for Oracle’s
products remains at the sole discretion of Oracle.
4. Are you Dimitri?..
§ Yes, it's me :-)
§ Hello from Paris! ;-)
§ Passionated by Systems and Databases Performance
§ Previous 15 years @Sun Benchmark Center
§ Started working on MySQL Performance since v3.23
§ But during all that time just for fun only ;-)
§ Since last years officially @MySQL Performance full time now
§ http://dimitrik.free.fr/blog / @dimitrik_fr
5. Agenda
§ Overview
§ Analyzing MySQL Workload
§ Analyzing and Understanding of MySQL Internals
§ Performance improvements in MySQL 5.6 (and 5.7)
§ Benchmark results
§ Pending issues..
§Q&A
11. Why benchmarking MySQL?..
●
A good benchmark testing may help you understand ahead the
resistance of your solution to incoming potential problems ;-)
12. Why benchmarking MySQL?..
●
But keep it in mind:
●
Even a very powerful solution but
leaved in wrong hands may still be
easily broken!... :-)
13. The Main MySQL Performance Tuning
#1 Best Practice is... ???..
14. The Main MySQL Performance Tuning
#1 Best Practice is... ???..
USE YOUR BRAIN !!! :-)
15. The Main MySQL Performance Tuning
#1 Best Practice is... ???..
USE YOUR BRAIN !!! :-)
AND THIS IS THE
AND THIS IS THE
MAIN SLIDE! ;-))
MAIN SLIDE! ;-))
16. Before we started..
●
Please, keep in mind:
●
NOBODY knows everything ;-))
●
There is no absolute true in any topic around..
●
The best answer in most cases will be probably “It depends..” ;-))
●
So, again, “USE YOUR BRAIN!” is the best advice and the best option
●
Also, knowledge and understanding of problems are changing all the
time..
●
And probably even what I'll tell you today is already obsolete. ;-))
●
Enjoy thinking and digging problems deeply ;-))
●
MySQL Performance is a very fun topic (specially current days ;-))
17. Different Approach for different problems
●
You are discovering a production workload..
●
●
You are trying to understand why your production is running
slower time to time..
●
●
Tracing, debugging, analyzing, discovering of new problems ;-)
You are looking for a new platform for existing production
workload (or new apps under dev.)..
●
●
Full discovery..
Workload simulation, benchmarking, discovering of the next level
issues..
etc...
18. They all have something in common!
●
Monitoring !..
●
Choose a tool you're familiar with (or install one and become familiar)
●
Use a tool you can completely trust ;-)
●
●
●
●
Keep in mind that sometimes you may need a 5-10sec interval
measurements (or even less).. - not every tool is allowing..
Keep a history of your monitoring to be able to compare “good” and
“bad” cases..
When something is starting to go wrong, usually it'll be not in the place
which was always problematic, but in the place started to have a
different behavior.. - and your goal is to find it ;-)
Always monitor your HW and OS !!!
19. MySQL Enterprise Monitor (MEM) v.3.0
●
Absolutely fantastic product!
●
Try it! (and buy it if you like it! ;-) - improve your daily work experience!)
20. Monitoring & Analyzing with dim_STAT (as you ask ;)
●
All my graphs are built with it (download: http://dimitrik.free.fr)
●
All System load stats (CPU, I/O, Network, RAM, Processes,...)
●
Manly for Solaris & Linux, but any other UNIX too :-)
●
Add-Ons for Oracle, MySQL, PostgreSQL, Java, etc.
●
MySQL Add-Ons:
–
–
mysqlLOAD : compact data, multi-host monitoring oriented
–
mysqlWAITS : top wait events from Performance SCHEMA
–
InnodbSTAT : most important data from “show innodb status”
–
innodbMUTEX : monitoring InnoDB mutex waits
–
●
mysqlSTAT : all available data from “show status”
innodbMETRICS : all counters from the METRICS table
And any other you want to add! :-)
21. Think “Database Performance” from the beginning!
●
Server:
●
●
OS is important! - Linux, Solaris, etc.. (and Windows too!)
●
●
Having faster CPU is still better! 32 cores is good enough ;-)
Right malloc() lib!! (Linux: jemalloc, Solaris: libumem)
Storage:
●
●
SSD helping random access! (index/data) more and more cheaper
●
FS is important! - ZFS, UFS, QFS, VxFS, EXT3, EXT4, XFS, etc..
●
O_DIRECT or not O_DIRECT, AIO or not AIO, and be aware of bugs! ;-)
●
●
Don't use slow disks! (except if this is a test validation goal :-))
Do some generic I/O tests first (Sysbench, IObench, iozone, etc.)
Don't forget network !! :-) (faster is better, 10Gbit is great!)
22. Seek for your best option..
Performance
Lower Price
Security
23. What to monitor on Linux?..
●
First of all use the best Linux for you!
●
●
Install & use “jemalloc”; if XFS has problems, use EXT4 (nobarrier!)
●
●
Or ORACLE Linux if you don't know which one to choose ;-)
Use AIO + O_DIRECT, don't use “cfq” IO scheduler!..
Always keep an eye on:
●
RunQueue(!), CPU, RAM, Swap in/outProcesses: vmstat, top, psSTAT
●
Storage level: iostat, ..
●
Network: netLOAD, nicstat, …
●
Overall system activity: # perf top -z
–
●
perf: excellent profiler!
IMPORTANT : system monitoring usually helps to dig DB issues!
24. Know/ test/ check your platform limits / “features”..
●
My backup is finished on Linux faster than on Solaris same HW
●
●
●
Be sure first there is really no more I/O activity once backup is “finished”
Keep in mind Linux buffering..
Linux distro: MySQL Performance has x4 regression! Fix it!
●
How did you see it? – Our QA test is taking x4 times more time..
●
Which engine? – InnoDB..
●
What is innodb_flush_log_at_trx_commit value? – set to 1.. why?
●
Tried innodb_flush_log_at_trx_commit=2 ?.. – Oh! You fixed it!! Thanks!!
●
Wait, what did you “improve” recently in distro? – FS flushing, why?..
●
Well, the test in fact is proving that you did not “sync” on every fsync()
before, that's all.. But now in your FS flushing you get it fixed ;-)
25. The Infinitive Loop of Database Tuning...
Application
Application
DB Engine
DB Engine
OS
OS
Server
Server
Storage
Storage
#1 Monitoring
●#1 Monitoring
●#2 Tuning
●#2 Tuning
●#3 Optimization
●#3 Optimization
●#4 Improvement(s)
●#4 Improvement(s)
●#5 …
●#5 …
●...
●...
●goto #1
●goto #1
●
26. The Infinitive Loop of Database Tuning...
Even if in
Even if in
95% cases
95% cases
the problem
the problem
Is here!!! :-)
Is here!!! :-)
Application
Application
DB Engine
DB Engine
OS
OS
Server
Server
Storage
Storage
#1 Monitoring
●#1 Monitoring
●#2 Tuning
●#2 Tuning
●#3 Optimization
●#3 Optimization
●#4 Improvement(s)
●#4 Improvement(s)
●#5 …
●#5 …
●...
●...
●goto #1
●goto #1
●
28. MySQL Design
●
Multi-Threaded database
●
●
Simplified data access!
●
●
Fast context switch!
Concurrent access?.. Scalability?..
Storage Engines
●
Initially: MyISAM only
●
Then, with InnoDB: started to match expectations of a “true RDBMS” ;-)
●
Many other engines (MEMORY, CSV, NDB, PBXT, etc.)
●
CREATE TABLE ... ENGINE=<NAME_OF_ENGINE>
●
ALTER TABLE ... ENGINE=<NAME_OF_ENGINE>
●
Did you choose a right Engine?..
29. MyISAM Engine (since 1994)
●
Non-transactional! / No fast recovery! :-)
●
Cache
●
●
Data => FS cache
●
●
Index only
mysql> flush tables;
Single Writer @Table
●
Main bottleneck! => single writer
●
Solutions: delayed inserts, low priority
●
Query plan: Index forcing may be necessary (hint)
●
Extremely simple and lightweight
30. Why MySQL + MyISAM was successful ?..
●
Full Text search queries out-of-the-box!
●
SELECT count(*) ... :-))
●
Extremely SIMPLE!
●
my.conf => configuration parameters; mysql.server start / stop
●
Database => directory
●
Table => directory/Table.MYD, Table.MYI, Table.frm
●
$ cp Base1/Table.* /other/mysql/Base2
●
Data binary compatibility! (ex: reports via NFS)
●
Replication ready!
●
Very FAST! (until some limit :-))
●
RW workload is killing.. (but on 2CPU servers it was ok ;-))
32. InnoDB changing the game (since 2001)
●
Row-level locking
●
Index-only reads
●
True transactions / UNDO
●
Auto recovery
●
Double write / Checksums
●
Tablespaces or File-per-Table option
●
Buffer pool
●
Multi-threaded
●
Currently the fastest transactional disk-based MySQL Storage
Engine!
33. MySQL Performance (traditionally, in the past)
●
Choose the right Engine for each of your table/database
●
Read-Only / Text search => MyISAM
●
Read+Write / Transactions => InnoDB
●
Short/Small Transactions + DB fits in RAM => NDB
●
Tune / Optimize your queries
●
Once scalability limit is reached => go for Distributed:
●
●
Master / Slave(s) => role-based workload
●
●
Sharding
Any other similar :-)
Scalability = Main Performance Problem!...
●
But with Big Users on that time anyway: Google, Facebook, Amazon..
34. Things are changing constantly, stay tuned ;-)
●
MySQL/InnoDB Scalability:
●
●
2008 : up to 4CPU cores
●
2009 : up to 16CPU cores (+Sun)
●
2010 : up to 32CPU cores (+Oracle)
●
2012 : up to 48CPU cores..
●
2014 : …?? ;-)
●
●
2007 : up to 2CPU...
NOTE: on the same HW performance is better from version to version!
InnoDB today:
●
At least x4-8 times better performance than 2-3 years ago ;-)
●
Capable of over 100K 300K 500K QPS(!) + FTS & Memcached
38. How easy is to see the same
in Production now?.. ;-)
39. Starting points
●
What are your network limits?..
●
●
●
Latency? Max throughput? What CPU% is spent just for network?
Do you use prepared statements? (reducing traffic)
Can you use persistent connections?
●
●
Greatly improved in 5.6, yet more in 5.7 (55K Connect/s in 5.7 currently)
●
Higher QPS if more queries executed before disconnect!
●
●
Connect / Disconnect has its limits..
Thread cache size matters!
Do you use transactions on read-only requests?..
●
●
QPS is improved since 5.6 and yet more in 5.7
But you cannot get a rid from a traffic overhead due BEGIN / COMMIT
exchanges
40. Analyzing MySQL Workload
●
Understand the load first :
●
Hot queries <== could be improved?..
●
Hot tables / files <== storage ok? DB design?..
●
Bad query execution plans.. <== improve, force index, etc.
●
Row Lock contentions due Application Design <== will not scale..
●
Deadlocks due Application Logic.. <== will not scale..
●
NOTE: be sure you're not hitting some HW / OS limits (and MySQL is in
fact out of scope ;-))
41. Performance Schema since MySQL 5.6: Gold Mine!
●
Query digest (enabled by default) :
●
●
SELECT all queries having > N rows read
●
●
SELECT all queries with execution time > N ms
SELECT queries having table scans, not using indexes, etc..
FILE_IO (enabled by default) :
●
●
●
Time spent on every IO operation for every database file
Amount of each kind of IO operations for every file
Table Locks (enabled by default) :
●
●
See which tables are the most accessed
==> Just with these 3 metrics you already have an idea if things
are still going well or not.. - and MEM is excellent here! ;-)
42. Classic MySQL Monitoring
●
SHOW Commands:
●
●
mysql> show global status ;
●
mysql> show processlist ;
●
mysql> show engine innodb status ;
●
mysql> show engine innodb mutex ;
●
●
mysql> status ;
INFORMATION_SCHEMA.* , InnoDB METRICS table, etc..
Important :
●
only PFS instrumentation / query is truly lock free..
●
every query during its execution uses 1 CPU core full time!
●
excessive requesting may significantly lower an overall performance!
43. So far, what do you have to look on?..
●
MySQL Server general:
●
●
Query/sec, Select/sec, Commit/sec, Connect/sec, Connections, Abort...
InnoDB:
●
BP usage/ dirty%/ page hit%
●
Checkpoint Age, REDO logs rates (MB/sec, Writes/sec, Sync time/sec)
●
Adaptive Flushing rates, Sync Flushing rates, Sync Flushing waits
●
LRU Flushing stats, User Threads LRU Flushing, ..
●
History List Length (purge)
●
Mutex Waits (InnoDB, PFS)
●
File IO Waits (PFS)
●
etc...
44. Suspecting a problem?.. - Benchmark!
●
Have a clear goal!
●
●
Otherwise: I've obtained all these results, and now... so what?..
Want to simulate your production workload?..
●
●
●
Then just simulate it! (many SW available, not always OSS/free)
Hard to simulate? - adapt some generic tests
Want to know capacity limits of a given platform?
●
●
Still try to focus on the test which are most significant for you!
Want just to validate config settings impacts?
●
●
●
Focus on tests which are potentially depending on these settings
Or any, if the goal to prove there are not depending ;-)
Well, just keep thinking about what you're doing ;-)
45. Test Workload
●
Before to do something complex...
●
Be sure first you're comfortable with
“basic” operations!
●
●
Many tables?
●
Short queries?
●
●
Single table?
Long queries?
Remember: any complex load just
represents a mix of simple operations..
●
So, start from as simple as possible..
●
And then increase complexity progressively..
46. Popular “Generic” Test Workloads @MySQL
●
Sysbench
●
●
OLTP, RO/RW, 1-table, since v0.5 N-table, lots load options, deadlocks
DBT2 / TPCC-like
●
●
●
OLTP, RW, very complex, growing db, no options, deadlocks
In fact using mostly only 2 tables! (thanks Performance Schema ;-))
dbSTRESS
●
●
linkbench (Facebook)
●
●
OLTP, RO/RW, several tables, one most hot, configurable, no deadlocks
OLTP, RW, very intensive
DBT3
●
DWH, complex heavy query, loved by Optimizer Team ;-)
47. MySQL Performance: No Silver Bullet !!!
Internal Limits..
Internal Limits..
There is
There is
No Silver
No Silver
Bullet!!!
Bullet!!!
MySQL
MySQL
Configuration Settings..
Configuration Settings..
InnoDB
InnoDB
Server, OS, FS
Server, OS, FS
Query Optimization..
Query Optimization..
Storage
Storage
BBU, SSD
BBU, SSD
Application Contentions..
Application Contentions..
48. MySQL Config settings
●
Ask yourself right questions and start with some basic params:
●
●
Double write buffer? Checksums?
●
innodb_flush_log_at_trx_commit= 1 / 2 ??
●
Flush Method = O_DIRECT + ASYNC
●
Binlog Sync? - binlog group commit is since MySQL 5.6 only!
●
File per table?
●
IO capacity = 2000
●
●
Buffer Pool size / Buffer Pool Instances
Etc..
Then adapt then to discovered your HW/OS and MySQL
Internal limits!..
51. Example: Buffer Pool Instances
●
RW intensive workload:
●
BP instances = 1/ 2/ 4/ 8
52. Workload: Read-Only oriented
●
Bigger Buffer Pool (BP) is better
●
BP < dataset = IO-bound
●
TRX list (kernel_mutex, since 5.6: trx_sys mutex)
●
Read view
●
Auto-commit or transactions?..
●
●
Prepared statements
●
●
Grouping many queries within a single transaction may also largely
reduce MDL locking, but still keep them short ! (check with PFS)
Observed 10% performance improvement in 5.6 (while Parser time is
not more than 3% according to profiler)..
Read-Only transactions!
53. InnoDB: Read-Only Transactions in 5.6
●
Sysbench OLTP_RO Point-Selects:
●
Concurrent user sessions: 1, 2, 4 .. 1024
●
Using of transactions in sysbench = 0 / 1
54. InnoDB: Read-Only Transactions in 5.6 (Apr.2013)
●
Sysbench OLTP_RO Point-Selects:
●
Concurrent user sessions: 1, 2, 4 .. 1024
●
Using of transactions in sysbench = 0
55. InnoDB : false sharing of cache-line = true killer
●
RO or RW Workloads
●
Same symptoms in 5.5 & 5.6 : no QPS improvement between 16 and 32
user sessions:
56. InnoDB : false sharing of cache-line fixed!
●
RO or RW Workloads
●
“G5” patch! :-)
●
Over x2(!) times better on Sysbench OLTP_RO,
●
x6(!) times better on SIMPLE-Ranges!
●
NOTE: the fix is not applicable on 5.5..
57. MySQL Internals: “killer” LOCK_open mutex
●
MySQL 5.5 and before:
●
Keep “table_open_cache” setting big enough!
●
Monitor global status for '%opened%'
●
●
Once this contention become the most hot – well, time to upgrade to
5.6 ;-))
Since MySQL 5.6:
●
Fixed: several table open cache instances
●
But it doesn't mean you can use a small “table_open_cache” either ;-)
●
Monitor PFS Waits!
●
Monitor “table_open_cache%” status variables!
●
Keep “table_open_cache_instances” at least bigger than 1
58. MySQL 5.6 Internals : low table_open_cache
●
MySQL 5.6 :
●
Not big enough “table_open_cache” setting
59. MySQL 5.6 Internals : low table_open_cache (2)
●
MySQL 5.6 :
●
Not big enough “table_open_cache” setting
●
PFS Waits monitoring: LOCK_table_cache become the most hot:
●
Table_open_cache% status:
60. MySQL 5.6 Internals : table_open_cache_instances
●
MySQL 5.6 :
●
When LOCK_table_cache wait is on top, the gain is usually well visible:
61. Workload: Read-Write
●
RW activity
●
●
Updates only? Insert? Delete? R/W %ratio?
Bigger Buffer Pool (BP) is still better
●
BP < dataset = IO-bound Reads(!) or R+W
●
BP > dataset = CPU-bound or IO-bound Writes(!)
●
REDO size matters a lot! (up to 2TB in 5.6)
●
Adaptive Flushing matters a lot!
●
LRU flushing matters a lot as well!
●
Tip: Neighbor Pages flushing = off / on
62. But let me tell you now the
whole story first! ;-)
63. Jan.2009 : Long RW Intensive Test
●
RW Workload:
●
128 concurrent users, 500M REDO, dirty pages= 15%
●
But let's get a look on the real state of BP:
64. InnoDB Internals: Dirty pages
●
How does it work?..
●
●
SQL> show innodb
status G
But why my dirty
pages% setting
is ignored?...
●
●
Buffer pool
> free
> data
innodb_buffer_pool_size = M
> dirty
innodb_max_dirty_pages_pct = 15%
Mystery?...
All votes:
it's impossible ;-)
REDO
DATA / INDEX
innodb_log_file_size = 500M
65. InnoDB Internals: Dirty pages and REDO?..
●
What if I'll reduce REDO size now?..
●
REDO: 500M => 128M
Buffer pool
> free
> data
innodb_buffer_pool_size = M
> dirty
innodb_max_dirty_pages_pct = 15%
REDO
DATA / INDEX
innodb_log_file_size = 128M
66. InnoDB Internals: Dirty pages and REDO?..
●
What if I'll reduce REDO size now?..
●
●
REDO: 500M => 128M
Forcing lower
Dirty Pages Amount!
Buffer pool
> free
> data
innodb_buffer_pool_size = M
> dirty
innodb_max_dirty_pages_pct = 15%
REDO
DATA / INDEX
innodb_log_file_size = 128M
67. Any Changes on RW Test now?..
●
REDO = 500M
●
REDO = 128M
68. Fine, but..
●
Remained questions:
●
Why finally Dirty Pages% setting is completely ignored?...
●
While, after all, any dangers to have many dirty pages?...
●
And what is the impact of REDO logs size?..
71. InnoDB Internals: Impact of REDO size
●
RW Intensive Load
●
●
Result: 6000 TPS => 8000 TPS! 30% better!!!
●
●
REDO size = 128M => 1024M
For such an improvement we may ignore Dirty Pages% ;-))
But : WHY these TPS drops?...
72. InnoDB Internals: Analyzing the code..
●
Master thread logic:
Master Thread
loop: //Main loop
...
if( dirty pct > limit)
flush_batch( 100% IO);
...
do {
pages= trx_purge();
if( 1sec passed ) flush_log();
} while (pages);
...
goto loop;
Buffer pool
> free
> data
> dirty
REDO
DATA / INDEX
73. InnoDB Internals: Analyzing the code..
●
Master thread may never leave purge loop!!!
Master Thread
loop: //Main loop
...
if( dirty pct > limit)
flush_batch( 100% IO);
...
do {
pages= trx_purge();
if( 1sec passed ) flush_log();
} while (pages);
...
goto loop;
Buffer pool
> free
> data
> dirty
REDO
DATA / INDEX
74. InnoDB Internals: Analyzing the code..
●
But if Master thread is never leave purge loop...
●
Who is then flushing Dirty Pages?...
Buffer pool
> free
> data
> dirty
REDO
DATA / INDEX
75. InnoDB Internals: Analyzing the code..
●
But if Master thread is never leave purge loop...
●
●
Who is then flushing Dirty Pages?...
Redo log constraints:
●
●
●
●
Cyclic, need free space
Checkpoint Age: diff between the
current LSN in redo and the oldest
dirty page LSN
Checkpoint Age cannot out-pass the max
checkpoint age (redo log size)
Buffer pool
> free
> data
> dirty
If Checkpoint Age >= 7/8 of Max Age
=> Flush ALL dirty pages regardless IO capacity!!!
REDO
(“Furious Flushing”)
DATA / INDEX
76. InnoDB Internals: Introducing Purge Thread
●
Purge Thread is the MUST !!!
Master Thread
loop: //Main loop
...
sleep( 1 );
...
if( dirty pct > limit)
flush_batch( 100% IO);
...
flush_log();
...
goto loop;
Purge Thread
loop:
sleep( ... );
do { pages=
trx_purge();
} while (pages);
goto loop;
Buffer pool
> free
> data
> dirty
REDO
DATA / INDEX
79. InnoDB Purge since MySQL 5.5
●
Purging has a cost! (similar to Garbage Collecting)
●
●
●
Since MySQL 5.5: single purge thread (off by default)
Since MySQL 5.6: several purge thread(s) (up to 32)
However, Purge may lag and do not follow workload..
●
●
Ex.: On aggressive RW got 400GB of undo records within few hours(!)
●
●
This is very bad when happens...
Then it took days to reach zero in History Length..
The main problem is the past – how to dose purging?..
●
●
Since 5.6: with many threads, Purge become auto-stable itself
Still missing a dynamic config option to say how many purge threads to
run in parallel right now (but it'll be fixed soon ;-))
80. InnoDB : Purge improvement in 5.6
●
Several Purge Threads :
●
NOTE: activation is auto-magical (I'm serious ;-))
81. InnoDB : Purge improvement in 5.6
●
Fixed max purge lag code!
●
●
●
innodb_max_purge_lag
innodb_max_purge_lag_delay <= configurable!
Setting innodb_max_purge_lag=1M:
82. InnoDB Internals: “Furious Flushing”
●
Direct dependence on REDO log size
●
NOTE:
●
●
●
●
No direct dependence
on amount of dirty
pages and REDO size!
Depends on workload!
Buffer pool
> free
> data
innodb_buffer_pool_size = M
> dirty
innodb_max_dirty_pages_pct = N
However, bigger REDO
allows more dirty pages..
And recovery is way
faster today!
REDO
DATA / INDEX
innodb_log_file_size = L
83. InnoDB: REDO log constraints
●
REDO log constraints: (Always monitor Checkpoint Age!!!)
●
●
●
Cyclic, need free space
Buffer pool
Checkpoint age: diff between the current LSN in REDO
> free
and the oldest dirty page LSN
Checkpoint age cannot out-pass the max checkpoint
age (redo log size)
●
●
●
If Checkpoint age >= 7/8 of Max age => Flush ALL dirty!
> data
> dirty
=> AKA “furious flushing”...
Adaptive Flushing:
●
Keep REDO under Max age
●
Respecting IO capacity limit
REDO
DATA / INDEX
84. InnoDB: Adaptive Flushing
●
MySQL 5.5:
●
●
●
Estimation based
Sometimes works ;-)
MySQL 5.6 :
●
●
●
Based on REDO write rate + I/O capacity Max
Involving batch flushing with N pages to flush (progressive, depending
on REDO %free) + page age limit (according REDO rate)
Tuning:
●
innodb_io_capacity / innofb_io_capacity_max
●
innodb_adaptive_flushing_lwm / innodb_max_dirty_pages_pct_lwm
●
ALL are dynamic!
●
Monitor Checkpoint Age..
87. InnoDB Adaptive Flushing: Fine Tuning
●
Monitor your Flushing rate / capabilities..
●
Adapt IO capacity and REDO size :
88. InnoDB and I/O Performance
●
Keep in mind the nature of I/O operation!
●
●
Sequential Read (SR)
●
Random Write (RW)
●
●
Sequential Write (SW)
Random Read (RR)
InnoDB
Buffer pool
> free
> data
●
Data files <= SW,SR,RW,RR
●
Redo log <= SW
●
Bin log <= SW
●
Double write <= SW
> dirty
BINLOG
DATA / INDEX
double write buffer
REDO
89. InnoDB and I/O Performance
●
Avoid a hot-mix of I/O operations!
●
Random Read (RR) <= most painful & costly!!!
●
Place REDO on different LUNs/disks
●
●
Place BINLOG on a separated
storage array!
I/O Settings
●
I/O write threads
●
> data
I/O capacity
●
Buffer pool
> free
I/O read threads
> dirty
BINLOG
DATA / INDEX
double write buffer
REDO
90. InnoDB: Doublewrite Buffer
●
Protecting from partially written pages
●
Data first written into Doublewrite buffer (sys.tablespace)
●
Then flushed to the datafiles
●
●
On recovery: if partially written page discovered => use its image from
doublewrite buffer
What is the cost?..
●
Doublewrite I/O is sequential, so should be fast
●
Writes will do less sync calls:
–
Instead of sync on every page write
–
Sync once on doublewrite buffer write
–
Then once on the datafile(s) for the same chunk of pages
91. InnoDB: Doublewrite buffer real impact?
●
Usually:
●
●
●
performance remains the same (or better)
+ recovery guarantee!
In some cases:
●
Up to 30% performance degradation...
●
Why?...
92. InnoDB: Doublebuffer and I/O dependency
●
Random Reads are killing!
●
RR = ~5ms wait per operation on HD
●
Example:
–
Application is doing 30.000 IO op/s
–
All operations are SW/RW
–
Now 5% of Writes become RR
–
What about performance?..
Buffer pool
> free
> data
> dirty
BINLOG
DATA / INDEX
double write buffer
REDO
93. InnoDB: Doublebuffer and I/O dependency
●
Random Reads are killing!
●
RR = 5ms wait per operation
●
Example:
–
Application is doing 30.000 IO op/s
–
All operations are SW/RW
–
Now 5% of Writes become RR
–
Performance => 10.000 IO ops/s...
–
x3 times degradation!
–
100 SW= 100 x 0.1ms = 10ms
–
95 SW + 5 RR = 9.5ms + 25ms
Buffer pool
> free
> data
> dirty
BINLOG
DATA / INDEX
double write buffer
REDO
94. InnoDB: Doublebuffer and I/O dependency
●
Workaround: move doublewrite buffer on REDO disks
●
Have to set innodb_file_per_table initially for DB
●
Move system tablespace on REDO disks:
$ mv /DATA/ibdata1 /LOG
$ ln -s /LOG/ibdata1 /DATA
Buffer pool
> free
> data
●
Or just use SSD !!! ;-)
> dirty
BINLOG
DATA / INDEX
double write buffer
REDO
95. User Concurrency scenarios
●
Single user?..
●
●
●
With a bigger code path today 5.6 simply cannot be faster than 5.5
But then, why you're not considering Query Cache? ;-)
More users?..
●
●
●
Up to 8-16 concurrent users all internal contention are not yet hot
So, 5.6 will not be better yet..
More than 16 users?..
●
●
●
Then you'll feel a real difference, but if you have at least 16cores ;-)
Or if you have really a lot of concurrent users
But don't forget other 5.6 improvements either!
●
On-line DDL, Binlog group commit, Memcached, etc..
96. High Concurrency Tuning
●
●
If bottleneck is due a concurrent access on the same data (due
application design) – ask dev team to re-design ;-)
If bottleneck is due MySQL/InnoDB internal contentions, then:
●
If you cannot avoid it, then at least don't let them grow ;-)
●
Try to increase InnoDB spin wait delay (dynamic)
●
Try innodb_thread_concurrency=N (dynamic)
●
CPU taskset / prcset (Linux / Solaris, both dynamic)
●
Thread Pool
●
NOTE: things with contentions may radically change since 5.7, so stay
tuned ;-)
97. InnoDB Spin Wait Delay
●
RO/RW Workloads:
●
With more CPU cores internal contentions become more hot..
●
Bind mysqld to less cores helps, but the goal is to use more cores ;-)
●
Using innodb_thread_concurrency is not helping here anymore..
●
So, innodb_spin_wait_delay is entering in the game:
98. Tune InnoDB Spin Wait Delay
●
Notes :
●
is the max random delay on “sleep” within a spin loop in wait for lock..
●
Ideally should be auto.. while the same tuning works for 5.5 as well ;-)
●
General rule: default is 6, may need an increase with more cores
●
Test: 32-HT/ 32/ 24/ 16cores, spin delay = 6 / 96 :
99. Thread Pool @MySQL
●
None of these solutions will help to increase performance!
●
●
it'll just help to keep the peak level constant (and you yet need to
discover on which level of concurrency you're reaching your peak ;-))
ThreadPool in MySQL 5.5 and 5.6 is aware if I/O are involved!
●
So, better than innodb thread concurrency setting or taskset
●
May still require spin wait delay tuning!
●
The must for high concurrency loads!
●
●
May still start to show a difference since 32-128 concurrent users! (all
depends on workload)..
Keep in mind that OS scheduler is not aware how to manage user
threads most optimally, but ThreadPool does ;-)
103. InnoDB High Concurrency: AHI
●
Adaptive Hash Index (AHI)
●
Helps a lot on Read-Only workloads
●
In fact it helps always until itself become not actively modified
●
AHI contention is seen as its btr_search_latch RW-lock contetnion
●
So, on Read+Write become a huge bottleneck..
●
In many cases on RW the result is better with AHI=off..
●
NOTE: there is still a big mystery around AHI when it's having
btr_search_latch contention even when there is no changes at all (pure
RO in memory).. - expected to be fixed in 5.7 ;-)
104. Testing Apples-to-Apples...
●
Comparing MySQL 5.6 vs 5.5 :
●
Don't have G5: dead..
●
Don't have open table cache instances: bad..
●
Don't have improved Adaptive Flushing; bad..
●
Don't have fixed Purge & Lag: danger!..
●
Don't have binlog group commit and use binlog: dead..
●
Etc. etc. etc.
●
●
NOTE: some “improvement” are also fixes which are making stuff
working properly, but coming with additional overhead (like Purge)..
NOTE: when comparing 5.6 and 5.5 keep in mind that Performance
Schema is enabled by default in 5.6, and not in 5.5, so think to disable it
in both (as 5.5 also has a way less PFS instrumentation)..
115. MySQL 5.6: Pending issues
●
Index lock..
●
Lock_sys contention..
●
Trx_sys contention..
●
MDL scalability..
●
Flushing limits..
●
LRU flushing..
●
Design bug on block locking.. (was here from the beginning)
●
Not able yet to use 100% I/O capacity on a powerful storage..
●
“Mysterious” contentions on dbSTRESS..
●
etc..
116. MySQL 5.7: Work in progress.. ;-)
●
Index lock.. <== fixed !
●
Lock_sys contention.. <== lowered !
●
Trx_sys contention.. <== improved a lot !!!
●
MDL scalability.. <== in progress..
●
Flushing limits.. <== in progress..
●
LRU flushing.. <== in progress..
●
Design bug on block locking.. (was here from the beginning)
●
Not able yet to use 100% I/O capacity on a powerful storage..
●
“Mysterious” contentions on dbSTRESS..
●
Etc.. <== well, ALL in progress / investigation ;-)
117. MySQL 5.7: DMR2 (Sep.2013)
●
OLTP_RO Point-Selects 8-tables: 500K QPS !!!
●
UNIX socket, sysbench 0.4.8 (older, using less CPU)
118. MySQL 5.7: DMR2 (Sep.2013)
●
OLTP_RO Point-Selects 8-tables: 440K QPS
●
IP port, sysbench 0.4.13 (“common”, using more CPU)
119. MySQL 5.7: DMR2 (Sep.2013)
●
OLTP_RO Point-Selects-TRX 8-tables: 200K QPS
●
IP port, sysbench 0.4.13 (“common”, using more CPU)
120. MySQL 5.7: DMR2 (Sep.2013)
●
OLTP_RO Point-Selects 8-tables: Scalability..
●
UNIX socket, sysbench 0.4.8 (older, using less CPU)
121. MySQL 5.7: DMR2 (Sep.2013)
●
OLTP_RO Point-Selects 8-tables: Scalability..
●
IP port, sysbench 0.4.13 (using more CPU)
122. MySQL 5.7: DMR2 (Sep.2013)
●
OLTP_RO Point-Selects-TRX 8-tables: Scalability..
●
IP port, sysbench 0.4.13 (using more CPU)
123. MySQL 5.7: DMR2 (Sep.2013)
●
OLTP_RO 8-tables: 280K QPS
●
IP port, sysbench 0.4.13 (“common”, using more CPU)
124. MySQL 5.7: DMR2 (Sep.2013)
●
OLTP_RO 1-table: lower than 5.6...
●
Due higher MDL contentions, work in progress..
125. MySQL 5.7: DMR2 (Sep.2013)
●
OLTP_RW 8-tables: 265K QPS
●
IP port, sysbench 0.4.13 (“common”, using more CPU)
126. MySQL 5.7: DMR2 (Sep.2013)
●
OLTP_RW 1-table: lower than 5.6
●
MDL contention, work in progress..
127. THANK YOU !!!
●
All details about presented materials you may find on:
●
http://dimitrik.free.fr - dim_STAT, Benchmark Reports
●
http://dimitrik.free.fr/blog - Articles about MySQL Performance