The document is an agenda for a talk titled "The Accidental DBA" about MySQL database administration. The talk covers MySQL basics like installation, configuration, backups and replication. It is meant to introduce attendees to common MySQL administration tasks and help them get more comfortable with database administration even if they came to it by accident. The speaker is Jenni Snyder, an engineering manager at Yelp who became their first DBA in 2011.
How to Avoid Pitfalls in Schema Upgrade with Percona XtraDB ClusterSveta Smirnova
Percona XtraDB Cluster (PXC) is a 100% synchronized cluster in regards to DML operations. It is ensured by the optimistic locking model and ability to rollback transaction which cannot be applied on all nodes. However, DDL operations are not transactional in MySQL. This adds complexity when you need to change the schema of the database.
Changes made by DDL may affect the results of the queries. Therefore all modifications must replicate on all nodes prior to the next data access. For operations that run momentarily, it can be easily achieved, but schema changes may take hours to apply. Therefore in addition to the safest synchronous blocking schema upgrade method: TOI, - PXC supports more relaxed, though not safe, method RSU.
RSU: Rolling Schema Upgrade is advertised to be non-blocking. But you still need to take care of updates, running while you are performing such an upgrade. Surprisingly, even updates on not related tables and schema can cause RSU operation to fail.
In this talk, I will uncover nuances of PXC schema upgrades and point to details you need to take special care about.
Further Information
Schema change is a frequent task, and many do not expect any surprises with it. However, the necessity to replay the changes to all synchronized nodes adds complexity. I made a webinar on a similar topic which was recorded and available for replay. Now I have found that I share a link to the webinar to my Support customers approximately once per week. Not having a good understanding of how schema change works in the cluster leads to lockups and operation failures. This talk will provide a checklist that will help to choose the best schema change method.
Presented at Percona Live Online: https://perconaliveonline2020.sched.com/event/ePm2/how-to-avoid-pitfalls-in-schema-upgrade-with-percona-xtradb-cluster
Topics covered in this presentation:
1. The difference between traditional (e.g. MySQL) replication and Galera Cluster
2. General Galera Cluster principles
The document discusses managing data and archiving in MySQL 5.6 using new features like Portable Tablespace and EXCHANGE PARTITION. It provides an overview of using these features to move a tablespace or partition to a different table or server for purposes like data refresh or historical analysis. Specifically, it demonstrates exchanging a partition from one table to an external table, then moving that table to another server and loading new data before exchanging it back.
MySQL has multiple timeouts variables to control its operations. This presentation focus on the purpose of each timeout variables and how it can be used.
Replication Troubleshooting in Classic VS GTIDMydbops
This presentation talk will assist you in troubleshooting MySQL replication for the most common issues we might face with a simple comparison of how can we get them solved in the different replication methods (Classic VS GTID).
How to Avoid Pitfalls in Schema Upgrade with Percona XtraDB ClusterSveta Smirnova
Percona XtraDB Cluster (PXC) is a 100% synchronized cluster in regards to DML operations. It is ensured by the optimistic locking model and ability to rollback transaction which cannot be applied on all nodes. However, DDL operations are not transactional in MySQL. This adds complexity when you need to change the schema of the database.
Changes made by DDL may affect the results of the queries. Therefore all modifications must replicate on all nodes prior to the next data access. For operations that run momentarily, it can be easily achieved, but schema changes may take hours to apply. Therefore in addition to the safest synchronous blocking schema upgrade method: TOI, - PXC supports more relaxed, though not safe, method RSU.
RSU: Rolling Schema Upgrade is advertised to be non-blocking. But you still need to take care of updates, running while you are performing such an upgrade. Surprisingly, even updates on not related tables and schema can cause RSU operation to fail.
In this talk, I will uncover nuances of PXC schema upgrades and point to details you need to take special care about.
Further Information
Schema change is a frequent task, and many do not expect any surprises with it. However, the necessity to replay the changes to all synchronized nodes adds complexity. I made a webinar on a similar topic which was recorded and available for replay. Now I have found that I share a link to the webinar to my Support customers approximately once per week. Not having a good understanding of how schema change works in the cluster leads to lockups and operation failures. This talk will provide a checklist that will help to choose the best schema change method.
Presented at Percona Live Online: https://perconaliveonline2020.sched.com/event/ePm2/how-to-avoid-pitfalls-in-schema-upgrade-with-percona-xtradb-cluster
Topics covered in this presentation:
1. The difference between traditional (e.g. MySQL) replication and Galera Cluster
2. General Galera Cluster principles
The document discusses managing data and archiving in MySQL 5.6 using new features like Portable Tablespace and EXCHANGE PARTITION. It provides an overview of using these features to move a tablespace or partition to a different table or server for purposes like data refresh or historical analysis. Specifically, it demonstrates exchanging a partition from one table to an external table, then moving that table to another server and loading new data before exchanging it back.
MySQL has multiple timeouts variables to control its operations. This presentation focus on the purpose of each timeout variables and how it can be used.
Replication Troubleshooting in Classic VS GTIDMydbops
This presentation talk will assist you in troubleshooting MySQL replication for the most common issues we might face with a simple comparison of how can we get them solved in the different replication methods (Classic VS GTID).
The document summarizes a presentation given by Marco Tusa and Aleksandr Kuzminsky on using the Percona Data Recovery Tool for InnoDB to recover data from corrupted or missing InnoDB files. It introduces the speakers and outlines how the tool can be used to extract data from ibdata files, attach .ibd files after corruption, recover deleted records, recover dropped tables, and recover from wrong updates by parsing binary logs. Examples are provided of running the tool's page_parser, constraints_parser and sys_parser components to extract schema and data.
MySQL indexing is one of the areas where you can achieve the most performance gains. In this presentation we have a short look how we can improve MySQL performance with indexing...
Hotsos 2011: Mining the AWR repository for Capacity Planning, Visualization, ...Kristofferson A
The document discusses mining the Automatic Workload Repository (AWR) in Oracle databases for capacity planning, visualization, and other real-world uses. It introduces Karl Arao as a speaker and discusses topics he will cover including AWR, diagnosing performance issues using AWR data, visualization of AWR data, capacity planning, and tools for working with AWR data like scripts and linear regression. References and resources on working with AWR are also provided.
OpenWorld 2014 - Schema Management: versioning and automation with Puppet and...Frederic Descamps
This document discusses automating database schema changes and migrations. It introduces the MySQL Utilities and Percona Toolkit tools that can be used to compare and synchronize database schemas. It then describes an online-migration.py script that uses these tools to automate schema migrations in a way that can be integrated with Puppet for infrastructure automation. The script allows rolling back migrations, checking the schema version and differences, and performing migrations without blocking reads or writes to the database.
We will show the advantages of having a geo-distributed database cluster and how to create one using Galera Cluster for MySQL. We will also discuss the configuration and status variables that are involved and how to deal with typical situations on the WAN such as slow, untrusted or unreliable links, latency and packet loss. We will demonstrate a multi-region cluster on Amazon EC2 and perform some throughput and latency measurements in real-time (video http://galeracluster.com/videos/using-galera-replication-to-create-geo-distributed-clusters-on-the-wan-webinar-video-3/)
Scaling with sync_replication using Galera and EC2Marco Tusa
Challenging architecture design, and proof of concept on a real case of study using Syncrhomous solution.
Customer asks me to investigate and design MySQL architecture to support his application serving shops around the globe.
Scale out and scale in base to sales seasons.
How to migrate from MySQL to MariaDB without tearsSveta Smirnova
Presented at MariaDB Server Fest 2020: https://mariadb.org/fest2020/migrate-mysql/
MariaDB is a drop-in replacement for MySQL. Initial migration is simple: start MariaDB over the old MySQL datadir.
Later your application may notice that some features work differently than with MySQL. These are MariaDB improvements, so this is good and, likely the reason you migrated.
In this session, I will focus on the differences affecting application performance and behavior. In particular, features sharing the same name, but working differently.
- Galera is a MySQL clustering solution that provides true multi-master replication with synchronous replication and no single point of failure.
- It allows high availability, data integrity, and elastic scaling of databases across multiple nodes.
- Companies like Percona and MariaDB have integrated Galera to provide highly available database clusters.
The document discusses Cassandra Storage Engine (Cassandra SE) in MariaDB, which provides interoperability between MariaDB and Cassandra. It covers an introduction to Cassandra, how Cassandra SE works, data mapping between Cassandra and SQL, command mapping, use cases for Cassandra SE, and benchmarks. It aims to enable accessing Cassandra data from MariaDB using SQL queries.
Slides for the webinar held on January 21st 2014
Repair & Recovery for your MySQL, MariaDB & MongoDB / TokuMX Clusters
Galera Cluster, NDB Cluster, VIP with HAProxy and Keepalived, MongoDB Sharded Cluster, etc. all have their own availability models. We are aware of these availability models and will demonstrate in this webinar how to take corrective action in case of failures via our cluster management tool, ClusterControl.
In this webinar, Severalnines CTO Johan Andersson will show you how to leverage ClusterControl to detect failures in your database cluster and automatically repair them to maximize the availability of your database services. And Codership CEO Seppo Jaakola will be joining Johan to provide a deep-dive into Galera recovery internals.
Agenda:
Redundancy models for Galera, NDB and MongoDB/TokuMX
Failover & Recovery (Automatic vs Manual)
Zooming into Galera recovery procedures
Split brains in multi-datacenter setups
MariaDB 10.0 introduces domain-based parallel replication which allows transactions in different domains to execute concurrently on replicas. This can result in out-of-order transaction commit. MariaDB 10.1 adds optimistic parallel replication which maintains commit order. The document discusses various parallel replication techniques in MySQL and MariaDB including schema-based replication in MySQL 5.6 and logical clock replication in MySQL 5.7. It provides performance benchmarks of these techniques from Booking.com's database environments.
Proper Care and Feeding of a MySQL Database for Busy Linux AdministratorsDave Stokes
Do you 'also' have MySQL DBA responsibilities as part of your job but have no DBA training? This presentation covers a lot of the DBA level info for those who have a Linux admin background but are not DBAs
At this meetup Patrick McFadin, Solutions Architect at DataStax, will be discussing the most recently added features in Apache Cassandra 2.0, including: Lightweight transactions, eager retries, improved compaction, triggers, and CQL cursors. He'll also be touching on time series data with Apache Cassandra.
MariaDB 10.5 binary install (바이너리 설치)
- 네오클로바 DB지원사업부
1. About MariaDB
1.1 MariaDB 개요
1.2 MariaDB as a R-DBMS
1.3 Open Source Database System
2. 설치
2.1 설치 기본 정보
2.2 설치 준비
2.3 MariaDB 설치
2.4 MariaDB 시작 / 접속 / 종료
2.5 추가 설정
From these slides you'll learn how Galera integrates with MySQL 5.6 and Global Transaction IDs to enable cross-datacenter and cloud replication over high latency networks.
ABOUT GALERA CLUSTER
Galera Cluster for MySQL is a true multi-master MySQL replication plugin, and has been proven in mission-critical infrastructures of companies like Ping Identity, AVG Technologies, KPN and HP Cloud DNS. In this webcast you¹ll learn about the following Galera Cluster capabilities, including the latest innovations in the new 3.0 release:
Galera Cluster features and benefits
Support for MySQL 5.6
Integration with MySQL Global Transaction Identifiers
Mixing Galera synchronous replication and asynchronous MySQL replication
Deploying in WAN and Cloud environments
Handling high-latency networks
Management of Galera
The Proper Care and Feeding of a MySQL Database for Busy Linux Admins -- SCaL...Dave Stokes
If you are a Linux administrator and ALSO have to take care of a MySQL databases, this presentation if for you, While it will not turn you instantly into a DBA it will help you understand how to properly care and feed your instances
MySQL Performance Best Practices document discusses various techniques for optimizing MySQL performance. It covers monitoring performance with tools like MySQL Enterprise Monitor, optimizing configurations such as increasing the InnoDB buffer pool size, balancing data consistency versus performance by adjusting the innodb_flush_log_at_trx_commit setting, and summarizes that monitoring is essential to identify performance issues and test configuration changes before deploying to production.
This document discusses MySQL multi-source replication, which enables a replication slave to receive transactions from multiple masters simultaneously. It provides configuration steps for setting up a MariaDB/Percona Server database as a multi-source replication slave, including ensuring unique server IDs and GTID domains for each master, defining replication connections, and enabling parallel replication threads to optimize transaction processing from multiple sources.
The Proper Care and Feeding of MySQL DatabasesDave Stokes
Many Linux System Administrators are 'also' accidental database administrators. This is a guide for them to keep their MySQL database instances happy, health, and glowing
Linuxfest Northwest Proper Care and Feeding Of a MySQL for Busy Linux AdminsDave Stokes
This document provides tips and best practices for properly configuring and maintaining a MySQL server for busy Linux administrators. It recommends focusing on hardware resources like memory, storage, and networking. It also emphasizes the importance of backups, replication, monitoring tools, and security configurations. Proper configuration of these areas can help keep MySQL databases running smoothly and prevent issues that waste administrators' time.
The document summarizes a presentation given by Marco Tusa and Aleksandr Kuzminsky on using the Percona Data Recovery Tool for InnoDB to recover data from corrupted or missing InnoDB files. It introduces the speakers and outlines how the tool can be used to extract data from ibdata files, attach .ibd files after corruption, recover deleted records, recover dropped tables, and recover from wrong updates by parsing binary logs. Examples are provided of running the tool's page_parser, constraints_parser and sys_parser components to extract schema and data.
MySQL indexing is one of the areas where you can achieve the most performance gains. In this presentation we have a short look how we can improve MySQL performance with indexing...
Hotsos 2011: Mining the AWR repository for Capacity Planning, Visualization, ...Kristofferson A
The document discusses mining the Automatic Workload Repository (AWR) in Oracle databases for capacity planning, visualization, and other real-world uses. It introduces Karl Arao as a speaker and discusses topics he will cover including AWR, diagnosing performance issues using AWR data, visualization of AWR data, capacity planning, and tools for working with AWR data like scripts and linear regression. References and resources on working with AWR are also provided.
OpenWorld 2014 - Schema Management: versioning and automation with Puppet and...Frederic Descamps
This document discusses automating database schema changes and migrations. It introduces the MySQL Utilities and Percona Toolkit tools that can be used to compare and synchronize database schemas. It then describes an online-migration.py script that uses these tools to automate schema migrations in a way that can be integrated with Puppet for infrastructure automation. The script allows rolling back migrations, checking the schema version and differences, and performing migrations without blocking reads or writes to the database.
We will show the advantages of having a geo-distributed database cluster and how to create one using Galera Cluster for MySQL. We will also discuss the configuration and status variables that are involved and how to deal with typical situations on the WAN such as slow, untrusted or unreliable links, latency and packet loss. We will demonstrate a multi-region cluster on Amazon EC2 and perform some throughput and latency measurements in real-time (video http://galeracluster.com/videos/using-galera-replication-to-create-geo-distributed-clusters-on-the-wan-webinar-video-3/)
Scaling with sync_replication using Galera and EC2Marco Tusa
Challenging architecture design, and proof of concept on a real case of study using Syncrhomous solution.
Customer asks me to investigate and design MySQL architecture to support his application serving shops around the globe.
Scale out and scale in base to sales seasons.
How to migrate from MySQL to MariaDB without tearsSveta Smirnova
Presented at MariaDB Server Fest 2020: https://mariadb.org/fest2020/migrate-mysql/
MariaDB is a drop-in replacement for MySQL. Initial migration is simple: start MariaDB over the old MySQL datadir.
Later your application may notice that some features work differently than with MySQL. These are MariaDB improvements, so this is good and, likely the reason you migrated.
In this session, I will focus on the differences affecting application performance and behavior. In particular, features sharing the same name, but working differently.
- Galera is a MySQL clustering solution that provides true multi-master replication with synchronous replication and no single point of failure.
- It allows high availability, data integrity, and elastic scaling of databases across multiple nodes.
- Companies like Percona and MariaDB have integrated Galera to provide highly available database clusters.
The document discusses Cassandra Storage Engine (Cassandra SE) in MariaDB, which provides interoperability between MariaDB and Cassandra. It covers an introduction to Cassandra, how Cassandra SE works, data mapping between Cassandra and SQL, command mapping, use cases for Cassandra SE, and benchmarks. It aims to enable accessing Cassandra data from MariaDB using SQL queries.
Slides for the webinar held on January 21st 2014
Repair & Recovery for your MySQL, MariaDB & MongoDB / TokuMX Clusters
Galera Cluster, NDB Cluster, VIP with HAProxy and Keepalived, MongoDB Sharded Cluster, etc. all have their own availability models. We are aware of these availability models and will demonstrate in this webinar how to take corrective action in case of failures via our cluster management tool, ClusterControl.
In this webinar, Severalnines CTO Johan Andersson will show you how to leverage ClusterControl to detect failures in your database cluster and automatically repair them to maximize the availability of your database services. And Codership CEO Seppo Jaakola will be joining Johan to provide a deep-dive into Galera recovery internals.
Agenda:
Redundancy models for Galera, NDB and MongoDB/TokuMX
Failover & Recovery (Automatic vs Manual)
Zooming into Galera recovery procedures
Split brains in multi-datacenter setups
MariaDB 10.0 introduces domain-based parallel replication which allows transactions in different domains to execute concurrently on replicas. This can result in out-of-order transaction commit. MariaDB 10.1 adds optimistic parallel replication which maintains commit order. The document discusses various parallel replication techniques in MySQL and MariaDB including schema-based replication in MySQL 5.6 and logical clock replication in MySQL 5.7. It provides performance benchmarks of these techniques from Booking.com's database environments.
Proper Care and Feeding of a MySQL Database for Busy Linux AdministratorsDave Stokes
Do you 'also' have MySQL DBA responsibilities as part of your job but have no DBA training? This presentation covers a lot of the DBA level info for those who have a Linux admin background but are not DBAs
At this meetup Patrick McFadin, Solutions Architect at DataStax, will be discussing the most recently added features in Apache Cassandra 2.0, including: Lightweight transactions, eager retries, improved compaction, triggers, and CQL cursors. He'll also be touching on time series data with Apache Cassandra.
MariaDB 10.5 binary install (바이너리 설치)
- 네오클로바 DB지원사업부
1. About MariaDB
1.1 MariaDB 개요
1.2 MariaDB as a R-DBMS
1.3 Open Source Database System
2. 설치
2.1 설치 기본 정보
2.2 설치 준비
2.3 MariaDB 설치
2.4 MariaDB 시작 / 접속 / 종료
2.5 추가 설정
From these slides you'll learn how Galera integrates with MySQL 5.6 and Global Transaction IDs to enable cross-datacenter and cloud replication over high latency networks.
ABOUT GALERA CLUSTER
Galera Cluster for MySQL is a true multi-master MySQL replication plugin, and has been proven in mission-critical infrastructures of companies like Ping Identity, AVG Technologies, KPN and HP Cloud DNS. In this webcast you¹ll learn about the following Galera Cluster capabilities, including the latest innovations in the new 3.0 release:
Galera Cluster features and benefits
Support for MySQL 5.6
Integration with MySQL Global Transaction Identifiers
Mixing Galera synchronous replication and asynchronous MySQL replication
Deploying in WAN and Cloud environments
Handling high-latency networks
Management of Galera
The Proper Care and Feeding of a MySQL Database for Busy Linux Admins -- SCaL...Dave Stokes
If you are a Linux administrator and ALSO have to take care of a MySQL databases, this presentation if for you, While it will not turn you instantly into a DBA it will help you understand how to properly care and feed your instances
MySQL Performance Best Practices document discusses various techniques for optimizing MySQL performance. It covers monitoring performance with tools like MySQL Enterprise Monitor, optimizing configurations such as increasing the InnoDB buffer pool size, balancing data consistency versus performance by adjusting the innodb_flush_log_at_trx_commit setting, and summarizes that monitoring is essential to identify performance issues and test configuration changes before deploying to production.
This document discusses MySQL multi-source replication, which enables a replication slave to receive transactions from multiple masters simultaneously. It provides configuration steps for setting up a MariaDB/Percona Server database as a multi-source replication slave, including ensuring unique server IDs and GTID domains for each master, defining replication connections, and enabling parallel replication threads to optimize transaction processing from multiple sources.
The Proper Care and Feeding of MySQL DatabasesDave Stokes
Many Linux System Administrators are 'also' accidental database administrators. This is a guide for them to keep their MySQL database instances happy, health, and glowing
Linuxfest Northwest Proper Care and Feeding Of a MySQL for Busy Linux AdminsDave Stokes
This document provides tips and best practices for properly configuring and maintaining a MySQL server for busy Linux administrators. It recommends focusing on hardware resources like memory, storage, and networking. It also emphasizes the importance of backups, replication, monitoring tools, and security configurations. Proper configuration of these areas can help keep MySQL databases running smoothly and prevent issues that waste administrators' time.
This document discusses strategies for maintaining very large MySQL tables that have grown too big. It recommends creating a new database server with different configuration settings like InnoDB file per table to reduce size, using tools like MySQLTuner and tuning-primer to analyze settings, archiving old historical data with ptArchiver to reduce table sizes, and considering partitioning or changing the MySQL version. Monitoring tools like InnoDB status, global status, cacti and innotop are recommended to analyze server performance.
The Peoper Care and Feeding of a MySQL Server for Busy Linux AdminDave Stokes
Webcast 16 September 2015 with a big thanks to SolarWinds. This is a collection of best practices for Linux Admin who 'also have' database responsibility but are not DBAs
The document discusses three common ways to improve performance of a MySQL database that is experiencing high load:
1. Upgrade hardware by adding more RAM, faster disks, or more powerful CPUs. This provides a temporary fix but can become exponentially more expensive and does not address underlying issues.
2. Change MySQL configuration settings like tmp_table_size or sort_buffer_size to optimize for specific bottlenecks shown in global status variables, but there are no "silver bullets" and misconfigurations must be addressed.
3. Improve indexing and tune queries by addressing issues like temporary tables on disk, full table scans, and lack of indexes causing full joins or sorting, which can have long term benefits over simply adding resources
Training Slides: 203 - Backup & RecoveryContinuent
Watch this 36min training to learn about planning for backups, what some of the methods and tools are, how to restore backups and more.
TOPICS COVERED
- How to develop a backup plan
- Methods and tools for taking a backup
- Verifying the backup contains the last binary position, and the importance of this
- Restore backups into the cluster
- Provision a replica from an existing datasource
The document discusses new features and improvements in MySQL 5.6, including significant performance gains over MySQL 5.5. Key highlights include improved InnoDB performance through features like online DDL and buffer pool pre-loading, up to 151-234% performance gains on benchmarks. Other enhancements cover full-text search in InnoDB, NoSQL support through memcached integration, replication improvements with GTIDs and crash-safe slaves, and strengthened security with audit logging and password policies.
MySQL has a set of utilities written in Python that can do some amazing things for your MySQL instances from setting up replication with automatic fail over to copying database
A Backup Today Saves Tomorrow is a presentation from Percona Live 2013 that provides insight into planning and the tools used today to capture MySQL backups.
The document provides an overview of how to summarize and interpret information from MySQL server status and variable outputs to understand server performance and optimize configuration. It explains that status variables show current server activity levels, while global and session variables display configuration settings. Comparing status outputs over time calculates rates like queries/second. Key metrics help identify bottlenecks like a small key buffer size if the key read cache miss rate is high.
MySQL 5.7 proposes several changes to improve performance and consistency including:
1. Making replication durable by default by setting sync_binlog and repository options.
2. Deprecating features like INNODB monitor tables and ALTER IGNORE TABLE in favor of newer standards.
3. Simplifying and restricting SQL modes to encourage stricter querying and remove ambiguous options. Explanations for errors and modes will also be improved.
Breaking the Monolith - Microservice Extraction at SoundCloudJan Kischkel
This document summarizes SoundCloud's journey from a monolithic Rails application to a microservices architecture. It describes SoundCloud's initial architecture using Rails and MySQL. As traffic grew, they scaled vertically by adding caching, workers, and database optimizations. They eventually extracted services like messages into independent Scala/Finagle services for improved scalability, deployability and maintenance. The summary discusses the risks of integrating new services and retiring old components, highlighting the need for load testing, rollback plans, and handling stale clients during the migration.
Some internal tools were relying on deprecated statements and behavior that changed in MySQL 5.6. The presenter had to update the tools to use the proper START SLAVE/STOP SLAVE statements and account for new information logged in binlogs due to configuration changes. Testing in pre-production helped uncover these issues so they could be addressed before upgrading production servers.
MariaDB 10.4 became General Available (GA = ready for production) this summer. So it is time to look at the new Features in MariaDB 10.4. After a short intro about history we look for the reason of broad usage of MariaDB nowadays. Most important improvements where in User Authentication, InnoDB improvements, and Optimizer enhancements. A completely New Feature is Application-Time Period Tables. Backup got a new Locking behaviour so LVM snapshots are possible and officially supported now. And last but not least MariaDB 10.4 comes with Galera 4.
- The document discusses managing a large OLTP database at PayPal, including capacity management, planned maintenance, performance management, and troubleshooting. It provides details on monitoring the database infrastructure, conducting maintenance such as patching and switchovers, and optimizing performance for Oracle RAC environments. The goal is to support business needs and provide uninterrupted service through proactive management of the database tier.
The MySQL sys schema was integrated fully into MySQL Server from version 5.7.7 and has been improved in MySQL 8.0. Whether you are a DBA trying to determine where the resources are being used on your database instance and by whom, or a developer trying to figure out why your MySQL statements are running too slowly, the MySQL sys schema can help. Join this session to learn how to better use the MySQL sys schema to answer your day-to-day questions—from the original developer of the MySQL sys schema.
Upcoming Changes in MySQL 5.7
Morgan Tocker, MySQL Community Manager
The presentation outlines several proposed changes and deprecations in MySQL 5.7, including making replication more durable by default, replacing some InnoDB monitor tables with performance schema instrumentation, simplifying SQL modes, and deprecating features like ALTER IGNORE TABLE, the \N NULL synonym, and the Federated storage engine. Feedback is sought on these proposed changes to help guide MySQL's future development.
Similar to Percona Live '18 Tutorial: The Accidental DBA (20)
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Full-RAG: A modern architecture for hyper-personalizationZilliz
Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.
Programming Foundation Models with DSPy - Meetup SlidesZilliz
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!SOFTTECHHUB
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024Neo4j
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slackshyamraj55
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
Best 20 SEO Techniques To Improve Website Visibility In SERPPixlogix Infotech
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
HCL Notes and Domino License Cost Reduction in the World of DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
Infrastructure Challenges in Scaling RAG with Custom AI modelsZilliz
Building Retrieval-Augmented Generation (RAG) systems with open-source and custom AI models is a complex task. This talk explores the challenges in productionizing RAG systems, including retrieval performance, response synthesis, and evaluation. We’ll discuss how to leverage open-source models like text embeddings, language models, and custom fine-tuned models to enhance RAG performance. Additionally, we’ll cover how BentoML can help orchestrate and scale these AI components efficiently, ensuring seamless deployment and management of RAG systems in the cloud.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
3. 9:30am: Welcome & Let's talk about the basics
11:00am: Break
11:15am: Running MySQL in the real world
12:15pm:Final wrap up
12:30pm: Lunch
Administrative
3
4. Part 1:
0. Brief Introduction
1. MySQL History, Philosophy, Community
2. Installation, Basic Configuration, Launching
3. Using it, Backing it up
4. Replication basics, Monitoring
5. Database Defense & fast feedback to developers
Agenda in More Detail
4
5. Part 2:
● Running in Production #realtalk
● Being Prepared
● Advanced Monitoring and Investigation
● Fantastic Errors and Where to Find Them
● When Things Are "Slow"
● Advanced Tuning
● Replication and Scaling Out
● Discussion
Agenda in More Detail
5
6. ● My name is Jenni
● Engineering Manager @ Yelp
● First DBA there in 2011
Who am I Why am I Here?
6
21. When it's a:
● log
● queue
● cache
● photo
● disk directory
● spreadsheet
● data warehouse/cube/shed/lake/garage/whatever
Use the right tool for the job.
When Not to Use a Database
21
23. ● MySQL - Wikipedia
● First released in 1995
● Pluggable storage engines
● Written in the age of spinning rust
● Transactional engine InnoDB released in 2001
● Oracle buys InnoDB in 2005
● Bought by Sun in 2008
● Oracle bought Sun in 2010
Brief History of MySQL
23
24. ● It is meant to work best the most of the time
● It is hard impossible to optimize for all cases
● That sometimes surprises you
"Performs extremely well in the average case" - Wikipedia
MySQL - Why Does it Do What It Does
24
25. ● The manual is awesome
● DBA StackExchange
● So many forums - "just <search engine> for it"
● Conferences
● Lots of great open source
The MySQL Community
25
26. ● MySQL Manual - Server System Variables - 500+
● 400+ status metrics
Don't freak out. You don't have to care about these yet :)
Highly Configurable
26
33. ● service mysql start
or
● /etc/init.d/mysql start
● Runs on port 3306
● Throw a party
● Or just connect to it
○ Use the mysql command
Let's Fire This Puppy Up
33
35. jsnyder@db:~$ mysql -u root
Welcome to the MySQL monitor. Commands end with ; or g.
Your MySQL connection id is 1124438
Server version: 5.6.32-78.1-log Percona Server (GPL), Yelp Release 1
Copyright (c) 2009-2017 Percona LLC and/or its affiliates
Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights
reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or 'h' for help. Type 'c' to clear the current input
statement.
mysql>
35
36. mysql> show databases ;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
4 rows in set (0.02 sec)
36
37. ● Find the start script
● Find that err.log
● Run mysqld_safe manually
When it Won't Start
37
43. mysql> show grants for 'root'@'%';
+-------------------------------------------------------------+
| Grants for root@% |
+-------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION |
| GRANT PROXY ON ''@'' TO 'root'@'%' WITH GRANT OPTION |
+-------------------------------------------------------------+
2 rows in set (0.00 sec)
mysql> show grants for current_user ;
+-------------------------------------------------------------+
| Grants for root@% |
+-------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION |
| GRANT PROXY ON ''@'' TO 'root'@'%' WITH GRANT OPTION |
+-------------------------------------------------------------+
2 rows in set (0.00 sec)
43
44. ● Make sure MySQL is running
● On that host & port
● Match user & host in error with grants
● Read err.log
● Use skip-grant-tables to start over
Debugging Connection Issues
44
45. ● Add skip-grant-tables to your my.cnf
● Restart MySQL
ALTER USER 'root'@'localhost' IDENTIFIED BY 'MyNewPass';
What if You Don't Know the Password?
45
46. ● Start with broad grants ('user'@'%')
● Refine from there
Debugging MySQL Connection Errors
46
47. mysql -u myuser -p mypass -h myhost -P myport
jsnyder@db:~$ mysql -u myuser -h myhost
ERROR 1045 (28000): Access denied for user myuser@'10.10.10.10'
(using password: NO)
jsnyder@db:~$ mysql -u myuser -p -h myhost
Enter password:
ERROR 1045 (28000): Access denied for user
'myuser'@'10.10.10.10' (using password: YES)
jsnyder@db:~$ mysql -u myuser -p -h myhost
Enter password:
ERROR 2005 (HY000): Unknown MySQL server host 'myhost' (0)
47
49. We use:
To ORM or not to ORM?
Developing Against MySQL
49
Language Python
MySQL Client Library mysqlclient-python (there is also
Connector/Python)
Object Relational Mapping (ORM) SQLAlchemy
Schema Manager Hand-rolled system
50. ● Have developers read up on normal form
○ It's OK to denormalize for performance (later)
● Use one table for one thing
● Try not to get too fancy right away
Reference: Database Normalization Basics
About That Schema
50
52. ● Don't swallow database client warnings/errors
● Track client error rates
● Make sure devs get error feedback
Build feedback to go directly to developers.
When Developing Against MySQL
52
53. ● You can start simple
● Reads and writes scale differently
● Use discovery, load balancer or proxy
○ Smartstack
○ ProxySQL
○ MySQL Router
Use technology you already have if possible.
Application Connections
53
58. ● Spoiler alert: it's SQL
● mysqldump
● Best run on an offline/not replicating DB
● Sometimes the only way to upgrade
Logical Backups
58
59. ● Logs of all changes
● Can be used to "roll" a DB forward in time
● More on these later...
● Copy & compress
Sample command to look at these:
mysqlbinlog --base64-output=DECODE-ROWS -v mysql-bin.000001
Backing up the Binary Logs
59
60. ● Backups are only as good as your ability to use them
● Test your restores
Testing Backups
60
61. ● mysqlbinlog turns binary logs into SQL
● Use the coordinates you saved
● It's documented, but I'm not aware of a great tool
Point in Time Recovery With Binary Logs
61
63. ● Try it before you need it
● We call these "war games"
○ And have failover exercises
● The best plan is one you know will work!
More on replication next.
Practice!
63
65. MySQL uses the terms MASTER and SLAVE.
It is modern and inclusive to use PRIMARY and REPLICA.
This talk uses MASTER and REPLICA.
But First, a Word About Words
65
66. ● Basics
○ One master + multiple replicas
○ Client-initiated, client configured
○ The replication user also needs grants
GRANT REPLICATION CLIENT, REPLICATION SLAVE
ON *.* TO 'replication'@'10.%' ...
Replication
66
67. ● Master executes statement, writes it to:
○ Binary log
● Replica's IO_THREAD reads from master, writes it to:
○ Relay log
● Replica's SQL_THREAD executes statement, writes it to:
○ Binary log
Replicas can have their own replicas!
Logs & Threads
67
68. ● Unique server-id is a must
● Binary log formats
○ STATEMENT
○ ROW
○ MIXED
● Log-slave-updates = ON
● master_info_repository = TABLE
MySQL won't execute events from it's own server-id.
Basic Configuration
68
72. mysql> SHOW SLAVE STATUS G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 10.10.10.44
Master_User: replication_user
Master_Port: 3306
Master_Log_File: mysql-bin.000008
Read_Master_Log_Pos: 914832470
Relay_Log_File: relay-bin.000014
Relay_Log_Pos: 914832633
Relay_Master_Log_File: mysql-bin.000008
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
...
72
73. mysql> SHOW SLAVE STATUS G
… (continued)
Last_Errno: 0
Last_Error:
...
Exec_Master_Log_Pos: 914832470
...
Seconds_Behind_Master: 0
...
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
73
74. ● Replication errors in two places
● err.log
2017-10-28 17:22:06 5061 [ERROR] Slave SQL: Error 'Duplicate entry
'14' for key 'PRIMARY'' on query. Default database: 'yelp_db'.
Query: 'INSERT INTO user (id, name) VALUES ('14', 'Barbara')',
Error_code: 1062
● SHOW SLAVE STATUS:
Last_SQL_Errno: 1062
Last_SQL_Error:'Duplicate entry '14' for key 'PRIMARY'' on query.
Default database: 'yelp_db'. Query: 'INSERT INTO user (id, name)
VALUES ('14', 'Barbara')'
What Could Possibly go Wrong?
74
75. ● Who had the problem?
[ERROR] Slave SQL: Error
● What was it trying to do?
Query: 'INSERT INTO user (id, name) VALUES ('14', 'Barbara')'
● What happened?
'Duplicate entry '14' for key 'PRIMARY'' on query.
● Where was this?
Default database: 'yelp_db
Try Breaking Errors Up
75
76. ● Wrong binary log coordinates
● Inconsistent data
Common Causes of Errors
76
78. ● If you care about it, monitor it
● Lots of open source options
○ Built-in to your monitoring solution
○ percona-toolkit
○ Cheap scripts
○ Look on github
Humans like pictures graphs!
Monitoring - Server
78
79. ● Disk space:
○ We file Jira ticket at 85%
○ And page at 95%
● Is MySQL running, does it seem OK?
● Are it's cron jobs completing?
● Does it's config match running state?
Host Stuff
79
80. ● Auto-increment id limits
● Binary log growth
● Killed queries
● Replication hierarchy
● read_only
● Temporary tables on disk
● Threads connected
● Threads running
● More fancy replication checks
MySQL Server Stuff
80
81. ● Have your application:
○ Track client errors
○ Track query times
○ Bonus: query annotations
select id, name from user where id = 1 /* host ; code_path.py */
MySQL Client Stuff
81
83. Kill and Log:
● Long-running queries
● Long-running transactions
● Too many of the same queries
Send feedback directly to developers via the client.
Automated Defense
83
84. ● MySQL client will receive errors
○ And retry
○ Or fail gracefully
● Track client errors & statistics
● Alert devs where reasonable
Let developers see immediate feedback of a change!
Handling Client Errors
84
85. Before devs push to production they can test for:
● Queries that will be slow
● Transactions that do too much
Send feedback directly to developers via the client.
Automated Testing
85
86. Seems like a lot of work!
Let's moment to reflect on humans and feelings.
Is All This Really Worth it?
86
90. If you get paged:
● And it was because of someone else's change
● You might feel #negative_emotion
● Especially if it happens again…
Think About it - Ops Perspective
90
91. If someone tells you they got paged:
● You do something about it, you didn't know
● But if it happens again…
● You might feel #negative_emotion
Think About It - Dev Perspective
91
92. Will:
● Let developers discover issues & fix on their own
● Faster iteration while developing
● Confidence when pushing
● Keep the pager for when you're really needed
<3 <3 <3
Useful Automation/Monitoring
92
95. Questions, comments, quemments?
We covered:
1. MySQL History, Philosophy, Community
2. Installation, Basic Configuration, Launching
3. Using it, Backing it up
4. Replication basics, Monitoring
5. Database Defense & fast feedback to developers
Let's Talk About It
95
96. Here is what we'll cover when we get back
● Running in Production
● Being Prepared
● Advanced Monitoring and Investigation
● Fantastic Errors and Where to Find Them
● When Things Are "Slow"
● Advanced Tuning
● Replication and Scaling Out
● Discussion
Before we Break
96
98. Part 2a:
● Running in Production
● Being Prepared
● Advanced Monitoring and Investigation
● Fantastic Errors and Where to Find Them
And we're back
98
99. Part 2b:
● When Things Are "Slow"
● Advanced Tuning
● Replication and Scaling Out
● Discussion
And we're back
99
101. Some things to think about:
● Expectations
● SLOs - uptime and query time
● Caching where you can
● Degrading gracefully
Be prepared, or prepare to be surprised!
Welcome to Production!
101
102. Nothing is up 100%
● Get PM's and devs involved
● Put features behind toggles
● Maintenance mode(s)
● Read-only mode
Because We Measure Uptime in 9's
102
104. Embrace your fears:
● Think about what can go wrong
● Plan a response - write Runbooks!
● Practice
I don't always test in production...
Preparation For the Worst
104
105. Sometimes you have to do this anyway:
● Upgrades
● New hardware
● New features mean new tables & queries
Try to use the same procedures every time.
It's Not All Bad
105
106. ● Host failures
● Network failures
● Provider failures
● Overwhelming usage
What Could go Wrong?
106
108. Normalize maintenance & disaster recovery
● Replacing a replica
○ Same as building a new one
● Replacing a master
○ Same as promoting a new one
● Cloning a database
○ Same as restoring a backup
Best Practices - Server
108
109. ● Give developers insight into DB health during:
○ Pushing code
○ Traffic surges
Best Practices - Application
109
113. ● Monitoring & alerting is pretty much
○ Collecting data
○ Using it
Advanced Monitoring and Investigation
113
114. Examples of things to monitor on a DB server:
● Is MySQL running?
● Load % CPU Idle
● Disk usage
● Critical processes
● Critical jobs
Monitoring the Server
114
115. ● Auto-increment id limits
● Binary log growth
● Killed queries
● Replication delay, hierarchy
● read_only
● Temporary tables on disk
● Threads connected
● Threads running
● Fancy replication checks
Monitoring MySQL
115
116. SHOW GLOBAL STATUS has counters & gauges
● Queries per second
● Threads running
● Threads connected
● Slow queries
● Open transactions
Monitoring MySQL Usage
116
117. You can monitor these through our client:
● Error rate
○ Rates per type
● Log all warnings & errors
Monitoring MySQL Client
117
121. Make sure the alert:
● Goes to the right team!
● Contains
○ A link to the runbook/guide
○ The first step to take
Make your Alerts Actionable
121
122. Nothing is perfect the first time around
● Tune thresholds
● Keep track of alerts and response
● Handover to the next person down the line
If it happens consistently, automate the response!
Iterate and Pay it Forward
122
125. ● Start with the read some logs/part
● Iterate, refine, shorten
Have a Runbook/Guide
125
126. Cheapa$$ cron jobs capture this every minute:
● ps
● mysql -e "SHOW PROCESSLIST"
● mysql -e "SHOW ENGINE INNODB STATUS"
And this once a day:
● mysql -e "SHOW TABLE STATUS"
● mysqldump --no-data
What happened just before I was paged?
Tracking More Than Metrics
126
128. Collect forensic data about MySQL when problems occur.
● Can run as a daemon
● Highly configurable
● Waits for a trigger condition
● Collects a ton of information
pt-stalk
128
130. ● Connection Errors
● Client Errors
● Server Errors
● Replication Errors
2013, 'Lost connection to MySQL server during query'
Fantastic Errors...
130
131. ● Your application's log
● MySQL's error_log
● SHOW SLAVE STATUS
● SHOW ENGINE INNODB STATUS
Just <insert search engine here> for it...
And Where to Find Them
131
132. Traceback (most recent call last):
File "/blahblah/blah/blah.py", line 357, in run
return self.execute()
File "/blah/blah.py", line 359, in execute
return action(**match)
File "/blah/blaah.py", line 172, in blaah_html
return self.xtmpl('blaah.blaah',
env=self.blaah_env(blaah_form_obj))
File "/lib/blah-blah/orm/session.py", line 470, in __exit__
self.rollback()
File "/lib/blah-blah/bleh.py", line 238, in commit
self._delegate.commit()
OperationalError: (2013, 'Lost connection to MySQL
server during query') None None
132
133. ● Try running the query itself
○ Does it return a huge number of rows? Sometimes?
○ Return a large amount of data?
○ Increase max_allowed_packet
● Does it go away if you increase net_read_timeout?
● Is the application server overloaded?
● It might be the network
Debugging Lost Connections
133
134. ● Sometimes MySQL goes away…
OperationalError: (2006, 'MySQL server has gone away')
MySQL Goes Away...
134
136. A sensitive subject:
● You may store sensitive information in your DB
○ Email address
○ First & last names
○ Payment data
● You probably don't want it in your logs
● pt-fingerprint
When an Error Contains a Query
136
137. You'll find these in the error_log
● log_error_verbosity = 1
● start/stop messages
● replication information
● crashes
Server Errors
137
138. InnoDB: Warning: a long semaphore wait:
--Thread 140375092094720 has waited at row0purge.cc line 770
for 241.00 seconds the semaphore:
S-lock on RW-latch at 0x1346820 '&dict_operation_lock'
a writer (thread id 140375041206016) has reserved it in mode
exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file row0purge.cc line 770
Last time write locked in file
/dist/work/percona-server/storage/innobase/row/row0mysql.cc
line 3827
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print
diagnostic info:
InnoDB: Pending preads 5, pwrites 0
138
139. ● MySQL will dump what info it can to the log
● mysqld_safe will attempt to restart MySQL
● Crashes are not normal
MySQL Crashes
139
140. 16:22:00 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that
this binary or one of the libraries it was linked against blah
blahblah blah blah blah blah blah.
key_buffer_size=268435456
read_buffer_size=131072
max_used_connections=8
max_threads=41
thread_count=2
connection_count=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size +
sort_buffer_size)*max_threads = 309930 K bytes of memory
Hope that's ok; if not, decrease some variables in the
equation.
140
141. ● OOM Killer
● Corrupted Data
● Hardware Failure
● Bug
Common Crash Causes
141
143. Lots of things can cause this:
● Individual statements can be slow
● Lots of queries
● Lots of slow-ish queries
Find out what resource is in contention.
When Things Are "Slow"
143
144. Examples here are from our open dataset.
Not queries we use in real-life :)
Tuning an Individual Query or Table
144
145. # Time: 2017-10-27T19:38:53.671918Z
# User@Host: root[root] @ localhost [] Id: 13
# Query_time: 4.947666 Lock_time: 0.000416 Rows_sent: 5
Rows_examined: 610939
SET timestamp=1509305933;
select count(*), r.stars
from review r
join business b on r.business_id = b.id
join category c on b.id = c.business_id
where c.category = 'Soul Food'
group by r.stars ;
145
146. mysql> select count(*), r.stars from review r join
business b on r.business_id = b.id join category c on
b.id = c.business_id where c.category = 'Soul Food'
group by r.stars ;
+----------+-------+
| count(*) | stars |
+----------+-------+
| 2052 | 1 |
| 1785 | 2 |
| 2356 | 3 |
| 5432 | 4 |
| 8779 | 5 |
+----------+-------+
5 rows in set (4.95 sec)
146
147. mysql> EXPLAIN select count(*), r.stars from review r join
business b on r.business_id = b.id join /*snip*/ ... G
*********************** 1. row ***********************
id: 1
select_type: SIMPLE
table: category
partitions: NULL
type: ALL
possible_keys: fk_categories_business1_idx
key: NULL
key_len: NULL
ref: NULL
rows: 587969
filtered: 10.00
Extra: Using where; Using temporary; Using
filesort 147
150. mysql> pager fgrep rows
PAGER set to 'fgrep rows'
mysql> EXPLAIN select count(*), r.stars from review r join
business b on r.business_id = b.id join category c on b.id =
c.business_id where c.category = 'Soul Food' group by r.stars
G
rows: 587969
rows: 1
rows: 25
3 rows in set, 1 warning (0.00 sec)
150
151. mysql> nopager
PAGER set to stdout
mysql> select 587969 * 1 * 25 ;
+-----------------+
| 587969 * 1 * 25 |
+-----------------+
| 14699225 |
+-----------------+
1 row in set (0.00 sec)
# From our slow query log, the numbers can be pretty off!
# Query_time: 4.947666 Lock_time: 0.000416 Rows_sent: 5
Rows_examined: 610939
151
154. mysql> select count(*), r.stars from review r join business b
on r.business_id = b.id join category c on b.id = c.business_id
where c.category = 'Soul Food' group by r.stars ;
+----------+-------+
| count(*) | stars |
+----------+-------+
| 2052 | 1 |
| 1785 | 2 |
| 2356 | 3 |
| 5432 | 4 |
| 8779 | 5 |
+----------+-------+
5 rows in set (0.03 sec)
154
155. ● Long ALTER TABLE's can hurt performance
● pt-online-schema-change
● gh-ost
Changing the Schema IRL
155
156. There's a a great manual page How MySQL Uses Indexes
PERFORMANCE_SCHEMA can give you stats on index usage.
Keep in mind:
● Adding indexes slows down writes a bit
● Cardinality
● Leftmost-prefixing
And Just a Bit More on Indexes
156
159. Find them using SHOW PROCESSLIST
Or SELECT … FROM information_schema.processlist
You can use cheap bash commands to sort & find culprits.
Lots of Queries at Once
159
160. mysql -uroot -e "show full processlist" | awk '{print $5}' |
sort | uniq -c | sort -gr | head
jsnyder@db:~$ mysql -uroot -e "show full processlist" | awk
'{print $5}' | cut -c 1-70 | sort | uniq -c | sort -gr | head
521 Sleep
109 select count(*), r.stars from review r j
3 Query
2 Connect
1 Command
# kill them with pt-kill. Use --print to test first!
pt-kill --config /etc/mysql/emergency-kill.conf --kill
--match-info 'your query' --daemonize --run-time=60
160
161. You can:
● Some slow queries run infrequently
● Some slowish queries run all the time
● Visualization tools like Anemometer, VividCortex
Take a tcpdump & analyze with pt-query-digest:
pt-query-digest --type tcpdump tcpdump.txt
Finding the Next Thing To Optimize
161
164. InnoDB has a ton of knobs and dials
● Innodb_io_capacity
● Optimize your filesystem
● Add more memory, larger/more buffer pools
● Tinker with innodb_flush_method
There is a whole manual page dedicated to optimizing
InnoDB disk I/O.
I/O is Always the Slowest
164
165. Think of the InnoDB buffer pool as InnoDB's cache
● Increase the amount of memory to reduce I/O
Rough math:
innodb_buffer_pool_instances * innodb_log_buffer_size
+ (read_buffer_size + sort_buffer_size) * max_threads
MySQL Memory Calculator looks out of date, but can give
you a pretty good idea...
How InnoDB Uses Memory
165
166. To view running transactions:
● SHOW ENGINE INNODB STATUS
● SELECT * FROM information_schema.INNODB_TRX G
On Transactions
166
167. mysql> SHOW ENGINE INNODB STATUS ;
...
------------
TRANSACTIONS
------------
Trx id counter 50806393604
Purge done for trx's n:o < 50806389839 undo n:o < 0 state:
running but idle
History list length 527
LIST OF TRANSACTIONS FOR EACH SESSION:
...
167
168. ...
---TRANSACTION 50806392298, ACTIVE 12 sec starting index read
mysql tables in use 1, locked 1
314 lock struct(s), heap size 46632, 358 row lock(s), undo log
entries 353
MySQL thread id 23649033, OS thread handle 0x7f6bd31c7700,
query id 32707038971 10.10.10.40 application_user updating
UPDATE business SET ...
Trx read view will not see trx with id >= 50806392299, sees <
50806389830
168
169. Reduce lock contention by:
● Decreasing the number of locks your transactions use
● The total time of your transactions
● Making writes faster
See InnoDB Locking Explained with Stick Figures
Improving Transaction Efficiency
169
173. Ways to use replication:
● Take load off the master
● Send reads to the replicas
● Having a separate reporting database
● Delayed replica with pt-slave-delay
Replication and Scaling Out
173
174. ● Master writes binary logs
● Each replica has 2 threads:
○ IO_THREAD: reads from master, writes relay logs
○ SQL_THREAD: runs SQL from relay logs
● Serial per database with parallel replication
How MySQL Replication Works Again
174
175. ● master-master
○ One master "live", or
○ auto_increment_increment, offset
● master, intermediate master
● master(s) with many replicas
Some Replication Strategies
175
184. ● Don't write to sketchy databases
● Don't let your application write to replicas
● use read_only=1 to prevent writes to replicas
● SHOW MASTER STATUS, SHOW SLAVE STATUS
Remember, a host that goes away may wander back...
When in Doubt, Throw it Out
184
185. 1. Set old master to read_only=1
2. CHANGE MASTER TO if necessary
3. Configure your application to write to new master
4. Set new master to read_only=0
Never allow writes to a replica unless you know that's what you
want!
Sample Master Promotion Plan
185
187. Common causes:
● IO_Thread can't keep up
○ Local load or slow/lossy network
● SQL_Thread can't keep up
○ Local load, long statements or transactions
○ mysqlbinlog to look at transactions
Speaking of Lag...
187
188. You can always phone a friend:
● MySQL has a big, friendly online community
● Oracle MySQL, Percona, MariaDB offer support
Get the most out of your interactions with experts by
providing lots of context and what you've already tried.
When You're Really Stuck
188
190. We covered:
● Running in Production
● Being Prepared
● Advanced Monitoring and Investigation
● Fantastic Errors and Where to Find Them
● When Things Are "Slow"
● Advanced Tuning
● Replication and Scaling Out
Let's Talk About It
190
193. ● MySQL Manual
● MySQL Utilities
● Percona Toolkit
● Orchestrator: MySQL Replication Topology Manager
● ProxySQL: High Performance MySQL Proxy
● mysql-sys - tools for using performance_schema
● Anemometer - Slow Query Monitor
Appendix - Links to Resources
193
194. ● Database Reliability Engineering by Laine Campbell,
Charity Majors
● High Performance MySQL by Baron Schwartz, Peter
Zaitsev & Vadim Tkachenko
Appendix - Books
194
195. ● Database Defense in Depth by @geodbz
● GrossQueryChecker: Don't get surprised by query
performance in production! by @jcsuperstar
● InnoDB Locking Explained with Stick Figures by @billkarwin
● Blameless PostMortems and a Just Culture by @allspaw
● Literally any Percona Blog Post by @svetsmirnova
Appendix - Blog Posts
195
196. ● Pexels - Images licensed under CC0
● Icons from The Noun Project
● #WOCInTechChat
● The Jopwell Collection
Appendix - Image Sources
196
197. ● Yelp Dataset
Other datasets:
● MySQL's Employees Sample Database
● Percona's list of Sample datasets
Appendix - Yelp Dataset
197