This document provides an overview and comparison of different methods for performing online schema changes in databases. It discusses native online DDL capabilities in MySQL/MariaDB and TokuDB, as well as alternative methods like rolling schema updates, downtime windows, and the pt-online-schema-change tool. The document outlines features, limitations, and special cases to consider for different workloads and replication scenarios.
MySQL Administrator
Basic course
- MySQL 개요
- MySQL 설치 / 설정
- MySQL 아키텍처 - MySQL 스토리지 엔진
- MySQL 관리
- MySQL 백업 / 복구
- MySQL 모니터링
Advanced course
- MySQL Optimization
- MariaDB / Percona
- MySQL HA (High Availability)
- MySQL troubleshooting
네오클로바
http://neoclova.co.kr/
This tutorial covers all parallel replication implementation in MariaDB 10.0 and 10.1 and MySQL 5.6, 5.7 and 8.0 (including how it works in Group Replication).
MySQL and MariaDB have different types of parallel replication. In this tutorial, we present the different implementations that allow us to understand their limitations and tuning parameters. We cover how to make parallel replication faster and what to avoid for maximizing its benefits. We also present tests from Booking.com workloads.
Some of the subjects that are covered are group commit and optimistic parallel replication in MariaDB, the parallelism interval of MySQL and its Write Set optimization, and the ?slowing down the master to speed up the slave? optimization.
After this tutorial, you will know everything you need to implement and tune parallel replication in your environment. But more importantly, we will show how you can test parallel replication benefit in a non-disruptive way before deployment.
MySQL Administrator
Basic course
- MySQL 개요
- MySQL 설치 / 설정
- MySQL 아키텍처 - MySQL 스토리지 엔진
- MySQL 관리
- MySQL 백업 / 복구
- MySQL 모니터링
Advanced course
- MySQL Optimization
- MariaDB / Percona
- MySQL HA (High Availability)
- MySQL troubleshooting
네오클로바
http://neoclova.co.kr/
This tutorial covers all parallel replication implementation in MariaDB 10.0 and 10.1 and MySQL 5.6, 5.7 and 8.0 (including how it works in Group Replication).
MySQL and MariaDB have different types of parallel replication. In this tutorial, we present the different implementations that allow us to understand their limitations and tuning parameters. We cover how to make parallel replication faster and what to avoid for maximizing its benefits. We also present tests from Booking.com workloads.
Some of the subjects that are covered are group commit and optimistic parallel replication in MariaDB, the parallelism interval of MySQL and its Write Set optimization, and the ?slowing down the master to speed up the slave? optimization.
After this tutorial, you will know everything you need to implement and tune parallel replication in your environment. But more importantly, we will show how you can test parallel replication benefit in a non-disruptive way before deployment.
Performance Schema is a powerful diagnostic instrument for:
- Query performance
- Complicated locking issues
- Memory leaks
- Resource usage
- Problematic behavior, caused by inappropriate settings
- More
It comes with hundreds of options which allow precisely tuning what to instrument. More than 100 consumers store collected data.
In this tutorial, we will try all the important instruments out. We will provide a test environment and a few typical problems which could be hardly solved without Performance Schema. You will not only learn how to collect and use this information but have experience with it.
Tutorial at Percona Live Austin 2019
This is the presentation delivered by Karthik.P.R at MySQL User Camp Bangalore on 09th June 2017. ProxySQL is a high performance MySQL Load Balancer Designed to scale database servers.
MariaDB 10.5 binary install (바이너리 설치)
- 네오클로바 DB지원사업부
1. About MariaDB
1.1 MariaDB 개요
1.2 MariaDB as a R-DBMS
1.3 Open Source Database System
2. 설치
2.1 설치 기본 정보
2.2 설치 준비
2.3 MariaDB 설치
2.4 MariaDB 시작 / 접속 / 종료
2.5 추가 설정
ProxySQL - High Performance and HA Proxy for MySQLRené Cannaò
High Availability proxy designed to solve real issues of MySQL setups from small to very large production environments.
Presentation at Percona Live Amsterdam 2015
Replication Troubleshooting in Classic VS GTIDMydbops
This presentation talk will assist you in troubleshooting MySQL replication for the most common issues we might face with a simple comparison of how can we get them solved in the different replication methods (Classic VS GTID).
Almost Perfect Service Discovery and Failover with ProxySQL and OrchestratorJean-François Gagné
Of course there is no such thing as perfect service discovery, and we will see why in the talk. However, the way ProxySQL is deployed in this case minimizes the risk for split-brains, and this is why I qualify it as almost perfect. But let’s step back a little...
MySQL alone is not a high availability solution. To provide resilience to primary failure, other components need to be integrated with MySQL. At MessageBird, these additional components are ProxySQL and Orchestrator. In this talk, we describe how ProxySQL is architectured to provide close to perfect Service Discovery and how this, combined with Orchestrator, allows for automatic failover. The talk presents the details of the integration of MySQL, ProxySQL and Orchestrator in Google Cloud (and it would be easy to re-implement a similar architecture at other cloud vendors or on-premises). We will also cover lessons learned for the 2 years this architecture has been in production. Come to this talk to learn more about MySQL high availability, ProxySQL and Orchestrator.
24시간 365일 서비스를 위한 MySQL DB 이중화.
MySQL 이중화 방안들에 대해 알아보고 운영하면서 겪은 고민들을 이야기해 봅니다.
목차
1. DB 이중화 필요성
2. 이중화 방안
- HW 이중화
- MySQL Replication 이중화
3. 이중화 운영 장애
4. DNS와 VIP
5. MySQL 이중화 솔루션 비교
대상
- MySQL을 서비스하고 있는 인프라 담당자
- MySQL 이중화에 관심 있는 개발자
Maxscale switchover, failover, and auto rejoinWagner Bianchi
How the MariaDB Maxscale Switchover, Failover, and Rejoin works under the hood by Esa Korhonen and Wagner Bianchi.
You can watch the video of the presentation at
https://www.linkedin.com/feed/update/urn:li:activity:6381185640607809536
Using all of the high availability options in MariaDBMariaDB plc
MariaDB provides a number of high availability options, including replication with automatic failover and multi-master clustering. In this session Wagner Bianchi, Principal Remote DBA, provides a comprehensive overview of the high availability features in MariaDB, highlights their impact on consistency and performance, discusses advanced failover strategies and introduces new features such as casual reads and transparent connection failover.
MySQL Parallel Replication: inventory, use-case and limitationsJean-François Gagné
In the last 24 months, MySQL replication speed has improved a lot thanks to implementing parallel replication. MySQL and MariaDB have different types of parallel replication; in this talk, I present in details the different implementations, with their limitations and the corresponding tuning parameters. I also present benchmark results from real Booking.com workloads. Finally, I discuss some deployments at Booking.com that benefits from parallel replication speed improvements.
MySQL Parallel Replication: inventory, use-cases and limitationsJean-François Gagné
In the last 24 months, MySQL replication speed has improved a lot thanks to implementing parallel replication. MySQL and MariaDB have different types of parallel replication; in this talk, I present in detail the different implementations, with their limitations and the corresponding tuning parameters (covering MySQL 5.6, MariaDB 10.0, MariaDB 10.1 and MySQL 5.7). I also present benchmark results from real Booking.com workloads. Finally, I discuss some deployments at Booking.com that benefits from parallel replication speed improvements.
Performance Schema is a powerful diagnostic instrument for:
- Query performance
- Complicated locking issues
- Memory leaks
- Resource usage
- Problematic behavior, caused by inappropriate settings
- More
It comes with hundreds of options which allow precisely tuning what to instrument. More than 100 consumers store collected data.
In this tutorial, we will try all the important instruments out. We will provide a test environment and a few typical problems which could be hardly solved without Performance Schema. You will not only learn how to collect and use this information but have experience with it.
Tutorial at Percona Live Austin 2019
This is the presentation delivered by Karthik.P.R at MySQL User Camp Bangalore on 09th June 2017. ProxySQL is a high performance MySQL Load Balancer Designed to scale database servers.
MariaDB 10.5 binary install (바이너리 설치)
- 네오클로바 DB지원사업부
1. About MariaDB
1.1 MariaDB 개요
1.2 MariaDB as a R-DBMS
1.3 Open Source Database System
2. 설치
2.1 설치 기본 정보
2.2 설치 준비
2.3 MariaDB 설치
2.4 MariaDB 시작 / 접속 / 종료
2.5 추가 설정
ProxySQL - High Performance and HA Proxy for MySQLRené Cannaò
High Availability proxy designed to solve real issues of MySQL setups from small to very large production environments.
Presentation at Percona Live Amsterdam 2015
Replication Troubleshooting in Classic VS GTIDMydbops
This presentation talk will assist you in troubleshooting MySQL replication for the most common issues we might face with a simple comparison of how can we get them solved in the different replication methods (Classic VS GTID).
Almost Perfect Service Discovery and Failover with ProxySQL and OrchestratorJean-François Gagné
Of course there is no such thing as perfect service discovery, and we will see why in the talk. However, the way ProxySQL is deployed in this case minimizes the risk for split-brains, and this is why I qualify it as almost perfect. But let’s step back a little...
MySQL alone is not a high availability solution. To provide resilience to primary failure, other components need to be integrated with MySQL. At MessageBird, these additional components are ProxySQL and Orchestrator. In this talk, we describe how ProxySQL is architectured to provide close to perfect Service Discovery and how this, combined with Orchestrator, allows for automatic failover. The talk presents the details of the integration of MySQL, ProxySQL and Orchestrator in Google Cloud (and it would be easy to re-implement a similar architecture at other cloud vendors or on-premises). We will also cover lessons learned for the 2 years this architecture has been in production. Come to this talk to learn more about MySQL high availability, ProxySQL and Orchestrator.
24시간 365일 서비스를 위한 MySQL DB 이중화.
MySQL 이중화 방안들에 대해 알아보고 운영하면서 겪은 고민들을 이야기해 봅니다.
목차
1. DB 이중화 필요성
2. 이중화 방안
- HW 이중화
- MySQL Replication 이중화
3. 이중화 운영 장애
4. DNS와 VIP
5. MySQL 이중화 솔루션 비교
대상
- MySQL을 서비스하고 있는 인프라 담당자
- MySQL 이중화에 관심 있는 개발자
Maxscale switchover, failover, and auto rejoinWagner Bianchi
How the MariaDB Maxscale Switchover, Failover, and Rejoin works under the hood by Esa Korhonen and Wagner Bianchi.
You can watch the video of the presentation at
https://www.linkedin.com/feed/update/urn:li:activity:6381185640607809536
Using all of the high availability options in MariaDBMariaDB plc
MariaDB provides a number of high availability options, including replication with automatic failover and multi-master clustering. In this session Wagner Bianchi, Principal Remote DBA, provides a comprehensive overview of the high availability features in MariaDB, highlights their impact on consistency and performance, discusses advanced failover strategies and introduces new features such as casual reads and transparent connection failover.
MySQL Parallel Replication: inventory, use-case and limitationsJean-François Gagné
In the last 24 months, MySQL replication speed has improved a lot thanks to implementing parallel replication. MySQL and MariaDB have different types of parallel replication; in this talk, I present in details the different implementations, with their limitations and the corresponding tuning parameters. I also present benchmark results from real Booking.com workloads. Finally, I discuss some deployments at Booking.com that benefits from parallel replication speed improvements.
MySQL Parallel Replication: inventory, use-cases and limitationsJean-François Gagné
In the last 24 months, MySQL replication speed has improved a lot thanks to implementing parallel replication. MySQL and MariaDB have different types of parallel replication; in this talk, I present in detail the different implementations, with their limitations and the corresponding tuning parameters (covering MySQL 5.6, MariaDB 10.0, MariaDB 10.1 and MySQL 5.7). I also present benchmark results from real Booking.com workloads. Finally, I discuss some deployments at Booking.com that benefits from parallel replication speed improvements.
We will show how Galera Cluster executes DDLs in a safe, consistent manner across all the nodes in the cluster, and the differences with stand-alone MySQL. We will discuss how to prepare for and successfully carry out a schema upgrade and the considerations that need to be taken into account during the process.
A presentation about how to make MySQL highly available, presented at the San Francisco MySQL Meetup (http://www.sfmysql.org/events/15760472/) on January 26th, 2011.
A video recording of this presentation is available from Ustream: http://ustre.am/fyLk
Pseudo GTID and Easy MySQL Replication Topology ManagementShlomi Noach
Use Pseudo GTID to refactor your MySQL replication topology even in face of master/intermediate master craches; use open source orchestrator to automate, visualize and persist your topologies
In this presentation we discuss the New Features of MariaDB 10.4. First we give a short overview of the MariaDB Branches and Forks. Then we talk about the announced IPO. Technically we cover topics like Authentication, Accounts, InnoDB, Optimizer improvements, Application-Time Period Tables the new Backup Stage Galera 4 and other changes...
MariaDB 10.4 became General Available (GA = ready for production) this summer. So it is time to look at the new Features in MariaDB 10.4. After a short intro about history we look for the reason of broad usage of MariaDB nowadays. Most important improvements where in User Authentication, InnoDB improvements, and Optimizer enhancements. A completely New Feature is Application-Time Period Tables. Backup got a new Locking behaviour so LVM snapshots are possible and officially supported now. And last but not least MariaDB 10.4 comes with Galera 4.
This ppt was used by Devrim at pgDay Asia 2017. He talked about some important facts about WAL - Transaction Logs or xlogs in PostgreSQL. Some of these can really come handy on a bad day
Yahoo: Experiences with MySQL GTID and Multi Threaded ReplicationYashada Jadhav
Our experiences running GTID and parallel replication in production, advantages, limitations, lessons learnt and tips and tricks we discovered along the way at Oracle Openworld 2015
Similar to M|18 Battle of the Online Schema Change Methods (20)
SkySQL is the first and only database-as-a-service (DBaaS) to perform workload analysis with advanced deep learning models, identifying and classifying discrete workload patterns so DBAs can better understand database workloads, identify anomalies and predict changes.
In this session, we’ll explain the concepts behind workload analysis and show how it can be used in the real world (and with sample real-world data) to improve database performance and efficiency by identifying key metrics and changes to cyclical patterns.
SkySQL uses best-of-breed software, and when it comes to metrics and monitoring that means Prometheus and Grafana. SkySQL Monitor is built on both, and provides customers with interactive dashboards for both real-time and historic metrics monitoring. In addition, it meets the same high availability and security requirements as other SkySQL components, ensuring metrics are always available and always secure.
In this session, we’ll explain how SkySQL Monitor works, walk through its dashboards and show how to monitor key metrics for performance and replication.
Introducing the R2DBC async Java connectorMariaDB plc
Not too long ago, a reactive variant of the JDBC driver was released, known as Reactive Relational Database Connectivity (R2DBC for short). While R2DBC started as an experiment to enable integration of SQL databases into systems that use reactive programming models, it now specifies a full-fledged service-provider interface that can be used to retrieve data from a target data source.
In this session, we’ll take a look at the new MariaDB R2DBC connector and examine the advantages of fully reactive, non-blocking development with MariaDB. And, of course, we’ll dive in and get a first-hand look at what it’s like to use the new connector with some live coding!
The capabilities and features of MariaDB Platform continue to expand, resulting in larger and more sophisticated production deployments – and the need for better tools. To provide DBAs with comprehensive, consolidating tooling, we created MariaDB Enterprise Tools: an easy-to-use, modular command-line interface for interacting with any part of MariaDB Platform.
In this session, we will provide a preview of the MariaDB Enterprise Client, walk through current and planned modules and discuss future plans for MariaDB Enterprise Tools – including SkySQL modules and the ability to create custom modules.
Faster, better, stronger: The new InnoDBMariaDB plc
For MariaDB Enterprise Server 10.5, the default transactional storage engine, InnoDB, has been significantly rewritten to improve the performance of writes and backups. Next, we removed a number of parameters to reduce unnecessary complexity, not only in terms of configuration but of the code itself. And finally, we improved crash recovery thanks to better consistency checks and we reduced memory consumption and file I/O thanks to an all new log record format.
In this session, we’ll walk through all of the improvements to InnoDB, and dive deep into the implementation to explain how these improvements help everything from configuration and performance to reliability and recovery.
SkySQL implements a groundbreaking, state-of-the-art architecture based on Kubernetes and ServiceNow, and with a strong emphasis on cloud security – using compartmentalization and indirect access to secure and protect customer databases.
In this session, we’ll walk through the architecture of SkySQL and discuss how MariaDB leverages an advanced Kubernetes operator and powerful ServiceNow configuration/workflow management to deploy and manage databases on cloud infrastructure.
StarCompliance is a leading firm specializing in the recovery of stolen cryptocurrency. Our comprehensive services are designed to assist individuals and organizations in navigating the complex process of fraud reporting, investigation, and fund recovery. We combine cutting-edge technology with expert legal support to provide a robust solution for victims of crypto theft.
Our Services Include:
Reporting to Tracking Authorities:
We immediately notify all relevant centralized exchanges (CEX), decentralized exchanges (DEX), and wallet providers about the stolen cryptocurrency. This ensures that the stolen assets are flagged as scam transactions, making it impossible for the thief to use them.
Assistance with Filing Police Reports:
We guide you through the process of filing a valid police report. Our support team provides detailed instructions on which police department to contact and helps you complete the necessary paperwork within the critical 72-hour window.
Launching the Refund Process:
Our team of experienced lawyers can initiate lawsuits on your behalf and represent you in various jurisdictions around the world. They work diligently to recover your stolen funds and ensure that justice is served.
At StarCompliance, we understand the urgency and stress involved in dealing with cryptocurrency theft. Our dedicated team works quickly and efficiently to provide you with the support and expertise needed to recover your assets. Trust us to be your partner in navigating the complexities of the crypto world and safeguarding your investments.
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...Subhajit Sahu
Abstract — Levelwise PageRank is an alternative method of PageRank computation which decomposes the input graph into a directed acyclic block-graph of strongly connected components, and processes them in topological order, one level at a time. This enables calculation for ranks in a distributed fashion without per-iteration communication, unlike the standard method where all vertices are processed in each iteration. It however comes with a precondition of the absence of dead ends in the input graph. Here, the native non-distributed performance of Levelwise PageRank was compared against Monolithic PageRank on a CPU as well as a GPU. To ensure a fair comparison, Monolithic PageRank was also performed on a graph where vertices were split by components. Results indicate that Levelwise PageRank is about as fast as Monolithic PageRank on the CPU, but quite a bit slower on the GPU. Slowdown on the GPU is likely caused by a large submission of small workloads, and expected to be non-issue when the computation is performed on massive graphs.
Explore our comprehensive data analysis project presentation on predicting product ad campaign performance. Learn how data-driven insights can optimize your marketing strategies and enhance campaign effectiveness. Perfect for professionals and students looking to understand the power of data analysis in advertising. for more details visit: https://bostoninstituteofanalytics.org/data-science-and-artificial-intelligence/
As Europe's leading economic powerhouse and the fourth-largest hashtag#economy globally, Germany stands at the forefront of innovation and industrial might. Renowned for its precision engineering and high-tech sectors, Germany's economic structure is heavily supported by a robust service industry, accounting for approximately 68% of its GDP. This economic clout and strategic geopolitical stance position Germany as a focal point in the global cyber threat landscape.
In the face of escalating global tensions, particularly those emanating from geopolitical disputes with nations like hashtag#Russia and hashtag#China, hashtag#Germany has witnessed a significant uptick in targeted cyber operations. Our analysis indicates a marked increase in hashtag#cyberattack sophistication aimed at critical infrastructure and key industrial sectors. These attacks range from ransomware campaigns to hashtag#AdvancedPersistentThreats (hashtag#APTs), threatening national security and business integrity.
🔑 Key findings include:
🔍 Increased frequency and complexity of cyber threats.
🔍 Escalation of state-sponsored and criminally motivated cyber operations.
🔍 Active dark web exchanges of malicious tools and tactics.
Our comprehensive report delves into these challenges, using a blend of open-source and proprietary data collection techniques. By monitoring activity on critical networks and analyzing attack patterns, our team provides a detailed overview of the threats facing German entities.
This report aims to equip stakeholders across public and private sectors with the knowledge to enhance their defensive strategies, reduce exposure to cyber risks, and reinforce Germany's resilience against cyber threats.
Show drafts
volume_up
Empowering the Data Analytics Ecosystem: A Laser Focus on Value
The data analytics ecosystem thrives when every component functions at its peak, unlocking the true potential of data. Here's a laser focus on key areas for an empowered ecosystem:
1. Democratize Access, Not Data:
Granular Access Controls: Provide users with self-service tools tailored to their specific needs, preventing data overload and misuse.
Data Catalogs: Implement robust data catalogs for easy discovery and understanding of available data sources.
2. Foster Collaboration with Clear Roles:
Data Mesh Architecture: Break down data silos by creating a distributed data ownership model with clear ownership and responsibilities.
Collaborative Workspaces: Utilize interactive platforms where data scientists, analysts, and domain experts can work seamlessly together.
3. Leverage Advanced Analytics Strategically:
AI-powered Automation: Automate repetitive tasks like data cleaning and feature engineering, freeing up data talent for higher-level analysis.
Right-Tool Selection: Strategically choose the most effective advanced analytics techniques (e.g., AI, ML) based on specific business problems.
4. Prioritize Data Quality with Automation:
Automated Data Validation: Implement automated data quality checks to identify and rectify errors at the source, minimizing downstream issues.
Data Lineage Tracking: Track the flow of data throughout the ecosystem, ensuring transparency and facilitating root cause analysis for errors.
5. Cultivate a Data-Driven Mindset:
Metrics-Driven Performance Management: Align KPIs and performance metrics with data-driven insights to ensure actionable decision making.
Data Storytelling Workshops: Equip stakeholders with the skills to translate complex data findings into compelling narratives that drive action.
Benefits of a Precise Ecosystem:
Sharpened Focus: Precise access and clear roles ensure everyone works with the most relevant data, maximizing efficiency.
Actionable Insights: Strategic analytics and automated quality checks lead to more reliable and actionable data insights.
Continuous Improvement: Data-driven performance management fosters a culture of learning and continuous improvement.
Sustainable Growth: Empowered by data, organizations can make informed decisions to drive sustainable growth and innovation.
By focusing on these precise actions, organizations can create an empowered data analytics ecosystem that delivers real value by driving data-driven decisions and maximizing the return on their data investment.
Techniques to optimize the pagerank algorithm usually fall in two categories. One is to try reducing the work per iteration, and the other is to try reducing the number of iterations. These goals are often at odds with one another. Skipping computation on vertices which have already converged has the potential to save iteration time. Skipping in-identical vertices, with the same in-links, helps reduce duplicate computations and thus could help reduce iteration time. Road networks often have chains which can be short-circuited before pagerank computation to improve performance. Final ranks of chain nodes can be easily calculated. This could reduce both the iteration time, and the number of iterations. If a graph has no dangling nodes, pagerank of each strongly connected component can be computed in topological order. This could help reduce the iteration time, no. of iterations, and also enable multi-iteration concurrency in pagerank computation. The combination of all of the above methods is the STICD algorithm. [sticd] For dynamic graphs, unchanged components whose ranks are unaffected can be skipped altogether.
4. ● “Online” can mean several things:
● Changes only happen to metadata.
● Changes are made in-place, and the software allows concurrent DML.
● Table is rebuilt with table copy, but the software allows concurrent DML.
● In some cases (e.g., TokuDB), changes are made in memory and not pushed to disk until
the row is accessed. More on that later!
Definition
7. Native Online DDL - InnoDB/XtraDB - Overview
● Supported since MariaDB 10.0 / MySQL 5.6
● Information about progress is limited
● MariaDB >= 10.1.1: SHOW STATUS LIKE 'innodb_onlineddl%'; and error log
● MySQL >= 5.7.6: Performance schema stage/innodb/alter%
8. Native Online DDL - InnoDB/XtraDB - Settings
● Algorithm and locking methods
● e.g., ALTER TABLE test ADD INDEX name_idx (name), ALGORITHM=inplace, LOCK=none;
● You get an error message if the specified algorithm/locking cannot be used
● If not explicitly specified e.g., ALTER TABLE test ADD INDEX name_idx (name), the less expensive
combination will be used
● ALGORITHM=copy, LOCK=shared is the old behaviour (only reads allowed)
9. ● Some operations still require a table copy
● Change column data type
i. from MariaDB 10.2.2, increasing the size of a VARCHAR column can be done in-place
● Charset change
● Primary Key change
● DATE, DATETIME, TIMESTAMP columns created before MariaDB 10.1.2 / MySQL 5.6
● Online DDL not really “online” on replicas
● Operation needs to finish on the master before being replicated
● SQL thread is “blocked” while replica is executing the DDL operation
● Lag spike at the end
Native Online DDL - InnoDB/XtraDB - Limitations
10. ● Works only with InnoDB
● Can’t pause an online DDL operation
● Rollback can be expensive
● LOCK=none not supported with fulltext indexes
● https://bugs.mysql.com/bug.php?id=81819
● Instant ADD COLUMN for InnoDB in MariaDB 10.3+ is in RC
● https://jira.mariadb.org/browse/MDEV-11369
https://mariadb.com/kb/en/library/online-ddl-overview/
https://dev.mysql.com/doc/refman/5.7/en/innodb-create-index-overview.html
Native Online DDL - InnoDB/XtraDB - More Limitations
11. ● “TokuDB enables you to add or delete columns in an existing table, expand
char, varchar, varbinary, and integer type columns in an existing table, or
rename an existing column in a table with little blocking of other updates
and queries. … The work of altering the table for column addition,
deletion, or expansion is performed as subsequent operations touch
parts of the Fractal Tree, both in the primary index and secondary
indexes.”
Native Online DDL - TokuDB - Overview
Note: tokudb_version=5.6.37-82.2
12. ● Don’t change too much at once. For example, in changing column name, don’t change other
elements, or it will revert to a regular mysql (not hot copy) alter. This was originally intcol1
int(11) default null.
Native Online DDL - TokuDB - Limitations
ALTER TABLE test.t1
CHANGE intcol1 intone
INT(11) DEFAULT NULL;
Query OK, 0 rows affected (0.01 sec)
Records: 0 Duplicates: 0 Warnings: 0
ALTER TABLE test.t1
CHANGE intcol1 inttwo
INT(11) NOT NULL;
Query OK, 56777 rows affected (1.52 sec)
Records: 56777 Duplicates: 0 Warnings: 0
Note: tokudb_version=5.6.37-82.2
13. ● Don’t try to do multiple types of changes in one statement, or multiple renames.
Native Online DDL - TokuDB - HCADER - Limitations
These are not executed online:
ALTER TABLE test.t2
DROP number4,
ADD number7 int(11) default null;
Query OK, 56777 rows affected (0.93 sec)
Records: 56777 Duplicates: 0 Warnings: 0
ALTER TABLE test.t2
CHANGE number4 num4 int(11) default null,
CHANGE number7 num7 int(11) default null;
Query OK, 56777 rows affected (1.53 sec)
Records: 56777 Duplicates: 0 Warnings: 0
Although these are (because natively supported):
ALTER TABLE test.t2
ADD number5 int(11) default null,
ADD number6 int(11) default null;
Query OK, 0 rows affected (0.02 sec)
Records: 0 Duplicates: 0 Warnings: 0
ALTER TABLE test.t2
DROP number5,
DROP number6;
Query OK, 0 rows affected (0.18 sec)
Records: 0 Duplicates: 0 Warnings: 0
14. ● If dropping a column with an index, drop the index first, then drop the column.
Native Online DDL - TokuDB - HCADER - Limitations
This is not executed online, because number4 has an
associated index:
ALTER TABLE test.t1
DROP column number4;
alter table test.t1 drop column number4;
Query OK, 159002 rows affected (4.52 sec)
Records: 159002 Duplicates: 0 Warnings: 0
Separating the statements allows the drop to happen
online:
ALTER TABLE test.t1
DROP index idx4;
Query OK, 0 rows affected (0.37 sec)
Records: 0 Duplicates: 0 Warnings: 0
ALTER TABLE test.t1
DROP column number4;
Query OK, 0 rows affected (0.18 sec)
Records: 0 Duplicates: 0 Warnings: 0
15. ● Changing the size of an int, char, varchar, or varbinary is supported online, but only if the
column doesn’t have an index attached to it.
Native Online DDL - TokuDB - HCADER - Limitations
This is not executed online, because string1 has an
associated index:
...
`string1` char(2) DEFAULT NULL,
...
KEY `idxstr1` (`string1`)
ALTER TABLE test.t2
change column string1 string1 char(3)
default null;
Query OK, 56777 rows affected (6.39 sec)
Records: 56777 Duplicates: 0 Warnings: 0
This is executed online, because string2 has no index:
...
`string2` char(2) DEFAULT NULL,
ALTER TABLE test.t2
change column string2 string2 char(3)
default null;
Query OK, 0 rows affected (0.26 sec)
Records: 0 Duplicates: 0 Warnings: 0
16. ● A note about performance: “The time that the table lock is held can vary. The table-locking time
for HCADER is dominated by the time it takes to flush dirty pages, because MySQL closes the
table after altering it. If a checkpoint has happened recently, this operation is fast (on the order
of seconds). However, if the table has many dirty pages, then the flushing stage can take
on the order of minutes.”
● Another note about performance: “The work of altering the table for column addition, deletion,
or expansion is performed as subsequent operations touch parts of the Fractal Tree, both in
the primary index and secondary indexes.” Think about very busy workloads.
● Take this very seriously, especially in the presence of replication! Test early and often.
Native Online DDL - TokuDB - HCADER - Limitations
17. ● Adding indexes is done online, allowing concurrent inserts and updates
by:
1. Setting the variable:
set tokudb_create_index_online=on
2. Using the syntax:
create index index_name on table_name (column_name);
BUT will block replication on replicas: it is long-running even though non-blocking
Native Online DDL - TokuDB - Adding Indexes
18. All changes require a table copy, except create or drop secondary indexes.
More may be supported soon. https://github.com/facebook/mysql-5.6/issues/47
Native Online DDL - Native Online DDL - RocksDB
19. ● Traditional online DDL
● Aurora Fast DDL
● updates INFORMATION_SCHEMA system table with the new schema
● records the old schema into a Schema Version Table with the timestamp
● change is propagated to read replicas
● on subsequent DML operations, check to see if the affected data page has a pending
schema operation by comparing the log sequence number (LSN) timestamp for the page
with the LSN timestamp of schema changes.
● update the page to the new schema before applying the DML statement
https://aws.amazon.com/blogs/database/amazon-aurora-under-the-hood-fast-ddl/
Native Online DDL: RDS MySQL / Aurora - Overview
20. ● Problems with traditional online DDL
● Temp table can fill ephemeral storage
● Fast DDL
● only supports adding nullable columns to the end of a table
● no support for partitioned tables
● no support for the older REDUNDANT row format
● Only usable if the max possible record size is smaller than half the page size
Native Online DDL: RDS MySQL / Aurora - Limitations
22. ● Stop replica
● Perform the alter on the replica
● Start the replica and allow it to catch up
● (repeat...)
● Promote replica as master + deploy new code release
● Perform the alter on the old master
● Promote the old master back (optional)
Rolling Schema Updates - Overview
23. ● Time-consuming manual process
● Complexity proportional to number of slaves
● Schema is inconsistent
● If using slaves for reads, the application needs to be forward/backward compatible
● Some changes can break replication
● Potential problems with GTID
Rolling Schema Updates - Limitations
24. ● Run a DDL change on slave A
● The DDL change is applied to other slaves (B, C, …)
● Promote slave A as master
● Possible outcomes:
● Got fatal error 1236 from master when reading data from binary log:
'The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION
= 1, but the master has purged binary logs containing GTIDs that the
slave requires
● If server A still has the binlog with the DDL change, the transaction is sent (again) to all
current slaves
Rolling Schema Updates - Limitations - Errant Tx
25. ● Avoid errant transactions
● Disable binlog for the session
● Run the changes
● How to spot them
● Check executed_gtid_set in the slave (show slave status)
● Check executed_gtid_set in the master (show master status)
● Use gtid_substract(‘slave_set’, ‘master_set’) function
Rolling Schema Updates - Limitations - Errant Tx
27. ● Schedule application downtime
● Do changes during code rollout maintenance
● Rarely used these days, but sometimes can choose to do impactful
changes during relatively quiet times to reduce length of running change
Downtime - Overview
29. ● Create an empty copy of the table to alter
● Modify the new table
● Create triggers from old to new table (_ins, _upd, _del)
● Copy rows from the original table into the new table (in chunks)
● Move away the original table and replace it with the new one
pt-osc - Overview
https://www.percona.com/doc/percona-toolkit/LATEST/pt-online-schema-change.html
30. ● Chunk size controls
● --chunk-time
i. Adjusts chunk size dynamically based on row copy time
● --chunk-size
i. Specify a fixed chunk size
● Triggers
● --preserve-triggers (MySQL 5.7 only)
● Finding replicas
● --recursion-method
i. processlist
ii. hosts (Uses SHOW SLAVE HOSTS)
iii. dsn=DSN
iv. none
pt-osc - Settings
31. ● Throttling
● Master
i. --max-load Threads_connected=800,Threads_running=80
ii. --critical-load
iii.--pause-file
● Slave lag
i. --max-lag
ii. --check-interval
iii.--check-slave-lag or DSN to monitor specific replica(s)
pt-osc - Settings
32. ● Potential stalls after dropping a big table
● Use --no-drop-old-table
● Trigger-related
● Tool cannot be completely paused
● Adding/dropping triggers is expensive!
i. Control metadata lock time
1. --set-vars=lock_wait_timeout=10
ii. Use retries
1. --tries create_triggers:5:0.5,drop_triggers:5:0.5 (try 5 times, wait 0.5 sec
between tries)
pt-osc - Limitations
33. ● Dealing with foreign keys
● --alter-foreign-keys-method
i. drop_swap: no atomic rename
ii. rebuild_constraints: rebuild is done using normal blocking DDL
iii. auto: let pt-osc decide
● FK Names: adds/removes underscores at the beginning based on the first FK of the table
● Be careful with underscores in FK names
● Try to avoid (in the same table) things like:
i. _FK1
ii. __FK2
iii. FK3
● https://bugs.launchpad.net/percona-toolkit/+bug/1428812
pt-osc - More Limitations
35. ● Create an empty copy (“shadow”) of the table to alter with the new
structure
● Create change capture (“deltas”) table
● All columns of the source table
● Extra integer auto-increment column to track order of changes
● Extra integer column to track DML type
● Create 3 triggers from old to “deltas” table
● Inserts
● Deletes
● Updates
Facebook OSC - Overview
36. ● Dump chunks of rows from the original table & load to new table
● Replay changes from deltas table
● Checksum
● Cut-over
● Lock the existing table
● Final round of replay
● Swap the tables
Facebook OSC - Overview
37. Facebook OSC - Unique features
● Developed for rolling schema updates
● Tool runs with sql-log-bin=0
● Change replay is asynchronous
● Use of SELECT INTO OUTFILE / LOAD DATA INFILE to avoid gap lock
● Manage schema via source control
● Relies on a file containing CREATE TABLE to run the schema change
● Supports MyRocks
● Implemented as standalone Python class
● Interact with OSC from your code
38. Facebook OSC - Limitations
● No support for triggers, FK, rename column or RBR
● Only one OSC can run at a time
● Requires Python 2.7
● Difficult to install in older distributions
● No atomic rename
● 'table not found' errors can happen for a short period
● Implications of using sql-log-bin=0
https://github.com/facebookincubator/OnlineSchemaChange
40. ● Create an empty copy of the table to alter
● Modify the new table
● Hook up as a MySQL replica (stream binlog events)
● Copy rows from the original table into the new table in chunks
● Apply events on the new table
● When the copy is complete, move away the original table and replace it
with the new one
gh-ost - Overview
41. ● No triggers!
● Can be completely paused
● Low overhead and less locking
● Multiple concurrent migrations
● Dynamic reconfiguration
● Can offload some operations to a slave
● reading binlog events
● queries required for (optional) accurate progress counter
● Supports testing or migrating on a slave
● --test-on-replica
● --migrate-on-replica
gh-ost - Unique features
42. ● Throttling
● Master
i. --max-load=Threads_connected=800,Threads_running=80
ii. --critical-load
iii.--critical-load-interval-millis=10000 (2nd chance)
iv. --critical-load-hibernate-seconds=300 (don’t panic, hibernate instead)
● Slave lag
i. --max-lag-millis
ii. --throttle-control-replicas
● Pause
i. echo throttle | nc -U /tmp/gh-ost.test.sock
gh-ost - Settings
44. gh-ost - Limitations
● No support for FK or triggers
● Needs row based binlogs on at least one slave
● No support for new MySQL 5.7 columns
● Generated
● POINT
● No JSON PKs
● Master-master is supported but only active-passive
46. Use Case: RDS MySQL / Aurora - Overview
● Black-box approach
● RDS replicas similar to traditional slaves
● Aurora replicas share the same storage with the master
● STOP/START SLAVE done through procedures
● mysql.rds_stop_replication
● mysql.rds_start_replication
48. Use Case: RDS MySQL / Aurora - Complications
● Privileges are limited
● No SUPER
● Detecting replicas is tricky
● processlist gives you the internal IP
● Aurora read replicas are “special”
● Read_only can’t be modified
● Replication filters are used
● Replicate_Ignore_Table: mysql.plugin,mysql.rds_monitor
● Binlogs are disabled by default
50. Use Case: Aurora - gh-ost on master
● Make sure binary logs are enabled
● Set backup retention >= 1 day
● set binlog_format=ROW in parameter group
● use --assume-rbr
● Specify the Aurora writer endpoint in --host
● use --allow-on-master
51. ./gh-ost
…
--alter=”add column b int”
--assume-rbr
--allow-on-master
--host=<aurora writer endpoint>
--database="testdb"
--table="mytable"
--verbose
--execute
Use Case: Aurora - gh-ost on master
52. Use Case: RDS MySQL - gh-ost on replica
● Make sure binary logs are enabled on the replica
● Set backup retention >= 1 day
● Set binlog_format=ROW in the parameter group
● Use --assume-rbr
● Specify the replica endpoint in --host
● Specify the master’s endpoint in --assume-master-host
53. Use Case: RDS MySQL - gh-ost on replica
./gh-ost
…
--alter=”add column b int”
--assume-rbr
--assume-master-host=<master endpoint>
--host=<replica endpoint>
--max-lag-millis=5000
--database="testdb"
--table="mytable"
--verbose
--execute
55. ● Tungsten uses an external replicator process
● Reads from the binary logs (row or statement-based data)
● Converts transaction data to db-agnostic format
● Writes into Transaction History Log (THL) files
● Hosts are not aware they are replicating
● SHOW SLAVE STATUS is empty
Use case: Tungsten - Overview
57. ● Issues with pt-osc
● Replicas have to be manually specified (DSN table)
● Binlog format considerations (triggers)
■ STATEMENT: works out of the box
■ ROW: use Tungsten plugin for Percona toolkit
■ MIXED: do not use!
● Issues with gh-ost
● Can’t detect the master/replicas automatically
● Can’t start/stop replication
● Can’t switch binlog format
Use case: Tungsten - Complications
60. ● Pass the slave to --host
● Use --assume-master-host to specify the actual master
● Specify --tungsten argument
● Slave needs binlog_format=ROW
● Use Tungsten log-slave-updates parameter
○ tpm configure property=log-slave-updates=true
○ hang at INFO Waiting for tables to be in place otherwise
Use case: Tungsten - gh-ost using a slave
63. ● Can’t benefit from online DDL
● Total order isolation (TOI)
● Blocks all writes to the cluster - no thanks!
● Rolling schema updates (RSU)
● Block one node at a time
● New and old schema definitions need to be compatible
● Manual operation
● Adjust gcache big enough to allow IST after
Use case: Galera - Overview
65. ● pt-osc
● Only InnoDB tables are supported
● gh-ost
● No official support for Galera/XtraDB Cluster
i. Works in Galera 5.6 using --cut-over=two-step
ii. Doesn’t work with Galera 5.7
iii. https://github.com/github/gh-ost/issues/224
Use case: Galera - Complications
67. ● Set wsrep_OSU_method to TOI
● ALTER of _new table needs to replicate to all nodes at the same time
● No other special arguments are required
● Throttling
● No replication lag in Galera
● --max-flow-ctl (percentage)
i. Check average time cluster spent pausing for Flow Control and make pt-osc pause if
it goes over the percentage indicated
ii. available for Galera >=5.6
Use case: Galera - pt-osc
69. Multisource replication is commonly found in MariaDB because it has been
supported in the product for a while. One replica supports multiple sources.
Use case: Multisource Replication - Overview
m1
m2
s1
72. Use case: Multisource Replication
Previous version required running
gh-ost on the master. See
https://github.com/github/gh-
ost/issues/225. Now we can use
assume-master-host.
m1
m2
s1
gh-ost
./gh-ost
--max-load=Threads_running=25
--critical-load=Threads_running=1000
--chunk-size=1000
--throttle-control-replicas="192.168.56.12"
--max-lag-millis=1500
--user="ghost"
--password="ghost"
--allow-master-master
--assume-master-host=192.168.56.10
--database="d1"
--table="t1"
--verbose
--alter="change column name name char(50)"
--cut-over=default
--default-retries=120
--panic-flag-file=/tmp/ghost.panic.flag
--postpone-cut-over-flag-file=/tmp/ghost.postpone.flag
--initially-drop-ghost-table
--initially-drop-old-table
--execute
74. Daisy-chained replicas are commonly used in complex topologies to scale a
large number of second-level replicas while avoiding load on the primary from
the replication threads. The key is to use “log-slave-updates” so that changes
are passed to the second-level replicas.
Use case: Daisy-chained replication - Overview
m1 s1 s1a
log-slave-updates
81. Use case: Very large tables - Complications
● Problems with DDL changes are exacerbated
● Replication lag
● Expensive rollback
● Info about progress can be limited
● Long queries
● Disk space
● Memory usage
83. Use case: Very large tables
● External tools are more desirable than online DDL
● pt-osc is usually faster than gh-ost
● Experiment with --chunk-size and --chunk-time vs default auto
● Set high timeouts
● Retry failed operations
● Throttle
● If table is extremely big, consider dump/load into an altered table
84. Use case: Very large tables - pt-osc
pt-online-schema-change --set-
vars=lock_wait_timeout=10,innodb_lock_wait_timeout=10
--critical-load Threads_running=1000
--max-load Threads_running=50
--max-lag 30
--alter-foreign-keys-method=drop_swap
--chunk-time 1
--alter '...'
D=my_db,t=my_table
--execute
86. Use case: Very busy tables - Overview
● Busy tables receive many changes to data over short period of time
● Often combined with very large tables use case
88. Use case: Very busy tables - Complications
● Usual problems with online DDL
● No throttling
● Progress info is limited
● Long queries, IO contention
● Problem with buffering of ongoing changes
● Innodb_online_alter_log_max_size parameter
■ stores data inserted, updated, or deleted in the table during the DDL operation
■ If a temporary log file exceeds the upper size limit, operation fails
■ A large value extends the period of time at the end of the DDL operation when the
table is locked to apply the data from the log
90. Use case: Very busy tables
● Use a replica to offload traffic from the master
● Use small chunks
● Set low timeouts to reduce impact
● Configure retry of failed operations
● Configure throttling
● Trigger the cut-over manually on a low traffic period
93. Use case: Metadata-only - Overview
● Metadata-only changes touch the .frm -- the table definition -- not the
data files
● Examples
● dropping an index
● set column default value
● change auto_increment
● change column name
● FK changes with foreign_key_checks=0
(Valerie and Ivan)
Hi, I’m Valerie. I’m in the open source database cluster at Pythian.
Hi, I’m Ivan...
Today we’re here to referee a battle of the online schema change methods.
(Valerie starts)
First we’ll walk through a few definitions,
give a basic overview of online DDL,
and then discuss alternate methods and tools.
Then we’ll engage in battle!
What does it mean when we say we want to do an “online” schema change?
Online can mean several things.
The change may be happening only to metadata of the table, the data definition files.
The change might be happening in-place (not a table copy per se, but sometimes requiring reorganization), and allowing changes to the data (like inserts and updates) to happen at the same time.
The change might require a full table copy on disk, but also not block statements.
With some engines, this can even mean the changes are made in temporary memory structures, and written by row to the main storage as data is accessed.
So it means a lot of things at the database level. From the application’s perspective, though, it means statements aren’t blocked while your schema change is happening.
Why do we care about changes being online? Again, from the perspective of the application, statements can continue to run against the database, reading and writing without issue, while the change is happening. That means no downtime for your application.
There’s no promise that performance might not dip some, and there may be a brief pause at the beginning or end of a long-running change, but in general there will be no application failures.
As database technologies have matured, they’ve all started to allow more native capabilities for online changes.
Native online schema changes have been supported in MariaDB since version 10.0, and in MySQL since version 5.6.
Status output about progress of long-running changes has improved, but remains limited. Remember this when we get to the battle: when a change has been running for 2 days, a manager might ask you when it’s going to be done.
You can specify three alter algorithms: default, copy, or inplace. If nothing is specified, then the default behavior will occur for this type of change.
You might specify “copy” if you want to force a table rebuild, but in general, you’ll want to choose inplace. If the alter doesn’t support inplace, you’ll receive an error message. If you get into the habit of specifying algorithm=inplace, you’ll be prepared if a change will require a table copy.
You can specify four alter locks: default, none, shared, exclusive. Again, get into the habit of using “none” to specify that that alter won’t block operations.
Her are some changes remaining that require table copies.
A critical thing to consider is how much replication lag will be incurred on any single-threaded replicas. That SQL thread will be blocked while the replica is running the change. If it’s a long-running change, that’s a lot of lag.
Another important thing to highlight is that you can’t pause a native online alter. Once it starts, you’re committed to completing it, or stopping it and waiting for what can be an expensive rollback.
Information about native online alters above was specific to InnoDB.
A newer engine, TokuDB, offers unique online alter functionality. The several available alters are completed in memory and then pushed down to storage as a part of regular Toku operations.
You just need to be careful about how you plan changes in Toku. Do changes one at a time; for example, don’t change names and data types at the same time.
If you need to do something like add a column and drop a column, separate those alters.
If you need to drop a column that has an index, drop the index online first, then drop the column online.
Changing the size of a basic datatype will be done online, but only if the column doesn’t have an index attached.
The most critical limitations of online alters in TokuDB are centered around performance.
Two things stand out: A table lock with a flush is done at the end of the change, so if there are many dirty pages to be flushed, that process can take several minutes. And, because Toku will push the changes down as part of its regular operations; if you have a very busy workload (or even a batch workload, if timed poorly), this background work can use a lot of system resources.
Take these limitations very seriously, especially in the presence of replication. Test your workload very well.
Adding indexes in TokuDB is a specific online alter case. It’s important to use the correct syntax, highlighted here.
Again, we will see a block with single-threaded replication, causing replication lag.
For another newer engine, RocksDB, all changes require a table copy, except creating/dropping indexes.
Community support may soon be saving the day here.
With Amazon RDS, traditional native online alter functionality is available.
In addition, Aurora Fast DDL allows online changes because of versioning. Consider overhead of these additional background processes.
Specific to RDS, if you have ephemeral storage, and you’re doing any change that requires temp space, the required temp space can fill your allocation.
Aurora Fast DDL is also limited in nonstandard cases, like on tables with partitions.
(Ivan takes over)
Okay. Online alters are pretty robust these days, with most standard table definitions and changes covered.
Alternate methods and tools exist to help with changes not covered by online alter functionality, and, for all changes, to allow for better insight and control.
We’ll cover two alternate methods briefly.
A common method used in the past often for large tables, or for replication scenarios requiring read replicas to be completely up-to-date, is to do a rolling schema update.
In this method, the replica is stopped and changes are made there. Then a failover is done in conjunction with the application change requiring the new schema (if the application code isn’t forward- and backward-compatible), the change is done on the old primary, and replication is restored.
The major limitation here is the time-consuming manual process, compounded in the presence of complex replication setups.
Any writes to the replica can cause serious data inconsistency when GTIDs are used, resulting in errant transactions that need to be manually resolved.
Here’s an example of how GTID modifies the rolling schema approach. If we are not careful we can end up with duplicate statements or broken slaves after we promote a new master
Just as a public service announcement, In case errant transactions occur, here’s how this would be resolved.
Another commonly used schema change process in the past was just to take complete downtime. The downside of downtime is… well, downtime, and it doesn’t work well in today 24x7 environments.
A “light” way to do this is to do changes during relative downtime. That is, do changes when the application is relatively quiet, to avoid performance degradation or replication lag.
Alternate tools were developed to address the limitations of online alters, before the native online DDL was quite robust enough.
These tools remain useful, however.
pt-online-schema-change is a well-known and widely used tool.
With this tool, a new table is created, rows are written in chunks, and concurrent changes are applied via triggers. At the end of the alter, a table rename is done.
pt-online-schema-change can be controlled in various ways when starting the change, including controlling the size of the chunks, and specifying which replicas will be monitored for replication lag.
When starting the tool, you can also specify the thresholds of variables related to workload and replication lag.
To avoid a major limitation, don’t drop the old table right after the table rename. Do this at a later time when load is quiet, to avoid stalls.
The use of triggers has been the most widely critiqued feature of pt-online-schema-change, that being a poorly optimized part of the database code.
Other limitations exist around foreign keys, but this can be configured to a certain degree.
Especially note the bug around foreign keys including an underscore in the name.
Another tool that is similar to pt-osc is Facebook’s OSC, created specifically for their environment, but open-sourced and available to all, so I think it’s interesting to look at.
Facebook osc works similarly to pt-osc. We create a table with the new structure and then add some triggers on the old table.
We start dumping rows in chunks from the source table and loading onto the new table, while replaying changes.
There is a checksum performed, doing a consistent snapshot at the beginning, and then a quick replay to fill the gap between shadow and existing table.
At the end, the old and new tables are swapped.
The most important thing to keep in mind is the tool is designed to run rolling schema updates, as changes do not propagate to the slaves.
The main difference with pt-osc is the change capture table that is written to by the triggers, rather than the target table itself, which allows the change replay to be asynchronous.
Also rather than insert/select as pt-osc does, the tool uses select into outfile and load data infile. This avoids the use of gap-lock mechanism thus reducing locking.
Since the tool relies on a file with a create table statement, you can easily integrate changes with your source control mechanism.
The main limitations of Facebook online schema change are no support for triggers and foreign keys.
Only one migration can be run at a time.
Python 2.7 support is tricky if you are running an older distribution.
Your application needs to be able to tolerate “table not found” errors for a short-period.
As mentioned before, the tool is meant to be run on each of your servers individually.
Issues with trigger-based tools led to the development of Github’s gh-ost, which is a triggerless online schema migration solution.
It is testable and provides the ability to pause changes, dynamic control/reconfiguration, auditing, and many operational perks.
Gh-ost works in the same way as other tools by copying the table rows in chunks, but rather than using triggers, gh-ost will attach itself as a virtual replica, and capture ongoing changes directly from the binary logs.
In addition to the benefits of not having triggers, gh-ost can be reconfigured on the fly by sending commands to a UNIX socket.
Gh-ost can attach to either the master or a slave, as long as log-slave-updates is set. An accurate progress counter, which requires a full scan of the table is also available to be offloaded to a slave.
gh-ost can operate directly on a slave, and also do a test migration. In that mode, the replication is left stopped so you can manually compare and checksum the old and new tables.
One of the available mechanisms for throttling is the usual check for threads connected/running. If you hit max-load, the operation is paused, while critical load is used to abort the change. There are some thresholds available to do a second check before aborting.
The throttling configuration can be set to check for slave lag; you can also specify which slaves are monitored.
Since there are no triggers, you can interactively pause an ongoing change by sending the command to the unix socket.
The dynamic reconfiguration is possible by sending the commands via unix socket. You can check the status and also query or modify the chunk size. The operation can also be throttled manually.
The cutover can be delayed by specifying a command line argument, and manually triggered on a low traffic period.
If the interactive commands don’t work because the -U flag is not available, be sure you have the correct version of netcat installed.
The main thing that prevents using gh-ost is If you have foreign keys or triggers.
You also need at least a slave running row-based replication, although gh-ost can flip the format for you if you let it.
The newer columns are also not supported yet, and if you have a master-master setup where you write to both masters at the same time that is also a problem.
Now we move on to some use cases, and let the battle begin!
RDS and Aurora take away many complexities of db administration, so you can think of your database as a black box
Unique to Aurora, consider that the underlying storage for nodes is shared.
Any online alter method or tool must take these two constraints into account.
Related, stopping and starting the replica is done through procedures, not directly in the database.
Let’s battle!
Given the differences in database administration statements, there are several complications.
In Amazon’s version of MySQL, either RDS MySQL or Aurora, privileges are limited. This often turns out to be an issue with operational tasks, and online schema changes are no exception.
Commands using “super” are disabled, processlist doesn’t give accessible IP information, filters are in place, read_only can’t be modified.
Also, binlogs aren’t enabled by default, but these can be configured by setting the backup retention appropriately.
With Amazon, gh-ost wins! The additional flexibility provided by dynamic reconfiguration, plus reduced overhead due to no triggers gives gh-ost the crown here, as long as you don’t have Foreign Keys
For Aurora, the only available option is to run against the master, as read replicas are effectively using the same underlying database files
Enable binlogs, and set binlog format to “row” in the parameter group.
For gh-ost, use assume RBR so it does not get stuck on trying to set the binlog format, and use allow on master (specify the master in the host flag).
And here’s an example of the gh-ost statement.
For rds the recommended way is to run the tool against a replica. We also need to manually give the master external ip address
And here’s an example of the gh-ost statement.
Next contestant up is Tungsten.
Tungsten uses an external replication process, as a replacement to the native built-in replication.
Notably when doing operational tasks such as online schema changes, replication status via “show slave status” is not seen inside the database.
Let’s battle!
The complications with native online DDL remain with Tungsten, and additional complications arise with external tools.
With pt-osc, because of the lack of replication information, the replicas have to be specified manually via the DSN table. And because of the triggers used in pt-osc, binlog format matters. With statement-based replication, pt-osc works out of the box. With row-based replication, you’ll need to use a Tungsten plugin. Mixed replication is not recommended.
With gh-ost, you can’t detect replication because of the lack of status within the database, and you can’t start or stop replication as well.
For Tungsten, gh-ost wins. It’s the least difficult to configure correctly for Tungsten, and allows more control than pt-osc.
Gh-ost can be configured with Tungsten to run against the master or a replica. Here’s an overview of the process when running against a replica, which is what we’ll recommend here.
Some important configuration notes: You will need to set log-slave-updates in the Tungsten configuration. That’s in the Tungsten configuration, distinct from the database configuration.
In the gh-ost statement, specify the master with assume-master-host, and use the tungsten flag.
Gh-ost requires binlog format to be set to row wherever it is running, so in this case, you’ll set the replica’s binlog-format to row. If this isn’t already set, you will need to restart the Tungsten replicator as well.
And here’s an example of the gh-ost configuration.
Up next, Galera.
Galera has built-in methods for making schema changes, but completely online DDL is not one of them.
You can choose total order isolation, but this blocks all writes to the cluster. Without taking application downtime, we don’t want this.
You can use rolling schema updates, removing one node at a time as described earlier in alternate methods. This method incurs the same complications as described earlier: it is a manual process and requires the application to be forward- and backward-compatible.
Maybe we should use an alternate tool.
Let’s battle!
Alternate tools have their own complications, but do a better job with online alters than the built-in methods.
With pt-osc, only innodb is supported, and the online schema upgrade configuration has to be set to total order isolation, so there will be minimal blocking.
With gh-ost, there’s no official support, so you can run into untested or undocumented issues. In Galera 5.6, there was a workaround to use the two-step cutover, but this does not work in Galera 5.7.
And pt-osc wins! It is the more straight forward option for Galera cluster.
To use pt-osc with Galera, set the online schema upgrade setting to total order isolation. This does not cause downtime, but indicates that Galera will process the changes on all nodes.
Other than that, no special arguments are required. You won’t see any replication lag.
(Valerie takes over)
Multisource replication can introduce several operational complexities, online schema change among them.
Here is what multisource replication looks like. It’s often found in environments running MariaDB, because it has been available for a while.
The main disadvantage to using native online alter with this setup is the space on the replica, which is already at a premium. If the change is not metadata-only, then you can get a surprise in disk utilization on the replica.
Let’s battle!
Gh-ost wins here, for the ability to control the change in what can be a complex replication environment. If we run into any surprises with disk utilization or otherwise, we can pause and sort it out.
Run gh-ost on the replica, specifying assume-master-host so gh-ost knows which master the table change should be run on.
Daisy-chained replication introduces similar complexities with operational tasks.
Here’s an example of daisy-chained replication. The middle replica has log-slave-updates turned on, and changes are sent down the line to another replica.
Let’s battle!
Because gh-ost is unwilling to set RBR when it detects a slave, you would need to set binlog_format=ROW on the intermediate slave beforehand.
To avoid that additional change, pt-osc wins here for changes that aren’t metadata only.
Run pt online schema change against the master, of course.
Daisy-chained replication can incur replication lag on a downstream replica as a matter of course. If that’s the case in your environment, exclude the downstream replica in any replication lag checks in the pt-osc configuration to avoid unnecessary pauses in the change.
Now for two use cases that are fairly common in our experience: very large tables, and very busy tables.
The first we’ll tackle is very large tables.
What is very large? Let’s define that somewhat vaguely as having many rows of a wide table. That means: it depends. But you know very large when you see it in your environment.
(We will exclude for the moment the once-in-a-lifetime very-very-very large table situation, for which there’s no canned recommendation.)
Let’s battle!
All the things that can complicate a schema change become exacerbated with a very large table. Replication lag becomes laggier. Expensive rollbacks become unaffordable. Disk space headroom is likely not available. You’re getting major resource contention.
In this case, pt-online-schema-change wins, because in both benchmarks and real world cases it has been proven to be faster.
When you’re considering all the complications above, reducing the time a change runs by even 12 hours can be positive.
Be absolutely sure you have the disk space required before starting, though (see very-very-very large table disclaimer above).
Again, test in your environment with your workload. Experiment with setting flags in a test environment (you won’t be able to change those on the fly).
Set high timeouts and retries to allow the tool to work without constant throttling.
Here are some pt-osc typical settings for a very large table.
The other common scenario is a very busy table. Unlike the case with very large tables, recommendations here can cover very-very-very busy tables as well.
What is “busy”? Busy tables are those with many changes in a short period of time. Think of an e-commerce site on Black Friday.
Busy tables can be large tables, and so this use case is often intertwined with the previous one. There’s a particular feature of busy tables to take into consideration, though, when making a recommendation.
Let’s battle!
Again, the typical complications with online changes become exacerbated with very busy tables.
There is an additional problem with native online DDL. If the change buffer used fills up, the operation will fail. If you configure this to be too high, the table lock at the end of the change will be extended.
In the case of very busy tables, gh-ost wins. This is even true when the table is very large, because when a table is very busy, there are typically times when it is more busy (think about the workload in the daytime versus middle-of-the-night), and you will want the ability to pause a change on a very busy table during a very-very-very busy time. So if you have a change against a very large table that will require pausing during a very busy time during the day, you’ll want to pause it.
Plan ahead, and use your testing to predict how long this change will take. Time it so that the cutover is done manually after the change is complete, to time it with any application changes and to avoid doing that during a very-very-very busy time.
REDUNDANT -- fix Valerie
Again, external tools are going to work better than native online DDL.
Configure the change to allow for small chunks. In this case, you’ll want low timeouts to avoid performance degradation, and configure for retries.
As mentioned above, schedule the cutover to happen manually at a time when there is low traffic workload.
And here’s how you would configure gh-ost to run a change with a very busy table.
We’ve refereed a battle for several setups you’re likely to run into. What we haven’t talked about is a change that isn’t going to require a table copy.
A handful of changes are only going to change the metadata. It’s useful to check documentation for your version of MariaDB or MySQL to see if a change is done only to that metadata, before engaging one of the complex processes described above.
A common example is dropping an index. This works on metadata only in most recent versions of both MariaDB and MySQL.
Let’s battle! We’ll say our battle involves a change to drop an index.
And in this case, native online DDL wins.
So keep an eye on release notes and documentation. Database products continue to mature, and continue to allow more online features.
OK, that wraps up our online schema change battle.
Are there any questions?