This tutorial covers all parallel replication implementation in MariaDB 10.0 and 10.1 and MySQL 5.6, 5.7 and 8.0 (including how it works in Group Replication).
MySQL and MariaDB have different types of parallel replication. In this tutorial, we present the different implementations that allow us to understand their limitations and tuning parameters. We cover how to make parallel replication faster and what to avoid for maximizing its benefits. We also present tests from Booking.com workloads.
Some of the subjects that are covered are group commit and optimistic parallel replication in MariaDB, the parallelism interval of MySQL and its Write Set optimization, and the ?slowing down the master to speed up the slave? optimization.
After this tutorial, you will know everything you need to implement and tune parallel replication in your environment. But more importantly, we will show how you can test parallel replication benefit in a non-disruptive way before deployment.
24시간 365일 서비스를 위한 MySQL DB 이중화.
MySQL 이중화 방안들에 대해 알아보고 운영하면서 겪은 고민들을 이야기해 봅니다.
목차
1. DB 이중화 필요성
2. 이중화 방안
- HW 이중화
- MySQL Replication 이중화
3. 이중화 운영 장애
4. DNS와 VIP
5. MySQL 이중화 솔루션 비교
대상
- MySQL을 서비스하고 있는 인프라 담당자
- MySQL 이중화에 관심 있는 개발자
This tutorial covers all parallel replication implementation in MariaDB 10.0 and 10.1 and MySQL 5.6, 5.7 and 8.0 (including how it works in Group Replication).
MySQL and MariaDB have different types of parallel replication. In this tutorial, we present the different implementations that allow us to understand their limitations and tuning parameters. We cover how to make parallel replication faster and what to avoid for maximizing its benefits. We also present tests from Booking.com workloads.
Some of the subjects that are covered are group commit and optimistic parallel replication in MariaDB, the parallelism interval of MySQL and its Write Set optimization, and the ?slowing down the master to speed up the slave? optimization.
After this tutorial, you will know everything you need to implement and tune parallel replication in your environment. But more importantly, we will show how you can test parallel replication benefit in a non-disruptive way before deployment.
24시간 365일 서비스를 위한 MySQL DB 이중화.
MySQL 이중화 방안들에 대해 알아보고 운영하면서 겪은 고민들을 이야기해 봅니다.
목차
1. DB 이중화 필요성
2. 이중화 방안
- HW 이중화
- MySQL Replication 이중화
3. 이중화 운영 장애
4. DNS와 VIP
5. MySQL 이중화 솔루션 비교
대상
- MySQL을 서비스하고 있는 인프라 담당자
- MySQL 이중화에 관심 있는 개발자
Almost Perfect Service Discovery and Failover with ProxySQL and OrchestratorJean-François Gagné
Of course there is no such thing as perfect service discovery, and we will see why in the talk. However, the way ProxySQL is deployed in this case minimizes the risk for split-brains, and this is why I qualify it as almost perfect. But let’s step back a little...
MySQL alone is not a high availability solution. To provide resilience to primary failure, other components need to be integrated with MySQL. At MessageBird, these additional components are ProxySQL and Orchestrator. In this talk, we describe how ProxySQL is architectured to provide close to perfect Service Discovery and how this, combined with Orchestrator, allows for automatic failover. The talk presents the details of the integration of MySQL, ProxySQL and Orchestrator in Google Cloud (and it would be easy to re-implement a similar architecture at other cloud vendors or on-premises). We will also cover lessons learned for the 2 years this architecture has been in production. Come to this talk to learn more about MySQL high availability, ProxySQL and Orchestrator.
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)Jean-François Gagné
To get better replication speed and less lag, MySQL implements parallel replication in the same schema, also known as LOGICAL_CLOCK. But fully benefiting from this feature is not as simple as just enabling it.
In this talk, I explain in detail how this feature works. I also cover how to optimize parallel replication and the improvements made in MySQL 8.0 and back-ported in 5.7 (Write Sets), greatly improving the potential for parallel execution on replicas (but needing RBR).
Come to this talk to get all the details about MySQL 5.7 and 8.0 Parallel Replication.
MySQL Database Architectures - InnoDB ReplicaSet & ClusterKenny Gryp
Following MySQL InnoDB Cluster as our first, fully integrated MySQL High Availability solution based on Group Replication, MySQL Shell 8.0.19 includes MySQL InnoDB ReplicaSet which delivers another complete solution, this time based on MySQL Replication.
The basic idea for InnoDB ReplicaSet is to do the same for classic MySQL Replication as InnoDB Cluster did for Group Replication. We take a strong technology that is very powerful but can be complex, and provide an easy-to-use AdminAPI for it in the MySQL Shell.
In just a few easy to use Shell commands, a MySQL Replication database architecture can be configured from scratch including:
Data provisioning using MySQL CLONE, Setting up replication,
Performing manual switchover/failover.
Almost Perfect Service Discovery and Failover with ProxySQL and OrchestratorJean-François Gagné
Of course there is no such thing as perfect service discovery, and we will see why in the talk. However, the way ProxySQL is deployed in this case minimizes the risk for split-brains, and this is why I qualify it as almost perfect. But let’s step back a little...
MySQL alone is not a high availability solution. To provide resilience to primary failure, other components need to be integrated with MySQL. At MessageBird, these additional components are ProxySQL and Orchestrator. In this talk, we describe how ProxySQL is architectured to provide close to perfect Service Discovery and how this, combined with Orchestrator, allows for automatic failover. The talk presents the details of the integration of MySQL, ProxySQL and Orchestrator in Google Cloud (and it would be easy to re-implement a similar architecture at other cloud vendors or on-premises). We will also cover lessons learned for the 2 years this architecture has been in production. Come to this talk to learn more about MySQL high availability, ProxySQL and Orchestrator.
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)Jean-François Gagné
To get better replication speed and less lag, MySQL implements parallel replication in the same schema, also known as LOGICAL_CLOCK. But fully benefiting from this feature is not as simple as just enabling it.
In this talk, I explain in detail how this feature works. I also cover how to optimize parallel replication and the improvements made in MySQL 8.0 and back-ported in 5.7 (Write Sets), greatly improving the potential for parallel execution on replicas (but needing RBR).
Come to this talk to get all the details about MySQL 5.7 and 8.0 Parallel Replication.
MySQL Database Architectures - InnoDB ReplicaSet & ClusterKenny Gryp
Following MySQL InnoDB Cluster as our first, fully integrated MySQL High Availability solution based on Group Replication, MySQL Shell 8.0.19 includes MySQL InnoDB ReplicaSet which delivers another complete solution, this time based on MySQL Replication.
The basic idea for InnoDB ReplicaSet is to do the same for classic MySQL Replication as InnoDB Cluster did for Group Replication. We take a strong technology that is very powerful but can be complex, and provide an easy-to-use AdminAPI for it in the MySQL Shell.
In just a few easy to use Shell commands, a MySQL Replication database architecture can be configured from scratch including:
Data provisioning using MySQL CLONE, Setting up replication,
Performing manual switchover/failover.
【DLゼミ】XFeat: Accelerated Features for Lightweight Image Matchingharmonylab
公開URL:https://arxiv.org/pdf/2404.19174
出典:Guilherme Potje, Felipe Cadar, Andre Araujo, Renato Martins, Erickson R. ascimento: XFeat: Accelerated Features for Lightweight Image Matching, Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
概要:リソース効率に優れた特徴点マッチングのための軽量なアーキテクチャ「XFeat(Accelerated Features)」を提案します。手法は、局所的な特徴点の検出、抽出、マッチングのための畳み込みニューラルネットワークの基本的な設計を再検討します。特に、リソースが限られたデバイス向けに迅速かつ堅牢なアルゴリズムが必要とされるため、解像度を可能な限り高く保ちながら、ネットワークのチャネル数を制限します。さらに、スパース下でのマッチングを選択できる設計となっており、ナビゲーションやARなどのアプリケーションに適しています。XFeatは、高速かつ同等以上の精度を実現し、一般的なラップトップのCPU上でリアルタイムで動作します。
セル生産方式におけるロボットの活用には様々な問題があるが,その一つとして 3 体以上の物体の組み立てが挙げられる.一般に,複数物体を同時に組み立てる際は,対象の部品をそれぞれロボットアームまたは治具でそれぞれ独立に保持することで組み立てを遂行すると考えられる.ただし,この方法ではロボットアームや治具を部品数と同じ数だけ必要とし,部品数が多いほどコスト面や設置スペースの関係で無駄が多くなる.この課題に対して音𣷓らは組み立て対象物に働く接触力等の解析により,治具等で固定されていない対象物が組み立て作業中に運動しにくい状態となる条件を求めた.すなわち,環境中の非把持対象物のロバスト性を考慮して,組み立て作業条件を検討している.本研究ではこの方策に基づいて,複数物体の組み立て作業を単腕マニピュレータで実行することを目的とする.このとき,対象物のロバスト性を考慮することで,仮組状態の複数物体を同時に扱う手法を提案する.作業対象としてパイプジョイントの組み立てを挙げ,簡易な道具を用いることで単腕マニピュレータで複数物体を同時に把持できることを示す.さらに,作業成功率の向上のために RGB-D カメラを用いた物体の位置検出に基づくロボット制御及び動作計画を実装する.
This paper discusses assembly operations using a single manipulator and a parallel gripper to simultaneously
grasp multiple objects and hold the group of temporarily assembled objects. Multiple robots and jigs generally operate
assembly tasks by constraining the target objects mechanically or geometrically to prevent them from moving. It is
necessary to analyze the physical interaction between the objects for such constraints to achieve the tasks with a single
gripper. In this paper, we focus on assembling pipe joints as an example and discuss constraining the motion of the
objects. Our demonstration shows that a simple tool can facilitate holding multiple objects with a single gripper.
12. Multi-Source Replication
mysql> CHANGE MASTER TO master_host = 'xxx', master_port = xxx, m
aster_user = 'xxx', master_password = 'xxx', master_auto_positio
n = 1 FOR CHANNEL 'site_1';
mysql> SHOW SLAVE STATUSG
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: xxx
Master_User: xxx
Master_Port: xxx
..
Channel_Name: site_1
11/84
13. Multi-Source Replicationの監視
SHOW SLAVE STATUS だと全チャンネル出⼒されるので、今ま
でのがそのまま使えない
SHOW SLAVE STATUS FOR CHANNNEL 'site_1', SHOW SLAVE STATUS
FOR CHANNEL 'site_2', .. と分割するか
-
そういえば5.7からperformance̲schemaにレプリケーション関連の
テーブル追加されたよねって思ったけど
SELECT iothread.channel_name, iothread.service_state AS io_thread,
sqlthread.service_state AS sql_thread FROM
performance_schema.replication_connection_status AS iothread JOIN
performance_schema.replication_applier_status_by_worker AS sqlthread で
*_Running: Yes 的なところは取れるんだけど、 Seconds_Behind_Master が取れな
い。。
“SHOW SLAVE STATUS Information Not In the Replication Tables”
MySQL :: MySQL 5.7 Reference Manual :: 23.9.11 Performance Schema
Replication Tables
Oh..
-
12/84
15. ⽴ちはだかるConnector/Jの壁
MySQL Bugs: #80631: ResultSet.getString return
garbled result with json type data
Connector/Jだとマルチバイト⽂字が化ける-
まだ直ってない-
マルチバイトもテストしてくれよおおおお-
14/84
20. InnoDB FTS + mecab-ipadic-neologd
InnoDB FTSは 既に⾊々踏み抜いておいたので 今のところ
問題なし
MySQL Bugs: #76120 (アクセス権なし)-
MySQL Bugs: #76121: Warning 1235, “FTS auxiliary tables
will not be flushed” is printed twice.
-
MySQL Bugs: #76139 (アクセス権なし)-
MySQL Bugs: #76164: InnoDB FTS with MeCab parser prints
empty error message
-
MySQL Bugs: #80755 (アクセス権なし)-
MySQL Bugs: #80760: Reverse Engineer fails to load table
which has “WITH PARSER” clause
-
19/84
32. ボコられ案(5-1-2)
AP
[Not supported by viewer] Connector/J
Master Slave
mysqlfabric
Monitor/Demote
Monitor/Promote
Lookup Group Query
Routing
Routing
AP
AP Connector/J
31/84
62. MySQL Fabricの現実
もうずっと⻑いことMySQL WorkbenchからMySQL Fabric
に接続できない
⼀時期セミナーで「MySQL WorkbenchからMySQL Fabricが管理で
きます︕」と謳っていた時期があったのに…
-
MySQL Bugs: #74894: Failure to connect to MySQL Fabric
from a windows installed workbench.
-
とはいえ慣れれば mysqlfabric コマンドでも何とかなる
けど、監視⽤途にはパースが超めんどくさいので、昔みたいにJSONで返してくれるオ
プションも欲しかった。。
-
61/84
63. MySQL Fabricの現実
MySQL Bugs: #73206: MySQL Fabric should report a
warning when MySQL Event Scheduler is disabled
MySQL Bugs: #74894: Failure to connect to MySQL
Fabric from a windows installed workbench.
MySQL Bugs: #81557: MySQL Fabric uses wrong
argument of MAKETIME in prune̲log Event
MySQL Bugs: #81558: prune̲log event doesnʼt use
any index
MySQL Bugs: #81559: Incorrect WHERE clause in
dump̲servers fanction
62/84
66. MySQL Routerの現実
APから mysqlrouter がESTAB1.
mysqlrouter と mysqlfabric は非同期通信でキャッシュを更
新
2.
mysqlrouter はキャッシュを⾒てMySQL ServerとESTAB3.
AP => mysqlrouter => MySQL Server とNATされる。遅
延は10usくらい。
4.
何故か mysqlfabric からキャッシュの更新通知が⾏って
も、 mysqlrouter => MySQL ServerのESTABが 切れない
5.
65/84
67. MySQL Routerの現実
シングルスレッドで、パケットを全てルーティングする
(NATな動き)ので、1万QPSとか叩くと mysqlrouter がボ
トルネックになって詰まる
それくらいの規模になったら複数の mysqlrouter プロセスを上げるし
かないけど
-
そんなトラフィックが来る予定はない
-
mysqlrouter の max_connections を1000以上にするとクラッシュす
るらしい
MySQL Bugs: #80260: MySQL Router is down with more than 1000
concurrent connections
-
66/84
78. (別の環境含め) MySQL 5.7でやったこと
SET GLOBAL innodb_buffer_pool_size = .. 経験済み
そこまで悪いものでもなかった (⼼臓には悪かった)-
MySQL Bugs: #77564: SIGABRT during resizing the InnoDB
Buffer Pool Online with memory full condition
-
sync_binlog= 1 でも5.6ほどひどくない (気がする)
sys スキーマ美味しいです
5.6にもガリガリインストールしてるから余計ありがたみはない-
77/84
79. (別の環境含め) MySQL 5.7でやったこと
むしろ既存の5.6をアップグレードした5.7でオンライン
gtid_mode= ON に移⾏できた。うれしい。
暗黙のテンポラリーテーブル
MyISAMにしてる( internal_tmp_disk_storage_engine= MyISAM )-
performance_schema_*_size とか
performance_schema_*_instances のデフォルトがautosize
になってるので、テキトーな値を秘伝のタレに追加
でないと運⽤中に思った以上にメモリー使⽤量が増えていく
で、 SET GLOBAL innodb_buffer_pool_size = .. でちょっと減らした。。
SHOW ENGINE performance_schema STATUS で⾒られるよ
-
78/84