Successfully reported this slideshow.
Your SlideShare is downloading. ×

OpenSUSE Conf 2020 MySQL Clone

More Related Content

Related Books

Free with a 30 day trial from Scribd

See all

OpenSUSE Conf 2020 MySQL Clone

  1. 1. Georgi Kodinov, MySQL team lead MySQL Clone A better way to migrate databases
  2. 2. Georgi “Joro” Kodinov • Doing MySQL server development since 2006 • Based in Plovdiv, Bulgaria • Works on security features, MySQL wire protocol, plugins • Speaker at OpenSUSE conferences since 2013 2 © 2020 Oracle
  3. 3. What is MySQL Clone ? A simple and efficient way to create a MySQL instance replica © 2020 Oracle3
  4. 4. WhyYet Another Method ? • It’s easy to use: a single command • It’s a data snapshot with consistent GTID and binary log position • Data can be cloned locally or from remote • It’s resumable ! © 2020 Oracle4
  5. 5. It’s FASTER Too ! © 2020 Oracle5 https://mydbops.wordpress.com/2019/11/14/ mysql-clone-plugin-speed-test/ 2 servers CPU: 2 Cores RAM: 4 GB DISK: 200 GB SSD approx. 122.40 GB data from sysbench
  6. 6. Clone: Use Cases © 2020 Oracle6
  7. 7. Provision a new replication node © 2020 Oracle7 Source Replica Replica2
  8. 8. Re-provision a returning node © 2020 Oracle8 BinlogSource ReplicaData ??
  9. 9. Clone: Operation © 2020 Oracle9
  10. 10. Terminology © 2020 Oracle10 Donor RecipientData
  11. 11. The big picture © 2020 Oracle11 Client Network Network Clone Plugin Innodb SE-2 [Future] Clone Plugin Innodb SE-2 [Future] Recipient Server Donor Server Clone SQL mysql> CLONE INSTANCE FROM clone_user@donor.host.com:3306 IDENTIFIED BY "clone_password"; Connect
  12. 12. Prerequisites on the Donor • Install the plugin: INSTALL PLUGIN clone SONAME 'mysql_clone.so'; • Grant BACKUP_ADMIN to a user account CREATE USER foo …; GRANT BACKUP_ADMIN TO foo; © 2020 Oracle12
  13. 13. Prerequisites on the Recipient • Install the plugin: INSTALL PLUGIN clone SONAME 'mysql_clone.so'; • Grant CLONE_ADMIN to a user account CREATE USER bar …; GRANT CLONE_ADMIN TO bar; • Allow the donor host SET GLOBAL clone_valid_donor_list='donor_computer:3306'; © 2020 Oracle13
  14. 14. On the recipient, as “bar” CLONE INSTANCE FROM foo@donor_computer:3306 IDENTIFIED BY 'foo_pwd‘ DATA DIRECTORY = 'clone_dir' REQUIRE SSL © 2020 Oracle14
  15. 15. Clone: What Has Just Happened ? © 2020 Oracle15
  16. 16. Clone Stages © 2020 Oracle16 Init Data Files File Copy Modified Page IDs Page Copy Redo Logs Redo Copy Done
  17. 17. Clone: Network Failure Recovery © 2020 Oracle17 Donor RecipientData Less than 5 min
  18. 18. Clone Monitoring: performance_schema.clone_status mysql> SELECT STATE FROM performance_schema.clone_status; +-----------+ | STATE | +-----------+ | Completed | +-----------+ © 2020 Oracle18
  19. 19. Clone Monitoring: performance_schema.clone_progress mysql> SELECT STAGE, STATE, END_TIME FROM performance_schema.clone_progress; +-----------+-----------+----------------------------+ | stage | state | end_time | +-----------+-----------+----------------------------+ | DROP DATA | Completed | 2020-09-27 14:45:43.141261 | | FILE COPY | Completed | 2020-09-27 14:45:44.457572 | | PAGE COPY | Completed | 2020-09-27 14:45:44.577330 | | REDO COPY | Completed | 2020-09-27 14:45:44.679570 | | FILE SYNC | Completed | 2020-09-27 14:45:44.918547 | | RESTART | Completed | 2020-09-27 14:45:48.583565 | | RECOVERY | Completed | 2020-09-27 14:45:49.626595 | +-----------+-----------+----------------------------+ © 2020 Oracle19
  20. 20. Performance And Resource Use Control © 2020 Oracle20 Variable Type Meaning clone_max_concurrency Int (Maximum) number of cloning threads clone_autotune_concurrency Bool Enable dynamic spawning of cloning threads clone_max_network_bandwidth Int, Mbps (recipient only) limit network usage clone_max_data_bandwidth Int, Mbps (recipient only) limit clone transfer rate
  21. 21. Clone Limitations • Does not clone configuration values • Does not clone binary logs • Supports only InnoDB • Blocks all concurrent DDLs on the Donor • Will FULLY override recipient data • Only to the SAME server version • Only on the same OS and platform • Doesn’t work over the X protocol © 2020 Oracle21
  22. 22. Further Reading • Reference Manual • https://dev.mysql.com/doc/refman/8.0/en/clone-plugin.html • Blogs • https://mysqlserverteam.com/clone-create-mysql-instance-replica/ • https://mysqlhighavailability.com/mysql-innodb-cluster-automatic-node-provisioning • https://mysqlhighavailability.com/a-breakthrough-in-usability-automatic-node-provisioning • https://mysqlhighavailability.com/automatic-provisioning-in-group-replication • InnoDB cluster video • https://lefred.be/content/migrate-from-a-single-mysql-instance-to-mysql-innodb-cluster- using-clone-plugin/ © 2020 Oracle22
  23. 23. Thank you ! Georgi Kodinov, Software Development Director Georgi.Kodinov@oracle.com https://www.slideshare.net/GeorgiKodinov © 2020 Oracle23

Editor's Notes

  • * The node is far behind and binary log sync would take long time.
    * Binary logs are not available to sync
  • Recipient: Server instance, where the clone statement is executed.
    Donor: The source server instance from which the data is going to be cloned. The Donor collects all its data and metadata stored in the InnoDB storage engine at a consistent snapshot to replace the data in Recipient.
    After successful execution of a CLONE SQL statement the Recipient server is restarted automatically. The restart involves recovery. Once restarted, recipient is a clone of donor server!
  • clone_ssl_ca/clone_ssl_cert/clone_ssl_key
  • Donor dynamic snapshot is transferred in three stages.
    File Copy
    Page Copy
    Redo Copy
  • Recipient: Server instance, where the clone statement is executed.
    Donor: The source server instance from which the data is going to be cloned. The Donor collects all its data and metadata stored in the InnoDB storage engine at a consistent snapshot to replace the data in Recipient.
    After successful execution of a CLONE SQL statement the Recipient server is restarted automatically. The restart involves recovery. Once restarted, recipient is a clone of donor server!
  • One or zero rows
    ID
    A unique cloning operation identifier in the current MySQL server instance.
    PID
    Process list ID of the session executing the cloning operation.
    STATE
    Current state of the cloning operation. Values include Not Started, In Progress, Completed, and Failed.
    BEGIN_TIME
    A timestamp in 'YYYY-MM-DD hh:mm:ss[.fraction]' format that shows when the cloning operation started.
    END_TIME
    A timestamp in 'YYYY-MM-DD hh:mm:ss[.fraction]' format that shows when the cloning operation finished. Reports NULL if the operation has not ended.
    SOURCE
    The donor MySQL server address in 'HOST:PORT' format. The column displays 'LOCAL INSTANCE' for a local cloning operation.
    DESTINATION
    The directory being cloned to.
    ERROR_NO
    The error number reported for a failed cloning operation.
    ERROR_MESSAGE
    The error message string for a failed cloning operation.
    BINLOG_FILE
    The name of the binary log file up to which data is cloned.
    BINLOG_POSITION
    The binary log file offset up to which data is cloned.
    GTID_EXECUTED
    The GTID value for the last cloned transaction.
  • ID
    A unique cloning operation identifier in the current MySQL server instance.
    STAGE
    The name of the current cloning stage. Stages include DROP DATA, FILE COPY, PAGE_COPY, REDO_COPY, FILE_SYNC, RESTART, and RECOVERY.
    STATE
    The current state of the cloning stage. States include Not Started, In Progress, and Completed.
    BEGIN_TIME
    A timestamp in 'YYYY-MM-DD hh:mm:ss[.fraction]' format that shows when the cloning stage started. Reports NULL if the stage has not started.
    END_TIME
    A timestamp in 'YYYY-MM-DD hh:mm:ss[.fraction]' format that shows when the cloning stage finished. Reports NULL if the stage has not ended.
    THREADS
    The number of concurrent threads used in the stage.
    ESTIMATE
    The estimated amount of data for the current stage, in bytes.
    DATA
    The amount of data transferred in current state, in bytes.
    NETWORK
    The amount of network data transferred in the current state, in bytes.
    DATA_SPEED
    The current actual speed of data transfer, in bytes per second. This value may differ from the requested maximum data transfer rate defined by clone_max_data_bandwidth.
    NETWORK_SPEED
    The current speed of network transfer in bytes per second.
  • Clone plugin uses multiple connections from recipient to donor server.
    - clone_max_concurrency limits maximum connections
    Clone operation starts with 1 connection thread and threads are increased in steps. Auto tuning checks and stop spawning new connections if no significant performance improvement is noticed.
    - clone_autotune_concurrency
    If cpu, i/o, Network resource is saturated then increasing connections/threads could degrade performance and could also impact donor performance. Auto tuning helps here.

×