Easy MySQL Replication Setup and Troubleshooting

1. Easy MySQL Replication Setup and Troubleshooting Bob Burgess Salesforce ExactTarget Marketing Cloud (Radian6)

2. • Introduction • Description • Configuration Items • Commands • Setup • Beyond the Basics • Troubleshooting Faults • Gotchas • Closing

4. Slides • I will post these slides on the conference site and slideshare • No need to copy them down

5. Who Am I? • MySQL user since 2007 • Radian6: Social Media Monitoring • Now part of Salesforce ExactTarget Marketing Cloud • Large Data Set • Dozens of Databases • Replication is CRUCIAL for us

6. Why Am I Doing This? • Replication isn’t hard • Show simplest replication case • Share how we deal with common problems

7. Our First Replication • Set up a separate database for running reports, to separate back-end from user-facing activity • Alexander Rubin of MySQL Support in 2008

8. What is “Replication”?

9. Why Replication? • Database Failure • Share Read Traffic (Scale-Out) • Reporting/Analytics Databases • Network Load • Backups

10. Replication Compared • DRBD – Block-level – Replica is unusable until switched to • Galera / Percona Cluster – More complicated – Not WAN-Friendly for High Transaction Rate • Tungsten Replicator • Others..?

12. Replication Described Master Replica

13. Replication Described Master table1 table2 mysql Update table1 set... Insert into table2... Binlogs: • Events in commit order • Contains timestamps and auto-increment IDs client Replica table1 table2 IO Thread SQL Thread Update table1 set... Insert into table2... mysql

15. Replication Described log_bin=/data/log/mysql-bin Config items Master table1 table2 mysql Update table1 set... Insert into table2... client Replica table1 table2 IO Thread SQL Thread Update table1 set... Insert into table2... mysql

16. Replication Described server_id=1 server_id=2 Config items (1) Master table1 table2 mysql Update table1 set... Insert into table2... client Replica (2) table1 table2 IO Thread SQL Thread Update table1 set... Insert into table2... mysql

18. Replication Commands • change master to • start slave • stop slave • reset slave • show slave statusG • show master status • mysqlbinlog • grant replication slave

19. Replication Commands change master to... Which server? Port? Credentials? Log file & position? (1) Master table1 table2 mysql client Replica (2) table1 table2 IO Thread SQL Thread Update table1 set... Insert into table2... mysql Update table1 set... Insert into table2...

20. Replication Commands • change master to... Which server? Port? Credentials? Log file & position? • change master to master_host=‘Master’, master_port=3306, master_user=‘repl’, master_password=‘pass’, master_log_file=‘mysql_bin.000001’, master_log_pos=4563744;

21. Replication Commands (1) Master table1 table2 mysql Update table1 set... Insert into table2... client Replica (2) table1 table2 mysql

22. Replication Commands change master to master_host=‘Master’, master_port=3306, master_user=‘repl’, master_password=‘pass’, master_log_file=‘mysql_bin.000001’, master_log_pos=4563744; (1) Master table1 table2 mysql Update table1 set... Insert into table2... client Replica (2) table1 table2 IO Thread Update table1 set... Insert into table2... mysql

23. Replication Commands start slave; (1) Master table1 table2 mysql Update table1 set... Insert into table2... client Replica (2) table1 table2 IO Thread SQL Thread Update table1 set... Insert into table2... mysql

24. Replication Commands stop slave; (1) Master table1 table2 mysql Update table1 set... Insert into table2... client Replica (2) table1 table2 Update table1 set... Insert into table2... mysql

25. Replication Commands reset slave; (1) Master table1 table2 mysql Update table1 set... Insert into table2... client Replica (2) table1 table2 mysql

27. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1

28. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1 What it’s doing

29. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1 Where it’s connected

30. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1 Where the IO Thread has read up to

31. Master table1 table2 mysql Update table1 set... Insert into table2... client Replica table1 table2 IO Thread SQL Thread Update table1 set... Insert into table2... mysql Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843

32. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1 Where the SQL Thread has executed up to (in relaylog)

33. Master table1 table2 mysql Update table1 set... Insert into table2... client Replica table1 table2 IO Thread SQL Thread Update table1 set... Insert into table2... mysql Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548

34. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1 Where the SQL Thread has executed up to (corresponding position in Master’s binlog)

35. Master table1 table2 mysql Update table1 set... Insert into table2... client Replica table1 table2 IO Thread SQL Thread Update table1 set... Insert into table2... mysql Relay_Master_Log_File: mysql-bin.000001 Exec_Master_Log_Pos: 23456234

36. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1 If the IO and SQL Threads are running

37. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1 Replication Filters in effect

38. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1 Replication Lag (how long ago was timestamp of latest-processed query?) Can be “NULL”

39. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1 Replication Error (SQL Thread)

40. Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1 Replication Error (IO Thread)

42. Basic Setup Walk-Through • Unique server_id values my.cnf on master [mysqld] server_id=1 my.cnf on replica [mysqld] server_id=2

43. Basic Setup Walk-Through • Replication User On Master: grant replication slave on *.* to ‘repl’ identified by ‘pass’;

44. Basic Setup Walk-Through • Point to the correct binlog position • On master: show master status; ...once both databases are equal • On Replica: change master to ...

45. Basic Setup Walk-Through • Start it up • On replica: start slave; • Check it • On replica: show slave statusG

46. And that’s it.

48. Limiting Tables or Schemas • If you don’t need all the tables • In my.cnf: replicate-do-table=schema.tableName replicate-skip-table=schema.otherTable • If you don’t need all the schemas • In my.cnf replication-do-db=schemaOne replication_skip_db=schemaTwo

49. Limiting Tables replicate-do-table=schema.tableName (1) Master table1 table2 mysql Update table1 set... Insert into table2... client Replica (2) table1 table2 IO Thread SQL Thread with filter Update table1 set... Insert into table2... mysql

50. Limiting Columns or Rows • If you don’t need all columns • Subset of table for reporting or analytics

51. Limiting Columns or Rows • Original table create table x (a int primary key, b int, c int) engine=InnoDB; • Blackhole table on replica like master table create table x (a int, b int, c int) engine=blackhole; • Target table on replica create table y (a int primary key, b int) engine=InnoDB;

52. Limiting Columns or Rows • Before-insert trigger on replica to put data into second “real” table • Only columns a and b, and only for b=1 create trigger x_bi before insert on x for each row insert into y (a, b) select new.a, new.b from dual where new.b=1;

53. Limiting Columns or Rows (1) Master table1 table2 mysql Update table1 set... Insert into table2... client Replica (2) x y IO Thread SQL Thread Update table1 set... Insert into table2... mysql TRG

54. Multi-Level • Local and Remote DR Site • Spreading load over many servers M R M R M R R R R R R R R ...

55. Multi-Level • Config log_slave_updates=1

56. Multi-Level Master Replica table1 table2 table1 table2 mysql IO Thread SQL Thread Update table1 set... Insert into table2... Update table1 set... Insert into table2... Replica table1 table2 IO Thread SQL Thread Update table1 set... Insert into table2... Update table1 set... Insert into table2... mysql

57. Types of Binlogs • Statement-Based – Contains the actual statements run – Data drift is more likely (code modified by triggers, etc) • Row-based – Replicates changes for each row – Mostly unreadable by humans – Unaffected by how the change is done (data drift less likely) – mysqlbinlog -v

59. Fault: Lag • Long-running transactions • High concurrency on master becomes single session on replica • show slave statusG • Easy to alarm on

60. Fault: Stopping from Error • Any error stops SQL Thread! • Duplicate Key • Unknown Function • slave_exec_mode=idempotent ...suppresses duplicate-key and no-key-found errors • show slave statusG

61. Faults: Can’t Connect to Master • Fault shows in slave status as soon as you start slave. • Did you create replication slave user? • Correct user/password? • Correct hostname/port? • Master is running and reachable?

62. Fault: Can’t Find Binlog • Clearly stated in the slave status • Can happen if replica has been stopped (or is lagged) and the binlogs are removed from the master • Execute from archive OR • Rebuild replica and restart from a known good binlog position

63. Faults: Configuration • replication-do-table • replication-do-schema • log_slave_updates

64. Faults: server_id • Must be unique! • If equal: “Seconds behind” jumps everywhere.

66. Gotcha: functions • Functions will be run on the replica even if only appearing in a select on the master • Make sure all functions exist on replica DB, even if only a dummy version create function myFtn (a int, b int) returns int deterministic return 0;

67. Gotcha: Reconnection • When a replica is stopped and reset, until the database is restarted, replication may reconnect to a master after “reset slave” change master to master_port=1234;

68. Gotcha: Master Crash • Replication becomes unstable if master crashes • Error: unknown instruction at position xxx • Often need to use mysqlbinlog to find a suitable position in the binlog to restart replication from stop slave; reset slave; change master to ...; start slave;

69. Gotcha: MySQL Versions • Replication not completely compatible between versions (especially later=>earlier) • 5.5 master to 5.1 replica: character set error “sometimes” • Stops replication

70. Impossible • One replica can have only one master (until 5.7) • Can’t change table or schema filters dynamically • Can’t put binlogs back on master server, for recovery purposes

72. The Book

73. Alternatives • Tungsten Replicator from Continuent • Others?

74. Questions?

75. Thank you! Bob Burgess rburgess@salesforce.com Don’t forget your session evaluation!

Easy MySQL Replication Setup and Troubleshooting

You are reading a preview.

Check these out next

Easy MySQL Replication Setup and Troubleshooting

More Related Content

Slideshows for you (20)

Similar to Easy MySQL Replication Setup and Troubleshooting (20)

Recently uploaded (20)