The document discusses best practices for handling network glitches in group replication. It recommends checking replication status using Performance Schema and MySQL Shell to diagnose issues. It also suggests adapting group replication settings to faulty networks by increasing timeouts to avoid expels. These adaptations include increasing write concurrency and transaction size limits to handle higher latencies. The document also recommends configuring rejoin attempts and quorum timeout to deal with failures and prevent unstable members from interfering.

2. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Group Replication: Handling Network Glitches
Best Practices
Pedro Gomes
Software Developer
MySQL Replication Team Censored Dolphins
January 30, 2020
Confidential – Oracle Internal/Restricted/Highly Restricted

3. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, timing, and pricing of any
features or functionality described for Oracle’s products may change and remains at the
sole discretion of Oracle Corporation.
Confidential – Oracle Internal/Restricted/Highly Restricted 3

4. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
●@distributedpete
●MySQL Replication Developer
●MySQL Developer since 2012
●Portuguese born and raised 🇵🇹
Pedro Gomes
Confidential – Oracle Internal/Restricted/Highly Restricted 4

5. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
What you can expect
Checking the status
Diagnostics
Adapt
Deal with Failures
1
2
3
4
Confidential – Oracle Internal/Restricted/Highly Restricted 5

6. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Checking the status
Diagnostics
Adapt
Deal with Failures
1
2
3
4
Confidential – Oracle Internal/Restricted/Highly Restricted 6

7. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Checking the status
• Where to look?
– The usual suspect: Performance_Schema
• MySQL Shell is your friend
The first step in solving an issue is knowing what the issue is
Confidential – Oracle Internal/Restricted/Highly Restricted 7

8. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Checking the status
Performance Schema
Confidential – Oracle Internal/Restricted/Highly Restricted 8

9. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Checking the status
Performance Schema
Confidential – Oracle Internal/Restricted/Highly Restricted 9

10. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Checking the status
Performance Schema
Confidential – Oracle Internal/Restricted/Highly Restricted 10

11. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Checking the status
MySQL Shell
Confidential – Oracle Internal/Restricted/Highly Restricted 11

12. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Checking the status
MySQL Shell – extended information
Confidential – Oracle Internal/Restricted/Highly Restricted 12

13. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Check the status
Diagnostics
Adapt
Deal with failures
1
2
3
4
Confidential – Oracle Internal/Restricted/Highly Restricted 13

14. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Diagnostics
• Identify the scenario
• Identify the cause
What do I do with this diagnostic data?
Confidential – Oracle Internal/Restricted/Highly Restricted 14

15. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Diagnostics
Easily identified scenarios
Confidential – Oracle Internal/Restricted/Highly Restricted 15
• Loosing a single server

16. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Diagnostics
Easily identified
scenarios
Confidential – Oracle Internal/Restricted/Highly Restricted 16
• No Quorum

17. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Diagnostics
• There is a delay or network latency fluctuations on the secondaries
– What stats can I check?
Evaluating delays
Confidential – Oracle Internal/Restricted/Highly Restricted 17

18. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Diagnostics
• From replication_applier_status_by_worker and replication_connection_status
• Replication delay - 'rep delay (sec)':
– SELECT LAST_APPLIED_TRANSACTION_END_APPLY_TIMESTAMP -
LAST_APPLIED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP
• Transport time - 'transport time':
– LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP -
LAST_QUEUED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP
• Time to queue on the relay log - 'time RL' :
– LAST_QUEUED_TRANSACTION_END_QUEUE_TIMESTAMP -
LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP
• Time for application - 'apply time' :
– LAST_APPLIED_TRANSACTION_END_APPLY_TIMESTAMP -
LAST_APPLIED_TRANSACTION_START_APPLY_TIMESTAMP
Evaluating delays
Confidential – Oracle Internal/Restricted/Highly Restricted 18

19. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Diagnostics
Evaluating delays
Confidential – Oracle Internal/Restricted/Highly Restricted 19

20. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Diagnostics
Evaluating delays
Confidential – Oracle Internal/Restricted/Highly Restricted 20
• You can code this as a shell plugin
• Check it at https://github.com/lefred/mysqlshell-plugins

21. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Diagnostics (non network issues)
• Can your workload be executed in parallel?
– Check if your parallel applier workers are idle
• Can the group handle the load?
– COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE
– COUNT_TRANSACTIONS_IN_QUEUE
What if I see no delays?
Confidential – Oracle Internal/Restricted/Highly Restricted 21

22. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Check the status
Diagnostics
Adapt
Deal with the failures
1
2
3
4
Confidential – Oracle Internal/Restricted/Highly Restricted 22

23. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Adapt
• To avoid frequent member expels due to faulty networks make use of:
– group_replication_member_expel_timeout
Adapt to faulty networks
Confidential – Oracle Internal/Restricted/Highly Restricted 23

24. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Adapt
Adapt to higher latencies
Confidential – Oracle Internal/Restricted/Highly Restricted 24
group replication write concurrency

25. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Adapt
Adapt to higher latencies
Confidential – Oracle Internal/Restricted/Highly Restricted 25
group replication write concurrency

26. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Adapt
Adapt to higher latencies
Confidential – Oracle Internal/Restricted/Highly Restricted 26

27. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Adapt
Adapt to higher latencies
Confidential – Oracle Internal/Restricted/Highly Restricted 27

28. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Adapt
Adapt to higher latencies
Confidential – Oracle Internal/Restricted/Highly Restricted 28

29. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Adapt
Adapt to higher latencies
Confidential – Oracle Internal/Restricted/Highly Restricted 29
• Also be aware of big transactions:
– group_replication_communication_max_message_size
●
●
●
●
Alternative strategy
– group_replication_transaction_size_limit

30. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Check the status
Diagnostics
Adapt
Deal with failures
1
2
3
4
Confidential – Oracle Internal/Restricted/Highly Restricted 30

31. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Deal with failures
• Configure your rejoin attempts for each member
– group_replication_autorejoin_tries
• If you don’t want to persist it
Retry retry retry
Confidential – Oracle Internal/Restricted/Highly Restricted 31

32. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Deal with failures
• Configure whom shall the next primary be
Whom shall be next
Confidential – Oracle Internal/Restricted/Highly Restricted 32

33. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Deal with failures
• Configure your timeout when quorum is not achievable
Prevent other members from rising from the dead
Confidential – Oracle Internal/Restricted/Highly Restricted 33

34. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Deal with failures
• What is the behavior when a member goes to error?
– Read only mode : no writes, can still read from it
– Offline mode: no reads or writes, can still access the server with
admin privileges (CONNECTION_ADMIN or SUPER)
– Abort server: the name is self explanatory
Update your exit state action
Confidential – Oracle Internal/Restricted/Highly Restricted 34

35. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Thank you!
Questions?
Confidential – Oracle Internal/Restricted/Highly Restricted 35