Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MySQL Group Replication: Handling Network Glitches - Best Practices

86 views

Published on

preFOSDEM MySQL Day 2020
Presentation from Pedro Gomes

Published in: Technology
  • Be the first to comment

MySQL Group Replication: Handling Network Glitches - Best Practices

  1. 1. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Group Replication: Handling Network Glitches Best Practices Pedro Gomes Software Developer MySQL Replication Team Censored Dolphins January 30, 2020 Confidential – Oracle Internal/Restricted/Highly Restricted
  2. 2. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a  commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. Confidential – Oracle Internal/Restricted/Highly Restricted 3
  3. 3. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | ●@distributedpete ●MySQL Replication Developer ●MySQL Developer since 2012 ●Portuguese born and raised 🇵🇹 Pedro Gomes Confidential – Oracle Internal/Restricted/Highly Restricted 4
  4. 4. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | What you can expect Checking the status Diagnostics Adapt Deal with Failures 1 2 3 4 Confidential – Oracle Internal/Restricted/Highly Restricted 5
  5. 5. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Checking the status Diagnostics Adapt Deal with Failures 1 2 3 4 Confidential – Oracle Internal/Restricted/Highly Restricted 6
  6. 6. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Checking the status • Where to look? – The usual suspect: Performance_Schema • MySQL Shell is your friend The first step in solving an issue is knowing what the issue is Confidential – Oracle Internal/Restricted/Highly Restricted 7
  7. 7. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Checking the status Performance Schema Confidential – Oracle Internal/Restricted/Highly Restricted 8
  8. 8. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Checking the status Performance Schema Confidential – Oracle Internal/Restricted/Highly Restricted 9
  9. 9. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Checking the status Performance Schema Confidential – Oracle Internal/Restricted/Highly Restricted 10
  10. 10. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Checking the status MySQL Shell Confidential – Oracle Internal/Restricted/Highly Restricted 11
  11. 11. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Checking the status MySQL Shell – extended information Confidential – Oracle Internal/Restricted/Highly Restricted 12
  12. 12. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Check the status Diagnostics Adapt Deal with failures 1 2 3 4 Confidential – Oracle Internal/Restricted/Highly Restricted 13
  13. 13. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Diagnostics • Identify the scenario • Identify the cause What do I do with this diagnostic data? Confidential – Oracle Internal/Restricted/Highly Restricted 14
  14. 14. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Diagnostics Easily identified scenarios Confidential – Oracle Internal/Restricted/Highly Restricted 15 • Loosing a single server
  15. 15. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Diagnostics Easily identified scenarios Confidential – Oracle Internal/Restricted/Highly Restricted 16 • No Quorum
  16. 16. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Diagnostics • There is a delay or network latency fluctuations on the secondaries – What stats can I check? Evaluating delays Confidential – Oracle Internal/Restricted/Highly Restricted 17
  17. 17. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Diagnostics • From replication_applier_status_by_worker and replication_connection_status • Replication delay - 'rep delay (sec)': – SELECT LAST_APPLIED_TRANSACTION_END_APPLY_TIMESTAMP - LAST_APPLIED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP • Transport time - 'transport time': – LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP - LAST_QUEUED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP • Time to queue on the relay log - 'time RL' : – LAST_QUEUED_TRANSACTION_END_QUEUE_TIMESTAMP - LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP • Time for application - 'apply time' : – LAST_APPLIED_TRANSACTION_END_APPLY_TIMESTAMP - LAST_APPLIED_TRANSACTION_START_APPLY_TIMESTAMP Evaluating delays Confidential – Oracle Internal/Restricted/Highly Restricted 18
  18. 18. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Diagnostics Evaluating delays Confidential – Oracle Internal/Restricted/Highly Restricted 19
  19. 19. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Diagnostics Evaluating delays Confidential – Oracle Internal/Restricted/Highly Restricted 20 • You can code this as a shell plugin • Check it at https://github.com/lefred/mysqlshell-plugins
  20. 20. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Diagnostics (non network issues) • Can your workload be executed in parallel? – Check if your parallel applier workers are idle • Can the group handle the load? – COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE – COUNT_TRANSACTIONS_IN_QUEUE What if I see no delays? Confidential – Oracle Internal/Restricted/Highly Restricted 21
  21. 21. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Check the status Diagnostics Adapt Deal with the failures 1 2 3 4 Confidential – Oracle Internal/Restricted/Highly Restricted 22
  22. 22. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Adapt • To avoid frequent member expels due to faulty networks make use of: – group_replication_member_expel_timeout Adapt to faulty networks Confidential – Oracle Internal/Restricted/Highly Restricted 23
  23. 23. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Adapt Adapt to higher latencies Confidential – Oracle Internal/Restricted/Highly Restricted 24 group replication write concurrency
  24. 24. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Adapt Adapt to higher latencies Confidential – Oracle Internal/Restricted/Highly Restricted 25 group replication write concurrency
  25. 25. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Adapt Adapt to higher latencies Confidential – Oracle Internal/Restricted/Highly Restricted 26
  26. 26. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Adapt Adapt to higher latencies Confidential – Oracle Internal/Restricted/Highly Restricted 27
  27. 27. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Adapt Adapt to higher latencies Confidential – Oracle Internal/Restricted/Highly Restricted 28
  28. 28. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Adapt Adapt to higher latencies Confidential – Oracle Internal/Restricted/Highly Restricted 29 • Also be aware of big transactions: – group_replication_communication_max_message_size ● ● ● ● Alternative strategy – group_replication_transaction_size_limit
  29. 29. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Check the status Diagnostics Adapt Deal with failures 1 2 3 4 Confidential – Oracle Internal/Restricted/Highly Restricted 30
  30. 30. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Deal with failures • Configure your rejoin attempts for each member – group_replication_autorejoin_tries • If you don’t want to persist it Retry retry retry Confidential – Oracle Internal/Restricted/Highly Restricted 31
  31. 31. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Deal with failures • Configure whom shall the next primary be Whom shall be next Confidential – Oracle Internal/Restricted/Highly Restricted 32
  32. 32. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Deal with failures • Configure your timeout when quorum is not achievable Prevent other members from rising from the dead Confidential – Oracle Internal/Restricted/Highly Restricted 33
  33. 33. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Deal with failures • What is the behavior when a member goes to error? – Read only mode : no writes, can still read from it – Offline mode: no reads or writes, can still access the server with admin privileges (CONNECTION_ADMIN or SUPER) – Abort server: the name is self explanatory Update your exit state action Confidential – Oracle Internal/Restricted/Highly Restricted 34
  34. 34. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Thank you! Questions? Confidential – Oracle Internal/Restricted/Highly Restricted 35

×