This document provides an overview of advanced MySQL replication techniques. It begins with an introduction to replication basics like principles, setup, and monitoring. It then discusses challenges with replication including master switchovers during failures and slaves falling behind. The document outlines various replication power techniques and features that can help address these issues like row-based and semi-synchronous replication. It also covers tools for replication and techniques for replication outside of the traditional setup.
FOSDEM 2013 : Getting Started with Couchhbase Server 2.0Tugdual Grall
Learn about Couchbase Server 2.0 the Open Source NoSQL database.
This presentation was delivered on Feb 3rd during the FOSDEM conference ( http://fosdem.org )
FOSDEM 2013 : Getting Started with Couchhbase Server 2.0Tugdual Grall
Learn about Couchbase Server 2.0 the Open Source NoSQL database.
This presentation was delivered on Feb 3rd during the FOSDEM conference ( http://fosdem.org )
Many questions on database newsgroups and forums can be answered with uses of outer joins. Outer joins are part of the standard SQL language and supported by all RDBMS brands. Many programmers are expected to use SQL in their work, but few know how to use outer joins effectively.
Learn to use this powerful feature of SQL, increase your employability, and amaze your friends!
Karwin will explain outer joins, show examples, and demonstrate a Sudoku puzzle solver implemented in a single SQL query.
Java MySQL Connector & Connection Pool Features & OptimizationKenny Gryp
This talk will give an overview of the different available Java MySQL connectors (Connector/J, MariaDB Java Connector) and connection pools (Commons-DBCP, C3P0, ...).
One of the things with the default configuration of these solutions is that they are very chatty. This can have a noticeable impact on application performance and database load. I've seen many environments where over 50% of the total amount of queries are caused by such behavior. This behavior will be explained through examples seen on production systems as well as recommendations on optimization will be made.
Load balancing and transparent failover solutions will be described for both MySQL's traditional asynchronous replication and Galera based replication (Percona XtraDB Cluster or MariaDB Galera Cluster).
Designing an extensible, flexible schema that supports user customization is a common requirement, but it's easy to paint yourself into a corner.
Examples of extensible database requirements:
- A database that allows users to declare new fields on demand.
- Or an e-commerce catalog with many products, each with distinct attributes.
- Or a content management platform that supports extensions for custom data.
The solutions we use to meet these requirements is overly complex and the performance is terrible. How should we find the right balance between schema and schemaless database design?
I'll briefly cover the disadvantages of Entity-Attribute-Value (EAV), a problematic design that's an example of the antipattern called the Inner-Platform Effect, That is, modeling an attribute-management system on top of the RDBMS architecture, which already provides attributes through columns, data types, and constraints.
Then we'll discuss the pros and cons of alternative data modeling patterns, with respect to developer productivity, data integrity, storage efficiency and query performance, and ease of extensibility.
- Class Table Inheritance
- Serialized BLOB
- Inverted Indexing
Finally we'll show tools like pt-online-schema-change and new features of MySQL 5.6 that take the pain out of schema modifications.
A presentation about how to make MySQL highly available, presented at the San Francisco MySQL Meetup (http://www.sfmysql.org/events/15760472/) on January 26th, 2011.
A video recording of this presentation is available from Ustream: http://ustre.am/fyLk
"Disaster is inevitable" and "To move forward you must first backup" should be known to all software developers. This presentation will discuss all the options for your valuable data assets in MySQL, and highlight how to maintain site reliability of your data
A New Architecture for Group Replication in Data GridEditor IJCATR
Nowadays, grid systems are vital technology for programs running with high performance and problems solving with largescale
in scientific, engineering and business. In grid systems, heterogeneous computational resources and data should be shared
between independent organizations that are scatter geographically. A data grid is a kind of grid types that make relations computational
and storage resources. Data replication is an efficient way in data grid to obtain high performance and high availability by saving
numerous replicas in different locations e.g. grid sites. In this research, we propose a new architecture for dynamic Group data
replication. In our architecture, we added two components to OptorSim architecture: Group Replication Management component
(GRM) and Management of Popular Files Group component (MPFG). OptorSim developed by European Data Grid projects for
evaluate replication algorithm. By using this architecture, popular files group will be replicated in grid sites at the end of each
predefined time interval.
MySQL Group Replication - HandsOn TutorialKenny Gryp
During this tutorial, attendees have their hands on virtual machines and migrate standard Master - Slave architecture to the new MySQL native Group Replication.
After explaining briefly what is group replication and how this is important for MySQL HA architecture. We will cover how to verify the workload and the scheme to how GR can be used and configured.
Then we will go trough the migration steps with minimal impact on the live system.
Basic administration tasks are covered such as add/remove a node from the cluster. We also play with performance_schema to monitor our Group Replication cluster and understand how to control it.
MySQL Replication Performance Tuning for Fun and Profit!Vitor Oliveira
MySQL Replication, in addition to bringing high-availability, is the foundation to build high-performance MySQL database systems. Using read scale-out and sharding one can design systems that go from the capacity of a single server to supporting the largest internet sites. But to design and operate high-performance, efficient, manageable and reliable deployments requires knowing the intricacies of the underlying technologies.
This session will provide insights on the main factors that affect the performance of Asynchronous Replication and Group Replication, and how to configure them to make the most out of the underlying computing system. It will also show the latest developments in MySQL 5.7 and 8.0, in areas spanning from group communication to the multi-threaded slave applier, and how effective they are in helping meet the performance requirements in terms of throughput, latency and durability to support the most demanding workload types.
MySQL High Availability with Group ReplicationNuno Carvalho
MySQL Group Replication is a MySQL plugin that combines distributed systems technologies with database replication. The outcome of fusing such technologies is a resilient, multimaster update-everywhere replication solution for MySQL. At its core is a group communication service providing reliable and dependable message-passing for a group of MySQL servers. This enables automatic and consistent coordination between all the participants in a group. Such automatic orchestration of MySQL servers results in a fault-tolerant replicated state machine providing a highly available MySQL data store.
Every website wants to become successful. Few websites however undertake the basic and fundamental steps to build a rock solid foundation to ensure a scalable
MySQL High Availability and Disaster Recovery with Continuent, a VMware companyContinuent
Users seeking high availability, disaster recovery and zero downtime maintenance operation for business-critical MySQL applications face confusing choices. Is multi-master or master/slave clustering better? What about synchronous versus asynchronous replication? Using a plain vanilla, stock MySQL or a modified version of it? Which of these choices are right for data-driven businesses that depend on fast, reliable data access?
This no-BS webinar cuts through the FUD to explore the real trade-offs between the different clustering and replication methods, thens show you how Continuent's asynchronous master/slave clusters support these important capabilities for business-critical applications:
- High application write rates Master/slave clustering with Continuent
- Mixed workloads consisting of large and small transactions
- Data across multiple geographically distributed locations
- Failures and more importantly recovery from them
- Zero downtime maintenance and software upgrades
- Use of off-the-shelf MySQL/MariaDB to avoid application changes and allow clusters to improve as MySQL itself does.
We illustrate key points with demonstrations and case studies from deployed systems.
Presentation about front end performance improvements with specific hints if you're running Ruby on Rails. Original presentation at the Boston-rb group in April 12, 2011
Many questions on database newsgroups and forums can be answered with uses of outer joins. Outer joins are part of the standard SQL language and supported by all RDBMS brands. Many programmers are expected to use SQL in their work, but few know how to use outer joins effectively.
Learn to use this powerful feature of SQL, increase your employability, and amaze your friends!
Karwin will explain outer joins, show examples, and demonstrate a Sudoku puzzle solver implemented in a single SQL query.
Java MySQL Connector & Connection Pool Features & OptimizationKenny Gryp
This talk will give an overview of the different available Java MySQL connectors (Connector/J, MariaDB Java Connector) and connection pools (Commons-DBCP, C3P0, ...).
One of the things with the default configuration of these solutions is that they are very chatty. This can have a noticeable impact on application performance and database load. I've seen many environments where over 50% of the total amount of queries are caused by such behavior. This behavior will be explained through examples seen on production systems as well as recommendations on optimization will be made.
Load balancing and transparent failover solutions will be described for both MySQL's traditional asynchronous replication and Galera based replication (Percona XtraDB Cluster or MariaDB Galera Cluster).
Designing an extensible, flexible schema that supports user customization is a common requirement, but it's easy to paint yourself into a corner.
Examples of extensible database requirements:
- A database that allows users to declare new fields on demand.
- Or an e-commerce catalog with many products, each with distinct attributes.
- Or a content management platform that supports extensions for custom data.
The solutions we use to meet these requirements is overly complex and the performance is terrible. How should we find the right balance between schema and schemaless database design?
I'll briefly cover the disadvantages of Entity-Attribute-Value (EAV), a problematic design that's an example of the antipattern called the Inner-Platform Effect, That is, modeling an attribute-management system on top of the RDBMS architecture, which already provides attributes through columns, data types, and constraints.
Then we'll discuss the pros and cons of alternative data modeling patterns, with respect to developer productivity, data integrity, storage efficiency and query performance, and ease of extensibility.
- Class Table Inheritance
- Serialized BLOB
- Inverted Indexing
Finally we'll show tools like pt-online-schema-change and new features of MySQL 5.6 that take the pain out of schema modifications.
A presentation about how to make MySQL highly available, presented at the San Francisco MySQL Meetup (http://www.sfmysql.org/events/15760472/) on January 26th, 2011.
A video recording of this presentation is available from Ustream: http://ustre.am/fyLk
"Disaster is inevitable" and "To move forward you must first backup" should be known to all software developers. This presentation will discuss all the options for your valuable data assets in MySQL, and highlight how to maintain site reliability of your data
A New Architecture for Group Replication in Data GridEditor IJCATR
Nowadays, grid systems are vital technology for programs running with high performance and problems solving with largescale
in scientific, engineering and business. In grid systems, heterogeneous computational resources and data should be shared
between independent organizations that are scatter geographically. A data grid is a kind of grid types that make relations computational
and storage resources. Data replication is an efficient way in data grid to obtain high performance and high availability by saving
numerous replicas in different locations e.g. grid sites. In this research, we propose a new architecture for dynamic Group data
replication. In our architecture, we added two components to OptorSim architecture: Group Replication Management component
(GRM) and Management of Popular Files Group component (MPFG). OptorSim developed by European Data Grid projects for
evaluate replication algorithm. By using this architecture, popular files group will be replicated in grid sites at the end of each
predefined time interval.
MySQL Group Replication - HandsOn TutorialKenny Gryp
During this tutorial, attendees have their hands on virtual machines and migrate standard Master - Slave architecture to the new MySQL native Group Replication.
After explaining briefly what is group replication and how this is important for MySQL HA architecture. We will cover how to verify the workload and the scheme to how GR can be used and configured.
Then we will go trough the migration steps with minimal impact on the live system.
Basic administration tasks are covered such as add/remove a node from the cluster. We also play with performance_schema to monitor our Group Replication cluster and understand how to control it.
MySQL Replication Performance Tuning for Fun and Profit!Vitor Oliveira
MySQL Replication, in addition to bringing high-availability, is the foundation to build high-performance MySQL database systems. Using read scale-out and sharding one can design systems that go from the capacity of a single server to supporting the largest internet sites. But to design and operate high-performance, efficient, manageable and reliable deployments requires knowing the intricacies of the underlying technologies.
This session will provide insights on the main factors that affect the performance of Asynchronous Replication and Group Replication, and how to configure them to make the most out of the underlying computing system. It will also show the latest developments in MySQL 5.7 and 8.0, in areas spanning from group communication to the multi-threaded slave applier, and how effective they are in helping meet the performance requirements in terms of throughput, latency and durability to support the most demanding workload types.
MySQL High Availability with Group ReplicationNuno Carvalho
MySQL Group Replication is a MySQL plugin that combines distributed systems technologies with database replication. The outcome of fusing such technologies is a resilient, multimaster update-everywhere replication solution for MySQL. At its core is a group communication service providing reliable and dependable message-passing for a group of MySQL servers. This enables automatic and consistent coordination between all the participants in a group. Such automatic orchestration of MySQL servers results in a fault-tolerant replicated state machine providing a highly available MySQL data store.
Every website wants to become successful. Few websites however undertake the basic and fundamental steps to build a rock solid foundation to ensure a scalable
MySQL High Availability and Disaster Recovery with Continuent, a VMware companyContinuent
Users seeking high availability, disaster recovery and zero downtime maintenance operation for business-critical MySQL applications face confusing choices. Is multi-master or master/slave clustering better? What about synchronous versus asynchronous replication? Using a plain vanilla, stock MySQL or a modified version of it? Which of these choices are right for data-driven businesses that depend on fast, reliable data access?
This no-BS webinar cuts through the FUD to explore the real trade-offs between the different clustering and replication methods, thens show you how Continuent's asynchronous master/slave clusters support these important capabilities for business-critical applications:
- High application write rates Master/slave clustering with Continuent
- Mixed workloads consisting of large and small transactions
- Data across multiple geographically distributed locations
- Failures and more importantly recovery from them
- Zero downtime maintenance and software upgrades
- Use of off-the-shelf MySQL/MariaDB to avoid application changes and allow clusters to improve as MySQL itself does.
We illustrate key points with demonstrations and case studies from deployed systems.
Presentation about front end performance improvements with specific hints if you're running Ruby on Rails. Original presentation at the Boston-rb group in April 12, 2011
Advanced MySQL Replication Architectures - Luis SoaresMySQL Brasil
Arquiteturas avançadas e usos curiosos do recurso nativo de replicação do MySQL. Também são detalhadas algumas novidades da versão 5.6 que elevam a Replicação MySQL a outro nível.
Conçu en 2007 par REQUEA avec l'aide du labo LSR/IMAG (D.Donsez et M.Desertot), Dysoweb est une plateforme de composition d'applications Web dynamiques utilisée pour réaliser des applications d'entreprise.
La plateforme est aujourd'hui au coeur d'une cinquantaine de projets en production, dont certains depuis plus de 4 ans. Ces projets ont été réalisés par une cinquantaine d'ingénieurs, développeurs, consultants qui ont été formés et exposés à OSGi à travers l'utilisation de dysoweb.
Les applications d'entreprise Dysoweb sont utilisées quotidiennement et souvent de manière intensive par environ 15000 utilisateurs dans des environnements variés. Déployées aussi bien en mode SaaS que chez sur les serveurs des clients, sur tout type d'environnement (Windows, Linux, Aix, Solaris) avec une grande variété de moteurs de base de données (Oracle, MSSQL, MySQL) et de serveurs applicatifs (Tomcat, OracleAS, Weblogic, Jonas).
Ces projets ont permis à Dysoweb d'acquérir une maturité certaine au travers de cas d'usage diversifiés où l'utilisation d'OSGi a été déterminante.
These are the slides from my talk about the AppScale project at the SBonRails meetup. It covers AppScale as well as Google App Engine and the research projects have come out of it, including Neptune, a Ruby DSL focused on computation-heavy workloads.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
Advanced mysql replication techniques
1. Advanced MySQL
Replication Techniques
Giuseppe Maxia, QA Director, Continuent, Inc
Facts. And Demos. Possibly fun
Monday, April 11, 2011 1
2. about me - Giuseppe Maxia
• a.k.a. The Data Charmer
• QA Director at Continuent, Inc
• Long time hacking with MySQL features
• Formerly, community manager,db consultant, designer,
coder.
• A passion for QA and open source
• Blogger
• http://datacharmer.blogspot.com
Monday, April 11, 2011 2
3. Agenda
• Replication basics
• Replication blues
• Master switch and failover
• Slave lagging
• Gotchas
• Replication power techniques and features
• row-based, semi-synch, delayed
• Circular, bi-directional
• leveraging replication
• Tools and tips
• Replication outside the box
•
Monday, April 11, 2011
seamless failover, parallel, multi-master
3
4. DA
AG
EN Replication basics
• Principles
• How to set up replication
• Monitoring replication
Monday, April 11, 2011 4
10. DA
AG
EN replication setup
Monday, April 11, 2011 10
11. 1 SHUT DOWN THE DATABASE SERVER
Master
11
Monday, April 11, 2011 11
12. 2 MAKE A BACKUP COPY
Master
12
Monday, April 11, 2011 12
13. 3 ENABLE THE MASTER
Master
Configuration file
[mysqld]
log-bin=mysql-bin
server-id=1
13
Monday, April 11, 2011 13
14. 4 RESTART THE MASTER
Master
14
Monday, April 11, 2011 14
15. 5 CREATE REPLICATION USER
Master
SQL command
GRANT REPLICATION SLAVE ON *.*
to 'slave_user@'10.10.100.%'
IDENTIFIED BY 'slave_pass';
15
Monday, April 11, 2011 15
16. 6 INSTALL MySQL on the slave
Slave 1
Make sure that:
• You're using the same version of MySQL
• You have the same directory structure
• The server is not started yet
16
Monday, April 11, 2011 16
17. 7 COPY THE MASTER DATA to the slave
Slave 1
17
Monday, April 11, 2011 17
19. 9 START THE SLAVE SERVER
Slave 1
19
Monday, April 11, 2011 19
20. 10 INITIALIZE THE SLAVE
Slave 1
SQL command
CHANGE MASTER TO
MASTER_HOST=master_IP,
MASTER_PORT=3306,
MASTER_USER=slave_user,
MASTER_PASSWORD='slave_pwd';
20
Monday, April 11, 2011 20
21. 11 START THE SLAVE SERVICE
Slave 1
SQL command
START SLAVE;
21
Monday, April 11, 2011 21
22. 12 CHECK THE SLAVE
Slave 1
SQL command
SHOW SLAVE STATUS G
...
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
...
22
Monday, April 11, 2011 22
23. Troubleshooting
• SHOW SLAVE STATUS says
SLAVE_IO_RUNNING=No
• Make sure that the slave host can connect
to the master
• Make sure that master and slave have
different Server-id
• Check the error log of both master and slave
Monday, April 11, 2011 23
24. Testing the slave
• Create a table in the master.
• Make sure that the slave has replicated
the table.
• Insert data in the master
• read that data in the slave
Monday, April 11, 2011 24
26. Sample get slave
status
monitoring
slave
Running? No
Yes
master
Get master
binlog and
position
Same or later
No
Yes
binlog/position?
check table
contents
alert 26
Monday, April 11, 2011 26
27. monitoring replication
master> SHOW MASTER STATUS
slave> SHOW SLAVE STATUS
FULL SCRIPTS:
http://datacharmer.blogspot.com/2011/04/refactored-again-poor-mans-
mysql.html
http://forge.mysql.com/tools/tool.php?id=6
Monday, April 11, 2011 27
28. show master status
File: mysql-bin.000002
Position: 78045744
Binlog_Do_DB:
Binlog_Ignore_DB:
Monday, April 11, 2011 28
29. show slave status
Slave_IO_State: Waiting for master to send event
Master_Host: 127.0.0.1
Master_User: rsandbox
Master_Port: 27371
Connect_Retry: 60
Master_Log_File: mysql-bin.000002
Read_Master_Log_Pos: 78045744
Relay_Log_File: mysql_sandbox27372-relay-bin.000055
Relay_Log_Pos: 78045889
Relay_Master_Log_File: mysql-bin.000002
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Monday, April 11, 2011 29
30. show slave status
...
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
...
Monday, April 11, 2011 30
31. show slave status
...
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 78045744
Relay_Log_Space: 78046100
...
Seconds_Behind_Master: 0
...
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Monday, April 11, 2011 31
32. DA
AG
EN Replication blues
• Master switch and failover
• Slave lagging
• Gotchas
Monday, April 11, 2011 32
33. DA
AG
EN Master switch and failover
Monday, April 11, 2011 33
34. Replacing the Master
master crashe
s
Let all slaves
catch up with
execution STOP
replication in
all slaves
FIND the most make it the
up to date slave master
FIND which run missing
connect all
transactions are transactions
slaves to the
missing from to other
new master
other slaves slaves
Stop
34
Monday, April 11, 2011 34
35. Planned master switch
• Stop accepting writes
• Wait until all slaves have caught up
• Stop replication in all slaves
• Promote a slave to master
• Point all slaves to the new master
Monday, April 11, 2011 35
36. Changing a failed master
• Pre-requisite:
• log_bin and log_slave_updates must be
enabled in all the slaves
• If not, there is more manual labor
Monday, April 11, 2011 36
37. Changing a failed master (1)
• Wait until all slaves have caught up
• Identify the most advanced slave
• Make that slave the new master
• ... so far, so good
Monday, April 11, 2011 37
38. Changing a failed master (2)
• For each remaining slave:
• Find the missing statements
• Find the LAST statement replicated by the slave
• Find the same statement in the new master
binlog (*)
• get the position of the NEXT statement
(*) if log_slave_updates was not enabled, you
need to convert the relay log statements to SQL
and do the next step manually
Monday, April 11, 2011 38
39. Changing a failed master (3)
• For each remaining slave:
• Apply the missing statements
• CHANGE MASTER TO
master_host=”new_master_hostname”,
master_port=new_master_port,
master_log_file=”mysql-bin.xxxxxx”,
master_log_pos=YYYY
Monday, April 11, 2011 39
41. What is a global transaction ID
• A unique identifier of a transaction
• Unique for the whole cluster, not for each node
• Generated by the ultimate source (the master)
• Does not change when the transaction goes
through an intermediate master
Monday, April 11, 2011 41
42. Why should you care
• Failure recovery
• MySQL DOES NOT have it
Monday, April 11, 2011 42
47. Need to synch the
missing transactions
New master
Slave 1 Slave 2
Binary log
Transactions
========
Binary log Binary log ========
========
Transactions Transactions ========
======== ========
======== ========
======== ========
======== ========
Monday, April 11, 2011 47
48. Transaction position in
the new master binlog
New master
Slave 1 Slave 2
Binary log
Transactions
========
Binary log Binary log ========
========
Transactions Transactions ========
======== ========
======== ========
======== ? ======== ?
======== ========
Monday, April 11, 2011 48
49. An existing solution
• Google has made a patch to implement global
transaction IDs
• HOWEVER
• It only works with 5.0
• It doesn't work with row based replication
Monday, April 11, 2011 49
51. CREATE TABLE Global_Trans_ID (
number INT UNSIGNED AUTO_INCREMENT PRIMARY KEY
) ENGINE = MyISAM;
CREATE TABLE Last_Exec_Trans (
server_id INT UNSIGNED,
trans_id INT UNSIGNED
) ENGINE = InnoDB;
Monday, April 11, 2011 51
52. CREATE PROCEDURE commit_trans ()
BEGIN
DECLARE trans_id, server_id INT UNSIGNED;
SET SQL_LOG_BIN = 0;
INSERT INTO global_trans_id() values ();
SELECT LAST_INSERT_ID() INTO trans_id,
@@server_id INTO server_id;
DELETE FROM global_trans_id where number < trans_Id;
SET SQL_LOG_BIN = 1;
INSERT INTO last_exec_trans(server_id, trans_id)
VALUES (server_id, trans_id);
COMMIT;
END
Monday, April 11, 2011 52
54. Simplifying the scheme
• No MyISAM table and LAST_INSERT_ID()
• Use UUID_SHORT() instead
• Replace the InnoDB table with a BlackHole one
Monday, April 11, 2011 54
55. Advantages
• 10 times faster
• No additional tables and storage required
Monday, April 11, 2011 55
56. Disadvantages
• The master SERVER ID must be < 256
Monday, April 11, 2011 56
57. Another solution
• We’ll see that in the second part of the
talk
Monday, April 11, 2011 57
58. DA
AG
EN Slave lagging
Monday, April 11, 2011 58
59. Slave lagging
• Common causes:
• Excess of traffic
• Slave restart
• Long DDL operations
Monday, April 11, 2011 59
60. Slave lagging
transaction
transaction
MySQL transaction
transaction
transaction
DBMS transaction
transaction
transaction
transaction
transaction
transaction
transaction
BINARY LOG
ns act act tio n
tra csio s ac ctio
ac ion ion n
an n s an
nr tr tra aniton
tsa a n csio
nts ct n
rn a o
taa scti
r
trnsa
n
n
tio
t
tra
tra
REPLICATION
MySQL replication
is SINGLE THREAD
transaction
transaction
MySQL
DBMS
Monday, April 11, 2011 60
62. statement-based replication
master> set global binlog_format=‘statement’;
master> insert into test.t1 select count(*) from
information_schema.columns;
master> show global status like 'opened_tables';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Opened_tables | 41 |
+---------------+-------+
slave> show global status like 'opened_tables';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Opened_tables | 41 |
+---------------+-------+
Monday, April 11, 2011 62
63. row-based replication
master> set global binlog_format=‘row’;
master> insert into test.t1 select count(*) from
information_schema.columns;
master> show global status like 'opened_tables';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Opened_tables | 41 |
+---------------+-------+
slave> show global status like 'opened_tables';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Opened_tables | 17 |
+---------------+-------+
Monday, April 11, 2011 63
64. Expensive query split
# instead of this:
insert into t1 select benchmark(10000000, sha('abc'));
Query OK, 1 row affected (4.19 sec)
Records: 1 Duplicates: 0 Warnings: 0
# you may do this
set @result=(select benchmark(10000000, sha('abc')));
Query OK, 0 rows affected (4.19 sec)
insert into t1 values (@result);
Query OK, 1 row affected (0.00 sec)
Monday, April 11, 2011 64
65. Expensive query split
# at 471
#110410 8:21:21 server id 1 end_log_pos 518
User_var
SET @`result`:=0/*!*/;
# at 518
#110410 8:21:21 server id 1 end_log_pos 612 Query
thread_id=3 exec_time=0 error_code=0
SET TIMESTAMP=1302448881/*!*/;
insert into t1 values (@result)
Monday, April 11, 2011 65
66. Slave query prefetch
• What is it?
• a technique that fetches queries from the
relay logs, and runs them as SELECT
statements, to warm up the indexes
• How do you do it?
• mk-slave-prefetch (http://www.maatkit.org)
• Slave readahead http://sourceforge.net/
projects/slavereadahead/
Monday, April 11, 2011 66
68. Gotchas
• When you thought that you had figured
out everything ...
Monday, April 11, 2011 68
69. Gotchas: LOAD DATA
• LOAD DATA INFILE is replicated as a
temporary file
• The slave needs three times the size of
the data:
• the relay log
• the temporary file
• the data itself
Monday, April 11, 2011 69
70. Gotchas: default engine
• If you define storage_engine in the
master, it does not get replicated
Monday, April 11, 2011 70
71. Gotchas: binlog format
• In circular replication, and relay slaves
• Changes to binlog_format are not
propagated
Monday, April 11, 2011 71
72. Gotchas: binlog format
server id: 101 insert into t1
format: statement values (@@server_id) 101
server id: 102
format: row
102
server id: 103 102
format: row
Monday, April 11, 2011 72
73. Gotchas: set global not really global
• set global binlog_format=row
• Works for all new connections after the
change
• It does not work for the event
scheduler!
Monday, April 11, 2011 73
74. Gotchas: triggers and row replication
• In statement based replication, triggers
are fired both on master and slave
• in row-based replication, triggers are
fired on master only, and the resulting
rows are moved to the slave
Monday, April 11, 2011 74
75. DA
AG
EN Replication power techniques
and features
• row-based
• semi-synch
• delayed
• Circular
• bi-directional
• Leveraging replication
Monday, April 11, 2011 75
76. DA
AG
EN row-based replication
Monday, April 11, 2011 76
77. row-based replication
• Available in 5.1 and later
• can be changed using “binlog_format”
• row
• mixed
• statement (default)
Monday, April 11, 2011 77
78. row-based replication
• DO
• when you have expensive queries in the
master
• when you are using non-deterministic
functions
• DON’T
• When you are updating a lot of rows
Monday, April 11, 2011 78
79. DA
AG
EN semi-synch replication
Monday, April 11, 2011 79
80. semi-synchronous
replication
• Available in 5.5 and higher
• Makes sure that at least one slave has
copied the data.
• Increases reliability
Monday, April 11, 2011 80
81. client
master transaction
1 with regular
commit replication
execute 2
slave
returns
4 to client
binary log 3
5 replication
81
Monday, April 11, 2011 81
82. client transaction
master
1 with semi-
commit synchronous
replication
execute 2
slave
7 returns
to client
binary log 3
sends
4 transaction relay log 5
to slave
gets
6 acknowledgement
82
Monday, April 11, 2011 82
83. semi-synchronous
replication in practice
• installation:
• it’s a plugin.
• Actually, two plugins
Monday, April 11, 2011 83
84. semi-synch replication install
# in the master
plugin-load=rpl_semi_sync_master=semisync_master.so
rpl_semi_sync_master_enabled=1
# in each slave
plugin-load=rpl_semi_sync_slave=semisync_slave.so
rpl_semi_sync_slave_enabled=1
# restar all servers
Monday, April 11, 2011 84
85. semi-synch replication check
# in the master
show variables like 'rpl_semi%';
+------------------------------------+-------+
| Variable_name | Value |
+------------------------------------+-------+
| rpl_semi_sync_master_enabled | ON |
| rpl_semi_sync_master_timeout | 10000 |
| rpl_semi_sync_master_trace_level | 32 |
| rpl_semi_sync_master_wait_no_slave | ON |
+------------------------------------+-------+
Monday, April 11, 2011 85
86. semi-synch replication check
show status like "rpl_semi_%tx";
+-----------------------------+-------+
| variable_name | value |
+-----------------------------+-------+
| RPL_SEMI_SYNC_MASTER_NO_TX | 0 |
| RPL_SEMI_SYNC_MASTER_YES_TX | 0 |
+-----------------------------+-------+
Monday, April 11, 2011 86
87. semi-synch replication test
master> create table t1 ( i int);
Query OK, 0 rows affected (0.01 sec)
master> show status like "rpl_semi_%tx";
+-----------------------------+-------+
| Variable_name | Value |
+-----------------------------+-------+
| Rpl_semi_sync_master_no_tx | 0 |
| Rpl_semi_sync_master_yes_tx | 1 |
+-----------------------------+-------+
Monday, April 11, 2011 87
88. disabling semi-synch
# for each slave
set global rpl_semi_sync_slave_enabled=0;
stop slave io_thread;
start slave io_thread;
Monday, April 11, 2011 88
89. disabled semi-synch replication test
master> insert into t1 values (1);
Query OK, 1 row affected (10.00 sec)
master> show status like "rpl_semi_%tx";
+-----------------------------+-------+
| Variable_name | Value |
+-----------------------------+-------+
| Rpl_semi_sync_master_no_tx | 1 |
| Rpl_semi_sync_master_yes_tx | 1 |
+-----------------------------+-------+
2 rows in set (0.00 sec)
Monday, April 11, 2011 89
90. disabled semi-synch replication test
master> insert into t1 values (2);
Query OK, 1 row affected (0.01 sec)
master> show status like "rpl_semi_%tx";
+-----------------------------+-------+
| Variable_name | Value |
+-----------------------------+-------+
| Rpl_semi_sync_master_no_tx | 2 |
| Rpl_semi_sync_master_yes_tx | 1 |
+-----------------------------+-------+
2 rows in set (0.00 sec)
Monday, April 11, 2011 90
91. re-enabling semi-synch
# in one slave
set global rpl_semi_sync_slave_enabled=1;
stop slave io_thread;
start slave io_thread;
Monday, April 11, 2011 91
92. reenabled semi-synch replication test
master> insert into t1 values (3);
Query OK, 1 row affected (0.01 sec)
master> show status like "rpl_semi_%tx";
+-----------------------------+-------+
| Variable_name | Value |
+-----------------------------+-------+
| Rpl_semi_sync_master_no_tx | 2 |
| Rpl_semi_sync_master_yes_tx | 2 |
+-----------------------------+-------+
2 rows in set (0.00 sec)
Monday, April 11, 2011 92
93. DA
AG
EN delayed replication
Monday, April 11, 2011 93
94. delayed replication in practice
STOP SLAVE;
change master to master_delay=60;
START SLAVE;
Monday, April 11, 2011 94
103. Partition pruning
1a - unpartitioned table - SINGLE RECORD
I
D select *
N from table_name
A
D where colx =
T
E 120
A
X
Monday, April 11, 2011 103
104. Partition pruning
2a - table partitioned by colx - SINGLE REC
1-99
100-199 select *
from table_name
200-299
where colx =
120
300-399
400-499
500-599
Monday, April 11, 2011 104
105. Partition pruning
1a - unpartitioned table - RANGE
select *
I from
D
N table_name
A where colx
D between 120
T
E and 230
A
X
Monday, April 11, 2011 105
106. Partition pruning
2a - table partitioned by colx - RANGE
1-99
select *
100-199 from
table_name
200-299 where colx
between 120
and 230
300-399
400-499
500-599
Monday, April 11, 2011 106
108. Benchmarking results
engine 6 month range
InnoDB 4 min 30s
MyISAM 25.03s
Archive 22 min 25s
InnoDB partitioned by month 13.19
MyISAM partitioned by year 6.31
MyISAM partitioned by month 4.45
Archive partitioned by year 16.67
Archive partitioned by month 8.97
Monday, April 11, 2011 108
109. Partitions limits
§ Little concurrency
§ Need of using the partitioning column
Monday, April 11, 2011 109
113. Replication
=
let the slave do the dirty work
Monday, April 11, 2011 113
114. master
backup
slaves
STOP SLAVE
remove slave
perform
from load
backup
balancer
attach slave
START Let slave
to load
SLAVE catch up
balancer
Monday, April 11, 2011 114
115. master
make
summary
tables
slaves
STOP SLAVE
calculate remove slave
summary from load
tables balancer
attach slave
Let slave
START SLAVE to load
catch up
balancer
Monday, April 11, 2011 115
116. master
Partitions
for heavy innodb
non partitioned
statistics
slave slave
innodb
innodb non partitioned
partitioned by range
slave
MyISAM
partitioned by range
Monday, April 11, 2011 116
117. Simulating master
multiple innodb
non partitioned
dimensions
slave
slave
innodb
non partitioned
ARCHIVE
partitioned by range
(date)
slave
slave
ARCHIVE
partitioned by range
ARCHIVE (location)
partitioned by range
(product)
Monday, April 11, 2011 117
118. DA
AG
EN Tools and tips
• MySQL Sandbox
• Replaying binary logs
• Filtering binary logs
• Baron Schwartz’s Maatkit
• Shlomi Noach’s openark kit
• Facebook’s Online Schema Change
• Measuring replication speed
• Detecting if replication is on
Monday, April 11, 2011 118
119. DA
AG
EN MySQL Sandbox
Monday, April 11, 2011 119
120. MySQL Sandbox
http://mysqlsandbox.net
• Free software (Perl under GPL)
• One (unix) host
• Many database servers
• Single or multiple sandboxes
• Customized scripts to use the servers
• Standard or circular replication
• Installs IN SECONDS
Monday, April 11, 2011 120
121. overview
MySQL MySQL
server server
Data DB1 Data DB1
DB2 DB3
DATA DIRECTORY DB2 DB3
PORT
SOCKET
Monday, April 11, 2011 121
122. overview
MySQL MySQL
server server
Data DB1
SAME Data DB1
DB2 DB3
DATA DB2 DB3
DIRECTORY?
/var/lib/mysql /var/lib/mysql
DATA CORRUPTION
Monday, April 11, 2011 122
123. overview
MySQL MySQL
server server
SAME
PORT or
SOCKET?
3306 3306
/tmp/mysql.sock /tmp/mysql.sock
DOES NOT START
Monday, April 11, 2011 123
126. The easy way
$ make_sandbox
/path/to/mysql-5.1.54_linux.tar.gz
# it should work always
Monday, April 11, 2011 126
127. The easier way
$ make_sandbox 5.1.54
# Needs some preliminary work
Monday, April 11, 2011 127
128. The easiest way
$ sb 5.1.54
# Needs the same preliminary work
Monday, April 11, 2011 128
129. MySQL Sandbox
VERSION
MySQL
server
Data DB1
DB2 DB3
$SANDBOX_HOME/msb_VERSION/data
VERSION
/tmp/mysql_VERSION.sock
Monday, April 11, 2011 129
130. MySQL Sandbox
5.1.54
MySQL
server
Data DB1
DB2 DB3
$SANDBOX_HOME/msb_5_1_54/data
5154
/tmp/mysql_5154.sock
Monday, April 11, 2011 130
131. MySQL Sandbox
5.5.9
MySQL
server
Data DB1
DB2 DB3
$SANDBOX_HOME/msb_5_5_09/data
5509
/tmp/mysql_5509.sock
Monday, April 11, 2011 131
132. Single Sandbox
MySQL customized scripts
server
start
stop
restart
status
clear
send_kill
use
Monday, April 11, 2011 132
133. Multiple Sandbox
MySQL customized scripts
server
start_all
stop_all
restart_all m n1
status_all s1 n2
clear_all s2 n3
send_kill_a
ll
use_all
Monday, April 11, 2011 133
134. Where do you get it
•from CPAN
sudo cpan MySQL::Sandbox
•from launchpad
http://launchpad.net/mysql-sandbox
Monday, April 11, 2011 134
135. The easy replication way
$ make_replication_sandbox
/path/to/mysql-5.1.54_linux.tar.gz
# or, after some preparation
$ make_replication_sandbox 5.1.54
Monday, April 11, 2011 135
136. default architecture
$HOME
/sandboxes opt
expanded
tarballs
$SANDBOX_HOME
mysql
$SANDBOX_BINARY
installed
sandboxes
Monday, April 11, 2011 136
139. creating a single sanbox
make_sandbox
/path/to/mysql-X.X.XX-OS.tar.gz
Monday, April 11, 2011 139
140. using a single sanbox
# after
# make_sandbox
# /path/to/mysql-X.X.XX-OS.tar.gz
$ cd $SANDBOX_HOME/msb_X_X_XX
$ ./use
Monday, April 11, 2011 140
141. creating a single sanbox
with a specific options file
make_sandbox
/path/to/mysql-X.X.XX-OS.tar.gz
--my_file=/path/to/my.cnf
Monday, April 11, 2011 141
142. easily create a sandbox after the first
one
The long way
$ cd $HOME/opt/mysql # $SANDBOX_BINARY
$ gunzip -c
/path/to/mysql-5.1.34-osx10.5-x86.tar.gz
| tar -xf -
$ mv mysql-5.1.34-osx10.5-x86 5.1.34
$ make sandbox 5.1.34
Monday, April 11, 2011 142
143. easily create a sandbox after the first
one
The short way
$ make_sandbox
path/to/mysql-5.1.34-osx10.5-x86.tar.gz
--export_binaries
Monday, April 11, 2011 143
144. starting a single sanbox
$ cd $SANDBOX_HOME/msb_X_X_XX
$ ./start
Monday, April 11, 2011 144
145. starting a single sanbox
with temporary options
$ cd $SANDBOX_HOME/msb_X_X_XX
$ ./start --option=value
$ ./restart --option=value
$ ./start --key-buffer=20000000
Monday, April 11, 2011 145
146. creating a sandbox with custom port
and directory
$ make_sandbox 5.1.34
--sandbox_port=7800
--sandbox_directory=mickeymouse
Monday, April 11, 2011 146
147. creating a sandbox with automatic
port checking
$ make_sandbox 5.1.34 --check_port
# if 5.1.34 is free
# port=5134
# directory=msb_5_1_34
# else
# port=5135 (or the first free)
# directory=msb_5_1_34_a
Monday, April 11, 2011 147
148. create a replication sandbox
$ make_replication_sandbox
path/to/mysql-5.1.34-osx10.5-x86.tar.gz
Monday, April 11, 2011 148
149. create a circular replication sandbox
$ make_replication_sandbox
--circular=4
path/to/mysql-5.1.34-osx10.5-x86.tar.gz
Monday, April 11, 2011 149
150. changing port to an existing sandbox
$ sbtool -o port
-s /path/to/source/sandbox
--new_port=XXXX
Monday, April 11, 2011 150
151. installing the innodb plugin
$ sbtool -o plugin
--plugin=innodb
-s /path/to/source/sandbox
Monday, April 11, 2011 151
152. creating a replication sandbox with
new base port
$ make_replication_sandbox
--replication_directory=newwdir
--check_base_port 5.0.79
# Creates a replication directory under
# $SANDBOX_HOME/newdir
# The previous one is preserved.
# No conflicts happen
Monday, April 11, 2011 152
153. DA
AG
EN Replaying binary logs
Monday, April 11, 2011 153
154. DA
AG
EN Filtering binary logs
Monday, April 11, 2011 154
156. maatkit
• mk-heartbeat Monitor MySQL replication delay.
• mk-purge-logs Purge binary logs on a master based on purge rules.
• mk-slave-delay Make a MySQL slave server lag behind its master.
• mk-slave-find Find and print replication hierarchy tree of MySQL
slaves.
• mk-slave-move Move a MySQL slave around in the replication
hierarchy.
• mk-slave-prefetch Pipeline relay logs on a MySQL slave to pre-warm
caches.
• mk-slave-restart Watch and restart MySQL replication after errors.
• mk-table-checksum Perform an online replication consistency check,
or checksum MySQL tables efficiently on one or many servers.
• mk-table-sync Synchronize MySQL table data efficiently.
• http://www.maatkit.org/
Monday, April 11, 2011 156
158. Facebook Online Schema Change
• Based on one of Shlomi Noach OpenArk utilities
• Developed in PHP by Facebook
• http://www.facebook.com/note.php?note_id=430801045932
Monday, April 11, 2011 158
160. openark
• oak-get-slave-lag: print slave replication lag and
terminate with respective exit code.
• oak-online-alter-table: perform a non-blocking ALTER
TABLE operation.
• oak-purge-master-logs: purge master logs, depending
on the state of replicating slaves.
• oak-show-replication-status: show how far behind are
replicating slaves on a given master.
• http://code.openark.org/forge/openark-kit
Monday, April 11, 2011 160
161. DA
AG
EN Check replication speed
Monday, April 11, 2011 161
162. check replication speed
• Write a time to the master
• Retrieve it from the slave
• Compare times
• http://forge.mysql.com/tools/tool.php?
id=5
Monday, April 11, 2011 162
163. DA
AG
EN Replication outside the box
• Seamless failover
• Parallel replication
• Multiple source replication
• Multiple master replication
Monday, April 11, 2011 163
164. Advanced MySQL
Replication for the
masses
Monday, April 11, 2011 164
165. Once Upon A Time,
In The Life Of A
Database
Consultant ...
Monday, April 11, 2011 165
166. The story of a steel
foundry
• Used MySQL databases to store
production monitoring data
• Inserted a zillion records per second.
• Slaves often lagged behind
Monday, April 11, 2011 166
167. transaction
transaction
MySQL transaction
transaction
transaction
DBMS transaction
transaction
transaction
transaction
transaction
transaction
transaction
BINARY LOG
ns act act tio n
tra ansio ns sac ntio
ac ion ion n
ac
nr tr tra aniton
tsa a n csio
nts ct n
rn a o
taa scti
r
trnsa
n
n
tio
ct
tra
tra
REPLICATION
transaction
transaction
MySQL
DBMS
Monday, April 11, 2011 167
168. The story of a shoe
maker
• Had a successful business, spread to a
dozen stores.
• Needed to aggregate the data from the
stores in his headquarters database.
Monday, April 11, 2011 168
169. MySQL
MySQL
DBMS MySQL
DBMS
DBMS
store
store store
MySQL
DBMS
headquarters
Monday, April 11, 2011 169
170. The story of a widgets
seller
• Had a successful business, designed for one
server.
• Products were created in several sites.
• Needed to allow insertions from more
than one master at the time
Monday, April 11, 2011 170
171. MySQL MySQL
DBMS DBMS
master master
MySQL MySQL
DBMS DBMS
master master
Monday, April 11, 2011 171
172. All these stories tell us:
Nice dream, but MySQL
can’t do it
Monday, April 11, 2011 172
176. What can it do?
• Easy failover
• Multiple masters
• Multiple sources to a single slave
• Parallel replication
• Replicate to Oracle and PostgreSQL database
Monday, April 11, 2011 176
189. The steel foundry dream
or parallel replication
From here ...
Monday, April 11, 2011 189
190. The steel foundry dream
or parallel replication
To here.
Monday, April 11, 2011 190
191. Parallel replication facts
• Sharded by database
• Good choice for slave lag problems
• Bad choice for single database projects
Monday, April 11, 2011 191
213. Installation:
the long way
• Get the binaries
• Expand the tarball
• Check the requirements in the manual
• Run ./configure
• Run ./configure-service
Monday, April 11, 2011 213
214. Installation:
the quick way
• Get the tarball
• Get the tungsten deployer from Google
Code
• Sit back and enjoy!
Monday, April 11, 2011 214
216. replicator
• It’s the service provider
• You launch it once when you start
• You may restart it when you change config
Monday, April 11, 2011 216
217. trepctl
• Tungsten Replicator ConTroLler
• It’s the driving seat for your replication
• You can start, update, and stop services
• You can get specific info
Monday, April 11, 2011 217
218. thl
• Transaction History List
• Gives you access to the Tungsten relay logs
Monday, April 11, 2011 218
219. DA
AG
EN More info on replication
Monday, April 11, 2011 219
225. Talks
• Monday, 1:30pm “Learn how to cure replication
deprivation with Tungsten”
• Tuesday 10:50am “Replication Update”
• Tuesday 2:00pm “Diagnosing replication failures”
• Tuesday 3:05 “Advanced replication monitoring”
• Thursday 10:50am “Replication for Availability &
Durability with MySQL and Amazon RDS”
• Thursday 11:55am “Build your own PaaS for MySQL
with Tungsten Enterprise”
• Thursday 2:00pm “Error detection and correction with
MySQL Replication”
• Thursday 2:50pm “Performance comparisons and
trade-offs for various MySQL replication schemes”
Monday, April 11, 2011 225