©Continuent 2014
Tungsten University: 
Set Up and Operate
Tungsten Replication
Robert Hodges, CEO
©Continuent 2014
Introducing Continuent
2
• The leading provider of clustering and
replication for open source DBMS
• Our ...
©Continuent 2014
Introducing Tungsten Replicator
3
©Continuent 2014
Tungsten Replicator Overview
4
Master
(Transactions + Metadata)
Slave
THL
DBMS	

Logs
Replicator
(Transac...
©Continuent 2014
Master Replication Service
5
Extract Filter Apply
Stage
Extract Filter Apply
Stage
Pipeline
MySQL	

Maste...
©Continuent 2014
Slave Replication Service
6
Extract Filter Apply
Stage
Extract Filter Apply
Stage
Extract Filter Apply
St...
©Continuent 2014
EC2 Installation Using Github project
1.Set up Github vagrant project
git clone https://github.com/contin...
©Continuent 2014
Tpm installation overview
8
/opt/continuent/etc/tungsten/tungsten.ini
check prereqs	

[copy code]	

config...
©Continuent 2014
What Is Tungsten Doing?
9
©Continuent 2014
Commands to Check Replication
• trepctl services -- All services
• trepctl status -- Overall pipeline sta...
©Continuent 2014
Changing Replicator Configuration
11
©Continuent 2014
Reconfiguring a Replicator
• Edit /etc/tungsten/tungsten.ini
• No restart on local host
trepctl offline!
...
©Continuent 2014
Enabling Parallel Replication
13
©Continuent 2014
Parallel Replication of Sharded Data
14
Extract Filter Apply
Stage
Extract Filter Apply
Stage
Stage
Pipel...
©Continuent 2014
Fixing Problems
15
©Continuent 2014
The Scene of the Crime
16
trepctl -host db2 status
Processing status command...
NAME VALUE
---- -----
......
©Continuent 2014
Correcting problems
1.Look at the transaction (thl list -seqno NNN)
2.Look at the replicator message log
...
©Continuent 2014
Tungsten Logs
18
©Continuent 2014
Simple Facts about Logs
• Replicator log = THL (Transaction History Log)
• One THL per replication servic...
©Continuent 2014
Resetting Replication
20
©Continuent 2014
Resetting/Reinstalling Automatically
Reset
1.tpm reset
Reinstall
1.tpm uninstall --i-am-sure
2.Rerun inst...
©Continuent 2014
Resetting/Reinstalling Manually
Reset
1.Stop replicator
2.Remove THL files
3.Remove tungsten_<svc> databa...
©Continuent 2014
Backup and Restore
23
©Continuent 2014
What Backup Tools Does MySQL Have?
24
MySQL
Server
Process
File System
DBMS Files
mysqldump	

(dataset < ...
©Continuent 2014
Advanced Installation
Patterns
25
©Continuent 2014
How Do I Install Fan-In Replication?
26
db1
Replicator
db3
Service db1
db2
Replicator
Service db2
Replica...
©Continuent 2014
How Do I install Multi-Master?
27
db1 Replicator
Service db1
Service db2
Service db3
db3
Service db1
Serv...
©Continuent 2014
Tpm Understands Topologies!
• Use --topology argument!
28
Fan-In StarAll-Masters
--topology=fan-in --topo...
©Continuent 2014
Tpm install from staging directory
29
db1
db2
db3
Staging copy 	

of files
check prereqs	

copy code	

con...
©Continuent 2014
Commands to Perform Staging Install
30
$ wget http://downloads.tungsten-replicator.org/download.php?
file...
©Continuent 2014
Wrapping Up
31
©Continuent 2014
Tungsten Replicator is 100% Open
Source
32
• Project home:
http://code.google.com/p/tungsten-replicator/
...
©Continuent 2014
Heterogeneous
OracleMySQL Oracle MySQL MySQL
star
master-slave
fan-in slave all-masters
MySQLOracle Oracle
©Continuent 2014
We Do Clustering Too
34
Tungsten clusters combine off-
the-shelf open source DBMS
servers into data servi...
©Continuent 2014
Conclusion
35
• Tungsten Replicator is an advanced, open source
replicator
• Use tpm to install either fr...
©Continuent 2014
Continuent Web Page:	

http://www.continuent.com	

!
Tungsten Replicator:	

http://code.google.com/p/tung...
Upcoming SlideShare
Loading in...5
×

Set Up & Operate Tungsten Replicator

181

Published on

Tungsten Replicator is an innovative and reliable tool that can solve your most complex replication problems. In this webinar we introduce Replicator installation and show you how to use key Replicator features effectively with MySQL.

Course Topics:

- Checking host and MySQL prerequisites
- Downloading code from http://code.google.com/p/tungsten-replicator/
- Installation using the tpm utility
- Transaction filtering using standard filters as well as customized filters you write yourself
- Enabling and managing parallel replication
- Configuring multi-master and fan-in using multiple replication services
- Backup and restore integration
- Troubleshooting replication problems
- Logging bugs and participating in the Tungsten Replicator community

Replication is a powerful technology that takes knowledge and planning to use effectively. This webinar gives you the background that makes replication easier to set up and allows you to take full advantage of the Tungsten Replicator benefits.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
181
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Set Up & Operate Tungsten Replicator

  1. 1. ©Continuent 2014 Tungsten University:  Set Up and Operate Tungsten Replication Robert Hodges, CEO
  2. 2. ©Continuent 2014 Introducing Continuent 2 • The leading provider of clustering and replication for open source DBMS • Our Product: Continuent Tungsten • Clustering - Commercial-grade HA, performance scaling and data management for MySQL • Replication - Flexible, high-performance data movement
  3. 3. ©Continuent 2014 Introducing Tungsten Replicator 3
  4. 4. ©Continuent 2014 Tungsten Replicator Overview 4 Master (Transactions + Metadata) Slave THL DBMS Logs Replicator (Transactions + Metadata) THLReplicator Download transactions via network Apply using JDBC
  5. 5. ©Continuent 2014 Master Replication Service 5 Extract Filter Apply Stage Extract Filter Apply Stage Pipeline MySQL Master Transaction History Log In-Memory Queue Slave ReplicatorsBinlog tcp/ip
  6. 6. ©Continuent 2014 Slave Replication Service 6 Extract Filter Apply Stage Extract Filter Apply Stage Extract Filter Apply Stage Pipeline Transaction History Log In-Memory Queue Slave DBMS Master Replicator tcp/ip
  7. 7. ©Continuent 2014 EC2 Installation Using Github project 1.Set up Github vagrant project git clone https://github.com/continuent/ continuent-vagrant 2.Pick EC2 Vagrantfile and fill in keys 3.Copy examples/MasterSlave/default.pp to manifests/ 4.Run launch.sh 7 (Follow directions in README.md)
  8. 8. ©Continuent 2014 Tpm installation overview 8 /opt/continuent/etc/tungsten/tungsten.ini check prereqs [copy code] configure topology [defaults]! user=tungsten! home-directory=/opt/continuent! datasource-user=tungsten! datasource-password=secret! skip-validation-check=MySQLPermissionsCheck! start-and-report=true! ! [university]! master=db1! slaves=db2,db3! topology=master-slave! tpm update (Installation)
  9. 9. ©Continuent 2014 What Is Tungsten Doing? 9
  10. 10. ©Continuent 2014 Commands to Check Replication • trepctl services -- All services • trepctl status -- Overall pipeline status • trepctl status -name stages -- Stage summary • trepctl status -name stores -- Store summary • trepctl status -name tasks -- Task detail • trepctl [-service name] heartbeat -- Is it Alive? 10
  11. 11. ©Continuent 2014 Changing Replicator Configuration 11
  12. 12. ©Continuent 2014 Reconfiguring a Replicator • Edit /etc/tungsten/tungsten.ini • No restart on local host trepctl offline! tpm update --no-restart! trepctl configure! trepctl online! ! • With automatic restart tpm update 12 Don’t change replicator configuration files directly!
  13. 13. ©Continuent 2014 Enabling Parallel Replication 13
  14. 14. ©Continuent 2014 Parallel Replication of Sharded Data 14 Extract Filter Apply Stage Extract Filter Apply Stage Stage Pipeline Transaction History Log Parallel Queue Slave DBMS Extract Filter Apply Extract Filter Apply Extract Filter Apply Master Replicator tcp/ip
  15. 15. ©Continuent 2014 Fixing Problems 15
  16. 16. ©Continuent 2014 The Scene of the Crime 16 trepctl -host db2 status Processing status command... NAME VALUE ---- ----- ... pendingError : Event application failed: seqno=1957469 fragno=0 message=java.sql.SQLException: Statement failed on slave but succeeded on master pendingErrorCode : NONE pendingErrorEventId : mysql-bin.000008:0000000461736666;0 pendingErrorSeqno : 1957469 pendingExceptionMessage: java.sql.SQLException: Statement failed on slave but succeeded on master create table trouble(id int primary key) /* ___SERVICE___ = [university] */ ... Finished status command...
  17. 17. ©Continuent 2014 Correcting problems 1.Look at the transaction (thl list -seqno NNN) 2.Look at the replicator message log 3.Fix the problem and go online OR 4.Skip over the problem and go online OR 5.Reprovision the slave 17
  18. 18. ©Continuent 2014 Tungsten Logs 18
  19. 19. ©Continuent 2014 Simple Facts about Logs • Replicator log = THL (Transaction History Log) • One THL per replication service • 2x the size of MySQL binlog • Default retention of 7 days • Take service offline to move or delete logs safely • The thl tool shows you log contents 19
  20. 20. ©Continuent 2014 Resetting Replication 20
  21. 21. ©Continuent 2014 Resetting/Reinstalling Automatically Reset 1.tpm reset Reinstall 1.tpm uninstall --i-am-sure 2.Rerun installation 21
  22. 22. ©Continuent 2014 Resetting/Reinstalling Manually Reset 1.Stop replicator 2.Remove THL files 3.Remove tungsten_<svc> databases Reinstall 1.All of the above, plus delete /opt/continuent/* 2.Rerun installation 22
  23. 23. ©Continuent 2014 Backup and Restore 23
  24. 24. ©Continuent 2014 What Backup Tools Does MySQL Have? 24 MySQL Server Process File System DBMS Files mysqldump (dataset < 10Gb) xtrabackup (10Gb < dataset < 1Tb tar/rsync (dataset < 1Tb) file system snapshot (1Tb << dataset)
  25. 25. ©Continuent 2014 Advanced Installation Patterns 25
  26. 26. ©Continuent 2014 How Do I Install Fan-In Replication? 26 db1 Replicator db3 Service db1 db2 Replicator Service db2 Replicator Service db1 Service db2
  27. 27. ©Continuent 2014 How Do I install Multi-Master? 27 db1 Replicator Service db1 Service db2 Service db3 db3 Service db1 Service db2 Service db3 db2 Replicator Service db1 Service db2 Service db3
  28. 28. ©Continuent 2014 Tpm Understands Topologies! • Use --topology argument! 28 Fan-In StarAll-Masters --topology=fan-in --topology=all-masters --topology=star
  29. 29. ©Continuent 2014 Tpm install from staging directory 29 db1 db2 db3 Staging copy of files check prereqs copy code configure topology tpm configure ... tpm update
  30. 30. ©Continuent 2014 Commands to Perform Staging Install 30 $ wget http://downloads.tungsten-replicator.org/download.php? file=tungsten-replicator-2.2.1-403.tar.gz ! $ tar -xvzf tungsten-replicator-2.2.1-403.tar.gz ! $ ln -s tungsten-replicator-2.2.1-403 tungsten ! $ tungsten/tools/tpm configure defaults --reset --user=tungsten --home-directory=/opt/continuent --datasource-user=tungsten --datasource-password=secret --skip-validation-check=MySQLPermissionsCheck --start-and-report=true ! $ tungsten/tools/tpm configure fanin --members=db1,db2,db3 --masters=db1,db2 --master-services=db1,db2 --topology=fan-in ! $ tungsten/tools/tpm update
  31. 31. ©Continuent 2014 Wrapping Up 31
  32. 32. ©Continuent 2014 Tungsten Replicator is 100% Open Source 32 • Project home: http://code.google.com/p/tungsten-replicator/ • Log bugs, find builds, post in discussion group • Documentation: https://docs.continuent.com • Install automation: https://github.com/continuent !
  33. 33. ©Continuent 2014 Heterogeneous OracleMySQL Oracle MySQL MySQL star master-slave fan-in slave all-masters MySQLOracle Oracle
  34. 34. ©Continuent 2014 We Do Clustering Too 34 Tungsten clusters combine off- the-shelf open source DBMS servers into data services with: ! • 24x7 data access • Scaling of load on replicas • Simple management commands ! ...without app changes or data migration Amazon US West apache /php GonzoPortal.com Connector Connector
  35. 35. ©Continuent 2014 Conclusion 35 • Tungsten Replicator is an advanced, open source replicator • Use tpm to install either from staging or using batch commands • Use trepctl to manage replication • Tungsten Replicator has capabilities far beyond what we showed today Use your imagination!
  36. 36. ©Continuent 2014 Continuent Web Page: http://www.continuent.com ! Tungsten Replicator: http://code.google.com/p/tungsten-replicator Our Blogs: http://scale-out-blog.blogspot.com http://datacharmer.org/blog http://www.continuent.com/news/blogs http://flyingclusters.blogspot.com/ 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009 e-mail: sales@continuent.com
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×