Yet Another Replication Tool
                          RubyRep




                           /               Denish Patel...
Who am I?




                     •   With OmniTi for more than 3 years

                     •   Manage high traffic dat...
Next 30 minutes ..



                     •   Replication

                     •   Various Tools

                     •...
Replication

                     •   Types

                         •   Synchronous

                             •   Se...
Tools

                     Program           Type        Method     Based on

                   PgCluster -II Synchronou...
Why not Slony?




                     •   Replicated tables MUST need PK or UK

                     •   Doesn’t support...
RubyRep Toolset




                     •   Scan

                     •   Sync

                     •   Replicate




F...
RubyRep Key Features

                     •   Easy to install, setup, configure and monitor

                     •   Pla...
Install



                     •   Standard Ruby Version

                         •   Install Ruby Environment , Install...
Scan

                     •   Features

                         •   Compare “Left” and “Right” Databases/Tables

       ...
Scan

                     •   To scan all tables starting with ‘c’:

                         •   rubyrep scan -c omniti....
Config
                         RR::Initializer::run do |config|config.left = {

                         :adapter => 'pos...
Sync


                         •   Compare and Sync

                         •   :sync_conflict_handling - :ignore (defa...
Sync
                         RR::Initializer::run do |config|config.left = {

                         :adapter => 'postg...
Replicate

                     •   Replicate Database/Tables

                     •   Global vs Table specific settings
...
Replicate
                         RR::Initializer::run do |config|config.left = {

                         :adapter => '...
Master-Slave Config


                         config.include_tables /./ # regexp matching all tables in the database

   ...
Customize as needed ...

                     •   Tables without PK

                         •   config.options[:auto_key...
References

                     •   Installation Notes

                         •   http://rubyforge.org/frs/?group_id=7...
Thanks




                     •   Arndt Lehmann

                     •   Thank you!!




Friday, March 26, 2010        ...
Questions ?




Friday, March 26, 2010                 21
Upcoming SlideShare
Loading in …5
×

Yet Another Replication Tool: RubyRep

3,530 views

Published on

Presentation given PgEast 2010 on 26th March 2010.

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
3,530
On SlideShare
0
From Embeds
0
Number of Embeds
94
Actions
Shares
0
Downloads
45
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Yet Another Replication Tool: RubyRep

  1. 1. Yet Another Replication Tool RubyRep / Denish Patel Database Architect Friday, March 26, 2010 1
  2. 2. Who am I? • With OmniTi for more than 3 years • Manage high traffic database systems • Replication database system deployments • Not a core hacker of RubyRep • “Oh, We are hiring!!” • Contact : denish@omniti.com Friday, March 26, 2010 2
  3. 3. Next 30 minutes .. • Replication • Various Tools • Slony? Why another tool ? • RubyRep • Install • Features • Examples • Tweaking replication policies Friday, March 26, 2010 3
  4. 4. Replication • Types • Synchronous • Semi-synchronous (O My!) • Asynchronous • Methods • Master - Slave • Master - Master • Needs • Horizontal Scaling • Standby Database • Backup Database Friday, March 26, 2010 4
  5. 5. Tools Program Type Method Based on PgCluster -II Synchronous M-M Shared Disk Slony-I Asynchronous M-S Trigger Bucardo Asynchronous M-M, M-S Trigger Londiste Asynchronous M-S Trigger Mammoth Asynchronous M-S Log RubyRep Asynchronous M-M, M-S Trigger Friday, March 26, 2010 5
  6. 6. Why not Slony? • Replicated tables MUST need PK or UK • Doesn’t support large objects • Doesn’t support synchronizing tables outside of replication • Limitations of version compatibility • Difficult to setup • Doesn’t support Master - Master • Difficult to monitor and manage Friday, March 26, 2010 6
  7. 7. RubyRep Toolset • Scan • Sync • Replicate Friday, March 26, 2010 7
  8. 8. RubyRep Key Features • Easy to install, setup, configure and monitor • Platform independent • Table design independent • Simple PK • Combined PK • No PK • Supports large objects i.e bytea, tsvector • Database Version Independent • Database independent • PostgreSQL • MySQL Friday, March 26, 2010 8
  9. 9. Install • Standard Ruby Version • Install Ruby Environment , Install DB drivers , Install RubyRep • Difficult to install • Slower performance • JRuby Version • Install JVM, Download and Extract gzip file • Easy to install • Faster performance Friday, March 26, 2010 9
  10. 10. Scan • Features • Compare “Left” and “Right” Databases/Tables • Customize batch size • Load on client server • Steps: • Generate config file • ./rubyrep generate omniti.conf • Configure • Run the Scan • ./rubyrep scan -c omniti.conf • Production Results • Users table with 60M rows and 45 columns • 3 hours • Running from monitor server Friday, March 26, 2010 10
  11. 11. Scan • To scan all tables starting with ‘c’: • rubyrep scan -c omniti.conf /^c/ • Scan with default options will produce output like this: • clients 100% ......................... 0 • clients_details 100% ......................... 5 • Options • -s, --summary[=detailed] • -d, --detailed[=mode] • full , keys, diff • -b, --progress-bar[=length] • -c, --config=CONFIG_FILE • --help Friday, March 26, 2010 11
  12. 12. Config RR::Initializer::run do |config|config.left = { :adapter => 'postgresql', # or 'mysql' :database => 'omniti', :username => 'dba', :password => 'securepasswd', :host => '127.0.0.1', :schema_search_path => 'omniti’ } config.right = { :adapter => 'postgresql', :database => 'omniti', :username => 'dba', :password => 'securepasswd', :host => '127.0.0.1', :port => '5484' } config.include_tables 'users,users_backup' config.include_tables /^o/ # regexp matches all tables starting with o end Friday, March 26, 2010 12
  13. 13. Sync • Compare and Sync • :sync_conflict_handling - :ignore (default), :left_wins, :right_wins • :sync_record_handling • :left_record_handling , :right_record_handling • :ignore, :delete, :insert(default) • Logging • :ignored_changes , :all_changes , • :ignored_conflicts , :all_conflicts • Others • :row_buffer_size , :commit_frequency • http://www.rubyrep.org/configuration.html Friday, March 26, 2010 13
  14. 14. Sync RR::Initializer::run do |config|config.left = { :adapter => 'postgresql', # or 'mysql' :database => 'omniti', :username => 'dba', :password => 'securepasswd', :host => '127.0.0.1', :schema_search_path => 'omniti’ } config.right = { :adapter => 'postgresql', :database => 'omniti', :username => 'dba', :password => 'securepasswd', :host => '127.0.0.1', :port => '5484' } # To match all tables except those ending with ‘backup’: config.include_tables /./ config.exclude_tables /backup$/ #exclude tables ending with backup end Friday, March 26, 2010 14
  15. 15. Replicate • Replicate Database/Tables • Global vs Table specific settings • rubyrep replicate -c omniti.conf • Setup necessary infrastructure tables • Setup necessary triggers for the replicated tables • Initial scan is executed and synched • Tweak sequences • Left -Odd , Right -Even for M-M replication • Default M-M replication setting • :replicate_record _handling = :replicate • Replication Conflict handling • :later_wins, :earlier_wins • Automatically Sync newly created table(Restart required) • Uninstall • rubyrep uninstall -c omniti.conf Friday, March 26, 2010 15
  16. 16. Replicate RR::Initializer::run do |config|config.left = { :adapter => 'postgresql', # or 'mysql' :database => 'omniti', :username => 'dba', :password => 'securepasswd', :host => '127.0.0.1', :schema_search_path => 'omniti‘ } config.right = { :adapter => 'postgresql', :database => 'omniti', :username => 'dba', :password => 'securepasswd', :host => '127.0.0.1', :port => '5484' } config.options[:sync_conflict_handling] = :left_wins config.options[:replication_conflict_handling] = :left_wins config.add_table_options 'admin', :sync_conflict_handling => :right_wins, :replication_conflict_handling => :right_wins config.include_tables /./ end Friday, March 26, 2010 16
  17. 17. Master-Slave Config config.include_tables /./ # regexp matching all tables in the database config.options[:auto_key_limit] = 40 config.options[:adjust_sequences] = false config.options[:sequence_increment] = 1 config.options[:replication_conflict_handling] = :left_wins config.options[:logged_replication_events] = [ :ignored_changes, :ignored_conflicts ] end Friday, March 26, 2010 17
  18. 18. Customize as needed ... • Tables without PK • config.options[:auto_key_limit] = 2 • config.add_table_options /_/, :auto_key_limit => 2 • config.include_tables 'articles_tags', :key => ['article_id', 'tag_id'] • Checks before and after • :before_table_sync => "SET foreign_key_checks = 0" • :after_table_sync => "SET foreign_key_checks = 1" • Low bandwidth with RubyRep Proxy • Benefits only Scan and Compare ; only difference is transferred • :proxy_host => '172.16.1.5', , :proxy_port => '9876' • Cascading replication • :rep_prefix to use separate triggers and tables Friday, March 26, 2010 18
  19. 19. References • Installation Notes • http://rubyforge.org/frs/?group_id=7932 • User Guide • http://www.rubyrep.org/documentation.html • API Documentation • http://rubyrep.rubyforge.org/ • Tutorials • http://www.rubyrep.org/tutorial.html • Blogs • http://denishjpatel.blogspot.com/2009/08/yet-another- postgresql-replication-tool.html Friday, March 26, 2010 19
  20. 20. Thanks • Arndt Lehmann • Thank you!! Friday, March 26, 2010 20
  21. 21. Questions ? Friday, March 26, 2010 21

×