Your SlideShare is downloading. ×
Yet Another Replication Tool: RubyRep
Yet Another Replication Tool: RubyRep
Yet Another Replication Tool: RubyRep
Yet Another Replication Tool: RubyRep
Yet Another Replication Tool: RubyRep
Yet Another Replication Tool: RubyRep
Yet Another Replication Tool: RubyRep
Yet Another Replication Tool: RubyRep
Yet Another Replication Tool: RubyRep
Yet Another Replication Tool: RubyRep
Yet Another Replication Tool: RubyRep
Yet Another Replication Tool: RubyRep
Yet Another Replication Tool: RubyRep
Yet Another Replication Tool: RubyRep
Yet Another Replication Tool: RubyRep
Yet Another Replication Tool: RubyRep
Yet Another Replication Tool: RubyRep
Yet Another Replication Tool: RubyRep
Yet Another Replication Tool: RubyRep
Yet Another Replication Tool: RubyRep
Yet Another Replication Tool: RubyRep
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Yet Another Replication Tool: RubyRep

2,981

Published on

Presentation given PgEast 2010 on 26th March 2010.

Presentation given PgEast 2010 on 26th March 2010.

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,981
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
44
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Yet Another Replication Tool RubyRep / Denish Patel Database Architect Friday, March 26, 2010 1
  • 2. Who am I? • With OmniTi for more than 3 years • Manage high traffic database systems • Replication database system deployments • Not a core hacker of RubyRep • “Oh, We are hiring!!” • Contact : denish@omniti.com Friday, March 26, 2010 2
  • 3. Next 30 minutes .. • Replication • Various Tools • Slony? Why another tool ? • RubyRep • Install • Features • Examples • Tweaking replication policies Friday, March 26, 2010 3
  • 4. Replication • Types • Synchronous • Semi-synchronous (O My!) • Asynchronous • Methods • Master - Slave • Master - Master • Needs • Horizontal Scaling • Standby Database • Backup Database Friday, March 26, 2010 4
  • 5. Tools Program Type Method Based on PgCluster -II Synchronous M-M Shared Disk Slony-I Asynchronous M-S Trigger Bucardo Asynchronous M-M, M-S Trigger Londiste Asynchronous M-S Trigger Mammoth Asynchronous M-S Log RubyRep Asynchronous M-M, M-S Trigger Friday, March 26, 2010 5
  • 6. Why not Slony? • Replicated tables MUST need PK or UK • Doesn’t support large objects • Doesn’t support synchronizing tables outside of replication • Limitations of version compatibility • Difficult to setup • Doesn’t support Master - Master • Difficult to monitor and manage Friday, March 26, 2010 6
  • 7. RubyRep Toolset • Scan • Sync • Replicate Friday, March 26, 2010 7
  • 8. RubyRep Key Features • Easy to install, setup, configure and monitor • Platform independent • Table design independent • Simple PK • Combined PK • No PK • Supports large objects i.e bytea, tsvector • Database Version Independent • Database independent • PostgreSQL • MySQL Friday, March 26, 2010 8
  • 9. Install • Standard Ruby Version • Install Ruby Environment , Install DB drivers , Install RubyRep • Difficult to install • Slower performance • JRuby Version • Install JVM, Download and Extract gzip file • Easy to install • Faster performance Friday, March 26, 2010 9
  • 10. Scan • Features • Compare “Left” and “Right” Databases/Tables • Customize batch size • Load on client server • Steps: • Generate config file • ./rubyrep generate omniti.conf • Configure • Run the Scan • ./rubyrep scan -c omniti.conf • Production Results • Users table with 60M rows and 45 columns • 3 hours • Running from monitor server Friday, March 26, 2010 10
  • 11. Scan • To scan all tables starting with ‘c’: • rubyrep scan -c omniti.conf /^c/ • Scan with default options will produce output like this: • clients 100% ......................... 0 • clients_details 100% ......................... 5 • Options • -s, --summary[=detailed] • -d, --detailed[=mode] • full , keys, diff • -b, --progress-bar[=length] • -c, --config=CONFIG_FILE • --help Friday, March 26, 2010 11
  • 12. Config RR::Initializer::run do |config|config.left = { :adapter => 'postgresql', # or 'mysql' :database => 'omniti', :username => 'dba', :password => 'securepasswd', :host => '127.0.0.1', :schema_search_path => 'omniti’ } config.right = { :adapter => 'postgresql', :database => 'omniti', :username => 'dba', :password => 'securepasswd', :host => '127.0.0.1', :port => '5484' } config.include_tables 'users,users_backup' config.include_tables /^o/ # regexp matches all tables starting with o end Friday, March 26, 2010 12
  • 13. Sync • Compare and Sync • :sync_conflict_handling - :ignore (default), :left_wins, :right_wins • :sync_record_handling • :left_record_handling , :right_record_handling • :ignore, :delete, :insert(default) • Logging • :ignored_changes , :all_changes , • :ignored_conflicts , :all_conflicts • Others • :row_buffer_size , :commit_frequency • http://www.rubyrep.org/configuration.html Friday, March 26, 2010 13
  • 14. Sync RR::Initializer::run do |config|config.left = { :adapter => 'postgresql', # or 'mysql' :database => 'omniti', :username => 'dba', :password => 'securepasswd', :host => '127.0.0.1', :schema_search_path => 'omniti’ } config.right = { :adapter => 'postgresql', :database => 'omniti', :username => 'dba', :password => 'securepasswd', :host => '127.0.0.1', :port => '5484' } # To match all tables except those ending with ‘backup’: config.include_tables /./ config.exclude_tables /backup$/ #exclude tables ending with backup end Friday, March 26, 2010 14
  • 15. Replicate • Replicate Database/Tables • Global vs Table specific settings • rubyrep replicate -c omniti.conf • Setup necessary infrastructure tables • Setup necessary triggers for the replicated tables • Initial scan is executed and synched • Tweak sequences • Left -Odd , Right -Even for M-M replication • Default M-M replication setting • :replicate_record _handling = :replicate • Replication Conflict handling • :later_wins, :earlier_wins • Automatically Sync newly created table(Restart required) • Uninstall • rubyrep uninstall -c omniti.conf Friday, March 26, 2010 15
  • 16. Replicate RR::Initializer::run do |config|config.left = { :adapter => 'postgresql', # or 'mysql' :database => 'omniti', :username => 'dba', :password => 'securepasswd', :host => '127.0.0.1', :schema_search_path => 'omniti‘ } config.right = { :adapter => 'postgresql', :database => 'omniti', :username => 'dba', :password => 'securepasswd', :host => '127.0.0.1', :port => '5484' } config.options[:sync_conflict_handling] = :left_wins config.options[:replication_conflict_handling] = :left_wins config.add_table_options 'admin', :sync_conflict_handling => :right_wins, :replication_conflict_handling => :right_wins config.include_tables /./ end Friday, March 26, 2010 16
  • 17. Master-Slave Config config.include_tables /./ # regexp matching all tables in the database config.options[:auto_key_limit] = 40 config.options[:adjust_sequences] = false config.options[:sequence_increment] = 1 config.options[:replication_conflict_handling] = :left_wins config.options[:logged_replication_events] = [ :ignored_changes, :ignored_conflicts ] end Friday, March 26, 2010 17
  • 18. Customize as needed ... • Tables without PK • config.options[:auto_key_limit] = 2 • config.add_table_options /_/, :auto_key_limit => 2 • config.include_tables 'articles_tags', :key => ['article_id', 'tag_id'] • Checks before and after • :before_table_sync => "SET foreign_key_checks = 0" • :after_table_sync => "SET foreign_key_checks = 1" • Low bandwidth with RubyRep Proxy • Benefits only Scan and Compare ; only difference is transferred • :proxy_host => '172.16.1.5', , :proxy_port => '9876' • Cascading replication • :rep_prefix to use separate triggers and tables Friday, March 26, 2010 18
  • 19. References • Installation Notes • http://rubyforge.org/frs/?group_id=7932 • User Guide • http://www.rubyrep.org/documentation.html • API Documentation • http://rubyrep.rubyforge.org/ • Tutorials • http://www.rubyrep.org/tutorial.html • Blogs • http://denishjpatel.blogspot.com/2009/08/yet-another- postgresql-replication-tool.html Friday, March 26, 2010 19
  • 20. Thanks • Arndt Lehmann • Thank you!! Friday, March 26, 2010 20
  • 21. Questions ? Friday, March 26, 2010 21

×