Denish Patel, a database architect at OmniTi, gave a presentation on the RubyRep database replication tool. RubyRep is an asynchronous master-master and master-slave replication tool that is easy to install, platform independent, and supports large objects and tables without primary keys. It was developed as an alternative to Slony because Slony has limitations such as requiring tables to have primary keys and not supporting large objects or master-master replication. The presentation covered how to install, configure, scan, sync, and replicate databases using RubyRep.

1. Yet Another Replication Tool
RubyRep
/ Denish Patel
Database Architect
Friday, March 26, 2010 1

2. Who am I?
• With OmniTi for more than 3 years
• Manage high traffic database systems
• Replication database system deployments
• Not a core hacker of RubyRep
• “Oh, We are hiring!!”
• Contact : denish@omniti.com
Friday, March 26, 2010 2

3. Next 30 minutes ..
• Replication
• Various Tools
• Slony? Why another tool ?
• RubyRep
• Install
• Features
• Examples
• Tweaking replication policies
Friday, March 26, 2010 3

4. Replication
• Types
• Synchronous
• Semi-synchronous (O My!)
• Asynchronous
• Methods
• Master - Slave
• Master - Master
• Needs
• Horizontal Scaling
• Standby Database
• Backup Database
Friday, March 26, 2010 4

5. Tools
Program Type Method Based on
PgCluster -II Synchronous M-M Shared Disk
Slony-I Asynchronous M-S Trigger
Bucardo Asynchronous M-M, M-S Trigger
Londiste Asynchronous M-S Trigger
Mammoth Asynchronous M-S Log
RubyRep Asynchronous M-M, M-S Trigger
Friday, March 26, 2010 5

6. Why not Slony?
• Replicated tables MUST need PK or UK
• Doesn’t support large objects
• Doesn’t support synchronizing tables outside of replication
• Limitations of version compatibility
• Difficult to setup
• Doesn’t support Master - Master
• Difficult to monitor and manage
Friday, March 26, 2010 6

7. RubyRep Toolset
• Scan
• Sync
• Replicate
Friday, March 26, 2010 7

8. RubyRep Key Features
• Easy to install, setup, configure and monitor
• Platform independent
• Table design independent
• Simple PK
• Combined PK
• No PK
• Supports large objects i.e bytea, tsvector
• Database Version Independent
• Database independent
• PostgreSQL
• MySQL
Friday, March 26, 2010 8

9. Install
• Standard Ruby Version
• Install Ruby Environment , Install DB drivers , Install RubyRep
• Difficult to install
• Slower performance
• JRuby Version
• Install JVM, Download and Extract gzip file
• Easy to install
• Faster performance
Friday, March 26, 2010 9

10. Scan
• Features
• Compare “Left” and “Right” Databases/Tables
• Customize batch size
• Load on client server
• Steps:
• Generate config file
• ./rubyrep generate omniti.conf
• Configure
• Run the Scan
• ./rubyrep scan -c omniti.conf
• Production Results
• Users table with 60M rows and 45 columns
• 3 hours
• Running from monitor server
Friday, March 26, 2010 10

11. Scan
• To scan all tables starting with ‘c’:
• rubyrep scan -c omniti.conf /^c/
• Scan with default options will produce output like this:
• clients 100% ......................... 0
• clients_details 100% ......................... 5
• Options
• -s, --summary[=detailed]
• -d, --detailed[=mode]
• full , keys, diff
• -b, --progress-bar[=length]
• -c, --config=CONFIG_FILE
• --help
Friday, March 26, 2010 11

12. Config
RR::Initializer::run do |config|config.left = {
:adapter => 'postgresql', # or 'mysql'
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:schema_search_path => 'omniti’ }
config.right = {
:adapter => 'postgresql',
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:port => '5484'
}
config.include_tables 'users,users_backup'
config.include_tables /^o/ # regexp matches all tables starting with o
end
Friday, March 26, 2010 12

13. Sync
• Compare and Sync
• :sync_conflict_handling - :ignore (default), :left_wins, :right_wins
• :sync_record_handling
• :left_record_handling , :right_record_handling
• :ignore, :delete, :insert(default)
• Logging
• :ignored_changes , :all_changes ,
• :ignored_conflicts , :all_conflicts
• Others
• :row_buffer_size , :commit_frequency
• http://www.rubyrep.org/configuration.html
Friday, March 26, 2010 13

14. Sync
RR::Initializer::run do |config|config.left = {
:adapter => 'postgresql', # or 'mysql'
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:schema_search_path => 'omniti’ }
config.right = {
:adapter => 'postgresql',
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:port => '5484' }
# To match all tables except those ending with ‘backup’:
config.include_tables /./
config.exclude_tables /backup$/ #exclude tables ending with backup
end
Friday, March 26, 2010 14

15. Replicate
• Replicate Database/Tables
• Global vs Table specific settings
• rubyrep replicate -c omniti.conf
• Setup necessary infrastructure tables
• Setup necessary triggers for the replicated tables
• Initial scan is executed and synched
• Tweak sequences
• Left -Odd , Right -Even for M-M replication
• Default M-M replication setting
• :replicate_record _handling = :replicate
• Replication Conflict handling
• :later_wins, :earlier_wins
• Automatically Sync newly created table(Restart required)
• Uninstall
• rubyrep uninstall -c omniti.conf
Friday, March 26, 2010 15

16. Replicate
RR::Initializer::run do |config|config.left = {
:adapter => 'postgresql', # or 'mysql'
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:schema_search_path => 'omniti‘ }
config.right = {
:adapter => 'postgresql',
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:port => '5484' }
config.options[:sync_conflict_handling] = :left_wins
config.options[:replication_conflict_handling] = :left_wins
config.add_table_options 'admin',
:sync_conflict_handling => :right_wins,
:replication_conflict_handling => :right_wins
config.include_tables /./
end
Friday, March 26, 2010 16

17. Master-Slave Config
config.include_tables /./ # regexp matching all tables in the database
config.options[:auto_key_limit] = 40
config.options[:adjust_sequences] = false
config.options[:sequence_increment] = 1
config.options[:replication_conflict_handling] = :left_wins
config.options[:logged_replication_events] = [
:ignored_changes,
:ignored_conflicts
]
end
Friday, March 26, 2010 17

18. Customize as needed ...
• Tables without PK
• config.options[:auto_key_limit] = 2
• config.add_table_options /_/, :auto_key_limit => 2
• config.include_tables 'articles_tags', :key => ['article_id', 'tag_id']
• Checks before and after
• :before_table_sync => "SET foreign_key_checks = 0"
• :after_table_sync => "SET foreign_key_checks = 1"
• Low bandwidth with RubyRep Proxy
• Benefits only Scan and Compare ; only difference is transferred
• :proxy_host => '172.16.1.5', , :proxy_port => '9876'
• Cascading replication
• :rep_prefix to use separate triggers and tables
Friday, March 26, 2010 18

19. References
• Installation Notes
• http://rubyforge.org/frs/?group_id=7932
• User Guide
• http://www.rubyrep.org/documentation.html
• API Documentation
• http://rubyrep.rubyforge.org/
• Tutorials
• http://www.rubyrep.org/tutorial.html
• Blogs
• http://denishjpatel.blogspot.com/2009/08/yet-another-
postgresql-replication-tool.html
Friday, March 26, 2010 19

20. Thanks
• Arndt Lehmann
• Thank you!!
Friday, March 26, 2010 20

21. Questions ?
Friday, March 26, 2010 21