Presentation given PgEast 2010 on 26th March 2010.

1. Yet Another Replication Tool
RubyRep
/ Denish Patel
Database Architect
Friday, March 26, 2010 1

2. Who am I?
• With OmniTi for more than 3 years
• Manage high traffic database systems
• Replication database system deployments
• Not a core hacker of RubyRep
• “Oh, We are hiring!!”
• Contact : denish@omniti.com
Friday, March 26, 2010 2

3. Next 30 minutes ..
• Replication
• Various Tools
• Slony? Why another tool ?
• RubyRep
• Install
• Features
• Examples
• Tweaking replication policies
Friday, March 26, 2010 3

4. Replication
• Types
• Synchronous
• Semi-synchronous (O My!)
• Asynchronous
• Methods
• Master - Slave
• Master - Master
• Needs
• Horizontal Scaling
• Standby Database
• Backup Database
Friday, March 26, 2010 4

5. Tools
Program Type Method Based on
PgCluster -II Synchronous M-M Shared Disk
Slony-I Asynchronous M-S Trigger
Bucardo Asynchronous M-M, M-S Trigger
Londiste Asynchronous M-S Trigger
Mammoth Asynchronous M-S Log
RubyRep Asynchronous M-M, M-S Trigger
Friday, March 26, 2010 5

6. Why not Slony?
• Replicated tables MUST need PK or UK
• Doesn’t support large objects
• Doesn’t support synchronizing tables outside of replication
• Limitations of version compatibility
• Difficult to setup
• Doesn’t support Master - Master
• Difficult to monitor and manage
Friday, March 26, 2010 6

7. RubyRep Toolset
• Scan
• Sync
• Replicate
Friday, March 26, 2010 7

8. RubyRep Key Features
• Easy to install, setup, configure and monitor
• Platform independent
• Table design independent
• Simple PK
• Combined PK
• No PK
• Supports large objects i.e bytea, tsvector
• Database Version Independent
• Database independent
• PostgreSQL
• MySQL
Friday, March 26, 2010 8

9. Install
• Standard Ruby Version
• Install Ruby Environment , Install DB drivers , Install RubyRep
• Difficult to install
• Slower performance
• JRuby Version
• Install JVM, Download and Extract gzip file
• Easy to install
• Faster performance
Friday, March 26, 2010 9

10. Scan
• Features
• Compare “Left” and “Right” Databases/Tables
• Customize batch size
• Load on client server
• Steps:
• Generate config file
• ./rubyrep generate omniti.conf
• Configure
• Run the Scan
• ./rubyrep scan -c omniti.conf
• Production Results
• Users table with 60M rows and 45 columns
• 3 hours
• Running from monitor server
Friday, March 26, 2010 10

11. Scan
• To scan all tables starting with ‘c’:
• rubyrep scan -c omniti.conf /^c/
• Scan with default options will produce output like this:
• clients 100% ......................... 0
• clients_details 100% ......................... 5
• Options
• -s, --summary[=detailed]
• -d, --detailed[=mode]
• full , keys, diff
• -b, --progress-bar[=length]
• -c, --config=CONFIG_FILE
• --help
Friday, March 26, 2010 11

12. Config
RR::Initializer::run do |config|config.left = {
:adapter => 'postgresql', # or 'mysql'
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:schema_search_path => 'omniti’ }
config.right = {
:adapter => 'postgresql',
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:port => '5484'
}
config.include_tables 'users,users_backup'
config.include_tables /^o/ # regexp matches all tables starting with o
end
Friday, March 26, 2010 12

13. Sync
• Compare and Sync
• :sync_conflict_handling - :ignore (default), :left_wins, :right_wins
• :sync_record_handling
• :left_record_handling , :right_record_handling
• :ignore, :delete, :insert(default)
• Logging
• :ignored_changes , :all_changes ,
• :ignored_conflicts , :all_conflicts
• Others
• :row_buffer_size , :commit_frequency
• http://www.rubyrep.org/configuration.html
Friday, March 26, 2010 13

14. Sync
RR::Initializer::run do |config|config.left = {
:adapter => 'postgresql', # or 'mysql'
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:schema_search_path => 'omniti’ }
config.right = {
:adapter => 'postgresql',
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:port => '5484' }
# To match all tables except those ending with ‘backup’:
config.include_tables /./
config.exclude_tables /backup$/ #exclude tables ending with backup
end
Friday, March 26, 2010 14

15. Replicate
• Replicate Database/Tables
• Global vs Table specific settings
• rubyrep replicate -c omniti.conf
• Setup necessary infrastructure tables
• Setup necessary triggers for the replicated tables
• Initial scan is executed and synched
• Tweak sequences
• Left -Odd , Right -Even for M-M replication
• Default M-M replication setting
• :replicate_record _handling = :replicate
• Replication Conflict handling
• :later_wins, :earlier_wins
• Automatically Sync newly created table(Restart required)
• Uninstall
• rubyrep uninstall -c omniti.conf
Friday, March 26, 2010 15

16. Replicate
RR::Initializer::run do |config|config.left = {
:adapter => 'postgresql', # or 'mysql'
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:schema_search_path => 'omniti‘ }
config.right = {
:adapter => 'postgresql',
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:port => '5484' }
config.options[:sync_conflict_handling] = :left_wins
config.options[:replication_conflict_handling] = :left_wins
config.add_table_options 'admin',
:sync_conflict_handling => :right_wins,
:replication_conflict_handling => :right_wins
config.include_tables /./
end
Friday, March 26, 2010 16

17. Master-Slave Config
config.include_tables /./ # regexp matching all tables in the database
config.options[:auto_key_limit] = 40
config.options[:adjust_sequences] = false
config.options[:sequence_increment] = 1
config.options[:replication_conflict_handling] = :left_wins
config.options[:logged_replication_events] = [
:ignored_changes,
:ignored_conflicts
]
end
Friday, March 26, 2010 17

18. Customize as needed ...
• Tables without PK
• config.options[:auto_key_limit] = 2
• config.add_table_options /_/, :auto_key_limit => 2
• config.include_tables 'articles_tags', :key => ['article_id', 'tag_id']
• Checks before and after
• :before_table_sync => "SET foreign_key_checks = 0"
• :after_table_sync => "SET foreign_key_checks = 1"
• Low bandwidth with RubyRep Proxy
• Benefits only Scan and Compare ; only difference is transferred
• :proxy_host => '172.16.1.5', , :proxy_port => '9876'
• Cascading replication
• :rep_prefix to use separate triggers and tables
Friday, March 26, 2010 18

19. References
• Installation Notes
• http://rubyforge.org/frs/?group_id=7932
• User Guide
• http://www.rubyrep.org/documentation.html
• API Documentation
• http://rubyrep.rubyforge.org/
• Tutorials
• http://www.rubyrep.org/tutorial.html
• Blogs
• http://denishjpatel.blogspot.com/2009/08/yet-another-
postgresql-replication-tool.html
Friday, March 26, 2010 19

20. Thanks
• Arndt Lehmann
• Thank you!!
Friday, March 26, 2010 20

21. Questions ?
Friday, March 26, 2010 21