Denish Patel One of the key features any Enterprise considers when choosing a database technology for their architecture solution stack is that of replication. Oracle and MySQL both have built in replication solutions, but as of yet PostgreSQL doesn't support a built in replication solution. There are many replication solutions available however, and different companies are using different solutions customized for their needs. Among all of the solutions, Slony is probably the most widely tested and deployed within organizations, although it does have the following limitations: * Replicated tables must have a unique or primary key * It does not support replication of large objects * Schema changes are not propagated (though they can be coordinated) * It does not support synchronizing databases outside of replication * There are limitations on version compatability; you can not replicate from PostgreSQL 8.2 to PostgreSQL 8.4 for example * It is more difficult to set up than many other replication solutions One new alternative to Slony is a project known as RubyRep, which is designed to avoid some of the limitations of Slony. RubyRep has full stack solution from comparing two databases to start replicating them. In the talk, I will explore the RubyRep usage with examples for following actions: - Compare - Sync - Replicate

1. Yet Another Replication Tool
RubyRep
/ Denish Patel
Database Architect
Friday, March 26, 2010 1

2. Who am I?
• With OmniTi for more than 3 years
• Manage high traffic database systems
• Replication database system deployments
• Not a core hacker of RubyRep
• “Oh, We are hiring!!”
• Contact : denish@omniti.com
Friday, March 26, 2010 2

3. Next 30 minutes ..
• Replication
• Various Tools
• Slony? Why another tool ?
• RubyRep
• Install
• Features
• Examples
• Tweaking replication policies
Friday, March 26, 2010 3

4. Replication
• Types
• Synchronous
• Semi-synchronous (O My!)
• Asynchronous
• Methods
• Master - Slave
• Master - Master
• Needs
• Horizontal Scaling
• Standby Database
• Backup Database
Friday, March 26, 2010 4

5. Tools
Program Type Method Based on
PgCluster -II Synchronous M-M Shared Disk
Slony-I Asynchronous M-S Trigger
Bucardo Asynchronous M-M, M-S Trigger
Londiste Asynchronous M-S Trigger
Mammoth Asynchronous M-S Log
RubyRep Asynchronous M-M, M-S Trigger
Friday, March 26, 2010 5

6. Why not Slony?
• Replicated tables MUST need PK or UK
• Doesn’t support large objects
• Doesn’t support synchronizing tables outside of replication
• Limitations of version compatibility
• Difficult to setup
• Doesn’t support Master - Master
• Difficult to monitor and manage
Friday, March 26, 2010 6

7. RubyRep Toolset
• Scan
• Sync
• Replicate
Friday, March 26, 2010 7

8. RubyRep Key Features
• Easy to install, setup, configure and monitor
• Platform independent
• Table design independent
• Simple PK
• Combined PK
• No PK
• Supports large objects i.e bytea, tsvector
• Database Version Independent
• Database independent
• PostgreSQL
• MySQL
Friday, March 26, 2010 8

9. Install
• Standard Ruby Version
• Install Ruby Environment , Install DB drivers , Install RubyRep
• Difficult to install
• Slower performance
• JRuby Version
• Install JVM, Download and Extract gzip file
• Easy to install
• Faster performance
Friday, March 26, 2010 9

10. Scan
• Features
• Compare “Left” and “Right” Databases/Tables
• Customize batch size
• Load on client server
• Steps:
• Generate config file
• ./rubyrep generate omniti.conf
• Configure
• Run the Scan
• ./rubyrep scan -c omniti.conf
• Production Results
• Users table with 60M rows and 45 columns
• 3 hours
• Running from monitor server
Friday, March 26, 2010 10

11. Scan
• To scan all tables starting with ‘c’:
• rubyrep scan -c omniti.conf /^c/
• Scan with default options will produce output like this:
• clients 100% ......................... 0
• clients_details 100% ......................... 5
• Options
• -s, --summary[=detailed]
• -d, --detailed[=mode]
• full , keys, diff
• -b, --progress-bar[=length]
• -c, --config=CONFIG_FILE
• --help
Friday, March 26, 2010 11

12. Config
RR::Initializer::run do |config|config.left = {
:adapter => 'postgresql', # or 'mysql'
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:schema_search_path => 'omniti’ }
config.right = {
:adapter => 'postgresql',
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:port => '5484'
}
config.include_tables 'users,users_backup'
config.include_tables /^o/ # regexp matches all tables starting with o
end
Friday, March 26, 2010 12

13. Sync
• Compare and Sync
• :sync_conflict_handling - :ignore (default), :left_wins, :right_wins
• :sync_record_handling
• :left_record_handling , :right_record_handling
• :ignore, :delete, :insert(default)
• Logging
• :ignored_changes , :all_changes ,
• :ignored_conflicts , :all_conflicts
• Others
• :row_buffer_size , :commit_frequency
• http://www.rubyrep.org/configuration.html
Friday, March 26, 2010 13

14. Sync
RR::Initializer::run do |config|config.left = {
:adapter => 'postgresql', # or 'mysql'
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:schema_search_path => 'omniti’ }
config.right = {
:adapter => 'postgresql',
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:port => '5484' }
# To match all tables except those ending with ‘backup’:
config.include_tables /./
config.exclude_tables /backup$/ #exclude tables ending with backup
end
Friday, March 26, 2010 14

15. Replicate
• Replicate Database/Tables
• Global vs Table specific settings
• rubyrep replicate -c omniti.conf
• Setup necessary infrastructure tables
• Setup necessary triggers for the replicated tables
• Initial scan is executed and synched
• Tweak sequences
• Left -Odd , Right -Even for M-M replication
• Default M-M replication setting
• :replicate_record _handling = :replicate
• Replication Conflict handling
• :later_wins, :earlier_wins
• Automatically Sync newly created table(Restart required)
• Uninstall
• rubyrep uninstall -c omniti.conf
Friday, March 26, 2010 15

16. Replicate
RR::Initializer::run do |config|config.left = {
:adapter => 'postgresql', # or 'mysql'
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:schema_search_path => 'omniti‘ }
config.right = {
:adapter => 'postgresql',
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:port => '5484' }
config.options[:sync_conflict_handling] = :left_wins
config.options[:replication_conflict_handling] = :left_wins
config.add_table_options 'admin',
:sync_conflict_handling => :right_wins,
:replication_conflict_handling => :right_wins
config.include_tables /./
end
Friday, March 26, 2010 16

17. Master-Slave Config
config.include_tables /./ # regexp matching all tables in the database
config.options[:auto_key_limit] = 40
config.options[:adjust_sequences] = false
config.options[:sequence_increment] = 1
config.options[:replication_conflict_handling] = :left_wins
config.options[:logged_replication_events] = [
:ignored_changes,
:ignored_conflicts
]
end
Friday, March 26, 2010 17

18. Customize as needed ...
• Tables without PK
• config.options[:auto_key_limit] = 2
• config.add_table_options /_/, :auto_key_limit => 2
• config.include_tables 'articles_tags', :key => ['article_id', 'tag_id']
• Checks before and after
• :before_table_sync => "SET foreign_key_checks = 0"
• :after_table_sync => "SET foreign_key_checks = 1"
• Low bandwidth with RubyRep Proxy
• Benefits only Scan and Compare ; only difference is transferred
• :proxy_host => '172.16.1.5', , :proxy_port => '9876'
• Cascading replication
• :rep_prefix to use separate triggers and tables
Friday, March 26, 2010 18

19. References
• Installation Notes
• http://rubyforge.org/frs/?group_id=7932
• User Guide
• http://www.rubyrep.org/documentation.html
• API Documentation
• http://rubyrep.rubyforge.org/
• Tutorials
• http://www.rubyrep.org/tutorial.html
• Blogs
• http://denishjpatel.blogspot.com/2009/08/yet-another-
postgresql-replication-tool.html
Friday, March 26, 2010 19

20. Thanks
• Arndt Lehmann
• Thank you!!
Friday, March 26, 2010 20

21. Questions ?
Friday, March 26, 2010 21