Replication - Nick Carboni - ManageIQ Design Summit 2016
•Download as PPTX, PDF•
0 likes•740 views
Database Replication in ManageIQ by Nick Carboni at ManageIQ Design Summit 2016
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Replication - Nick Carboni - ManageIQ Design Summit 2016
Similar to Replication - Nick Carboni - ManageIQ Design Summit 2016 (20)
More from ManageIQ
More from ManageIQ (20)
Recently uploaded
Recently uploaded (20)
Replication - Nick Carboni - ManageIQ Design Summit 2016
- 2. Introduction Nick Carboni Software Engineer at Red Hat Appliance and database for ManageIQ (Platform Team)
- 3. Replication in ManageIQ Multiple databases can be used for far away environments Database replication allows for easy reporting and visibility Database 1 Database 2 Global Database Replication!
- 5. rubyrep Ruby + ActiveRecord Triggers = Developer headaches SQL Configuration is … hardelsif commands.include? command status = commands[command][:command].run(args.slice(1, 1_000_000))
- 6. rubyrep in ManageIQ We use our own fork https://github.com/ManageIQ/rubyrep Mainly support for new versions of ActiveRecord
- 7. rubyrep in ManageIQ PR to add support for Rails 4.2 March 27, 2015 “Yup. It's basically dead. Our changes are also SOOO crazy different, that this is like a Frankenstein project now. I wonder if we should just write something different from scratch given our experience...or perhaps we should rethink replication altogether and come up with a plan.” - @Fryguy
- 8. rubyrep in ManageIQ Worker process Database Synchronization role Worker will run on the server with the role Passive global database
- 9. pglogical http://2ndquadrant.com/en-us/resources/pglogical/ https://github.com/2ndQuadrant/pglogical/ Inclusion into PostgreSQL core in progress 9.7+??
- 10. pglogical PostgreSQL extension Process runs on database server Managed by PostgreSQL service SQL stored procedure API postgresql.conf configuration
- 11. pglogical PostgreSQL logical decoding https://www.postgresql.org/docs/9.4/static/logicaldecoding.html Output plugin pglogical_output Consumer extension pglogical Replication slots https://www.postgresql.org/docs/9.4/static/warm-
- 12. pglogical in ManageIQ AR connection adapter extension Exposes a ruby method per pglogical stored procedure Configure “subscriptions” to remote regions through global region UI Exclude table configuration unchanged “Advanced Settings” tab in remote region UI
- 14. Performance testing How long does it take to replicate a “backlog” of data? Vary amount of data and artificial network latency Measure time until “backlog” is zero rubyrep - number of rows in rr_pending_changes table pglogical - transaction log location compared to flush location SELECT pg_xlog_location_diff(pg_current_xlog_location(), flush_location) AS lag_bytes FROM pg_stat_replication;
- 15. Insert Performance Rows pglogical rubyrep 1,000 0.158 (σ = 0.006) 0.189 (σ = 0.011) 10,000 1.499 (σ = 0.095) 1.84 (σ = 0.093) 100,000 16.105 (σ = 0.296) 18.042 (σ = 0.320) 1,000,000 147.508 (σ = 3.557) 195.399 (σ = 18.317) Sample size = 10
- 16. No Latency
- 17. 25ms Latency (50ms ping time)
- 18. 100ms Latency (200ms ping time)
- 19. Q&A
Editor's Notes
- Replication as a feature, not for failover - that’s Brett’s talk next Better local performance (using the product normally) by keeping the database close to the provider / other infrastructure it is managing No one wants multiple reports because of performance concerns Replicate the data to the “global” database and report there
- The ruby rep website is even older - 2009
- Typically just doing more than it probably needs to. Used active record so it could support more than just postgresql Renamed tables cause issues with triggers We don’t have a standard interface to adding / removing them when we change things Not everyone that is making a database change is also thinking about replication Talk about even streams table - Table was renamed Had to remember that replication was a thing, Had to rename all the existing rows in the pending changes and the state table Forgot about the triggers The old trigger stayed around and we got a change entry for both table names every time something happened SQL as opposed to … Was trying to track down how the configuration worked Our config file was a two line ruby script that was passed as an argument to a rubyrep gem method (run) That config file called out to a class which set config options from within the gem
- Found the PR to our fork for adding support for Rails 4.2 while looking back in the repo Found a comment We also added support for rails 5
- Don’t want worker running on a server that has a bad connection to both databases. Extra worker takes up memory on the appliance We have to monitor when replication is running (heartbeat) Additional process outside of our worker actually runs the replication Upside - passive global Replication is just another database connection writing data
- 9.7 estimate from a contributor on the project I spoke with Still not looking likely, but the extension is fine
- Starts and stops with the database once it is configured No worrying about roles Will always run on the database server Stored procs are a lot nicer to deal with than a separate process or changing that config
- Logical decoding is a way to capture more detailed data from the WAL logs Needs an output plugin to present the data Replication slots track when changes have been consumed and retain WAL logs accordingly Before replication slots users had to come up with a WAL archiving strategy to make sure the logs were still there when they needed to be replicated.
- The user now configures the credentials in one place (global)
- Latency achieved with `tc` command to inject a delay before network traffic is queued
- (standard deviation) Triggers have some overhead on insert
- rows pglogical rubyrep Pgl-stddev Rr-stddev 1000 0.2s 8.769s 0 0.0959687449 10000 0.76s 18.524s 0.0843274043 0.5560015987 100000 5.88s 114.736s (2min) 0.267498702 3.5969128739 1000000 69.087s 1281.661s (21min) 11.417237504 239.7990555745
- Rows pglogical rubyrep Pgl-stddev Rr-stddev 1000 0.62s 70.294s 0.0632455532 6.4325823396 10000 1.67s 592.32s (~10 min) 0.6535373831 76.2543952832 100000 6.841s 5530.42s (1hr 30 min) 0.5028463428 360.8224483039 1000000 62.69s 8.1968882985
- Rows pglogical rubyrep Pgl-stddev Rr-stddev 1,000 2.05s 233.655s (3min 52s) 0.2065591118 4.6881179593 10,000 3.011s 2058.525s (34min) 0.2521221045 4.2779960262 100,000 11.739s 20506.11s (5h 40min) 7.7121526754 1,000,000 118.233s (2min) 6.5350627813