Bucardo
Upcoming SlideShare
Loading in...5
×
 

Bucardo

on

  • 1,387 views

Selena Deckelmann...

Selena Deckelmann

Bucardo is a mature replication system for PostgreSQL that supports asynchronous replication for both master-slave and multi-master systems. Originally designed for slow and unreliable networks, it has remarkable recovery ability, an easy to use command-line interface and development is active! Uses for Bucardo include: a slave read-only database, multi-master replication, data warehousing and just having fun moving your data around! Will include overview replication for PostgreSQL in general, a tour of features, a basic configuration walk through and the much feared live demo!

Statistics

Views

Total Views
1,387
Views on SlideShare
1,387
Embed Views
0

Actions

Likes
1
Downloads
13
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Bucardo Bucardo Presentation Transcript

  • Bucardo: Replication with tiny little goats Selena Deckelmann selena@endpoint.com en dp oi nt .c om http://www.flickr.com/photos/kevincollins/
  • Hi! ★ Software Engineer http://endpoint.com ★ User Group Organizer http://pugs.postgresql.org/pdxpug ★ Conference Organizer http://opensourcebridge.org en dp oi nt .c om
  • SEAPUG? • subscribe: email to: majordomo@postgresql.org message body: subscribe seapug • Talk to me after! en dp oi nt .c om
  • en dp http://www.telegraph.co.uk/science/science-news/4409958/Extinct-ibex-is- oi nt .c resurrected-by-cloning.html om
  • What can it do? en dp oi nt .c http://www.flickr.com/photos/8602783@N06/2294028540 om
  • Master-slave • Scaling/Redundancy • UPGRADES (8.x -> 8.4) • Reporting databases • Data warehousing • Now handles SEQUENCES! en dp oi nt .c om
  • Master-master • Write two two databases! • Sync type: SWAP • Conflict resolution through hooks! en dp oi nt .c om
  • Custom code • Respond to DML changes • Filter • Run arbitrary code! en dp oi nt .c om
  • STORY TIME! item ski item ski item ski item item ski item ski item onhand ski item skis 10 ski item ski item ski item ski on hand = count(item) en dp oi nt .c om
  • prod db reporting db new! item NOTIFY bucardo ski sync B new! item syn ski cC on hand cA Y TIF +1 syn NO custom code count(items) and other stuff en dp oi nt .c om goat courtesy of: http://www.flickr.com/photos/nuritwilde/394603312
  • Strengths • Drop-in, no changes to Postgres • Control daemon/DB can run anywhere • EXTREME async • No locking of slaves • Command-line controls rock! en dp oi nt .c om
  • The Fine Print • No smooth DDL handling • Not for failover (aka high availability) • bloat: pg_listener, q, bucardo_delta, bucardo_track • No locking of slaves • No Windows support • One developer en dp oi nt .c om http://www.flickr.com/photos/ncarey/135901120/
  • http://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling en dp oi nt .c om
  • Installing en dp oi nt .c om http://www.flickr.com/photos/kendrick/361173694
  • Bucardo Requirements • Postgres (8.3 or higher recommended) • Perl & some packages: • DBD::Pg 2.0 • DBI 1.51 • DBIx::Safe 1.2.4 • ExtUtils::MakeMaker 6.32 • pl/perlu and pl/pgsql en dp oi nt .c om
  • Simple set up MASTER setup: bucardo_ctl add db MyDB name=master bucardo_ctl add all tables bucardo_ctl add all sequences SLAVE setup: bucardo_ctl add db MyDB name=slave port=6543 bucardo_ctl add herd all_tables psql# INSERT INTO herdmap (herd, goat) SELECT ‘all_tables’, id FROM goat WHERE (reltype = ‘table’ OR reltype = ‘sequence’) en AND db = ‘master’; dp oi nt .c om
  • Setting up, cont. TEST: bucardo_ctl validate all SYNC: bucardo_ctl add sync delta type=pushdelta source=all_tables targetdb=slave (installs triggers!) en dp oi nt .c om
  • Setting up • Initiate a one-time-copy: onetimecopy=[1|2] • Set up ongoing syncs: DEFAULT (can set timeouts) • MANY configuration options en dp oi nt .c om
  • Admin stuff • .bucardorc (set things like: dbport) • Only need one ‘bucardo’ database • Doesn’t need to be your prod databases • System uses the ‘bucardo’ schema for ease of admin, removal en dp oi nt .c om
  • Smooth upgrades! bucardo_ctl upgrade • Stop bucardo (bucardo_ctl stop) • applies DDL changes, functions • see also UPGRADE file en dp oi nt .c om
  • Stuff to watch out for • search_path for ‘bucardo’ user • PL/Perlu & pl/pgsql must be installed • Uses prepared statements, turn off with: bucardo_ctl update db MyDB server_side_prepares=0 en dp oi nt .c om
  • Internals en dp oi nt http://www.flickr.com/photos/chelseagirlphotos .c om
  • Internals bucardo=# d List of relations Schema | Name | Type | Owner ---------+-------------------------------+----------+--------- bucardo | audit_pid | table | bucardo bucardo | audit_pid_id_seq | sequence | bucardo bucardo | bucardo_config | table | bucardo bucardo | bucardo_custom_trigger | table | bucardo bucardo | bucardo_custom_trigger_id_seq | sequence | bucardo bucardo | bucardo_log_message | table | bucardo bucardo | bucardo_rate | table | bucardo bucardo | customcode | table | bucardo bucardo | customcode_id_seq | sequence | bucardo bucardo | customcode_map | table | bucardo bucardo | db | table | bucardo bucardo | db_connlog | table | bucardo bucardo | dbgroup | table | bucardo bucardo | dbmap | table | bucardo bucardo | goat | table | bucardo bucardo | goat_id_seq | sequence | bucardo bucardo | herd | table | bucardo bucardo | herdmap | table | bucardo bucardo | q | table | bucardo bucardo | sync | table | bucardo bucardo | upgrade_log | table | bucardo en (21 rows) dp oi nt .c om
  • goats Table "bucardo.goat" Column | Type | ----------------------+--------------------------+ id | integer | db | text | schemaname | text | tablename | text | reltype | text | pkey | text | qpkey | text | pkeytype | text | has_delta | boolean | ping | boolean | ... en dp oi nt .c om
  • goats # bucardo_ctl list tables Table: archive.course_logging DB: odw_master PK: none Table: archive.messages DB: odw_master PK: none Table: archive.student_assignments DB: odw_master PK: none Table: archive.student_courses DB: odw_master PK: none Table: logging.activity DB: odw_master PK: activity_name (varchar) Table: logging.course_logging DB: odw_master PK: course_logging_id (int4) Table: public.ecash_transaction DB: odw_master PK: trans_id (int4) Table: public.feedback DB: odw_master PK: date (timestamp) Table: public.holidays DB: odw_master PK: customernum|month|day| year (int4|int4|int4|int4) ... en dp oi nt .c om
  • bucardo_ctl list sequences Sequence 63: logging.logging_seq Sequence 64: public.trans_id_seq Sequence 65: public.num_seq Sequence 66: public.upload_seq en dp oi nt .c om
  • bucardo database bucardo=# d List of relations Schema | Name | Type | Owner ---------+-------------------------------+----------+--------- ... bucardo | db | table | bucardo bucardo | db_connlog | table | bucardo bucardo | dbgroup | table | bucardo bucardo | dbmap | table | bucardo bucardo | goat | table | bucardo bucardo | goat_id_seq | sequence | bucardo bucardo | herd | table | bucardo bucardo | herdmap | table | bucardo bucardo | q | table | bucardo bucardo | sync | table | bucardo bucardo | upgrade_log | table | bucardo (21 rows) en dp oi nt .c om
  • Internals # bucardo_ctl status Days back: 3 User: bucardo Database: bucardo Port: 8000 PID of Bucardo MCP: 6876 Name Type State PID Last_good Time I/U/D Last_bad Time ========+=====+=====+====+=========+=====+=====+========+==== a_delta | P |idle |6886|1m4s |0s |0/0/0|unknown | b_delta | P |idle |6885|1m4s |0s |0/0/0|9h32m47s|0s c_delta | P |idle |6887|1m4s |0s |0/0/0|unknown | d_delta | P |idle |6888|54s |0s |0/0/0|2m6s |3s e_delta | P |idle |6890|1m4s |0s |0/0/0|9h32m47s|0s f_delta | P |idle |6889|1m4s |0s |0/0/0|9h32m47s|0s en dp oi nt .c om
  • # bucardo_ctl status a_delta Days back: 3 User: bucardo Database: bucardo Port: 8000 ====================================================================== Sync name: a_delta Current state: idle (PID = 6888) Type: pushdelta Source herd/database: all_a / a_master Target database: a_slave Tables in sync: 26 Last good: 3m 34s (time to run: 0s) Last good time: Oct 16, 2009 17:52:20 Target: a_slave Ins/Upd/Del: 20 / 0 / 50 Last bad: 4m 47s (time to run: 3s) Last bad time: Oct 16, 2009 17:51:08 Target: a_slave Latest bad reason: ? PID file: /var/run/bucardo/bucardo.ctl.sync.a_delta.pid PID file created: Fri Oct 16 17:52:09 2009 Status: active Limitdbs: 0 Priority: 0 Checktime: none Overdue time: 00:00:00 Expired time: 00:00:00 Stayalive: yes Kidsalive: yes Rebuild index: 0 Do_listen: yes Ping: yes Makedelta: no en dp Onetimecopy: 0 oi nt .c om
  • How Bucardo replicates en dp oi nt .c om
  • Triggers INSERT, UPDATE, DELETE bucardo_add_delta IN SE RT bucardo_delta bucardo_triggerkick_[sync] NOTIFY! en dp oi nt .c om
  • NOTIFY / LISTEN • Simple interprocess communication psql# LISTEN cupcakes; psql# NOTIFY cupcakes; Asynchronous notification "cupcakes" received from server process with PID 1337. • Any Postgres process can listen in en dp oi nt .c om
  • Actual Triggers! Table "public.test" Column | Type | Modifiers --------+-----------------------+----------- test | character varying(15) | not null Indexes: "test_pkey" PRIMARY KEY, btree ("test") Triggers: bucardo_add_delta AFTER INSERT OR DELETE OR UPDATE ON test FOR EACH ROW EXECUTE PROCEDURE bucardo_add_delta_role() bucardo_triggerkick_a_delta AFTER INSERT OR DELETE OR UPDATE ON test FOR EACH STATEMENT EXECUTE PROCEDURE bucardo_triggerkick_a_delta() en dp oi nt .c om
  • On your master DB Tables: • bucardo_delta table has: ID, txn_time • bucardo_track table: who’s done what? en dp oi nt .c om
  • Processes! • MCP, CTL, KID • MCP: master control process • CTL: controller (track, kick off/kill KIDs) • KID: sync processes en dp oi nt .c om
  • syncs • A set of tables (herd) that are replicated together en dp oi nt .c om
  • Initiating syncs • NOTIFY vs. timeout vs. kick • NOTIFY - from master db to MCP • timeout - controller detects • kick - manual/from MCP to controller en dp oi nt .c om
  • Bucardo NOTIFY NOTIFY NOTIFY MCP CTL KID TRIGGER INSERT INTO “q” SELECT from “q” UPDATE “q” Flow of control REPLICATION en EVENT! dp oi nt .c om
  • Administrativa • Totally ok to kill KIDs • CTL cleans up after you • May need to kill open database handles (for unfinished COPYs, etc) en dp oi nt .c om
  • Sync types • Fullcopy. Timeout/kick. TRUNCATE or DELETE, then COPY • Pushdelta. Trigger. One way master-slave. • Swap. Trigger. Two-way. Conflict resolution. en dp oi nt .c om
  • Bucardo anti-trigger • canonical way to disable triggers: ALTER TABLE DISABLE TRIGGER ALL; :( • pre 8.3 way: UPDATE pg_class.. • SET session_replication_role = ‘replica’; • Applies to just the current session • Thanks, Jan! en dp oi nt .c om
  • Bucardo hooks • LIVE DATA CHANGES! • pass in a hashref (ROW or ROWINFO) • Returned value - bitmapped (a/b or both!) • Affect the replication event, side effects • Example: cache invalidation en dp oi nt .c om
  • Bucardo Hooks • customcode & customcode_map • Conflict handlers (SWAP syncs) • Exception - fix and try again • Pre and post trigger en dp oi nt .c om
  • Development • bucardo.org • mailing lists: http://bucardo.org/wiki/ Bucardo_mailing_list • Release early, often. • Please submit patches! bug reports! tests! en dp oi nt .c om
  • Questions? .com nt oi dp en
  • Thanks! http://www.slideshare.net/selenamarie/bucardo selena@endpoint.com @selenamarie on Twitter rss: http://www.chesnok.com/daily en dp oi nt .c om