Your SlideShare is downloading. ×
0
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Bucardo
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Bucardo

1,099

Published on

Selena Deckelmann …

Selena Deckelmann

Bucardo is a mature replication system for PostgreSQL that supports asynchronous replication for both master-slave and multi-master systems. Originally designed for slow and unreliable networks, it has remarkable recovery ability, an easy to use command-line interface and development is active! Uses for Bucardo include: a slave read-only database, multi-master replication, data warehousing and just having fun moving your data around! Will include overview replication for PostgreSQL in general, a tour of features, a basic configuration walk through and the much feared live demo!

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,099
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
14
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Bucardo: Replication with tiny little goats Selena Deckelmann selena@endpoint.com en dp oi nt .c om http://www.flickr.com/photos/kevincollins/
  • 2. Hi! ★ Software Engineer http://endpoint.com ★ User Group Organizer http://pugs.postgresql.org/pdxpug ★ Conference Organizer http://opensourcebridge.org en dp oi nt .c om
  • 3. SEAPUG? • subscribe: email to: majordomo@postgresql.org message body: subscribe seapug • Talk to me after! en dp oi nt .c om
  • 4. en dp http://www.telegraph.co.uk/science/science-news/4409958/Extinct-ibex-is- oi nt .c resurrected-by-cloning.html om
  • 5. What can it do? en dp oi nt .c http://www.flickr.com/photos/8602783@N06/2294028540 om
  • 6. Master-slave • Scaling/Redundancy • UPGRADES (8.x -> 8.4) • Reporting databases • Data warehousing • Now handles SEQUENCES! en dp oi nt .c om
  • 7. Master-master • Write two two databases! • Sync type: SWAP • Conflict resolution through hooks! en dp oi nt .c om
  • 8. Custom code • Respond to DML changes • Filter • Run arbitrary code! en dp oi nt .c om
  • 9. STORY TIME! item ski item ski item ski item item ski item ski item onhand ski item skis 10 ski item ski item ski item ski on hand = count(item) en dp oi nt .c om
  • 10. prod db reporting db new! item NOTIFY bucardo ski sync B new! item syn ski cC on hand cA Y TIF +1 syn NO custom code count(items) and other stuff en dp oi nt .c om goat courtesy of: http://www.flickr.com/photos/nuritwilde/394603312
  • 11. Strengths • Drop-in, no changes to Postgres • Control daemon/DB can run anywhere • EXTREME async • No locking of slaves • Command-line controls rock! en dp oi nt .c om
  • 12. The Fine Print • No smooth DDL handling • Not for failover (aka high availability) • bloat: pg_listener, q, bucardo_delta, bucardo_track • No locking of slaves • No Windows support • One developer en dp oi nt .c om http://www.flickr.com/photos/ncarey/135901120/
  • 13. http://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling en dp oi nt .c om
  • 14. Installing en dp oi nt .c om http://www.flickr.com/photos/kendrick/361173694
  • 15. Bucardo Requirements • Postgres (8.3 or higher recommended) • Perl & some packages: • DBD::Pg 2.0 • DBI 1.51 • DBIx::Safe 1.2.4 • ExtUtils::MakeMaker 6.32 • pl/perlu and pl/pgsql en dp oi nt .c om
  • 16. Simple set up MASTER setup: bucardo_ctl add db MyDB name=master bucardo_ctl add all tables bucardo_ctl add all sequences SLAVE setup: bucardo_ctl add db MyDB name=slave port=6543 bucardo_ctl add herd all_tables psql# INSERT INTO herdmap (herd, goat) SELECT ‘all_tables’, id FROM goat WHERE (reltype = ‘table’ OR reltype = ‘sequence’) en AND db = ‘master’; dp oi nt .c om
  • 17. Setting up, cont. TEST: bucardo_ctl validate all SYNC: bucardo_ctl add sync delta type=pushdelta source=all_tables targetdb=slave (installs triggers!) en dp oi nt .c om
  • 18. Setting up • Initiate a one-time-copy: onetimecopy=[1|2] • Set up ongoing syncs: DEFAULT (can set timeouts) • MANY configuration options en dp oi nt .c om
  • 19. Admin stuff • .bucardorc (set things like: dbport) • Only need one ‘bucardo’ database • Doesn’t need to be your prod databases • System uses the ‘bucardo’ schema for ease of admin, removal en dp oi nt .c om
  • 20. Smooth upgrades! bucardo_ctl upgrade • Stop bucardo (bucardo_ctl stop) • applies DDL changes, functions • see also UPGRADE file en dp oi nt .c om
  • 21. Stuff to watch out for • search_path for ‘bucardo’ user • PL/Perlu & pl/pgsql must be installed • Uses prepared statements, turn off with: bucardo_ctl update db MyDB server_side_prepares=0 en dp oi nt .c om
  • 22. Internals en dp oi nt http://www.flickr.com/photos/chelseagirlphotos .c om
  • 23. Internals bucardo=# d List of relations Schema | Name | Type | Owner ---------+-------------------------------+----------+--------- bucardo | audit_pid | table | bucardo bucardo | audit_pid_id_seq | sequence | bucardo bucardo | bucardo_config | table | bucardo bucardo | bucardo_custom_trigger | table | bucardo bucardo | bucardo_custom_trigger_id_seq | sequence | bucardo bucardo | bucardo_log_message | table | bucardo bucardo | bucardo_rate | table | bucardo bucardo | customcode | table | bucardo bucardo | customcode_id_seq | sequence | bucardo bucardo | customcode_map | table | bucardo bucardo | db | table | bucardo bucardo | db_connlog | table | bucardo bucardo | dbgroup | table | bucardo bucardo | dbmap | table | bucardo bucardo | goat | table | bucardo bucardo | goat_id_seq | sequence | bucardo bucardo | herd | table | bucardo bucardo | herdmap | table | bucardo bucardo | q | table | bucardo bucardo | sync | table | bucardo bucardo | upgrade_log | table | bucardo en (21 rows) dp oi nt .c om
  • 24. goats Table "bucardo.goat" Column | Type | ----------------------+--------------------------+ id | integer | db | text | schemaname | text | tablename | text | reltype | text | pkey | text | qpkey | text | pkeytype | text | has_delta | boolean | ping | boolean | ... en dp oi nt .c om
  • 25. goats # bucardo_ctl list tables Table: archive.course_logging DB: odw_master PK: none Table: archive.messages DB: odw_master PK: none Table: archive.student_assignments DB: odw_master PK: none Table: archive.student_courses DB: odw_master PK: none Table: logging.activity DB: odw_master PK: activity_name (varchar) Table: logging.course_logging DB: odw_master PK: course_logging_id (int4) Table: public.ecash_transaction DB: odw_master PK: trans_id (int4) Table: public.feedback DB: odw_master PK: date (timestamp) Table: public.holidays DB: odw_master PK: customernum|month|day| year (int4|int4|int4|int4) ... en dp oi nt .c om
  • 26. bucardo_ctl list sequences Sequence 63: logging.logging_seq Sequence 64: public.trans_id_seq Sequence 65: public.num_seq Sequence 66: public.upload_seq en dp oi nt .c om
  • 27. bucardo database bucardo=# d List of relations Schema | Name | Type | Owner ---------+-------------------------------+----------+--------- ... bucardo | db | table | bucardo bucardo | db_connlog | table | bucardo bucardo | dbgroup | table | bucardo bucardo | dbmap | table | bucardo bucardo | goat | table | bucardo bucardo | goat_id_seq | sequence | bucardo bucardo | herd | table | bucardo bucardo | herdmap | table | bucardo bucardo | q | table | bucardo bucardo | sync | table | bucardo bucardo | upgrade_log | table | bucardo (21 rows) en dp oi nt .c om
  • 28. Internals # bucardo_ctl status Days back: 3 User: bucardo Database: bucardo Port: 8000 PID of Bucardo MCP: 6876 Name Type State PID Last_good Time I/U/D Last_bad Time ========+=====+=====+====+=========+=====+=====+========+==== a_delta | P |idle |6886|1m4s |0s |0/0/0|unknown | b_delta | P |idle |6885|1m4s |0s |0/0/0|9h32m47s|0s c_delta | P |idle |6887|1m4s |0s |0/0/0|unknown | d_delta | P |idle |6888|54s |0s |0/0/0|2m6s |3s e_delta | P |idle |6890|1m4s |0s |0/0/0|9h32m47s|0s f_delta | P |idle |6889|1m4s |0s |0/0/0|9h32m47s|0s en dp oi nt .c om
  • 29. # bucardo_ctl status a_delta Days back: 3 User: bucardo Database: bucardo Port: 8000 ====================================================================== Sync name: a_delta Current state: idle (PID = 6888) Type: pushdelta Source herd/database: all_a / a_master Target database: a_slave Tables in sync: 26 Last good: 3m 34s (time to run: 0s) Last good time: Oct 16, 2009 17:52:20 Target: a_slave Ins/Upd/Del: 20 / 0 / 50 Last bad: 4m 47s (time to run: 3s) Last bad time: Oct 16, 2009 17:51:08 Target: a_slave Latest bad reason: ? PID file: /var/run/bucardo/bucardo.ctl.sync.a_delta.pid PID file created: Fri Oct 16 17:52:09 2009 Status: active Limitdbs: 0 Priority: 0 Checktime: none Overdue time: 00:00:00 Expired time: 00:00:00 Stayalive: yes Kidsalive: yes Rebuild index: 0 Do_listen: yes Ping: yes Makedelta: no en dp Onetimecopy: 0 oi nt .c om
  • 30. How Bucardo replicates en dp oi nt .c om
  • 31. Triggers INSERT, UPDATE, DELETE bucardo_add_delta IN SE RT bucardo_delta bucardo_triggerkick_[sync] NOTIFY! en dp oi nt .c om
  • 32. NOTIFY / LISTEN • Simple interprocess communication psql# LISTEN cupcakes; psql# NOTIFY cupcakes; Asynchronous notification "cupcakes" received from server process with PID 1337. • Any Postgres process can listen in en dp oi nt .c om
  • 33. Actual Triggers! Table "public.test" Column | Type | Modifiers --------+-----------------------+----------- test | character varying(15) | not null Indexes: "test_pkey" PRIMARY KEY, btree ("test") Triggers: bucardo_add_delta AFTER INSERT OR DELETE OR UPDATE ON test FOR EACH ROW EXECUTE PROCEDURE bucardo_add_delta_role() bucardo_triggerkick_a_delta AFTER INSERT OR DELETE OR UPDATE ON test FOR EACH STATEMENT EXECUTE PROCEDURE bucardo_triggerkick_a_delta() en dp oi nt .c om
  • 34. On your master DB Tables: • bucardo_delta table has: ID, txn_time • bucardo_track table: who’s done what? en dp oi nt .c om
  • 35. Processes! • MCP, CTL, KID • MCP: master control process • CTL: controller (track, kick off/kill KIDs) • KID: sync processes en dp oi nt .c om
  • 36. syncs • A set of tables (herd) that are replicated together en dp oi nt .c om
  • 37. Initiating syncs • NOTIFY vs. timeout vs. kick • NOTIFY - from master db to MCP • timeout - controller detects • kick - manual/from MCP to controller en dp oi nt .c om
  • 38. Bucardo NOTIFY NOTIFY NOTIFY MCP CTL KID TRIGGER INSERT INTO “q” SELECT from “q” UPDATE “q” Flow of control REPLICATION en EVENT! dp oi nt .c om
  • 39. Administrativa • Totally ok to kill KIDs • CTL cleans up after you • May need to kill open database handles (for unfinished COPYs, etc) en dp oi nt .c om
  • 40. Sync types • Fullcopy. Timeout/kick. TRUNCATE or DELETE, then COPY • Pushdelta. Trigger. One way master-slave. • Swap. Trigger. Two-way. Conflict resolution. en dp oi nt .c om
  • 41. Bucardo anti-trigger • canonical way to disable triggers: ALTER TABLE DISABLE TRIGGER ALL; :( • pre 8.3 way: UPDATE pg_class.. • SET session_replication_role = ‘replica’; • Applies to just the current session • Thanks, Jan! en dp oi nt .c om
  • 42. Bucardo hooks • LIVE DATA CHANGES! • pass in a hashref (ROW or ROWINFO) • Returned value - bitmapped (a/b or both!) • Affect the replication event, side effects • Example: cache invalidation en dp oi nt .c om
  • 43. Bucardo Hooks • customcode & customcode_map • Conflict handlers (SWAP syncs) • Exception - fix and try again • Pre and post trigger en dp oi nt .c om
  • 44. Development • bucardo.org • mailing lists: http://bucardo.org/wiki/ Bucardo_mailing_list • Release early, often. • Please submit patches! bug reports! tests! en dp oi nt .c om
  • 45. Questions? .com nt oi dp en
  • 46. Thanks! http://www.slideshare.net/selenamarie/bucardo selena@endpoint.com @selenamarie on Twitter rss: http://www.chesnok.com/daily en dp oi nt .c om

×