Pg amqp
Upcoming SlideShare
Loading in...5
×
 

Pg amqp

on

  • 2,997 views

Theo Schlossnagle...

Theo Schlossnagle

Often times, when we make changes in a database we also want to "do something useful." Simply sending and email or triggering an ETL into a non-postgres system like Lucene is also a bit convoluted and painful. By using asynchronous messaging, we can make this simple. There have been a few attempts in the past to add into postgres the ability to queue external tasks, but with AMQP you get a complete package.

pg_amqp marries the AMQP standard into PostgreSQL providing transaction-aware message queueing. We'll talk about how easy it can be to get an AMQP enabled postgresql up and running with pg_amqp. We'll also discuss the caveats that exist in pg_amqp (currently implementing AMQP 0.8) and how AMQP 1.0 will eventually solve them.

Statistics

Views

Total Views
2,997
Views on SlideShare
2,993
Embed Views
4

Actions

Likes
1
Downloads
8
Comments
0

2 Embeds 4

http://www.slashdocs.com 3
http://www.docshut.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Pg amqp Pg amqp Presentation Transcript

    • PostgreSQL: meet your Queue / Presentation / Theo Schlossnagle 1
    • PostgreSQL is Awesome • Fast. • Extensible. • Tablespaces. • Robust data types. • Partitioning (albeit fake). • Partial and functional indexes. • Extremely supportive community. • Extremely compliant with database standards. 2
    • PostgreSQL is not the “world” • Inevitably, we must interact with the rest of the world. • “non-SQL” components: • nosql systems • caching systems • search systems (solr/lucene) • management processes 3
    • Appropiare Typicalis • Enforce in the application: • the application code that updates the price or description of a product in the products table; • the application submits the updates to the search index system. • The flaw: • psql# UPDATE products SET description = REPLACE(description, ‘behaviour’, ‘behavior’); • Administrative fixes like that require out-of-band dependency handling. 4
    • A Solution • Ideally, the database would notify all of these systems. • The most common case I see: memcached. • app: pull from memcached user::jesus@omniti.com if not found: select * from users where email=‘jesus@omniti.com‘ put row in memcached at user::jesus@omniti.com • app: update users set mood=‘happy‘ where email=‘jesus@omniti.com‘ (a) purge memcached record (b) get full row and replace in memcached • hence: pgmemcache • Problem: • need a Postgres module for each remote component 5
    • Enter Queueing • Queueing? • A generic message bus that allows PostgreSQL to communicate with other components in the architecture. • Enter AMQP: “Advanced Message Queueing Protocol” • Why not STOMP? • Why not Starling? • AMQP has been around the block, and the specification is quite complete. • Almost every “real” message broker implementation supports AMQP 6
    • Setups: Installing • svn export https://labs.omniti.com/pgtreats/trunk/contrib/pg_amqp • cd pg_amqp • make USE_PGXS=1 • make install • add to postgresql.conf: shared_preload_libraries = 'pg_amqp.so' • (re)start postgres • load the pg_amqp.sql file into your database. 7
    • Setup: configuring your broker INSERT INTO amqp.broker (host,port,vhost,username,password) VALUES (‘localhost’,5672,NULL,‘guest’,‘guest’) RETURNING broker_id 8
    • Setup: declaring an exchange • This can often be done outside of the AMQP client • using an AMQP management process (that is just, in fact, an AMQP client) • Often, another component has already created the exchange. • If you really need to do it within PostgreSQL: SELECT amqp.declare_exchange(broker_id, exchange_name, exchange_type, passive, durable, auto_delete) 9
    • Usage: sending messages • How do I connect? • It’s implicit... you don’t need to. • How do I disconnect? • Broker connections are cached and live across transactions. • If you want to explicitly disconnect: SELECT amqp.disconnect(broker_id); 10
    • Usage: sending messages for real • Sending messages as a part of my transaction: SELECT amqp.publish(broker_id, exchange, routing_key, message); • This will publish the “message” over the specified “exchange” using the specified “routing_key,” but only on transaction commit within Postgres. • Sending messages NOW: SELECT amqp.autonomous_publish(broker_id, exchange, routing_key, message); • Publish the same message, but do it immediately. 11
    • A dark side: unsafe? WTF? • Currently, pg_amqp uses the AMQP 0.8 specification. • The AMQP 1.0 specification introduces formal 2PC. • It is possible in the current implementation to have AMQP transaction fail when we commit it (in postgres’s commit txn hook) in a fashion that we cannot act on. Ouch. • This only happens when the AMQP broker crashes between the last in-transaction publish call and the COMMIT on the database side. • Once RabbitMQ supports AMQP 1.0, I’ll update the driver to use 2PC. 12
    • Thank you for listening. https://labs.omniti.com/pgtreats/trunk/contrib/pg_amqp/ / Presentation 13