Uploaded on

Theo Schlossnagle …

Theo Schlossnagle

Often times, when we make changes in a database we also want to "do something useful." Simply sending and email or triggering an ETL into a non-postgres system like Lucene is also a bit convoluted and painful. By using asynchronous messaging, we can make this simple. There have been a few attempts in the past to add into postgres the ability to queue external tasks, but with AMQP you get a complete package.

pg_amqp marries the AMQP standard into PostgreSQL providing transaction-aware message queueing. We'll talk about how easy it can be to get an AMQP enabled postgresql up and running with pg_amqp. We'll also discuss the caveats that exist in pg_amqp (currently implementing AMQP 0.8) and how AMQP 1.0 will eventually solve them.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. PostgreSQL: meet your Queue / Presentation / Theo Schlossnagle 1
  • 2. PostgreSQL is Awesome • Fast. • Extensible. • Tablespaces. • Robust data types. • Partitioning (albeit fake). • Partial and functional indexes. • Extremely supportive community. • Extremely compliant with database standards. 2
  • 3. PostgreSQL is not the “world” • Inevitably, we must interact with the rest of the world. • “non-SQL” components: • nosql systems • caching systems • search systems (solr/lucene) • management processes 3
  • 4. Appropiare Typicalis • Enforce in the application: • the application code that updates the price or description of a product in the products table; • the application submits the updates to the search index system. • The flaw: • psql# UPDATE products SET description = REPLACE(description, ‘behaviour’, ‘behavior’); • Administrative fixes like that require out-of-band dependency handling. 4
  • 5. A Solution • Ideally, the database would notify all of these systems. • The most common case I see: memcached. • app: pull from memcached user::jesus@omniti.com if not found: select * from users where email=‘jesus@omniti.com‘ put row in memcached at user::jesus@omniti.com • app: update users set mood=‘happy‘ where email=‘jesus@omniti.com‘ (a) purge memcached record (b) get full row and replace in memcached • hence: pgmemcache • Problem: • need a Postgres module for each remote component 5
  • 6. Enter Queueing • Queueing? • A generic message bus that allows PostgreSQL to communicate with other components in the architecture. • Enter AMQP: “Advanced Message Queueing Protocol” • Why not STOMP? • Why not Starling? • AMQP has been around the block, and the specification is quite complete. • Almost every “real” message broker implementation supports AMQP 6
  • 7. Setups: Installing • svn export https://labs.omniti.com/pgtreats/trunk/contrib/pg_amqp • cd pg_amqp • make USE_PGXS=1 • make install • add to postgresql.conf: shared_preload_libraries = 'pg_amqp.so' • (re)start postgres • load the pg_amqp.sql file into your database. 7
  • 8. Setup: configuring your broker INSERT INTO amqp.broker (host,port,vhost,username,password) VALUES (‘localhost’,5672,NULL,‘guest’,‘guest’) RETURNING broker_id 8
  • 9. Setup: declaring an exchange • This can often be done outside of the AMQP client • using an AMQP management process (that is just, in fact, an AMQP client) • Often, another component has already created the exchange. • If you really need to do it within PostgreSQL: SELECT amqp.declare_exchange(broker_id, exchange_name, exchange_type, passive, durable, auto_delete) 9
  • 10. Usage: sending messages • How do I connect? • It’s implicit... you don’t need to. • How do I disconnect? • Broker connections are cached and live across transactions. • If you want to explicitly disconnect: SELECT amqp.disconnect(broker_id); 10
  • 11. Usage: sending messages for real • Sending messages as a part of my transaction: SELECT amqp.publish(broker_id, exchange, routing_key, message); • This will publish the “message” over the specified “exchange” using the specified “routing_key,” but only on transaction commit within Postgres. • Sending messages NOW: SELECT amqp.autonomous_publish(broker_id, exchange, routing_key, message); • Publish the same message, but do it immediately. 11
  • 12. A dark side: unsafe? WTF? • Currently, pg_amqp uses the AMQP 0.8 specification. • The AMQP 1.0 specification introduces formal 2PC. • It is possible in the current implementation to have AMQP transaction fail when we commit it (in postgres’s commit txn hook) in a fashion that we cannot act on. Ouch. • This only happens when the AMQP broker crashes between the last in-transaction publish call and the COMMIT on the database side. • Once RabbitMQ supports AMQP 1.0, I’ll update the driver to use 2PC. 12
  • 13. Thank you for listening. https://labs.omniti.com/pgtreats/trunk/contrib/pg_amqp/ / Presentation 13