meet your Queue
/ Presentation / Theo Schlossnagle
2. PostgreSQL is Awesome
• Robust data types.
• Partitioning (albeit fake).
• Partial and functional indexes.
• Extremely supportive community.
• Extremely compliant with database standards.
3. PostgreSQL is not the “world”
• Inevitably, we must interact with the rest of the world.
• “non-SQL” components:
• nosql systems
• caching systems
• search systems (solr/lucene)
• management processes
4. Appropiare Typicalis
• Enforce in the application:
• the application code that updates the price or description of a
product in the products table;
• the application submits the updates to the search index system.
• The flaw:
• psql# UPDATE products
SET description =
REPLACE(description, ‘behaviour’, ‘behavior’);
• Administrative fixes like that require out-of-band dependency
5. A Solution
• Ideally, the database would notify all of these systems.
• The most common case I see: memcached.
• app: pull from memcached user::email@example.com
if not found:
select * from users where email=‘firstname.lastname@example.org‘
put row in memcached at user::email@example.com
• app: update users set mood=‘happy‘
(a) purge memcached record
(b) get full row and replace in memcached
• hence: pgmemcache
• need a Postgres module for each remote component
6. Enter Queueing
• A generic message bus that allows PostgreSQL to communicate
with other components in the architecture.
• Enter AMQP: “Advanced Message Queueing Protocol”
• Why not STOMP?
• Why not Starling?
• AMQP has been around the block, and the specification is quite
• Almost every “real” message broker implementation supports
7. Setups: Installing
• svn export
• cd pg_amqp
• make USE_PGXS=1
• make install
• add to postgresql.conf:
shared_preload_libraries = 'pg_amqp.so'
• (re)start postgres
• load the pg_amqp.sql file into your database.
8. Setup: configuring your broker
INSERT INTO amqp.broker (host,port,vhost,username,password)
9. Setup: declaring an exchange
• This can often be done outside of the AMQP client
• using an AMQP management process
(that is just, in fact, an AMQP client)
• Often, another component has already created the exchange.
• If you really need to do it within PostgreSQL:
passive, durable, auto_delete)
10. Usage: sending messages
• How do I connect?
• It’s implicit... you don’t need to.
• How do I disconnect?
• Broker connections are cached and live across transactions.
• If you want to explicitly disconnect:
11. Usage: sending messages for real
• Sending messages as a part of my transaction:
SELECT amqp.publish(broker_id, exchange, routing_key, message);
• This will publish the “message” over the specified “exchange” using
the specified “routing_key,” but only on transaction commit within
• Sending messages NOW:
SELECT amqp.autonomous_publish(broker_id, exchange,
• Publish the same message, but do it immediately.
12. A dark side: unsafe? WTF?
• Currently, pg_amqp uses the AMQP 0.8 specification.
• The AMQP 1.0 specification introduces formal 2PC.
• It is possible in the current implementation to have AMQP transaction
fail when we commit it (in postgres’s commit txn hook) in a fashion
that we cannot act on. Ouch.
• This only happens when the AMQP broker crashes between the last
in-transaction publish call and the COMMIT on the database side.
• Once RabbitMQ supports AMQP 1.0, I’ll update the driver to use 2PC.
13. Thank you for listening.