0
Integrating PostgreSQL with RabbitMQ
Gavin M. Roy
April 4th, 2014
About Me
VP of Architecture

AWeber Communications
!
Blog: http://gavinroy.com
Twitter: @Crad
GitHub: https://github.com/g...
RabbitMQ Primer
About RabbitMQ
• Message Oriented Middleware
• Primary identity is defined by AMQP 0-9-1
• Supports other protocols with Pl...
Why RabbitMQ?
• Create loosely coupled architectures
• Decouple database write operations
• Communicate across application...
Native AMQP Clients
C
Clojure
Cobol
Common Lisp
Delphi
Erlang
Go
Groovy
Haskell
Java
JavaScript
.NET
OCaml
Perl
PHP
Python...
Who Uses It?
Agora Games
Chef
Google AdMob
Instagram
MeetMe
Mercado Libre
Mozilla
Nasa
New York Times
NSF
Openstack
Reddit
RabbitMQ Concepts
AMQP Message
Body
Properties
Message Properties
app-id
content-encoding
content-type
correlation-id
delivery-mode
expiration
headers
message-id
priorit...
Publishers and Consumers
CP
Multiple Publishers
P
C
P
P
Multiple Consumers
P
C
C
C
AMQ Model
Exchange

defines routing behavior
Queue

stores messages in memory and
optionally on disk
Binding

defines relati...
Routing Keys
• Provided when publishing a message
• Compared against binding keys by exchanges
• Example uses of a Routing...
Built-In Exchange Types
Direct

String matching on Routing Key
Fanout

No routing key, messages sent to all bound queues
T...
Topic Exchange Binding Keys
namespace.delimited.keys
#

Receive all messages
namespace.#

Receive all messages in the name...
Example Exchange Plugins
Consistent Hashing

Distribute messages via
routing key hashed value
Pulse

Publish internal Rabb...
Exchange to Exchange Bindings
X
X
X
Queue
Queue
Queue
to RabbitMQ
pg_amqp
• PostgreSQL Extension
• User-defined functions to publish via AMQP
• Developed by OmnTI
• Invoke from user-defined ...
Using pg_amqp
• Has table of AMQP broker connections
• Publishes body only AMQP messages
• Transactional publishing via am...
https://github.com/gmr/On-Rabbits-and-ElephantsExample:
SELECT amqp.publish(broker_id,
'name',
'routing-key',
'message');
...
PgSQL Listen Exchange
• Exchange that issues LISTEN and publishes received
notifications to bound queues
• When routing mes...
Sending a Notification
psql (9.3.2)
Type "help" for help.
!
postgres=# timing
Timing is on.
postgres=# NOTIFY channel_name,...
Receiving a Notification
psql (9.3.2)
Type "help" for help.
!
postgres=# LISTEN channel_name;
!
(after NOTIFY issued)
!
Asy...
Example Notification Message
to PostgreSQL
No direct integrations yet
Chapter 14
Creating a pgsql-storage
RabbitMQ plugin
Other ways?
via Consumer apps in the language of your choice
via Apache Flume with RabbitMQ Source,
PostgreSQL Sink
Questions?
rabbitmq-pgsql-listen-exchange

https://github.com/aweber/rabbitmq-pgsql-listen-exchange
pg_ampq

http://pgxn.o...
Integrating PostgreSql with RabbitMQ
Upcoming SlideShare
Loading in...5
×

Integrating PostgreSql with RabbitMQ

3,501

Published on

Slides from my PgConf NYC 2014 talk on integrating PostgreSQL and RabbitMQ.

Published in: Data & Analytics, Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,501
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
51
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Transcript of "Integrating PostgreSql with RabbitMQ"

  1. 1. Integrating PostgreSQL with RabbitMQ Gavin M. Roy April 4th, 2014
  2. 2. About Me VP of Architecture
 AWeber Communications ! Blog: http://gavinroy.com Twitter: @Crad GitHub: https://github.com/gmr ! RabbitMQ in Depth (MEAP) http://manning.com/roy
  3. 3. RabbitMQ Primer
  4. 4. About RabbitMQ • Message Oriented Middleware • Primary identity is defined by AMQP 0-9-1 • Supports other protocols with Plugins:
 
 AMQP 1.0, HTTP, MQTT, STOMP, XMPP, UDP, WebStomp, and more • Written in Erlang/OTP, is Open Source (MPL), and is developed/maintained by Pivotal
  5. 5. Why RabbitMQ? • Create loosely coupled architectures • Decouple database write operations • Communicate across application platforms • Tap into pre-existing message flow for new purposes • Scale-out clustering for growth, throughput & HA • Federation for WAN latencies & network partitions
  6. 6. Native AMQP Clients C Clojure Cobol Common Lisp Delphi Erlang Go Groovy Haskell Java JavaScript .NET OCaml Perl PHP Python Ruby Scala
  7. 7. Who Uses It? Agora Games Chef Google AdMob Instagram MeetMe Mercado Libre Mozilla Nasa New York Times NSF Openstack Reddit
  8. 8. RabbitMQ Concepts
  9. 9. AMQP Message Body Properties
  10. 10. Message Properties app-id content-encoding content-type correlation-id delivery-mode expiration headers message-id priority reply-to timestamp type user-id Body Properties
  11. 11. Publishers and Consumers CP
  12. 12. Multiple Publishers P C P P
  13. 13. Multiple Consumers P C C C
  14. 14. AMQ Model Exchange
 defines routing behavior Queue
 stores messages in memory and optionally on disk Binding
 defines relationship between exchanges and queues X Queue Binding
  15. 15. Routing Keys • Provided when publishing a message • Compared against binding keys by exchanges • Example uses of a Routing Key: • Connotate the type of message • Designate the destination of a message • Categorize the content in the message
  16. 16. Built-In Exchange Types Direct
 String matching on Routing Key Fanout
 No routing key, messages sent to all bound queues Topic
 Pattern matching in Routing Key Headers
 No routing key, string matching in the message headers property
  17. 17. Topic Exchange Binding Keys namespace.delimited.keys #
 Receive all messages namespace.#
 Receive all messages in the namespace namespace.delimited.*
 Receive all namespace.delimited messages namespace.*.keys
 Receive all namespace messages ending in keys
  18. 18. Example Exchange Plugins Consistent Hashing
 Distribute messages via routing key hashed value Pulse
 Publish internal RabbitMQ metrics via AMQP Random
 Distribute messages across all bound queues randomly Recent History
 Stores last 20 messages to any bound queue Riak Storage 
 Store received messages in Riak" Script Exchange
 Calls out to external scripts for routing behavior
  19. 19. Exchange to Exchange Bindings X X X Queue Queue Queue
  20. 20. to RabbitMQ
  21. 21. pg_amqp • PostgreSQL Extension • User-defined functions to publish via AMQP • Developed by OmnTI • Invoke from user-defined function • Available on PGXN
  22. 22. Using pg_amqp • Has table of AMQP broker connections • Publishes body only AMQP messages • Transactional publishing via amqp.publish • Non-transactional via amqp.autonomous_publish • Needs some love for additional features
  23. 23. https://github.com/gmr/On-Rabbits-and-ElephantsExample: SELECT amqp.publish(broker_id, 'name', 'routing-key', 'message'); Publishing a Message
  24. 24. PgSQL Listen Exchange • Exchange that issues LISTEN and publishes received notifications to bound queues • When routing messages, bindings are checked for the routing key to match the NOTIFY channel • Unobtrusive to PostgreSQL environment, native constructs for publishing messages • Requires PostgreSQL 9.0 and greater • Body only messages*
  25. 25. Sending a Notification psql (9.3.2) Type "help" for help. ! postgres=# timing Timing is on. postgres=# NOTIFY channel_name, 
 ‘Test notification'; NOTIFY Time: 0.160 ms
  26. 26. Receiving a Notification psql (9.3.2) Type "help" for help. ! postgres=# LISTEN channel_name; ! (after NOTIFY issued) ! Asynchronous notification "channel_name" with payload "Test notification" received from server process with PID 16749.
  27. 27. Example Notification Message
  28. 28. to PostgreSQL
  29. 29. No direct integrations yet
  30. 30. Chapter 14 Creating a pgsql-storage RabbitMQ plugin
  31. 31. Other ways? via Consumer apps in the language of your choice via Apache Flume with RabbitMQ Source, PostgreSQL Sink
  32. 32. Questions? rabbitmq-pgsql-listen-exchange
 https://github.com/aweber/rabbitmq-pgsql-listen-exchange pg_ampq
 http://pgxn.org/dist/pg_amqp/ RabbitMQ in Depth 
 44% off code pgcf (includes other books as well)
 http://manning.com/roy/ Follow me on Twitter
 @Crad
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×