Download as PDF, PPTX
Uploaded byGavin Roy
Integrating PostgreSql with RabbitMQ
This document discusses integrating RabbitMQ with PostgreSQL by using RabbitMQ as a messaging system to publish notifications from PostgreSQL. It describes pg_amqp, a PostgreSQL extension that allows publishing messages via AMQP from user-defined functions. It also describes a PostgreSQL Listen Exchange in RabbitMQ that issues LISTEN commands and publishes received notifications to queues based on routing keys matching the notification channel.
More Related Content
Similar to Integrating PostgreSql with RabbitMQ
Integrating PostgreSql with RabbitMQ
- 1. Integrating PostgreSQL withRabbitMQ Gavin M. Roy April 4th, 2014
- 2. About Me VP ofArchitecture AWeber Communications ! Blog: http://gavinroy.com Twitter: @Crad GitHub: https://github.com/gmr ! RabbitMQ in Depth (MEAP) http://manning.com/roy
- 3.
- 4. About RabbitMQ • MessageOriented Middleware • Primary identity is defined by AMQP 0-9-1 • Supports other protocols with Plugins: AMQP 1.0, HTTP, MQTT, STOMP, XMPP, UDP, WebStomp, and more • Written in Erlang/OTP, is Open Source (MPL), and is developed/maintained by Pivotal
- 5. Why RabbitMQ? • Createloosely coupled architectures • Decouple database write operations • Communicate across application platforms • Tap into pre-existing message flow for new purposes • Scale-out clustering for growth, throughput & HA • Federation for WAN latencies & network partitions
- 6. Native AMQP Clients C Clojure Cobol CommonLisp Delphi Erlang Go Groovy Haskell Java JavaScript .NET OCaml Perl PHP Python Ruby Scala
- 7. Who Uses It? AgoraGames Chef Google AdMob Instagram MeetMe Mercado Libre Mozilla Nasa New York Times NSF Openstack Reddit
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14. AMQ Model Exchange defines routingbehavior Queue stores messages in memory and optionally on disk Binding defines relationship between exchanges and queues X Queue Binding
- 15. Routing Keys • Providedwhen publishing a message • Compared against binding keys by exchanges • Example uses of a Routing Key: • Connotate the type of message • Designate the destination of a message • Categorize the content in the message
- 16. Built-In Exchange Types Direct Stringmatching on Routing Key Fanout No routing key, messages sent to all bound queues Topic Pattern matching in Routing Key Headers No routing key, string matching in the message headers property
- 17. Topic Exchange BindingKeys namespace.delimited.keys # Receive all messages namespace.# Receive all messages in the namespace namespace.delimited.* Receive all namespace.delimited messages namespace.*.keys Receive all namespace messages ending in keys
- 18. Example Exchange Plugins ConsistentHashing Distribute messages via routing key hashed value Pulse Publish internal RabbitMQ metrics via AMQP Random Distribute messages across all bound queues randomly Recent History Stores last 20 messages to any bound queue Riak Storage Store received messages in Riak" Script Exchange Calls out to external scripts for routing behavior
- 19. Exchange to ExchangeBindings X X X Queue Queue Queue
- 20.
- 21. pg_amqp • PostgreSQL Extension •User-defined functions to publish via AMQP • Developed by OmnTI • Invoke from user-defined function • Available on PGXN
- 22. Using pg_amqp • Hastable of AMQP broker connections • Publishes body only AMQP messages • Transactional publishing via amqp.publish • Non-transactional via amqp.autonomous_publish • Needs some love for additional features
- 23.
- 24. PgSQL Listen Exchange •Exchange that issues LISTEN and publishes received notifications to bound queues • When routing messages, bindings are checked for the routing key to match the NOTIFY channel • Unobtrusive to PostgreSQL environment, native constructs for publishing messages • Requires PostgreSQL 9.0 and greater • Body only messages*
- 25. Sending a Notification psql(9.3.2) Type "help" for help. ! postgres=# timing Timing is on. postgres=# NOTIFY channel_name, ‘Test notification'; NOTIFY Time: 0.160 ms
- 26. Receiving a Notification psql(9.3.2) Type "help" for help. ! postgres=# LISTEN channel_name; ! (after NOTIFY issued) ! Asynchronous notification "channel_name" with payload "Test notification" received from server process with PID 16749.
- 28.
- 29.
- 30.
- 31. Chapter 14 Creating apgsql-storage RabbitMQ plugin
- 32. Other ways? via Consumerapps in the language of your choice via Apache Flume with RabbitMQ Source, PostgreSQL Sink
- 33. Questions? rabbitmq-pgsql-listen-exchange https://github.com/aweber/rabbitmq-pgsql-listen-exchange pg_ampq http://pgxn.org/dist/pg_amqp/ RabbitMQ in Depth 44% off code pgcf (includes other books as well) http://manning.com/roy/ Follow me on Twitter @Crad