This document discusses using RabbitMQ and message queues to process events from connected devices in a microservices architecture. RabbitMQ can be used to implement common messaging patterns like publish/subscribe and push/pull to distribute processing of device data across multiple services. It provides examples of using RabbitMQ exchanges, queues, and bindings to route messages based on topics and load balance work. Error handling is also addressed through retry queues and dedicated error services.

1. Event processing with Node.js and RabbitMQ
by Zlatko Duric

2. Disclaimer (AKA about me)
● Experience
– Node.js? Lots
– NoSQL? Lots.
– Messaging? Little
● Backend Developer at ThinxNet.com
(http://thinxnet.com/#jobs)

3. Connected cars?
● Hardware device
● Various sensors – lots of data
● “Cloud” processing

4. Ready to serve. How?
● Event collection and processing
● Webapp, mobile app
● Backoffice and support apps, notification services etc.
● Reporting, third party software integration etc
● REST approach:
– Lots of requests
– Parsing of data is blocking
– Avoid choking devices
– Communication overhead
– Error management

5. What service? Microservice!
● Redis PUBSUB
● ZeroMQ, MQTT, AMQP
● RabbitMQ, ActiveMQ

6. About RabbitMQ
● Messaging broker based on AMQP protocol
● Platform/language independent
● Extensible (Erlang)
● Persistent message queues
● Node.js drivers (node-amqp, rabbit.js...)

7. Hello world

8. Exchanges and topics
● Publisher and subscriber. Simple as that.
● Get message, ack or reject (nack) it.
● Subscribers listen on a queue
● Publishers talk to an exchange
● Exchange routes to queues based on topics

9. Example

10. Channels and connections
● 1 node process = 1 connection
● Many channels, share same connection

11. PUBSUB
● Simple PUBSUB, as used elsewhere
● All listeners get the same message
● Suboptimal

12. Example

13. PUSH-PULL/WORKER
● Round-robin distribution
● Worker socket.ack() is explicit (rabbit.js
driver)
● Prefetch sets the number of msgs received
● Problem: concurrent/sequential
modification?

14. Example

15. Content-based routing
● Erlang extensions? Not standard.
● App-side:
– Keep internal mini-queue
– Do not ack a msg if it's not first on local
queue
– Hold internal messages based on
internal “topic”, ie “user.<ID>.save” -
hold msgs for that user
– Item.update() instead of item.save()
● Gets nasty fast, should minimize
these scenarios

16. Example

17. REST API with MQ
● REQUEST/REPLY for immediate responses
● Other patterns for direct access
– REST API just forwards the message
– replies will go by other route
● Pub/sub is easy!

18. Example

19. Error handling
● Uncaught exceptions are covered
● Retry logic is simplified – push incomplete
messages to a 'fixer' queue, requeue failed
jobs or when no resources
● Dedicated error handling

20. Example

21. Error handling
● Uncaught exceptions are covered
● Retry logic is simplified – push incomplete
messages to a 'fixer' queue, requeue failed
jobs or when no resources
● Dedicated error handling

22. Testing
● No good known mocks, please help
● RabbitMQ always on on Jenkins machine
● Test handlers directly

23. Buzzwords index
● Cloud
● Connected car
● Microservice
● Patterns
:)

24. Info sources
● https://www.rabbitmq.com/
● http://www.squaremobius.net/rabbit.js/
● http://www.manning.com/videla/

25. Note to self
● Thank the audience for patience
● Invite audience for a discussion and questions