Slides used in a presentation on data integration using message systems with RabbitMQ and Celery.

1. Data Integration with
RabbitMQ and Celery
Guaranteed Delivery
@edytarcio

2. Message Styles - Recap
 Scp. RSync. Database replication. Rest/Soap. RPC. Messages patterns

3. What is RabbitMQ
RabbitMQ
 Broker/ Message Server
 Erlang
 AMQP
 Language Agnostic
Celery
● Dynamic high level lib
● Synchronous/Asynchronous task
queue processing
● Scheduling tasks
● Monitoring based
● Python
● RabbitMQ client
● It’s also a server

4. Some RabbitMQ Features
 No message loss
 Persistent messages
 Publish confirms
 Message acknowledgment
 Mirrored queues
 Message priority
 Dead letter exchange
 Alternate exchange
 Messages and Queues TTLs
 Federation

5. Technologies & Responsabilities

6. Message flow

7. Messages Patterns - Publish/Subscribe

8. Messages Patterns - Request/Reply

9. Topologies - We started with Canonical Data Models
 Do not go for “fully connected”, Canonical approach is best!

10. What exactly are we trying to achieve?
 Low latency. The message shouldn’t take long
to get to its destination.
 Leaky buffer control. Acknowledgement.
 Resilient system. Able to recover from failures.
You don’t lose data.
 Connection failure. When the server is offline
● Fault tolerance. Celery Retry/ Dead Letter

11. Refining RabbitMq Model...
High Availability
 Clustering.
 Load balancer.
Which way to go?

12. Federation & Shovel for rescue
… but how do they
help us?

13. That’s how Federation/Shovel helps us

14. Distributed RabbitMq using Federation/Shovel

15. Anatomy of a message
Numbers
max msg publish/s
149910
max msg consume/s
64315
To conclude...

16. Thanks!
 https://www.rabbitmq.com/
 http://zeromq.org/
 http://www.celeryproject.org/
 https://derickbailey.com/email-courses/rabbitmq-patterns-for-applications/
 http://www.enterpriseintegrationpatterns.com/
 https://www.google.com.br/
 Pictures were taken from the WEB!
edytarcio.silva@armazemparaiba.com.br

17. Wait there’s more...
Brokerless
● Peer-to-peer
● Multi-connection