RabbitMQ is a message broker that allows for asynchronous and reliable messaging. It supports features like guaranteed delivery, acknowledgments, and mirroring. Celery is a task queue library for Python that uses RabbitMQ. It allows for scheduling, monitoring, and retrying of asynchronous tasks. RabbitMQ and Celery can be used together to build distributed systems with low latency, fault tolerance, and resilience through acknowledgments, clustering, load balancing, and retry mechanisms.

1. Data Integration with
RabbitMQ and Celery
Guaranteed Delivery
@edytarcio

2. Message Styles - Recap
 Scp. RSync. Database replication. Rest/Soap. RPC. Messages patterns

3. What is RabbitMQ
RabbitMQ
 Broker/ Message Server
 Erlang
 AMQP
 Language Agnostic
Celery
● Dynamic high level lib
● Synchronous/Asynchronous task
queue processing
● Scheduling tasks
● Monitoring based
● Python
● RabbitMQ client
● It’s also a server

4. Some RabbitMQ Features
 No message loss
 Persistent messages
 Publish confirms
 Message acknowledgment
 Mirrored queues
 Message priority
 Dead letter exchange
 Alternate exchange
 Messages and Queues TTLs
 Federation

5. Technologies & Responsabilities

6. Message flow

7. Messages Patterns - Publish/Subscribe

8. Messages Patterns - Request/Reply

9. Topologies - We started with Canonical Data Models
 Do not go for “fully connected”, Canonical approach is best!

10. What exactly are we trying to achieve?
 Low latency. The message shouldn’t take long
to get to its destination.
 Leaky buffer control. Acknowledgement.
 Resilient system. Able to recover from failures.
You don’t lose data.
 Connection failure. When the server is offline
● Fault tolerance. Celery Retry/ Dead Letter

11. Refining RabbitMq Model...
High Availability
 Clustering.
 Load balancer.
Which way to go?

12. Federation & Shovel for rescue
… but how do they
help us?

13. That’s how Federation/Shovel helps us

14. Distributed RabbitMq using Federation/Shovel

15. Anatomy of a message
Numbers
max msg publish/s
149910
max msg consume/s
64315
To conclude...

16. Thanks!
 https://www.rabbitmq.com/
 http://zeromq.org/
 http://www.celeryproject.org/
 https://derickbailey.com/email-courses/rabbitmq-patterns-for-applications/
 http://www.enterpriseintegrationpatterns.com/
 https://www.google.com.br/
 Pictures were taken from the WEB!
edytarcio.silva@armazemparaiba.com.br

17. Wait there’s more...
Brokerless
● Peer-to-peer
● Multi-connection