What is message queuing and what are the differences between gearman, kafka, mosquitto and rabbitmq.

1. Message Queuing
Message Queuing
Marian Marinov
mm@yuhu.biz

2. What are we going to review?
➢ Gearman (the most basic of the bunch)
➢ Mosquitto
➢ Kafka
➢ RabbitMQ

3. PUB/SUB model
➢ Gearman - client -> worker
➢ Mosquitto - publish -> subscribe
➢ Kafka - publish -> subscribe
➢ RabbitMQ - producer-> consumer
➢ All of them have a broker in the middle
➢ I intentionally omit ZeroMQ and POSIX Message Queues

4. PUB/SUB model
➢ Gearman - client -> worker
➢ Mosquitto - publish -> subscribe
➢ Kafka - publish -> subscribe
➢ RabbitMQ - producer-> consumer
➢ All of them have a broker in the middle
➢ I intentionally omit ZeroMQ and POSIX Message Queues
➢ Also intentionally skipped ActiveMQ and JBossMQ :)

5. But what is it?
app1 <-> broker
| <-> app2
| <-> app2
| <-> app2
<-> app2
main() {
user_data = get_user_data(user_id);
}
where main() will be app1 and get_user_data()
will be app2

6. But what is it?
app1 can be in any language
In the different brokers, it is called:
➢ Client (gearman)
➢ Publisher (Kafka/Mosquitto)
➢ Producer (RabbitMQ)

7. But what is it?
app2 can be in any language,
In the different brokers, it is called:
➢ Worker (gearman)
➢ Subscriber (Kafka/Mosquitto)
➢ Consumer (RabbitMQ)

8. But what is it?
The brokers can implement different schemas of
executions:
➢ Round-robin (sync/async)
➢ Weighted
➢ Fanout (sent to all connected workers), only
async
➢ Topic (or list of topics)

9. Why messaging?
➢ Lighter way to "call" an API

10. Why messaging?
➢ Lighter way to "call" an API
➢ Use multiple languages, seamlessly

11. Why messaging?
➢ Lighter way to "call" an API
➢ Use multiple languages, seamlessly
➢ Loose the possibility to send values by reference

12. Why messaging?
➢ Lighter way to "call" an API
➢ Use multiple languages, seamlessly
➢ Loose the possibility to send values by reference
➢ Self confined interfaces

13. Why messaging?
➢ Lighter way to "call" an API
➢ Use multiple languages, seamlessly
➢ Loose the possibility to send values by reference
➢ Self confined interfaces
➢ Parallelism and extreme scalability

14. Why messaging?
➢ Lighter way to "call" an API
➢ Use multiple languages, seamlessly
➢ Loose the possibility to send values by reference
➢ Self confined interfaces
➢ Parallelism and extreme scalability
➢ Control over resource usage

15. Why messaging?
➢ Lighter way to "call" an API
➢ Use multiple languages, seamlessly
➢ Loose the possibility to send values by reference
➢ Self confined interfaces
➢ Parallelism and extreme scalability
➢ Control over resource usage
➢ Easy access separation

16. Broker architecture
➢ Gearman - single server
➢ Mosquitto - single server (HA in Pro)
➢ Kafka - single or distributed
➢ RabbitMQ - distributed but can work on
single server
* Kafka and RabbitMQ have suggested minimum number of 3 servers

17. Message/Topic storage
➢ Gearman
➢ messages are retained until consumed
➢ own protocol (very lightweight)
➢ Mosquitto
➢ message expiration can be configured
➢ topic hierarchy sensors/+/temperature/#
➢ MQTT protocol (lightwight)
➢ Kafka
➢ messages are deleted if not consumed in 7d(default)
➢ own protocol
➢ RabbitMQ
➢ STOMP, MQTT, AMQP, HTTP and WebSockets
➢ message delivery and durability depends :)

18. Message/Topic delivery
➢ Gearman
➢ sync or async(round-robin)
➢ Mosquitto
➢ topic hierarchy sensors/+/temperature/#
➢ Kafka
➢ topics
➢ messages are not deleted on consumption
➢ RabbitMQ
➢ direct(sync), fanout(broadcast), topic or match

19. Actual communication
Broker
Server
Server
Server
Server
app1
app2
app2
app2
app2

20. Actual communication
Broker
Server
Server
Server
Server
app1
app2
app2
app2
app2
JSON/BSON/Protobuf
JSON/BSON/Protobuf

21. Actual communication
Broker
Server
Server
Server
Server
app1
app2
app2
app2
app2
JSON/BSON/Protobuf
JSON/BSON/Protobuf
get_client_info('{
“user_id”: 414895,
“type”:”payment
}');

22. Ecosystem - Kafka
➢ Very well integrated into Java and
generally the GOTO solution for messaging
➢ Kafka Streams is the library that you have
to take a look at

23. Ecosystem - Gearman
➢“Java gearman service” is a asynchronous
interface written entirely in Java
➢ Gearman Java a pure Java driver exists
on Launchpad

24. Ecosystem - Mosquitto
➢ Eclipse Paho Java Client

25. Ecosystem - RabbitMQ
➢ Spring AMQP
➢ Java Client
➢ JMS Client
➢ Hop
➢ Reactor

26. Thank you!
If you have any questions
find me at the back