Overview for RabbitMQ with AMQP protocol and a discussion for the AMQP model with message properties and publish and consume message types of exchange main properties of the queue

1. RabbitMQ Part1
By:Mohammed Shaban

2. Agenda
• Message Broker
• RabbitMQ Overview
• RabbitMQ Concepts
• AMQP
• RabbitMQ Workfolw
• Message Properties
• Publishing Messages
• Consuming Messages
• Rejecting Messages
• Controlling queues

3. Message Broker

4. Message Broker
A message broker is an architectural pattern for message
validation, transformation, and routing. It mediates
communication among applications, minimizing the mutual
awareness that applications should have of each other in order to
be able to exchange messages, effectively implementing
decoupling.

5. Message Broker Actions
• Route messages to one or more destinations
• Transform messages to an alternative representation
• Perform message aggregation, decomposing messages into multiple
messages and sending them to their destination, then recomposing the
responses into one message to return to the user
• Interact with an external repository to augment a message or store it
• Invoke web services to retrieve data
• Respond to events or errors
• Provide content and topic-based message routing using the publish–
subscribe pattern

6. List of message broker software
• Amazon Web Services (AWS) Simple Queue Service (SQS)
• Apache ActiveMQ
• Apache Kafka
• Apache Qpid
• Celery
• Cloverleaf (E-Novation Lifeline)
• Comverse Message Broker (Comverse Technology)
• Enduro/X Transactional Message Queue (TMQ)
• Financial Fusion Message Broker (Sybase)
• Fuse Message Broker (enterprise ActiveMQ)
• Gearman

7. List of message broker software Cont.
• HornetQ (Red Hat)
• IBM Integration Bus
• IBM Message Queues / IBM WebSphere MQ
• JBoss Messaging (JBoss)
• JORAM
• Microsoft Azure Service Bus (Microsoft)
• Microsoft BizTalk Server (Microsoft)
• NATS (MIT Open Source License, written in Go)
• Open Message Queue

8. List of message broker software Cont.
• Oracle Message Broker (Oracle Corporation)
• RabbitMQ (Mozilla Public License, written in Erlang)
• Redis An open source, in-memory data structure store, used as a
database, cache and message broker.
• SAP PI (SAP AG)
• Solace PubSub+
• Spread Toolkit
• Tarantool, a NoSQL database, with a set of stored procedures for message
queues
• TIBCO Enterprise Message Service
• WSO2 Message Broker

9. RabbitMQ Overview

10. RabbitMQ
RabbitMQ is an open source message broker written in Erlang,
currently under the wing of Pivotal Software. It’s based around
the AMQP open protocol, with official client libraries in Java,
.NET, Erlang, as well as libraries for most other popular
programming languages.

11. RabbitMQ’s features and benefit
• Open source
• Platform and vendor neutral
• Lightweight
• Client libraries for most modern languages
• Flexibility in controlling messaging trade-offs—
• Plugins for higher-latency environments—
• Third-party plugins
• Layers of security

12. RabbitMQ and Erlang
It was written in Erlang, the telco-grade, functional programming
language designed at the Ericsson Computer Science Laboratory in
the mid-to-late 1980s. Erlang was designed to be a distributed,
fault-tolerant, soft real-time system for applications that require
99.999% uptime. As a language and runtime system, Erlang
focuses on lightweight processes that pass messages between
each other, providing a high level of concurrency with no shared
state.

13. Message-oriented middleware (MOM)
Software or hardware infrastructure that allows for the sending and
receiving of messages from distributed systems. RabbitMQ fills this
role handily with functionality that provides advanced routing and
message distribution, even with wide area network (WAN) tolerances
to support reliable, distributed systems that interconnect with other
systems easily.

14. Loosely Coupled ArchitectureHigh Coupled Architecture
Loosely Coupled Architecture

15. Loosely Coupled ArchitectureHigh Coupled Architecture
Loosely Coupled Architecture

16. Loosely Coupled Architecture

17. Loosely Coupled Architecture

18. Loosely Coupled Architecture

19. Message Queuing model
• Exchange—The component of the message broker that routes
messages to queues
• Queue—A data structure on disk or in memory that stores
messages
• Binding—A rule that tells the exchange which queue the messages
should be stored in

20. Messaging Queue Model
Routing
Message Queuing model

21. RabbitMQ Concepts

22. RabbitMQ Overview

23. RabbitMQ Concepts
• Producer: Application that sends the messages.
• Consumer: Application that receives the messages.
• Queue: Buffer that stores messages.
• Message: Data that is sent from the producer to a consumer through RabbitMQ.
• Connection: A connection is a TCP connection between your application and the
RabbitMQ broker.
• Channel: A channel is a virtual connection inside a connection. When you are
publishing or consuming messages or subscribing to a queue, it's all done over a
channel.

24. RabbitMQ Concepts Cont.
• Exchange: Receives messages from producers and pushes them to queues
depending on rules defined by the exchange type. A queue needs to be bound to
at least one exchange to be able to receive messages.
• Binding: A binding is a link between a queue and an exchange.
• Routing key: The routing key is a key that the exchange looks at to decide how
to route the message to queues. You can think of the routing key as the
destination address of a message.

25. RabbitMQ Concepts Cont.
• AMQP: The Advanced Message Queuing Protocol is the primary protocol used by
RabbitMQ for messaging.
• Users: It's possible to connect to RabbitMQ with a given username and password.
Every user can be assigned permissions such as rights to read, write and
configure privileges within the instance. Users can also be assigned permissions
to specific virtual hosts.
• Vhost, virtual host: A virtual host provides a way to segregate applications that
are using the same RabbitMQ instance. Different users can have different access
privileges to different vhosts and queues, and exchanges can be created so that
they only exist in one vhost.

26. RabbitMQ Concepts Cont.
• Acknowledgments and Confirms: Acknowledgments and confirms indicate that
messages have been received or acted upon. Acknowledgments can be used in
both directions; A consumer can indicate to the server that it has
received/processed a message, and the server could report

27. AMQP
Advanced Message Queuing Protocol

29. AMQP as an RPC transport
• Kicking off the conversation
• The Client send a protocol header
• The server send Connection.Start command
• The client responds with the Connection.StartOK frame
• Tuning in to the right channel

30. AMQP commands
• AMQP uses classes and methods, referred to as AMQP commands,
to create a common language between clients and servers. The
classes in AMQP define a scope of functionality, and each class
contains methods that perform different tasks.
(Class)Connection.Start(Method)

31. AMQP frame components
• When commands are sent to and from RabbitMQ, all of the
arguments required to execute them are encapsulated in data
structures called frames that encode the data for transmission.

32. AMQP frame components

33. AMQP frame components
• Frame type
• Channel number
• Frame size in bytes
• Frame payload
• End-byte marker (ASCII value 206)
Frame Header

34. Types of frames
• The protocol header frame is only used once, when connecting to
RabbitMQ.
• A method frame carries with it the RPC request or response that’s
being sent to or received from RabbitMQ.
• A content header frame contains the size and properties for a
message.
• Body frames contain the content of messages.
• The heartbeat frame is sent to and from RabbitMQ as a check to ensure
that both sides of the connection are available and working properly.

35. Method Frame
Header Frame
Body Frame
Types of frames

36. Sample Method Frame

37. Sample Header Frame

38. Sample Body Frame

39. RabbitMQ Workfolw
Publish and Consume message

40. 1. The producer publishes a message to an
exchange. When you create the exchange, you
have to specify the type of it. The different
types of exchanges are explained in detail later
on.
2. The exchange receives the message and is now
responsible for the routing of the message. The
exchange looks at different message attributes
and keys depending on the exchange type.
3. In this case, we see two bindings to two different
queues from the exchange. The exchange routes
the message to the correct queue, depending on
its attributes.
4. The messages stay in the queue until a consumer
handles them.
5. The consumer handles the message, thus
removing it from the queue.
The message flow in RabbitMQ

41. Publish-Consumer Steps
• Declaring an exchange
• Declaring a queue
• Binding a queue to an exchange
• Publishing a message to RabbitMQ
• Consuming messages from RabbitMQ

42. Declaring an exchange
• Exchanges are AMQP entities where messages are
sent. Exchanges take a message and route it into
zero or more queues. The routing algorithm used
depends on the exchange type and rules called
bindings. AMQP 0-9-1 brokers provide four exchange
types:
• Direct : delivers messages to queues based on a message
routing key. In a direct exchange, the message is routed to
the queue whose binding key exactly matches the routing
key of the message.
• Fanout : routes messages to all of the queues that are
bound to it.
• Topic : performs a wildcard match between the routing key
and the routing pattern specified in the binding.
• Headers : use the message header attributes to do their
routing.

43. Declaring a queue
• They store messages that are consumed by applications. Queues share some
properties with exchanges, but also have some additional properties:
• Name
• Durable : A durable queue ensures that RabbitMQ never loses the queue
• Message TTL: The time a message published to a queue can live before it's discarded.Auto-
delete
• Auto-expire: The time a queue can be unused before it's automatically deleted.
• Max length: How many (ready) messages a queue can hold before it starts to drop them.
• Max length bytes: The total body size of ready messages a queue can contain before it starts
to
• drop them.

44. Binding a queue to an exchange
• Once the exchange and queue have been created, it’s time to bind
them together.

45. Publishing a message to RabbitMQ

46. Consuming messages from RabbitMQ
• To consume messages from a queue in RabbitMQ, a
consumer application subscribes to the queue in
RabbitMQ by issuing a Basic.Consume command.
• The server will respond with Basic.ConsumeOk
• The consumer will start receiving messages in the
unsurprising form of Basic.Deliver methods and their
content header and body frame counterparts

47. Writing a message publisher in .Net

48. Getting messages from RabbitMQ in .Net

49. Message Properties

50. Message Publishing

51. Message Properties in Header Frame

52. Property Type For use by Suggested or specified use
content-type short-string Application Specify the type of the message body using mime-types.
content-encoding short-string Application Specify whether your message body is encoded in some special
way, such as zlib, deflate, or Base64.
Message-id/Correlation-id :
enables the message to carry data in the header that uniquely identifies it as it flows through the various
components in a loosely coupled system.
correlation-id short-string Application If the message is in reference to some other message or uniquely
identifiable item, the correlation-id is a good way to indicate
what the message is referencing.
message-id short-string Application A unique identifier such as a UUID that your application can use to
identify the message.
timestamp timestamp Application An epoch or Unix timestamp value that can be used to indicate
when the message was created.
Expiration :
In declaring a queue, you can pass an x-message-ttl argument along with the queue definition.
expiration short-string Application An epoch or Unix timestamp value as a text string that indicates
when the message should expire.
Message Properties

53. Property Type For use by Suggested or specified use
delivery-mode octet RabbitMQ A value of 1 tells RabbitMQ it can keep the message in memory; 2
indicates it should also write it to disk.
app-id short-string Application Useful for defining the application publishing the messages.
user-id short-string Application A free-form string that, if used, RabbitMQ will validate against the
connected user and drop messages if they don’t match.
type short-string Application A text string your application can use to describe the message
type or payload.
reply-to short-string Application Can be used to carry a queu name or the routing key a consumer
should use when replying to a message implementing an RPC
pattern.
headers table Both A free-form key/value table that you can use to add additional
metadata about your message; RabbitMQ can route based upon
this if desired.
priority octet RabbitMQ A property for priority ordering in queues.
Message Properties Cont.

54. Publishing Messages
Practices

55. Publishing message facets
Performance Reliability

56. balance between high performance and
message safety
• How important is it that messages are guaranteed to be en-queued when published?
• Should a message be returned to a publisher if it can’t be routed?
• If a message can’t be routed, should it be sent somewhere else where it can later be reconciled?
• Is it okay if messages are lost when a RabbitMQ server crashes?
• Should RabbitMQ confirm that it has performed all requested routing and persistence tasks to a
publisher when it processes a new message?
• Should a publisher be able to batch message deliveries and then receive confirmation from
RabbitMQ that all requested routing and persistence tasks have been applied to all of the
messages in the batch?
• If you’re batching the publishing of messages that require confirmation of routing and
persistence, is there a need for true atomic commits to the destination queues for a message?
• Are there acceptable trade-offs in reliable delivery that your publishers can use to achieve
higher performance and message throughput?
• What other aspects of message publishing will impact message throughput and performance?

57. RabbitMQ won’t accept non-routable messages with mandatory set

58. Publisher Confirms as a lightweight alternative to transactions

59. Using alternate exchanges for unroutable messages

60. Batch processing with transactions

61. Surviving node failures with HA queues

62. Persisting messages to disk via delivery-mode 2
In addition to delivery-mode of 2, for messages to truly survive a restart of a RabbitMQ broker, your queues must be
declared as durable when they’re created.
delivery-mode: 1
delivery-mode: 2

63. Consuming Messages
Practices

64. Basic.Get
Get is acting as Pull message from the queue if there is any messages on it
If There is a messages If There is not any messages

65. Basic.Consume
Consume is acting as push message from the queue to the consumer

66. Consuming message facets

67. With no-Ack if the socket
buffer is not full and the
rabbitmq can write to it
the rabbitmq will write the
new messages but if the
ack is required the
rabbitmq will not write a
new message until the
consumer send ack
Using no-ack mode for faster throughput

68. Controlling consumer prefetching via quality of service settings
AMQP specifies the basic.qos method
to allow you to limit the number of
unacknowledged messages on a
channel (or connection) when
consuming (aka "prefetch count").
Simple benchmark results for consuming with no QoS set and
different prefetch count values

69. Using transactions with consumers
Transactions don’t work for consumers
with acknowledgments disabled.
Message velocities when using transactions compared to non-
transactional message velocities

70. Rejecting Messages

71. Rejecting messagesRejecting Message
Basic.Reject
Basic.Nack
Dead Letter Exhange

72. A consumer can acknowledge, reject, or negatively acknowledge a message. Basic.Nack allows for multiple messages to
be rejected at once, whereas Basic.Reject allows just one message to be rejected at a time.

73. A rejected message can be routed as a dead-letter
message through another exchange.
Dead Letter Exchange
• The message is rejected (basic.reject
or basic.nack) with requeue=false,
• The TTL for the message expires; or
• The queue length limit is exceeded.
If the dead letter exchange is missing
then, the messages will be silently
dropped.

74. Controlling queues

75. Temporary queues
• Automatically deleting queues
• the queue will only delete itself when there are no more consumers
listening to it.
• Allowing only a single consumer
• An exclusive queue will also automatically be deleted when the channel
that the queue was created on is closed
• Automatically expiring queues
• RabbitMQ will delete the queue if it has gone unused for some length of
time.

76. Permanent queues
• Queue durability
• When declaring a queue that should persist across server restarts.
• Auto-expiration of messages in a queue
• The stale data that should be removed after its usefulness has expired
• Maximum length queues
• once the queue messages reaches the maximum size, RabbitMQ will drop
messages from the front of the queue as new messages are added.

77. Queue settings
Argument name Purpose
x-dead-letter-exchange An exchange to which non-requeued rejected messages are routed
x-dead-letter-routing-key An optional routing key for dead-lettered messages
x-expires Queue is removed after the specified number of milliseconds
x-ha-policy When creating HA queues, specifies the mode for enforcing HA
across nodes
x-ha-nodes The nodes that an HA queue is distributed across
x-max-length The maximum message count for a queue
x-message-ttl Message expiration in milliseconds, enforced at the queue level
x-max-priority Enables priority sorting of a queue with a maximum priority value of
255

78. Thanks