Your SlideShare is downloading. ×
0
Real World Messaging
With Apache ActiveMQ
    Bruce Snyder
    bsnyder@apache.org
    7 Nov 2008
    New Orleans, Louisiana
Do You Use JMS?




                  2
Agenda

• Common questions
• ActiveMQ features




                      3
What is ActiveMQ?

• Message-oriented middleware
• Apache project
  – http://activemq.apache.org/
• Apache licensed
• JMS ...
What kind of hardware is
needed for a throughput of
X number of messages per
         second?



                       5
What Are Your
Performance Objectives?




                     6
What Trade-Offs
Can You Accept?




                  7
Common Trade-Offs


•   Messaging Domain
•   Durability v. Persistence
•   Message acks v. transactions
•   Message consum...
Messaging Domains

• Do all consumers need a copy of the
  messages?
• What if consumers are disconnected?
• Can consumers...
Durability v. Persistence

• What quality of service do you need?

  – Durability - should messages be held while
    cons...
Message Acks v.
          Transactions
• When messages are received, the
  consumer must acknowledge the
  message
  – The...
Synchronous Message
        Consumption
public class MyConsumer {
...

public void doCreateConsumer() {

    Destination d...
Asynchronous Message
    Consumption
public class MyConsumer {
...

public void doCreateConsumer() {

    Destination dest...
How can the broker be
    clustered to improve
   scalability or guarantee
        availability?
• Let’s talk about two fe...
Broker Options

•   Connectors
•   Persistence
•   Master/slave
•   Security




                            15
Two Types of Transports




                     16
Transport Connectors

• Client-to-broker connections
  – Similar to JDBC connections to a database
• Various protocols are...
Clients Should Be
    Configured For Failover
• Use the failover protocol:

failover:(tcp://broker1:61616,tcp://broker2:61...
Network of Brokers

• Broker-to-broker connections
• Various protocols supported:
  – Static
  – Failover
  – Multicast
  ...
Networks of Brokers

• Provides large scalability
• ActiveMQ store-and-forward allows
  messages to traverse brokers
  – D...
Topology Example




                   21
Topology Example




                   22
Topology Example




                   23
Topology Example




                   24
Topology Example




                   25
Master/Slave
Broker Configurations




                    26
Three Types of
          Master/Slave
• Pure master/slave
• Shared filesystem master/slave
• JDBC master/slave




       ...
Pure Master/Slave

• Shared nothing, fully replicated topology
  – Does not depend on shared filesystem or
    database
• ...
Pure Master/Slave

• Master broker will only respond to client
  when a message exchange has been
  successfully passed to...
Pure Master/Slave

• If the master fails, the slave optionally
  has two modes of operation:
  – Start up all it’s network...
Shared Filesystem
         Master/Slave
• Utilizes a directory on a shared
  filesystem
• No restriction on number of brok...
JDBC Master/Slave

• Recommended when using a shared
  database
• No restriction on the number of brokers
• Simple configu...
Client Connectivity
       With Master/Slave
• Clients should use failover transport:

     failover://(tcp://masterhost:6...
How can the broker be
    clustered to improve
   scalability or guarantee
        availability?
• Create a network of bro...
High Availability and
         Fault Tolerance
•   RAIDed disks
•   A Storage Area Network
•   Clustered relational databa...
What's the real difference
   between message
 persistence strategies?




                       36
AMQ Message Store

• Transactional message storage solution
• Fast and reliable
• Composed of two parts:
  – Data Store - ...
Non-Journaled JDBC

• Transactional message storage solution
• Reliable but not fast
  – JDBC connection overhead is prohi...
Journaled JDBC

• Transactional message storage solution
• Reliable and faster than non-journaled
• Two-piece store
  – Jo...
Message Cursors

• Messages are no longer stored in
  memory
  – Previous to 5.1, message references were
    stored in me...
What security options are
        available?


• Authentication
  – I.e., are you allowed to connect to ActiveMQ?
• Author...
Authentication

• File based
• JAAS based




                         42
Authorization

• Destination level
• Message level via custom plugin




                                    43
What's the best way to
 connect non-Java clients?


• Let’s talk about wire formats




                                  ...
Wire Formats

• OpenWire
  – The default in ActiveMQ; a binary protocol
• STOMP
  – Simple Text Oriented Messaging Protoco...
OpenWire

• A binary wire level protocol for
  marshaling objects to/from byte arrays
• Designed for performance
  – Sacri...
STOMP

• A text-based wire level protocol
• Designed to be very easy to understand
  – Encourages implementation in additi...
What's the best approach if
 a producer or consumer
     may be down for
        some time?




                       48
Use the failover Protocol
• Utilize the failover options
   – initialReconnectDelay - wait before reconnect
   – maxReconn...
Use Producer Flow Control

• Prevents slow consumers from being
  flooded

        ...
        <destinationPolicy>
       ...
Use Message Persistence

• Quality of service to preserve messages
  in the event of broker failure




                  ...
Consumer Options

• Message prefetch
•   Consumer dispatch async
•   Exclusive consumer
•   Consumer priority
•   Message ...
Message Prefetch

• Used for slow consumer situations
  – Consumer is flooded by messages from the broker
• FIFO buffer on...
Async Dispatch

• Asynchronous message delivery to
  consumers
  – Default is true
• Useful for slow consumers
  – Incurs ...
Exclusive Consumers

• Anytime more than one consumer is consuming
  from a queue, message order is lost
• Allows a single...
Consumer Priority

• Just like it sounds
  – Gives a consumer priority for message delivery
  – Allows for the weighting o...
Message Groups

• Uses the JMSXGroupID property to define which
  message group a message belongs
  – Guarantees ordered p...
Redelivery Policy

• Messages are redelivered to a client
  when:
  – A transacted session is rolled back
  – A transacted...
Retroactive Consumer

• Message replay at start of a subscription
  – At the start of every subscription, send any old
   ...
Wildcards on Destinations

• Price.>
• Price.Stock.>
• Price.Stock.NASDAQ.*
• Price.Stock.*.IBM

            ...
         ...
Selectors

• Used to attach a filter to a subscription
• Defined using a subset SQL 92 syntax
• JMS selectors
   – Filters...
Slow Consumers
   Strategies




                 62
Slow Consumer Strategies

• Various configurable strategies for handling
  slow consumers
• Slow consumer situations are v...
Use Message Limit
           Strategies
• PendingMessageLimitStrategy
  – Calculates the max number of pending messages
  ...
Use Prefetch and
         an Eviction Policy
• Use the prefetch policy
  – The prefetch policy has a property named
    ma...
Use Destination Policies

• Configured on the destination policies in
  the ActiveMQ XML configuration file
• Combined wit...
Additional Tips

• Consider configuring message cursors
• The status of slow consumers can be monitored via
  JMX properti...
How to monitor individual
 messages, queue depths,
 or other broker statistics?

• JMX
• ActiveMQ web console
• Additional...
What is Apache Camel?




 http://activemq.apache.org/camel/
Camel Components
Fluent Java API

RouteBuilder MyRoute = new RouteBuilder() {
   public void configure() {
     from(quot;activemq:TEST.QUE...
XML Config

<camelContext id=quot;camelquot;
  xmlns=quot;http://activemq.apache.org/camel/schema/springquot;>
  <package>...
Information Overload Yet?
Thank You For Attending!




 Questions?
Messaging With ActiveMQ
Upcoming SlideShare
Loading in...5
×

Messaging With ActiveMQ

25,854

Published on

Presentation from ApacheCon US 2008

Published in: Technology
2 Comments
30 Likes
Statistics
Notes
  • very informative,
    thanks
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • awesome overview.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
25,854
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
1,066
Comments
2
Likes
30
Embeds 0
No embeds

No notes for slide

Transcript of "Messaging With ActiveMQ"

  1. 1. Real World Messaging With Apache ActiveMQ Bruce Snyder bsnyder@apache.org 7 Nov 2008 New Orleans, Louisiana
  2. 2. Do You Use JMS? 2
  3. 3. Agenda • Common questions • ActiveMQ features 3
  4. 4. What is ActiveMQ? • Message-oriented middleware • Apache project – http://activemq.apache.org/ • Apache licensed • JMS 1.1 compliant • Goal: – Standards-based, message-oriented application integration across many languages and platforms 4
  5. 5. What kind of hardware is needed for a throughput of X number of messages per second? 5
  6. 6. What Are Your Performance Objectives? 6
  7. 7. What Trade-Offs Can You Accept? 7
  8. 8. Common Trade-Offs • Messaging Domain • Durability v. Persistence • Message acks v. transactions • Message consumption types 8
  9. 9. Messaging Domains • Do all consumers need a copy of the messages? • What if consumers are disconnected? • Can consumers deal with missing messages? • Do you need request/reply messaging? 9
  10. 10. Durability v. Persistence • What quality of service do you need? – Durability - should messages be held while consumer is offline? – Persistence - used to preserve messages in the event of a JMS provider failure 10
  11. 11. Message Acks v. Transactions • When messages are received, the consumer must acknowledge the message – There is no unacknowledge! • Transactions allow for: – Rollback for error handling – Batching for speed 11
  12. 12. Synchronous Message Consumption public class MyConsumer { ... public void doCreateConsumer() { Destination destination = consumer.getSession().createQueue(quot;JOBS.quot; + job); MessageConsumer messageConsumer = consumer.getSession().createConsumer(destination); while ((message = consumer.receive(timeout)) != null) { processMessage(message); } } ... } 12
  13. 13. Asynchronous Message Consumption public class MyConsumer { ... public void doCreateConsumer() { Destination destination = consumer.getSession().createQueue(quot;JOBS.quot; + job); MessageConsumer messageConsumer = consumer.getSession().createConsumer(destination); messageConsumer.setMessageListener(new MyMessageListener(job)); } ... } 13
  14. 14. How can the broker be clustered to improve scalability or guarantee availability? • Let’s talk about two features – Network of brokers – Master/slave 14
  15. 15. Broker Options • Connectors • Persistence • Master/slave • Security 15
  16. 16. Two Types of Transports 16
  17. 17. Transport Connectors • Client-to-broker connections – Similar to JDBC connections to a database • Various protocols are supported: – VM – TCP – NIO – UDP – SSL – HTTP/S 17
  18. 18. Clients Should Be Configured For Failover • Use the failover protocol: failover:(tcp://broker1:61616,tcp://broker2:61616, tcp://broker3:61616)?initialReconnectDelay=100 18
  19. 19. Network of Brokers • Broker-to-broker connections • Various protocols supported: – Static – Failover – Multicast – Zeroconf – Peer – Fanout – Discovery 19
  20. 20. Networks of Brokers • Provides large scalability • ActiveMQ store-and-forward allows messages to traverse brokers – Demand-based forwarding – Some people call this distributed queues • Many possible configurations or topologies are supported 20
  21. 21. Topology Example 21
  22. 22. Topology Example 22
  23. 23. Topology Example 23
  24. 24. Topology Example 24
  25. 25. Topology Example 25
  26. 26. Master/Slave Broker Configurations 26
  27. 27. Three Types of Master/Slave • Pure master/slave • Shared filesystem master/slave • JDBC master/slave 27
  28. 28. Pure Master/Slave • Shared nothing, fully replicated topology – Does not depend on shared filesystem or database • A Slave broker consumes all message states from the Master broker (messages, acks, tx states) • Slave does not start any networking or transport connectors 28
  29. 29. Pure Master/Slave • Master broker will only respond to client when a message exchange has been successfully passed to the slave broker 29
  30. 30. Pure Master/Slave • If the master fails, the slave optionally has two modes of operation: – Start up all it’s network and transport connectors • All clients connected to failed Master resume on Slave – Close down completely • Slave is simply used to duplicate state from Master 30
  31. 31. Shared Filesystem Master/Slave • Utilizes a directory on a shared filesystem • No restriction on number of brokers • Simple configuration (point to the data dir) • One master selected at random 31
  32. 32. JDBC Master/Slave • Recommended when using a shared database • No restriction on the number of brokers • Simple configuration • Clustered database negates single point of failure • One master selected at random 32
  33. 33. Client Connectivity With Master/Slave • Clients should use failover transport: failover://(tcp://masterhost:61616, tcp://slavehost:61616)?randomize=false 33
  34. 34. How can the broker be clustered to improve scalability or guarantee availability? • Create a network of brokers • Use master/slave for broker failover 34
  35. 35. High Availability and Fault Tolerance • RAIDed disks • A Storage Area Network • Clustered relational databases • Clustered JDBC via C-JDBC – http://c-jdbc.objectweb.org/ 35
  36. 36. What's the real difference between message persistence strategies? 36
  37. 37. AMQ Message Store • Transactional message storage solution • Fast and reliable • Composed of two parts: – Data Store - holds messages in a transactional journal – Reference store - stores message locations for fast retrieval • The default message store in ActiveMQ 5 37
  38. 38. Non-Journaled JDBC • Transactional message storage solution • Reliable but not fast – JDBC connection overhead is prohibitively slow 38
  39. 39. Journaled JDBC • Transactional message storage solution • Reliable and faster than non-journaled • Two-piece store – Journal - A high-performance, transactional journal – Database - A relational database of your choice • Default database in ActiveMQ 4.x is Apache Derby 39
  40. 40. Message Cursors • Messages are no longer stored in memory – Previous to 5.1, message references were stored in memory • Messages are paged in from storage when space is available in memory 40
  41. 41. What security options are available? • Authentication – I.e., are you allowed to connect to ActiveMQ? • Authorization – I.e., do you have permission to use that ActiveMQ resource? 41
  42. 42. Authentication • File based • JAAS based 42
  43. 43. Authorization • Destination level • Message level via custom plugin 43
  44. 44. What's the best way to connect non-Java clients? • Let’s talk about wire formats 44
  45. 45. Wire Formats • OpenWire – The default in ActiveMQ; a binary protocol • STOMP – Simple Text Oriented Messaging Protocol; a text based protocol • XMPP – The Jabber XML protocol • REST – HTTP POST and GET • AMQP – Not yet fully supported 45
  46. 46. OpenWire • A binary wire level protocol for marshaling objects to/from byte arrays • Designed for performance – Sacrificed some ease of implementation • Clients for C++, Java and .NET – The default is Java 46
  47. 47. STOMP • A text-based wire level protocol • Designed to be very easy to understand – Encourages implementation in additional languages • Can be easily demonstrated via telnet • Clients for C, Javascript, Perl, PHP, Python, Ruby and more 47
  48. 48. What's the best approach if a producer or consumer may be down for some time? 48
  49. 49. Use the failover Protocol • Utilize the failover options – initialReconnectDelay - wait before reconnect – maxReconnectDelay - max time between reconnects – useExponentialBackOff - exponentially grow time between reconnects – backOffMultiplier - multiplier for backoff – maxReconnectAttempts - max number of reconnects – randomize - randomly choose from list of brokers – backup - start and hold a hot standby connection 49
  50. 50. Use Producer Flow Control • Prevents slow consumers from being flooded ... <destinationPolicy> <policyMap> <policyEntries> <policyEntry topic=quot;FOO.>quot; producerFlowControl=quot;falsequot; memoryLimit=quot;128mbquot;> </policyEntries> </policyMap> </destinationPolicy> ... 50
  51. 51. Use Message Persistence • Quality of service to preserve messages in the event of broker failure 51
  52. 52. Consumer Options • Message prefetch • Consumer dispatch async • Exclusive consumer • Consumer priority • Message groups • Redeliery policies • Retroactive consumer • Selectors • Some slow consumer strategies 52
  53. 53. Message Prefetch • Used for slow consumer situations – Consumer is flooded by messages from the broker • FIFO buffer on the consumer side 53
  54. 54. Async Dispatch • Asynchronous message delivery to consumers – Default is true • Useful for slow consumers – Incurs a bit of overhead 54
  55. 55. Exclusive Consumers • Anytime more than one consumer is consuming from a queue, message order is lost • Allows a single consumer to consume all messages on a queue to maintain message ordering 55
  56. 56. Consumer Priority • Just like it sounds – Gives a consumer priority for message delivery – Allows for the weighting of consumers to optimize network traversal for message delivery 56
  57. 57. Message Groups • Uses the JMSXGroupID property to define which message group a message belongs – Guarantees ordered processing of related messages across a single destination – Load balancing of message processing across multiple consumers – HA/failover if consumer goes down 57
  58. 58. Redelivery Policy • Messages are redelivered to a client when: – A transacted session is rolled back – A transacted session is closed before commit – A session is using CLIENT_ACKNOWLEDGE and Session.recover() is explicitly called 58
  59. 59. Retroactive Consumer • Message replay at start of a subscription – At the start of every subscription, send any old messages that the consumer may have missed – Configurable via policies 59
  60. 60. Wildcards on Destinations • Price.> • Price.Stock.> • Price.Stock.NASDAQ.* • Price.Stock.*.IBM ... <destinationPolicy> <policyMap> <policyEntries> <policyEntry topic=quot;Price.Stock.>quot; memoryLimit=quot;128mbquot;> </policyEntries> </policyMap> </destinationPolicy> ... 60
  61. 61. Selectors • Used to attach a filter to a subscription • Defined using a subset SQL 92 syntax • JMS selectors – Filters only message properties • JMSType = ‘stock’ and trader = ‘bob’ and price < ‘105’ • XPath selectors – Filters message bodies that contain XML • ‘/message/cheese/text() = 'swiss'’ 61
  62. 62. Slow Consumers Strategies 62
  63. 63. Slow Consumer Strategies • Various configurable strategies for handling slow consumers • Slow consumer situations are very common • Caused by: – Slow network connections – Unreliable network connections – Busy network situations – Busy JVM situations – Half disconnects with sockets 63
  64. 64. Use Message Limit Strategies • PendingMessageLimitStrategy – Calculates the max number of pending messages to be held in memory for a consumer above its prefetch size • ConstantPendingMessageLimitStrategy – A constant limit for all consumers • PrefetchRatePendingMessageLimitStrategy – Calculates the max number of pending messages using a multiplier of the consumers prefetch size 64
  65. 65. Use Prefetch and an Eviction Policy • Use the prefetch policy – The prefetch policy has a property named maximumPendingMessageLimit that can be used on a per connection or per consumer basis • Use a message eviction policy – oldestMessageEvictionStrategy - Evict the oldest messages first – oldestMessageWithLowestPriorityEvictionStrategy - Evict the oldest messages with the lowest priority first 65
  66. 66. Use Destination Policies • Configured on the destination policies in the ActiveMQ XML configuration file • Combined with wildcards, this is very powerful 66
  67. 67. Additional Tips • Consider configuring message cursors • The status of slow consumers can be monitored via JMX properties – discarded - The count of how many messages have been discarded during the lifetime of the subscription due to it being a slow consumer – matched - The current number of messages matched and to be dispatched to the subscription as soon as some capacity is available in the prefetch buffer. So a non-zero value implies that the prefetch buffer is full for this subscription 67
  68. 68. How to monitor individual messages, queue depths, or other broker statistics? • JMX • ActiveMQ web console • Additional consumers – Camel routes • SpringSource AMS – Based on Hyperic • IONA FuseHQ 68 – Based on Hyperic
  69. 69. What is Apache Camel? http://activemq.apache.org/camel/
  70. 70. Camel Components
  71. 71. Fluent Java API RouteBuilder MyRoute = new RouteBuilder() { public void configure() { from(quot;activemq:TEST.QUEUEquot;). to(quot;file:///Users/bsnyder/camelinbox/text.txtquot;). to(quot;log:MyLogquot;); } };
  72. 72. XML Config <camelContext id=quot;camelquot; xmlns=quot;http://activemq.apache.org/camel/schema/springquot;> <package>com.mycompany</package> <route> <from uri=quot;activemq:example.Aquot; /> <to uri=quot;file:///Users/bsnyder/camelinbox/text.txtquot; /> <to uri=”log:MyLog?showProperties=true” /> </route> </camelContext>
  73. 73. Information Overload Yet?
  74. 74. Thank You For Attending! Questions?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×