中间件基础服务团队 鼬神
Von Gosling@web world
MOM those things
Message-Oriented Middleware
Agenda
•Part 1
 MOM intro.
 MOM spec.
•Part 2
 MOM core tech.
 MOM advanced features
. Part 3
 MOM geomancy
 References
Part 1
MOM intro.
I. why
II. what
III. when
MOM spec.
I. JMS
II. AMQP
III. STOMP
IV. XMPP
V. MQTT
MOM intro. - why
1970s single protocol connecting mainframe
MOM intro. - why
1980s many protocol connecting mainframe
MOM intro. - why
• not easy since data format , hardware etc.
• so adapter coms out, but still face many problems,
maintainable?
• so many technologies comes out , such as
RPC(COM/COM+,CORBA,EJB) – sync. call
Message Queue – Async. event notification
Also ,many Enterprise messaging killer coming, such
as WebSphere MQ, SonicMQ , TIBCO Rendezvous,
SUN JCAPS…
MOM intro. - what
Message-oriented middleware (MOM) is software or
hardware infrastructure supporting sending and
receiving messages between distributed systems. MOM
allows application modules to be distributed over
heterogeneous platforms and reduces the complexity of
developing applications that span multiple operating
systems and network protocols…
-- From Wikipedia
MOM intro. - what
Messaging mediator
MOM intro. - when
Heterogeneous application integration
Decoupled between applications(No RPC)
Event-driven backbone(CEP)
MOM spec. - JMS
•JMS 1.1 released in in 2002 (JMS 1.0.2 spec. since
1998)
•JMS 2.0 launched in 2011 as JSR 343
•JMS 2.0 released in 2013 (with Java EE 7)
MOM spec - JMS
JMS client (MessageProducer, MessageConsumer ,
JMSConsumer , JMSProducer)
Non-JMS client (Instead of the JMS API , using JMS
provider’s native client API)
JMS producer
JMS consumer
JMS provider
JMS message
JMS domains
Administered objects(Connection factory,Destination)
MOM spec - JMS
MessageConsumer glance
MOM spec. - JMS
Non-JMS client - HornetQ
MOM spec - JMS
JMS message
MOM spec. - JMS
MOM spec. - JMS
How to use it ? Selector(SQL 92 subset ,exactly any retrieval tech.
you can image)
MOM spec - JMS
Payload
MOM spec. - JMS
JMS domains PK.
MOM spec. - JMS
JMS 1.1 Core API
MOM spec. - JMS
MOM spec. - JMS
Snippet with JMS 1.1 spec.
MOM spec. - JMS
Snippet with JMS 2.0 spec.
MOM spec - JMS
Concurrency
ConnectionFactory,Connection,Destination
Session,MessageProducer,MessageConsumer
MOM spec. - AMQP
2012.10 1.0 release
MOM spec. - AMQP
MOM spec. - AMQP
•Types
•Transport
•Messaging
•Transactions
•Security
MOM spec. - AMQP
Class Diagram of Communication Endpoints
MOM spec. - AMQP
Version Negotiation examples
MOM spec - AMQP
Protocol frames
MOM spec - AMQP
AMQP Frame Layout
MOM spec - AMQP
AMQP connection state Diagram
MOM spec - AMQP
AMQP message
MOM spec - AMQP
How to parser?
MOM spec - AMQP
How to use?
MOM spec. - STOMP
STOMP 1.2 Released on 2012 10
STOMP 1.1
STOMP 1.0
In a way, It’s seems combing memcached protocol with
REST style operation(such as
send,subscribe,unsubscribe,begin,commit,abort,ack,n
ack,disconnect).
Part 2
MOM core tech.
I. Transport
II. Persistent
III. Transaction
IV. Integration
MOM advanced features
I. HA
II. Performance
III. Administering and monitoring
IV. Plugins
MOM core tech. - Transport
AMQ Transport configuration snippets
MOM core tech. - Transport
AMQ log
MOM core tech. - Transport
Hornetq Transport configuration snippets
MOM core tech. - Persistent
RDB store
Memory store
File store
Sequential disk access can in some cases be faster
than random memory access! – from ACM Queue
article
MOM core tech. - Persistent
MOM core tech. - Persistent
MOM core tech. - Persistent
File store structure
MOM core tech. - Persistent
AMQ store structure
MOM core tech. - Transaction
MOM core tech. - Transaction
MOM core tech. - Transaction
2PC , 3PC or Paxos Why not choice?
Complex, Round Trip…
MOM Core Tech. - Transaction
Client DB
Broker DB
①
send
halfmessage
②Op
③
Commit/Rollback
①store half message
③, ⑤
commit: update message status
rollback:delete message
④
periodiccheck
uncommittedmessage
⑤Commit/Rollback
Local Tx
Local Tx
MOM core tech. - Integration
Jetty,Tomcat
Hadoop
Storm
ESB
ServiceMix
MOM advanced features- HA
Static Discovery
Dynamic Discovery – ZK/UDP/JGROUP
Master-Master
Master-Slave
MOM advanced features- HA
Shared store
MOM advanced features- HA
Replication
MOM advanced features- HA
Which one?
MOM advanced features- Performance
Broker performance
Client performance
TCP kernel
Protocol
MOM advanced features- Performance
Broker performance
 Static scales better than dynamic
 Asynchronous network connection
establishment
 Concurrent store and dispatch
 Store optimization
MOM advanced features- Performance
Client performance
 setDisableMessageID
 setDisableMessageTimeStamp
 Avoid ObjectMessage
 Avoid AUTO_ACKNOWLEDGE
 Avoid durable messages
 Send messages non blocking
 ProducerWindowSize(sendFailIfNoSpaceAft
erTimeout)
 Prefetch limit
MOM advanced features- Performance
TCP kernel
MOM advanced features- Performance
Protocol
OpenWire Text-based
No silver bullet...
MOM advanced features- Performance
MOM advanced features- Performance
Performance differs greatly depending on
many different factors
 the network topology
 transport protocols used
 quality of service
 hardware, network, JVM and operating system
 number of producers, number of consumers
 distribution of messages across destinations along
with message size
So, No one is absolutely winner!
Benchmark testing(SPECjms2007) is a Galileo
thing !
MOM advanced features- Administering and monitoring
Metrics
MDC
JMX-HTTP
MOM advanced features- Plugins
Router
Visualization
Statistics
MOM advanced features- Plugins
Router
MOM advanced features- Plugins
Statistics
MOM advanced features- Plugins
AMQ plugins architecture
MOM advanced features- Plugins
AMQ - How plugins work
Part3
MOM geomancy
I. ZMQ
II. Local features continue
III. Cloud Messaging
References
MOM geomancy - ZMQ
C++ Networking library
ZMQ Pattern:
Request-reply, which connects a set of clients to a set of services.
This is a remote procedure call and task distribution pattern.
Publish-subscribe, which connects a set of publishers to a set of
subscribers. This is a data distribution pattern.
Pipeline, which connects nodes in a fan-out/fan-in pattern that can
have multiple steps and loops. This is a parallel task distribution
and collection pattern.
MOM geomancy - ZMQ
MOM geomancy - ZMQ
MOM geomancy - ZMQ
MOM geomancy - ZMQ
ØMQ’s Built-in Proxy Function code snippet
MOM geomancy - local features continue
Close to latest stable spec. ,such as JMS 2.0,AMQP 1.0,STOMP
1.2 etc.;
Promote communication kernel ,such as AIO, Actor pattern etc.;
Full stack PUSH model , support web, android, ios etc.;
More Cloud features…
MQ
(Broker)
Broker WebSocket Connection
Service
WebSocket
Engine
JMSWebSocket
HTTPServer
Java* Application
JMS Client Runtime
WebSocket Connection Handler
Java API for WebSocket
Tyrus Implementation
MOM geomancy - local features continue
Kestrel kernel
MOM geomancy - local features continue
But, if you biased towards jms,How to give your
views about jms ?
•Join and contribute to users@jms-spec.java.net
•Submit ideas to JMS spec issue tracker (or comment
on existing ideas)
•Contact nigel.deakin@oracle.com
•See jms-spec.java.net for details
MOM geomancy - Cloud Messaging
MOM geomancy - Cloud Messaging
MOM geomancy - References
MOM geomancy - References
•http://en.wikipedia.org/wiki/Message-oriented_middleware
•http://augustl.com/blog/2013/zeromq_instead_of_http/
•http://kafka.apache.org/documentation.html#design
•http://www.oschina.net/translate/client-side-messaging-essentials
•http://www.iron.io/mq
•http://java.dzone.com/articles/guaranteed-messaging-topics
•http://download.progress.com/5331/open/adobe/prc/psc/perf_tuning_activemq/index
.htm
•http://www.cs.cornell.edu/courses/cs614/2007fa/Slides/FLP_and_Paxos.pdf
MOM geomancy - References
•Enterprise Integration with WSO2 ESB
•http://lingo.codehaus.org/
•http://activespace.codehaus.org/
•https://engineering.groupon.com/2013/page/2/
•http://blog.x-aeon.com/2013/04/10/a-quick-message-queue-benchmark-activemq-
rabbitmq-hornetq-qpid-apollo/
•http://www.slashroot.in/linux-network-tcp-performance-tuning-sysctl
•http://activemq.apache.org/performance.html
•http://activemq.apache.org/apollo/documentation/openwire-manual.html
•http://snarfed.org/transactions_across_datacenters_io.html
MOM geomancy - References
•JMS Performance Comparison
•amqp-core-complete-v1.0-os.pdf
•HornetQ User Manual.pdf
•ActiveMQ In Action(Manning-2011).pdf
•Building a High Availability and Disaster Recovery Solution using AlwaysOn
Availability Groups.docx
•RocketMQ_design.pdf
•ZeroMQ: Messaging for Many Applications.pdf
•ZeroMQ.pdf
Q & A
你问我答
共创,共建
Questions?
Thanks
!

Mom those things v1

  • 1.
    中间件基础服务团队 鼬神 Von Gosling@webworld MOM those things Message-Oriented Middleware
  • 2.
    Agenda •Part 1  MOMintro.  MOM spec. •Part 2  MOM core tech.  MOM advanced features . Part 3  MOM geomancy  References
  • 3.
    Part 1 MOM intro. I.why II. what III. when MOM spec. I. JMS II. AMQP III. STOMP IV. XMPP V. MQTT
  • 4.
    MOM intro. -why 1970s single protocol connecting mainframe
  • 5.
    MOM intro. -why 1980s many protocol connecting mainframe
  • 6.
    MOM intro. -why • not easy since data format , hardware etc. • so adapter coms out, but still face many problems, maintainable? • so many technologies comes out , such as RPC(COM/COM+,CORBA,EJB) – sync. call Message Queue – Async. event notification Also ,many Enterprise messaging killer coming, such as WebSphere MQ, SonicMQ , TIBCO Rendezvous, SUN JCAPS…
  • 7.
    MOM intro. -what Message-oriented middleware (MOM) is software or hardware infrastructure supporting sending and receiving messages between distributed systems. MOM allows application modules to be distributed over heterogeneous platforms and reduces the complexity of developing applications that span multiple operating systems and network protocols… -- From Wikipedia
  • 8.
    MOM intro. -what Messaging mediator
  • 9.
    MOM intro. -when Heterogeneous application integration Decoupled between applications(No RPC) Event-driven backbone(CEP)
  • 10.
    MOM spec. -JMS •JMS 1.1 released in in 2002 (JMS 1.0.2 spec. since 1998) •JMS 2.0 launched in 2011 as JSR 343 •JMS 2.0 released in 2013 (with Java EE 7)
  • 11.
    MOM spec -JMS JMS client (MessageProducer, MessageConsumer , JMSConsumer , JMSProducer) Non-JMS client (Instead of the JMS API , using JMS provider’s native client API) JMS producer JMS consumer JMS provider JMS message JMS domains Administered objects(Connection factory,Destination)
  • 12.
    MOM spec -JMS MessageConsumer glance
  • 13.
    MOM spec. -JMS Non-JMS client - HornetQ
  • 14.
    MOM spec -JMS JMS message
  • 15.
  • 16.
    MOM spec. -JMS How to use it ? Selector(SQL 92 subset ,exactly any retrieval tech. you can image)
  • 17.
    MOM spec -JMS Payload
  • 18.
    MOM spec. -JMS JMS domains PK.
  • 19.
    MOM spec. -JMS JMS 1.1 Core API
  • 20.
  • 21.
    MOM spec. -JMS Snippet with JMS 1.1 spec.
  • 22.
    MOM spec. -JMS Snippet with JMS 2.0 spec.
  • 23.
    MOM spec -JMS Concurrency ConnectionFactory,Connection,Destination Session,MessageProducer,MessageConsumer
  • 24.
    MOM spec. -AMQP 2012.10 1.0 release
  • 25.
  • 26.
    MOM spec. -AMQP •Types •Transport •Messaging •Transactions •Security
  • 27.
    MOM spec. -AMQP Class Diagram of Communication Endpoints
  • 28.
    MOM spec. -AMQP Version Negotiation examples
  • 29.
    MOM spec -AMQP Protocol frames
  • 30.
    MOM spec -AMQP AMQP Frame Layout
  • 31.
    MOM spec -AMQP AMQP connection state Diagram
  • 32.
    MOM spec -AMQP AMQP message
  • 33.
    MOM spec -AMQP How to parser?
  • 34.
    MOM spec -AMQP How to use?
  • 35.
    MOM spec. -STOMP STOMP 1.2 Released on 2012 10 STOMP 1.1 STOMP 1.0 In a way, It’s seems combing memcached protocol with REST style operation(such as send,subscribe,unsubscribe,begin,commit,abort,ack,n ack,disconnect).
  • 36.
    Part 2 MOM coretech. I. Transport II. Persistent III. Transaction IV. Integration MOM advanced features I. HA II. Performance III. Administering and monitoring IV. Plugins
  • 37.
    MOM core tech.- Transport AMQ Transport configuration snippets
  • 38.
    MOM core tech.- Transport AMQ log
  • 39.
    MOM core tech.- Transport Hornetq Transport configuration snippets
  • 40.
    MOM core tech.- Persistent RDB store Memory store File store Sequential disk access can in some cases be faster than random memory access! – from ACM Queue article
  • 41.
    MOM core tech.- Persistent
  • 42.
    MOM core tech.- Persistent
  • 43.
    MOM core tech.- Persistent File store structure
  • 44.
    MOM core tech.- Persistent AMQ store structure
  • 45.
    MOM core tech.- Transaction
  • 46.
    MOM core tech.- Transaction
  • 47.
    MOM core tech.- Transaction 2PC , 3PC or Paxos Why not choice? Complex, Round Trip…
  • 48.
    MOM Core Tech.- Transaction Client DB Broker DB ① send halfmessage ②Op ③ Commit/Rollback ①store half message ③, ⑤ commit: update message status rollback:delete message ④ periodiccheck uncommittedmessage ⑤Commit/Rollback Local Tx Local Tx
  • 49.
    MOM core tech.- Integration Jetty,Tomcat Hadoop Storm ESB ServiceMix
  • 50.
    MOM advanced features-HA Static Discovery Dynamic Discovery – ZK/UDP/JGROUP Master-Master Master-Slave
  • 51.
    MOM advanced features-HA Shared store
  • 52.
    MOM advanced features-HA Replication
  • 53.
  • 54.
    MOM advanced features-Performance Broker performance Client performance TCP kernel Protocol
  • 55.
    MOM advanced features-Performance Broker performance  Static scales better than dynamic  Asynchronous network connection establishment  Concurrent store and dispatch  Store optimization
  • 56.
    MOM advanced features-Performance Client performance  setDisableMessageID  setDisableMessageTimeStamp  Avoid ObjectMessage  Avoid AUTO_ACKNOWLEDGE  Avoid durable messages  Send messages non blocking  ProducerWindowSize(sendFailIfNoSpaceAft erTimeout)  Prefetch limit
  • 57.
    MOM advanced features-Performance TCP kernel
  • 58.
    MOM advanced features-Performance Protocol OpenWire Text-based No silver bullet...
  • 59.
  • 60.
    MOM advanced features-Performance Performance differs greatly depending on many different factors  the network topology  transport protocols used  quality of service  hardware, network, JVM and operating system  number of producers, number of consumers  distribution of messages across destinations along with message size So, No one is absolutely winner! Benchmark testing(SPECjms2007) is a Galileo thing !
  • 61.
    MOM advanced features-Administering and monitoring Metrics MDC JMX-HTTP
  • 62.
    MOM advanced features-Plugins Router Visualization Statistics
  • 63.
    MOM advanced features-Plugins Router
  • 64.
    MOM advanced features-Plugins Statistics
  • 65.
    MOM advanced features-Plugins AMQ plugins architecture
  • 66.
    MOM advanced features-Plugins AMQ - How plugins work
  • 67.
    Part3 MOM geomancy I. ZMQ II.Local features continue III. Cloud Messaging References
  • 68.
    MOM geomancy -ZMQ C++ Networking library ZMQ Pattern: Request-reply, which connects a set of clients to a set of services. This is a remote procedure call and task distribution pattern. Publish-subscribe, which connects a set of publishers to a set of subscribers. This is a data distribution pattern. Pipeline, which connects nodes in a fan-out/fan-in pattern that can have multiple steps and loops. This is a parallel task distribution and collection pattern.
  • 69.
  • 70.
  • 71.
  • 72.
    MOM geomancy -ZMQ ØMQ’s Built-in Proxy Function code snippet
  • 73.
    MOM geomancy -local features continue Close to latest stable spec. ,such as JMS 2.0,AMQP 1.0,STOMP 1.2 etc.; Promote communication kernel ,such as AIO, Actor pattern etc.; Full stack PUSH model , support web, android, ios etc.; More Cloud features… MQ (Broker) Broker WebSocket Connection Service WebSocket Engine JMSWebSocket HTTPServer Java* Application JMS Client Runtime WebSocket Connection Handler Java API for WebSocket Tyrus Implementation
  • 74.
    MOM geomancy -local features continue Kestrel kernel
  • 75.
    MOM geomancy -local features continue But, if you biased towards jms,How to give your views about jms ? •Join and contribute to users@jms-spec.java.net •Submit ideas to JMS spec issue tracker (or comment on existing ideas) •Contact nigel.deakin@oracle.com •See jms-spec.java.net for details
  • 76.
    MOM geomancy -Cloud Messaging
  • 77.
    MOM geomancy -Cloud Messaging
  • 78.
    MOM geomancy -References
  • 79.
    MOM geomancy -References •http://en.wikipedia.org/wiki/Message-oriented_middleware •http://augustl.com/blog/2013/zeromq_instead_of_http/ •http://kafka.apache.org/documentation.html#design •http://www.oschina.net/translate/client-side-messaging-essentials •http://www.iron.io/mq •http://java.dzone.com/articles/guaranteed-messaging-topics •http://download.progress.com/5331/open/adobe/prc/psc/perf_tuning_activemq/index .htm •http://www.cs.cornell.edu/courses/cs614/2007fa/Slides/FLP_and_Paxos.pdf
  • 80.
    MOM geomancy -References •Enterprise Integration with WSO2 ESB •http://lingo.codehaus.org/ •http://activespace.codehaus.org/ •https://engineering.groupon.com/2013/page/2/ •http://blog.x-aeon.com/2013/04/10/a-quick-message-queue-benchmark-activemq- rabbitmq-hornetq-qpid-apollo/ •http://www.slashroot.in/linux-network-tcp-performance-tuning-sysctl •http://activemq.apache.org/performance.html •http://activemq.apache.org/apollo/documentation/openwire-manual.html •http://snarfed.org/transactions_across_datacenters_io.html
  • 81.
    MOM geomancy -References •JMS Performance Comparison •amqp-core-complete-v1.0-os.pdf •HornetQ User Manual.pdf •ActiveMQ In Action(Manning-2011).pdf •Building a High Availability and Disaster Recovery Solution using AlwaysOn Availability Groups.docx •RocketMQ_design.pdf •ZeroMQ: Messaging for Many Applications.pdf •ZeroMQ.pdf
  • 82.
  • 83.