Global Scale ESB with Mule      Andrew D Kennedy      grkvlt@apache.org         March 2011
Who Am I•   Enterprise Java Developer•   Apache Qpid Committer•   Previously Security Engineer•   Over 15 Years Java Exper...
This Presentation•   What is an ESB•   What is Mule•   Mule Applications•   Mule Scalability
What is an ESB• Enterprise Service Bus• SOA  – Service Oriented Architecture• SEDA  – Staged Event-Driven Architecture• EI...
Service Oriented Architecture• Services  – Shared Business Functions• Decoupling  – Local and Remote Services• Distributed...
EIP• Similar to Design Patterns    – Gang of Four Book•   Shared Terminology•   Names for Common Concepts•   Allows Discus...
(Some) Common Patterns•   Aggregator (268)•   Channel Adapter (127)•   Content Based Router (230)•   Dead Letter Channel (...
What is Mule• Open Source Project• MuleSoft  –   Mule Forge  –   Mule Enterprise  –   Mule MQ  –   Tcat Server• 2.x - Lega...
Transports• Carry Messages between Services• Connector  – Configuration  – Threading  – Retry• Endpoint  – Connects Servic...
Endpoint Examples<file:inbound-endpoint path="/esb/in"   comparator="org.mule.transport.file.comparator.Older   FirstCompa...
VM• Communication within VM• Synchronous or Asynchronous• Transactional<vm:connector name="async"  queueEvents="true” /><v...
JMS• Queues• Topics• Request-Response<jms:connector name=”jms" specification="1.1"  connectionFactory-ref=”qpid"  username...
Web Services• HTTP and HTTPS  – Custom Data Formats• CXF  – JAX-WS  – SOAP  – WSDL• REST
CXF JAX-WS<inbound-endpoint   address="http://localhost:63081/hello" /><cxf:jaxws-service   serviceClass="org.example.Serv...
File System• Poll Directory for Input<file:connector name="input" fileAge="500"   autoDelete="true" pollingFrequency="100"...
Other Transports•   AJAX              •   TCP•   jBPM              •   UDP•   JDBC              •   FTP•   Quartz         ...
Transformers•   Change message content•   XML to POJO via JAXB•   Applied to endpoints•   Efficient mechanism for processi...
Transformer Types•   Encryption and Decryption•   ByteArray to Object•   Expression Evaluator•   XML and JSON•   Compressi...
Filtering• Apply to endpoints• Payload Type• Expression    – XPath, OGNL, JXPath•   RegEx•   Wildcard•   Message Property•...
Filtering Examples<and-filter>    <message-property-filter        pattern="JMSCorrelationID=1234567890"/>    <message-prop...
Routers•   Control message flow at endpoints•   Used to link services•   Both inbound and outbound•   Specify Filtering an...
Selective Consumer<inbound>    <selective-consumer-router>        <mulexml:jxpath-filter          expression="msg/header/r...
Idempotent Filter<inbound>    <idempotent-receiver-router      idExpression="#[message:id]-#[header:foo]">        <simple-...
Filtering Router<outbound>     <forwarding-catch-all-strategy>            <jms:outbound-endpoint queue="error.queue"/>    ...
Services• Combine Endpoints, Routers and  Components• Mule 2.0 idiom• Replaced with Flows and Patterns  – Simpler to use a...
Flows• Mule 3 idiom• Inbound endpoint• Message processors  – Chained execution  – Components• Outbound endpoint for one-wa...
Components• Carry out actions• Business logic in a flow• Simple components  – Logging  – Passthrough  – Testing• Spring or...
Message Processors•   Aggregator and Splitter•   Message Filters•   Recipient List•   Resequencer•   Round Robin•   Wire Tap
Example Flow<flow name=”OrderFlow">      <file:inbound-endpoint path="/incoming">              <file:filename-filter name=...
JMS Flow<flow name=”JmsFlow">      <jms:inbound-endpoint queue="in">            <jms:transaction action="ALWAYS_BEGIN" /> ...
Complex Flow<flow name=”TwitterFlow”>      <poll frequency="5000">             <twitter:search query=”example" />      </p...
Patterns• Specific integration Features• Simple Service  – Exposes Components as Web Services• Web Service Proxy• Bridge  ...
Questions?
Mule ApplicationMule 2 Application Design
Mule Scalability• Global Enterprises• Performance• Multiple Data Locations, Sources,  Destinations and Owners• Services  –...
Problems• Performance  –   Computation  –   Latency  –   Throughput  –   Infrastructure• Integration  – Adding Functionali...
Transactions• ACID  – Atomicity, Consistency, Isolation, Durability• Easy on one system• Hard with distributed systems  – ...
Computation• Parallel Processing• Asynchronous Tasks• Orchestration  – Workflow• Pooling and Threading• More and faster is...
Bottlenecks• Bandwidth  – Networking  – Latency and Throughput• External service providers• Data sources  – Database• Mess...
Messaging Backbone• Sets scalability limits• Message flow through system• Infrastructure choices  –   JMS  –   Web based  ...
Clustering• Various external components  – Database  – Messaging  – Web Services• Business logic components  – Terracotta ...
Load Balancing• Web Services• Software• Hardware  – XML Transformation  – SSL• Round Robin
Questions?Thanks for ListeningAndrew D Kennedygrkvlt@apache.org
Upcoming SlideShare
Loading in...5
×

Global Scale ESB with Mule

3,138

Published on

Presentation on Mule 2 and 3 and associated scalability issues and bottlenecks

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,138
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
106
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Global Scale ESB with Mule

  1. 1. Global Scale ESB with Mule Andrew D Kennedy grkvlt@apache.org March 2011
  2. 2. Who Am I• Enterprise Java Developer• Apache Qpid Committer• Previously Security Engineer• Over 15 Years Java Experience• Worked with various Investment Banks• Interested in Data Mining and Visualisation
  3. 3. This Presentation• What is an ESB• What is Mule• Mule Applications• Mule Scalability
  4. 4. What is an ESB• Enterprise Service Bus• SOA – Service Oriented Architecture• SEDA – Staged Event-Driven Architecture• EIP – Enterprise Integration Patterns
  5. 5. Service Oriented Architecture• Services – Shared Business Functions• Decoupling – Local and Remote Services• Distributed Applications• Directory and Discovery
  6. 6. EIP• Similar to Design Patterns – Gang of Four Book• Shared Terminology• Names for Common Concepts• Allows Discussion• Simplifies Design
  7. 7. (Some) Common Patterns• Aggregator (268)• Channel Adapter (127)• Content Based Router (230)• Dead Letter Channel (119)• Selective Consumer (515)• Message Translator (85)• Guaranteed Delivery (122)
  8. 8. What is Mule• Open Source Project• MuleSoft – Mule Forge – Mule Enterprise – Mule MQ – Tcat Server• 2.x - Legacy Applications• 3.1.1 - Current Community Release• 3.2.0 - Latest Developer Build
  9. 9. Transports• Carry Messages between Services• Connector – Configuration – Threading – Retry• Endpoint – Connects Services – Transform Messages – Transaction Boundary
  10. 10. Endpoint Examples<file:inbound-endpoint path="/esb/in" comparator="org.mule.transport.file.comparator.Older FirstComparator" reverseOrder="true” /><jetty:endpoint name="server" host="localhost" port="60203" path="services/Lookup" /><imaps:endpoint name=”mail" host="localhost" password=”hunter2" port="123" user=”adk"/><quartz:endpoint name=”batch" repeatCount="10" repeatInterval=”60000" jobName="job"/><jms:inbound-endpoint queue="test.queue"/>
  11. 11. VM• Communication within VM• Synchronous or Asynchronous• Transactional<vm:connector name="async" queueEvents="true” /><vm:inbound-endpoint path="in" connector-ref="async"/>
  12. 12. JMS• Queues• Topics• Request-Response<jms:connector name=”jms" specification="1.1" connectionFactory-ref=”qpid" username=”adk" password=”hunter2” /><jms:outbound-endpoint queue=”audit” /><jms:inbound-endpoint topic=”uk.*.gbp” />
  13. 13. Web Services• HTTP and HTTPS – Custom Data Formats• CXF – JAX-WS – SOAP – WSDL• REST
  14. 14. CXF JAX-WS<inbound-endpoint address="http://localhost:63081/hello" /><cxf:jaxws-service serviceClass="org.example.ServiceImpl” /><cxf:jaxws-client clientClass=”com.example.Client” wsdlPort="SoapPort” wsdlLocation="classpath:/save.wsdl” operation=”output” /><outbound-endpoint address="http://www.example.com/services/save” />
  15. 15. File System• Poll Directory for Input<file:connector name="input" fileAge="500" autoDelete="true" pollingFrequency="100" moveToDirectory="/backup" moveToPattern="#[header:originalFilename].backup” />• Write Output File<file:connector name="output" outputAppend="true" outputPattern="#[function:datestamp]- #[header:originalFilename]" />
  16. 16. Other Transports• AJAX • TCP• jBPM • UDP• JDBC • FTP• Quartz • POP3• STDIO • IMAP• RMI • SMTP• Servlet • XMPP
  17. 17. Transformers• Change message content• XML to POJO via JAXB• Applied to endpoints• Efficient mechanism for processing• Similar to AOP
  18. 18. Transformer Types• Encryption and Decryption• ByteArray to Object• Expression Evaluator• XML and JSON• Compression• Encoding and Decoding
  19. 19. Filtering• Apply to endpoints• Payload Type• Expression – XPath, OGNL, JXPath• RegEx• Wildcard• Message Property• Logical Operators – And, Or, Not
  20. 20. Filtering Examples<and-filter> <message-property-filter pattern="JMSCorrelationID=1234567890"/> <message-property-filter pattern="JMSReplyTo=null"/></and-filter><or-filter> <payload-type-filter expectedType="java.lang.String"/> <payload-type-filter expectedType="java.lang.StringBuffer"/></or-filter><not-filter> <wildcard-filter pattern=”com.example.*"/></not-filter>
  21. 21. Routers• Control message flow at endpoints• Used to link services• Both inbound and outbound• Specify Filtering and Transformations
  22. 22. Selective Consumer<inbound> <selective-consumer-router> <mulexml:jxpath-filter expression="msg/header/resultcode = success"/> </selective-consumer-router> <forwarding-catch-all-strategy> <jms:endpoint topic="error.topic"/> </forwarding-catch-all-strategy></inbound>
  23. 23. Idempotent Filter<inbound> <idempotent-receiver-router idExpression="#[message:id]-#[header:foo]"> <simple-text-file-store directory="./idempotent” /> </idempotent-receiver-router></inbound>
  24. 24. Filtering Router<outbound> <forwarding-catch-all-strategy> <jms:outbound-endpoint queue="error.queue"/> </forwarding-catch-all-strategy> <filtering-router> <smtp:outbound-endpoint to=”adk@example.com"/> <payload-type-filter expectedType="java.lang.Exception"/> </filtering-router> <filtering-router> <jms:outbound-endpoint queue="string.queue"/> <and-filter> <payload-type-filter expectedType="java.lang.String"/> <regex-filter pattern="the quick brown (.*)"/> </and-filter> </filtering-router></outbound>
  25. 25. Services• Combine Endpoints, Routers and Components• Mule 2.0 idiom• Replaced with Flows and Patterns – Simpler to use and configure – Some features still not available
  26. 26. Flows• Mule 3 idiom• Inbound endpoint• Message processors – Chained execution – Components• Outbound endpoint for one-way• Response for request-response
  27. 27. Components• Carry out actions• Business logic in a flow• Simple components – Logging – Passthrough – Testing• Spring or POJO• Web services
  28. 28. Message Processors• Aggregator and Splitter• Message Filters• Recipient List• Resequencer• Round Robin• Wire Tap
  29. 29. Example Flow<flow name=”OrderFlow"> <file:inbound-endpoint path="/incoming"> <file:filename-filter name=”order-*.xml"/> </file:inbound-endpoint> <xml:xslt-transformer xsl-file=”order-transform.xsl"/> <splitter expression="xpath://order"/> <!-- The following message processors will be invoked for each order in the xml file --> <expression-filter expression="xpath://order[@type=book]"/> <component class=”com.example.BookOrderProcessor"/> <smtp:outbound-endpoint subject="Order Confirmation" address="#[variable:email]"/> <jdbc:outbound-endpoint ref=“saveOrder"/ > <default-exception-strategy> <jms:outbound-endpoint queue="failedOrders"/> </default-exception-strategy></flow>
  30. 30. JMS Flow<flow name=”JmsFlow"> <jms:inbound-endpoint queue="in"> <jms:transaction action="ALWAYS_BEGIN" /> </jms:inbound-endpoint> <component class="com.example.ProcessMessage" /> <jms:outbound-endpoint queue="out"> <jms:transaction action="ALWAYS_JOIN" /> </jms:outbound-endpoint> <default-exception-strategy> <commit-transaction exception-pattern="com.example.ExampleException” /> <jms:outbound-endpoint queue="dead.letter"> <jms:transaction action="JOIN_IF_POSSIBLE" /> </jms:outbound-endpoint> </default-exception-strategy></flow>
  31. 31. Complex Flow<flow name=”TwitterFlow”> <poll frequency="5000"> <twitter:search query=”example" /> </poll> <splitter evaluator="json" expression="results" /> <idempotent-message-filter idExpression="#[json:id]" /> <enricher target="#[variable:userDescription]" source="#[json:description]"> <twitter:user userId="#[json:from_user]" /> </enricher> <salesforce:create type="Opportunity"> <salesforce:sObject> <salesforce:field key="Name" value="#[json:from_user]" /> <salesforce:field key="Description" value="#[variable:userDescription]" /> </salesforce:sObject> </salesforce:create></flow>
  32. 32. Patterns• Specific integration Features• Simple Service – Exposes Components as Web Services• Web Service Proxy• Bridge – Message Adapter and Transformer• Validator – Validates inbound messages synchronously – Processes asynchronously
  33. 33. Questions?
  34. 34. Mule ApplicationMule 2 Application Design
  35. 35. Mule Scalability• Global Enterprises• Performance• Multiple Data Locations, Sources, Destinations and Owners• Services – Software as a Service – Infrastructure as a Service – Platform as a Service
  36. 36. Problems• Performance – Computation – Latency – Throughput – Infrastructure• Integration – Adding Functionality – Collecting Data• Security and Identity• Transactions
  37. 37. Transactions• ACID – Atomicity, Consistency, Isolation, Durability• Easy on one system• Hard with distributed systems – Pick your boundaries• Robust failure handling
  38. 38. Computation• Parallel Processing• Asynchronous Tasks• Orchestration – Workflow• Pooling and Threading• More and faster is better
  39. 39. Bottlenecks• Bandwidth – Networking – Latency and Throughput• External service providers• Data sources – Database• Messaging
  40. 40. Messaging Backbone• Sets scalability limits• Message flow through system• Infrastructure choices – JMS – Web based – AMQP – XMPP – TIBCO• Hardware solutions available
  41. 41. Clustering• Various external components – Database – Messaging – Web Services• Business logic components – Terracotta – GigaSpaces XAP
  42. 42. Load Balancing• Web Services• Software• Hardware – XML Transformation – SSL• Round Robin
  43. 43. Questions?Thanks for ListeningAndrew D Kennedygrkvlt@apache.org
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×