Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Fast SOA with Apache Synapse


Published on

Presented at ApacheCon Europe 2008: this presentation introduces the ApacheCon Synapse ESB and the performance benefits of the design.

Published in: Technology
  • esper on slide 15
    Are you sure you want to  Yes  No
    Your message goes here

Fast SOA with Apache Synapse

  1. 1. High Speed SOA with Apache Synapse Paul Fremantle VP, Apache Synapse CTO, WSO2 [email_address]
  2. 2. Contents <ul><li>What is Apache Synapse </li></ul><ul><li>History of the project </li></ul><ul><li>Some examples </li></ul><ul><li>Delving into the model </li></ul><ul><li>Performance </li></ul><ul><li>Connectors, Adaptors </li></ul><ul><li>Getting started </li></ul><ul><li>Getting involved </li></ul>
  3. 3. What is Apache Synapse? <ul><li>Is it? </li></ul><ul><ul><li>A highly performant XML Gateway </li></ul></ul><ul><ul><li>An Enterprise Service Bus </li></ul></ul><ul><ul><li>An HTTP proxy server </li></ul></ul><ul><ul><li>A protocol switcher </li></ul></ul><ul><ul><li>A way of hosting message manipulation logic </li></ul></ul>
  4. 4. What is Apache Synapse? <ul><li>A lightweight Enterprise Services Bus (ESB) </li></ul><ul><ul><li>Available as a WAR file, NT Service, Linux Daemon </li></ul></ul><ul><ul><li>Runs as a process with its own Listeners, Tasks and Senders </li></ul></ul><ul><ul><li>Can be deployed standalone or part of a cluster or distributed network </li></ul></ul><ul><ul><li>High performance, asynchronous, streaming design </li></ul></ul><ul><ul><li>Can initiate work – scheduled tasks </li></ul></ul><ul><ul><li>Supports multiple transports including HTTP, JMS, TCP, SMTP and (S)FTP </li></ul></ul><ul><ul><li>Simple to extend </li></ul></ul>
  5. 5. Flows JMS SMTP inflow log xslt send outflow send xslt log HTTP 8080 HTTPS 8443 JMS SMTP HTTP 8080 HTTPS 8443
  6. 6. Synapse startup <ul><li>Using Bouncy castle JAR for Java 1.5 </li></ul><ul><li>Starting Synapse/Java ... </li></ul><ul><li>Using SYNAPSE_HOME: C:SYNAPS~1.1in.. </li></ul><ul><li>Using JAVA_HOME: c:jdk </li></ul><ul><li>Using SYNAPSE_XML: -Dsynapse.xml=&quot;C:SYNAPS~1.1in.. epositoryconfsynapse.xml&quot; </li></ul><ul><li>2007-11-12 12:16:58,250 [-] [main] INFO ServerManager Using the Axis2 Repository C:SYNAPS~1.1in.. epository </li></ul><ul><li>2007-11-12 12:17:01,921 [-] [main] INFO SynapseInitializationModule Initializing Synapse at : Mon Nov 12 12:17:01 GMT 2007 </li></ul><ul><li>2007-11-12 12:17:01,937 [] [main] INFO SynapseInitializationModule Loading mediator extensions... </li></ul><ul><li>2007-11-12 12:17:01,937 [] [main] INFO SynapseInitializationModule Initializing the Synapse configuration ... </li></ul><ul><li>2007-11-12 12:17:01,968 [] [main] INFO XMLConfigurationBuilder Generating the Synapse configuration model by parsing the XML configuration </li></ul><ul><li>(some deleted) </li></ul><ul><li>2007-11-12 12:17:04,359 [] [main] INFO HttpCoreNIOSender HTTP Sender starting </li></ul><ul><li>2007-11-12 12:17:04,968 [] [main] INFO HttpCoreNIOListener HTTPS Listener starting on port : 8443 </li></ul><ul><li>2007-11-12 12:17:04,968 [] [main] INFO ServerManager Starting transport https on port 8443 </li></ul><ul><li>2007-11-12 12:17:05,046 [] [main] INFO ServerManager Ready for processing </li></ul>
  7. 7. History of the project <ul><li>August 2005 Kicked off as an incubator proposal </li></ul><ul><li>December 2005 Milestone 1 released </li></ul><ul><li>July 2006 Milestone 2 released </li></ul><ul><li>December 2006 Synapse 0.90 released </li></ul><ul><li>January 2007 Graduates from Incubator </li></ul><ul><li>May 2007 Released 1.0 </li></ul><ul><li>December 2007 Becomes a Top Level Project </li></ul><ul><li>January 2008 Synapse 1.1.1 released </li></ul>
  8. 8. A really simple example <ul><li>Exposing an existing SOAP endpoint as XML/HTTP, XML/JMS </li></ul><ul><li><definitions> </li></ul><ul><li><proxy name=“example&quot; transports=“http jms&quot;> </li></ul><ul><li><target> </li></ul><ul><li><endpoint> </li></ul><ul><li> <wsdl uri=““ </li></ul><ul><li>service=“RemoteService&quot; </li></ul><ul><li>port=“RemotePort&quot;/> </li></ul><ul><li></endpoint> </li></ul><ul><li><outSequence> </li></ul><ul><li><send/> </li></ul><ul><li> </outSequence> </li></ul><ul><li></target> </li></ul><ul><li></proxy> </li></ul><ul><li></definitions> </li></ul>
  9. 9. Some other things you can do “out of the box” <ul><li>Content-based routing </li></ul><ul><ul><li>Direct messages based on properties or XPath expressions </li></ul></ul><ul><li>Convert from existing formats (CSV, Fixed records, etc) to and from XML </li></ul><ul><li>Transform XML </li></ul><ul><ul><li>Using XSLT or XQuery </li></ul></ul><ul><li>Validate using XML Schema </li></ul><ul><li>Update databases and enhance messages with extra data </li></ul><ul><ul><li>DBLookup and DBReport mediators – any JDBC database </li></ul></ul><ul><li>Switch between SOAP and non-SOAP, any transport </li></ul><ul><ul><li>HTTP, HTTPS, SMTP/POP3, XMPP, JMS, TCP, Filesystem, FTP, AMQP </li></ul></ul><ul><li>Add or handle WS-* support including </li></ul><ul><ul><li>WS-Security, WS-SecureConversation, WS-Trust, WS-ReliableMessaging </li></ul></ul><ul><li>Load balancing, failover, throttling </li></ul>
  10. 10. Registries and Dynamism Synapse “Registry” HTTP extensible read and cache notify <ul><li>“ Transactional updates” </li></ul><ul><li>existing flows continue to use old config </li></ul><ul><li>Also graceful restart </li></ul><ul><li>switch off transport cleanly </li></ul><ul><li>restart Synapse when all work is done </li></ul>
  11. 11. A more detailed example NY Variable Format records London Existing trade database D2243578XX910 D6233578YYYY9 <FEED> <CAK>2243578</CAK> <AccNo>3289019</AccNo> <CAT>AK</CAT> … </FEED> WS-Security (encryption and digital signature) MQ Validation
  12. 12. Overview of the flow <ul><li><sequence name=“new york”> </li></ul><ul><li>Flatpack with Variable record config </li></ul><ul><li>E4X script to reformat XML </li></ul><ul><li>Send to London </li></ul><ul><li></sequence> </li></ul><ul><li><sequence name=“london”> </li></ul><ul><li>DBreport </li></ul><ul><li>Filter </li></ul><ul><li>Send to MQ </li></ul><ul><li></sequence> </li></ul>
  13. 13. Results <ul><li>Replicated a 3 month project is 3 days </li></ul><ul><ul><li>Included enhancing the Flatpack mediator to support Variable Records </li></ul></ul><ul><ul><li>Added WS-Security over original scenario </li></ul></ul><ul><ul><li>1200 lines of config reduced to less than 300 </li></ul></ul><ul><ul><li>Much more maintainable – clean separation of concerns between departments </li></ul></ul>
  14. 14. Google Spreadsheet and CSV CSV Synapse Http Poller Job CSV to XML Atom
  15. 15. Esper <ul><li>Event Stream Processing project </li></ul><ul><ul><li> </li></ul></ul><ul><li>Allows you to define queries on sets of “events” </li></ul><ul><ul><li>SELECT AVG(price) FROM OrderEvent.WIN:TIME(30 sec) </li></ul></ul><ul><li>EsperMediator for Synapse makes it simple to add CEP to your SOA </li></ul>
  16. 16. Ganglia, Quartz, Esper Synapse Quartz Scheduler Ganglia gmond telnet 8649 <GANGLIA_XML> GMondPoller Job Esper Mediator SELECT AVG(CPU_USER) FROM GANGLIA urn:gmond Alert notification IM Atom or JMS urn:cpu
  17. 17. <ul><li><definitions xmlns=&quot;;> </li></ul><ul><li><task class=&quot;org.fremantle.gmond.GMonTask“ name=“Gmon”> </li></ul><ul><li> <trigger interval=&quot;5000&quot;/> </li></ul><ul><li> <property name=&quot;hostname&quot; value=&quot;localhost&quot;/> </li></ul><ul><li> <property name=&quot;port&quot; value=&quot;8649&quot;/> </li></ul><ul><li><property name=&quot;to&quot; value=&quot;urn:gmond&quot;/> </li></ul><ul><li></task> </li></ul><ul><li><in> </li></ul><ul><li> <filter source=&quot;get-property('To')&quot; regex=&quot;urn:gmond&quot;> </li></ul><ul><li><log/> </li></ul><ul><li><class name=&quot;org.fremantle.esper.EsperMediator&quot;> </li></ul><ul><li><property name=&quot;Configuration&quot; </li></ul><ul><li>value=“./repository/conf/esper.conf.xml&quot;/> </li></ul><ul><li><property name=&quot;statement&quot; </li></ul><ul><li>value=&quot;select avg(cpu_user) from &quot;/> </li></ul><ul><li><property name=&quot;EventToAddress&quot; value=&quot;urn:cpu&quot;/> </li></ul><ul><li></class> </li></ul><ul><li></filter> </li></ul><ul><li><filter source=&quot;get-property('To')&quot; regex=&quot;urn:cpu&quot;> </li></ul><ul><li> <send> </li></ul><ul><li><endpoint> </li></ul><ul><li><address uri=&quot;http://localhost:9090/notfication&quot;/> </li></ul><ul><li></endpoint> </li></ul><ul><li></send> </li></ul><ul><li></filter> </li></ul><ul><li></in> </li></ul><ul><li></definitions> </li></ul>
  18. 18. Performance <ul><li>Non-blocking IO </li></ul><ul><ul><li>Apache HTTPCore </li></ul></ul><ul><ul><li> </li></ul></ul><ul><li>Streaming XML and Binary </li></ul><ul><ul><li>Apache Axiom </li></ul></ul><ul><ul><li> </li></ul></ul>
  19. 19. Non-blocking graphically <ul><li>This model means: </li></ul><ul><li>Synapse threads never blocked during normal processing </li></ul><ul><li>Number of sockets open >> number of threads </li></ul>TIME Thread2 Incoming req Socket open Thread1 Socket open Request processing Response processing Outgoing resp Outgoing req Incoming resp Synapse
  20. 20. Demonstrating Performance <ul><li>Synapse by default runs </li></ul><ul><ul><li>2 listener threads </li></ul></ul><ul><ul><li>2 sender threads </li></ul></ul><ul><ul><li>8 worker threads </li></ul></ul><ul><li>Added a 100ms thread sleep to the server </li></ul><ul><li>Ran 250 concurrent clients for 10000 runs </li></ul><ul><ul><li>Simply would not have run without NIO </li></ul></ul><ul><li>Also did a simple test comparing: </li></ul><ul><ul><li>346 bytes in/ 1,170 bytes out </li></ul></ul><ul><ul><li>Direct to Axis2: 7.4ms </li></ul></ul><ul><ul><li>Via Synapse: 8.1ms – diff = 0.710ms !! </li></ul></ul>
  21. 21. XSLT performance
  22. 22. Extending Synapse <ul><li>Java Mediators </li></ul><ul><ul><li>Implement a simple Java API </li></ul></ul><ul><li>Java Commands </li></ul><ul><ul><li>Pure POJO with mapping into messages using XPath </li></ul></ul><ul><li>Scripts </li></ul><ul><ul><li>Using the Apache BSF project you can write scripts in JavaScript, Groovy and Ruby </li></ul></ul><ul><li>Synapse extensions </li></ul><ul><ul><li>First class mediators with their own “Domain” XML configuration </li></ul></ul><ul><ul><li>Simply drop the JAR into the classpath </li></ul></ul><ul><li>Tasks </li></ul><ul><ul><li>Repetitive jobs can be run at set intervals </li></ul></ul><ul><li>Registries </li></ul><ul><ul><li>Synapse can dynamically load its configuration from a remote registry </li></ul></ul><ul><li>Transports </li></ul><ul><ul><li>Create new connectors to remote systems </li></ul></ul>
  23. 23. Simple Mediator: CSV->XML <ul><li>public boolean mediate(MessageContext mc) { </li></ul><ul><ul><li>DataHandler dh = PayloadHelper. getBinaryPayload (mc); </li></ul></ul><ul><ul><li>BufferedReader br; </li></ul></ul><ul><ul><li>new BufferedReader(new InputStreamReader(dh.getInputStream())); </li></ul></ul><ul><ul><li>CSVReader csvReader = new CSVReader(br); </li></ul></ul><ul><ul><li>OMFactory fac = OMAbstractFactory. getOMFactory (); </li></ul></ul><ul><ul><li>OMElement el = fac.createOMElement(&quot;csv&quot;, csvNS); </li></ul></ul><ul><ul><li>// create element to hold data </li></ul></ul><ul><ul><li>while ((nextLine = csvReader.readNext()) != null) { </li></ul></ul><ul><ul><li>rownum++; </li></ul></ul><ul><ul><li>// add elements to XML </li></ul></ul><ul><ul><li>} </li></ul></ul><ul><ul><li>br.close(); </li></ul></ul><ul><ul><li>PayloadHelper. setXMLPayload (mc, el); </li></ul></ul><ul><ul><li>return true; </li></ul></ul><ul><li>} </li></ul>
  24. 24. Tasks – Quartz integration <ul><li>Simple repetitive actions </li></ul><ul><li>Can also be used to start a long-running activity at startup </li></ul><ul><li>Uses the Quartz Scheduler to run items </li></ul><ul><ul><li> </li></ul></ul><ul><li>Tasks must implement the Task interface </li></ul><ul><ul><ul><li>package org.apache.synapse.startup; </li></ul></ul></ul><ul><ul><ul><li>public interface Task </li></ul></ul></ul><ul><ul><ul><li>{ </li></ul></ul></ul><ul><ul><ul><li>public abstract void execute(); </li></ul></ul></ul><ul><ul><ul><li>} </li></ul></ul></ul><ul><li>Tasks may implement the ManagedLifecycle interface </li></ul><ul><li>Properties are set by injection (String and XML) </li></ul>
  25. 25. Sample task - MessageInjector <ul><li>public class MessageInjector implements Task, ManagedLifecycle </li></ul><ul><li>{ </li></ul><ul><li>public void setTo(String url) </li></ul><ul><li>{ to = url; } </li></ul><ul><li>public void setMessage(OMElement elem) </li></ul><ul><li>{ message = elem; } </li></ul><ul><li>public void execute() { </li></ul><ul><li> MessageContext mc = </li></ul><ul><li>synapseEnvironment.createMessageContext(); </li></ul><ul><li>mc.setTo(new EndpointReference(to)); </li></ul><ul><li>PayloadHelper.setXMLPayload(mc, </li></ul><ul><li> message.cloneOMElement()); </li></ul><ul><li>synapseEnvironment.injectMessage(mc); </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>
  26. 26. Script example <script language=&quot;js&quot;><![CDATA[ var rowset = mc.getPayloadXML() var ns = new Namespace(&quot;http://...&quot;); var d6 = rowset..ns::row.(@recordname==&quot;D6&quot;); var output = <FEEDS/> for (var i=0; i<d6.length(); i++) { var row = d6[i]; var cak = row.ns::entry.(@name==“CAK&quot;).text(); … var feed = <FEED> <CAK>{cak}</CAK> <AccNo>{accno}</AccNo> </FEED>; output.appendChild(feed); } mc.setPayloadXML(output); </script>
  27. 27. Community Site for extensions
  28. 28. Connectors, Adapters, Other projects <ul><li>CSV, Fixed and Variable records – FlatPack, OpenCSV </li></ul><ul><li>FIX transport using QuickFix/J </li></ul><ul><li>AMQP transport using Apache QPid </li></ul><ul><li>Complex Event Processing based on Esper </li></ul><ul><li>Atom using Apache Abdera </li></ul><ul><li>Rules using Drools </li></ul><ul><li>EDI using Smooks </li></ul><ul><li>BPEL using Apache ODE </li></ul><ul><li>Hessian Binary XML using Hessian </li></ul><ul><li>URLRewriter </li></ul><ul><li>IM Mediator MSN, ICQ, Yahoo, Jabber </li></ul>
  29. 29. User quotes <ul><li>&quot;Synapse's simplicity of configuration, ease of extensibility, and integration were the key factors in our choice. Synapse's speed and stability, non-blocking I/O support, and responsive development community then sealed the decision,&quot; </li></ul><ul><li>&quot;Synapse passed all our performance benchmark, load and memory leak tests with flying colours.&quot; </li></ul><ul><li>“ Thanks for the quick fix. Synapse continues to rock.” </li></ul>
  30. 30. Futures <ul><li>We continue to add new mediators and transports </li></ul><ul><li>Better support for complete HTTP Proxy </li></ul><ul><ul><li>Jointly done with HTTPCore </li></ul></ul><ul><li>More work on a non-XML DSL for Synapse </li></ul><ul><li>Better support for REST </li></ul><ul><ul><li>DELETE and PUT support </li></ul></ul><ul><li>Unifying support for events </li></ul><ul><ul><li>WS-Eventing, Atom, RSS, XMPP </li></ul></ul><ul><li>Improved cluster management </li></ul><ul><ul><li>Graceful restart </li></ul></ul><ul><li>Improved JMX management </li></ul><ul><ul><li>Better granularity </li></ul></ul>
  31. 31. Getting Started
  32. 32. Getting Involved <ul><li>Join us </li></ul><ul><ul><li>[email_address] </li></ul></ul><ul><ul><li>[email_address] </li></ul></ul><ul><li>Raise a JIRA for an improvement you would like </li></ul><ul><ul><li> </li></ul></ul><ul><li>Submit a patch </li></ul><ul><li>Contribute a mediator or a task definition </li></ul>
  33. 33. Questions?