Fast SOA with Apache Synapse

Slide 1: High Speed SOA with Apache Synapse Paul Fremantle VP, Apache Synapse CTO, WSO2 pzf@apache.org

Slide 2: Contents • What is Apache Synapse • History of the project • Some examples • Delving into the model • Performance • Connectors, Adaptors • Getting started • Getting involved

Slide 3: What is Apache Synapse? • Is it? – A highly performant XML Gateway – An Enterprise Service Bus – An HTTP proxy server – A protocol switcher – A way of hosting message manipulation logic

Slide 4: What is Apache Synapse? • A lightweight Enterprise Services Bus (ESB) – Available as a WAR file, NT Service, Linux Daemon – Runs as a process with its own Listeners, Tasks and Senders – Can be deployed standalone or part of a cluster or distributed network – High performance, asynchronous, streaming design – Can initiate work – scheduled tasks – Supports multiple transports including HTTP, JMS, TCP, SMTP and (S)FTP – Simple to extend

Slide 5: Flows HTTP HTTP 8080 8080 inflow HTTPS HTTPS 8443 8443 log xslt send JMS JMS SMTP outflow SMTP send xslt log

Slide 6: Synapse startup Using Bouncy castle JAR for Java 1.5 Starting Synapse/Java ... Using SYNAPSE_HOME: C:SYNAPS~1.1bin.. Using JAVA_HOME: c:jdk Using SYNAPSE_XML: -Dsynapse.xml="C:SYNAPS~1.1bin..repositoryconfsynapse.xml" 2007-11-12 12:16:58,250 [-] [main] INFO ServerManager Using the Axis2 Repository C:SYNAPS~1.1bin..repository 2007-11-12 12:17:01,921 [-] [main] INFO SynapseInitializationModule Initializing Synapse at : Mon Nov 12 12:17:01 GMT 2007 2007-11-12 12:17:01,937 [127.0.0.1-pzfdell] [main] INFO SynapseInitializationModule Loading mediator extensions... 2007-11-12 12:17:01,937 [127.0.0.1-pzfdell] [main] INFO SynapseInitializationModule Initializing the Synapse configuration ... 2007-11-12 12:17:01,968 [127.0.0.1-pzfdell] [main] INFO XMLConfigurationBuilder Generating the Synapse configuration model by parsing the XML configuration (some deleted) 2007-11-12 12:17:04,359 [127.0.0.1-pzfdell] [main] INFO HttpCoreNIOSender HTTP Sender starting 2007-11-12 12:17:04,968 [127.0.0.1-pzfdell] [main] INFO HttpCoreNIOListener HTTPS Listener starting on port : 8443 2007-11-12 12:17:04,968 [127.0.0.1-pzfdell] [main] INFO ServerManager Starting transport https on port 8443 2007-11-12 12:17:05,046 [127.0.0.1-pzfdell] [main] INFO ServerManager Ready for processing

Slide 7: History of the project • August 2005 Kicked off as an incubator proposal • December 2005 Milestone 1 released • July 2006 Milestone 2 released • December 2006 Synapse 0.90 released • January 2007 Graduates from Incubator • May 2007 Released 1.0 • December 2007 Becomes a Top Level Project • January 2008 Synapse 1.1.1 released

Slide 8: A really simple example Exposing an existing SOAP endpoint as XML/HTTP, XML/JMS <definitions> <proxy name=“example" transports=“http jms"> <target> <endpoint> <wsdl uri=“http://remote.com/service?wsdl“ service=“RemoteService" port=“RemotePort"/> </endpoint> <outSequence> <send/> </outSequence> </target> </proxy> </definitions>

Slide 9: Some other things you can do “out of the box” • Content-based routing – Direct messages based on properties or XPath expressions • Convert from existing formats (CSV, Fixed records, etc) to and from XML • Transform XML – Using XSLT or XQuery • Validate using XML Schema • Update databases and enhance messages with extra data – DBLookup and DBReport mediators – any JDBC database • Switch between SOAP and non-SOAP, any transport – HTTP, HTTPS, SMTP/POP3, XMPP, JMS, TCP, Filesystem, FTP, AMQP • Add or handle WS-* support including – WS-Security, WS-SecureConversation, WS-Trust, WS- ReliableMessaging • Load balancing, failover, throttling

Slide 10: Registries and Dynamism read Synapse and cache “Registry” notify HTTP extensible “Transactional updates” -existing flows continue to use old config Also graceful restart -switch off transport cleanly -restart Synapse when all work is done

Slide 11: A more detailed example WS-Security (encryption and digital signature) NY London MQ <FEED> <CAK>2243578</CAK> <AccNo>3289019</AccNo> <CAT>AK</CAT> Validation … </FEED> Variable Existing Format trade records database D2243578XX910 D6233578YYYY9

Slide 12: Overview of the flow <sequence name=“new york”> Flatpack with Variable record config E4X script to reformat XML Send to London </sequence> <sequence name=“london”> DBreport Filter Send to MQ </sequence>

Slide 13: Results • Replicated a 3 month project is 3 days – Included enhancing the Flatpack mediator to support Variable Records – Added WS-Security over original scenario – 1200 lines of config reduced to less than 300 – Much more maintainable – clean separation of concerns between departments

Slide 14: Google Spreadsheet and CSV Synapse Google.com Http Poller CSV to XML Atom CSV Job

Slide 15: Esper • Event Stream Processing project – http://esper.codehaus.org • Allows you to define queries on sets of “events” SELECT AVG(price) FROM OrderEvent.WIN:TIME(30 sec) • EsperMediator for Synapse makes it simple to add CEP to your SOA

Slide 16: Ganglia, Quartz, Esper Ganglia Synapse gmond Quartz Esper Alert Scheduler Mediator notification SELECT AVG(CPU_USER) telnet 8649 urn:gmond FROM GANGLIA urn:cpu IM <GANGLIA_XML> GMondPoller Atom Job or JMS

Slide 17: <definitions xmlns="http://ws.apache.org/ns/synapse"> <task class="org.fremantle.gmond.GMonTask“ name=“Gmon”> <trigger interval="5000"/> <property name="hostname" value="localhost"/> <property name="port" value="8649"/> <property name="to" value="urn:gmond"/> </task> <in> <filter source="get-property('To')" regex="urn:gmond"> <log/> <class name="org.fremantle.esper.EsperMediator"> <property name="Configuration" value=“./repository/conf/esper.conf.xml"/> <property name="statement" value="select avg(cpu_user) from Ganglia.win:length_batch(10) "/> <property name="EventToAddress" value="urn:cpu"/> </class> </filter> <filter source="get-property('To')" regex="urn:cpu"> <send> <endpoint> <address uri="http://localhost:9090/notfication"/> </endpoint> </send> </filter> </in> </definitions>

Slide 18: Performance • Non-blocking IO – Apache HTTPCore – http://hc.apache.org/httpcomponents-core/htt • Streaming XML and Binary – Apache Axiom – http://ws.apache.org/commons/axiom

Slide 19: Non-blocking graphically Incoming req Synapse Request Thread1 processing Outgoing req Socket open Socket open TIME Incoming resp Response Thread2 processing Outgoing resp This model means: 2. Synapse threads never blocked during normal processing 3. Number of sockets open >> number of threads

Slide 20: Demonstrating Performance • Synapse by default runs – 2 listener threads – 2 sender threads – 8 worker threads • Added a 100ms thread sleep to the server • Ran 250 concurrent clients for 10000 runs – Simply would not have run without NIO • Also did a simple test comparing: – 346 bytes in/ 1,170 bytes out – Direct to Axis2: 7.4ms – Via Synapse: 8.1ms – diff = 0.710ms!!

Slide 21: XSLT performance

Slide 22: Extending Synapse • Java Mediators – Implement a simple Java API • Java Commands – Pure POJO with mapping into messages using XPath • Scripts – Using the Apache BSF project you can write scripts in JavaScript, Groovy and Ruby • Synapse extensions – First class mediators with their own “Domain” XML configuration – Simply drop the JAR into the classpath • Tasks – Repetitive jobs can be run at set intervals • Registries – Synapse can dynamically load its configuration from a remote registry • Transports – Create new connectors to remote systems

Slide 23: Simple Mediator: CSV->XML public boolean mediate(MessageContext mc) { DataHandler dh = PayloadHelper.getBinaryPayload(mc); BufferedReader br; new BufferedReader(new InputStreamReader(dh.getInputStream())); CSVReader csvReader = new CSVReader(br); OMFactory fac = OMAbstractFactory.getOMFactory(); OMElement el = fac.createOMElement("csv", csvNS); // create element to hold data while ((nextLine = csvReader.readNext()) != null) { rownum++; // add elements to XML } br.close(); PayloadHelper.setXMLPayload(mc, el); return true; }

Slide 24: Tasks – Quartz integration • Simple repetitive actions • Can also be used to start a long-running activity at startup • Uses the Quartz Scheduler to run items – www.opensymphony.com/quartz • Tasks must implement the Task interface package org.apache.synapse.startup; public interface Task { public abstract void execute(); } • Tasks may implement the ManagedLifecycle interface • Properties are set by injection (String and XML)

Slide 25: Sample task - MessageInjector public class MessageInjector implements Task, ManagedLifecycle { public void setTo(String url) { to = url; } public void setMessage(OMElement elem) { message = elem; } public void execute() { MessageContext mc = synapseEnvironment.createMessageContext(); mc.setTo(new EndpointReference(to)); PayloadHelper.setXMLPayload(mc, message.cloneOMElement()); synapseEnvironment.injectMessage(mc); } }

Slide 26: Script example <script language="js"><![CDATA[ var rowset = mc.getPayloadXML() var ns = new Namespace("http://..."); var d6 = rowset..ns::row.(@recordname=="D6"); var output = <FEEDS/> for (var i=0; i<d6.length(); i++) { var row = d6[i]; var cak = row.ns::entry.(@name==“CAK").text(); … var feed = <FEED> <CAK>{cak}</CAK> <AccNo>{accno}</AccNo> </FEED>; output.appendChild(feed); } mc.setPayloadXML(output); </script>

Slide 27: Community Site for extensions http://esbsite.org

Slide 28: Connectors, Adapters, Other projects • CSV, Fixed and Variable records – FlatPack, OpenCSV • FIX transport using QuickFix/J • AMQP transport using Apache QPid • Complex Event Processing based on Esper • Atom using Apache Abdera • Rules using Drools • EDI using Smooks • BPEL using Apache ODE • Hessian Binary XML using Hessian • URLRewriter • IM Mediator MSN, ICQ, Yahoo, Jabber

Slide 29: User quotes "Synapse's simplicity of configuration, ease of extensibility, and integration were the key factors in our choice. Synapse's speed and stability, non-blocking I/O support, and responsive development community then sealed the decision," "Synapse passed all our performance benchmark, load and memory leak tests with flying colours." “Thanks for the quick fix. Synapse continues to rock.”

Slide 30: Futures • We continue to add new mediators and transports • Better support for complete HTTP Proxy – Jointly done with HTTPCore • More work on a non-XML DSL for Synapse • Better support for REST – DELETE and PUT support • Unifying support for events – WS-Eventing, Atom, RSS, XMPP • Improved cluster management – Graceful restart • Improved JMX management – Better granularity

Slide 31: Getting Started

Slide 32: Getting Involved • Join us – dev@synapse.apache.org – user@synapse.apache.org • Raise a JIRA for an improvement you would like – http://issues.apache.org/jira/browse/SYNAPSE • Submit a patch • Contribute a mediator or a task definition

Slide 33: Questions?