Fast SOA with Apache Synapse

High Speed SOA with Apache Synapse Paul Fremantle VP, Apache Synapse CTO, WSO2 [email_address]

Contents

What is Apache Synapse

History of the project

Some examples

Delving into the model

Performance

Connectors, Adaptors

Getting started

Getting involved

What is Apache Synapse?

Is it?

A highly performant XML Gateway

An Enterprise Service Bus

An HTTP proxy server

A protocol switcher

A way of hosting message manipulation logic

What is Apache Synapse?

A lightweight Enterprise Services Bus (ESB)

Available as a WAR file, NT Service, Linux Daemon

Runs as a process with its own Listeners, Tasks and Senders

Can be deployed standalone or part of a cluster or distributed network

High performance, asynchronous, streaming design

Can initiate work – scheduled tasks

Supports multiple transports including HTTP, JMS, TCP, SMTP and (S)FTP

Simple to extend

Flows JMS SMTP inflow log xslt send outflow send xslt log HTTP 8080 HTTPS 8443 JMS SMTP HTTP 8080 HTTPS 8443

Synapse startup

Using Bouncy castle JAR for Java 1.5

Starting Synapse/Java ...

Using SYNAPSE_HOME: C:SYNAPS~1.1in..

Using JAVA_HOME: c:jdk

Using SYNAPSE_XML: -Dsynapse.xml="C:SYNAPS~1.1in.. epositoryconfsynapse.xml"

2007-11-12 12:16:58,250 [-] [main] INFO ServerManager Using the Axis2 Repository C:SYNAPS~1.1in.. epository

2007-11-12 12:17:01,921 [-] [main] INFO SynapseInitializationModule Initializing Synapse at : Mon Nov 12 12:17:01 GMT 2007

2007-11-12 12:17:01,937 [127.0.0.1-pzfdell] [main] INFO SynapseInitializationModule Loading mediator extensions...

2007-11-12 12:17:01,937 [127.0.0.1-pzfdell] [main] INFO SynapseInitializationModule Initializing the Synapse configuration ...

2007-11-12 12:17:01,968 [127.0.0.1-pzfdell] [main] INFO XMLConfigurationBuilder Generating the Synapse configuration model by parsing the XML configuration

(some deleted)

2007-11-12 12:17:04,359 [127.0.0.1-pzfdell] [main] INFO HttpCoreNIOSender HTTP Sender starting

2007-11-12 12:17:04,968 [127.0.0.1-pzfdell] [main] INFO HttpCoreNIOListener HTTPS Listener starting on port : 8443

2007-11-12 12:17:04,968 [127.0.0.1-pzfdell] [main] INFO ServerManager Starting transport https on port 8443

2007-11-12 12:17:05,046 [127.0.0.1-pzfdell] [main] INFO ServerManager Ready for processing

History of the project

August 2005 Kicked off as an incubator proposal

December 2005 Milestone 1 released

July 2006 Milestone 2 released

December 2006 Synapse 0.90 released

January 2007 Graduates from Incubator

May 2007 Released 1.0

December 2007 Becomes a Top Level Project

January 2008 Synapse 1.1.1 released

A really simple example

Exposing an existing SOAP endpoint as XML/HTTP, XML/JMS

<definitions>

<proxy name=“example" transports=“http jms">

<target>

<endpoint>

<wsdl uri=“http://remote.com/service?wsdl“

service=“RemoteService"

port=“RemotePort"/>

</endpoint>

<outSequence>

<send/>

</outSequence>

</target>

</proxy>

</definitions>

Some other things you can do “out of the box”

Content-based routing

Direct messages based on properties or XPath expressions

Convert from existing formats (CSV, Fixed records, etc) to and from XML

Transform XML

Using XSLT or XQuery

Validate using XML Schema

Update databases and enhance messages with extra data

DBLookup and DBReport mediators – any JDBC database

Switch between SOAP and non-SOAP, any transport

HTTP, HTTPS, SMTP/POP3, XMPP, JMS, TCP, Filesystem, FTP, AMQP

Add or handle WS-* support including

WS-Security, WS-SecureConversation, WS-Trust, WS-ReliableMessaging

Load balancing, failover, throttling

Registries and Dynamism Synapse “Registry” HTTP extensible read and cache notify

“ Transactional updates”

existing flows continue to use old config

Also graceful restart

switch off transport cleanly

restart Synapse when all work is done

A more detailed example NY Variable Format records London Existing trade database D2243578XX910 D6233578YYYY9 <FEED> <CAK>2243578</CAK> <AccNo>3289019</AccNo> <CAT>AK</CAT> … </FEED> WS-Security (encryption and digital signature) MQ Validation

Overview of the flow

<sequence name=“new york”>

Flatpack with Variable record config

E4X script to reformat XML

Send to London

</sequence>

<sequence name=“london”>

DBreport

Filter

Send to MQ

</sequence>

Results

Replicated a 3 month project is 3 days

Included enhancing the Flatpack mediator to support Variable Records

Added WS-Security over original scenario

1200 lines of config reduced to less than 300

Much more maintainable – clean separation of concerns between departments

Google Spreadsheet and CSV Google.com CSV Synapse Http Poller Job CSV to XML Atom

Esper

Event Stream Processing project

http://esper.codehaus.org

Allows you to define queries on sets of “events”

SELECT AVG(price) FROM OrderEvent.WIN:TIME(30 sec)

EsperMediator for Synapse makes it simple to add CEP to your SOA

Ganglia, Quartz, Esper Synapse Quartz Scheduler Ganglia gmond telnet 8649 <GANGLIA_XML> GMondPoller Job Esper Mediator SELECT AVG(CPU_USER) FROM GANGLIA urn:gmond Alert notification IM Atom or JMS urn:cpu

<definitions xmlns="http://ws.apache.org/ns/synapse">

<task class="org.fremantle.gmond.GMonTask“ name=“Gmon”>

<trigger interval="5000"/>

<property name="hostname" value="localhost"/>

<property name="port" value="8649"/>

<property name="to" value="urn:gmond"/>

</task>

<in>

<filter source="get-property('To')" regex="urn:gmond">

<log/>

<class name="org.fremantle.esper.EsperMediator">

<property name="Configuration"

value=“./repository/conf/esper.conf.xml"/>

<property name="statement"

value="select avg(cpu_user) from Ganglia.win:length_batch(10) "/>

<property name="EventToAddress" value="urn:cpu"/>

</class>

</filter>

<filter source="get-property('To')" regex="urn:cpu">

<send>

<endpoint>

<address uri="http://localhost:9090/notfication"/>

</endpoint>

</send>

</filter>

</in>

</definitions>

Performance

Non-blocking IO

Apache HTTPCore

http://hc.apache.org/httpcomponents-core/httpcore-nio/index.html

Streaming XML and Binary

Apache Axiom

http://ws.apache.org/commons/axiom

Non-blocking graphically

This model means:

Synapse threads never blocked during normal processing

Number of sockets open >> number of threads

TIME Thread2 Incoming req Socket open Thread1 Socket open Request processing Response processing Outgoing resp Outgoing req Incoming resp Synapse

Demonstrating Performance

Synapse by default runs

2 listener threads

2 sender threads

8 worker threads

Added a 100ms thread sleep to the server

Ran 250 concurrent clients for 10000 runs

Simply would not have run without NIO

Also did a simple test comparing:

346 bytes in/ 1,170 bytes out

Direct to Axis2: 7.4ms

Via Synapse: 8.1ms – diff = 0.710ms !!

XSLT performance

Extending Synapse

Java Mediators

Implement a simple Java API

Java Commands

Pure POJO with mapping into messages using XPath

Scripts

Using the Apache BSF project you can write scripts in JavaScript, Groovy and Ruby

Synapse extensions

First class mediators with their own “Domain” XML configuration

Simply drop the JAR into the classpath

Tasks

Repetitive jobs can be run at set intervals

Registries

Synapse can dynamically load its configuration from a remote registry

Transports

Create new connectors to remote systems

Simple Mediator: CSV->XML

public boolean mediate(MessageContext mc) {

DataHandler dh = PayloadHelper. getBinaryPayload (mc);

BufferedReader br;

new BufferedReader(new InputStreamReader(dh.getInputStream()));

CSVReader csvReader = new CSVReader(br);

OMFactory fac = OMAbstractFactory. getOMFactory ();

OMElement el = fac.createOMElement("csv", csvNS);

// create element to hold data

while ((nextLine = csvReader.readNext()) != null) {

rownum++;

// add elements to XML

}

br.close();

PayloadHelper. setXMLPayload (mc, el);

return true;

}

Tasks – Quartz integration

Simple repetitive actions

Can also be used to start a long-running activity at startup

Uses the Quartz Scheduler to run items

www.opensymphony.com/quartz

Tasks must implement the Task interface

package org.apache.synapse.startup;

public interface Task

{

public abstract void execute();

}

Tasks may implement the ManagedLifecycle interface

Properties are set by injection (String and XML)

Sample task - MessageInjector

public class MessageInjector implements Task, ManagedLifecycle

{

public void setTo(String url)

{ to = url; }

public void setMessage(OMElement elem)

{ message = elem; }

public void execute() {

MessageContext mc =

synapseEnvironment.createMessageContext();

mc.setTo(new EndpointReference(to));

PayloadHelper.setXMLPayload(mc,

message.cloneOMElement());

synapseEnvironment.injectMessage(mc);

}

}

Script example <script language="js"><![CDATA[ var rowset = mc.getPayloadXML() var ns = new Namespace("http://..."); var d6 = rowset..ns::row.(@recordname=="D6"); var output = <FEEDS/> for (var i=0; i<d6.length(); i++) { var row = d6[i]; var cak = row.ns::entry.(@name==“CAK").text(); … var feed = <FEED> <CAK>{cak}</CAK> <AccNo>{accno}</AccNo> </FEED>; output.appendChild(feed); } mc.setPayloadXML(output); </script>

Community Site for extensions http://esbsite.org

Connectors, Adapters, Other projects

CSV, Fixed and Variable records – FlatPack, OpenCSV

FIX transport using QuickFix/J

AMQP transport using Apache QPid

Complex Event Processing based on Esper

Atom using Apache Abdera

Rules using Drools

EDI using Smooks

BPEL using Apache ODE

Hessian Binary XML using Hessian

URLRewriter

IM Mediator MSN, ICQ, Yahoo, Jabber

User quotes

"Synapse's simplicity of configuration, ease of extensibility, and integration were the key factors in our choice. Synapse's speed and stability, non-blocking I/O support, and responsive development community then sealed the decision,"