Succeeding with the Apache SOA stackAnd debunking some SOA mythsJohan EdstromSOA Executive and Apache developer.jedstrom@ savoirtech.comjoed@ apache.orgJeff GenenderPrincipaljgenender@ savoirtech.com
Agenda What is an ESB? • Where would an “ESB” fit in? • How do you choose the ESB? • Reasons for picking the Apache ServiceMix stack Modular Stable Cluster capable How to design software and the build system for an ESB • Karaf and ServiceMix are OSGi based, ServiceMix also can handle JBI. • The Apache projects are modularized using Maven. Get over it and accept it :)
Key parts to scalability The right resources in the right place Messaging • Or any other asynchronous system State driven Conversational Effectively aggregating Transactionally safe Correctly utilizing EE
W is an ESB? hat In my world it is simply a runtime for deploying asynchronous, event driven code. • So, yes; of course most of you have deployed and built ESB solutions. • Yes, of course we are partial to “ESB” solutions. What is an ESB in marketing slides? • Components • Data-transformations • Scalability and performance (You can randomly wave your hands here) • A Bus This is a mythical, magical, illusionary fantastically awesome thing that transparently just makes everything work.
Where does the marketing ESB fit in? Picking up a file from a file area, transforming it with XSLT, publishing to a web-service. Consuming a web-service, using a RoutingSlip and sending to a product, then inventory queue on a JMS provider Consuming a web-service payload and transforming the data to an insert in a JDBC component Providing “Enterprise routing infrastructure” Providing “Enterprise orchestration”
And what about a real world one? Where you need to do any of the marketing ESB stuff • Of course they are valid use-cases Where you want web-apps to be able to share resources • ServiceMix / Karaf happily will work as a servlet container Long running processing, “Orchestration” Any type of data transformation scenarios
Real one Multi-Protocol input/output Complex publish subscribe systems Where you need to process in parallel In Springism’s - anywhere you have a Bean-Ref you could start breaking things apart.
So how is that “ESB” designed? This is not far from an EE container • Add OpenEJB if you want and you’ll have a full stack It contains a web-services layer Dependency injection services OSGi integration and legacy (JBI) support Camel for routing, transformation and mediation Aries and Spring for EE support, JNDI, TX, JPA Web-app deployment support
So why is that so cool? Well.... You can treat it in the classical manner Or as a really kick-ass funky container Or - as a nice fusion of them both!
Apache ActiveMQ Communications platform of choice, providing high performance, scalability, and mission-critical reliability for distributed enterprise computing. Apache ActiveMQ supports based on specs: • JMS 1.1 • J2EE 1.4 integration-related components including JDBC, JCA, and EJBs; dependent specifications such as JTA and JNDI; • AJAX, REST, HTTP, TCP, SSL, NIO+SSL, UDP, multicast, JGroups and JXTA transport protocols
ActiveMQ “Instability” Instability... what is it? • Most common term used by end users • Generally refers to ActiveMQ being frozen or completely seized Refers to producers being unable to send message Refers to queues being frozen/seized or stuck messages Generally temporarily is fixed by the holdover from the Microsoft days • The 3-finger salute (ctrl-alt-delete) • Restart of ActiveMQ Followed by complaints that ActiveMQ is trash • Blog by David Pollak: “ActiveMQ: not ready for prime time” http://goodstuff.im/activemq-not-ready-for-prime-time • Is ActiveMQ trash or not properly written? Not trash...its a very properly written... its extremely powerful • If I have these problems, what do you do? Diagnose it first ;-)
Apache ActiveMQ This is another presentation :) • So you need to attend that one if you want to debate ActiveMQ.
Apache ActiveMQ - Additional functionality Advisory Queues and Topics are administrative channels. • The Broker periodically posts information about the status of known destinations • Administration focused programs can subscribe to advisory messages like any other Topic Useful life-cycle and supplementary notifications Advisories are generated for the following: • Starting and stopping consumers and producers • Creating and destroying temporary destinations • Expiring messages on topics and queues • Sending messages from brokers to destinations with no consumers • Starting and stopping connections
Apache ActiveMQ, Easy to scale Can bridge, multicast, “cluster” as well as integrate with other JMS providers
Apache ActiveMQ - High Availability Pro ConReplicated No “single” point of Supports only a single failure slave. Requires manual restart and synchronization to bring back a failed masterShared File No “single” point of Requires a networked failure. Can run file-system solution multiple slaves, with locking automatic recovery.JDBC No “single” point of Slow failure. Can run multiple slaves,
Apache Camel A powerful rule based routing and mediation engine. Provides a POJO-based implementation of Enterprise Integration Patterns using Java DSL (Domain Specific Language) or Spring / Aries Blueprint to configure routing and mediation rules. Uses generics, annotations and URIs so that it can easily work directly with any kind of transport or messaging model such as HTTP, JMS, JBI, SCA, MINA or CXF Bus API without mandating a normalized message API. Designed as a small library which has “minimal” dependencies for easy embedding in any Java application.
Apache Camel Features Camel is a Domain Specific Language (DSL) focussed on implementing Enterprise Integration Patterns (EIPs) • Examples: multicast, splitter, content-based router, routing slip, dynamic routers, aggregator EIPs are implemented by Camel “Processors” • Users can develop their own processors • A processor is a fundamental building block • Bean language and bindings exists so that not a single piece of Apache Camel Imports will be necessary when integrating your existing code Camel can connect to a wide variety of integration technologies • Examples: JMS, HTTP, FTP, SOAP, File - There are ~ 120 components • Integrations are implemented by Camel “Components” • Users can develop their own components
And it is a pretty healthy library! Camel makes application design & development easy: • Reduces time to create integration flows • Reduces complexity of integration flows and makes them easier to maintain • Routes can be deployed in mainstream typical containers (OSGi, JEE, Servlet, etc.) Used in several commercial offerings Used in “competing” offerings About as close to an Integrations standard as you can get The community stays at a high level: • 503 subscribers to the users mailing list • 183 subscribers to the dev mailing list The community is active • ~863 mails per month on users mailing list in Mar 2011 - May 2012 (+25 %) • ~167 mails per month on the dev mailing list in Mar 2011 - May 2012 (-15 %) • Avg. 535 commits per month in Mar 2011 - May 2012 (+5 %)
CXF Simple, flexible, interceptor based, standards compliant framework for building JAX-WS and JAX-RS services and consumers
Additional - Application development tools Spring-DM and Spring Aries Blueprint Google Guice Felix iPojo And so on.....
Do I need all of that? Nope, many solutions will need just a few things • jaxb, camel, jms, soap, rest and perhaps jdbc • Cut the container down to fit your needs • We don’t need to load all of the 100+ Apache Camel components • Pick and choose! Should I run that messaging solution inside the “ESB” • Entirely up to you, let us look a little deeper at that in a sec. Can I test these solutions or am I stuck with System.out.println and a remote debugger?
I’ve heard that OSGi sucks? Yes, It is a bit more complex to manage a full lifecycle • On the other hand you get reliable hot-deployment and things like configuration, service management and subscriptions out of the box. Yes, package level import export can be confusing • So make sure you use packages in a nice modular way. Yes, it takes a while to learn the tooling • Then did you wake up one day just knowing that WAR or EAR layout? Yes, you need to learn the classpath and a bit about classloading • Which’ll make you a better developer in the end of the day. • Many problems in this area actually are created by applying EE classloaders to an OSGi environment.
More suckage You have to write modular software!!!! • Yes, you’ll benefit from writing modular software Heh :) Yes, some third party libraries are “Fluster-Clucks” in OSGi • You can always re-package em’ or use something else. • ServiceMix bundles Yes, logging is strange and odd • So Pax logging is provided Yes, It was hard to get Hibernate working at one point in time • And there are like a bazillion blogs about getting it working.....
W is the simplest way of addressing teh SUKZ? hat Use Apache Maven. • Just accept it. Designate a maven fascist • Keep your builds nice and clean • Physically and monetary make sure that you abuse people breaking builds • Structure your projects as early as you can • Don’t be afraid of using a truckload of modules • Get familiar with the OSGi plugins, let your modules inherit base settings Go steal that build setup for Apache Karaf or Apache Camel! Use a whiteboard and perhaps do a few diagrams every now and then? Other ways of 100% guaranteed suckcess is to rely only on blogs, random advice on irc and never reading source code.
And keep this in mind...Spring-DM... it’s a bit like Trump’s hair...Spring-DM... it’s a bit like Trump’s hair...Its there... it covers quite a bit...but it’s kinda hard to recommend.Looks great... but needs a lot of maintenanceor it becomes a big mess.
Let’s say we liked the Apache stack! And let us say we have a very simple and silly use-case • We want to accept incoming SOAP requests • We need to inspect those HTTP headers and then use them to route • We want this to be dynamic and flexible We might want to add a more complex routing engine later • We want this to be modular and possible to run across multiple systems This tells us we could use • Spring or Aries Blueprint, Google Guice or plain old Java • We will use Apache CXF for the WS part • We will use Apache Camel for connecting this and routing • We will use JMS via Apache Camel-JMS
Proof of concept outline Let us use some EIP pattern symbols to outline this • We haven’t really made any technology statements here We are going to use Apache ActiveMQ because I said so. It could easily have been any other JMS provider. • We know we’ll have a synchronous part Web-services by nature are. • We’ll try and make all the other parts asynchronous • We’ll also make sure that the participants don’t need strong integration or visibility of each other, thus we’ll (hopefully) end up with some pretty re-usable modules • Another good approach is to also use sequence diagrams
Let’s craft some code! We’ll start with the Maven setup • Use properties • Define a parent • Like said previously, steal a good setup • Inherit, inherit, inherit • Use modules We also ripped a WSDL from Apache CXF • SOAP Entry point • Gives something to code-gen against • Common use-case
Maven Complicated parent pom Very simple sub-modules
W else do we want here? hat A “Dynamic” Router! • We’ll craft a simple header based Dynamic router from eip patterns. Technically it is in between a routing slip with termination and a real control channel pattern, we utilize JMS as our control channel for this. We use this example to play with destinations from SoapUI
The “Dynamic Router” Yep, it is just a Java class. • Imagine how easily you could integrate this with.... Anything!
W build that type of a route? hy We are able to turn “endpoints”, “routing”, “systems” or whatever you want to call it into simple destination names. As we get more complex we only need to control the hop- building. We can break all these modules apart
So let’s sum up what we have at this point We defined a CXF (JAX-WS) Endpoint, generated stubs and tied it into Camel • The MEP (Message Exchange Pattern) is IN/OUT since we use CXF We then dropped in a router “Our Dynamic Router” • It’ll listen to a JMS queue called recognizer • Then it’ll start inspecting headers for a location
Header inspections A simple processor, if we wanted to we could make similar transformations in XML, Simple, Scala and so on. We are just modifying the message header info.
W about that testing? hat We are going to be dealing with an OSGi environment. • Like an EE container it makes testing a tad less fun.... Full blown container testing with something like SoapUI • Can be scripted and automated in Maven Full OSGi tests with Pax-Exam • Allows you to wire up a full container Simple OSGi registry tests with PojoSr • Let’s you simulate an environment so you can test BluePrint! Functional Junit/TestNG tests with Camel-Test. • Should be combined with at least one of the above!
Let’s start with a Camel-T T est est. Several flavors • Junit, Junit4, TestNG, Spring and so on. • Exposes easy access to context, components and inlined route-builders
W got slightly distracted in looking at T Options... e est First of, let us just do a Camel Test, we want to verify that the Dynamic routing works as expected. • As you can see, none of these routes know about each other.
Then we want to check that we can load the WS Blueprint context - We rely on some AMQ Magic as well!
And our wiring “Test” First off, the “Dummy Response” • We simulate the real route that only grabs a message and dumps it on a queue, we add 2 things. Response building A Mock destination for validation.
T method est With this we have a full integrations test including • JMS, CXF, OSGi ServiceRegistry, SPI lookup, Component Discovery, JAXB validation and schema generation....
Green Bars!!! We’ve tested most functionality, all we really need to do • Write the “response-builder” • Make that response builder a bit more dynamic so it can work Against several queues With different responses Yes, it is a dummy class. But since it is a Java Object it is pretty darn easy to let your imagination run free.
Dynamic config data in Blueprint Works like springs property placeholder Allows us to set defaults A persistent-id corresponds to a config file in ./etc in ServiceMix or Karaf.
How we’ll be using this dynamic data We basically can use this to write completely dynamic steps that will restart on a configuration change from • File changes • Code changes • JMX changes
Proof of concept outline We now have the following modules • customer-service : A model library • customer-service-ws : The webservices endpoint • dynamic-routing : A generic module for routing • response-builder : A simple module that returns a payload • That means that all of the needed patterns are fulfilled. • Competing consumer is a natural thing in JMS • Eventing is a natural part of Apache Camel • Request Reply is “Handled for us” based on endpoint and MEP
T sum everything up o Simple OSGi build, we added exports in one place. Completely modular design Re-usable bundles We can deploy this on one or many machines We can “cluster” a network of brokers We can easily change transport protocols We are covered by tests
Thank you very much for your attention! Questions? Links • http://camel.apache.org/ • http://servicemix.apache.org/ • http://cxf.apache.org/ • http://activemq.apache.org/ • http://karaf.apache.org/