Download as PDF, PPTX
![I. IS Integration heuristics
Just do it !
Just coding simple things might be just not so simple …
public class FileCopier {
public static void main(String args[]) throws Exception {
File inboxDirectory = new File("data/inbox");
File outboxDirectory = new File("data/outbox");
outboxDirectory.mkdir();
File[] files = inboxDirectory.listFiles();
for (File source : files) {
if (source.isFile()) {
File dest = new File(
outboxDirectory.getPath()
+ File.separator
+ source.getName());
copyFile(source, dest);
}
}
}
private static void copyFile(File source, File dest)
throws IOException {
OutputStream out = new FileOutputStream(dest);
byte[] buffer = new byte[(int) source.length()];
FileInputStream in = new FileInputStream(source);
in.read(buffer);
try {
out.write(buffer);
} finally {
out.close();
in.close();
}
}
}](https://image.slidesharecdn.com/2011-04-apachecamel-131101081020-phpapp01/85/Apache-Camel-Stephane-Kay-April-2011-5-320.jpg)
![II. Camel’s bleat | routing
Java DSL : Builder pattern
The Builder Pattern (GOF) is central to the Camel Java DSL
public class FileCopierWithCamel {
public static void main(String args[]) throws Exception {
// create CamelContext
CamelContext context = new DefaultCamelContext();
// add our route to the CamelContext
context.addRoutes(new RouteBuilder() {
public void configure() {
from("file:data/inbox?noop=true").to("file:data/outbox");
}
});
// start the route and let it do its work
context.start();
Thread.sleep(10000);
// stop the CamelContext
context.stop();
}
}](https://image.slidesharecdn.com/2011-04-apachecamel-131101081020-phpapp01/85/Apache-Camel-Stephane-Kay-April-2011-18-320.jpg)
Uploaded byJUG Lausanne
Apache Camel - Stéphane Kay - April 2011
Apache Camel is a powerful open source integration framework based on known enterprise integration patterns. It provides a simple Java domain specific language for defining routing and mediation logic. Camel includes over 70 components, 18 data formats, and support for many languages for expressions and predicates. It also includes tools for testing, transactions, concurrency, and exceptions handling. Camel's architecture uses services like routes, endpoints, components and data formats that are accessible through its container.
More Related Content
Similar to Apache Camel - Stéphane Kay - April 2011
Apache Camel - Stéphane Kay - April 2011
- 1. Apache Camel Stephane Kay Villede Lausanne
- 2. Scope IS Integration EIP Routing Mediation Camel Container Unit testing Hands-on Real Use Case JMS / Active MQ JBI / NMR OSGI / ServiceMix Scala
- 3. Table of contents ISIntegration heuristics I. ◦ ◦ Common issues Paradigms Camel’s bleat II. ◦ ◦ Concepts responding to integration needs Getting started What’s under the humps? III. ◦ ◦ Camel architecture Distribution Camel in Action IV. ◦ ◦ Persistence, transactions, concurrency, failover Aggregation A Real Use-Case V. ◦ Description, issues & achievement
- 4. I. IS IntegrationHeuristics
- 5. I. IS Integrationheuristics Just do it ! Just coding simple things might be just not so simple … public class FileCopier { public static void main(String args[]) throws Exception { File inboxDirectory = new File("data/inbox"); File outboxDirectory = new File("data/outbox"); outboxDirectory.mkdir(); File[] files = inboxDirectory.listFiles(); for (File source : files) { if (source.isFile()) { File dest = new File( outboxDirectory.getPath() + File.separator + source.getName()); copyFile(source, dest); } } } private static void copyFile(File source, File dest) throws IOException { OutputStream out = new FileOutputStream(dest); byte[] buffer = new byte[(int) source.length()]; FileInputStream in = new FileInputStream(source); in.read(buffer); try { out.write(buffer); } finally { out.close(); in.close(); } } }
- 6. I. IS Integrationheuristics A Framework? The bad : ◦ Learning effort consumes time. ◦ Adds anarchical dependencies. ◦ Risk of coming with a new need / not implemented The good : ◦ Gain value after a training period ◦ Leverage your code to a higher abstraction ◦ Potentially a future standard ? Indicator : community commits & posts curve. The plus-value ◦ Not obvious
- 7. I. IS Integrationheuristics Separation of concerns IS Integration involves 1. Endpoints A common endpoint interface abstraction 2. Data Processing Using Message / exchange abstraction and conversion tool 3. Routing A common vocabulary / pattern as DSL foundation.
- 8. I. IS Integration| endpoint Message Endpoint pattern Routes should refer to a common abstract Message Endpoint rather than directly using their technical interface A good Endpoint abstraction is a URI description
- 9. II. Camel’s bleat| data Uniformed Exchange Message’s container during routing
- 10. I. IS Integration| data Data Transformation Data Format Transformation ◦ Message body is transformed from one format to another. CSV -> XML Data Type Transformation ◦ Java Type of message body is transformed/casted into another. java.lang.String -> java.jms.TextMessage java.io.File -> java.lang.String
- 11. I. IS Integration| routing Use case : Filtering Route ◦ How can a component avoid receiving uninteresting messages ? ◦ …coming from a different system
- 12. I. IS Integration| routing EIP? A Catalog of Design patterns for common IS integration issues http://camel.apache.org/enterprise-integration-patterns.html
- 13. I. IS Integration| routing EIPs Samples… Filter ◦ Splitter ◦ How can a component avoid receiving uninteresting messages? How can we process a message if it contains multiple elements, each of which may have to be processed in a different way? Router ◦ How can you decouple individual processing steps so that messages can be passed to different filters depending on a set of conditions? Aggregator ◦ How do we combine the results of individual, but related messages so that they can be processed as a whole?
- 14. I. IS Integrationheuristics Putting all needs together Route definition attributes : ◦ Endpoints Producer / Consumer(s) ◦ Processor(s) Uniformed exchange processing interface ◦ Predicate(s) A route language should provide those key abstractions Endpoint A = endpoint(“activemq:queue:quote”); Endpoint B = endpoint(“mq:quote”); Predicate isWidget = xpath(“/quote/product = „widget‟”); from(A).filter(isWidget).to(B);
- 15. I. IS Integrationheuristics I. Review IS Integration paradigms ◦ Endpoint abstraction -> URIs? ◦ Data Tx -> Functions // Java beans ? ◦ Routing -> EIPs
- 16. II. Camel’s bleat ApacheCamel is a Powerful Open Source Integration Framework based on known Enterprise Integration Patterns * Mission Statement : Making integration easier and more accessible to developers
- 17. II. Camel’s bleat| endpoint Endpoint URI A camel endpoint URI consists of three parts : ftp://riders.com/orders?username=rider&password=secret Scheme Context path Options
- 18. II. Camel’s bleat| routing Java DSL : Builder pattern The Builder Pattern (GOF) is central to the Camel Java DSL public class FileCopierWithCamel { public static void main(String args[]) throws Exception { // create CamelContext CamelContext context = new DefaultCamelContext(); // add our route to the CamelContext context.addRoutes(new RouteBuilder() { public void configure() { from("file:data/inbox?noop=true").to("file:data/outbox"); } }); // start the route and let it do its work context.start(); Thread.sleep(10000); // stop the CamelContext context.stop(); } }
- 19. II. Camel’s bleat| data Data Format transformations Marshal / unmarshal from("activemq:QueueWithJavaObjects”) .marshal().jaxb() .to("mq:QueueWithXmlMessages"); from("activemq:queue:MY_QUEUE").unmarshal().zip().proce ss(new UnZippedMessageProcessor()); http://camel.apache.org/data-format.html
- 20. II. Camel’s bleat| data Message Translator EIP Aka Adapter (GOF) Using a Processor from("direct:start").process(new Processor() { public void process(Exchange exchange) { exchange.getIn().setBody(in.getBody(String.class) + " World!"); } }).to("mock:result"); Using Bean binding from("direct:start") .bean(MyBean.class, "hello(String,String)") .to("mock:result"); Using transform() DSL from("direct:start").transform(body().regexReplaceAll(“n”,”<br/>”)) .to("mock:result");
- 21. II. Camel’s bleat| first ride Getting started : IDE Eclipse Maven integration (m2eclipse) ◦ http://m2eclipse.sonatype.org/sites/m2e ◦ Check your eclipse.ini : locate your JDK -vm C:/dev/java/jdk1.6.0_18/bin Optional : Fuse Camel plugin for Eclipse ◦ http://repo.fusesource.com/beta/eclipse/u pdate/
- 22. II. Camel’s bleat| first ride Getting started : Maven Basic Maven configuration
- 23. II. Camel’s bleat| first ride First Ride Copy contents from one folder to another, control final name / dir JUnit ◦ ◦ ◦ ◦ Startup context Route definition : File component Processor / Exchange Simple Language (File Component) Delegate JUnit test to Main class
- 24. II. Camel’s bleat| first ride Second Ride Add some features to JUnit Class ◦ ◦ ◦ ◦ Log Add failure if file already exist Predicate : Simple Language Error handling Router EIP (Choice)
- 25. II. Camel’s bleat| spring Spring XML : content-based router <camelContext> <route> <from uri="activemq:NewOrders"/> <choice> <when> <xpath>/order/product = 'widget'</xpath> <to uri="activemq:Orders.Widgets"/> </when> <otherwise> <to uri="activemq:Orders.Gadgets"/> </otherwise> </choice> </route> </camelContext>
- 26. II. Camel’s bleat II.Review Endpoint / Component URIs Java DSL How to start / process exchanges Log, error handling / routing
- 27. III. What’s underthe humps? High-level architecture ◦ Container & Services Toolkit provided out-of-the box Distribution
- 28. III. What’s underthe humps? High-level view Container responsibilities : ◦ Create and activate the necessary Endpoint instances using the available Component implementations
- 29. III. What’s underthe humps? Camel Context Services Components ◦ Endpoints ◦ Contains the loaded data formats. Registry ◦ Contains the loaded type converters. Camel has a mechanism that allows you to manually or automatically convert from one type to another. Data formats ◦ Contains the routes that have been added. Type converters ◦ Contains the endpoints that have been created. Routes ◦ Loads components on the fly either by autodiscovery on the classpath or when a new bundle is activated in an OSGi container. Contains a registry that allows you to look up beans. By default, this will be a JNDI registry. If you’re using Camel from Spring, this will be the Spring ApplicationContext. It can also be an OSGi registry if you use Camel in an OSGi container. Languages ◦ Contains the loaded languages. Camel defines 3 different DSLs (Java, XML/Spring, and Scala) and allows you to use many different languages to create expressions.
- 30. III. What’s underthe humps? Out of the box : EIPs 50+ Routing Patterns based on EIPs http://camel.apache.org/enterprise-integration-patterns.html
- 31. III. What’s underthe humps? Out of the box : Components 70+ Endpoint impl. aka « Components » All available via URI/scheme paradigm http://camel.apache.org/components.html
- 32. III. What’s underthe humps? Out of the box : Data Format 18 Data Formats - uniformed usage : from(“file://input").marshal().zip().to("activemq:queue:MY_QUEUE"); http://camel.apache.org/data-format.html
- 33. III. What’s underthe humps? Out of the box : Languages Predicates & Expressions http://camel.apache.org/languages.html
- 34. Camel distribution http://camel.apache.org/download.html Components sources Examples Maven Central (current v2.7) ◦ Snapshot Repo <repository> <id>apache.snapshots</id> <name>Apache Development Snapshot Repository</name> <url>https://repository.apache.org/content/repositories/snapshots/</url> <releases> <enabled>false</enabled> </releases> <snapshots> <enabled>true</enabled> </snapshots> </repository>
- 35. III. Review Camel isan integration framework Based on Enterprise Integration Patterns Routing and mediation Easy to use DSL to define routes No heavy specification No container dependency Payload agnostic Connectivity to a great wealth of transports
- 36. IV. Camel inAction Test support Transactions Exceptions Concurrency Persistence Aggregation Runtime
- 37. IV. Camel inAction| test Camel Test Support Provides context/registry support MockEndpoint assertions resultEndpoint.expectedBodiesReceived(expectedBody); template.sendBodyAndHeader(expectedBody, "foo", "bar"); resultEndpoint.assertIsSatisfied();
- 38. IV. Camel inAction| test Using test support CamelTestSupport Junit @EndpointInject @Produce Filter Pattern
- 39. IV. Camel inAction | tx Transactions Component level ◦ Polling strategy public interface PollingConsumerPollStrategy { boolean begin(Consumer consumer, Endpoint endpoint); void commit(Consumer consumer, Endpoint endpoint, int polledMessages); boolean rollback(Consumer consumer, Endpoint endpoint, int retryCounter, Exception cause) throws Exception; } Routing level ◦ Transactional Client EIP http://camel.apache.org/transactional-client.html ◦ Transacted DSL from("jetty:http://localhost/myservice/order").transacted() .to("bean:validateOrder").to("jms:queue:order");
- 40. IV. Camel inAction | tx Propagation Define transaction behaviour <bean id="PROPAGATION_REQUIRED" class="org.apache.camel.spring.spi.SpringTransactionPoli cy"> <property name="transactionManager" ref="txManager"/> <property name="propagationBehaviorName" value="PROPAGATION_REQUIRED"/> </bean> <bean id="PROPAGATION_REQUIRES_NEW" class="org.apache.camel.spring.spi.SpringTransactionPolicy"> <property name="transactionManager" ref="txManager"/> <property name="propagationBehaviorName" value="PROPAGATION_REQUIRES_NEW"/> </bean> Use transacted DSL from("direct:mixed") .transacted("PROPAGATION_REQUIRED“) .setBody(constant("Tiger in Action")) .beanRef("bookService") .setBody(constant("Elephant in Action")) .beanRef("bookService“).to("direct:mixed2");
- 41. IV. Camel inAction | errors Exceptions onException clause new RouteBuilder() { public void configure() { onException(IOException.class).maximumRedeliveries(3); from("direct:start") .process("processor1") .process("processor2") // <--- throws a ConnectException .to("mock:theEnd"); } ◦ Note : By default, a producer will not redeliver Handle onException(ValidationException.class) .handled(true).transform(body(constant("INVALID ORDER"))); from("jetty:http://localhost/myservice/order").transacted() .to("bean:validateOrder").to("jms:queue:order");
- 42. IV. Camel inAction | errors Dead Letter EIP The Error Handler propagates error back to the client. Dead Letter Channel will always moved failed exchanges to this queue. useOriginalMessage is used for routing the original input message instead of the current message that potentially is modified during routing. from("jms:queue:order:input").to("bean:validateOrder") .to("bean:transformOrder").to("bean:handleOrder"); errorHandler(deadLetterChannel("jms:queue:dead") .useOriginalMessage().mamimumRedeliveries(5) .redeliverDelay(5000);
- 43. IV. Camel inAction | toolkit Concurency Dealing with data flows means mastering concurency ◦ Seda Component : from("seda:bar?concurrentConsumers=5").to("file://output"); ◦ Thread DSL How do we avoid the messages that does not expect a reply to block ? from("jms:queue:order") // do some sanity check validation .to("bean:validateOrder") .to("mock:validate").threads(20) // do some CPU heavy processing of the message (we simulate and delay) .unmarshal(mySecureDataFormat).delay(1000) .to("bean:handleOrder").to("mock:order");
- 44. IV. Camel inAction | toolkit Sync to Async toAsync Processor from("direct:start").to("mock:a").toAsync("direct:bar", 5).to("mock:result"); from("direct:bar").to("mock:b").transform(constant("Bye World")); How does this work? void process(Exchange exchange, AsyncCallback callback) throws Exception; ◦ With AsyncCallback interface : void onTaskCompleted(Exchange exchange); Http Async Producer public class JettyHttpProducer extends DefaultProducer implements AsyncProcessor ◦ Then invoke the callback when reply is ready on the Exchange. ◦ Routing with Spring DSL <route> <!-- we route from a direct endpoint --> <from uri="direct:start"/> <!-- log the request --> <to uri="log:+++ request +++"/> <!-- then doing a non blocking async request/reply to the http server with a pool of 10 threads --> <to uri="jetty://http://0.0.0.0:9123/myapp" async="true" poolSize="10"/> <!-- the reply from the server is logged --> <to uri="log:+++ reply +++"/> <!-- and to our mock so we can assert we got all responses --> <to ref="result"/> </route>
- 45. IV. Camel inAction | toolkit Persistence SQL, JDBC and JPA components provided out-of the box File idempotent consumer with pluggable repository from("file://inbox?idempotent=true").to("..."); <bean id="fileStore" class="org.apache.camel.processor.idempotent.FileIdempotentRepository"> <!-- the filename for the store --> <property name="fileStore" value="target/fileidempotent/.filestore.dat"/> <!-- the max filesize in bytes for the file. Camel will trunk and flush the cache if the file gets bigger --> <property name="maxFileStoreSize" value="512000"/> <!-- the number of elements in our store --> <property name="cacheSize" value="250"/> </bean> <camelContext xmlns="http://camel.apache.org/schema/spring"> <route> <from uri="file://target/fileidempotent/?idempotent=true&idempotentRepository= #fileStore&move=done/${file:name}"/> <to uri="mock:result"/> </route> Stateful EIP : Aggregator
- 46. IV. Camel inAction | toolkit Aggregator EIP Combine a number of messages together into a single message. ◦ http://camel.apache.org/aggregator2.html Completion predicates from("direct:start“). .aggregate(header("id"), new BodyInAggregatingStrategy()).completionTimeout(3000) .to("mock:aggregated"); from("direct:start“).aggregate(header("id"), new BodyInAggregatingStrategy()).completionSize(3) .to("mock:aggregated"); from("direct:start“).aggregate(header("id"), new BodyInAggregatingStrategy()) .eagerCheckCompletion().completionPredicate(body().isEqualTo("END")) .to("mock:aggregated"); from("direct:start“).aggregate(header("id"), new BodyInAggregatingStrategy()). completionSize(header("mySize")) .to("mock:aggregated");
- 47. IV. Camel inAction | toolkit Aggregator Strategy & Fail Over Strategy public class NumberAggregationStrategy implements AggregationStrategy { public Exchange aggregate(Exchange oldExchange, Exchange newExchange) { if (oldExchange == null) { return newExchange; } Integer num1 = oldExchange.getIn().getBody(Integer.class); Integer num2 = newExchange.getIn().getBody(Integer.class); Integer num3 = (num1 != null ? num1 : 0) + (num2 != null ? num2 : 0); oldExchange.getIn().setBody(num3); return oldExchange; } } http://camel.apache.org/aggregate-example.html
- 48. IV. Camel inAction | runtime Running Camel Known Deployment Options ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ Standalone Java Application Java Web Start Spring Application Web Application Google App Engine J2EE Application JBI OSGi
- 49. IV. Riding Camel| runtime Running inside App Server Servlet component camel-example-servlet-tomcat CXF Component camel-example-cxf-tomcat Camel Web console actually « Web Component »
- 50. IV. Review Camel isnot a server Camel is lightweight and embeddable Camel supports Enterprise Integration paradigms from the JEE/Spring world. Camel is not an ESB though it implements EIPs. Camel doesn’t need but supports any messaging infrastructure (JMS / MQ) Camel supports messaging at JBI abstraction level
- 51. V. A realUse-Case Multiple source FTP import to a centralized repository ◦ Collect files (0-5 x 2-20 Megs) from N different FTP endpoints, every hours on a 24/7 basis ◦ End-user configuration ◦ Logs : technical + business ◦ Monitor failures FTP SAN OPCON
- 52. Constraints Big binary files: maybe JMS would not fit Implementation should fit some common corporate approach to entreprise intergation. Currently only SAN @ client. Do not need App Server yet. Client is technical-oriented : would like hands-on. Keep intact current logs / traces, but must be possible to connect on JDBC. Would be interested in indexing meta –infos within Alfresco (JSR-170). Central schedule/alerting infrastructure must be connected. Nagios could be an option at mid-term.
- 53. Real issues Multiple sourceendpoints pattern : ◦ not found (!!!). FTP component is an active poller; no easy way to stop polling after having consuming all files. No easy way to put into configuration the context path of Endpoint URIs
- 54. Other applications… Camel asan ETL : Excel sources PeopleSoft to ActiveDirectory Bidirectional Federal <-> City Entreprises DB official data Reconcilliation of mutations streams
- 55. Conclusions Open-Source ◦ Help yourselfthrough debug mode on sources Maven-based development ◦ « Real » modularity Real Standards ◦ EIPs ◦ Maven, Spring, Java6. ◦ JBI (Glassfish) Gateway to the future ◦ OSGI (Fuse ESB) Camel. What else?
- 56. More information needed? Camelwebsite: ◦ http://camel.apache.org Camel article: ◦ http://architects.dzone.com/articles/apachecamel-integration FuseSource website: ◦ http://fusesource.com Camel in Action book: ◦ http://manning.com/ibsen Camel in Action Source code: ◦ http://code.google.com/p/camelinaction/
- 57.