• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
JBoss OneDayTalk 2013: "NoSQL Integration with Apache Camel - MongoDB, CouchDB, Neo4j, HBase, Cassandra, AWS S3, Hazelcast, etc."
 

JBoss OneDayTalk 2013: "NoSQL Integration with Apache Camel - MongoDB, CouchDB, Neo4j, HBase, Cassandra, AWS S3, Hazelcast, etc."

on

  • 2,013 views

SQL cannot solve several problems emerging with big data. A distributed, fault-tolerant architecture is necessary. NoSQL comes to the rescue, but therefore it does not use SQL as its query language or ...

SQL cannot solve several problems emerging with big data. A distributed, fault-tolerant architecture is necessary. NoSQL comes to the rescue, but therefore it does not use SQL as its query language or give full ACID guarantees. Thus, in the future you will have to learn new concepts and integrate these NoSQL databases as you integrate SQL databasestoday. The open source integration framework Apache Camel is already prepared for this challenging task.

Apache Camel implements the well-known Enteprise Integration Patterns (EIP) and therefore offers a standardized, domain-specific language to integrate applications and clouds. It can be used in almost every integration project within the JVM environment. All integration projects can be realized in a consistent way without redundant boilerplate code.

This session demonstrates the elegance of Apache Camel for NoSQL integration. Several examples are shown for all different concepts by integrating NoSQL databases from CouchDB (Document Store), HBase (Column-oriented), Neo4j (Graph), Amazon Web Services (Key Value Store), and others.

If the required NoSQL database is not supported by Apache Camel, you can easily create your own Camel component with very low effort. This procedure is explained at the end of the session.

Statistics

Views

Total Views
2,013
Views on SlideShare
1,722
Embed Views
291

Actions

Likes
0
Downloads
0
Comments
0

2 Embeds 291

http://www.kai-waehner.de 290
https://www.rebelmouse.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    JBoss OneDayTalk 2013: "NoSQL Integration with Apache Camel - MongoDB, CouchDB, Neo4j, HBase, Cassandra, AWS S3, Hazelcast, etc." JBoss OneDayTalk 2013: "NoSQL Integration with Apache Camel - MongoDB, CouchDB, Neo4j, HBase, Cassandra, AWS S3, Hazelcast, etc." Presentation Transcript

    • Systems Integration in the NoSQL Era with Apache Camel Kai Wähner kwaehner@talend.com @KaiWaehner www.kai-waehner.de 10/23/2013 © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Kai Wähner Main Tasks Requirements Engineering Enterprise Architecture Management Business Process Management Architecture and Development of Applications Service-oriented Architecture Integration of Legacy Applications Cloud Computing Big Data Consulting Developing Coaching Speaking Writing Contact Email: kwaehner@talend.com Blog: www.kai-waehner.de/blog Twitter: @KaiWaehner Social Networks: Xing, LinkedIn © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • What is the problem? Growth • Applications • Interfaces • Technologies • Products © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • A new era: NoSQL © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Solution: systems integration All Roads lead to Rome ... © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Wishes for integrators • Standardized Modeling • Efficient Realization • Automatic Testing © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Systems integration in the NoSQL era © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • What is the key message? © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Key messages NoSQL cannot be avoided, and must be integrated! NoSQL integration is already possible! Apache Camel helps a lot! © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Document-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Graph-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components Live Demos © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Document-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Graph-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • The evolving database landscape © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Complementary Concepts NoSQL Cloud Big Data © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Document-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Graph-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Alternatives for systems integration API Integration Framework Enterprise Service Bus (ESB) Complexity of Integration Low High Apache Camel vs. Spring Integration vs. Mule http://www.kai-waehner.de/blog/2012/01/10/spoilt-for-choice-which-integration-framework-to-use-spring-integration-mule-esb-or-apache-camel/ © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • More details about integration frameworks... http://www.kai-waehner.de/blog/2013/09/25/how-to-choose-the-right-open-sourceintegration-framework-apache-camel-jboss-talend-spring-integration-pivotal-or-muleesb-javaone-2013/ © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Enterprise Integration Patterns (EIP) Apache Camel Implements the EIPs © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Enterprise Integration Patterns (EIP) © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Enterprise Integration Patterns (EIP) © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Architecture http://java.dzone.com/articles/apache-camel-integration © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Choose your favorite DSL XML (not production-ready yet) © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Choose your required components TCP SQL SMTP Netty RMI FTP Lucene JMX MQ Atom LDAP Quartz XSLT Akka Many many more © Talend 2013 IRC Log HTTP File EJB AMQP RSS AWS-S3 Jetty JDBC Bean-Validation JMS CXF “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner Custom Components
    • Deploy it wherever you need Standalone Application Server Web Container Spring Container OSGi Cloud © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Enterprise-ready • Open Source • Scalability • Error Handling • Transaction • Monitoring • Tooling • Commercial Support © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Community  Camel rocks! Mailing Lists? Forums? Blogs? Articles? Conference talks? ESBs? Professionals? Jobs? Knowledge? © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Live demo Apache Camel in action... © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Document-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Graph-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Document-oriented database © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Document-oriented database • 10gen • stores structured data as JSON-like documents with dynamic schemas • REST API and several SDKs (Java, .NET, Ruby, PHP, Python, etc.) • Ad hoc queries, indexing, replication, load balancing • Powerful, but also easy to use and flexible • Example: Disney persists state information of online games in a common object repository. © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Code example: MongoDB Java Driver © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Code example: camel-mongodb component // Producer from(“jms:FlightDocumentQueue") .to("mongodb:myDb?database=flights &collection=tickets &operation=insert"); // Consumer from("mongodb:myDb?database=flights &collection=cancellations &tailTrackIncreasingField=departureTime") .to(“jms:CancelledFlightsQueue"); © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Live demo Integration of a document-oriented database in action... © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Document-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Graph-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Key-Value database © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Key-Value database • Part of Amazon Web Services (AWS) • Online storage web service • Store arbitrary objects (computer files) up to 5 terabytes • REST and SOAP API • SDKs for Java, .NET, PHP, Ruby, etc. • Highly-scalable, reliable, and low-latency • Alternative for Hadoop‘s file system HDFS • Example: DigitalChalk offers creating, delivering and managing training videos © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Code example: AWS S3 Java SDK AmazonS3 s3 = new AmazonS3Client(new PropertiesCredentials( S3Sample.class.getResourceAsStream("AwsCredentials.properties"))); String bucketName = "my-first-s3-bucket-" + UUID.randomUUID(); String key = "MyObjectKey"; try { s3.createBucket(bucketName); s3.putObject(new PutObjectRequest(bucketName, key, createSampleFile())); S3Object object = s3.getObject(new GetObjectRequest(bucketName, key)); ObjectListing objectListing = s3.listObjects(new ListObjectsRequest() .withBucketName(bucketName) .withPrefix("My")); s3.deleteObject(bucketName, key); s3.deleteBucket(bucketName); } catch (AmazonServiceException ase) { // error handling... } catch (AmazonClientException ace) { // error handling... } © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Code example: camel-aws component // Producer from(„jms:toS3Queue") .setHeader(S3Constants.KEY, simple(“order.txt")) .to("aws-s3://myBucket?accessKey=" + a + "&secretKey= " + s) // Consumer from("aws-s3://myBucket?accessKey=“ + a + "&secretKey=" + s) .to("log:S3logging") © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Open source tooling on top of Camel: Talend ESB Development Runtime Operation Management Mediation & Integration Mediation & Integration Loadbalancing & High Availability Configuration Message Broker Business Rules Project Repository Service Container Deployment Repository Testing Build & Deploy Performance & Availability Talend Administration Center Security Talend ESB Web Services Stack Talend ESB Studio Service Development Documentation & Examples 24x7 Support Training & Certification Maintenance Professional Services © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner Indemnification Certified Partners
    • Open source tooling on top of Camel: Talend ESB Development Runtime Operation Security Route Designer Apache Camel Service Locator & Service Act. Monitoring Mediation Service Designer Integration Apache Maven Build & Deploy Mediation Distributed Registry / Tracking Message Broker Apache Archiva Artifact repository Apache Karaf / Cellar Eclipse Equinox OSGi / Clustering OSGi Apache ActiveMQ Service Activity Monitoring and Service Locator UI Repository Metadata & Projects vFabric Hyperic HQ Talend Administration Center REST & Web Services Management, Configuration & Monitoring Secure Token Server Talend ESB Runtime Apache CXF Talend ESB Studio Eclipse STP/WTP soapUI Documentation & Examples 24x7 Support Training & Certification Maintenance Professional Services © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner Indemnification Certified Partners
    • Open source tooling on top of Camel: Talend ESB Route Builder     Endpoints EIPs Processors Custom components Configuration  Components  Endpoints Code Generation  100% Java  Camel Code  Packaged as OSGi Bundles Execution in the IDE  Debugging  Live statistics  Short dev cycles © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Live demo Integration of a key-value database in action... © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Alternative to Vendor APIs ➜ © Talend 2013 Generic APIs “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • jClouds (Generic API) Generic API for IaaS © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • jClouds (Generic API) Compute API Blobstore API © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • jClouds (Generic API) Several different Cloud providers supported © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • jClouds (Generic API) – AWS S3 Blobstore (Java) Use another provider? Just change this line! © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Code example: camel-jclouds component from("direct:toJcloudsAwsS3") .setHeader(JcloudsConstants.BLOB_NAME, "jclouds-demo-tutorial.txt") .setHeader(JcloudsConstants.CONTAINER_NAME, "kw-s3-data") .to("jclouds:blobstore:aws-s3") from("direct:toJcloudsMicrosoftAzure") .setHeader(JcloudsConstants.BLOB_NAME, "jclouds-demo-tutorial.txt") .setHeader(JcloudsConstants.CONTAINER_NAME, "kw-s3-data") .to("jclouds:blobstore:azureblob") © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Document-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Graph-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • In-memory database © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • In-memory database • In-memory data grid • Clustering and highly scalable data distribution solution for Java platform • Architecture is peer-to-peer • Distributed Java data structures (Queue, Set, List, Map, Lock, Topic) • Java and REST API © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Code example: Hazelcast Java API © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Code example: camel-hazelcast component // Producer from("direct:add") .setHeader(HazelcastConstants.OPERATION, „add“) .to("hazelcast:queue:foo"); // Consumer from("hazelcast:queue:foo") .log(“content of object foo: ${body}"); © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Live demo Integration of an in-memory database in action... © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Document-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Graph-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Graph-oriented database © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Graph-oriented database • Neo Technology • Graphs rather than tables • Nodes, edges, and properties to represent and store data • Index-free adjacency • REST API and many SDKs (Java, .NET, Ruby, PHP, Python, etc.) • Embedded, disk-based, fully transactional • Powerful tool for graph-like queries • Example: Facebook friends © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Code example: Neo4j Ruby API © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Code example: camel-neo4j component // Producer from("jms:createNewNeo4jNode") .to("neo4j:http://Neo4jServer:7474/data"); // Consumer from(„neo4j://todo)... Not implemented in current Camel release (2.12)   Use Camel‘s REST components (shown in some minutes...) © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Document-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Graph-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Column-oriented database © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • HBase • Modeled after Google's BigTable • Runs on top of HDFS (Hadoop Distributed Filesystem) • Can serve as the input and output for MapReduce jobs run in Hadoop • Stores data tables as sections of columns of data rather than as rows of data • Java API plus REST, Avro or Thrift gateway APIs • Use HBase when you need random, realtime read/write access to your Big Data • Example: Advantages for DWHs, CRMs, and other ad-hoc inquiry systems where aggregates are computed over large numbers of similar data items. © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Code example: HBase Java API © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Code example: camel-hbase component © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Document-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Graph-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Custom NoSQL components Component Creates Endpoint Consumer © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner Producer
    • Live demo Custom NoSQL components in action... © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Alternative for custom NoSQL components • SOAP • REST © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Code example: REST API for Salesforce object store // Salesforce Query (SOQL) via REST API from("direct:salesforceViaHttpLIST") .setHeader("X-PrettyPrint", 1) .setHeader("Authorization", accessToken) .setHeader(Exchange.CONTENT_TYPE, "application/json") .to("https://na14.salesforce.com/services/data/v20.0/query?q=SELECT+name+from +Article__c") // Salesforce CREATE via REST API from("direct:salesforceViaHttpCREATE") .setHeader("X-PrettyPrint", 1) .setHeader("Authorization", accessToken) .setHeader(Exchange.CONTENT_TYPE, "application/json“) .to("https://na14.salesforce.com/services/data/v20.0/sobjects/Article__c") © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • SQL is still very alive, of course... Camel SQL components: • sql:statement • jdbc:dataSourceName • jpa://entityName • mybatis://statementName • hibernate://entityName © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Did you get the key message? © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Key messages NoSQL cannot be avoided, and must be integrated! NoSQL integration is already possible! Apache Camel helps a lot! © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Did you get the key message? © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
    • Thank you for your attention. Questions? kwaehner@talend.com www.kai-waehner.de LinkedIn / Xing @KaiWaehner