Systems Integration in the NoSQL Era
with Apache Camel

Kai Wähner
kwaehner@talend.com
@KaiWaehner
www.kai-waehner.de
10/2...
Kai Wähner
Main Tasks
Requirements Engineering
Enterprise Architecture Management
Business Process Management
Architecture...
What is the problem?

Growth
• Applications
• Interfaces

• Technologies
• Products

© Talend 2013

“Systems Integration i...
A new era: NoSQL

© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
Solution: systems integration

All Roads lead
to Rome ...
© Talend 2013

“Systems Integration in the NoSQL Era with Apache...
Wishes for integrators

• Standardized Modeling
• Efficient Realization
• Automatic Testing
© Talend 2013

“Systems Integr...
Systems integration in the NoSQL era

© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähne...
What is the key message?

© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
Key messages

NoSQL cannot be avoided, and must be integrated!

NoSQL integration is already possible!
Apache Camel helps ...
Agenda

1) Introduction to NoSQL
2) Introduction to Apache Camel
3) Integration of a Document-oriented Database
4) Integra...
Agenda

1) Introduction to NoSQL
2) Introduction to Apache Camel
3) Integration of a Document-oriented Database
4) Integra...
The evolving database landscape

© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
Complementary Concepts

NoSQL

Cloud

Big Data
© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by ...
Agenda

1) Introduction to NoSQL
2) Introduction to Apache Camel
3) Integration of a Document-oriented Database
4) Integra...
Alternatives for systems integration

API

Integration
Framework

Enterprise Service Bus (ESB)

Complexity
of Integration
...
More details about integration frameworks...

http://www.kai-waehner.de/blog/2013/09/25/how-to-choose-the-right-open-sourc...
Enterprise Integration Patterns (EIP)

Apache Camel
Implements the EIPs
© Talend 2013

“Systems Integration in the NoSQL E...
Enterprise Integration Patterns (EIP)

© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähn...
Enterprise Integration Patterns (EIP)

© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähn...
Architecture

http://java.dzone.com/articles/apache-camel-integration

© Talend 2013

“Systems Integration in the NoSQL Er...
Choose your favorite DSL

XML

(not production-ready yet)

© Talend 2013

“Systems Integration in the NoSQL Era with Apach...
Choose your required components
TCP

SQL

SMTP

Netty
RMI

FTP

Lucene

JMX

MQ

Atom
LDAP

Quartz

XSLT

Akka
Many many m...
Deploy it wherever you need

Standalone

Application Server

Web Container
Spring Container
OSGi
Cloud
© Talend 2013

“Sys...
Enterprise-ready

• Open Source
• Scalability
• Error Handling
• Transaction
• Monitoring
• Tooling
• Commercial Support
©...
Community  Camel rocks!

Mailing Lists?
Forums?
Blogs?
Articles?
Conference talks?
ESBs?
Professionals?
Jobs?
Knowledge?
...
Live demo

Apache Camel in action...
© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
Agenda

1) Introduction to NoSQL
2) Introduction to Apache Camel
3) Integration of a Document-oriented Database
4) Integra...
Document-oriented database

© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
Document-oriented database

• 10gen
• stores structured data as JSON-like
documents with dynamic schemas
• REST API and se...
Code example: MongoDB Java Driver

© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
Code example: camel-mongodb component
// Producer
from(“jms:FlightDocumentQueue")
.to("mongodb:myDb?database=flights
&coll...
Live demo

Integration of a document-oriented database in action...
© Talend 2013

“Systems Integration in the NoSQL Era w...
Agenda

1) Introduction to NoSQL
2) Introduction to Apache Camel
3) Integration of a Document-oriented Database
4) Integra...
Key-Value database

© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
Key-Value database

• Part of Amazon Web Services (AWS)
• Online storage web service
• Store arbitrary objects (computer f...
Code example: AWS S3 Java SDK
AmazonS3 s3 = new AmazonS3Client(new PropertiesCredentials(
S3Sample.class.getResourceAsStre...
Code example: camel-aws component

// Producer
from(„jms:toS3Queue")
.setHeader(S3Constants.KEY, simple(“order.txt"))
.to(...
Open source tooling on top of Camel: Talend ESB
Development

Runtime

Operation
Management

Mediation &
Integration

Media...
Open source tooling on top of Camel: Talend ESB
Development

Runtime

Operation

Security

Route Designer

Apache Camel

S...
Open source tooling on top of Camel: Talend ESB
Route Builder





Endpoints
EIPs
Processors
Custom components

Config...
Live demo

Integration of a key-value database in action...
© Talend 2013

“Systems Integration in the NoSQL Era with Apac...
Alternative to Vendor APIs

➜

© Talend 2013

Generic APIs

“Systems Integration in the NoSQL Era with Apache Camel” by Ka...
jClouds (Generic API)
Generic API for IaaS

© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by Kai...
jClouds (Generic API)

Compute API
Blobstore API
© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” b...
jClouds (Generic API)

Several different
Cloud providers
supported
© Talend 2013

“Systems Integration in the NoSQL Era wi...
jClouds (Generic API) – AWS S3 Blobstore (Java)

Use another provider?
Just change this line!

© Talend 2013

“Systems Int...
Code example: camel-jclouds component

from("direct:toJcloudsAwsS3")
.setHeader(JcloudsConstants.BLOB_NAME, "jclouds-demo-...
Agenda

1) Introduction to NoSQL
2) Introduction to Apache Camel
3) Integration of a Document-oriented Database
4) Integra...
In-memory database

© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
In-memory database

• In-memory data grid
• Clustering and highly scalable data
distribution solution for Java platform
• ...
Code example: Hazelcast Java API

© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
Code example: camel-hazelcast component

// Producer
from("direct:add")
.setHeader(HazelcastConstants.OPERATION, „add“)
.t...
Live demo

Integration of an in-memory database in action...
© Talend 2013

“Systems Integration in the NoSQL Era with Apa...
Agenda

1) Introduction to NoSQL
2) Introduction to Apache Camel
3) Integration of a Document-oriented Database
4) Integra...
Graph-oriented database

© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
Graph-oriented database

• Neo Technology
• Graphs rather than tables
• Nodes, edges, and properties to represent
and stor...
Code example: Neo4j Ruby API

© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
Code example: camel-neo4j component

// Producer
from("jms:createNewNeo4jNode")
.to("neo4j:http://Neo4jServer:7474/data");...
Agenda

1) Introduction to NoSQL
2) Introduction to Apache Camel
3) Integration of a Document-oriented Database
4) Integra...
Column-oriented database

© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
HBase

• Modeled after Google's BigTable
• Runs on top of HDFS (Hadoop Distributed
Filesystem)
• Can serve as the input an...
Code example: HBase Java API

© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
Code example: camel-hbase component

© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
Agenda

1) Introduction to NoSQL
2) Introduction to Apache Camel
3) Integration of a Document-oriented Database
4) Integra...
Custom NoSQL components

Component

Creates

Endpoint

Consumer

© Talend 2013

“Systems Integration in the NoSQL Era with...
Live demo

Custom NoSQL components in action...
© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by...
Alternative for custom NoSQL components

• SOAP
• REST
© Talend 2013

“Systems Integration in the NoSQL Era with Apache Ca...
Code example: REST API for Salesforce object store
// Salesforce Query (SOQL) via REST API

from("direct:salesforceViaHttp...
SQL is still very alive, of course...

Camel SQL components:
• sql:statement
• jdbc:dataSourceName
• jpa://entityName
• my...
Did you get the key message?

© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
Key messages

NoSQL cannot be avoided, and must be integrated!

NoSQL integration is already possible!
Apache Camel helps ...
Did you get the key message?

© Talend 2013

“Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
Thank you for your attention. Questions?

kwaehner@talend.com
www.kai-waehner.de
LinkedIn / Xing
@KaiWaehner
Upcoming SlideShare
Loading in …5
×

JBoss OneDayTalk 2013: "NoSQL Integration with Apache Camel - MongoDB, CouchDB, Neo4j, HBase, Cassandra, AWS S3, Hazelcast, etc."

2,847
-1

Published on

SQL cannot solve several problems emerging with big data. A distributed, fault-tolerant architecture is necessary. NoSQL comes to the rescue, but therefore it does not use SQL as its query language or give full ACID guarantees. Thus, in the future you will have to learn new concepts and integrate these NoSQL databases as you integrate SQL databasestoday. The open source integration framework Apache Camel is already prepared for this challenging task.

Apache Camel implements the well-known Enteprise Integration Patterns (EIP) and therefore offers a standardized, domain-specific language to integrate applications and clouds. It can be used in almost every integration project within the JVM environment. All integration projects can be realized in a consistent way without redundant boilerplate code.

This session demonstrates the elegance of Apache Camel for NoSQL integration. Several examples are shown for all different concepts by integrating NoSQL databases from CouchDB (Document Store), HBase (Column-oriented), Neo4j (Graph), Amazon Web Services (Key Value Store), and others.

If the required NoSQL database is not supported by Apache Camel, you can easily create your own Camel component with very low effort. This procedure is explained at the end of the session.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,847
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

JBoss OneDayTalk 2013: "NoSQL Integration with Apache Camel - MongoDB, CouchDB, Neo4j, HBase, Cassandra, AWS S3, Hazelcast, etc."

  1. 1. Systems Integration in the NoSQL Era with Apache Camel Kai Wähner kwaehner@talend.com @KaiWaehner www.kai-waehner.de 10/23/2013 © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  2. 2. Kai Wähner Main Tasks Requirements Engineering Enterprise Architecture Management Business Process Management Architecture and Development of Applications Service-oriented Architecture Integration of Legacy Applications Cloud Computing Big Data Consulting Developing Coaching Speaking Writing Contact Email: kwaehner@talend.com Blog: www.kai-waehner.de/blog Twitter: @KaiWaehner Social Networks: Xing, LinkedIn © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  3. 3. What is the problem? Growth • Applications • Interfaces • Technologies • Products © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  4. 4. A new era: NoSQL © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  5. 5. Solution: systems integration All Roads lead to Rome ... © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  6. 6. Wishes for integrators • Standardized Modeling • Efficient Realization • Automatic Testing © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  7. 7. Systems integration in the NoSQL era © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  8. 8. What is the key message? © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  9. 9. Key messages NoSQL cannot be avoided, and must be integrated! NoSQL integration is already possible! Apache Camel helps a lot! © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  10. 10. Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Document-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Graph-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components Live Demos © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  11. 11. Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Document-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Graph-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  12. 12. The evolving database landscape © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  13. 13. Complementary Concepts NoSQL Cloud Big Data © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  14. 14. Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Document-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Graph-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  15. 15. Alternatives for systems integration API Integration Framework Enterprise Service Bus (ESB) Complexity of Integration Low High Apache Camel vs. Spring Integration vs. Mule http://www.kai-waehner.de/blog/2012/01/10/spoilt-for-choice-which-integration-framework-to-use-spring-integration-mule-esb-or-apache-camel/ © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  16. 16. More details about integration frameworks... http://www.kai-waehner.de/blog/2013/09/25/how-to-choose-the-right-open-sourceintegration-framework-apache-camel-jboss-talend-spring-integration-pivotal-or-muleesb-javaone-2013/ © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  17. 17. Enterprise Integration Patterns (EIP) Apache Camel Implements the EIPs © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  18. 18. Enterprise Integration Patterns (EIP) © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  19. 19. Enterprise Integration Patterns (EIP) © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  20. 20. Architecture http://java.dzone.com/articles/apache-camel-integration © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  21. 21. Choose your favorite DSL XML (not production-ready yet) © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  22. 22. Choose your required components TCP SQL SMTP Netty RMI FTP Lucene JMX MQ Atom LDAP Quartz XSLT Akka Many many more © Talend 2013 IRC Log HTTP File EJB AMQP RSS AWS-S3 Jetty JDBC Bean-Validation JMS CXF “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner Custom Components
  23. 23. Deploy it wherever you need Standalone Application Server Web Container Spring Container OSGi Cloud © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  24. 24. Enterprise-ready • Open Source • Scalability • Error Handling • Transaction • Monitoring • Tooling • Commercial Support © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  25. 25. Community  Camel rocks! Mailing Lists? Forums? Blogs? Articles? Conference talks? ESBs? Professionals? Jobs? Knowledge? © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  26. 26. Live demo Apache Camel in action... © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  27. 27. Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Document-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Graph-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  28. 28. Document-oriented database © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  29. 29. Document-oriented database • 10gen • stores structured data as JSON-like documents with dynamic schemas • REST API and several SDKs (Java, .NET, Ruby, PHP, Python, etc.) • Ad hoc queries, indexing, replication, load balancing • Powerful, but also easy to use and flexible • Example: Disney persists state information of online games in a common object repository. © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  30. 30. Code example: MongoDB Java Driver © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  31. 31. Code example: camel-mongodb component // Producer from(“jms:FlightDocumentQueue") .to("mongodb:myDb?database=flights &collection=tickets &operation=insert"); // Consumer from("mongodb:myDb?database=flights &collection=cancellations &tailTrackIncreasingField=departureTime") .to(“jms:CancelledFlightsQueue"); © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  32. 32. Live demo Integration of a document-oriented database in action... © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  33. 33. Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Document-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Graph-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  34. 34. Key-Value database © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  35. 35. Key-Value database • Part of Amazon Web Services (AWS) • Online storage web service • Store arbitrary objects (computer files) up to 5 terabytes • REST and SOAP API • SDKs for Java, .NET, PHP, Ruby, etc. • Highly-scalable, reliable, and low-latency • Alternative for Hadoop‘s file system HDFS • Example: DigitalChalk offers creating, delivering and managing training videos © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  36. 36. Code example: AWS S3 Java SDK AmazonS3 s3 = new AmazonS3Client(new PropertiesCredentials( S3Sample.class.getResourceAsStream("AwsCredentials.properties"))); String bucketName = "my-first-s3-bucket-" + UUID.randomUUID(); String key = "MyObjectKey"; try { s3.createBucket(bucketName); s3.putObject(new PutObjectRequest(bucketName, key, createSampleFile())); S3Object object = s3.getObject(new GetObjectRequest(bucketName, key)); ObjectListing objectListing = s3.listObjects(new ListObjectsRequest() .withBucketName(bucketName) .withPrefix("My")); s3.deleteObject(bucketName, key); s3.deleteBucket(bucketName); } catch (AmazonServiceException ase) { // error handling... } catch (AmazonClientException ace) { // error handling... } © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  37. 37. Code example: camel-aws component // Producer from(„jms:toS3Queue") .setHeader(S3Constants.KEY, simple(“order.txt")) .to("aws-s3://myBucket?accessKey=" + a + "&secretKey= " + s) // Consumer from("aws-s3://myBucket?accessKey=“ + a + "&secretKey=" + s) .to("log:S3logging") © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  38. 38. Open source tooling on top of Camel: Talend ESB Development Runtime Operation Management Mediation & Integration Mediation & Integration Loadbalancing & High Availability Configuration Message Broker Business Rules Project Repository Service Container Deployment Repository Testing Build & Deploy Performance & Availability Talend Administration Center Security Talend ESB Web Services Stack Talend ESB Studio Service Development Documentation & Examples 24x7 Support Training & Certification Maintenance Professional Services © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner Indemnification Certified Partners
  39. 39. Open source tooling on top of Camel: Talend ESB Development Runtime Operation Security Route Designer Apache Camel Service Locator & Service Act. Monitoring Mediation Service Designer Integration Apache Maven Build & Deploy Mediation Distributed Registry / Tracking Message Broker Apache Archiva Artifact repository Apache Karaf / Cellar Eclipse Equinox OSGi / Clustering OSGi Apache ActiveMQ Service Activity Monitoring and Service Locator UI Repository Metadata & Projects vFabric Hyperic HQ Talend Administration Center REST & Web Services Management, Configuration & Monitoring Secure Token Server Talend ESB Runtime Apache CXF Talend ESB Studio Eclipse STP/WTP soapUI Documentation & Examples 24x7 Support Training & Certification Maintenance Professional Services © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner Indemnification Certified Partners
  40. 40. Open source tooling on top of Camel: Talend ESB Route Builder     Endpoints EIPs Processors Custom components Configuration  Components  Endpoints Code Generation  100% Java  Camel Code  Packaged as OSGi Bundles Execution in the IDE  Debugging  Live statistics  Short dev cycles © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  41. 41. Live demo Integration of a key-value database in action... © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  42. 42. Alternative to Vendor APIs ➜ © Talend 2013 Generic APIs “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  43. 43. jClouds (Generic API) Generic API for IaaS © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  44. 44. jClouds (Generic API) Compute API Blobstore API © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  45. 45. jClouds (Generic API) Several different Cloud providers supported © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  46. 46. jClouds (Generic API) – AWS S3 Blobstore (Java) Use another provider? Just change this line! © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  47. 47. Code example: camel-jclouds component from("direct:toJcloudsAwsS3") .setHeader(JcloudsConstants.BLOB_NAME, "jclouds-demo-tutorial.txt") .setHeader(JcloudsConstants.CONTAINER_NAME, "kw-s3-data") .to("jclouds:blobstore:aws-s3") from("direct:toJcloudsMicrosoftAzure") .setHeader(JcloudsConstants.BLOB_NAME, "jclouds-demo-tutorial.txt") .setHeader(JcloudsConstants.CONTAINER_NAME, "kw-s3-data") .to("jclouds:blobstore:azureblob") © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  48. 48. Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Document-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Graph-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  49. 49. In-memory database © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  50. 50. In-memory database • In-memory data grid • Clustering and highly scalable data distribution solution for Java platform • Architecture is peer-to-peer • Distributed Java data structures (Queue, Set, List, Map, Lock, Topic) • Java and REST API © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  51. 51. Code example: Hazelcast Java API © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  52. 52. Code example: camel-hazelcast component // Producer from("direct:add") .setHeader(HazelcastConstants.OPERATION, „add“) .to("hazelcast:queue:foo"); // Consumer from("hazelcast:queue:foo") .log(“content of object foo: ${body}"); © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  53. 53. Live demo Integration of an in-memory database in action... © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  54. 54. Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Document-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Graph-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  55. 55. Graph-oriented database © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  56. 56. Graph-oriented database • Neo Technology • Graphs rather than tables • Nodes, edges, and properties to represent and store data • Index-free adjacency • REST API and many SDKs (Java, .NET, Ruby, PHP, Python, etc.) • Embedded, disk-based, fully transactional • Powerful tool for graph-like queries • Example: Facebook friends © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  57. 57. Code example: Neo4j Ruby API © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  58. 58. Code example: camel-neo4j component // Producer from("jms:createNewNeo4jNode") .to("neo4j:http://Neo4jServer:7474/data"); // Consumer from(„neo4j://todo)... Not implemented in current Camel release (2.12)   Use Camel‘s REST components (shown in some minutes...) © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  59. 59. Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Document-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Graph-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  60. 60. Column-oriented database © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  61. 61. HBase • Modeled after Google's BigTable • Runs on top of HDFS (Hadoop Distributed Filesystem) • Can serve as the input and output for MapReduce jobs run in Hadoop • Stores data tables as sections of columns of data rather than as rows of data • Java API plus REST, Avro or Thrift gateway APIs • Use HBase when you need random, realtime read/write access to your Big Data • Example: Advantages for DWHs, CRMs, and other ad-hoc inquiry systems where aggregates are computed over large numbers of similar data items. © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  62. 62. Code example: HBase Java API © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  63. 63. Code example: camel-hbase component © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  64. 64. Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Document-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Graph-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  65. 65. Custom NoSQL components Component Creates Endpoint Consumer © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner Producer
  66. 66. Live demo Custom NoSQL components in action... © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  67. 67. Alternative for custom NoSQL components • SOAP • REST © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  68. 68. Code example: REST API for Salesforce object store // Salesforce Query (SOQL) via REST API from("direct:salesforceViaHttpLIST") .setHeader("X-PrettyPrint", 1) .setHeader("Authorization", accessToken) .setHeader(Exchange.CONTENT_TYPE, "application/json") .to("https://na14.salesforce.com/services/data/v20.0/query?q=SELECT+name+from +Article__c") // Salesforce CREATE via REST API from("direct:salesforceViaHttpCREATE") .setHeader("X-PrettyPrint", 1) .setHeader("Authorization", accessToken) .setHeader(Exchange.CONTENT_TYPE, "application/json“) .to("https://na14.salesforce.com/services/data/v20.0/sobjects/Article__c") © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  69. 69. SQL is still very alive, of course... Camel SQL components: • sql:statement • jdbc:dataSourceName • jpa://entityName • mybatis://statementName • hibernate://entityName © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  70. 70. Did you get the key message? © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  71. 71. Key messages NoSQL cannot be avoided, and must be integrated! NoSQL integration is already possible! Apache Camel helps a lot! © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  72. 72. Did you get the key message? © Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  73. 73. Thank you for your attention. Questions? kwaehner@talend.com www.kai-waehner.de LinkedIn / Xing @KaiWaehner

×