• Save
Systems Integration in the NoSQL Era with Apache Camel (Neo4j, CouchDB, AWS S3, Riak, MongoDB, HBase, Hazelcast)
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Systems Integration in the NoSQL Era with Apache Camel (Neo4j, CouchDB, AWS S3, Riak, MongoDB, HBase, Hazelcast)

on

  • 3,128 views

SQL cannot solve several problems emerging with big data. In the future you will have to integrate NoSQL databases, too. The open source integration framework Apache Camel is already prepared for this ...

SQL cannot solve several problems emerging with big data. In the future you will have to integrate NoSQL databases, too. The open source integration framework Apache Camel is already prepared for this challenging task. Several examples are shown for integrating NoSQL databases from CouchDB (Document Store), HBase (Column-oriented), Neo4j (Graph), Amazon Web Services (Key Value Store), and others.

Statistics

Views

Total Views
3,128
Views on SlideShare
2,686
Embed Views
442

Actions

Likes
7
Downloads
0
Comments
1

2 Embeds 442

http://www.kai-waehner.de 441
http://translate.googleusercontent.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Systems Integration in the NoSQL Era with Apache Camel (Neo4j, CouchDB, AWS S3, Riak, MongoDB, HBase, Hazelcast) Presentation Transcript

  • 1. Systems Integration in the NoSQL Era with Apache Camel Kai Wähner kwaehner@talend.com @KaiWaehner www.kai-waehner.de 2/28/2013© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 2. Kai Wähner Main Tasks Requirements Engineering Enterprise Architecture Management Business Process Management Architecture and Development of Applications Service-oriented Architecture Integration of Legacy Applications Cloud Computing Big Data Consulting Contact Developing Coaching Email: kwaehner@talend.com Speaking Blog: www.kai-waehner.de/blog Writing Twitter: @KaiWaehner Social Networks: Xing, LinkedIn© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 3. What is the problem? Growth • Applications • Interfaces • Technologies • Products© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 4. A new era: NoSQL© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 5. Solution: systems integration All Roads lead to Rome ...© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 6. Wishes for integrators • Standardized Modeling • Efficient Realization • Automatic Testing© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 7. Systems integration in the NoSQL era© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 8. What is the key message?© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 9. Key messages NoSQL cannot be avoided, and must be integrated! NoSQL integration is already possible! Apache Camel helps a lot!© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 10. Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Graph-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Document-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components Live Demos© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 11. Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Graph-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Document-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 12. The evolving database landscape© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 13. Complementary Concepts NoSQL Cloud Big Data© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 14. Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Graph-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Document-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 15. Alternatives for systems integration Enterprise Service Bus (ESB) Integration Framework API Complexity of Integration Low High Apache Camel vs. Spring Integration vs. Mule http://www.kai-waehner.de/blog/2012/01/10/spoilt-for-choice-which-integration-framework-to-use-spring-integration-mule-esb-or-apache-camel/© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 16. Enterprise Integration Patterns (EIP) Apache Camel Implements the EIPs© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 17. Enterprise Integration Patterns (EIP)© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 18. Enterprise Integration Patterns (EIP)© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 19. Architecture http://java.dzone.com/articles/apache-camel-integration© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 20. Choose your favorite DSL XML (not production-ready yet)© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 21. Choose your required components TCP SQL SMTP JMS Netty Jetty RMI FTP Lucene JDBC EJB Bean-Validation MQ IRC JMX Quartz RSS AMQP Atom Log AWS-S3 HTTP XSLT LDAP File Akka Many many more CXF Custom Components© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 22. Deploy it wherever you need Standalone Application Server Web Container Spring Container OSGi Cloud© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 23. Enterprise-ready • Open Source • Scalability • Error Handling • Transaction • Monitoring • Tooling • Commercial Support© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 24. Community  Camel rocks! Mailing Lists? Forums? Blogs? Articles? Conference talks? ESBs? Professionals? Jobs? Knowledge?© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 25. Live demo Apache Camel in action...© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 26. Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Graph-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Document-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 27. Graph-oriented database© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 28. Graph-oriented database • Neo Technology • Graphs rather than tables • Nodes, edges, and properties to represent and store data • Index-free adjacency • REST API and many SDKs (Java, .NET, Ruby, PHP, Python, etc.) • Embedded, disk-based, fully transactional • Powerful tool for graph-like queries • Example: Facebook friends© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 29. Code example: Neo4j Ruby API© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 30. Code example: camel-neo4j component // Producer from("jms:createNewNeo4jNode") .to("neo4j:http://Neo4jServer:7474/data"); // Consumer from(„neo4j://todo)... Not implemented in current Camel release (2.11)   Use Camel‘s REST components (shown in some minutes...)© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 31. Live demo Integration of a graph-oriented database in action...© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 32. Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Graph-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Document-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 33. Key-Value database© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 34. Key-Value database • Part of Amazon Web Services (AWS) • Online storage web service • Store arbitrary objects (computer files) up to 5 terabytes • REST and SOAP API • SDKs for Java, .NET, PHP, Ruby, etc. • Highly-scalable, reliable, and low-latency • Alternative for Hadoop‘s file system HDFS • Example: DigitalChalk offers creating, delivering and managing training videos© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 35. Code example: AWS S3 Java SDK AmazonS3 s3 = new AmazonS3Client(new PropertiesCredentials( S3Sample.class.getResourceAsStream("AwsCredentials.properties"))); String bucketName = "my-first-s3-bucket-" + UUID.randomUUID(); String key = "MyObjectKey"; try { s3.createBucket(bucketName); s3.putObject(new PutObjectRequest(bucketName, key, createSampleFile())); S3Object object = s3.getObject(new GetObjectRequest(bucketName, key)); ObjectListing objectListing = s3.listObjects(new ListObjectsRequest() .withBucketName(bucketName) .withPrefix("My")); s3.deleteObject(bucketName, key); s3.deleteBucket(bucketName); } catch (AmazonServiceException ase) { // error handling... } catch (AmazonClientException ace) { // error handling... }© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 36. Code example: camel-aws component // Producer from(„jms:toS3Queue") .setHeader(S3Constants.KEY, simple(“order.txt")) .to("aws-s3://myBucket?accessKey=" + a + "&secretKey= " + s) // Consumer from("aws-s3://myBucket?accessKey=“ + a + "&secretKey=" + s) .to("log:S3logging")© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 37. Tooling on top of Camel: Talend ESB Development Runtime Operation Talend Administration Center Service Development Web Services Stack Security Management Mediation & Mediation & Loadbalancing & High Configuration Integration Integration Availability Talend ESB Studio Project Testing Message Broker Business Rules Repository Talend ESB Deployment Performance & Build & Deploy Service Container Repository Availability Documentation & Examples 24x7 Support Training & Certification Indemnification Maintenance Professional Services Certified Partners© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 38. Tooling on top of Camel: Talend ESB Development Runtime Operation Management, Eclipse STP/WTP Apache CXF Secure Token Server Talend Administration Center Configuration & soapUI REST & Web Services Security Monitoring Service Locator & Talend ESB Runtime Route Designer Apache Camel Service Act. Service Activity Monitoring and Service Locator UI Talend ESB Studio Mediation Mediation Monitoring Distributed Registry / Tracking Service Designer Apache ActiveMQ Apache Archiva Repository Integration Message Broker Artifact repository Metadata & Projects Apache Maven Apache Karaf / Cellar Eclipse Equinox vFabric Build & Deploy OSGi / Clustering OSGi Hyperic HQ Documentation & Examples 24x7 Support Training & Certification Indemnification Maintenance Professional Services Certified Partners© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 39. Tooling on top of Camel: Talend ESB Route Builder  Endpoints  EIPs  Processors  Custom components Configuration  Components  Endpoints Code Generation  100% Java  Camel Code  Packaged as OSGi Bundles Execution in the IDE  Debugging  Live statistics  Short dev cycles© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 40. Live demo Integration of a key-value database in action...© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 41. Alternative to Vendor APIs ➜ Generic APIs© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 42. jClouds (Generic API) Generic API for IaaS© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 43. jClouds (Generic API) Compute API Blobstore API© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 44. jClouds (Generic API) Several different Cloud providers supported© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 45. jClouds (Generic API) – AWS S3 Blobstore (Java) Use another provider? Just change this line!© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 46. Code example: camel-jclouds component from("direct:toJcloudsAwsS3") .setHeader(JcloudsConstants.BLOB_NAME, "jclouds-demo-tutorial.txt") .setHeader(JcloudsConstants.CONTAINER_NAME, "kw-s3-data") .to("jclouds:blobstore:aws-s3") from("direct:toJcloudsMicrosoftAzure") .setHeader(JcloudsConstants.BLOB_NAME, "jclouds-demo-tutorial.txt") .setHeader(JcloudsConstants.CONTAINER_NAME, "kw-s3-data") .to("jclouds:blobstore:azureblob")© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 47. Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Graph-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Document-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 48. In-memory database© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 49. In-memory database • In-memory data grid • Clustering and highly scalable data distribution solution for Java platform • Architecture is peer-to-peer • Distributed Java data structures (Queue, Set, List, Map, Lock, Topic) • Java and REST API© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 50. Code example: Hazelcast Java API© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 51. Code example: camel-hazelcast component // Producer from("direct:add") .setHeader(HazelcastConstants.OPERATION, „add“) .to("hazelcast:queue:foo"); // Consumer from("hazelcast:queue:foo") .log(“content of object foo: ${body}");© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 52. Live demo Integration of an in-memory database in action...© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 53. Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Graph-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Document-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 54. Document-oriented database© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 55. Document-oriented database • 10gen • stores structured data as JSON-like documents with dynamic schemas • REST API and several SDKs (Java, .NET, Ruby, PHP, Python, etc.) • Ad hoc queries, indexing, replication, load balancing • Powerful, but also easy to use and flexible • Example: Disney persists state information of online games in a common object repository.© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 56. Code example: MongoDB Java Driver© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 57. Code example: camel-mongodb component // Producer from(“jms:FlightDocumentQueue") .to("mongodb:myDb?database=flights &collection=tickets &operation=insert"); // Consumer from("mongodb:myDb?database=flights &collection=cancellations &tailTrackIncreasingField=departureTime") .to(“jms:CancelledFlightsQueue");© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 58. Live demoIntegration of a document-oriented database in action...© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 59. Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Graph-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Document-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 60. Column-oriented database© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 61. HBase • Modeled after Googles BigTable • Runs on top of HDFS (Hadoop Distributed Filesystem) • Can serve as the input and output for MapReduce jobs run in Hadoop • Stores data tables as sections of columns of data rather than as rows of data • Java API plus REST, Avro or Thrift gateway APIs • Use HBase when you need random, realtime read/write access to your Big Data • Example: Advantages for DWHs, CRMs, and other ad-hoc inquiry systems where aggregates are computed over large numbers of similar data items.© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 62. Code example: HBase Java API© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 63. Code example: camel-hbase component© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 64. Live demo Integration of a column-oriented database in action...© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 65. Agenda 1) Introduction to NoSQL 2) Introduction to Apache Camel 3) Integration of a Graph-oriented Database 4) Integration of a Key-Value Database 5) Integration of an In-Memory Database 6) Integration of a Document-oriented Database 7) Integration of a Column-oriented Database 8) Custom NoSQL Components© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 66. Custom NoSQL components Component Creates Endpoint Consumer Producer© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 67. Live demo Custom NoSQL components in action...© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 68. Alternative for custom NoSQL components • SOAP • REST© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 69. Code example: REST API for Salesforce object store // Salesforce Query (SOQL) via REST API from("direct:salesforceViaHttpLIST") .setHeader("X-PrettyPrint", 1) .setHeader("Authorization", accessToken) .setHeader(Exchange.CONTENT_TYPE, "application/json") .to("https://na14.salesforce.com/services/data/v20.0/query?q=SELECT+name+from +Article__c") // Salesforce CREATE via REST API from("direct:salesforceViaHttpCREATE") .setHeader("X-PrettyPrint", 1) .setHeader("Authorization", accessToken) .setHeader(Exchange.CONTENT_TYPE, "application/json“) .to("https://na14.salesforce.com/services/data/v20.0/sobjects/Article__c")© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 70. Live demo NoSQL integration via REST in action...© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 71. SQL is still very alive, of course... Camel SQL components: • sql:statement • jdbc:dataSourceName • jpa://entityName • mybatis://statementName • hibernate://entityName© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 72. Did you get the key message?© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 73. Key messages NoSQL cannot be avoided, and must be integrated! NoSQL integration is already possible! Apache Camel helps a lot!© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 74. Did you get the key message?© Talend 2013 “Systems Integration in the NoSQL Era with Apache Camel” by Kai Wähner
  • 75. Thank you for your attention. Questions? kwaehner@talend.com www.kai-waehner.de LinkedIn / Xing @KaiWaehner