Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Kafka & Couchbase
Integration Patterns
Manuel Hurtado
Couchbase Solutions Engineer
©2016 Couchbase Inc. 2
Agenda
• Couchbase Introduction
• Kafka Introduction
• Kafka Connect & Couchbase Kafka Connector
• ...
©2016 Couchbase Inc. 3
Agenda
• Couchbase Introduction
• Kafka Introduction
• Kafka Connect & Couchbase Kafka Connector
• ...
©2016 Couchbase Inc. 4
What is Couchbase?
Couchbase delivers the Data Platform for the Digital Economy
• Products: Couchba...
©2016 Couchbase Inc. 5
Growing number of use cases
5
Catalog Metadata
Operational
Dashboarding
User Profile
Database
Sessi...
©2016 Couchbase Inc. 6
Why customers choose Couchbase?
6
Memory-first
Architecture
Full SQL Query
Language
Active-Active
G...
©2016 Couchbase Inc. 7
Agenda
• Couchbase Introduction
• Kafka Introduction
• Kafka Connect & Couchbase Kafka Connector
• ...
©2016 Couchbase Inc. 8
Kafka Introduction
What is Kafka?
• Kafka is a distributed, partitioned, replicated, log
service de...
©2016 Couchbase Inc. 9
Kafka Architecture
©2016 Couchbase Inc. 10
Kafka Architecture
©2016 Couchbase Inc. 11
Agenda
• Couchbase Introduction
• Kafka Introduction
• Kafka Connect & Couchbase Kafka Connector
•...
©2016 Couchbase Inc. 12
The challenge: Stream Data Platform with Kafka
Source: https://www.confluent.io/blog/announcing-ka...
©2016 Couchbase Inc. 13
Kafka Connect
Source: https://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-...
©2016 Couchbase Inc. 14
Kafka Connect
Source: https://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-...
©2016 Couchbase Inc. 15
Couchbase Connector in Kafka
©2016 Couchbase Inc. 16
Database Change Protocol (DCP)
©2016 Couchbase Inc. 17
Agenda
• Couchbase Introduction
• Kafka Introduction
• Kafka Connect & Couchbase Kafka Connector
•...
©2016 Couchbase Inc. 18
Couchbase & Kafka Use Cases
©2016 Couchbase Inc. 19
Couchbase & Kafka Use Cases
Currently using Kafka, but not Couchbase
Reading from Couchbase with a...
©2016 Couchbase Inc. 20
Couchbase & Kafka Use Cases
Currently using Couchbase, but not Kafka
Backup of the data
○ Need to ...
©2016 Couchbase Inc. 21
Couchbase & Kafka Use Cases
Currently using Kafka and Couchbase
Integrate more naturally with Kafk...
©2016 Couchbase Inc. 22
©2016 Couchbase Inc. 23
Demo: Couchbase as Data Source Basic Scenario
ConsoleConsumer
• OOTB Couchbase Connector
• Bucket ...
©2016 Couchbase Inc. 24
Demo: Couchbase as Data Source + Data Sink Basic
Scenario
• OOTB Couchbase Connector
• Bucket “tra...
©2016 Couchbase Inc. 25
Demo: Couchbase as Data Source + Data Sink Custom
Scenario
• Custom Schema Converter
• Custom Filt...
©2016 Couchbase Inc. 26
Demo: Twitter Data Source + Couchbase as Data Sink
• Custom Kafka Producer & Consumer
• Bucket “tw...
©2016 Couchbase Inc. 27
Couchbase Connector in Confluent Platform
©2016 Couchbase Inc. 28
Confluent Platform
©2016 Couchbase Inc. 29
Resources
• Couchbase Kafka Connector
https://developer.couchbase.com/documentation/server/current...
Q & A
Upcoming SlideShare
Loading in …5
×

Kafka & Couchbase Integration Patterns

428 views

Published on

Presentation from a meetup in Madrid on 9 March 2017

Published in: Data & Analytics
  • Hello! High Quality And Affordable Essays For You. Starting at $4.99 per page - Check our website! https://vk.cc/82gJD2
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Kafka & Couchbase Integration Patterns

  1. 1. Kafka & Couchbase Integration Patterns Manuel Hurtado Couchbase Solutions Engineer
  2. 2. ©2016 Couchbase Inc. 2 Agenda • Couchbase Introduction • Kafka Introduction • Kafka Connect & Couchbase Kafka Connector • Use cases
  3. 3. ©2016 Couchbase Inc. 3 Agenda • Couchbase Introduction • Kafka Introduction • Kafka Connect & Couchbase Kafka Connector • Use cases
  4. 4. ©2016 Couchbase Inc. 4 What is Couchbase? Couchbase delivers the Data Platform for the Digital Economy • Products: Couchbase Server & Couchbase Mobile • Open source NoSQL, JSON document database • Founded 2010 • 500+ enterprise customers, including 20+ Fortune 100 UNIFIED ADMINISTRATION UNIFIED PROGRAMMING INTERFACE Data Query Index SearchMobileReplication Analytics {N1QL}
  5. 5. ©2016 Couchbase Inc. 5 Growing number of use cases 5 Catalog Metadata Operational Dashboarding User Profile Database Session Database Inventory & Availability Entitlement Management Field Service Enablement Customer 360 Asset/Resource Management Device User Data Management Endpoint Data Management
  6. 6. ©2016 Couchbase Inc. 6 Why customers choose Couchbase? 6 Memory-first Architecture Full SQL Query Language Active-Active Global Data Replication Multi-dimensional scaling Mobile
  7. 7. ©2016 Couchbase Inc. 7 Agenda • Couchbase Introduction • Kafka Introduction • Kafka Connect & Couchbase Kafka Connector • Use cases
  8. 8. ©2016 Couchbase Inc. 8 Kafka Introduction What is Kafka? • Kafka is a distributed, partitioned, replicated, log service developed by LinkedIn and open sourced in 2011. • Basically it is a massively scalable pub/sub message queue architected as a distributed transaction log. It was created to provide “a unified platform for handling all the real-time data feeds a large company might have”. • It powers a large number of high-profile companies including LinkedIn, Yahoo and Netflix.
  9. 9. ©2016 Couchbase Inc. 9 Kafka Architecture
  10. 10. ©2016 Couchbase Inc. 10 Kafka Architecture
  11. 11. ©2016 Couchbase Inc. 11 Agenda • Couchbase Introduction • Kafka Introduction • Kafka Connect & Couchbase Kafka Connector • Use cases
  12. 12. ©2016 Couchbase Inc. 12 The challenge: Stream Data Platform with Kafka Source: https://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines/
  13. 13. ©2016 Couchbase Inc. 13 Kafka Connect Source: https://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines/
  14. 14. ©2016 Couchbase Inc. 14 Kafka Connect Source: https://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines/
  15. 15. ©2016 Couchbase Inc. 15 Couchbase Connector in Kafka
  16. 16. ©2016 Couchbase Inc. 16 Database Change Protocol (DCP)
  17. 17. ©2016 Couchbase Inc. 17 Agenda • Couchbase Introduction • Kafka Introduction • Kafka Connect & Couchbase Kafka Connector • Use cases
  18. 18. ©2016 Couchbase Inc. 18 Couchbase & Kafka Use Cases
  19. 19. ©2016 Couchbase Inc. 19 Couchbase & Kafka Use Cases Currently using Kafka, but not Couchbase Reading from Couchbase with almost no code (Source) ○ Adopt default message format given by connector ○ Write custom converter ○ Use Kafka Streams to process events from Couchbase, and write them back. Writing to Couchbase without touching SDK (Sink) ○ Provide document in the generic form (it will be converted into JSON)
  20. 20. ©2016 Couchbase Inc. 20 Couchbase & Kafka Use Cases Currently using Couchbase, but not Kafka Backup of the data ○ Need to be filtered/analyzed/transformed by the business logic ○ Checking integrity Turn realtime data into history ○ e.g. Couchbase keeps the exchange rates, but it is necessary to emit stream of the deltas, or log of the changes.
  21. 21. ©2016 Couchbase Inc. 21 Couchbase & Kafka Use Cases Currently using Kafka and Couchbase Integrate more naturally with Kafka infrastructure. A lot of the custom code may be moved into connector. ○ Filtering ○ Transformation ○ Process/Task management
  22. 22. ©2016 Couchbase Inc. 22
  23. 23. ©2016 Couchbase Inc. 23 Demo: Couchbase as Data Source Basic Scenario ConsoleConsumer • OOTB Couchbase Connector • Bucket “travel-sample” • Output all-content • Consume messages to console • Insert/Update/Delete • JMX Monitor
  24. 24. ©2016 Couchbase Inc. 24 Demo: Couchbase as Data Source + Data Sink Basic Scenario • OOTB Couchbase Connector • Bucket “travel-sample” to “receiver” • Raw content • Write messages to Couchbase • JMX Monitor
  25. 25. ©2016 Couchbase Inc. 25 Demo: Couchbase as Data Source + Data Sink Custom Scenario • Custom Schema Converter • Custom Filter • “travel-sample” to “receiver”: only “airlines” • JSON content • Write messages to Couchbase • JMX Monitor AirlineConverter AirlineFilter
  26. 26. ©2016 Couchbase Inc. 26 Demo: Twitter Data Source + Couchbase as Data Sink • Custom Kafka Producer & Consumer • Bucket “twitter” • Write messages to Couchbase • JMX Monitor TwitterProducer Couchbase Consumer
  27. 27. ©2016 Couchbase Inc. 27 Couchbase Connector in Confluent Platform
  28. 28. ©2016 Couchbase Inc. 28 Confluent Platform
  29. 29. ©2016 Couchbase Inc. 29 Resources • Couchbase Kafka Connector https://developer.couchbase.com/documentation/server/current/connectors/kafka- 3.1/kafka-intro.html • Couchbase Kafka Connector (code) https://github.com/couchbase/kafka-connect-couchbase • Demo code repositories https://github.com/mahurtado/KafkaCouchbaseConnectorSample https://github.com/mahurtado/TwitterKafkaCouchbasePipeline • Kafka Connect Confluent https://www.confluent.io/product/connectors/
  30. 30. Q & A

×