Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
RealTime Messages at Scale
with Apache Kafka
Will Gardella
Product Manager
©2015 Couchbase Inc. 2
©2015 Couchbase Inc. 3
©2015 Couchbase Inc. 4
©2015 Couchbase Inc. 5
©2015 Couchbase Inc. 6
Agenda
 You might need Kafka if…
 Kafka architecture
 Background - Couchbase
 Couchbase & Kafka...
What’s Apache Kafka for?
You might need Kafka if…
©2015 Couchbase Inc. 8
You might need Kafka if…
Photo Credit: Cory Doctorow
https://www.flickr.com/photos/doctorow/14638938
©2015 Couchbase Inc. 9
Different speeds for different
systems
 NoSQL
 RDBMS
 Cache
 Search
 Apps
 Metrics
 Logs
 H...
Source:
Confluen
t
Typical Kafka Use Cases
©2015 Couchbase Inc. 12
Kafka Architecture
Broker 1
Consumer
Producer
Producer
Producer
Consumer
Consumer
Consumer
Kafka C...
©2015 Couchbase Inc. 13
Kafka Architecture
Broker 1
Consumer
Zookeeper
Producer
Producer
Producer
Consumer
Consumer
Consum...
Couchbase Server 4.0
A brief Introduction
©2015 Couchbase Inc. 15
Couchbase Server 4.0 for modern applications
Combines the flexibility of JSON, the power of SQL, a...
©2015 Couchbase Inc. 16
Couchbase Server Defined
The first NoSQL database that enables you to develop with
agility and ope...
©2015 Couchbase Inc. 17
The Power Of The Flexible JSON Schema
Ability to store data in multiple ways
• Denormalized single...
©2015 Couchbase Inc. 18
Couchbase and Other Big Data Systems
data scientist / engineersup to 1010 application
users
NoSQL
...
Kafka & Couchbase Use Cases
©2015 Couchbase Inc. 21
Couchbase & Kafka Use Cases
 Couchbase as the Master Database
– Changes in the bucket update data...
The Couchbase Kafka Connector
How it works
©2015 Couchbase Inc. 26
Database Change Protocol (DCP)
Couchbase Server’s internal data sync mechanism since Couchbase Ser...
An Example Producer and Consumer
ConnectingCouchbase via Kafka to anApplication
©2015 Couchbase Inc. 35
Kafka Generator Example
©2015 Couchbase Inc. 36
Kafka Producer Example
©2015 Couchbase Inc. 37
Kafka Producer Example
©2015 Couchbase Inc. 38
A Kafka Consumer Example
©2015 Couchbase Inc. 39
A Kafka Consumer Example
Demo
©2015 Couchbase Inc. 41
Couchbase Kafka Connector Roadmap
Available Now: 1.2 GA
 Kafka Producer or Consumer
 Stream even...
©2015 Couchbase Inc. 42
Learn More - Couchbase Kafka Connector
Confluent’s Ewen Cheslack-Postava at Couchbase Connect 2015...
Thank you.
will.gardella@couchbase.com
Twitter: @WillGardella
Upcoming SlideShare
Loading in …5
×

Real time Messages at Scale with Apache Kafka and Couchbase

2,002 views

Published on

Kafka is a scalable, distributed publish subscribe messaging system that's used as a data transmission backbone in many data intensive digital businesses. Couchbase Server is a scalable, flexible document database that's fast, agile, and elastic. Because they both appeal to the same type of customers, Couchbase and Kafka are often used together.

This presentation from a meetup in Mountain View describes Kafka's design and why people use it, Couchbase Server and its uses, and the use cases for both together. Also covered is a description and demo of Couchbase Server writing documents to a Kafka topic and consuming messages from a Kafka topic. using the Couchbase Kafka Connector.

Published in: Software
  • Be the first to comment

Real time Messages at Scale with Apache Kafka and Couchbase

  1. 1. RealTime Messages at Scale with Apache Kafka Will Gardella Product Manager
  2. 2. ©2015 Couchbase Inc. 2
  3. 3. ©2015 Couchbase Inc. 3
  4. 4. ©2015 Couchbase Inc. 4
  5. 5. ©2015 Couchbase Inc. 5
  6. 6. ©2015 Couchbase Inc. 6 Agenda  You might need Kafka if…  Kafka architecture  Background - Couchbase  Couchbase & Kafka  Behind the Scenes  Demo  An Example Producer and Consumer
  7. 7. What’s Apache Kafka for? You might need Kafka if…
  8. 8. ©2015 Couchbase Inc. 8 You might need Kafka if… Photo Credit: Cory Doctorow https://www.flickr.com/photos/doctorow/14638938
  9. 9. ©2015 Couchbase Inc. 9 Different speeds for different systems  NoSQL  RDBMS  Cache  Search  Apps  Metrics  Logs  Hadoop  Relational Data Warehouse
  10. 10. Source: Confluen t
  11. 11. Typical Kafka Use Cases
  12. 12. ©2015 Couchbase Inc. 12 Kafka Architecture Broker 1 Consumer Producer Producer Producer Consumer Consumer Consumer Kafka Cluster Broker 2 Broker 3
  13. 13. ©2015 Couchbase Inc. 13 Kafka Architecture Broker 1 Consumer Zookeeper Producer Producer Producer Consumer Consumer Consumer Kafka Cluster Broker 2 Broker 3 Topic 1 – Partition 1 Topic 2 – Partition 2 Topic 2 – Partition 1 Topic 3 – Partition 1 Topic 1 – Partition 2 Topic 3 – Partition 2
  14. 14. Couchbase Server 4.0 A brief Introduction
  15. 15. ©2015 Couchbase Inc. 15 Couchbase Server 4.0 for modern applications Combines the flexibility of JSON, the power of SQL, and the scale of NoSQL Develop with Agility Operate at Any Scale  Flexible JSON data model  Dynamic schema support  Powerful query language that extends SQL to JSON  Sub-millisecond latencies at scale  Elastic scaling on commodity servers  High availability
  16. 16. ©2015 Couchbase Inc. 16 Couchbase Server Defined The first NoSQL database that enables you to develop with agility and operate at any scale. Managed Cache Key-Value Store Document Database Embedded Database Sync Management
  17. 17. ©2015 Couchbase Inc. 17 The Power Of The Flexible JSON Schema Ability to store data in multiple ways • Denormalized single document, as opposed to normalizing data across multiple table • Dynamic Schema to add new values when needed
  18. 18. ©2015 Couchbase Inc. 18 Couchbase and Other Big Data Systems data scientist / engineersup to 1010 application users NoSQL Database 101- 102 Kafka Hadoop Spark Elasticsearch EDW
  19. 19. Kafka & Couchbase Use Cases
  20. 20. ©2015 Couchbase Inc. 21 Couchbase & Kafka Use Cases  Couchbase as the Master Database – Changes in the bucket update data elsewhere  Triggers / Event Handling – Handle events like deletions / expirations externally – E.g. expiration & replicated session tokens  Real-time Data Integration – Extract from Couchbase, transform and load data in real-time  Real-time Data Processing – Extract from a bucket, process in real-time and load back to another bucket
  21. 21. The Couchbase Kafka Connector How it works
  22. 22. ©2015 Couchbase Inc. 26 Database Change Protocol (DCP) Couchbase Server’s internal data sync mechanism since Couchbase Server 3.x  Used for – Intra-Cluster Replication – Indexing – XDCR (Cross Datacenter Replication for HA/DR) – Some connectors, including Kafka and Spark • Use Couchbase 2.x Java SDK JVM Core IO DCP handling library  Sends mutations – Mutations = creation, update, or delete of an item – Each mutation that occurs in a vBucket has a sequence number Important: DCP not supported for external clients!
  23. 23. An Example Producer and Consumer ConnectingCouchbase via Kafka to anApplication
  24. 24. ©2015 Couchbase Inc. 35 Kafka Generator Example
  25. 25. ©2015 Couchbase Inc. 36 Kafka Producer Example
  26. 26. ©2015 Couchbase Inc. 37 Kafka Producer Example
  27. 27. ©2015 Couchbase Inc. 38 A Kafka Consumer Example
  28. 28. ©2015 Couchbase Inc. 39 A Kafka Consumer Example
  29. 29. Demo
  30. 30. ©2015 Couchbase Inc. 41 Couchbase Kafka Connector Roadmap Available Now: 1.2 GA  Kafka Producer or Consumer  Stream events  Filters  Transform events 41 Code: https://github.com/couchbase/couchbase-kafka-connector/ Issues: https://issues.couchbase.com/projects/KAFKAC Docs: http://developer.couchbase.com/documentation/server/4.1/connectors/kafka-1.2/kafka-intro.html Planned  Monthly maintenance releases Under discussion  Merge code for Storm connector  Adopt Kafka Connect (Kafka 0.9)  ???
  31. 31. ©2015 Couchbase Inc. 42 Learn More - Couchbase Kafka Connector Confluent’s Ewen Cheslack-Postava at Couchbase Connect 2015  Great high level intro to Kafka in ~20 minutes  https://youtu.be/fFPVwYKUTHs Couchbase and Kafka - Up and Running in 10 Minutes  Run through the sample code yourself  http://blog.couchbase.com/2015/november/kafka-and-couchbase-up-and-running-in-10-minutes Product docs  http://developer.couchbase.com/documentation/server/4.1/connectors/kafka-1.2/kafka-intro.html Avalon Consulting blog and Github repo  http://blogs.avalonconsult.com/blog/big-data/purchase-transaction-alerting-with-couchbase-and- kafka/  https://github.com/Avalon-Consulting-LLC/couchbase-kafka 42
  32. 32. Thank you. will.gardella@couchbase.com Twitter: @WillGardella

×