Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Couchbase and Apache Kafka - Bridging the gap between RDBMS and NoSQL

1,014 views

Published on

Thousands of companies, from Uber and Netflix to Goldman Sachs and Cisco, use Apache Kafka to transform and reshape their data architectures. Kafka is frequently used as the bridge between legacy RDBMS and new NoSQL database systems, effectively transforming SQL table data into JSON documents and vice versa. Many companies also use Kafka for business-critical applications that drive real-time stream processing and analytics, intersystem messaging, high-volume data ingestion, and operational metrics collection.

Couchbase and Kafka can be used together to address high throughput, distributed data management, and transformation challenges.

In this webinar we’ll explore:

Where Kafka fits into the big data ecosystem
How companies are using Kafka for both real-time processing and as a bus for data exchange
An example of how Kafka can bridge legacy RDBMS and new NoSQL database systems
Several real-world use case architectures

Published in: Technology
  • Be the first to comment

Couchbase and Apache Kafka - Bridging the gap between RDBMS and NoSQL

  1. 1. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. COUCHBASE & CONFLUENT Bridging the Gap between RDBMS and NoSQL Tyler Mitchell, Senior Product Manager, Couchbase David Tucker, Director Partner Engineering and Alliances, Confluent
  2. 2. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. AGENDA 01/ 02/ 03/ 04/ Big Data Architecture Evolution Kafka as a Streaming Bus Real World Use Cases DEMO – Couchbase | Kafka | MySQL
  3. 3. BIG DATA ARCHITECTURE EVOLUTION1
  4. 4. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 5 Big Data Architecture Evolution Big Data version 1 focused on • ingest • archive
  5. 5. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 6 Big Data Architecture Evolution Big Data version 2 focused on • collate • analyze
  6. 6. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 7 Big Data Architecture Evolution Big Data version 3 ... • stream • remix
  7. 7. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 8 Big Data Architecture Evolution Big Data version 3 ... • stream • remix • ... engagement
  8. 8. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 9 Data Processing Platform • Critical connectivity • Analysis • Streaming HDFS DBMS Mobile Other Platforms Data Processing Platform
  9. 9. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 10 Stream Data Platform • Critical connectivity • Analysis • Streaming • Distributed • High throughput • Wider engagement HDFS DBMS Mobile Other Platforms Stream Data Platform
  10. 10. KAFKA AS A STREAMING BUS2
  11. 11. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 12 Couchbase & Kafka • Source • Sink • Custom Filter • Apache Kafka • Confluent Platform Kafka ?
  12. 12. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 13 How Organizations Handle Data Flow: A Giant Mess Data Warehouse Hadoop NoSQL Oracle SFDC Logging Bloomberg … any sink/source … and more OLTP ActiveMQ App App Caches OLTP OLTPAppAppApp Web Custom Apps Microservices Monitoring Analytics
  13. 13. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 14 Apache Kafka™: A Distributed Streaming Platform Apache Kafka Data Warehouse Hadoop NoSQL Oracle SFDC Twitter Bloomberg … any sink/source … any sink/source … and more Web Custom Apps Microservices Monitoring Analytics
  14. 14. REAL WORLD USE CASE3
  15. 15. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 16 Applications Across Industries Healthcare & Pharma Patient Monitoring, Pharma Substance control, Patient Relapse, Lab Results Alerts Banking & Capital Markets Fraud Detection, Trade Data Capture, Customer 360 Retail Inventory Management, Product Catalog, A/B Testing, Proactive Alerts Telecommunications Personalized Ads, Customer 360, Network Integrity Automotive Connected Car, Manufacturing Data Processing Travel & Leisure Visitor Segmentation, Fraud Detection
  16. 16. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 17 Confluent Platform Enterprise Streaming based on Apache Kafka™ Database Changes Log Events loT Data Web Events … CRM Data Warehouse Database Hadoop Data Integration … Monitoring Analytics Custom Apps Transformations Real-time Applications … Apache Open Source Confluent Open Source Confluent Enterprise Confluent Platform Apache Kafka™ Data Compatibility Monitoring & Administration Operations Clients Connectors Complete Open Trusted Enterprise Grade
  17. 17. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 18 Over 35% of Fortune 500 are using Apache Kafka™ 6 of top 10 Travel 7 of top 10 Global banks 8 of top 10 Insurance 9 of top 10 Telecom
  18. 18. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 19 Bank Reduces OpEx by $25M/year via Mainframe Offload Date Amount 1/27/2017 $4.56 1/22/2017 $32.14 Transaction Data Vendor Description Starbucks Coffee Walmart Blu-Ray Transaction Description Schema Website Microservices Match data and description Client profiles Lookup client profiles Mainframe MIPS = $$
  19. 19. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 20 Ingest, Process, Load, and Serve Data at a Global Scale Engagement Database … Engagement Database … Kafka cluster Applications Other data stores Kafka cluster Kafka Streams (Data Enrichment and Transformation) Kafka Connect (Connectors to Extract and Load data) Confluent Replicator Confluent Replicator Custom Replication Custom Replication Raw event data
  20. 20. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 21 How do I get streams of data into and out of my apps? Connect Clients REST
  21. 21. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 22 Kafka Connect – Streaming Data Capture JDBC IRC / Twitter CDC Couchbase NoSQL HDFS Kafka Connect API Kafka Pipeline Connector Connector Connector Connector Connector Connector Sources Sinks Fault tolerant Manage hundreds of data sources and sinks Preserves data schema Part of Apache Kafka project Integrated within Confluent Platform’s Control Center
  22. 22. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 23 Kafka Connect API Library of Connectors * Denotes Connectors developed at Confluent and distributed by Confluent. Extensive validation and testing have been performed. Databases * Analytics * Applications / Other Datastore / File Store * *
  23. 23. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 24 Kafka Streams API: A Part of Apache Kafka Kafka Streams API Producer Kafka Cluster Topic TopicTopic Consumer Consumer Key Benefits • No additional cluster • Easy to run as a service • Supports large aggregations and joins • Security and permissions fully integrated from Kafka Example Use Cases • Microservices • Continuous queries • Continuous transformations • Event-triggered processes
  24. 24. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 25 Architecture Example Before: Complexity for development and operations, heavy footprint 1 2 3 Capture business events in Kafka Must process events with separate, special- purpose clusters Write results back to Kafka Your Processing Job
  25. 25. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 26 Architecture Example With Kafka Streams: App-centric architecture that blends well into your existing infrastructure 1 2 3a Capture business events in Kafka Process events fast, reliably, securely with standard Java applications Write results back to Kafka Your App 3b External apps can directly query the latest results AppApp Kafka Streams API
  26. 26. DEMO COUCHBASE KAFKA & MYSQL4
  27. 27. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 28 Demo Scenario Powerful Streaming Data Pipeline Simulated transaction workload captured at scale to Couchbase • Sysbench workload running against a MySQL database • Debezium CDC Source Connector publishes change records to Kafka topic • Sink connector saves Kafka topic to Couchbase - with optional filtering • Key-value queries can aggregate RDBMS data or track explicit operations
  28. 28. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 29 Kafka Connect Demonstration Kafka Connect Apache Kafka Brokers Optional K-Streams app(s) Couchbase SinkConnector CDC SourceConnector 52 3 4 1
  29. 29. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. THANK YOU

×