Building a derived data store using Kafka

•Download as PPTX, PDF•

1 like•190 views

LinkedIn built a new derived data store called Venice to address limitations of their previous system Voldemort. Venice uses Kafka to enable scalable, fault-tolerant processing and replication of both batch-processed and incrementally updated derived data. It processes data through Hadoop jobs to Kafka topics, from which both batch-stored and real-time copies are maintained through Venice and Samza respectively. Kafka Mirror Maker replicates data across data centers for high availability.

Technology

Storage Infrastructure @ Linkedln
Deepshikha
SRE

Agenda
Why we need derived data stores
Nearline derived data
How we used Kafka to build a derived data
store

Derived Data Store
● What is derived data?
● Why do we need it?
● Why do we need specific stores for
derived data?

Voldemort Architecture
Voldemort
Colo1
Batch processing
Hadoop
Voldemort
Colo 2

Challenges with Voldemort
Inefficient batch
processing model
No support for
incremental
changes
Not fault tolerant

Near real-time streaming data
● Real Time insights drive engagement
and fuel a number of applications

Why incremental pushes?
● To serve near real time computation
● Doing a full cycle offline push to update partial records is
expensive wrt time & resource
● Atomic swap might result in not serving changed data

What do we need?
● Scalable, cost-effective replication of
batch processed data
● Pluggable support for stream processing
● Incremental pushes

Guess what? Kafka has it all!
Scalable, cost-effective replication of
batch processed data
Pluggable support for stream processing
Incremental pushes

It also has
Useful metrics
Cross-colo replication abstracted away
with Kafka Mirror Maker
Reduced overhead of setting up from
scratch!

Batch
processing
Venice architecture
Venice
Hadoop
Kafka
Venice Kafka
Kafka

Batch
processing
Venice architecture
Venice
Hadoop
Kafka
Venice Kafka
Kafka
Stream
processing
Samza

Replication via KMM
Venice corp
controller
corp kafka broker
hadoop

Replication via KMM
Venice corp
controller
corp kafka broker
hadoop
Colo 1
Kafka Mirror
Maker
Colo 2
Kafka Mirror
Maker
Colo 3
Kafka Mirror
Maker
Colo 4
Kafka Mirror
Maker

Replication via KMM
Corp Controller
Corp Kafka
Broker
Hadoop
Colo 1
Kafka Mirror
Maker
Colo 2
Kafka Mirror
Maker
Colo 3
Kafka Mirror
Maker
Colo 4
Kafka Mirror
Maker
Colo 1
Kafka Broker
Colo 3
Kafka Broker
Colo 2
Kafka Broker
Colo 4
Kafka Broker

Hybrid Mode
Store current batch’d
copy
Kafka Topics
Hadoop job
Venice

Hybrid Mode
Store current batch’d
copy
Kafka Topics
Hadoop job
Venice
Store streamed
copy
Samza job

Hybrid Mode
Store current batch’d
copy
Kafka Topics
Hadoop job
Venice
Store streamed
copy
Samza job
Store new batch’d copy

Evolution
Batch pushes
Hadoop Dependency
Limited Scalability
Tricky Expansion
Not fault tolerant
1.
2.
3.
4.
5.
Voldemort Venice
Batch & Incremental Pushes
Data derived from Kafka
No scaling issues
Easy to expand
Fault tolerant

What's hot

Using Kafka as a Database For Real-Time Transaction Processing | Chad Preisle...HostedbyConfluent

Putting Kafka Together with the Best of Google Cloud Platform confluent

Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...HostedbyConfluent

How Kafka and MemSQL Became the Dynamic Duo (Sarung Tripathi, MemSQL) Kafka S...HostedbyConfluent

Change Data Capture using KafkaAkash Vacher

Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...Nitin Kumar

Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...HostedbyConfluent

Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...HostedbyConfluent

Introduction to KafkaAkash Vacher

How to Lock Down Apache Kafka and Keep Your Streams Safeconfluent

Deploying Kafka at Dropbox, Mark Smith, Sean Fellowsconfluent

Capacity Planning Your Kafka Cluster | Jason Bell, DigitalisHostedbyConfluent

Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...confluent

StreamSQL Feature Store (Apache Pulsar Summit)Simba Khadder

HBaseCon2017 Splice Machine as a Service: Multi-tenant HBase using DCOS (Meso...HBaseCon

Kafka Connect - debeziumKasun Don

Kafka on Kubernetes: Keeping It Simple (Nikki Thean, Etsy) Kafka Summit SF 2019confluent

Performance Tuning RocksDB for Kafka Streams’ State Storesconfluent

Confluent Operations Training for Apache Kafkaconfluent

Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019 confluent

What's hot (20)

Using Kafka as a Database For Real-Time Transaction Processing | Chad Preisle...

Putting Kafka Together with the Best of Google Cloud Platform

Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...

How Kafka and MemSQL Became the Dynamic Duo (Sarung Tripathi, MemSQL) Kafka S...

Change Data Capture using Kafka

Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...

Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...

Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...

Introduction to Kafka

How to Lock Down Apache Kafka and Keep Your Streams Safe

Deploying Kafka at Dropbox, Mark Smith, Sean Fellows

Capacity Planning Your Kafka Cluster | Jason Bell, Digitalis

Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...

StreamSQL Feature Store (Apache Pulsar Summit)

HBaseCon2017 Splice Machine as a Service: Multi-tenant HBase using DCOS (Meso...

Kafka Connect - debezium

Kafka on Kubernetes: Keeping It Simple (Nikki Thean, Etsy) Kafka Summit SF 2019

Performance Tuning RocksDB for Kafka Streams’ State Stores

Confluent Operations Training for Apache Kafka

Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019

Similar to Building a derived data store using Kafka

Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...HostedbyConfluent

Real time Messages at Scale with Apache Kafka and CouchbaseWill Gardella

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Surviveconfluent

Introducing VeniceYan Yan

Can Apache Kafka Replace a Database? – The 2021 Update | Kai Waehner, ConfluentHostedbyConfluent

Introducing Venice - Strata NYC 2017Felix GV

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...Athens Big Data

Replication in real-time from Oracle and MySQL into data warehouses and analy...Continuent

Real-time Data Loading from Oracle and MySQL to Data Warehouses, AnalyticsContinuent

Replication in real-time from Oracle and MySQL into data warehouses and analy...Continuent

BBL KAPPA Lesfurets.comCedric Vidal

What is Apache Kafka and What is an Event Streaming Platform?confluent

Westpac Bank Tech Talk 1: Dive into Apache Kafkaconfluent

Replicate from Oracle to data warehouses and analyticsContinuent

AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017Monal Daxini

Liveperson DLD 2015 LivePerson

Apache kafkathe100rabh

Building Event-Driven Systems with Apache KafkaBrian Ritchie

Apache kafkaKumar Shivam

Can Apache Kafka Replace a Database?Kai Wähner

Similar to Building a derived data store using Kafka (20)

Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...

Real time Messages at Scale with Apache Kafka and Couchbase

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

Introducing Venice

Can Apache Kafka Replace a Database? – The 2021 Update | Kai Waehner, Confluent

Introducing Venice - Strata NYC 2017

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...

Replication in real-time from Oracle and MySQL into data warehouses and analy...

Real-time Data Loading from Oracle and MySQL to Data Warehouses, Analytics

Replication in real-time from Oracle and MySQL into data warehouses and analy...

BBL KAPPA Lesfurets.com

What is Apache Kafka and What is an Event Streaming Platform?

Westpac Bank Tech Talk 1: Dive into Apache Kafka

Replicate from Oracle to data warehouses and analytics

AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017

Liveperson DLD 2015

Apache kafka

Building Event-Driven Systems with Apache Kafka

Apache kafka

Can Apache Kafka Replace a Database?

Recently uploaded

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent

Key Features Of Token Development (1).pptxLBM Solutions

Benefits Of Flutter Compared To Other FrameworksSoftradix Technologies

SIEMENS: RAPUNZEL – A Tale About Knowledge GraphNeo4j

Install Stable Diffusion in windows machinePadma Pradeep

08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls

My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar

Slack Application Development 101 Slidespraypatel2

Swan(sea) Song – personal research during my six years at Swansea ... and bey...Alan Dix

Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 3652toLead Limited

How to Remove Document Management Hurdles with X-Docs?XfilesPro

Maximizing Board Effectiveness 2024 Webinar.pptxOnBoard

SQL Database Design For Developers at php[tek] 2024Scott Keck-Warren

GenCyber Cyber Security Day PresentationMichael W. Hawkins

Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j

Presentation on how to chat with PDF using ChatGPT code interpreternaman860154

Next-generation AAM aircraft unveiled by Supernal, S-A2Hyundai Motor Group

WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure servicePooja Nehwal

Breaking the Kubernetes Kill Chain: Host Path MountPuma Security, LLC

AI as an Interface for Commercial BuildingsMemoori

Recently uploaded (20)

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Key Features Of Token Development (1).pptx

Benefits Of Flutter Compared To Other Frameworks

SIEMENS: RAPUNZEL – A Tale About Knowledge Graph

Install Stable Diffusion in windows machine

08448380779 Call Girls In Civil Lines Women Seeking Men

My Hashitalk Indonesia April 2024 Presentation

Slack Application Development 101 Slides

Swan(sea) Song – personal research during my six years at Swansea ... and bey...

Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365

How to Remove Document Management Hurdles with X-Docs?

Maximizing Board Effectiveness 2024 Webinar.pptx

SQL Database Design For Developers at php[tek] 2024

GenCyber Cyber Security Day Presentation

Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...

Presentation on how to chat with PDF using ChatGPT code interpreter

Next-generation AAM aircraft unveiled by Supernal, S-A2

WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service

Breaking the Kubernetes Kill Chain: Host Path Mount

AI as an Interface for Commercial Buildings

Building a derived data store using Kafka

1. Storage Infrastructure @ Linkedln Deepshikha SRE

2. Agenda Why we need derived data stores Nearline derived data How we used Kafka to build a derived data store

3. Derived Data Store ● What is derived data? ● Why do we need it? ● Why do we need specific stores for derived data?

4. Legacy derived data store Voldemort

5. Voldemort Architecture Voldemort Colo1 Batch processing Hadoop Voldemort Colo 2

6. Challenges with Voldemort Inefficient batch processing model No support for incremental changes Not fault tolerant

7. Near real-time streaming data ● Real Time insights drive engagement and fuel a number of applications

8. Why incremental pushes? ● To serve near real time computation ● Doing a full cycle offline push to update partial records is expensive wrt time & resource ● Atomic swap might result in not serving changed data

9. What do we need? ● Scalable, cost-effective replication of batch processed data ● Pluggable support for stream processing ● Incremental pushes

10. Guess what? Kafka has it all! Scalable, cost-effective replication of batch processed data Pluggable support for stream processing Incremental pushes

11. It also has Useful metrics Cross-colo replication abstracted away with Kafka Mirror Maker Reduced overhead of setting up from scratch!

12. Brand new derived data store! Venice

13. Batch processing Venice architecture Venice Hadoop Kafka Venice Kafka Kafka

14. Batch processing Venice architecture Venice Hadoop Kafka Venice Kafka Kafka Stream processing Samza

15. Replication via KMM Venice corp controller corp kafka broker hadoop

16. Replication via KMM Venice corp controller corp kafka broker hadoop Colo 1 Kafka Mirror Maker Colo 2 Kafka Mirror Maker Colo 3 Kafka Mirror Maker Colo 4 Kafka Mirror Maker

17. Replication via KMM Corp Controller Corp Kafka Broker Hadoop Colo 1 Kafka Mirror Maker Colo 2 Kafka Mirror Maker Colo 3 Kafka Mirror Maker Colo 4 Kafka Mirror Maker Colo 1 Kafka Broker Colo 3 Kafka Broker Colo 2 Kafka Broker Colo 4 Kafka Broker

18. Replication via KMM Corp Controller Corp Kafka Broker Hadoop Colo 1 Kafka Mirror Maker Colo 2 Kafka Mirror Maker Colo 3 Kafka Mirror Maker Colo 4 Kafka Mirror Maker Colo 1 Kafka Broker Colo 3 Kafka Broker Colo 2 Kafka Broker Colo 4 Kafka Broker Venice subsystem Venice subsystem Venice subsystem Venice subsystem

19. Hybrid Mode Store current batch’d copy Kafka Topics Hadoop job Venice

20. Hybrid Mode Store current batch’d copy Kafka Topics Hadoop job Venice Store streamed copy Samza job

21. Hybrid Mode Store current batch’d copy Kafka Topics Hadoop job Venice Store streamed copy Samza job Store new batch’d copy

22. Hybrid Mode Store current batch’d copy Kafka Topics Hadoop job Venice Store streamed copy Samza job Store new batch’d copy

23. Evolution Batch pushes Hadoop Dependency Limited Scalability Tricky Expansion Not fault tolerant 1. 2. 3. 4. 5. Voldemort Venice Batch & Incremental Pushes Data derived from Kafka No scaling issues Easy to expand Fault tolerant

24. Questions?

Building a derived data store using Kafka

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Building a derived data store using Kafka

Similar to Building a derived data store using Kafka (20)

Recently uploaded

Recently uploaded (20)

Building a derived data store using Kafka