Kafka Summit NYC 2017 - Venice: A Distributed Database on top of Kafka

As a streaming platform, Apache Kafka provides low-latency, high-throughput, fault-tolerant publish and subscribe pipelines and excels at processing streams of real-time events. Kafka provides reliable, millisecond delivery for connecting downstream systems with real-time data. In this talk, we will show how easy it is to leverage Kafka and the Elasticsearch connector to keep your indices populated with the latest data from the rest of your enterprise, as it changes.

Presentation by Matthew Wise, LinkedIn

Software

Matthew Wise
Venice
A distributed database on top of Kafka

8
600+ Stores
100+ TB Pushed Daily
40+ Billion Queries/Day

Distributed K/V database
Kafka partitioning
Versioned pushes
Topic maps to version

MirrorMaker
Datacenter
Datacenter
Datacenter
Many
Datacenters

18
Venice Requirements
Message Buffer
Cross Datacenter Mirroring
Dynamic Topic Creation
Dynamic Topic Deletion
Mirrored Topic Creation
Mirrored Topic Deletion
Scalable Consumption
Data Guarantees

24
1 2 3 3 4 5
Data Guarantees:
At least once delivery

27
1 2 3 4 5
Data Guarantees:
Exactly once delivery

Data
Control
Distributed Systems:
Need Control

Matthew Wise – linkedin.com/in/matthewrswise
Venice
A distributed Database on top of Kafka

What's hot

Integrating Apache Kafka and Elastic Using the Connect Framework

Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation

Kafka Summit SF 2017 - Keynote - Go Against the Flow: Databases and Stream Pr...

Speakers: Ravi Dubey, Senior Manager, Software Engineering, Capital One + Jeff Sharpe, Software Engineer, Capital One Capital One supports interactions with real-time streaming transactional data using Apache Kafka®. Kafka helps deliver information to internal operation teams and bank tellers to assist with assessing risk and protect customers in a myriad of ways. Inside the bank, Kafka allows Capital One to build a real-time system that takes advantage of modern data and cloud technologies without exposing customers to unnecessary data breaches, or violating privacy regulations. These examples demonstrate how a streaming platform enables Capital One to act on their visions faster and in a more scalable way through the Kafka solution, helping establish Capital One as an innovator in the banking space. Join us for this online talk on lessons learned, best practices and technical patterns of Capital One’s deployment of Apache Kafka. -Find out how Kafka delivers on a 5-second service-level agreement (SLA) for inside branch tellers. -Learn how to combine and host data in-memory and prevent personally identifiable information (PII) violations of in-flight transactions. -Understand how Capital One manages Kafka Docker containers using Kubernetes. Watch the recording: https://videos.confluent.io/watch/6e6ukQNnmASwkf9Gkdhh69?.

Capital One Delivers Risk Insights in Real Time with Stream Processing

If a real-time dashboard takes 5 minutes to refresh, it’s not real-time. With data lakes increasingly enabling massive amounts of unprocessed data sets, delivering low-latency analytics is not for the faint-hearted. Learn how to stream massive amounts of data which used to be impossible to handle from Kafka, to serve real-time applications using lake-scale optimized approaches to storage and indexing.

Low-latency data applications with Kafka and Agg indexes | Tino Tereshko, Fir...

At Stripe, we operate a general ledger modeled as double-entry bookkeeping for all financial transactions. Warehousing such data is challenging due to its high volume and high cardinality of unique accounts. aFurthermore, it is financially critical to get up-to-date, accurate analytics over all records. Due to the changing nature of real time transactions, it is impossible to pre-compute the analytics as a fixed time series. We have overcome the challenge by creating a real time key-value store inside Pinot that can sustain half million QPS with all the financial transactions. We will talk about the details of our solution and the interesting technical challenges faced.

Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...

Lambda Architecture has been a common way to build data pipelines for a long time, despite difficulties in maintaining two complex systems. An alternative, Kappa Architecture, was proposed in 2014, but many companies are still reluctant to switch to Kappa. And there is a reason for that: even though Kappa generally provides a simpler design and similar or lower latency, there are a lot of practical challenges in areas like exactly-once delivery, late-arriving data, historical backfill and reprocessing. In this talk, I want to show how you can solve those challenges by embracing Apache Kafka as a foundation of your data pipeline and leveraging modern stream-processing frameworks like Apache Kafka Streams and Apache Flink.

It's Time To Stop Using Lambda Architecture | Yaroslav Tkachenko, Shopify

Kafka Summit SF 2017 - Fast Data in Supply Chain Planning

Companies are recognizing the importance of a low-latency, scalable, fault-tolerant data backbone, in the form of the Apache Kafka streaming platform. With Kafka, developers can integrate multiple sources and systems, enableing low latency analytics, event-driven architectures and the population of multiple downstream systems. These data pipelines can be built using configuration alone. In this talk we’ll see how easy it is to stream data from sources such as databases and into Kafka using the Kafka Connect API. We’ll use KSQL to filter, aggregate and join it to other data, and then stream this enriched data from Kafka out into targets such as Elasticsearch. All of this can be accomplished without a single line of code!

Streaming ETL to Elastic with Apache Kafka and KSQL

Apache kafka-a distributed streaming platform

We often need to build applications that analyze Kafka data to unlock the most value from event streams, so how can organizations build these real-time analytics applications? In this talk, we examine an indexing approach that enables fast SQL analytics on data from Kafka, without data flattening or denormalization. Rockset is the real-time indexing database that builds an inverted index, a columnar index and a row index on all fields of your Kafka messages, including nested fields and arrays. This Converged Index accelerates various types of analytic queries–search, aggregations and joins–without the need to denormalize or transform data for performance reasons. With indexing delivering significant gains in query performance, we also need to index new data in a timely manner. We discuss several strategies used for efficient ingestion and indexing from Kafka, including rollups, write optimizations on the underlying RocksDB storage engine, and the disaggregation of ingest and query compute.

Sub-Second SQL Search, Aggregations and Joins with Kafka and Rockset | Dhruba...

Milano Apache Kafka Meetup by Confluent (First Italian Kafka Meetup) on Wednesday, November 29th 2017. Il talk introduce Apache Kafka (incluse le APIs Kafka Connect e Kafka Streams), Confluent (la società creata dai creatori di Kafka) e spiega perché Kafka è un'ottima e semplice soluzione per la gestione di stream di dati nel contesto di due delle principali forze trainanti e trend industriali: Internet of Things (IoT) e Microservices.

Introduction to apache kafka, confluent and why they matter

Paolo Castagna

Introduction to Apache Kafka and Confluent... and why they matter

One of the key metrics to monitor when working with Apache Kafka, as a data pipeline or a streaming platform, is Consumer Groups Lag. Lag is the delta between the last produced message and the last committed message of a partition. In other words, lag indicates how far behind your application is in processing up-to-date information. For a long time, we used our own service to keep track of these metrics, collect them and visualize them. But this didn’t scale well. You had to perform many manual operations, redeploy it and to do other tedious manual tasks, but most importantly, the biggest gap for us, was that its output was represented in absolute numbers (e.g - your lag is 30K), which basically tells you nothing as a human being. We understood that we had to find a more suitable solution that will give us better visibility and will allow us to measure the lag in a time-based format that we all understand. In this talk, I’m going to go over the core concepts of Kafka offsets and lags, and explain why lag even matters and is an important KPI to measure. I’ll also talk about the kind of research we did to find the right tool, what the options in the market were at the time, and eventually why we chose Linkedin’s Burrow as the right tool for us. And finally, I’ll take a closer look at Burrow, its building blocks, how we build and deploy it, how we monitor better with it, and eventually the most important improvement - how we transformed its output from numbers to time-based metrics.

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020

Confluent Cloud makes Devops engineers lives a lot more easier. Yet moving 1500 microservices, 10K topics and 100K partitions to a multi-cluster Confluent cloud can be a challenge. In this talk you will hear about 5 lessons that Wix has learned in order to successfully meet this challenge. These lessons include: 1. Automation, Automation, Automation - all the process has to be completely automated at such scale 2. Prefer a gradual approach - E.g. migrate topics in small chunks and not all at once. Reduces risks if things go bad 3. Cleanup first - avoid migrating unused topics or topics with too many unnecessary partitions

5 lessons learned for successful migration to Confluent cloud | Natan Silinit...

Several different frameworks have been developed to draw data from Kafka and maintain standard SQL over continually changing data. This provides an easy way to query and transform data - now accessible by orders of magnitude more users. At the same time, using Standard SQL against changing data is a new pattern for many engineers and analysts. While the language hasn’t changed, we’re still in the early stages of understanding the power of SQL over Kafka - and in some interesting ways, this new pattern introduces some exciting new idioms. In this session, we’ll start with some basic use cases of how Standard SQL can be effectively used over events in Kafka- including how these SQL engines can help teams that are brand new to streaming data get started. From there, we’ll cover a series of more advanced functions and their implications, including: - WHERE clauses that contain time change the validity intervals of your data; you can programmatically introduce and retract records based on their payloads! - LATERAL joins turn streams of query arguments into query results; they will automatically share their query plans and resources! - GROUP BY aggregations can be applied to ever-growing data collections; reduce data that wouldn't even fit in a database in the first place. We'll review in-production examples where each of these cases make unmodified Standard SQL, run and maintain over data streams in Kafka, and provide the functionality of bespoke stream processors.

How to use Standard SQL over Kafka: From the basics to advanced use cases | F...

Activision Data team has been running a data pipeline for a variety of Activision games for many years. Historically we used a mix of micro-batch microservices coupled with classic Big Data tools like Hadoop and Hive for ETL. As a result, it could take up to 4-6 hours for data to be available to the end customers. In the last few years, the adoption of data in the organization skyrocketed. We needed to de-legacy our data pipeline and provide near-realtime access to data in order to improve reporting, gather insights faster, power web and mobile applications. I want to tell a story about heavily leveraging Kafka Streams and Kafka Connect to reduce the end latency to minutes, at the same time making the pipeline easier and cheaper to run. We were able to successfully validate the new data pipeline by launching two massive games just 4 weeks apart.

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

(Bruno Simic, Solutions Engineer, Couchbase) Breakout during Confluent’s streaming event in Munich. This three-day hands-on course focused on how to build, manage, and monitor clusters using industry best-practices developed by the world’s foremost Apache Kafka™ experts. The sessions focused on how Kafka and the Confluent Platform work, how their main subsystems interact, and how to set up, manage, monitor, and tune your cluster.

Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...

PayPal currently processes tens of billions of signals per day from different sources in batch and streaming mode. The data processing platform is the one powering these different analytical needs and use cases, not just at PayPal but our adjacencies like Venmo, Hyperwallet and iZettle. End users of this platform demand access to data insights with as much flexibility as possible to explore it with low processing latency. One such use case is where our Switchboard(data de-multiplexer) platform where we process approximately 20 billion events daily and provide data to different teams and platforms with-in PayPal and also to platform outside PayPal for more insights. When we started building this platform Kafka was just another asynchronous message processing platform for us but we have seen it evolving to a place where its adds value not just in terms of event processing but also for platform resiliency and scalability. Takeaway for the audience: Most people work with and have knowledge about data. With this talk I want to present information which is relevant and meaningful to the audience. Information and examples which will make it easier for attendees to understand our complex system and hopefully have some practical takeaways to use Kafka for similar problems on their hand.

Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...

Real-time Data Streaming from Oracle to Apache Kafka

Companies with batch and stream processing pipelines need to serve the insights they glean back to their users, an often-overlooked problem that can be hard to achieve reliably and at scale. Felix GV and Yan Yan offer an overview of Venice, a new data store capable of ingesting data from Hadoop and Kafka, merging it together, replicating it globally, and serving it online at low latency. Venice was designed to be the next-generation replacement of the Voldemort Read-Only system, with the intent to provide a broader feature set, better availability characteristics, and a more efficient architecture. Venice is designed for high-throughput ingestion from Hadoop and Kafka, and these data sources can be merged at ingestion time in order to provide semantics similar to those of a lambda architecture but with a simpler, faster, and more available read path. Robustness is a primary architectural concern and, as such, Venice provides highly available reads and writes, self-healing, stringent data validation guarantees, and the ability to roll back entire datasets in cases where bad data is pushed.

What's hot (20)

Integrating Apache Kafka and Elastic Using the Connect Framework

Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation

Kafka Summit SF 2017 - Keynote - Go Against the Flow: Databases and Stream Pr...

Capital One Delivers Risk Insights in Real Time with Stream Processing

Low-latency data applications with Kafka and Agg indexes | Tino Tereshko, Fir...

Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...

It's Time To Stop Using Lambda Architecture | Yaroslav Tkachenko, Shopify

Kafka Summit SF 2017 - Fast Data in Supply Chain Planning

Streaming ETL to Elastic with Apache Kafka and KSQL

Apache kafka-a distributed streaming platform

Sub-Second SQL Search, Aggregations and Joins with Kafka and Rockset | Dhruba...

Introduction to apache kafka, confluent and why they matter

Introduction to Apache Kafka and Confluent... and why they matter

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020

5 lessons learned for successful migration to Confluent cloud | Natan Silinit...

How to use Standard SQL over Kafka: From the basics to advanced use cases | F...

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...

Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...

Real-time Data Streaming from Oracle to Apache Kafka

Similar to Kafka Summit NYC 2017 - Venice: A Distributed Database on top of Kafka

Introducing Venice - Strata NYC 2017

Felix GV

Introducing Venice

Yan Yan

As your organization rapidly grows in scale, so do the amount of challenges. Growing scale comes in multiple dimensions - traffic, geographic presence, products portfolio, various technologies, amount of developers, etc. Coming up with an architecture that can handle all of the data flows in a universal, simple way is key. This talk is about Wix's Kafka based global data architecture and platform. How we made it very easy for Wix 2000 microservices to publish and subscribe to data, no matter where they are deployed in the world, or what technological stack they use. All the while offering various SDKs (some of them open-source), tools, and features for adapting to growing scale and insuring high resilience.

Dev Days Europe - Kafka based Global Data Mesh at Wix

Kafka based Global Data Mesh at Wix with Natan Silnitsky | Kafka Summit Londo...

Kafka Summit London - Kafka based Global Data Mesh at Wix

Devoxx Ukraine - Kafka based Global Data Mesh

As your organization rapidly grows in scale, so do the amount of challenges. Growing scale comes in multiple dimensions - traffic, geographic presence, products portfolio, various technologies, amount of developers, etc. Coming up with an architecture that can handle all of the data flows in a universal, simple way is key. This talk is about Wix's Kafka based global data architecture and platform. How we made it very easy for Wix 2000 microservices to publish and subscribe to data, no matter where they are deployed in the world, or what technological stack they use. All the while offering various tools and features for adapting to growing scale and insuring high resilience.

Kafka based Global Data Mesh at Wix

Can and should Apache Kafka replace a database? How long can and should I store data in Kafka? How can I query and process data in Kafka? These are common questions that come up more and more. This session explains the idea behind databases and different features like storage, queries, transactions, and processing to evaluate when Kafka is a good fit, and when it is not. The discussion includes different Kafka-native add-ons like Tiered Storage for long-term, cost-efficient storage, and ksqlDB as an event streaming database. The relation and trade-offs between Kafka and other databases are explored to complement each other instead of thinking about a replacement. This includes different options for pull and push-based bi-directional integration.

Can Apache Kafka Replace a Database? – The 2021 Update | Kai Waehner, Confluent

As Wix Kafka usage grew to 2.5B messages per day, >20K topics and >100K leader partitions serving 2000 microservices, we decided to migrate from self-operated single cluster per data-center to a managed cloud service (Like Amazon MSK or Confluent Cloud) with a multi-cluster setup. The classic approach would be to perform this transition when all incoming traffic is removed from the data center. But draining an entire data-center for an undetermined period of time, until all 2000 services complete the switch was too risky for us. This talk is about how we gradually migrated all of our Kafka consumers and producers with 0 downtime while they continued to handle regular traffic. You will learn practical steps you can take to greatly reduce the risks and speed up the migration timeline.

Migrating to Multi Cluster Managed Kafka - DevopStars 2022

Data Streaming with Apache Kafka & MongoDB - EMEA

Andrew Morgan

Webinar: Data Streaming with Apache Kafka & MongoDB

MongoDB

vCloud Technical deck - cb.ppt

juergenJaeckel

Data analytics at scale implementing stateful stream processing - publish

CodeValue

Migrating to Multi Cluster Managed Kafka - ApacheKafkaIL

Migrating to Multi Cluster Managed Kafka - Conf42 - CloudNative

Devoxx UK - Migrating to Multi Cluster Managed Kafka

“The mainframe is going away” is as true now as it was 10, 20 and 30 years ago. Mainframes are still crucial in handling critical business transactions, they were however built for an era where batch data movement was the norm and can be difficult to integrate into today’s data-driven, real-time, analytics-focused business processes as well as the environments that support them. Until now. Join experts from Confluent, Attunity, and Capgemini for a one-hour online talk session where you’ll learn how to: Unlock your mainframe data with unique change data capture (CDC) functionality without incurring the complexity and expense that come with sending ongoing queries into the mainframe database How using CDC benefits advanced analytics approaches such as deep machine learning and predictive analytics Deliver ongoing streams of data in real-time to the most demanding analytics environments Ensure that your analytics environment includes the broadest possible range of data sources and destinations while ensuring true enterprise-grade functionality Identify use cases that can help you get started delivering value to the business moving from POC to Pilot to Production

Leveraging Mainframe Data for Modern Analytics

Architecture patterns for distributed, hybrid, edge and global Apache Kafka deployments Multi-cluster and cross-data center deployments of Apache Kafka have become the norm rather than an exception. This session gives an overview of several scenarios that may require multi-cluster solutions and discusses real-world examples with their specific requirements and trade-offs, including disaster recovery, aggregation for analytics, cloud migration, mission-critical stretched deployments and global Kafka. Key takeaways: In many scenarios, one Kafka cluster is not enough. Understand different architectures and alternatives for multi-cluster deployments. Zero data loss and high availability are two key requirements. Understand how to realize this, including trade-offs. Learn about features and limitations of Kafka for multi cluster deployments Global Kafka and mission-critical multi-cluster deployments with zero data loss and high availability became the normal, not an exception.

Introduction to Apache Kafka and Confluent... and why they matter!

Paolo Castagna

Confluent and Elastic

Paolo Castagna

Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...

Kai Wähner

Similar to Kafka Summit NYC 2017 - Venice: A Distributed Database on top of Kafka (20)

Introducing Venice - Strata NYC 2017

Introducing Venice

Dev Days Europe - Kafka based Global Data Mesh at Wix

Kafka based Global Data Mesh at Wix with Natan Silnitsky | Kafka Summit Londo...

Kafka Summit London - Kafka based Global Data Mesh at Wix

Devoxx Ukraine - Kafka based Global Data Mesh

Kafka based Global Data Mesh at Wix

Can Apache Kafka Replace a Database? – The 2021 Update | Kai Waehner, Confluent

Migrating to Multi Cluster Managed Kafka - DevopStars 2022

Data Streaming with Apache Kafka & MongoDB - EMEA

Webinar: Data Streaming with Apache Kafka & MongoDB

vCloud Technical deck - cb.ppt

Data analytics at scale implementing stateful stream processing - publish

Migrating to Multi Cluster Managed Kafka - ApacheKafkaIL

Migrating to Multi Cluster Managed Kafka - Conf42 - CloudNative

Devoxx UK - Migrating to Multi Cluster Managed Kafka

Leveraging Mainframe Data for Modern Analytics

Introduction to Apache Kafka and Confluent... and why they matter!

Confluent and Elastic

Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...

More from confluent

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

Santander Stream Processing with Apache Flink

Unlocking the Power of IoT: A comprehensive approach to real-time insights

El Stream processing es un requisito previo de la pila de data streaming, que impulsa aplicaciones y pipelines en tiempo real. Permite una mayor portabilidad de datos, una utilización optimizada de recursos y una mejor experiencia del cliente al procesar flujos de datos en tiempo real. En nuestro taller práctico híbrido, aprenderás cómo filtrar, unir y enriquecer fácilmente datos en tiempo real dentro de Confluent Cloud utilizando nuestro servicio Flink sin servidor.

Workshop híbrido: Stream Processing con Flink

Our talk will explore the transformative impact of integrating Confluent, HiveMQ, and SparkPlug in Industry 4.0, emphasizing the creation of a Unified Namespace. In addition to the creation of a Unified Namespace, our webinar will also delve into Stream Governance and Scaling, highlighting how these aspects are crucial for managing complex data flows and ensuring robust, scalable IIoT-Platforms. You will learn how to ensure data accuracy and reliability, expand your data processing capabilities, and optimize your data management processes. Don't miss out on this opportunity to learn from industry experts and take your business to the next level.

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

La arquitectura impulsada por eventos (EDA) será el corazón del ecosistema de MAPFRE. Para seguir siendo competitivas, las empresas de hoy dependen cada vez más del análisis de datos en tiempo real, lo que les permite obtener información y tiempos de respuesta más rápidos. Los negocios con datos en tiempo real consisten en tomar conciencia de la situación, detectar y responder a lo que está sucediendo en el mundo ahora.

AWS Immersion Day Mapfre - Confluent

Eventos y Microservicios - Santander TechTalk

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

Citi TechTalk Session 2: Kafka Deep Dive

Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.

Build real-time streaming data pipelines to AWS with Confluent

Q&A with Confluent Professional Services: Confluent Service Mesh

Citi Tech Talk: Event Driven Kafka Microservices

An in depth look at how Confluent is being used in the financial services industry. Gain an understanding of how organisations are utilising data in motion to solve common problems and gain benefits from their real time data capabilities. It will look more deeply into some specific use cases and show how Confluent technology is used to manage costs and mitigate risks. This session is aimed at Solutions Architects, Sales Engineers and Pre Sales, and also the more technically minded business aligned people. Whilst this is not a deeply technical session, a level of knowledge around Kafka would be helpful.

Confluent & GSI Webinars series - Session 3

Transforming applications built with traditional messaging solutions such as TIBCO, MQ and Solace to be scalable, reliable and ready for the move to cloud How can applications built with traditional messaging technologies like TIBCO, Solace and IBM MQ be modernised and be made cloud ready? What are the advantages to Event Streaming approaches to pub/sub vs traditional message queues? What are the strengeths and weaknesses of both approaches, and what use cases and requirements are actually a better fit for messaging than Kafka?

Citi Tech Talk: Messaging Modernization

Citi Tech Talk: Data Governance for streaming and real time data

Confluent & GSI Webinars series: Session 2

Vous apprendrez également à : • Créer plus rapidement des produits et fonctionnalités à l’aide d’une suite complète de connecteurs et d’outils de gestion des flux, et à connecter vos environnements à des pipelines de données • Protéger vos données et charges de travail les plus critiques grâce à des garanties intégrées en matière de sécurité, de gouvernance et de résilience • Déployer Kafka à grande échelle en quelques minutes tout en réduisant les coûts et la charge opérationnelle associés

Data In Motion Paris 2023

Confluent Partner Tech Talk with Synthesis

The Future of Application Development - API Days - Melbourne 2023

The Playful Bond Between REST And Data Streams