Real-world Streaming Architectures

•

5 likes•1,314 views

This document discusses four streaming data architectures used by companies like Yelp, LINE, and Altspace VR. It begins by providing context on streaming data and the evolution from batch processing with Hadoop/MapReduce to stream processing with Apache Kafka and frameworks like Storm. The main case studies then describe how each company uses streaming to build real-time data pipelines and services. Key aspects discussed include using Kafka for messaging between services, Kafka Streams for stream processing, and storing streaming data in databases. The document concludes by promoting Apache Kafka as a platform for building scalable streaming applications.

1
@tlberglund
Four Streaming Data
Architectures

8Maria-José Viñas, NASA Earth Science News Team.

16
Yelp
https://engineeringblog.yelp.com/2016/07/billions-of-messages-a-day-yelps-real-time-data-pipeline.html
Justin C., Software Engineer
(May, 2016)

17
• 150 production services
• Metcalf’s Law
• REST scales poorly
• N+1 problem and bulk data services
A Streaming Backbone

20
• Stream-table duality
• But some services still want databases
• Individual services use MySQL
• Millions of rows across services
Streaming Databases

21
• In 2010, commitment to MapReduce was strong
• That fun was soon over
• Want to capitalize on schema translation tools
• Want near-real-time results
• Storm didn’t support enough Python
• Wrote their own!
Computing Streams

23
Yelp Conclusions
• Streaming pipeline ensures evolvability
• Stream/Table Duality
• Schema matters
• Yelp often ahead of the game!

24
LINE Corp
https://engineering.linecorp.com/en/blog/detail/80

25
Goals
• Unified data delivery pipeline between systems
• Replace a legacy, bespoke system for processing background tasks
• Had to pick a framework

28
KStreamBuilder builder = new KStreamBuilder();
// Read the input Kafka topic into a KStream instance.
KStream<byte[], String> textLines = builder.stream("TextLinesTopic");
// Convert to upper case (:: is Java 8 syntax)
KStream<byte[], String> uppercasedWithMapValues = textLines.mapValues(String::toUpperCase);
// Write the results to a new Kafka topic called "UppercasedTextLinesTopic".
uppercasedWithMapValues.to("UppercasedTextLinesTopic");
KafkaStreams streams = new KafkaStreams(builder, streamsConfiguration);
streams.start();
Streaming API for Kafka

29
Streaming API for Kafka
Stateful Transformations

30
Loopback Replicator
• One big input topic of all TalkOperations (1M messages/sec)
• Some services want a limited view of these
• Simple filtering

32
LINE Conclusions
• Classical Kafka Streams implementation
• Better to “buy” than build

39
Altspace Architecture
• Photon game engine
• Real-time play
• Capture for playback
• An entirely classical stream processing system

40
“Mirror User A to room R2”
Source: Altspace

41
6 months later: “Capture User A”
Source: Altspace

42
Playbacks
• Source capture data from Kafka or S3
• “Produce” it in a timed fashion

43
“Playback capture to room R2” Source: Altspace

69
Get Started with Apache Kafka Today!
THE place to start with Apache Kafka!
Thoroughly tested and quality
assured
More extensible developer
experience
Easy upgrade path to
Confluent Enterprise
https://www.confluent.io/download/

70
Confluent Community
Join us in Slack
https://slackpass.io/confluentcommunity
Find your local Meetup Group
https://www.confluent.io/apache-kafka-meetups/

My talk at Strata Data Conference, London, May 2017. https://conferences.oreilly.com/strata/strata-eu/public/schedule/detail/57619 Abstract: Modern businesses have data at their core, but this data is changing continuously. How can you harness this torrent of information in real time? The answer: stream processing. The core platform for streaming data is Apache Kafka, and thousands of companies are using Kafka to transform and reshape their industries, including Netflix, Uber, PayPal, Airbnb, Goldman Sachs, Cisco, and Oracle. Unfortunately, today’s common architectures for real-time data processing at scale suffer from complexity: to succeed, many technologies need to be stitched and operated together, and each individual technology is often complex by itself. This has led to a strong discrepancy between how we engineers would like to work and how we actually end up working in practice. Michael Noll explains how Apache Kafka helps you radically simplify your data processing architectures by building normal applications to serve your real-time processing needs rather than building clusters or similar special-purpose infrastructure—while still benefiting from properties typically associated exclusively with cluster technologies, like high scalability, distributed computing, and fault tolerance. Michael also covers Kafka’s Streams API, its abstractions for streams and tables, and its recently introduced interactive queries functionality. Along the way, Michael shares common use cases that demonstrate that stream processing in practice often requires database-like functionality and how Kafka allows you to bridge the worlds of streams and databases when implementing your own core business applications (for example, in the form of event-driven, containerized microservices). As you’ll see, Kafka makes such architectures equally viable for small-, medium-, and large-scale use cases.

Kafka Summit SF 2017 - Real-Time Document Rankings with Kafka Streams

confluent

Introducing Apache Kafka's Streams API - Kafka meetup Munich, Jan 25 2017

Michael Noll

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

HostedbyConfluent

Activision Data team has been running a data pipeline for a variety of Activision games for many years. Historically we used a mix of micro-batch microservices coupled with classic Big Data tools like Hadoop and Hive for ETL. As a result, it could take up to 4-6 hours for data to be available to the end customers. In the last few years, the adoption of data in the organization skyrocketed. We needed to de-legacy our data pipeline and provide near-realtime access to data in order to improve reporting, gather insights faster, power web and mobile applications. I want to tell a story about heavily leveraging Kafka Streams and Kafka Connect to reduce the end latency to minutes, at the same time making the pipeline easier and cheaper to run. We were able to successfully validate the new data pipeline by launching two massive games just 4 weeks apart.

Kafka Summit NYC 2017 - Data Processing at LinkedIn with Apache Kafka

confluent

A Tour of Apache Kafka

confluent

Stream Processing with Apache Kafka and .NET

confluent

Presentation from South Bay.NET meetup on 3/30. Speaker: Matt Howlett, Software Engineer at Confluent Apache Kafka is a scalable streaming platform that forms a key part of the infrastructure at many companies including Uber, Netflix, Walmart, Airbnb, Goldman Sachs and LinkedIn. In this talk Matt will give a technical overview of Kafka, discuss some typical use cases (from surge pricing to fraud detection to web analytics) and show you how to use Kafka from within your C#/.NET applications.

Putting Kafka Together with the Best of Google Cloud Platform

confluent

(Kir Titievsky, Google) Kafka Summit SF 2018 In this talk we will share some stories and patterns from customers who have built streaming pipelines and event-driven systems using Confluent Cloud in combination with Google Cloud Platform-native analytics tools, such as BigQuery and Dataflow. We will discuss what Confluent Cloud enables for hybrid deployments and how and why to mix and match platform-native and platform-neutral tools.

ETL as a Platform: Pandora Plays Nicely Everywhere with Real-Time Data Pipelines

confluent

ETL can be painful with dirty data and outdated batch processes slowing you down; there has to be a better way. In this talk we’ll discuss the benefits of introducing a streaming platform to your architecture including how it can greatly simplify complexity, speed up performance, and help your team deliver the features they need with real-time data integration. Pandora’s Lawrence Weikum will discuss what they’ve done to bring real-time data integration to the team. We’ll review their Kafka-powered data pipelines and how they make the most of Kafka’s Connect API to make it surprisingly system to keep systems in sync. Presented by: Lawrence Weikum, Senior Software Engineer, Pandora Gehrig Kunz, Technical Product Marketing Manager, Confluent

Introduction to apache kafka, confluent and why they matter

Paolo Castagna

Kafka, Killer of Point-to-Point Integrations, Lucian Lita

confluent

With 60+ products and over 24% of the US GDP flowing through it, system integration is a tough problem for Intuit. Seasonality, scale, and massive peaks in products like TurboTax, QuickBooks, and Mint.com add extra layers of difficulty when building shared data services around transaction and user graphs, clickstream processing, a/b testing, and personalization. To reduce complexity and latency, we’ve implemented Kafka as the backbone across these data services. This allows us to asynchronously trigger relevant processing, elegantly scaling up and down as needed around peaks, all without the need for point-to-point integrations. In this talk, we share what we’ve learned about Kafka at Intuit and describe our data services architecture. We found that Kafka is invaluable in achieving a scalable, clean architecture, allowing engineering teams to focus less on integration and more on product development.

Using Apache Kafka to Analyze Session Windows

confluent

Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day

Ankur Bansal

Building data pipelines is pretty hard! Building a multi-datacenter active-active real time data pipeline for multiple classes of data with different durability, latency and availability guarantees is much harder. Real time infrastructure powers critical pieces of Uber (think Surge) and in this talk we will discuss our architecture, technical challenges, learnings and how a blend of open source infrastructure (Apache Kafka and Samza) and in-house technologies have helped Uber scale.

Using Location Data to Showcase Keys, Windows, and Joins in Kafka Streams DSL...

confluent

Kafka Streams and the addition of KSQL has provided opportunities do stateful processing of data. Sometimes, the biggest challenge is determining how you can join that data. Keying and windowing are core concepts that need to be understood in order to properly and efficiently stream data. In this presentation, Neil will utilize geospatial data to showcase non-trivial joining; particularly, but not limited to, distance comparisons. The stream processing will be written in Kafka Streams DSL and in KSQL with the topologies being compared. KSQL 2.0 concepts of User Defined Functions (UDFs), nested AVRO structures, and ‘insert into’ functionality of KSQL will be showcased. The presentation will show a custom OpenSky Connector for obtaining real-time aircraft, a Streams application for processing that data, a D3 topojson application to visualize the data, and an addition KSQL implementation of the streams application for comparison. Expect a deep dive into the Streams DSL and KSQL implementations that will provide the bases into a discussion around Apache Kafka and stream processing.

Siphon - Near Real Time Databus Using Kafka, Eric Boyd, Nitin Kumar

confluent

Siphon is a highly available and reliable distributed pub/sub system built using Apache Kafka. It is used to publish, discover and subscribe to near real-time data streams for operational and product intelligence. Siphon is used as a “Databus” by a variety of producers and subscribers in Microsoft, and is compliant with security and privacy requirements. It has a built-in Auditing and Quality control. This session will provide an overview of the use of Kafka at Microsoft, and then deep dive into Siphon. We will describe an important business scenario and talk about the technical details of the system in the context of that scenario. We will also cover the design and implementation of the service, the scale, and real world production experiences from operating the service in the Microsoft cloud environment.

From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...

confluent

Mark Teehan, Principal Solutions Engineer, Confluent Use the Debezium CDC connector to capture database changes from a Postgres database - or MySQL or Oracle; streaming into Kafka topics and onwards to an external data store. Examine how to setup this pipeline using Docker Compose and Confluent Cloud; and how to use various payload formats, such as avro, protobuf and json-schema. https://www.meetup.com/Singapore-Kafka-Meetup/events/276822852/

From a Million to a Trillion Events Per Day: Stream Processing in Ludicrous M...

confluent

In this talk we’ll describe the evolution of stream processing at Tesla and the challenges that are specific to our needs, such as large skews in message-processing latencies. We’ll describe how we built a reliable and performant ingestion platform that allows us to take an idea from a whiteboard to production in just a matter of hours. We’ll also discuss the design principles, tools, and incident response processes that have enabled a small team to support Kafka and downstream services in highly-available and multi-tenant environments at scale.

Introduction to Apache Kafka and Confluent... and why they matter

confluent

Milano Apache Kafka Meetup by Confluent (First Italian Kafka Meetup) on Wednesday, November 29th 2017. Il talk introduce Apache Kafka (incluse le APIs Kafka Connect e Kafka Streams), Confluent (la società creata dai creatori di Kafka) e spiega perché Kafka è un'ottima e semplice soluzione per la gestione di stream di dati nel contesto di due delle principali forze trainanti e trend industriali: Internet of Things (IoT) e Microservices.

Stream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data Platform

confluent

Many enterprises have a large technical debt in legacy applications hosted in on-premises data centers. There is a strong desire to modernize and move to a cloud-based infrastructure, but the world won’t stop for you to transition. Existing applications need to be supported and enhanced; data from legacy platforms is required to make decisions that drive the business. On the other hand, data from cloud-based applications does not exist in a vacuum. Legacy applications need access to these cloud data sources and vice versa. Can an enterprise have it both ways? Can new applications be built in the cloud while existing applications are maintained in a private data center? Monsanto has adopted a cloud-first mentality—today most new development is focused on the cloud. However, this transition did not happen overnight. Chrix Finne and Bob Lehmann share their experience building and implementing a Kafka-based cross-data-center streaming platform to facilitate the move to the cloud—in the process, kick-starting Monsanto’s transition from batch to stream processing. Details include an overview of the challenges involved in transitioning to the cloud and a deep dive into the cross-data-center stream platform architecture, including best practices for running this architecture in production and a summary of the benefits seen after deploying this architecture.

Metrics Are Not Enough: Monitoring Apache Kafka and Streaming Applications

confluent

When you are running systems in production, clearly you want to make sure they are up and running at all times. But in a distributed system such as Apache Kafka… what does “up and running” even mean? Experienced Apache Kafka users know what is important to monitor, which alerts are critical and how to respond to them. They don’t just collect metrics - they go the extra mile and use additional tools to validate availability and performance on both the Kafka cluster and their entire data pipelines. In this presentation we’ll discuss best practices of monitoring Apache Kafka. We’ll look at which metrics are critical to alert on, which are useful in troubleshooting and what may actually be misleading. We’ll review a few “worst practices” - common mistakes that you should avoid. We’ll then look at what metrics don’t tell you - and how to cover those essential gaps.

Streaming Data and Stream Processing with Apache Kafka

confluent

What's hot

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

confluent

Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...

Michael Noll

Kafka Summit SF 2017 - Real-Time Document Rankings with Kafka Streams

confluent

Introducing Apache Kafka's Streams API - Kafka meetup Munich, Jan 25 2017

Michael Noll

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

HostedbyConfluent

Kafka Summit NYC 2017 - Data Processing at LinkedIn with Apache Kafka

confluent

A Tour of Apache Kafka

confluent

Stream Processing with Apache Kafka and .NET

confluent

Putting Kafka Together with the Best of Google Cloud Platform

confluent

ETL as a Platform: Pandora Plays Nicely Everywhere with Real-Time Data Pipelines

confluent

Introduction to apache kafka, confluent and why they matter

Paolo Castagna

Kafka, Killer of Point-to-Point Integrations, Lucian Lita

confluent

Using Apache Kafka to Analyze Session Windows

confluent

Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day

Ankur Bansal

Using Location Data to Showcase Keys, Windows, and Joins in Kafka Streams DSL...

confluent

Siphon - Near Real Time Databus Using Kafka, Eric Boyd, Nitin Kumar

confluent

From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...

confluent

From a Million to a Trillion Events Per Day: Stream Processing in Ludicrous M...

confluent

Introduction to Apache Kafka and Confluent... and why they matter

confluent

Stream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data Platform

confluent

What's hot (20)

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...

Kafka Summit SF 2017 - Real-Time Document Rankings with Kafka Streams

Introducing Apache Kafka's Streams API - Kafka meetup Munich, Jan 25 2017

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

Kafka Summit NYC 2017 - Data Processing at LinkedIn with Apache Kafka

A Tour of Apache Kafka

Stream Processing with Apache Kafka and .NET

Putting Kafka Together with the Best of Google Cloud Platform

ETL as a Platform: Pandora Plays Nicely Everywhere with Real-Time Data Pipelines

Introduction to apache kafka, confluent and why they matter

Kafka, Killer of Point-to-Point Integrations, Lucian Lita

Using Apache Kafka to Analyze Session Windows

Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day

Using Location Data to Showcase Keys, Windows, and Joins in Kafka Streams DSL...

Siphon - Near Real Time Databus Using Kafka, Eric Boyd, Nitin Kumar

From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...

From a Million to a Trillion Events Per Day: Stream Processing in Ludicrous M...

Introduction to Apache Kafka and Confluent... and why they matter

Stream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data Platform

Viewers also liked

Metrics Are Not Enough: Monitoring Apache Kafka and Streaming Applications

confluent

Streaming Data and Stream Processing with Apache Kafka

confluent

Reliability Guarantees for Apache Kafka

confluent

Kafka Summit NYC 2017 - Running Hundreds of Kafka Clusters with 5 People

confluent

Building Microservices with Apache Kafka

confluent

Building distributed systems is challenging. Luckily, Apache Kafka provides a powerful toolkit for putting together big services as a set of scalable, decoupled components. In this talk, I'll describe some of the design tradeoffs when building microservices, and how Kafka's powerful abstractions can help. I'll also talk a little bit about what the community has been up to with Kafka Streams, Kafka Connect, and exactly-once semantics. Presentation by Colin McCabe, Confluent, Big Data Day LA

Disaster Recovery Plans for Apache Kafka

confluent

Running Apache Kafka in production is only the first step in the Kafka operations journey. Professional Kafka users are ready to handle all possible disasters - because for most businesses having a disaster recovery plan is not optional. In this session, we’ll discuss disaster scenarios that can take down entire Kafka clusters and share advice on how to plan, prepare and handle these events. This is a technical session full of best practices - we want to make sure you are ready to handle the worst mayhem that nature and auditors can cause. Visit www.confluent.io for more information.

Kafka Summit NYC 2017 - Building Advanced Streaming Applications using the La...

confluent

Exactly-once Semantics in Apache Kafka

confluent

Apache Kafka's rise in popularity as a streaming platform has demanded a revisit of its traditional at-least-once message delivery semantics. In this talk, we present the recent additions to Kafka to achieve exactly-once semantics (EoS) including support for idempotence and transactions in the Kafka clients. The main focus will be the specific semantics that Kafka distributed transactions enable and the underlying mechanics which allow them to scale efficiently.

Common Patterns of Multi Data-Center Architectures with Apache Kafka

confluent

Whether you know you want to run Apache Kafka in multiple data centers and need practical advice or you are wondering why some organizations even need more than one cluster, this online talk is for you. In this short session, we’ll discuss the basic patterns of multi-datacenter Kafka architectures, explore some of the use-cases enabled by each architecture and show how Confluent Enterprise products make these patterns easy to implement. Visit www.confluent.io for more information.

Viewers also liked (9)

Metrics Are Not Enough: Monitoring Apache Kafka and Streaming Applications

Streaming Data and Stream Processing with Apache Kafka

Reliability Guarantees for Apache Kafka

Kafka Summit NYC 2017 - Running Hundreds of Kafka Clusters with 5 People

Building Microservices with Apache Kafka

Disaster Recovery Plans for Apache Kafka

Kafka Summit NYC 2017 - Building Advanced Streaming Applications using the La...

Exactly-once Semantics in Apache Kafka

Common Patterns of Multi Data-Center Architectures with Apache Kafka

Similar to Real-world Streaming Architectures

Kafka Connect and Streams (Concepts, Architecture, Features)

Kai Wähner

Typesafe & William Hill: Cassandra, Spark, and Kafka - The New Streaming Data...

DataStax Academy

Typesafe did a survey of Spark usage last year and found that a large percentage of Spark users combine it with Cassandra and Kafka. This talk focuses on streaming data scenarios that demonstrate how these three tools complement each other for building robust, scalable, and flexible data applications. Cassandra provides resilient and scalable storage, with flexible data format and query options. Kafka provides durable, scalable collection of streaming data with message-queue semantics. Spark provides very flexible analytics, everything from classic SQL queries to machine learning and graph algorithms, running in a streaming model based on "mini-batches", offline batch jobs, or interactive queries. We'll consider best practices and areas where improvements are needed.

AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017

Monal Daxini

Over 100 million subscribers from over 190 countries enjoy the Netflix service. This leads to over a trillion events, amounting to 3 PB, flowing through the Keystone infrastructure to help improve customer experience and glean business insights. The self-serve Keystone stream processing service processes these messages in near real-time with at-least once semantics in the cloud. This enables the users to focus on extracting insights, and not worry about building out scalable infrastructure. I’ll share the details about this platform, and our experience building it.

You Must Construct Additional Pipelines: Pub-Sub on Kafka at Blizzard

confluent

(Stephen Parente + Jeff Field, Blizzard) Kafka Summit SF 2018 Blizzard’s global data platform has become a driving force in both business and operational analytics. As more internal customers onboard with the system, there is increasing demand for custom applications to access this data in near real time. In order to avoid many independent teams with varying levels of Kafka expertise all accessing the firehose from our critical production Kafkas, we developed our own pub-sub system on top of Kafka to provide specific datasets to customers on their own cloud deployed Kafka clusters.

ksqlDB: A Stream-Relational Database System

confluent

Speaker: Matthias J. Sax, Software Engineer, Confluent ksqlDB is a distributed event streaming database system that allows users to express SQL queries over relational tables and event streams. The project was released by Confluent in 2017 and is hosted on Github and developed with an open-source spirit. ksqlDB is built on top of Apache Kafka®, a distributed event streaming platform. In this talk, we discuss ksqlDB’s architecture that is influenced by Apache Kafka and its stream processing library, Kafka Streams. We explain how ksqlDB executes continuous queries while achieving fault tolerance and high vailability. Furthermore, we explore ksqlDB’s streaming SQL dialect and the different types of supported queries. Matthias J. Sax is a software engineer at Confluent working on ksqlDB. He mainly contributes to Kafka Streams, Apache Kafka's stream processing library, which serves as ksqlDB's execution engine. Furthermore, he helps evolve ksqlDB's "streaming SQL" language. In the past, Matthias also contributed to Apache Flink and Apache Storm and he is an Apache committer and PMC member. Matthias holds a Ph.D. from Humboldt University of Berlin, where he studied distributed data stream processing systems. https://db.cs.cmu.edu/events/quarantine-db-talk-2020-confluent-ksqldb-a-stream-relational-database-system/

Etl is Dead; Long Live Streams

confluent

From a kafkaesque story to The Promised Land

Ran Silberman

Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...

confluent

(Bob Lehmann, Bayer) Kafka Summit SF 2018 You’ve built your streaming data platform. The early adopters are “all in” and have developed producers, consumers and stream processing apps for a number of use cases. A large percentage of the enterprise, however, has expressed interest but hasn’t made the leap. Why? In 2014, Bayer Crop Science (formerly Monsanto) adopted a cloud first strategy and started a multi-year transition to the cloud. A Kafka-based cross-datacenter DataHub was created to facilitate this migration and to drive the shift to real-time stream processing. The DataHub has seen strong enterprise adoption and supports a myriad of use cases. Data is ingested from a wide variety of sources and the data can move effortlessly between an on premise datacenter, AWS and Google Cloud. The DataHub has evolved continuously over time to meet the current and anticipated needs of our internal customers. The “cost of admission” for the platform has been lowered dramatically over time via our DataHub Portal and technologies such as Kafka Connect, Kubernetes and Presto. Most operations are now self-service, onboarding of new data sources is relatively painless and stream processing via KSQL and other technologies is being incorporated into the core DataHub platform. In this talk, Bob Lehmann will describe the origins and evolution of the Enterprise DataHub with an emphasis on steps that were taken to drive user adoption. Bob will also talk about integrations between the DataHub and other key data platforms at Bayer, lessons learned and the future direction for streaming data and stream processing at Bayer.

Lambda Architecture with Spark Streaming, Kafka, Cassandra, Akka, Scala

Helena Edelson

Scala Days, Amsterdam, 2015: Lambda Architecture - Batch and Streaming with Spark, Cassandra, Kafka, Akka and Scala; Fault Tolerance, Data Pipelines, Data Flows, Data Locality, Akka Actors, Spark, Spark Cassandra Connector, Big Data, Asynchronous data flows. Time series data, KillrWeather, Scalable Infrastructure, Partition For Scale, Replicate For Resiliency, Parallelism Isolation, Data Locality, Location Transparency

Streaming Analytics with Spark, Kafka, Cassandra and Akka

Helena Edelson

This talk will address how a new architecture is emerging for analytics, based on Spark, Mesos, Akka, Cassandra and Kafka (SMACK). Popular architecture like Lambda separate layers of computation and delivery and require many technologies which have overlapping functionality. Some of this results in duplicated code, untyped processes, or high operational overhead, let alone the cost (i.e. ETL). I will discuss the problem domain and what is needed in terms of strategies, architecture and application design and code to begin leveraging simpler data flows. We will cover how the particular set of technologies addresses common requirements and how collaboratively they work together to enrich and reinforce each other.

Data Stream Analytics - Why they are important

Paris Carbone

Big Data LDN 2017: Look Ma, No Code! Building Streaming Data Pipelines With A...

Matt Stubbs

Kafka Summit NYC 2017 Introduction to Kafka Streams with a Real-life Example

confluent

BBL KAPPA Lesfurets.com

Cedric Vidal

Streaming Analytics with Spark, Kafka, Cassandra and Akka by Helena Edelson

Spark Summit

Netflix Keystone—Cloud scale event processing pipeline

Monal Daxini

Pipes & Filters Architectural Pattern

Fredrik Kivi

Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...

Michael Noll

My talk at Google DevFest Switzerland, Fribourg, Oct 2017. https://devfest.ch/schedule/day1?sessionId=118 Abstract: Modern businesses have data at their core, and this data is changing continuously. How can we harness this torrent of information in real-time? The answer is stream processing, and the technology that has since become the core platform for streaming data is Apache Kafka. Among the thousands of companies that use Kafka to transform and reshape their industries are the likes of Netflix, Uber, PayPal, and AirBnB, but also established players such as Goldman Sachs, Cisco, and Oracle. Unfortunately, today’s common architectures for real-time data processing at scale suffer from complexity: there are many technologies that need to be stitched and operated together, and each individual technology is often complex by itself. This has led to a strong discrepancy between how we, as engineers, would like to work vs. how we actually end up working in practice. In this session we talk about how Apache Kafka helps you to radically simplify your data architectures. We cover how you can now build normal applications to serve your real-time processing needs — rather than building clusters or similar special-purpose infrastructure — and still benefit from properties such as high scalability, distributed computing, and fault-tolerance, which are typically associated exclusively with cluster technologies. We discuss common use cases to realize that stream processing in practice often requires database-like functionality, and how Kafka allows you to bridge the worlds of streams and databases when implementing your own core business applications (inventory management for large retailers, patient monitoring in healthcare, fleet tracking in logistics, etc), for example in the form of event-driven, containerized microservices. We will also give a brief shout-out to the recently launched KSQL, a streaming SQL engine for Apache Kafka.

Chicago Kafka Meetup

Cliff Gilmore

Confluent Kafka and KSQL: Streaming Data Pipelines Made Easy

Kairo Tavares

Similar to Real-world Streaming Architectures (20)

Kafka Connect and Streams (Concepts, Architecture, Features)

Typesafe & William Hill: Cassandra, Spark, and Kafka - The New Streaming Data...

AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017

You Must Construct Additional Pipelines: Pub-Sub on Kafka at Blizzard

ksqlDB: A Stream-Relational Database System

Etl is Dead; Long Live Streams

From a kafkaesque story to The Promised Land

Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...

Lambda Architecture with Spark Streaming, Kafka, Cassandra, Akka, Scala

Streaming Analytics with Spark, Kafka, Cassandra and Akka

Data Stream Analytics - Why they are important

Big Data LDN 2017: Look Ma, No Code! Building Streaming Data Pipelines With A...

Kafka Summit NYC 2017 Introduction to Kafka Streams with a Real-life Example

BBL KAPPA Lesfurets.com

Streaming Analytics with Spark, Kafka, Cassandra and Akka by Helena Edelson

Netflix Keystone—Cloud scale event processing pipeline

Pipes & Filters Architectural Pattern

Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...

Chicago Kafka Meetup

Confluent Kafka and KSQL: Streaming Data Pipelines Made Easy

More from confluent

Speed Wins: From Kafka to APIs in Minutes

confluent

Evolving Data Governance for the Real-time Streaming and AI Era

confluent

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

confluent

Santander Stream Processing with Apache Flink

confluent

Unlocking the Power of IoT: A comprehensive approach to real-time insights

confluent

Workshop híbrido: Stream Processing con Flink

confluent

El Stream processing es un requisito previo de la pila de data streaming, que impulsa aplicaciones y pipelines en tiempo real. Permite una mayor portabilidad de datos, una utilización optimizada de recursos y una mejor experiencia del cliente al procesar flujos de datos en tiempo real. En nuestro taller práctico híbrido, aprenderás cómo filtrar, unir y enriquecer fácilmente datos en tiempo real dentro de Confluent Cloud utilizando nuestro servicio Flink sin servidor.

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

confluent

Our talk will explore the transformative impact of integrating Confluent, HiveMQ, and SparkPlug in Industry 4.0, emphasizing the creation of a Unified Namespace. In addition to the creation of a Unified Namespace, our webinar will also delve into Stream Governance and Scaling, highlighting how these aspects are crucial for managing complex data flows and ensuring robust, scalable IIoT-Platforms. You will learn how to ensure data accuracy and reliability, expand your data processing capabilities, and optimize your data management processes. Don't miss out on this opportunity to learn from industry experts and take your business to the next level.

AWS Immersion Day Mapfre - Confluent

confluent

La arquitectura impulsada por eventos (EDA) será el corazón del ecosistema de MAPFRE. Para seguir siendo competitivas, las empresas de hoy dependen cada vez más del análisis de datos en tiempo real, lo que les permite obtener información y tiempos de respuesta más rápidos. Los negocios con datos en tiempo real consisten en tomar conciencia de la situación, detectar y responder a lo que está sucediendo en el mundo ahora.

Eventos y Microservicios - Santander TechTalk

confluent

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

confluent

Citi TechTalk Session 2: Kafka Deep Dive

confluent

Build real-time streaming data pipelines to AWS with Confluent

confluent

Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.

Q&A with Confluent Professional Services: Confluent Service Mesh

confluent

Citi Tech Talk: Event Driven Kafka Microservices

confluent

Confluent & GSI Webinars series - Session 3

confluent

An in depth look at how Confluent is being used in the financial services industry. Gain an understanding of how organisations are utilising data in motion to solve common problems and gain benefits from their real time data capabilities. It will look more deeply into some specific use cases and show how Confluent technology is used to manage costs and mitigate risks. This session is aimed at Solutions Architects, Sales Engineers and Pre Sales, and also the more technically minded business aligned people. Whilst this is not a deeply technical session, a level of knowledge around Kafka would be helpful.

Citi Tech Talk: Messaging Modernization

confluent

Transforming applications built with traditional messaging solutions such as TIBCO, MQ and Solace to be scalable, reliable and ready for the move to cloud How can applications built with traditional messaging technologies like TIBCO, Solace and IBM MQ be modernised and be made cloud ready? What are the advantages to Event Streaming approaches to pub/sub vs traditional message queues? What are the strengeths and weaknesses of both approaches, and what use cases and requirements are actually a better fit for messaging than Kafka?

Citi Tech Talk: Data Governance for streaming and real time data

confluent

Confluent & GSI Webinars series: Session 2

confluent

Data In Motion Paris 2023

confluent

Vous apprendrez également à : • Créer plus rapidement des produits et fonctionnalités à l’aide d’une suite complète de connecteurs et d’outils de gestion des flux, et à connecter vos environnements à des pipelines de données • Protéger vos données et charges de travail les plus critiques grâce à des garanties intégrées en matière de sécurité, de gouvernance et de résilience • Déployer Kafka à grande échelle en quelques minutes tout en réduisant les coûts et la charge opérationnelle associés

Confluent Partner Tech Talk with Synthesis

confluent

More from confluent (20)

Speed Wins: From Kafka to APIs in Minutes

Evolving Data Governance for the Real-time Streaming and AI Era

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

Santander Stream Processing with Apache Flink

Unlocking the Power of IoT: A comprehensive approach to real-time insights

Workshop híbrido: Stream Processing con Flink

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

AWS Immersion Day Mapfre - Confluent

Eventos y Microservicios - Santander TechTalk

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

Citi TechTalk Session 2: Kafka Deep Dive

Build real-time streaming data pipelines to AWS with Confluent

Q&A with Confluent Professional Services: Confluent Service Mesh

Citi Tech Talk: Event Driven Kafka Microservices

Confluent & GSI Webinars series - Session 3

Citi Tech Talk: Messaging Modernization

Citi Tech Talk: Data Governance for streaming and real time data

Confluent & GSI Webinars series: Session 2

Data In Motion Paris 2023

Confluent Partner Tech Talk with Synthesis

Recently uploaded

Globus Connect Server Deep Dive - GlobusWorld 2024

Globus

A Comprehensive Look at Generative AI in Retail App Testing.pdf

kalichargn70th171

Quarkus Hidden and Forbidden Extensions

Max Andersen

Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...

informapgpstrackings

Corporate Management | Session 3 of 3 | Tendenci AMS

Tendenci - The Open Source AMS (Association Management Software)

Experience our free, in-depth three-part Tendenci Platform Corporate Membership Management workshop series! In Session 1 on May 14th, 2024, we began with an Introduction and Setup, mastering the configuration of your Corporate Membership Module settings to establish membership types, applications, and more. Then, on May 16th, 2024, in Session 2, we focused on binding individual members to a Corporate Membership and Corporate Reps, teaching you how to add individual members and assign Corporate Representatives to manage dues, renewals, and associated members. Finally, on May 28th, 2024, in Session 3, we covered questions and concerns, addressing any queries or issues you may have. For more Tendenci AMS events, check out www.tendenci.com/events

Globus Compute wth IRI Workflows - GlobusWorld 2024

Globus

As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.

Navigating the Metaverse: A Journey into Virtual Evolution"

Donna Lenk

top nidhi software solution freedownload

vrstrong314

This presentation emphasizes the importance of data security and legal compliance for Nidhi companies in India. It highlights how online Nidhi software solutions, like Vector Nidhi Software, offer advanced features tailored to these needs. Key aspects include encryption, access controls, and audit trails to ensure data security. The software complies with regulatory guidelines from the MCA and RBI and adheres to Nidhi Rules, 2014. With customizable, user-friendly interfaces and real-time features, these Nidhi software solutions enhance efficiency, support growth, and provide exceptional member services. The presentation concludes with contact information for further inquiries.

Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...

Anthony Dahanne

Les Buildpacks existent depuis plus de 10 ans ! D’abord, ils étaient utilisés pour détecter et construire une application avant de la déployer sur certains PaaS. Ensuite, nous avons pu créer des images Docker (OCI) avec leur dernière génération, les Cloud Native Buildpacks (CNCF en incubation). Sont-ils une bonne alternative au Dockerfile ? Que sont les buildpacks Paketo ? Quelles communautés les soutiennent et comment ? Venez le découvrir lors de cette session ignite

AI Pilot Review: The World’s First Virtual Assistant Marketing Suite

Google

AI Pilot Review: The World’s First Virtual Assistant Marketing Suite 👉👉 Click Here To Get More Info 👇👇 https://sumonreview.com/ai-pilot-review/ AI Pilot Review: Key Features ✅Deploy AI expert bots in Any Niche With Just A Click ✅With one keyword, generate complete funnels, websites, landing pages, and more. ✅More than 85 AI features are included in the AI pilot. ✅No setup or configuration; use your voice (like Siri) to do whatever you want. ✅You Can Use AI Pilot To Create your version of AI Pilot And Charge People For It… ✅ZERO Manual Work With AI Pilot. Never write, Design, Or Code Again. ✅ZERO Limits On Features Or Usages ✅Use Our AI-powered Traffic To Get Hundreds Of Customers ✅No Complicated Setup: Get Up And Running In 2 Minutes ✅99.99% Up-Time Guaranteed ✅30 Days Money-Back Guarantee ✅ZERO Upfront Cost See My Other Reviews Article: (1) TubeTrivia AI Review: https://sumonreview.com/tubetrivia-ai-review (2) SocioWave Review: https://sumonreview.com/sociowave-review (3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review (4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review

SOCRadar Research Team: Latest Activities of IntelBroker

SOCRadar

The European Union Agency for Law Enforcement Cooperation (Europol) has suffered an alleged data breach after a notorious threat actor claimed to have exfiltrated data from its systems. Infamous data leaker IntelBroker posted on the even more infamous BreachForums hacking forum, saying that Europol suffered a data breach this month. The alleged breach affected Europol agencies CCSE, EC3, Europol Platform for Experts, Law Enforcement Forum, and SIRIUS. Infiltration of these entities can disrupt ongoing investigations and compromise sensitive intelligence shared among international law enforcement agencies. However, this is neither the first nor the last activity of IntekBroker. We have compiled for you what happened in the last few days. To track such hacker activities on dark web sources like hacker forums, private Telegram channels, and other hidden platforms where cyber threats often originate, you can check SOCRadar’s Dark Web News. Stay Informed on Threat Actors’ Activity on the Dark Web with SOCRadar!

GlobusWorld 2024 Opening Keynote session

Globus

Beyond Event Sourcing - Embracing CRUD for Wix Platform - Java.IL

Natan Silnitsky

In software engineering, the right architecture is essential for robust, scalable platforms. Wix has undergone a pivotal shift from event sourcing to a CRUD-based model for its microservices. This talk will chart the course of this pivotal journey. Event sourcing, which records state changes as immutable events, provided robust auditing and "time travel" debugging for Wix Stores' microservices. Despite its benefits, the complexity it introduced in state management slowed development. Wix responded by adopting a simpler, unified CRUD model. This talk will explore the challenges of event sourcing and the advantages of Wix's new "CRUD on steroids" approach, which streamlines API integration and domain event management while preserving data integrity and system resilience. Participants will gain valuable insights into Wix's strategies for ensuring atomicity in database updates and event production, as well as caching, materialization, and performance optimization techniques within a distributed system. Join us to discover how Wix has mastered the art of balancing simplicity and extensibility, and learn how the re-adoption of the modest CRUD has turbocharged their development velocity, resilience, and scalability in a high-growth environment.

2024 RoOUG Security model for the cloud.pptx

Georgi Kodinov

Providing Globus Services to Users of JASMIN for Environmental Data Analysis

Globus

JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.

Accelerate Enterprise Software Engineering with Platformless

WSO2

Key takeaways: Challenges of building platforms and the benefits of platformless. Key principles of platformless, including API-first, cloud-native middleware, platform engineering, and developer experience. How Choreo enables the platformless experience. How key concepts like application architecture, domain-driven design, zero trust, and cell-based architecture are inherently a part of Choreo. Demo of an end-to-end app built and deployed on Choreo.

Globus Compute Introduction - GlobusWorld 2024

Globus

Lecture 1 Introduction to games development

abdulrafaychaudhry

Enhancing Research Orchestration Capabilities at ORNL.pdf

Globus

Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.

RISE with SAP and Journey to the Intelligent Enterprise

Srikant77

Recently uploaded (20)

Globus Connect Server Deep Dive - GlobusWorld 2024

A Comprehensive Look at Generative AI in Retail App Testing.pdf

Quarkus Hidden and Forbidden Extensions

Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...

Corporate Management | Session 3 of 3 | Tendenci AMS

Globus Compute wth IRI Workflows - GlobusWorld 2024

Navigating the Metaverse: A Journey into Virtual Evolution"

top nidhi software solution freedownload

Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...

AI Pilot Review: The World’s First Virtual Assistant Marketing Suite

SOCRadar Research Team: Latest Activities of IntelBroker

GlobusWorld 2024 Opening Keynote session

Beyond Event Sourcing - Embracing CRUD for Wix Platform - Java.IL

2024 RoOUG Security model for the cloud.pptx

Providing Globus Services to Users of JASMIN for Environmental Data Analysis

Accelerate Enterprise Software Engineering with Platformless

Globus Compute Introduction - GlobusWorld 2024

Lecture 1 Introduction to games development

Enhancing Research Orchestration Capabilities at ORNL.pdf

RISE with SAP and Journey to the Intelligent Enterprise

Real-world Streaming Architectures

1. 1 @tlberglund Four Streaming Data Architectures

2. 2 The Old Story

3. 3

4. 4

5. 5

6. 6

7. 7 The Streaming Story

8. 8Maria-José Viñas, NASA Earth Science News Team.

9. 9Image credit: NASA/JPL/UCSD/JSC

10. 10

11. 11 Kafka

12. 12

13. 13

14. 14

15. 15 Examples

16. 16 Yelp https://engineeringblog.yelp.com/2016/07/billions-of-messages-a-day-yelps-real-time-data-pipeline.html Justin C., Software Engineer (May, 2016)

17. 17 • 150 production services • Metcalf’s Law • REST scales poorly • N+1 problem and bulk data services A Streaming Backbone

18. 18 💩 💩 💩 💩 💩 💩 💩

19. 19

20. 20 • Stream-table duality • But some services still want databases • Individual services use MySQL • Millions of rows across services Streaming Databases

21. 21 • In 2010, commitment to MapReduce was strong • That fun was soon over • Want to capitalize on schema translation tools • Want near-real-time results • Storm didn’t support enough Python • Wrote their own! Computing Streams

22. 22 PaaStorm Instance

23. 23 Yelp Conclusions • Streaming pipeline ensures evolvability • Stream/Table Duality • Schema matters • Yelp often ahead of the game!

24. 24 LINE Corp https://engineering.linecorp.com/en/blog/detail/80

25. 25 Goals • Unified data delivery pipeline between systems • Replace a legacy, bespoke system for processing background tasks • Had to pick a framework

26. 26 Option: Apache Samza™

27. 27 Option: Apache Kafka™ Streams

28. 28 KStreamBuilder builder = new KStreamBuilder(); // Read the input Kafka topic into a KStream instance. KStream<byte[], String> textLines = builder.stream("TextLinesTopic"); // Convert to upper case (:: is Java 8 syntax) KStream<byte[], String> uppercasedWithMapValues = textLines.mapValues(String::toUpperCase); // Write the results to a new Kafka topic called "UppercasedTextLinesTopic". uppercasedWithMapValues.to("UppercasedTextLinesTopic"); KafkaStreams streams = new KafkaStreams(builder, streamsConfiguration); streams.start(); Streaming API for Kafka

29. 29 Streaming API for Kafka Stateful Transformations

30. 30 Loopback Replicator • One big input topic of all TalkOperations (1M messages/sec) • Some services want a limited view of these • Simple filtering

31. 31 Decaton Source: LINE Corp

32. 32 LINE Conclusions • Classical Kafka Streams implementation • Better to “buy” than build

33. 33 Altspace VR

34. 34

35. 35

36. 36

37. 37

38. 38

39. 39 Altspace Architecture • Photon game engine • Real-time play • Capture for playback • An entirely classical stream processing system

40. 40 “Mirror User A to room R2” Source: Altspace

41. 41 6 months later: “Capture User A” Source: Altspace

42. 42 Playbacks • Source capture data from Kafka or S3 • “Produce” it in a timed fashion

43. 43 “Playback capture to room R2” Source: Altspace

44. 44

45. 45 The Dream Stream

46. 46

47. 47

48. 48

49. 49

50. 50

51. 51

52. 52

53. 53

54. 54

55. 55

56. 56

57. 57

58. 58

59. 59

60. 60

61. 61

62. 62

63. 63

64. 64

65. 65

66. 66

67. 67

68. 68

69. 69 Get Started with Apache Kafka Today! THE place to start with Apache Kafka! Thoroughly tested and quality assured More extensible developer experience Easy upgrade path to Confluent Enterprise https://www.confluent.io/download/

70. 70 Confluent Community Join us in Slack https://slackpass.io/confluentcommunity Find your local Meetup Group https://www.confluent.io/apache-kafka-meetups/

71. 71 Thank you @tlberglund

Real-world Streaming Architectures

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (9)

Similar to Real-world Streaming Architectures

Similar to Real-world Streaming Architectures (20)

More from confluent

More from confluent (20)

Recently uploaded

Recently uploaded (20)

Real-world Streaming Architectures