Kai Wähner, Technology Evangelist at Confluent: "Development of Scalable Machine Learning Microservices with Apache Kafka Streams and H2O.ai"

•

1 like•298 views

Kai Wähner, Technology Evangelist at Confluent: "Development of Scalable Machine Learning Microservices with Apache Kafka Streams and H2O.ai" Kai Wähner works as Technology Evangelist at Confluent. Kai’s main area of expertise lies within the fields of Big Data Analytics, Machine Learning, Integration, Microservices, Internet of Things, Stream Processing and Blockchain. He is regular speaker at international conferences such as JavaOne, O’Reilly Software Architecture or ApacheCon, writes articles for professional journals, and shares his experiences with new technologies on his blog (www.kai-waehner.de/blog). Contact and references: kontakt@kai-waehner.de / @KaiWaehner / www.kai-waehner.de Abstract: Big Data and Machine Learning are key for innovation in many industries today. The first part of this session explains how to build analytic models with R, Python or Scala leveraging open source machine learning / deep learning frameworks like Apache Spark, TensorFlow or H2O.ai. The second part discusses the deployment of these built analytic models to your own applications or microservices leveraging the Apache Kafka cluster and Kafka Streams. The session focuses on live demos and teaches lessons learned for executing analytic models in a highly scalable and performant way.

Data & Analytics

1Confidential
Apache Kafka + H2O.ai
Machine Learning Applied to Real Time Stream Processing
Kai Waehner
Technology Evangelist
kontakt@kai-waehner.de
LinkedIn
@KaiWaehner
www.kai-waehner.de

2Apache Kafka and Machine Learning
Agenda
1) Machine Learning and Real Time Applications
2) Building an Analytic Model with H2O.ai
3) Applying an Analytic Model with Apache Kafka

3Apache Kafka and Machine Learning
Agenda
1) Machine Learning and Real Time Applications
2) Building an Analytic Model with H2O.ai
3) Applying an Analytic Model with Apache Kafka

4Apache Kafka and Machine Learning
Machine Learning
... allows computers to find hidden insights without being
explicitly programmed where to look.

5Apache Kafka and Machine Learning
Real World Examples of Machine Learning
Spam Detection
Search Results +
Product Recommendation
Picture Detection
(Friends, Locations, Products)
Your Company
The Next Disruption:
Google Beats Go Champion

6Apache Kafka and Machine Learning
Leverage Machine Learning to Analyze and Act on Critical Business Moments
Seconds Minutes Hours
Price
Optimization
Predictive
Maintenance
Fraud
Detection
Cross Selling
Transportation
Rerouting
Customer
Service
Inventory
Management
Windows of Opportunity

7Apache Kafka and Machine Learning
Big Data Analytics for Actionable Insights
From Insight to Action
(continuous loop)

8Apache Kafka and Machine Learning
Streaming Platform
Big Data Analytics
Database
IoT Device
Streaming
Producer
…..
DWH
Data
Integration
C
O
N
N
E
C
T
C
O
N
N
E
C
T
Data Lake
Model
Building
Batch
Real
Time
Stream
Processing
REST
Interface
IoT Device
Mobile
App
Streaming
Consumer
C
O
N
N
E
C
T
C
O
N
N
E
C
T
BI Tool
Messaging
Web
Application
Model
Schema Registry
/ Governance
1) Data Producer
2) Analytics Platform
3) Streaming Platform
4) Data Consumer

9Apache Kafka and Machine Learning
Agenda
1) Machine Learning and Real Time Applications
2) Building an Analytic Model with H2O.ai
3) Applying an Analytic Model with Apache Kafka

10Apache Kafka and Machine Learning
STREAMING PLATFORM
BIG DATA ANALYTICS
Oracle DB
CoaP IoT
Kafka
Java Client
…..
HP Vertica
Data
Integration
F
L
U
M
E
H2O.ai,
TensorFlow
Batch
Real
Time
Confluent
REST Proxy
MQTT IoT
iPhone
App
Kafka
Go Client
C
K O
A N
F N
K E
A C
T
H
I
V
E
Grafana
Kafka
Java EE
Web App
Hadoop
C
K O
A N
F N
K E
A C
T
Confluent
Schema Registry
Kafka Streams
H2O.ai
Mesos
Kafka Streams
TensorFlow
Kubernetes
Avro
Avro
1) Data Producer
2) Analytics Platform
3) Streaming Platform
4) Data Consumer

11Apache Kafka and Machine Learning
Languages, Frameworks and Tools
Many more ….
Portable Format
for Analytics (PFA)

12Apache Kafka and Machine Learning
Live Demo
Use Case:
Airline Flight Delay Prediction
Machine Learning Algorithm:
Gradient Boosted Machines (GBM)
using Decision Trees
Technology:
H2O.ai

13Apache Kafka and Machine Learning
Agenda
1) Machine Learning and Real Time Applications
2) Building an Analytic Model with H2O.ai
3) Applying an Analytic Model with Apache Kafka

14Apache Kafka and Machine Learning
STREAMING PLATFORM
BIG DATA ANALYTICS
Oracle DB
CoaP IoT
Kafka
Java Client
…..
HP Vertica
Data
Integration
F
L
U
M
E
H2O.ai,
TensorFlow
Batch
Real
Time
Confluent
REST Proxy
MQTT IoT
iPhone
App
Kafka
Go Client
C
K O
A N
F N
K E
A C
T
H
I
V
E
Grafana
Kafka
Java EE
Web App
Hadoop
C
K O
A N
F N
K E
A C
T
Confluent
Schema Registry
Kafka Streams
H2O.ai
Mesos
Kafka Streams
TensorFlow
Kubernetes
Avro
Avro
1) Data Producer
2) Analytics Platform
3) Streaming Platform
4) Data Consumer

15Apache Kafka and Machine Learning
When to use Kafka Streams for Stream Processing?
No need for a
Big Data cluster
Deploy in your
existing infrastructure
Kafka manages
scalability / fail-over
Focus on development
of business logic
in your department

16Apache Kafka and Machine Learning
Use Case:
Airline Flight Delay Prediction
Machine Learning Algorithm:
Any! (in our example, H2O.ai GBM)
Streaming Platform:
Apache Kafka Core, Kafka’s Streams API
Live Demo with Open Source Technologies

17Apache Kafka and Machine Learning
H2O.ai Model + Kafka Streams
Filter
Map
1) Create H2O ML model
2) Configure Kafka Streams Application
3) Apply H2O ML model to Streaming Data
4) Start Kafka Streams App

18Apache Kafka and Machine Learning
Online Model Training
How to improve models?
1.Manual Update
2.Automated Batch
3.Real Time
à Apache Kafka for Messaging and Real Time Apps

19Apache Kafka and Machine Learning
Caveats for Online Model Training
• Processes and infrastructure not ready
• Validation needed before production
• Slows down the system
• Only a few ML implementations
supported
• Many use cases do not need it

20Apache Kafka and Machine Learning
Kai Waehner
Technology Evangelist
kontakt@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
LinkedIn
Questions? Feedback?
Please contact me!

To stay informed about the latest features in Confluent Platform 5.4 join Martijn Kieboom Solutions Engineer at Confluent, for the ‘What’s New in Confluent 5.4?’ on February 12 at 11 am GMT/ 12 Noon CET. Martijn will talk through the new features including: Role-Based Access Control and how it enables highly granular control of permissions and platform access Structured Audit Logs and how they enable the capture of authorization logs How Multi-Region Clusters deliver asynchronous replication at the topic level, allowing companies to run a single Kafka Cluster across multiple data-centres Schema validations role in enabling businesses that run Kafka at scale to deliver data compatibility across platforms

Time series-analysis-using-an-event-streaming-platform -_v3_final

confluent

(1) The document discusses using an event streaming platform like Apache Kafka for advanced time series analysis (TSA). Typical processing patterns are described for converting raw data into time series and reconstructing graphs and networks from time series data. (2) A challenge discussed is integrating data streams, experiments, and decision making. The document argues that stream processing using Kafka is better suited than batch processing for real-time applications and iterative research projects. (3) The document then covers approaches for TSA and network analysis using Kafka, including creating time series from event streams, creating graphs from time series pairs, and architectures using reusable building blocks for complex stream processing.

Data Transformations on Ops Metrics using Kafka Streams (Srividhya Ramachandr...

confluent

This document discusses using Kafka Streams to transform operational metrics data from Priceline applications before loading it into Splunk. It describes how the legacy monitoring system worked, and the motives for moving to Kafka and Kafka Streams. It then explains how data is collected from applications into Kafka topics, and how various transformations like formatting, key application, and aggregation are performed using Kafka Streams before loading to Splunk. It also discusses testing, monitoring, and debugging Kafka Streams applications.

Westpac Bank Tech Talk 2: Introduction to Streaming Data and Stream Processin...

confluent

Leveraging Mainframe Data for Modern Analytics

confluent

The document provides an overview of leveraging mainframe data for modern analytics using Attunity Replicate and Confluent streaming platform powered by Apache Kafka. It discusses the history of mainframes and data migration, how Attunity enables real-time data migration from mainframes, the Confluent streaming platform for building applications using data streams, and how Attunity and Confluent can be combined to modernize analytics using mainframe data streams. Use cases discussed include query offloading and cross-system customer data integration.

Kafka, Killer of Point-to-Point Integrations, Lucian Lita

confluent

With 60+ products and over 24% of the US GDP flowing through it, system integration is a tough problem for Intuit. Seasonality, scale, and massive peaks in products like TurboTax, QuickBooks, and Mint.com add extra layers of difficulty when building shared data services around transaction and user graphs, clickstream processing, a/b testing, and personalization. To reduce complexity and latency, we’ve implemented Kafka as the backbone across these data services. This allows us to asynchronously trigger relevant processing, elegantly scaling up and down as needed around peaks, all without the need for point-to-point integrations. In this talk, we share what we’ve learned about Kafka at Intuit and describe our data services architecture. We found that Kafka is invaluable in achieving a scalable, clean architecture, allowing engineering teams to focus less on integration and more on product development.

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020

HostedbyConfluent

One of the key metrics to monitor when working with Apache Kafka, as a data pipeline or a streaming platform, is Consumer Groups Lag. Lag is the delta between the last produced message and the last committed message of a partition. In other words, lag indicates how far behind your application is in processing up-to-date information. For a long time, we used our own service to keep track of these metrics, collect them and visualize them. But this didn’t scale well. You had to perform many manual operations, redeploy it and to do other tedious manual tasks, but most importantly, the biggest gap for us, was that its output was represented in absolute numbers (e.g - your lag is 30K), which basically tells you nothing as a human being. We understood that we had to find a more suitable solution that will give us better visibility and will allow us to measure the lag in a time-based format that we all understand. In this talk, I’m going to go over the core concepts of Kafka offsets and lags, and explain why lag even matters and is an important KPI to measure. I’ll also talk about the kind of research we did to find the right tool, what the options in the market were at the time, and eventually why we chose Linkedin’s Burrow as the right tool for us. And finally, I’ll take a closer look at Burrow, its building blocks, how we build and deploy it, how we monitor better with it, and eventually the most important improvement - how we transformed its output from numbers to time-based metrics.

The document discusses 3 realities of modern programming: 1) The rise of managed services where over half of Kafka users are using cloud versions. 2) Data is exploding in volume and streaming is needed. 3) Microservices have increased in popularity but communication between services can be complex; Kafka helps solve this as a backbone. Yelp moved to microservices and uses Kafka to connect over 70 services, saving $10M.

Maximize the Business Value of Machine Learning and Data Science with Kafka (...

confluent

Today, many companies that have lots of data are still struggling to derive value from machine learning (ML) and data science investments. Why? Accessing the data may be difficult. Or maybe it’s poorly labeled. Or vital context is missing. Or there are questions around data integrity. Or standing up an ML service can be cumbersome and complex. At Nuuly, we offer an innovative clothing rental subscription model and are continually evolving our ML solutions to gain insight into the behaviors of our unique customer base as well as provide personalized services. In this session, I’ll share how we used event streaming with Apache Kafka® and Confluent Cloud to address many of the challenges that may be keeping your organization from maximizing the business value of machine learning and data science. First, you’ll see how we ensure that every customer interaction and its business context is collected. Next, I’ll explain how we can replay entire interaction histories using Kafka as a transport layer as well as a persistence layer and a business application processing layer. Order management, inventory management, logistics, subscription management – all of it integrates with Kafka as the common backbone. These data streams enable Nuuly to rapidly prototype and deploy dynamic ML models to support various domains, including pricing, recommendations, product similarity, and warehouse optimization. Join us and learn how Kafka can help improve machine learning and data science initiatives that may not be delivered to their full potential.

Achieve Sub-Second Analytics on Apache Kafka with Confluent and Imply

confluent

Presenters: Rachel Pedreschi, Senior Director, Solutions Engineering, Imply.io + Josh Treichel, Partner Solutions Architect, Confluent Analytic pipelines running purely on batch processing systems can suffer from hours of data lag, resulting in accuracy issues with analysis and overall decision-making. Join us for a demo to learn how easy it is to integrate your Apache Kafka® streams in Apache Druid (incubating) to provide real-time insights into the data. In this online talk, you’ll hear about ingesting your Kafka streams into Imply’s scalable analytic engine and gaining real-time insights via a modern user interface. Register now to learn about: -The benefits of combining a real-time streaming platform with a comprehensive analytics stack -Building an analytics pipeline by integrating Confluent Platform and Imply -How KSQL, streaming SQL for Kafka, can easily transform and filter streams of data in real time -Querying and visualizing streaming data in Imply -Practical ways to implement Confluent Platform and Imply to address common use cases such as analyzing network flows, collecting and monitoring IoT data and visualizing clickstream data Confluent Platform, developed by the creators of Kafka, enables the ingest and processing of massive amounts of real-time event data. Imply, the complete analytics stack built on Druid, can ingest, store, query and visualize streaming data from Confluent Platform, enabling end-to-end real-time analytics. Together, Confluent and Imply can provide low latency data delivery, data transform, and data querying capabilities to power a range of use cases.

Dissolving the Problem (Making an ACID-Compliant Database Out of Apache Kafka®)

confluent

Presenter: Tim Berglund, Senior Director of Developer Experience, Confluent It has become a truism in the past decade that building systems at scale, using non-relational databases, requires giving up on the transactional guarantees afforded by the relational databases of yore. ACID transactional semantics are fine, but we all know you can’t have them all in a distributed system. Or can we? In this talk, I will argue that by designing our systems around a distributed log like Apache Kafka®, we can in fact achieve ACID semantics at scale. We can ensure that distributed write operations can be applied atomically, consistently, in isolation between services, and of course with durability. What seems to be a counterintuitive conclusion ends up being straightforwardly achievable using existing technologies, as an elusive set of properties becomes relatively easy to achieve with the right architectural paradigm underlying the application.

SIEM Modernization: Build a Situationally Aware Organization with Apache Kafka®

confluent

Watch this talk here: https://www.confluent.io/online-talks/siem-modernization-build-a-situationally-aware-organization-with-apache-kafka Of all security breaches, 85% are conducted with compromised credentials, often at the administration level or higher. A lot of IT groups think “security” means authentication, authorization and encryption (AAE), but these are often tick-boxes that rarely stop breaches. The internal threat surfaces of data streams or disk drives in a raidset in a data centre are not the threat surface of interest. Cyber or Threat organizations must conduct internal investigations of IT, subcontractors and supply chains without implicating the innocent. Therefore, they are organizationally air-gapped from IT. Some surveys indicate up to 10% of IT is under investigation at any given time. Deploying a signal processing platform, such as Confluent Platform, allows organizations to evaluate data as soon as it becomes available enabling them to assess and mitigate risk before it arises. In Cyber or Threat Intelligence, events can be considered signals, and when analysts are hunting for threat actors, these don't appear as a single needle in a haystack, but as a series of needles. In this paradigm, streams of signals aggregate into signatures. This session shows how various sub-systems in Apache Kafka can be used to aggregate, integrate and attribute these signals into signatures of interest. In this talk you will learn: -The current threat landscape -The difference between Security and Threat Intelligence -The value of Confluent platform as an ideal complement to hardware endpoint detection systems and batch-based SIEM warehouses

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

confluent

1. A streaming platform like Kafka can provide the benefits of Hadoop for batch processing but in a faster, real-time way by processing data as it arrives rather than storing all data. 2. Virtual reality applications require stream processing to power features like VR mirroring and capture in real-time. Kafka's stream processing capabilities address challenges like this for VR. 3. The document discusses how AltspaceVR uses Kafka stream processing for applications like VR mirroring and capture, presence tracking, scheduled tasks, and more to power their real-time VR experiences.

Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services

confluent

EDA Meets Data Engineering – What's the Big Deal?

confluent

Presenter: Guru Sattanathan, Systems Engineer, Confluent Event-driven architectures have been around for many years, much like Apache Kafka®, which first open sourced in 2011. The reality is that the true potential of Kafka is only being realised now. Kafka is becoming the central nervous system of many of today’s enterprises. It is bringing a profound paradigm shift to the way we think about enterprise IT. What has changed in Kafka to enable this paradigm shift? Is it not just a message broker, and how are enterprises using it today? This session will explore these key questions. Sydney: https://content.deloitte.com.au/20200221-tel-event-tech-community-syd-registration Melbourne: https://content.deloitte.com.au/20200221-tel-event-tech-community-mel-registration

Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...

confluent

Apache Kafka is critical to PayPal's analytics platform. It handles a stream of over 20 billion events per day across 300 partitions. To democratize access to analytics data, PayPal built a Connect platform leveraging Kafka to process and send data in real-time to tools of customers' choice. The platform scales to process over 40 billion events daily using reactive architectures with Akka and Alpakka Kafka connectors to consume and publish events within Akka streams. Some challenges include throughput limited by partitions and issues requiring tuning for optimal performance.

End to-end large messages processing with Kafka Streams & Kafka Connect

confluent

This document discusses processing large messages with Kafka Streams and Kafka Connect. It describes how large messages can exceed Kafka's maximum message size limit. It proposes using an S3-backed serializer to store large messages in S3 and send pointers to Kafka instead. This allows processing logic to remain unchanged while handling large messages. The serializer transparently retrieves messages from S3 during deserialization.

Kafka as your Data Lake - is it Feasible? (Guido Schmutz, Trivadis) Kafka Sum...

HostedbyConfluent

For a long time we discuss how much data we can keep in Kafka. Can we store data forever or do we remove data after a while and maybe having the history in a data lake on Object Storage or HDFS? With the advent of Tiered Storage in Confluent Enterprise Platform, storing data much longer in Kafka is much very feasible. So can we replace a traditional data lake with just Kafka? Maybe at least for the raw data? But what about accessing the data, for example using SQL? KSQL allows for processing data in a streaming fashion using an SQL like dialect. But what about reading all data of a topic? You can reset the offset and still use KSQL. But there is another family of products, so-called query engines for Big Data. They originate from the idea of reading Big Data sources such as HDFS, object storage or HBase, using the SQL language. Presto, Apache Drill and Dremio are the most popular solutions in that space. Lately these query engines also added support for Kafka topics as a source of data. With that you can read a topic as a table and join it with information available in other data sources. The idea of course is not real-time streaming analytics but batch analytics directly on the Kafka topic, without having to store it in a big data storage. This talk answers, how well these tools support Kafka as a data source. What serialization formats do they support? Is there some form of predicate push-down supported or do we have to always read the complete topic? How performant is a query against a topic, compared to a query against the same data sitting in HDFS or an object store? And finally, will this allow us to replace our data lake or at least part of it by Apache Kafka?

You Must Construct Additional Pipelines: Pub-Sub on Kafka at Blizzard

confluent

(Stephen Parente + Jeff Field, Blizzard) Kafka Summit SF 2018 Blizzard’s global data platform has become a driving force in both business and operational analytics. As more internal customers onboard with the system, there is increasing demand for custom applications to access this data in near real time. In order to avoid many independent teams with varying levels of Kafka expertise all accessing the firehose from our critical production Kafkas, we developed our own pub-sub system on top of Kafka to provide specific datasets to customers on their own cloud deployed Kafka clusters.

One Click Streaming Data Pipelines & Flows | Leveraging Kafka & Spark | Ido F...

HostedbyConfluent

The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems. Using real-world examples, this talk covers why Apache Kafka is an excellent choice for cloud-native and hybrid architectures, how to go about designing, implementing and maintaining ESP systems, best practices and patterns for migrating to the cloud or hybrid configurations, when to go with PaaS or IaaS, what options are available for running Kafka in cloud or hybrid environments and what you need to build and maintain successful ESP systems that are secure, performant, reliable, highly-available and scalable.

Apache Flink(tm) - A Next-Generation Stream Processor

Aljoscha Krettek

In diesem Vortrag wird es zunächst einen kurzen Überblick über den aktuellen Stand im Bereich der Streaming-Datenanalyse geben. Danach wird es mit einer kleinen Einführung in das Apache-Flink-System zur Echtzeit-Datenanalyse weitergehen, bevor wir tiefer in einige der interessanten Eigenschaften eintauchen werden, die Flink von den anderen Spielern in diesem Bereich unterscheidet. Dazu werden wir beispielhafte Anwendungsfälle betrachten, die entweder direkt von Nutzern stammen oder auf unserer Erfahrung mit Nutzern basieren. Spezielle Eigenschaften, die wir betrachten werden, sind beispielsweise die Unterstützung für die Zerlegung von Events in einzelnen Sessions basierend auf der Zeit, zu der ein Ereignis passierte (event-time), Bestimmung von Zeitpunkten zum jeweiligen Speichern des Zustands eines Streaming-Programms für spätere Neustarts, die effiziente Abwicklung bei sehr großen zustandsorientierten Streaming-Berechnungen und die Zugänglichkeit des Zustandes von außerhalb.

Processing Real-Time Data at Scale: A streaming platform as a central nervous...

confluent

(Marcus Urbatschek, Confluent) Presentation during Confluent’s streaming event in Munich. This three-day hands-on course focused on how to build, manage, and monitor clusters using industry best-practices developed by the world’s foremost Apache Kafka™ experts. The sessions focused on how Kafka and the Confluent Platform work, how their main subsystems interact, and how to set up, manage, monitor, and tune your cluster.

Kafka for Real-Time Event Processing in Serverless Environments

confluent

(Jeff Sharpe + Alex Srisuwan, Capital One) Kafka Summit SF 2018 Using Kafka as a platform messaging bus is common, but bridging communication between real-time and asynchronous components can become complicated, especially when dealing with serverless environments. This has become increasingly common in modern banking where events need to be processed at near-real-time speed. Serverless environments are well-suited to address these needs, and Kafka remains an excellent solution for providing the reliable, resilient communication layer between serverless components and dedicated stream processing services. In this talk, we will examine some of the strengths and weaknesses of using Kafka for real-time communication, some tips for efficient interactions with Kafka and AWS Lambda, and a number of useful patterns for maximizing the strengths of Kafka and serverless components.

What is Apache Kafka®?

confluent

Viktor Gamov, Confluent, Developer Advocate Apache Kafka is an open source distributed streaming platform that allows you to build applications and process events as they occur. Viktor Gamov (developer Advocate at Confluent) walks through how it works and important underlying concepts. As a real-time, scalable, and durable system, Kafka can be used for fault-tolerant storage as well as for other use cases, such as stream processing, centralized data management, metrics, log aggregation, event sourcing, and more. This talk will explain what a streaming platform such as Apache Kafka is and some of the use cases and design patterns around its use—including several examples of where it is solving real business problems. https://www.meetup.com/Chennai-Kafka/events/269942117/

Matching the Scale at Tinder with Kafka

confluent

(Krunal Vora, Tinder) Kafka Summit San Francisco 2018 At Tinder, we have been using Kafka for streaming and processing events, data science processes and many other integral jobs. Forming the core of the pipeline at Tinder, Kafka has been accepted as the pragmatic solution to match the ever increasing scale of users, events and backend jobs. We, at Tinder, are investing time and effort to optimize the usage of Kafka solving the problems we face in the dating apps context. Kafka forms the backbone for the plans of the company to sustain performance through envisioned scale as the company starts to grow in unexplored markets. Come, learn about the implementation of Kafka at Tinder and how Kafka has helped solve the use cases for dating apps. Engage in the success story behind the business case of Kafka at Tinder.

The Past, Present, and Future of Apache Flink®

Aljoscha Krettek

Flink has evolved from a batch processor to a unified stream and batch processing framework. It now supports event-time processing, state, and low-level streaming with ProcessFunction. Looking ahead, Flink aims to improve elasticity, fault tolerance, SQL support, and handling large state through incremental snapshots. It also plans to offer more control over resource allocation and scaling through both active and reactive modes.

INTRODUCING: CREATE PIPELINE

SingleStore

This document introduces MemSQL Pipelines, which allow for exactly-once data ingestion semantics when streaming data from Kafka into MemSQL. MemSQL Pipelines provide a native way to extract, transform, and load external data into MemSQL tables. They offer a scalable and highly performant ETL process across a distributed cluster. The document explains streaming semantics like at least once and exactly once delivery, and how MemSQL Pipelines coordinate with Kafka to enable exactly-once ingestion through offset tracking. It presents the architecture of MemSQL Pipelines and demonstrates their use through a live demo.

Machine Learning and Deep Learning Applied to Real Time with Apache Kafka Str...

confluent

This document discusses applying machine learning models to real-time stream processing using Apache Kafka. It covers building analytic models from historical data, applying those models to real-time streams without redevelopment, and techniques for online training of models. Live demos are presented using open source tools like Kafka Streams, Kafka Connect, and H2O to apply machine learning to streaming use cases like flight delay prediction. The key takeaway is that streaming platforms can leverage pre-built machine learning models to power real-time analytics and actions.

Apache Kafka Streams + Machine Learning / Deep Learning

Kai Wähner

What's hot

Kafka Summit NYC 2017 - Stream it Together: 3 Realities of Modern Programming

confluent

Maximize the Business Value of Machine Learning and Data Science with Kafka (...

confluent

Achieve Sub-Second Analytics on Apache Kafka with Confluent and Imply

confluent

Dissolving the Problem (Making an ACID-Compliant Database Out of Apache Kafka®)

confluent

SIEM Modernization: Build a Situationally Aware Organization with Apache Kafka®

confluent

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

confluent

Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services

confluent

EDA Meets Data Engineering – What's the Big Deal?

confluent

Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...

confluent

End to-end large messages processing with Kafka Streams & Kafka Connect

confluent

Kafka as your Data Lake - is it Feasible? (Guido Schmutz, Trivadis) Kafka Sum...

HostedbyConfluent

You Must Construct Additional Pipelines: Pub-Sub on Kafka at Blizzard

confluent

One Click Streaming Data Pipelines & Flows | Leveraging Kafka & Spark | Ido F...

HostedbyConfluent

Apache Flink(tm) - A Next-Generation Stream Processor

Aljoscha Krettek

Processing Real-Time Data at Scale: A streaming platform as a central nervous...

confluent

Kafka for Real-Time Event Processing in Serverless Environments

confluent

What is Apache Kafka®?

confluent

Matching the Scale at Tinder with Kafka

confluent

The Past, Present, and Future of Apache Flink®

Aljoscha Krettek

INTRODUCING: CREATE PIPELINE

SingleStore

What's hot (20)

Kafka Summit NYC 2017 - Stream it Together: 3 Realities of Modern Programming

Maximize the Business Value of Machine Learning and Data Science with Kafka (...

Achieve Sub-Second Analytics on Apache Kafka with Confluent and Imply

Dissolving the Problem (Making an ACID-Compliant Database Out of Apache Kafka®)

SIEM Modernization: Build a Situationally Aware Organization with Apache Kafka®

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services

EDA Meets Data Engineering – What's the Big Deal?

Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...

End to-end large messages processing with Kafka Streams & Kafka Connect

Kafka as your Data Lake - is it Feasible? (Guido Schmutz, Trivadis) Kafka Sum...

You Must Construct Additional Pipelines: Pub-Sub on Kafka at Blizzard

One Click Streaming Data Pipelines & Flows | Leveraging Kafka & Spark | Ido F...

Apache Flink(tm) - A Next-Generation Stream Processor

Processing Real-Time Data at Scale: A streaming platform as a central nervous...

Kafka for Real-Time Event Processing in Serverless Environments

What is Apache Kafka®?

Matching the Scale at Tinder with Kafka

The Past, Present, and Future of Apache Flink®

INTRODUCING: CREATE PIPELINE

Similar to Kai Wähner, Technology Evangelist at Confluent: "Development of Scalable Machine Learning Microservices with Apache Kafka Streams and H2O.ai"

Machine Learning and Deep Learning Applied to Real Time with Apache Kafka Str...

confluent

Apache Kafka Streams + Machine Learning / Deep Learning

Kai Wähner

Apache Kafka Open Source Ecosystem for Machine Learning at Extreme Scale (Apa...

Kai Wähner

This talk shows how to productionize Machine Learning models in mission-critical and scalable real time applications by leveraging Apache Kafka as streaming platform. The talk discusses the relation between Machine Learning frameworks such as TensorFlow, DeepLearning4J or H2O and the Apache Kafka ecosystem. A live demo shows how to build a mission-critical Machine Learning environment leveraging different Kafka components: Kafka messaging and Kafka Connect for data movement from and into different sources and sinks, Kafka Streams for model deployment and inference in real time, and KSQL for real time analytics of predictions, alerts and model accuracy. Updated slide deck and talk from September 2018 at ApacheCon Montreal.

How to Leverage the Apache Kafka Ecosystem to Productionize Machine Learning ...

Codemotion

This talk shows how to productionize Machine Learning models in mission-critical and scalable real time applications by leveraging Apache Kafka as streaming platform. The talk discusses the relation between Machine Learning frameworks such as TensorFlow, DeepLearning4J or H2O and the Apache Kafka ecosystem. A live demo shows how to build a Machine Learning environment leveraging different Kafka components: Kafka messaging and Kafka Connect for data movement, Kafka Streams for model deployment and inference in real time, and KSQL for real time analytics of predictions, accuracy and alerts.

Deep Learning at Extreme Scale (in the Cloud)  with the Apache Kafka Open Sou...

Kai Wähner

How to Build a Machine Learning Infrastructure with Kafka, Connect, Streams, KSQL, etc… This talk shows how to build Machine Learning models at extreme scale and how to productionize the built models in mission-critical real time applications by leveraging open source components in the public cloud. The session discusses the relation between TensorFlow and the Apache Kafka ecosystem - and why this is a great fit for machine learning at extreme scale. The Machine Learning architecture includes: Kafka Connect for continuous high volume data ingestion into the public cloud, TensorFlow leveraging Deep Learning algorithms to build an analytic model on powerful GPUs, Kafka Streams for model deployment and inference in real time, and KSQL for real time analytics of predictions, alerts and model accuracy. Sensor analytics for predictive alerting in real time is used as real world example from Internet of Things scenarios. A live demo shows the out-of-the-box integration and dynamic scalability of these components on Google Cloud. Key takeaways for the audience • Learn how to build a Machine Learning infrastructure at extreme scale and how to productionize the built models in mission-critical real time applications • Understand the benefits of a machine learning platform on the public cloud • Learn about an extreme scale Machine Learning architecture around the Apache Kafka open source ecosystem including Kafka Connect, Kafka Streams and KSQL • See a live demo for an Internet of Things use case: Sensor analytics for predictive alerting in real time

Machine Learning Trends of 2018 combined with the Apache Kafka Ecosystem

Kai Wähner

At OOP 2018 in Munich, I presented an updated version of my talk about building scalable, mission-critical microservices with the Apache Kafka ecosystem and Deep Learning frameworks like TensorFlow, DeepLearning4J or H2O. I want to share the updated slide deck and mention a few changes. The main story is the same as in my Confluent blog post about Kafka and Machine Learning: How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka. But I also discuss a few innovations in the ecosystem of Apache Kafka and trends in ML in the last months: KSQL, ONNX, Auto ML, ML platforms from Uber and Netflix. Let's take a look into these interesting topics and how this is related to each other.

Deep Learning Streaming Platform with Kafka Streams, TensorFlow, DeepLearning...

Kai Wähner

Talk from JavaOne 2017: Apache Kafka + Kafka Streams for Scalable, Mission Critical Deep Learning. Intelligent real time applications are a game changer in any industry. Deep Learning is one of the hottest buzzwords in this area. New technologies like GPUs combined with elastic cloud infrastructure enable the sophisticated usage of artificial neural networks to add business value in real world scenarios. Tech giants use it e.g. for image recognition and speech translation. This session discusses some real-world scenarios from different industries to explain when and how traditional companies can leverage deep learning in real time applications. This session shows how to deploy Deep Learning models into real time applications to do predictions on new events. Apache Kafka will be used to execute analytic models in a highly scalable and performant way. The first part introduces the use cases and concepts behind Deep Learning. It discusses how to build Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN) and Autoencoders leveraging open source frameworks like TensorFlow, DeepLearning4J or H2O. The second part shows how to deploy the built analytic models to real time applications leveraging Apache Kafka as streaming platform and Apache Kafka’s Streams API to embed the intelligent business logic into any external application or microservice. Some further material around Apache Kafka and Machine Learning: - Blog Post: How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka: https://www.confluent.io/blog/build-deploy-scalable-machine-learning-production-apache-kafka/ - Video: Build and Deploy Analytic Models with H2O.ai and Apache Kafka: https://www.youtube.com/watch?v=-q7CyIExBKM&feature=youtu.be - Code: Github Examples using Apache Kafka, TensorFlow, H2O, DeepLearning4J: https://github.com/kaiwaehner/kafka-streams-machine-learning-examples

Kai Waehner - Deep Learning at Extreme Scale in the Cloud with Apache Kafka a...

Codemotion

This talk shows how to build Machine Learning models at extreme scale and how to productionize the built models in mission-critical real time applications by leveraging open source components like TensorFlow and the Apache Kafka open source ecosystem in the public cloud - and why this is a great fit for machine learning at extreme scale. A live demo shows sensor analytics for predictive alerting in real time.

2019 04 seattle_meetup___kafka_machine_learning___kai_waehner

Nitin Kumar

A global automotive company builds a connected car infrastructure using Apache Kafka to enable machine learning applications. They preprocess sensor data from vehicles with KSQL, train models with TensorFlow, and deploy models to predict anomalies and power new services. The infrastructure spans multiple clouds and includes tools like Schema Registry for validation and Control Center for monitoring.

Introduction to Apache Kafka and why it matters - Madrid

Paolo Castagna

This document provides an introduction to Apache Kafka and discusses why it is an important distributed streaming platform. It outlines how Kafka can be used to handle streaming data flows in a reliable and scalable way. It also describes the various Apache Kafka APIs including Kafka Connect, Streams API, and KSQL that allow organizations to integrate Kafka with other systems and build stream processing applications.

Streaming Machine Learning with Python, Jupyter, TensorFlow, Apache Kafka and...

Kai Wähner

Data scientists and data engineers love Python for transforming, filtering, and processing data to train and deploy analytic models with frameworks such as TensorFlow. However, in real-world deployments, all of these steps require a scalable and reliable infrastructure. This session shows how data experts can use Python for data processing and model inference at scale, leveraging Python, Jupyter, Apache Kafka, and KSQL. Talk from Oracle Code One / Oracle World 2019 in San Francisco.

Unleashing Apache Kafka and TensorFlow in Hybrid Cloud Architectures

Kai Wähner

Talk at Strate Conference in London: Unleashing Apache Kafka and TensorFlow in Hybrid Cloud Architectures with Confluent: How do you leverage the flexibility and extreme scale of the public cloud and the Apache Kafka ecosystem to build scalable, mission-critical machine learning infrastructures that span multiple public clouds—or bridge your on-premises data centre to the cloud? Join Kai Wähner to learn how to use technologies such as TensorFlow with Kafka’s open source ecosystem for machine learning infrastructures. You’ll learn how to build a scalable, mission-critical machine learning infrastructure for data ingestion and processing, model training, deployment, and monitoring. The discussed architecture includes capabilities like scalable data preprocessing for training and predictions, a combination of different deep learning frameworks, data replication between data centers, intelligent real-time microservices running on Kubernetes, and local deployment of analytic models for offline predictions. Learn how the public cloud allows extreme scale for building analytic models and how the Apache Kafka open source ecosystem enables building a cloud-independent infrastructure for preprocessing and ingestion of data and inference and monitoring of analytic models in real time Understand why hybrid architectures and local model deployment are key for success in many scenarios and why you need a flexible machine learning architecture that supports different technologies and frameworks

Apache Kafka, Tiered Storage and TensorFlow for Streaming Machine Learning wi...

Kai Wähner

Don’t underestimate the Hidden Technical Debt in Machine Learning Systems. Leverage Apache Kafka’s open ecosystem as a scalable and flexible Event Streaming Platform to build one pipeline for real-time and batch use cases. Use Streaming Machine Learning with Apache Kafka, Tiered Storage, and TensorFlow IO to simplify your big data architecture. Tiered Storage for Kafka provides: - one platform for all data processing - an event-based source of truth for materialized views - no need for a pipeline between Kafka and a Data Lake like Hadoop Benefits: - cost reduction - long-term backup - performance isolation (real-time and historical analysis in the same cluster) Use Cases for Reprocessing Historical Events: - New consumer application - Error-handling - Compliance / regulatory processing - Query and analyze existing events - Model training

Apache Kafka, Tiered Storage and TensorFlow for Streaming Machine Learning wi...

confluent

Machine Learning (ML) is separated into model training and model inference. ML frameworks typically use a data lake like HDFS or S3 to process historical data and train analytic models. But it’s possible to completely avoid such a data store, using a modern streaming architecture. This talk compares a modern streaming architecture to traditional batch and big data alternatives and explains benefits like the simplified architecture, the ability of reprocessing events in the same order for training different models, and the possibility to build a scalable, mission-critical ML architecture for real time predictions with muss less headaches and problems. The talk explains how this can be achieved leveraging Apache Kafka, Tiered Storage and TensorFlow.

Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka

Kai Wähner

Streaming all over the World: Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka. Learn about various case studies for event streaming with Apache Kafka across industries. The talk explores architectures for real-world deployments from Audi, BMW, Disney, Generali, Paypal, Tesla, Unity, Walmart, William Hill, and more. Use cases include fraud detection, mainframe offloading, predictive maintenance, cybersecurity, edge computing, track&trace, live betting, and much more.

Overview of Apache Flink: the 4G of Big Data Analytics Frameworks

DataWorks Summit/Hadoop Summit

This document provides an overview of Apache Flink and discusses why it is suitable for real-world streaming analytics. The document contains an agenda that covers how Flink is a multi-purpose big data analytics framework, why streaming analytics are emerging, why Flink is suitable for real-world streaming analytics, novel use cases enabled by Flink, who is using Flink, and where to go from here. Key points include Flink innovations like custom memory management, its DataSet API, rich windowing semantics, and native iterative processing. Flink's streaming features that make it suitable for real-world use include its pipelined processing engine, stream abstraction, performance, windowing support, fault tolerance, and integration with Hadoop.

Overview of Apache Fink: the 4 G of Big Data Analytics Frameworks

Slim Baltagi

Slides of my talk at the Hadoop Summit Europe in Dublin, Ireland on April 13th, 2016. The talk introduces Apache Flink as both a multi-purpose Big Data analytics framework and real-world streaming analytics framework. It is focusing on Flink's key differentiators and suitability for streaming analytics use cases. It also shows how Flink enables novel use cases such as distributed CEP (Complex Event Processing) and querying the state by behaving like a key value data store.

Overview of Apache Fink: The 4G of Big Data Analytics Frameworks

Slim Baltagi

Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...

Kai Wähner

Talk from Kafka Summit San Francisco 2019 (https://kafka-summit.org/sessions/event-driven-model-serving-stream-processing-vs-rpc-kafka-tensorflow/). Video recording will be available for free on the Summit website. Event-based stream processing is a modern paradigm to continuously process incoming data feeds, e.g. for IoT sensor analytics, payment and fraud detection, or logistics. Machine Learning / Deep Learning models can be leveraged in different ways to do predictions and improve the business processes. Either analytic models are deployed natively in the application or they are hosted in a remote model server. In the latter you combine stream processing with RPC / Request-Response paradigm instead of direct doing direct inference within the application. This talk discusses the pros and cons of both approaches and shows examples of stream processing vs. RPC model serving using Kubernetes, Apache Kafka, Kafka Streams, gRPC and TensorFlow Serving. The trade-offs of using a public cloud service like AWS or GCP for model deployment are also discussed and compared to local hosting for offline predictions directly “at the edge”. Key takeaways • Machine Learning / Deep Learning models can be used in different ways to do predictions. Scalability and loose coupling are important success factors • Stream processing vs. RPC / Request-Response for model serving has many trade-offs – learn about alternatives and best practices for your different scenarios • Understand the alternatives and trade-offs of model deployment in modern infrastructures like Kubernetes or Cloud Services like AWS or GCP • See live demos with Java, gRPC, Apache Kafka, KSQL and TensorFlow Serving to understand the trade-offs

Event-Driven Model Serving: Stream Processing vs. RPC with Kafka and TensorFl...

confluent

Event-based stream processing is a modern paradigm to continuously process incoming data feeds, e.g. for IoT sensor analytics, payment and fraud detection, or logistics. Machine Learning / Deep Learning models can be leveraged in different ways to do predictions and improve the business processes. Either analytic models are deployed natively in the application or they are hosted in a remote model server. In the latter you combine stream processing with RPC / Request-Response paradigm instead of direct doing direct inference within the application. This talk discusses the pros and cons of both approaches and shows examples of stream processing vs. RPC model serving using Kubernetes, Apache Kafka, Kafka Streams, gRPC and TensorFlow Serving. The trade-offs of using a public cloud service like AWS or GCP for model deployment are also discussed and compared to local hosting for offline predictions directly "at the edge".

Similar to Kai Wähner, Technology Evangelist at Confluent: "Development of Scalable Machine Learning Microservices with Apache Kafka Streams and H2O.ai" (20)