A global automotive company builds a connected car infrastructure using Apache Kafka to enable machine learning applications. They preprocess sensor data from vehicles with KSQL, train models with TensorFlow, and deploy models to predict anomalies and power new services. The infrastructure spans multiple clouds and includes tools like Schema Registry for validation and Control Center for monitoring.
The Elephant in the Kubernetes Room - Team Interactions at Scale @ KubeCon No...Manuel Pais
Kubernetes helps us tame sprawling microservices architectures and address increased operational complexity. Kubernetes gives developers abstractions and APIs to deploy and run their services.
Yet, the elephant in the room is that to run, maintain and evolve Kubernetes clusters, we need more ops expertise and most likely a dedicated team to do so.
The question that begs to be asked is if we risk going back to pre-DevOps isolation between Dev and Ops teams? Is the tradeoff between better operational tools and introducing a new dependency layer on the path to production for application teams worthwhile? Are we making life easier for application teams or instead reducing their end-to-end ownership?
Manuel will then introduce Team Topologies, a balanced approach for thinking about teams responsibilities and interactions which can help get the most value out of your Kubernetes adoption.
"Building scalable applications for the cloud" is a presentation delivered by Petar Tahchiev, founder and CEO at Nemesis Software, to an audience of Java developers. The event was held at Cosmos Coworking space and it was organized by dev.bg - the largest developers community in Bulgaria.
VLSI stands for Very Large Scale Integration. Generally there are mainly 2 types of VLSI projects – 1. Projects in VLSI based System Design, 2. VLSI Design Projects. You might be confused to understand the difference between these 2 types of projects. Let me now explain to you.
Projects in VLSI based system design are the projects which involve the design of various types of digital systems that can be implemented on a PLD device like a FPGA or a CPLD.
Aliaksei Bahachuk - JavaScript and Solution ArchitectureAliaksei Bahachuk
Quite often the architecture in JavaScript reduces itself to the choice of a framework according to the frontend world’s recent trends. What if we told you that the choice of technology is just the step №7 on your way to developing project design? Every day many projects feel the draught or even fall to pieces because of the incorrectly chosen architecture.
We’ll share some stories that can help you look at the architecture and its meaning for the modern apps from the right perspective, as well as avoid mistakes that can simply ruin your project.
The Elephant in the Kubernetes Room - Team Interactions at Scale @ KubeCon No...Manuel Pais
Kubernetes helps us tame sprawling microservices architectures and address increased operational complexity. Kubernetes gives developers abstractions and APIs to deploy and run their services.
Yet, the elephant in the room is that to run, maintain and evolve Kubernetes clusters, we need more ops expertise and most likely a dedicated team to do so.
The question that begs to be asked is if we risk going back to pre-DevOps isolation between Dev and Ops teams? Is the tradeoff between better operational tools and introducing a new dependency layer on the path to production for application teams worthwhile? Are we making life easier for application teams or instead reducing their end-to-end ownership?
Manuel will then introduce Team Topologies, a balanced approach for thinking about teams responsibilities and interactions which can help get the most value out of your Kubernetes adoption.
"Building scalable applications for the cloud" is a presentation delivered by Petar Tahchiev, founder and CEO at Nemesis Software, to an audience of Java developers. The event was held at Cosmos Coworking space and it was organized by dev.bg - the largest developers community in Bulgaria.
VLSI stands for Very Large Scale Integration. Generally there are mainly 2 types of VLSI projects – 1. Projects in VLSI based System Design, 2. VLSI Design Projects. You might be confused to understand the difference between these 2 types of projects. Let me now explain to you.
Projects in VLSI based system design are the projects which involve the design of various types of digital systems that can be implemented on a PLD device like a FPGA or a CPLD.
Aliaksei Bahachuk - JavaScript and Solution ArchitectureAliaksei Bahachuk
Quite often the architecture in JavaScript reduces itself to the choice of a framework according to the frontend world’s recent trends. What if we told you that the choice of technology is just the step №7 on your way to developing project design? Every day many projects feel the draught or even fall to pieces because of the incorrectly chosen architecture.
We’ll share some stories that can help you look at the architecture and its meaning for the modern apps from the right perspective, as well as avoid mistakes that can simply ruin your project.
INTERFACE, by apidays - Apache Cassandra now speaks developer with Stargate ...apidays
INTERFACE, by apidays 2021 - It’s APIs all the way down
June 30, July 1 & 2, 2021
Apache Cassandra now speaks developer with Stargate: Rethinking database APIs
Ash Ryan Arnwine, Developer Experience Architect at Datastax
PuppetConf 2017 | Adobe Advertising Cloud: A Lean Puppet Workflow to Support ...Nicolas Brousse
Building and scaling a multi-cloud solution that's enabled for cloud bursting is not a trivial task, and requires a lot of automation. While experiencing hyper-growth on the Adobe Advertising Cloud, our operations engineering team had to frequently update and improve its workflow in order to stay nimble and allow fast delivery of new infrastructure. At TubeMogul/Adobe Advertising Cloud, we implemented a lean Puppet workflow that enables the operations engineering team to deploy and support a broad range of services in a complex environment that supports hundreds of billions of requests a day. With over 150 changes released per day on its production infrastructure, the team had to adjust and tune its processes to enforce quality, standards, to review, and to prevent systems from breaking. In this talk, you will learn how we implemented our infrastructure as code by leveraging tools like Puppet, Gerrit, Terraform, and Jenkins, which together enable our private and public cloud infrastructures across 12 locations and four continents.
INTERFACE, by apidays - C* made easy with Stargate APIs by Kirsten Hunter, D...apidays
INTERFACE, by apidays 2021 - It’s APIs all the way down
June 30, July 1 & 2, 2021
C* made easy with Stargate APIs
Kirsten Hunter, Author of "Irrestisible APIs", Developer Advocate at DataStax
apidays LIVE London 2021 - Getting started with Event-Driven APIs by Hugo Gue...apidays
apidays LIVE London 2021 - Reaching Maximum Potential in Banking & Insurance with API Mindset
October 27 & 28, 2021
The future API stack : GraphQL, gRPC and API specifications
Getting started with Event-Driven APIs
Hugo Guerrero, APIs & Messaging Developer Advocate at Red Hat
apidays LIVE Paris 2021 - Getting started with Event-Driven APis by Hugo Guer...apidays
apidays LIVE Paris 2021 - APIs and the Future of Software
December 7, 8 & 9, 2021
Getting started with Event-Driven APis
Hugo Guerrero, APIs & Messaging Developer Advocate at Red Hat
apidays LIVE New York 2021 - APIOps: automating API operations for speed and ...apidays
apidays LIVE New York 2021 - API-driven Regulations for Finance, Insurance, and Healthcare
July 28 & 29, 2021
APIOps: automating API operations for speed and quality at scale
Melissa van der Hecht, Field CTO at Kong
[APIdays INTERFACE 2021] Now that we have K8s, can we stop re-inventing API p...WSO2
Kubernetes has been called the "platform of platforms" and the final major evolutionary step of cloud native computing. What's needed to build an API Platform on it? A great developer experience? An API Marketplace for managing all APIs together in one place? Auto build and deploy onto multiple flavors of K8s? Multi-tenancy? SaaS model hosting with multi-tenancy? Team based development? Ability to create new microservices and APIs? Support for sync and async protocols? Analytics? Metering, monitoring, policy enforcement? What else? Are we done? Or will we need to rebuild the platform again on serverless functions?
Watch Recording : https://youtu.be/kQjETt_c8Ac
Smart edge ioT devices enable utility company to create new business segments...mfrancis
OSGi Community Event 2015
Nowadays utility companies face the situation that more and more customers equip their houses with energy storage systems trying to become self-sustaining with on-site energy production. Supplying electricity as a business model in this scenario does not work - it is neither sustainable nor extendable any more.</p>
EnBW - one of the biggest European energy supply companies - strikes a new path offering their energy know-how as a service to owners of on-site energy production systems.
EnergyBASE - an intelligent smart edge energy management device - helps to optimize in-house energy flows and to increase the percentage of self-containedness. It provides a self-learning system based on individual power production and personal household consumption characteristics and combines these data with additional external sources like weather data to calculate consumption prognosis in order to optimize in-house energy flows.
The EnergyBASE system consists of an ARM 450 MHz processor with 128 MB RAM and runs an embedded Linux operating system with integrated TCP/IP stack and SQL database. It provides LAN, WiFi and RS485 interfaces. The software stack contains Oracle Java Embedded SE 8 (ported by MicroDoc) and Prosyst mBS Smart Home OSGi.
In this talk we will present our experience using Java Embedded SE 8 and OSGi on an embedded device in a real-life project with demanding needs for computation performance (calculation of mathematical optimization models), handling of big data voluminas, various infrastructure needs (internet, sensors, powerline, housenet) and stability (24/7) requirements.
Speed-Up Kafka Delivery with AsyncAPI & Microcks | Hugo Guerrero, Red HatHostedbyConfluent
Apache Kafka is getting used as an event backbone in new organizations everyday. We would love to send every byte of data through the event bus. Like traditional REST APIs a contract-first approach is very useful when designing event-driven architectures. In the case of asynchronous APIs, we have the AsynAPI specification to document the endpoints where the schema of the records become the main part of the contract payload. Microcks allows us to deploy a testing and mocking platform to have a unified view of the endpoints to speed-up application delivery.
apidays LIVE Paris 2021 - API Attack Simulator - Find your API vulnerabilitie...apidays
apidays LIVE Paris 2021 - APIs and the Future of Software
December 7, 8 & 9, 2021
API Attack Simulator - Find your API vulnerabilities first
Sella Rafaeli, Full-Stack Web Developer at WIB
apidays LIVE Paris - SDK driven GraphQL by Nader Dabitapidays
apidays LIVE Paris - Responding to the New Normal with APIs for Business, People and Society
December 8, 9 & 10, 2020
SDK driven GraphQL
Nader Dabit, Senior Developer Advocate at Amazon Web Services
Automated Apache Kafka Mocking and Testing with AsyncAPI | Hugo Guerrero, Red...HostedbyConfluent
Apache Kafka is getting used as an event backbone in new organizations every day. We would love to send every byte of data through the event bus. Like traditional REST APIs, a contract-first approach is very useful when designing event-driven architectures. In the case of asynchronous APIs, we have the AsyncAPI specification to document the endpoints where the schema of the records become the main part of the contract payload. Microcks allows us to deploy a testing and mocking platform to have a unified view of the endpoints to speed-up application delivery.
In this session we will:
- Go over the evolution of API specifications
- Review the approach for contract-first design with Apache Kafka
- Introduce the AsyncAPI specification
- Take an overview of an implementation example for automated mocking and testing
apidays LIVE Jakarta - REST the events: REST APIs for Event-Driven Architectu...apidays
apidays LIVE Jakarta 2021 - Accelerating Digitisation
February 24, 2021
REST the events: REST APIs for Event-Driven Architecture
Mark Teehan, Principal Solution Engineer at Confluent APAC
Conclusion Code Cafe - Microcks for Mocking and Testing Async APIs (January 2...Lucas Jellema
Microcks is a tool for API Mocking and Testing. In this presentation an overview of the support in Microcks for asynchronous APIs - the event publishing and consuming behavior of services and applications
apidays LIVE New York 2021 - Solving API security through holistic obervabili...apidays
apidays LIVE New York 2021 - API-driven Regulations for Finance, Insurance, and Healthcare
July 28 & 29, 2021
Solving API security through holistic obervability
Jean-Baptiste Aviat, AppSec Staff Engineer at Datadog
Unleashing Apache Kafka and TensorFlow in Hybrid Cloud ArchitecturesKai Wähner
Talk at Strate Conference in London: Unleashing Apache Kafka and TensorFlow in Hybrid Cloud Architectures with Confluent:
How do you leverage the flexibility and extreme scale of the public cloud and the Apache Kafka ecosystem to build scalable, mission-critical machine learning infrastructures that span multiple public clouds—or bridge your on-premises data centre to the cloud?
Join Kai Wähner to learn how to use technologies such as TensorFlow with Kafka’s open source ecosystem for machine learning infrastructures. You’ll learn how to build a scalable, mission-critical machine learning infrastructure for data ingestion and processing, model training, deployment, and monitoring.
The discussed architecture includes capabilities like scalable data preprocessing for training and predictions, a combination of different deep learning frameworks, data replication between data centers, intelligent real-time microservices running on Kubernetes, and local deployment of analytic models for offline predictions.
Learn how the public cloud allows extreme scale for building analytic models and how the Apache Kafka open source ecosystem enables building a cloud-independent infrastructure for preprocessing and ingestion of data and inference and monitoring of analytic models in real time
Understand why hybrid architectures and local model deployment are key for success in many scenarios and why you need a flexible machine learning architecture that supports different technologies and frameworks
Kai Waehner - Deep Learning at Extreme Scale in the Cloud with Apache Kafka a...Codemotion
This talk shows how to build Machine Learning models at extreme scale and how to productionize the built models in mission-critical real time applications by leveraging open source components like TensorFlow and the Apache Kafka open source ecosystem in the public cloud - and why this is a great fit for machine learning at extreme scale. A live demo shows sensor analytics for predictive alerting in real time.
INTERFACE, by apidays - Apache Cassandra now speaks developer with Stargate ...apidays
INTERFACE, by apidays 2021 - It’s APIs all the way down
June 30, July 1 & 2, 2021
Apache Cassandra now speaks developer with Stargate: Rethinking database APIs
Ash Ryan Arnwine, Developer Experience Architect at Datastax
PuppetConf 2017 | Adobe Advertising Cloud: A Lean Puppet Workflow to Support ...Nicolas Brousse
Building and scaling a multi-cloud solution that's enabled for cloud bursting is not a trivial task, and requires a lot of automation. While experiencing hyper-growth on the Adobe Advertising Cloud, our operations engineering team had to frequently update and improve its workflow in order to stay nimble and allow fast delivery of new infrastructure. At TubeMogul/Adobe Advertising Cloud, we implemented a lean Puppet workflow that enables the operations engineering team to deploy and support a broad range of services in a complex environment that supports hundreds of billions of requests a day. With over 150 changes released per day on its production infrastructure, the team had to adjust and tune its processes to enforce quality, standards, to review, and to prevent systems from breaking. In this talk, you will learn how we implemented our infrastructure as code by leveraging tools like Puppet, Gerrit, Terraform, and Jenkins, which together enable our private and public cloud infrastructures across 12 locations and four continents.
INTERFACE, by apidays - C* made easy with Stargate APIs by Kirsten Hunter, D...apidays
INTERFACE, by apidays 2021 - It’s APIs all the way down
June 30, July 1 & 2, 2021
C* made easy with Stargate APIs
Kirsten Hunter, Author of "Irrestisible APIs", Developer Advocate at DataStax
apidays LIVE London 2021 - Getting started with Event-Driven APIs by Hugo Gue...apidays
apidays LIVE London 2021 - Reaching Maximum Potential in Banking & Insurance with API Mindset
October 27 & 28, 2021
The future API stack : GraphQL, gRPC and API specifications
Getting started with Event-Driven APIs
Hugo Guerrero, APIs & Messaging Developer Advocate at Red Hat
apidays LIVE Paris 2021 - Getting started with Event-Driven APis by Hugo Guer...apidays
apidays LIVE Paris 2021 - APIs and the Future of Software
December 7, 8 & 9, 2021
Getting started with Event-Driven APis
Hugo Guerrero, APIs & Messaging Developer Advocate at Red Hat
apidays LIVE New York 2021 - APIOps: automating API operations for speed and ...apidays
apidays LIVE New York 2021 - API-driven Regulations for Finance, Insurance, and Healthcare
July 28 & 29, 2021
APIOps: automating API operations for speed and quality at scale
Melissa van der Hecht, Field CTO at Kong
[APIdays INTERFACE 2021] Now that we have K8s, can we stop re-inventing API p...WSO2
Kubernetes has been called the "platform of platforms" and the final major evolutionary step of cloud native computing. What's needed to build an API Platform on it? A great developer experience? An API Marketplace for managing all APIs together in one place? Auto build and deploy onto multiple flavors of K8s? Multi-tenancy? SaaS model hosting with multi-tenancy? Team based development? Ability to create new microservices and APIs? Support for sync and async protocols? Analytics? Metering, monitoring, policy enforcement? What else? Are we done? Or will we need to rebuild the platform again on serverless functions?
Watch Recording : https://youtu.be/kQjETt_c8Ac
Smart edge ioT devices enable utility company to create new business segments...mfrancis
OSGi Community Event 2015
Nowadays utility companies face the situation that more and more customers equip their houses with energy storage systems trying to become self-sustaining with on-site energy production. Supplying electricity as a business model in this scenario does not work - it is neither sustainable nor extendable any more.</p>
EnBW - one of the biggest European energy supply companies - strikes a new path offering their energy know-how as a service to owners of on-site energy production systems.
EnergyBASE - an intelligent smart edge energy management device - helps to optimize in-house energy flows and to increase the percentage of self-containedness. It provides a self-learning system based on individual power production and personal household consumption characteristics and combines these data with additional external sources like weather data to calculate consumption prognosis in order to optimize in-house energy flows.
The EnergyBASE system consists of an ARM 450 MHz processor with 128 MB RAM and runs an embedded Linux operating system with integrated TCP/IP stack and SQL database. It provides LAN, WiFi and RS485 interfaces. The software stack contains Oracle Java Embedded SE 8 (ported by MicroDoc) and Prosyst mBS Smart Home OSGi.
In this talk we will present our experience using Java Embedded SE 8 and OSGi on an embedded device in a real-life project with demanding needs for computation performance (calculation of mathematical optimization models), handling of big data voluminas, various infrastructure needs (internet, sensors, powerline, housenet) and stability (24/7) requirements.
Speed-Up Kafka Delivery with AsyncAPI & Microcks | Hugo Guerrero, Red HatHostedbyConfluent
Apache Kafka is getting used as an event backbone in new organizations everyday. We would love to send every byte of data through the event bus. Like traditional REST APIs a contract-first approach is very useful when designing event-driven architectures. In the case of asynchronous APIs, we have the AsynAPI specification to document the endpoints where the schema of the records become the main part of the contract payload. Microcks allows us to deploy a testing and mocking platform to have a unified view of the endpoints to speed-up application delivery.
apidays LIVE Paris 2021 - API Attack Simulator - Find your API vulnerabilitie...apidays
apidays LIVE Paris 2021 - APIs and the Future of Software
December 7, 8 & 9, 2021
API Attack Simulator - Find your API vulnerabilities first
Sella Rafaeli, Full-Stack Web Developer at WIB
apidays LIVE Paris - SDK driven GraphQL by Nader Dabitapidays
apidays LIVE Paris - Responding to the New Normal with APIs for Business, People and Society
December 8, 9 & 10, 2020
SDK driven GraphQL
Nader Dabit, Senior Developer Advocate at Amazon Web Services
Automated Apache Kafka Mocking and Testing with AsyncAPI | Hugo Guerrero, Red...HostedbyConfluent
Apache Kafka is getting used as an event backbone in new organizations every day. We would love to send every byte of data through the event bus. Like traditional REST APIs, a contract-first approach is very useful when designing event-driven architectures. In the case of asynchronous APIs, we have the AsyncAPI specification to document the endpoints where the schema of the records become the main part of the contract payload. Microcks allows us to deploy a testing and mocking platform to have a unified view of the endpoints to speed-up application delivery.
In this session we will:
- Go over the evolution of API specifications
- Review the approach for contract-first design with Apache Kafka
- Introduce the AsyncAPI specification
- Take an overview of an implementation example for automated mocking and testing
apidays LIVE Jakarta - REST the events: REST APIs for Event-Driven Architectu...apidays
apidays LIVE Jakarta 2021 - Accelerating Digitisation
February 24, 2021
REST the events: REST APIs for Event-Driven Architecture
Mark Teehan, Principal Solution Engineer at Confluent APAC
Conclusion Code Cafe - Microcks for Mocking and Testing Async APIs (January 2...Lucas Jellema
Microcks is a tool for API Mocking and Testing. In this presentation an overview of the support in Microcks for asynchronous APIs - the event publishing and consuming behavior of services and applications
apidays LIVE New York 2021 - Solving API security through holistic obervabili...apidays
apidays LIVE New York 2021 - API-driven Regulations for Finance, Insurance, and Healthcare
July 28 & 29, 2021
Solving API security through holistic obervability
Jean-Baptiste Aviat, AppSec Staff Engineer at Datadog
Unleashing Apache Kafka and TensorFlow in Hybrid Cloud ArchitecturesKai Wähner
Talk at Strate Conference in London: Unleashing Apache Kafka and TensorFlow in Hybrid Cloud Architectures with Confluent:
How do you leverage the flexibility and extreme scale of the public cloud and the Apache Kafka ecosystem to build scalable, mission-critical machine learning infrastructures that span multiple public clouds—or bridge your on-premises data centre to the cloud?
Join Kai Wähner to learn how to use technologies such as TensorFlow with Kafka’s open source ecosystem for machine learning infrastructures. You’ll learn how to build a scalable, mission-critical machine learning infrastructure for data ingestion and processing, model training, deployment, and monitoring.
The discussed architecture includes capabilities like scalable data preprocessing for training and predictions, a combination of different deep learning frameworks, data replication between data centers, intelligent real-time microservices running on Kubernetes, and local deployment of analytic models for offline predictions.
Learn how the public cloud allows extreme scale for building analytic models and how the Apache Kafka open source ecosystem enables building a cloud-independent infrastructure for preprocessing and ingestion of data and inference and monitoring of analytic models in real time
Understand why hybrid architectures and local model deployment are key for success in many scenarios and why you need a flexible machine learning architecture that supports different technologies and frameworks
Kai Waehner - Deep Learning at Extreme Scale in the Cloud with Apache Kafka a...Codemotion
This talk shows how to build Machine Learning models at extreme scale and how to productionize the built models in mission-critical real time applications by leveraging open source components like TensorFlow and the Apache Kafka open source ecosystem in the public cloud - and why this is a great fit for machine learning at extreme scale. A live demo shows sensor analytics for predictive alerting in real time.
Apache Kafka Open Source Ecosystem for Machine Learning at Extreme Scale (Apa...Kai Wähner
This talk shows how to productionize Machine Learning models in mission-critical and scalable real time applications by leveraging Apache Kafka as streaming platform. The talk discusses the relation between Machine Learning frameworks such as TensorFlow, DeepLearning4J or H2O and the Apache Kafka ecosystem. A live demo shows how to build a mission-critical Machine Learning environment leveraging different Kafka components: Kafka messaging and Kafka Connect for data movement from and into different sources and sinks, Kafka Streams for model deployment and inference in real time, and KSQL for real time analytics of predictions, alerts and model accuracy.
Updated slide deck and talk from September 2018 at ApacheCon Montreal.
Deep Learning at Extreme Scale (in the Cloud) with the Apache Kafka Open Sou...Kai Wähner
How to Build a Machine Learning Infrastructure with Kafka, Connect, Streams, KSQL, etc…
This talk shows how to build Machine Learning models at extreme scale and how to productionize the built models in mission-critical real time applications by leveraging open source components in the public cloud. The session discusses the relation between TensorFlow and the Apache Kafka ecosystem - and why this is a great fit for machine learning at extreme scale.
The Machine Learning architecture includes: Kafka Connect for continuous high volume data ingestion into the public cloud, TensorFlow leveraging Deep Learning algorithms to build an analytic model on powerful GPUs, Kafka Streams for model deployment and inference in real time, and KSQL for real time analytics of predictions, alerts and model accuracy.
Sensor analytics for predictive alerting in real time is used as real world example from Internet of Things scenarios. A live demo shows the out-of-the-box integration and dynamic scalability of these components on Google Cloud.
Key takeaways for the audience
• Learn how to build a Machine Learning infrastructure at extreme scale and how to productionize the built models in mission-critical real time applications
• Understand the benefits of a machine learning platform on the public cloud
• Learn about an extreme scale Machine Learning architecture around the Apache Kafka open source ecosystem including Kafka Connect, Kafka Streams and KSQL
• See a live demo for an Internet of Things use case: Sensor analytics for predictive alerting in real time
How to Leverage the Apache Kafka Ecosystem to Productionize Machine Learning ...Codemotion
This talk shows how to productionize Machine Learning models in mission-critical and scalable real time applications by leveraging Apache Kafka as streaming platform. The talk discusses the relation between Machine Learning frameworks such as TensorFlow, DeepLearning4J or H2O and the Apache Kafka ecosystem. A live demo shows how to build a Machine Learning environment leveraging different Kafka components: Kafka messaging and Kafka Connect for data movement, Kafka Streams for model deployment and inference in real time, and KSQL for real time analytics of predictions, accuracy and alerts.
Data scientists and data engineers love Python for transforming, filtering, and processing data to train and deploy analytic models with frameworks such as TensorFlow. However, in real-world deployments, all of these steps require a scalable and reliable infrastructure. This session shows how data experts can use Python for data processing and model inference at scale, leveraging Python, Jupyter, Apache Kafka, and KSQL.
Talk from Oracle Code One / Oracle World 2019 in San Francisco.
Apache Kafka, Tiered Storage and TensorFlow for Streaming Machine Learning wi...Kai Wähner
Don’t underestimate the Hidden Technical Debt in Machine Learning Systems.
Leverage Apache Kafka’s open ecosystem as a scalable and flexible Event Streaming Platform to build one pipeline for real-time and batch use cases.
Use Streaming Machine Learning with Apache Kafka, Tiered Storage, and TensorFlow IO to simplify your big data architecture.
Tiered Storage for Kafka provides:
- one platform for all data processing
- an event-based source of truth for materialized views
- no need for a pipeline between Kafka and a Data Lake like Hadoop
Benefits:
- cost reduction
- long-term backup
- performance isolation (real-time and historical analysis in the same cluster)
Use Cases for Reprocessing Historical Events:
- New consumer application
- Error-handling
- Compliance / regulatory processing
- Query and analyze existing events
- Model training
Apache Kafka, Tiered Storage and TensorFlow for Streaming Machine Learning wi...confluent
Machine Learning (ML) is separated into model training and model inference. ML frameworks typically use a data lake like HDFS or S3 to process historical data and train analytic models. But it’s possible to completely avoid such a data store, using a modern streaming architecture.
This talk compares a modern streaming architecture to traditional batch and big data alternatives and explains benefits like the simplified architecture, the ability of reprocessing events in the same order for training different models, and the possibility to build a scalable, mission-critical ML architecture for real time predictions with muss less headaches and problems.
The talk explains how this can be achieved leveraging Apache Kafka, Tiered Storage and TensorFlow.
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesKai Wähner
This session introduces Apache Kafka, an event-driven open source streaming platform. Apache Kafka goes far beyond scalable, high volume messaging. In addition, you can leverage Kafka Connect for integration and the Kafka Streams API for building lightweight stream processing microservices in autonomous teams. The Confluent Platform adds further components such as a Schema Registry, REST Proxy, KSQL, Clients for different programming languages and Connectors for different technologies.
The session discusses how tech giants like LinkedIn, Ebay or Airbnb leverage Apache Kafka as event streaming platform to solve various different business problems and how to create a scalable, flexible microservice architecture. A live demo shows how you can easily process and analyze streams of events using Apache Kafka and KSQL.
Unleashing Apache Kafka and TensorFlow in the Cloud Kai Wähner
How can you leverage the flexibility and extreme scale in the public cloud combined with your Apache Kafka ecosystem to build scalable, mission-critical machine learning infrastructures, which span multiple public clouds or bridge your on-premise data centre to cloud?
This talk will discuss and demo how you can leverage machine learning technologies such as TensorFlow with your Kafka deployments in public cloud to build a scalable, mission-critical machine learning infrastructure for data preprocessing and ingestion, and model training, deployment and monitoring.
The discussed architecture includes capabilities like scalable data preprocessing for training and predictions, combination of different Deep Learning frameworks, data replication between data centres, intelligent real time microservices running on Kubernetes, and local deployment of analytic models for offline predictions.
Deep Learning UDF for KSQL for Streaming Anomaly Detection of MQTT IoT Sensor Data.:
I built a KSQL UDF for sensor analytics. It leverages the new API features of KSQL to build UDF / UDAF functions easily with Java to do continuous stream processing on incoming events.
Use Case: Connected Cars - Real Time Streaming Analytics using Deep Learning
Continuously process millions of events from connected devices (sensors of cars in this example).
Machine Learning and Deep Learning Applied to Real Time with Apache Kafka Str...confluent
Big Data and Machine Learning are key for innovation in many industries today. Large amounts of historical data are stored and analyzed in Hadoop, Spark or other clusters to find patterns and insights, e.g. for predictive maintenance, fraud detection or cross-selling.
This first part of the session explains how to build analytic models with R, Python and Scala leveraging open source machine learning / deep learning frameworks like Apache Spark, TensorFlow or H2O.ai. The second part discusses how to leverage these built analytic models in your own streaming applications or microservices; leveraging the Apache Kafka cluster and Kafka Streams instead of building an own stream processing cluster. The session focuses on live demos and teaches lessons learned for executing analytic models in a highly scalable and performant way.
The last part explains how Apache Kafka can help to move from a manual build and deployment of analytic models to continuous online model improvement in real time.
Apache Kafka Streams + Machine Learning / Deep LearningKai Wähner
Machine Learning and Deep Learning Applied to Real Time with Apache Kafka Streams...
Big Data and Machine Learning are key for innovation in many industries today. Large amounts of historical data are stored and analyzed in Hadoop, Spark or other clusters to find patterns and insights, e.g. for predictive maintenance, fraud detection or cross-selling.
This first part of the session explains how to build analytic models with R, Python and Scala leveraging open source machine learning / deep learning frameworks like Apache Spark, TensorFlow or H2O.ai. The second part discusses how to leverage these built analytic models in your own streaming applications or microservices; leveraging the Apache Kafka cluster and Kafka Streams instead of building an own stream processing cluster. The session focuses on live demos and teaches lessons learned for executing analytic models in a highly scalable and performant way.
The last part explains how Apache Kafka can help to move from a manual build and deployment of analytic models to continuous online model improvement in real time.
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniertconfluent
Für die Automobilindustrie ist die digitale Transformation wie für jede andere Branche zugleich eine digitale Revolution: Neue Marktspieler, neue Technologien und die in immer größeren Mengen anfallenden Daten schaffen neue Chancen, aber auch neue Herausforderungen – und erfordern neben neuen IT-Architekturen auch völlig neue Denkansätze.
60% der Fortune500-Unternehmen setzen zur Umsetzung ihrer Daten-Streaming-Projekte auf die umfassende verteilte Streaming-Plattform Apache Kafka®, darunter auch die AUDI AG.
Erfahren Sie in diesem Webinar:
Wie Kafka als Grundlage sowohl für Daten-Pipelines als auch für Anwendungen dient, die Echtzeit-Datenströme konsumieren und verarbeiten.
Wie Kafka Connect und Kafka Streams geschäftskritische Anwendungen unterstützt
Wie Audi mithilfe von Kafka und Confluent eine Fast Data IoT-Plattform umgesetzt hat, die den Bereich „Connected Car“ revolutioniert
Sprecher:
David Schmitz, Principal Architect, Audi Electronics Venture GmbH
Kai Waehner, Technology Evangelist, Confluent
Kafka and Machine Learning in Banking and Insurance IndustryKai Wähner
Streaming Machine Learning and Apache Kafka for real-time analytics-The Next Generation of Intelligent Software for Financial Services and Insurance Industries.
The slides cover use cases, architectures, and examples from various companies. Learn about Kafka + Machine Learning / Deep Learning for fraud detection and other use cases.
Apache Kafka® + Machine Learning for Supply Chain confluent
Watch this talk here: https://www.confluent.io/online-talks/apache-kafka-machine-learning-for-supply-chain
Automating multifaceted, complex workflows requires hybrid solutions like streaming analytics of IoT data, batch analytics like machine learning solutions, and real-time visualizations. Leaders in organizations who are responsible for global supply chain planning are responsible for working with and integrating with data from disparate sources around the world. Many of these data sources output information in real-time, which assists planners in operationalizing plans and interacting with manufacturing output. IoT sensors on manufacturing equipment and inventory control systems feed real-time processing pipelines to match actual production figures against planned schedules to calculate yield efficiency.
Using information from both real-time systems and batch optimization, supply chain managers are able to economize operations and automate tedious inventory and manufacturing accounting processes. Sitting on top of all these systems is a supply chain visualization tool, enabling users' visibility over the global supply chain. If you are responsible for key data integration initiatives, join for a detailed walk through of a customer's use of this system built using Confluent and Expero tools.
WHAT YOU'LL LEARN:
• See different use cases in automation industry and Industrial IoT (IIoT) where an event streaming platform adds business value.
• Understand different architecture options to leverage Apache Kafka and Confluent.
• How to leverage different analytics tools and machine learning frameworks in a flexible and scalable way.
• How real-time visualization ties together streaming and batch analytics for business users, interpreters, and analysts.
• Understand how streaming and batch analytics optimize the supply chain planning workflow.
• Conceptualize the intersection between resource utilization and manufacturing assets with long term planning and supply chain optimization.
IIoT with Kafka and Machine Learning for Supply Chain Optimization In Real Ti...Kai Wähner
I did a webinar with Confluent's partner Expero about "Apache Kafka and Machine Learning for Real Time Supply Chain Optimization". This is a great example for anybody in automation industry / Industrial IoT (IIoT) like automotive, manufacturing, logistics, etc.
We explain how a real time event streaming platform can integrate in real time with the legacy world and proprietary IIoT protocols (like Siemens S7, Modbus, Beckhoff ADS, OPC-UA, et al). You can process the data at scale and then ingest it into a modern database (like AWS S3, Snowflake or MongoDB) or analytic / machine learning framework (like TensorFlow, PyTorch or Azure Machine Learning Service).
Kappa vs Lambda Architectures and Technology ComparisonKai Wähner
Real-time data beats slow data. That’s true for almost every use case. Nevertheless, enterprise architects build new infrastructures with the Lambda architecture that includes separate batch and real-time layers.
This video explores why a single real-time pipeline, called Kappa architecture, is the better fit for many enterprise architectures. Real-world examples from companies such as Disney, Shopify, Uber, and Twitter explore the benefits of Kappa but also show how batch processing fits into this discussion positively without the need for a Lambda architecture.
The main focus of the discussion is on Apache Kafka (and its ecosystem) as the de facto standard for event streaming to process data in motion (the key concept of Kappa), but the video also compares various technologies and vendors such as Confluent, Cloudera, IBM Red Hat, Apache Flink, Apache Pulsar, AWS Kinesis, Amazon MSK, Azure Event Hubs, Google Pub Sub, and more.
Video recording of this presentation:
https://youtu.be/j7D29eyysDw
Further reading:
https://www.kai-waehner.de/blog/2021/09/23/real-time-kappa-architecture-mainstream-replacing-batch-lambda/
https://www.kai-waehner.de/blog/2021/04/20/comparison-open-source-apache-kafka-vs-confluent-cloudera-red-hat-amazon-msk-cloud/
https://www.kai-waehner.de/blog/2021/05/09/kafka-api-de-facto-standard-event-streaming-like-amazon-s3-object-storage/
Simplified Machine Learning Architecture with an Event Streaming Platform (Ap...Kai Wähner
Machine Learning is separated into model training and model inference. ML frameworks typically load historical data from a data store like HDFS or S3 to train models. This talk shows how you can completely avoid such a data store by ingesting streaming data directly via Apache Kafka from any source system into TensorFlow for model training and model inference using the capabilities of “TensorFlow I/O” add-on.
The talk compares this modern streaming architecture to traditional batch and big data alternatives and explains benefits like the simplified architecture, the ability of reprocessing events in the same order for training different models, and the possibility to build a scalable, mission-critical, real time ML architecture with muss less headaches and problems.
Key takeaways for the audience
• Scalable open source Machine Learning infrastructure
• Streaming ingestion into TensorFlow without the need for another data store like HDFS or S3 (leveraging TensorFlow I/O and its Kafka plugin)
• Stream Processing using analytic models in mission-critical deployments to act in Real Time
• Learn how Apache Kafka open source ecosystem including Kafka Connect, Kafka Streams and KSQL help to build, deploy, score and monitor analytic models
• Comparison and trade-offs between this modern streaming approach and traditional batch model training infrastructures
The Rise Of Event Streaming – Why Apache Kafka Changes EverythingKai Wähner
Business digitalization trends like microservices, the Internet of Things or Machine Learning are driving the need to process events at a whole new scale, speed and efficiency. Traditional solutions like ETL/data integration or messaging are not build to serve these needs.
Today, the open source project Apache Kafka® is being used by thousands of companies including over 60% of the Fortune 100 to power and innovate their businesses by focusing their data strategies around event-driven architectures leveraging event streaming.We will discuss the market and technology changes that have given rise to Kafka and to Event Streaming, and we will introduce the audience to the key aspects of building an Event streaming platform with Kafka. Examples of productive use cases from the automotive, manufacturing and transportation sector will showcase the power of event streaming.
Similar to 2019 04 seattle_meetup___kafka_machine_learning___kai_waehner (20)
Water scarcity is the lack of fresh water resources to meet the standard water demand. There are two type of water scarcity. One is physical. The other is economic water scarcity.
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdffxintegritypublishin
Advancements in technology unveil a myriad of electrical and electronic breakthroughs geared towards efficiently harnessing limited resources to meet human energy demands. The optimization of hybrid solar PV panels and pumped hydro energy supply systems plays a pivotal role in utilizing natural resources effectively. This initiative not only benefits humanity but also fosters environmental sustainability. The study investigated the design optimization of these hybrid systems, focusing on understanding solar radiation patterns, identifying geographical influences on solar radiation, formulating a mathematical model for system optimization, and determining the optimal configuration of PV panels and pumped hydro storage. Through a comparative analysis approach and eight weeks of data collection, the study addressed key research questions related to solar radiation patterns and optimal system design. The findings highlighted regions with heightened solar radiation levels, showcasing substantial potential for power generation and emphasizing the system's efficiency. Optimizing system design significantly boosted power generation, promoted renewable energy utilization, and enhanced energy storage capacity. The study underscored the benefits of optimizing hybrid solar PV panels and pumped hydro energy supply systems for sustainable energy usage. Optimizing the design of solar PV panels and pumped hydro energy supply systems as examined across diverse climatic conditions in a developing country, not only enhances power generation but also improves the integration of renewable energy sources and boosts energy storage capacities, particularly beneficial for less economically prosperous regions. Additionally, the study provides valuable insights for advancing energy research in economically viable areas. Recommendations included conducting site-specific assessments, utilizing advanced modeling tools, implementing regular maintenance protocols, and enhancing communication among system components.
Immunizing Image Classifiers Against Localized Adversary Attacksgerogepatton
This paper addresses the vulnerability of deep learning models, particularly convolutional neural networks
(CNN)s, to adversarial attacks and presents a proactive training technique designed to counter them. We
introduce a novel volumization algorithm, which transforms 2D images into 3D volumetric representations.
When combined with 3D convolution and deep curriculum learning optimization (CLO), itsignificantly improves
the immunity of models against localized universal attacks by up to 40%. We evaluate our proposed approach
using contemporary CNN architectures and the modified Canadian Institute for Advanced Research (CIFAR-10
and CIFAR-100) and ImageNet Large Scale Visual Recognition Challenge (ILSVRC12) datasets, showcasing
accuracy improvements over previous techniques. The results indicate that the combination of the volumetric
input and curriculum learning holds significant promise for mitigating adversarial attacks without necessitating
adversary training.
Final project report on grocery store management system..pdfKamal Acharya
In today’s fast-changing business environment, it’s extremely important to be able to respond to client needs in the most effective and timely manner. If your customers wish to see your business online and have instant access to your products or services.
Online Grocery Store is an e-commerce website, which retails various grocery products. This project allows viewing various products available enables registered users to purchase desired products instantly using Paytm, UPI payment processor (Instant Pay) and also can place order by using Cash on Delivery (Pay Later) option. This project provides an easy access to Administrators and Managers to view orders placed using Pay Later and Instant Pay options.
In order to develop an e-commerce website, a number of Technologies must be studied and understood. These include multi-tiered architecture, server and client-side scripting techniques, implementation technologies, programming language (such as PHP, HTML, CSS, JavaScript) and MySQL relational databases. This is a project with the objective to develop a basic website where a consumer is provided with a shopping cart website and also to know about the technologies used to develop such a website.
This document will discuss each of the underlying technologies to create and implement an e- commerce website.
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...Dr.Costas Sachpazis
Terzaghi's soil bearing capacity theory, developed by Karl Terzaghi, is a fundamental principle in geotechnical engineering used to determine the bearing capacity of shallow foundations. This theory provides a method to calculate the ultimate bearing capacity of soil, which is the maximum load per unit area that the soil can support without undergoing shear failure. The Calculation HTML Code included.
Explore the innovative world of trenchless pipe repair with our comprehensive guide, "The Benefits and Techniques of Trenchless Pipe Repair." This document delves into the modern methods of repairing underground pipes without the need for extensive excavation, highlighting the numerous advantages and the latest techniques used in the industry.
Learn about the cost savings, reduced environmental impact, and minimal disruption associated with trenchless technology. Discover detailed explanations of popular techniques such as pipe bursting, cured-in-place pipe (CIPP) lining, and directional drilling. Understand how these methods can be applied to various types of infrastructure, from residential plumbing to large-scale municipal systems.
Ideal for homeowners, contractors, engineers, and anyone interested in modern plumbing solutions, this guide provides valuable insights into why trenchless pipe repair is becoming the preferred choice for pipe rehabilitation. Stay informed about the latest advancements and best practices in the field.
NO1 Uk best vashikaran specialist in delhi vashikaran baba near me online vas...Amil Baba Dawood bangali
Contact with Dawood Bhai Just call on +92322-6382012 and we'll help you. We'll solve all your problems within 12 to 24 hours and with 101% guarantee and with astrology systematic. If you want to take any personal or professional advice then also you can call us on +92322-6382012 , ONLINE LOVE PROBLEM & Other all types of Daily Life Problem's.Then CALL or WHATSAPP us on +92322-6382012 and Get all these problems solutions here by Amil Baba DAWOOD BANGALI
#vashikaranspecialist #astrologer #palmistry #amliyaat #taweez #manpasandshadi #horoscope #spiritual #lovelife #lovespell #marriagespell#aamilbabainpakistan #amilbabainkarachi #powerfullblackmagicspell #kalajadumantarspecialist #realamilbaba #AmilbabainPakistan #astrologerincanada #astrologerindubai #lovespellsmaster #kalajaduspecialist #lovespellsthatwork #aamilbabainlahore#blackmagicformarriage #aamilbaba #kalajadu #kalailam #taweez #wazifaexpert #jadumantar #vashikaranspecialist #astrologer #palmistry #amliyaat #taweez #manpasandshadi #horoscope #spiritual #lovelife #lovespell #marriagespell#aamilbabainpakistan #amilbabainkarachi #powerfullblackmagicspell #kalajadumantarspecialist #realamilbaba #AmilbabainPakistan #astrologerincanada #astrologerindubai #lovespellsmaster #kalajaduspecialist #lovespellsthatwork #aamilbabainlahore #blackmagicforlove #blackmagicformarriage #aamilbaba #kalajadu #kalailam #taweez #wazifaexpert #jadumantar #vashikaranspecialist #astrologer #palmistry #amliyaat #taweez #manpasandshadi #horoscope #spiritual #lovelife #lovespell #marriagespell#aamilbabainpakistan #amilbabainkarachi #powerfullblackmagicspell #kalajadumantarspecialist #realamilbaba #AmilbabainPakistan #astrologerincanada #astrologerindubai #lovespellsmaster #kalajaduspecialist #lovespellsthatwork #aamilbabainlahore #Amilbabainuk #amilbabainspain #amilbabaindubai #Amilbabainnorway #amilbabainkrachi #amilbabainlahore #amilbabaingujranwalan #amilbabainislamabad
About
Indigenized remote control interface card suitable for MAFI system CCR equipment. Compatible for IDM8000 CCR. Backplane mounted serial and TCP/Ethernet communication module for CCR remote access. IDM 8000 CCR remote control on serial and TCP protocol.
• Remote control: Parallel or serial interface.
• Compatible with MAFI CCR system.
• Compatible with IDM8000 CCR.
• Compatible with Backplane mount serial communication.
• Compatible with commercial and Defence aviation CCR system.
• Remote control system for accessing CCR and allied system over serial or TCP.
• Indigenized local Support/presence in India.
• Easy in configuration using DIP switches.
Technical Specifications
Indigenized remote control interface card suitable for MAFI system CCR equipment. Compatible for IDM8000 CCR. Backplane mounted serial and TCP/Ethernet communication module for CCR remote access. IDM 8000 CCR remote control on serial and TCP protocol.
Key Features
Indigenized remote control interface card suitable for MAFI system CCR equipment. Compatible for IDM8000 CCR. Backplane mounted serial and TCP/Ethernet communication module for CCR remote access. IDM 8000 CCR remote control on serial and TCP protocol.
• Remote control: Parallel or serial interface
• Compatible with MAFI CCR system
• Copatiable with IDM8000 CCR
• Compatible with Backplane mount serial communication.
• Compatible with commercial and Defence aviation CCR system.
• Remote control system for accessing CCR and allied system over serial or TCP.
• Indigenized local Support/presence in India.
Application
• Remote control: Parallel or serial interface.
• Compatible with MAFI CCR system.
• Compatible with IDM8000 CCR.
• Compatible with Backplane mount serial communication.
• Compatible with commercial and Defence aviation CCR system.
• Remote control system for accessing CCR and allied system over serial or TCP.
• Indigenized local Support/presence in India.
• Easy in configuration using DIP switches.
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptxR&R Consult
CFD analysis is incredibly effective at solving mysteries and improving the performance of complex systems!
Here's a great example: At a large natural gas-fired power plant, where they use waste heat to generate steam and energy, they were puzzled that their boiler wasn't producing as much steam as expected.
R&R and Tetra Engineering Group Inc. were asked to solve the issue with reduced steam production.
An inspection had shown that a significant amount of hot flue gas was bypassing the boiler tubes, where the heat was supposed to be transferred.
R&R Consult conducted a CFD analysis, which revealed that 6.3% of the flue gas was bypassing the boiler tubes without transferring heat. The analysis also showed that the flue gas was instead being directed along the sides of the boiler and between the modules that were supposed to capture the heat. This was the cause of the reduced performance.
Based on our results, Tetra Engineering installed covering plates to reduce the bypass flow. This improved the boiler's performance and increased electricity production.
It is always satisfying when we can help solve complex challenges like this. Do your systems also need a check-up or optimization? Give us a call!
Work done in cooperation with James Malloy and David Moelling from Tetra Engineering.
More examples of our work https://www.r-r-consult.dk/en/cases-en/
1. 1Apache Kafka and Machine Learning – Kai Waehner
Scalable, Mission-Critical Machine Learning
Infrastructures with Apache Kafka
Unleashing Apache Kafka and TensorFlow in Hybrid Cloud Architectures
Kai Waehner
Technology Evangelist
kontakt@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de
2. 3Apache Kafka and Machine Learning – Kai Waehner
Disclaimer: This is a fictional story (but not far from reality)…
3. 4Apache Kafka and Machine Learning – Kai Waehner
Global automotive company builds connected car infrastructure
Digital Transformation
• Improve customer experience
• Increase revenue
• Reduce risk
Time
Today 2 years in the future3 years ago
Project begins Connected car
infrastructure in production
for first use cases
Improved processes
leveraging machine learning
4. 5Apache Kafka and Machine Learning – Kai Waehner
Analyze and act on critical business moments
Seconds Minutes Hours
Real Time
Tracking
Predictive
Maintenance
Fraud
Detection
Cross Selling
Transportation
Rerouting
Customer
Service
Inventory
Management
Windows of Opportunity
5. 6Apache Kafka and Machine Learning – Kai Waehner
Machine Learning (ML)
...allows computers to find hidden insights without being explicitly
programmed where to look.
Machine Learning
• Decision Trees
• Naïve Bayes
• Clustering
• Neural Networks
• Etc.
Deep Learning
• CNN
• RNN
• Autoencoder
• Etc.
6. 7Apache Kafka and Machine Learning – Kai Waehner
The First Analytic Models
How to deploy the models
in production?
…real-time processing?
…at scale?
…24/7 zero downtime?
7. 8Apache Kafka and Machine Learning – Kai Waehner
Hidden Technical Debt in Machine Learning Systems
https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
8. 10Apache Kafka and Machine Learning – Kai Waehner
Scalable, Technology-Agnostic ML Infrastructures
https://www.infoq.com/presentations/netflix-ml-meson
https://eng.uber.com/michelangelo
https://www.infoq.com/presentations/paypal-data-service-fraud
What is this
thing used everywhere?
9. 11Apache Kafka and Machine Learning – Kai Waehner
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
Apache Kafka—The Rise of a Streaming Platform
10. 12Apache Kafka and Machine Learning – Kai Waehner
Apache Kafka at Scale at Tech Giants
> 4.5 trillion messages / day > 6 Petabytes / day
“You name it”
* Kafka Is not just used by tech giants
** Kafka is not just used for big data
11. Confluents - Business Value per Use Case
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save
money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation & improved
in-car experience: Audi
Customer 360
Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log
aggregation, Splunk
replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka
Streams: Funding Circle
Detect Fraud & Prevent Fraud in Real
Time: PayPal
Kafka as a Service - A Tale of Security
and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$
Example Case Studies
(of many)
12. 14Apache Kafka and Machine Learning – Kai Waehner
Apache Kafka’s Open Source Ecosystem as Infrastructure for ML
13. 15Apache Kafka and Machine Learning – Kai Waehner
Apache Kafka’s Open Ecosystem as Infrastructure for ML
Kafka
Streams
Kafka
Connect
Rest Proxy
Schema Registry
Go/.NET /Python
Kafka Producer
KSQL
Kafka
Streams
14. 16Apache Kafka and Machine Learning – Kai Waehner
Getting Started
Okay, let’s build our own
ML infrastructure step by
step. Where do we start?
15. 17Apache Kafka and Machine Learning – Kai Waehner
Connected Car Infrastructure in Production on AWS
Kafka BrokerKafka BrokerKafka Broker
MQTT
ProxyMQTT
DevicesDevicesDevicesGateways
DevicesDevicesDevicesDevices MQTT
Real time tracking of the cars
to enable new, innovative digital services
The big data team
has the data already.
16. 19Apache Kafka and Machine Learning – Kai Waehner
Replication of IoT Data from AWS to GCP
Replication
Confluent Replicator
DevicesDevicesDevicesDevicesDevices
Analytics
We should also use
Kafka, but—oh no…GCP
is the strategic cloud
for the analytics team!
17. 21Apache Kafka and Machine Learning – Kai Waehner
Data Preprocessing
Preprocessing
Filter, transform, anonymize, extract features
Data needs to be
preprocessed at
scale and reusable!
Streams
• Use KSQL to preprocess data at scale without coding
• Use SQL statements for interactive analysis
+ deployment to production at scale
• Leverage e.g. Python with KSQL REST interface
Data Ready
for
Model Training
18. 22Apache Kafka and Machine Learning – Kai Waehner
Preprocessing with KSQL
SELECT car_id, event_id, car_model_id, sensor_input
FROM car_sensor c
LEFT JOIN car_models m ON c.car_model_id =
m.car_model_id
WHERE m.car_model_type ='Audi_A8';
19. 23Apache Kafka and Machine Learning – Kai Waehner
Data Ingestion
Connect
• “Kafka Benefits Under the Hood”
• Out-of-the-box connectivity
• Data format conversion
• Single message transformation
(including error-handling)
Preprocessed
Data
There isn’t just
one ML solution.
We need to be
flexible!
20. 24Apache Kafka and Machine Learning – Kai Waehner
Model Training
Let’s build some models
at extreme scale using
TensorFlow and TPUs!
Analytic Model
21. 25Apache Kafka and Machine Learning – Kai Waehner
Analytic Model (Autoencoder for Anomaly Detection)
22. 27Apache Kafka and Machine Learning – Kai Waehner
Replayability — a log never forgets!
Time
Model B Model XModel A
Producer
Distributed Commit Log
Different models with same data
Different ML frameworks
AutoML compatible
A/B testing
Google Cloud Storage HDFS
23. 28Apache Kafka and Machine Learning – Kai Waehner
The Need for Local Data Processing
Confluent
Replicator
PII data Local Processing
We are ready to use our
models for predictions,
BUT all the PII data needs to
be processed in our local data
center!
CLOUD
ON PREMISE
Analytic
Model
24. 31Apache Kafka and Machine Learning – Kai Waehner
Model Deployment - Option 1:
RPC communication to do model inference
Streams
Input Event
Prediction
Request
Response
Model Serving
TensorFlow Serving
gRPC
25. 32Apache Kafka and Machine Learning – Kai Waehner
Model Deployment - Option 2:
Model interference natively integrated into the App
Streams
Input Event
Prediction
26. 34Apache Kafka and Machine Learning – Kai Waehner
Confluent Schema Registry for Message Validation
Input Data
Schema
Registry
App 1
• “Kafka Benefits Under the Hood”
• Schema definition + evolution
• Forward and backward compatibility
• Multi data center deployment
I am a little bit worried.
How can we ensure every
team in every data center
produces and consumers
correct data?
App X
27. 35Apache Kafka and Machine Learning – Kai Waehner
Monitoring the infrastructure for ML
Kafka
Streams
Kafka
Connect
Rest Proxy
Schema Registry
Go / .NET / Python
Kafka Producer
KSQL
Kafka
Streams
Control Center
Build vs. Buy
Hosted vs. Managed
Basic vs. Advanced
28. 36Apache Kafka and Machine Learning – Kai Waehner
KSQL and Deep Learning (Auto Encoder) for Anomaly Detection
MQTT
Proxy
Elastic
search
Grafana
Kafka
Cluster
Kafka
Connect
KSQL
Car Sensors
Kafka Ecosystem
Other Components
Real Time
Emergency
System
All Data
PotentialDefect
Apply
Analytic
Model
Filter
Anomalies
On premise DCAt the edge
5858
KSQL and Deep Learning (Auto Encoder) for Anomaly Detection
MQTT
Proxy
Elastic
search
Grafana
Kafka
Cluster
Kafka
Connect
KSQL
Car Sensors
Kafka Ecosystem
Other Components
Real Time
Emergency
System
All Data
PotentialDefect
Apply
Analytic
Model
Filter
Anomalies
On premise DCAt the edge
29. 37Apache Kafka and Machine Learning – Kai Waehner
Model Training with Python, KSQL, TensorFlow, Keras and Jupyter
https://github.com/kaiwaehner/python-jupyter-apache-kafka-ksql-tensorflow-keras
30. 38Apache Kafka and Machine Learning – Kai Waehner
Model Deployment with Apache Kafka, KSQL and TensorFlow
“CREATE STREAM AnomalyDetection AS
SELECT sensor_id, detectAnomaly(sensor_values)
FROM car_engine;“
User Defined Function (UDF)
31. 39Apache Kafka and Machine Learning – Kai Waehner
Live Demo
End-to-End Sensor Analytics…
Python, Jupyter Notebook, TensorFlow, Keras, Apache Kafka, KSQL and MQTT
32. 40Apache Kafka and Machine Learning – Kai Waehner
Model Training with Python, KSQL, TensorFlow, Keras and Jupyter
https://github.com/kaiwaehner/python-jupyter-apache-kafka-ksql-tensorflow-keras
33. 41Apache Kafka and Machine Learning – Kai Waehner
Deep Learning UDF for KSQL for Streaming Anomaly Detection of MQTT IoT Sensor Data
https://github.com/kaiwaehner/ksql-udf-deep-learning-mqtt-iot
34. 44Apache Kafka and Machine Learning – Kai Waehner
Confluent Delivers a Mission-Critical Event Streaming Platform
Apache Kafka®
Core | Connect API | Streams API
Data Compatibility
Schema Registry
Enterprise Operations
Replicator | Auto Data Balancer | Connectors | MQTT Proxy | Kubernetes Operator
Database
Changes
Log Events IoT Data Web Events other events
Hadoop
Database
Data
Warehouse
CRM
other
DATA INTEGRATION
Transformations
Custom Apps
Analytics
Monitoring
other
REAL-TIME APPLICATIONS
COMMUNITY FEATURES COMMERCIAL FEATURES
Datacenter Public Cloud Confluent Cloud
Confluent Platform
Management & Monitoring
Control Center | Security
Development & Connectivity
Clients | Connectors | REST Proxy | KSQL
CONFLUENT FULLY-MANAGEDCUSTOMER SELF-MANAGED
35. 45Apache Kafka and Machine Learning – Kai Waehner
Best-of-breed Platforms, Partners and Services for Multi-cloud Streams
Private Cloud
Deploy on bare-metal, VMs,
containers or Kubernetes in your
datacenter with Confluent Platform
and Confluent Operator
Public Cloud
Implement self-managed in the public
cloud or adopt a fully managed service
with Confluent Cloud
Hybrid Cloud
Build a persistent bridge between
datacenter and cloud with
Confluent Replicator
Confluent
Replicator
VM
SELF MANAGED FULLY MANAGED
36. 46Apache Kafka and Machine Learning – Kai Waehner
Confluent’s Streaming Maturity Model - where are you?
Value
Maturity (Investment & time)
2
Enterprise
Streaming Pilot /
Early Production
Pub + Sub Store Process
5
Central Nervous
System
1
Developer
Interest
Pre-Streaming
4
Global
Streaming
3
SLA Ready,
Integrated
Streaming
Projects
Platform
37. 48Apache Kafka and Machine Learning – Kai Waehner
Kai Waehner
Technology Evangelist
kontakt@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Please contact me!