Uploaded byKai Wähner
Apache Kafka Open Source Ecosystem for Machine Learning at Extreme Scale (ApacheCon, Montreal, Sept 2018)
The document discusses building a scalable machine learning infrastructure using the Apache Kafka ecosystem, highlighting its ability to handle extreme scale data with high event volume and dynamic scalability. It covers topics including data ingestion, real-time model inference, and monitoring machine learning systems, emphasizing the importance of integration with various tools and frameworks. The document also provides real-world examples, use cases, and the necessary components of Kafka for effective machine learning applications.
More Related Content
Similar to Apache Kafka Open Source Ecosystem for Machine Learning at Extreme Scale (ApacheCon, Montreal, Sept 2018)
Apache Kafka Open Source Ecosystem for Machine Learning at Extreme Scale (ApacheCon, Montreal, Sept 2018)
- 1. 1Confidential Machine Learning atExtreme Scale with the Apache Kafka Open Source Ecosystem Kai Waehner Technology Evangelist kontakt@kai-waehner.de LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de How to build a Scalable, Mission-Critical Machine Learning Infrastructure
- 2. 2Apache Kafka andMachine Learning – Kai Waehner Poll In which of these environments, if at all, are you using Apache Kafka today? 1. On premise 2. A public cloud 3. Our own virtual private cloud 4. We do not currently use Apache Kafka
- 3. 3Apache Kafka andMachine Learning – Kai Waehner What is eXtreme Scale? • High Volume of Events (millions, billions, trillions) • Big Data Sets for Analytics (GB, TB, PB) • Dynamic Scalability for Training (minutes, hours, days) • Real Time Prediction Process for Deployment (ms) • Hybrid Deployments (different frameworks and clouds)
- 4. 4Apache Kafka andMachine Learning – Kai Waehner Agenda 1) Added Business Value via Machine Learning 2) Apache Kafka Ecosystem as Infrastructure for Machine Learning 3) Data Ingestion and Preprocessing with Apache Kafka for Model Training 4) Model Inference in Real Time with Kafka Streams and KSQL 5) DevOps and Monitoring of a Machine Learning Infrastructure
- 5. 5Apache Kafka andMachine Learning – Kai Waehner Agenda 1) Added Business Value via Machine Learning 2) Apache Kafka Ecosystem as Infrastructure for Machine Learning 3) Data Ingestion and Preprocessing with Apache Kafka for Model Training 4) Model Inference in Real Time with Kafka Streams and KSQL 5) DevOps and Monitoring of a Machine Learning Infrastructure
- 6. 6Apache Kafka andMachine Learning – Kai Waehner Machine Learning ... allows computers to find hidden insights without being explicitly programmed where to look. Machine Learning • Decision Trees • Naïve Bayes • Clustering • Neural Networks • etc. Deep Learning • CNN • RNN • Autoencoder • etc.
- 7. 7Apache Kafka andMachine Learning – Kai Waehner Real World Examples of Machine Learning Spam Detection Search Results + Product Recommendation Picture Detection (Friends, Locations, Products) Your Company The Next Disruption: Google Beats Go Champion
- 8. 8Apache Kafka andMachine Learning – Kai Waehner Leverage Machine Learning to Analyze and Act on Critical Business Moments Seconds Minutes Hours Price Optimization Predictive Maintenance Fraud Detection Cross Selling Transportation Rerouting Customer Service Inventory Management Windows of Opportunity
- 9. 9Apache Kafka andMachine Learning – Kai Waehner Live Demo – Building an Analytic Model Neural Networks in Action http://playground.tensorflow.org/
- 10. 10Apache Kafka andMachine Learning – Kai Waehner Languages, Frameworks and Tools for Machine Learning There is no Allrounder à ML-independent infrastructure needed! Portable Format for Analytics (PFA)
- 11. 11Apache Kafka andMachine Learning – Kai Waehner Machine Learning with H2O.ai H2O Engine R / Python / Scala / Flow UI Java Code
- 12. 12Apache Kafka andMachine Learning – Kai Waehner Live Demo – Building an Analytic Model Use Case: Airline Flight Delay Prediction Machine Learning Algorithm: Deep Learning using Neural Networks Technology: H2O.ai, TensorFlow
- 13. 13Apache Kafka andMachine Learning – Kai Waehner H2O Deep Water (TensorFlow, MXNet, …) https://h2o-release.s3.amazonaws.com/h2o/rel-vapnik/1/ docs-website/h2o-docs/booklets/DeepWaterBooklet.pdf Deep Water (H2O + TensorFlow) Pre-Defined Networks + User-Defined Networks
- 14. 14Apache Kafka andMachine Learning – Kai Waehner Hidden Technical Debt in Machine Learning Systems https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
- 15. 15Apache Kafka andMachine Learning – Kai Waehner Agenda 1) Added Business Value via Machine Learning 2) Apache Kafka Ecosystem as Infrastructure for Machine Learning 3) Data Ingestion and Preprocessing with Apache Kafka for Model Training 4) Model Inference in Real Time with Kafka Streams and KSQL 5) DevOps and Monitoring of a Machine Learning Infrastructure
- 16. 16Apache Kafka andMachine Learning – Kai Waehner The Log ConnectorsConnectors Producer Consumer Streaming Engine Apache Kafka – The Rise of a Streaming Platform
- 17. 17Apache Kafka andMachine Learning – Kai Waehner Apache Kafka at Scale https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/63921 https://qconlondon.com/london2018/presentation/cloud-native-and-scalable-kafka-architecture (2018) (2018)
- 18. 18Apache Kafka andMachine Learning – Kai Waehner Netflix’ Meson: Automation Engine for ML Pipelines https://www.infoq.com/presentations/netflix-ml-meson
- 19. 19Apache Kafka andMachine Learning – Kai Waehner Uber’s internal ML-as-a-Service Platform https://eng.uber.com/michelangelo • Cover the end-to-end ML workflow: manage data, train, evaluate, and deploy models, make predictions, and monitor predictions • Supports various AI technologies: Traditional ML models, time series forecasting, and deep learning
- 20. 20Apache Kafka andMachine Learning – Kai Waehner Paypal: Real Time Fraud Detection at Scale https://www.infoq.com/presentations/paypal-data-service-fraud “Scalable model / framework independent infrastructure for fraud detection”
- 21. 21Apache Kafka andMachine Learning – Kai Waehner Apache Kafka’s Open Source Ecosystem as Infrastructure for Machine Learning
- 22. 22Apache Kafka andMachine Learning – Kai Waehner Apache Kafka’s Open Source Ecosystem as Infrastructure for Machine Learning Kafka Streams Kafka Connect Rest Proxy Schema Registry Go / .NET / Python Kafka Producer KSQL Kafka Streams
- 23. 23Apache Kafka andMachine Learning – Kai Waehner Poll Which components of the Apache Kafka open source ecosystem do you use? 1) Kafka Connect 2) Kafka Streams 3) Confluent Open Source (Schema Registry, REST Proxy, KSQL, non-Java Clients, …) 4) Other Open Source Add-Ons (Kafka Manager, node-rdkafka Client, …) 5) Only Kafka for Messaging / Storage + Java Producers and Consumers
- 24. 24Apache Kafka andMachine Learning – Kai Waehner Replay-ability – A log never forgets! Time Model B Model XModel A Producer Distributed Commit Log Different models with same data Different ML Frameworks AutoML compatible A/B Testing
- 25. 25Apache Kafka andMachine Learning – Kai Waehner AutoML à No Data Scientist available for the ML Tasks? Hidden Technical Debt in Machine Learning Systems http://slideplayer.com/slide/10575150/ “One-Click Data-In Model-Out simplicity” This talk does not focus on building models, but scalable infrastructure for ML
- 26. 26Apache Kafka andMachine Learning – Kai Waehner Agenda 1) Added Business Value via Machine Learning 2) Apache Kafka Ecosystem as Infrastructure for Machine Learning 3) Data Ingestion and Preprocessing with Apache Kafka for Model Training 4) Model Inference in Real Time with Kafka Streams and KSQL 5) DevOps and Monitoring of a Machine Learning Infrastructure
- 27. 27Apache Kafka andMachine Learning – Kai Waehner Kafka Connect for Data Integration “Kafka benefits under the hood” Out-of-the-Box Connectivity Data Format Conversion Simple Message Transformation Connect Syslog (“you name the interface”)
- 28. 28Apache Kafka andMachine Learning – Kai Waehner Kafka Connect Configuration for MQTT Connector curl -s -X POST -H 'Content-Type: application/json' http://localhost:8083/connectors -d '{ "name" : "mqtt-source", "config" : { "connector.class" : "io.confluent.connect.mqtt.MqttSourceConnector", "tasks.max" : "1", "mqtt.server.uri" : "tcp://127.0.0.1:1883", "mqtt.topics" : "temperature", "kafka.topics" : "mqtt.", "transforms":"filter", "transforms.filter.type":"com.github.kaiwaehner.kafka.connect.smt.StringFilter", "transforms.filter.topic.format":"fraud" } }'
- 29. 29Apache Kafka andMachine Learning – Kai Waehner Kafka Streams / KSQL for Data Preprocessing Streams “Kafka benefits under the hood” Streaming ETL Same Pipeline for Training and Serving Input Data Output Data
- 30. 30Apache Kafka andMachine Learning – Kai Waehner KSQL – Continuous Query for Streaming ETL CREATE STREAM vip_actions AS SELECT userid, page, action FROM clickstream c LEFT JOIN users u ON c.userid = u.user_id WHERE u.level = 'Platinum';
- 31. 31Apache Kafka andMachine Learning – Kai Waehner Confluent Schema Registry for Message Validation Input Data Schema Registry Output Data“Kafka benefits under the hood” Schema Definition + Evolution Forward and Backward Compatibility
- 32. 32Apache Kafka andMachine Learning – Kai Waehner Kafka Connect for Data Ingestion Connect “Kafka benefits under the hood” Out-of-the-Box Connectivity Data Format Conversion Simple Message Transformation
- 33. 33Apache Kafka andMachine Learning – Kai Waehner Model Training Benefits of Public Cloud Extreme Scale Dynamic Instances Special Hardware
- 34. 34Apache Kafka andMachine Learning – Kai Waehner Agenda 1) Added Business Value via Machine Learning 2) Apache Kafka Ecosystem as Infrastructure for Machine Learning 3) Data Ingestion and Preprocessing with Apache Kafka for Model Training 4) Model Inference in Real Time with Kafka Streams and KSQL 5) DevOps and Monitoring of a Machine Learning Infrastructure
- 35. 35Apache Kafka andMachine Learning – Kai Waehner Model Serving / Inference / Deployment / Scoring Kafka Streams KSQL “Kafka benefits under the hood” Continuous Stream Processing Reuse Preprocessing Logic from Ingestion Pipeline Serving within the application (not via REST interface) Predictions in real time Streams Input Event Prediction
- 36. 36Apache Kafka andMachine Learning – Kai Waehner Stream Processing Data at Rest Data in Motion
- 37. 37Apache Kafka andMachine Learning – Kai Waehner Kafka Streams (shipped with Apache Kafka) / KSQL (Confluent Open Source)
- 38. 38Apache Kafka andMachine Learning – Kai Waehner Stream Processing Pipeline APIs Adapters / Channels Integration Messaging Stream Ingest Transformation Aggregation Enrichment Filtering Stream Preprocessing Process Management Analytics (Real Time) Applications & APIs Analytics / DW Reporting Stream Outcomes • Contextual Rules • Windowing • Patterns • Analytics • Machine Learning • … Stream Analytics Index / SearchNormalization Applying an Analytic Model is just a piece of the puzzle!
- 39. 39Apache Kafka andMachine Learning – Kai Waehner A complete streaming microservices, ready for production at large-scale Word Count App configuration Define processing (here: WordCount) Start processing
- 40. 40Apache Kafka andMachine Learning – Kai Waehner Why KSQL? Population CodingSophistication Realm of Stream Processing New, Expanded Realm BI Analysts Core Developers Data Engineers Core Developers who don’t like Java Kafka Streams KSQL
- 41. 41Apache Kafka andMachine Learning – Kai Waehner Trade-Offs • subscribe() • poll() • send() • flush() • mapValues() • filter() • punctuate() • Select…from… • Join…where… • Group by.. Flexibility Simplicity Kafka Streams KSQL Kafka / Consumer Producer
- 42. 42Apache Kafka andMachine Learning – Kai Waehner When to use Kafka Streams or KSQL for Stream Processing?
- 43. 43Apache Kafka andMachine Learning – Kai Waehner Kafka Streams (shipped with Apache Kafka) / KSQL (Confluent Open Source) Map, filter, aggregate, apply analytic model, „any business logic“ Input Stream (Kafka Topic) Kafka Cluster Output Stream (Kafka Topic) Kafka Cluster Stream Processing Microservice (Kafka Streams / KSQL) Deployed Anywhere Java App, Docker, Kubernetes, Mesos, “you-name-it”
- 44. 44Apache Kafka andMachine Learning – Kai Waehner Kafka Streams and KSQL Ok. Ok. Ok. are viable for S / M / L / XL / XXL use cases
- 45. 45Apache Kafka andMachine Learning – Kai Waehner Option 1: RPC communication to do model inference Streams Input Event Prediction Request Response Model Serving TensorFlow Serving RPC (e.g. gRPC, HTTP)
- 46. 46Apache Kafka andMachine Learning – Kai Waehner Option 2: Model interference natively integrated into the App Streams Input Event Prediction
- 47. 47Apache Kafka andMachine Learning – Kai Waehner Stream Processing vs. Request-Response for Model Serving Pros of a Model Server: • Simple integration with existing technologies and organizational processes • Easier to understand if you come from non- streaming world • Later migration to real streaming is also possible • Model management built-in for different models, versioning and A/B testing Cons (== Pros of Deployment in the Streaming App): • Worse latency as remote call instead of local inference • No offline inference (devices, edge processing, etc.) • Coupling the availability, scalability, and latency/throughput of your Kafka Streams application with the SLAs of the RPC interface • Side-effects (e.g., in case of failure) not covered by Kafka processing (e.g., exactly once)
- 48. 48Apache Kafka andMachine Learning – Kai Waehner Use Case: Airline Flight Delay Prediction Machine Learning Algorithm: Neural Network built with H2O and TensorFlow Streaming Platform: Apache Kafka and Kafka Streams Live Demo – Deployment of a Trained Model Streams
- 49. 49Apache Kafka andMachine Learning – Kai Waehner H2O.ai Model + Kafka Streams Filter Map 1) Create H2O DL model 2) Configure Kafka Streams Application 3) Apply H2O DL model to Streaming Data 4) Start Kafka Streams App
- 50. 50Apache Kafka andMachine Learning – Kai Waehner Github Examples: Kafka + Machine Learning https://www.confluent.io/blog/build-deploy-scalable-machine-learning-production-apache-kafka/ https://github.com/kaiwaehner/kafka-streams-machine-learning-examples 1) git clone à 2) mvn clean package à 3) look at implementations and unit tests
- 51. 51Apache Kafka andMachine Learning – Kai Waehner KSQL and Deep Learning (Autoencoder) for Fraud Detection “CREATE STREAM FraudDetection AS SELECT payment_id, applyFraudModel(payment_input) FROM payment_engine;“ User Defined Function (UDF) https://www.confluent.io/blog/build-udf-udaf-ksql-5-0 https://github.com/kaiwaehner/ksql-udf-deep-learning-mqtt-iot
- 52. 52Apache Kafka andMachine Learning – Kai Waehner Use Case: Anomaly Detection (Payment Fraud Detection) Machine Learning Algorithm: Autoencoder built with H2O Streaming Platform: Apache Kafka and KSQL Live Demo – Prebuilt Model Embedded in KSQL Function
- 53. 53Apache Kafka andMachine Learning – Kai Waehner Github Examples: KSQL + Deep Learning https://www.confluent.io/blog/build-deploy-scalable-machine-learning-production-apache-kafka/ https://github.com/kaiwaehner/ksql-udf-deep-learning-mqtt-iot https://github.com/kaiwaehner/ksql-fork-with-deep-learning-function + Kafka Connect + Elasticsearch
- 54. 54Apache Kafka andMachine Learning – Kai Waehner Agenda 1) Added Business Value via Machine Learning 2) Apache Kafka Ecosystem as Infrastructure for Machine Learning 3) Data Ingestion and Preprocessing with Apache Kafka for Model Training 4) Model Inference in Real Time with Kafka Streams and KSQL 5) DevOps and Monitoring of a Machine Learning Infrastructure
- 55. 55Apache Kafka andMachine Learning – Kai Waehner Automated Model Improvement with Apache Kafka and Kafka Streams How to deploy a model once, then monitor and improve it continuously?
- 56. 56Apache Kafka andMachine Learning – Kai Waehner Automated Model Improvement with Apache Kafka and Kafka Streams How to improve models? 1. Manual Update 2. Continuous Batch Updating 3. Real Time à Online Model Training Your choice… All possible with Kafka!
- 57. 57Apache Kafka andMachine Learning – Kai Waehner Caveats for Online Model Training • Processes and infrastructure not ready • Validation needed before production • Slows down the system • Only a few ML implementations à Build your own! • Only possible for unsupervised ML (e.g. clustering) • Many use cases do not need it à Do it only when feasible!
- 58. 58Apache Kafka andMachine Learning – Kai Waehner Continuous Batch Updating as “Best Feasible Option” DevOps Pipeline 1. Apply the model online to make predictions 2. Collect data and train a new model 3. Automated Re-Deployment (e.g. via a Kafka Topic) https://www.confluent.io/blog/predicting-flight-arrivals-with-the-apache-kafka-streams-api/ https://www.coveros.com/services/devops/
- 59. 59Apache Kafka andMachine Learning – Kai Waehner Kubernetes – The Winner of the Container and DevOps Wars! https://www.infoworld.com/article/3118345/cloud-computing/why-kubernetes-is-winning-the-container-war.html http://techgenix.com/year-of-kubernetes/ Docker, Inc
- 60. 60Apache Kafka andMachine Learning – Kai Waehner Kubernetes for Infrastructure Deployment https://kubernetes.io/blog/2016/10/kubernetes-and-openstack-at-yahoo-japan Stateful Backend • Zookeeper and Kafka Broker Pods • REST Proxy, Schema Registry • Persistent Volumes • Kubernetes Kafka / Confluent Operator Stateless and Stateful Clients • Java / .Net / Go / Python Kafka Clients • Kafka Streams / KSQL Apps • Scalability and Elasticity
- 61. 61Apache Kafka andMachine Learning – Kai Waehner Monitoring the Infrastructure for Machine Learning Kafka Streams Kafka Connect Rest Proxy Schema Registry Go / .NET / Python Kafka Producer KSQL Kafka Streams Control Center Build vs. Buy Hosted vs. Managed Basic vs. Advanced
- 62. 62Apache Kafka andMachine Learning – Kai Waehner Warning: Early Stage with focus on TensorFlow Training, TensorFlow Serving, Jupyter… Bigger ecosystem expected soon… Including Kafka components for ingestion, serving, monitoring… Kubernetes Deployment of ML Workflows https://github.com/kubeflow/kubeflow
- 63. 63Apache Kafka andMachine Learning – Kai Waehner Poll Which of the following use cases are you most likely to utilize Kafka for over the next year? 1. Data pipeline (not real time, e.g. batch in Spark) 2. Data pipeline (real time) 3. Stream processing (e.g. Kafka Streams, KSQL) 4. Stream processing with machine learning 5. Other (like microservices, event sourcing, storage)
- 64. 64Apache Kafka andMachine Learning – Kai Waehner Key Takeaways à Data Scientist and Developers have to work together continuously (org + tech!) à Mission critical, scalable production infrastructure is key for success of Machine Learning projects à Apache Kafka Ecosystem + Cloud = Machine Learning at Extreme Scale (Ingestion, Processing, Training, Inference, Monitoring)
- 65. 65Apache Kafka andMachine Learning – Kai Waehner Kai Waehner Technology Evangelist kontakt@kai-waehner.de @KaiWaehner www.kai-waehner.de www.confluent.io LinkedIn Questions? Feedback? Please contact me!