Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Apache Kafka Open Source Ecosystem for Machine Learning at Extreme Scale (ApacheCon, Montreal, Sept 2018)

309 views

Published on

This talk shows how to productionize Machine Learning models in mission-critical and scalable real time applications by leveraging Apache Kafka as streaming platform. The talk discusses the relation between Machine Learning frameworks such as TensorFlow, DeepLearning4J or H2O and the Apache Kafka ecosystem. A live demo shows how to build a mission-critical Machine Learning environment leveraging different Kafka components: Kafka messaging and Kafka Connect for data movement from and into different sources and sinks, Kafka Streams for model deployment and inference in real time, and KSQL for real time analytics of predictions, alerts and model accuracy.

Updated slide deck and talk from September 2018 at ApacheCon Montreal.

Published in: Technology
  • Be the first to comment

Apache Kafka Open Source Ecosystem for Machine Learning at Extreme Scale (ApacheCon, Montreal, Sept 2018)

  1. 1. 1Confidential Machine Learning at Extreme Scale with the Apache Kafka Open Source Ecosystem Kai Waehner Technology Evangelist kontakt@kai-waehner.de LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de How to build a Scalable, Mission-Critical Machine Learning Infrastructure
  2. 2. 2Apache Kafka and Machine Learning – Kai Waehner Poll In which of these environments, if at all, are you using Apache Kafka today? 1. On premise 2. A public cloud 3. Our own virtual private cloud 4. We do not currently use Apache Kafka
  3. 3. 3Apache Kafka and Machine Learning – Kai Waehner What is eXtreme Scale? • High Volume of Events (millions, billions, trillions) • Big Data Sets for Analytics (GB, TB, PB) • Dynamic Scalability for Training (minutes, hours, days) • Real Time Prediction Process for Deployment (ms) • Hybrid Deployments (different frameworks and clouds)
  4. 4. 4Apache Kafka and Machine Learning – Kai Waehner Agenda 1) Added Business Value via Machine Learning 2) Apache Kafka Ecosystem as Infrastructure for Machine Learning 3) Data Ingestion and Preprocessing with Apache Kafka for Model Training 4) Model Inference in Real Time with Kafka Streams and KSQL 5) DevOps and Monitoring of a Machine Learning Infrastructure
  5. 5. 5Apache Kafka and Machine Learning – Kai Waehner Agenda 1) Added Business Value via Machine Learning 2) Apache Kafka Ecosystem as Infrastructure for Machine Learning 3) Data Ingestion and Preprocessing with Apache Kafka for Model Training 4) Model Inference in Real Time with Kafka Streams and KSQL 5) DevOps and Monitoring of a Machine Learning Infrastructure
  6. 6. 6Apache Kafka and Machine Learning – Kai Waehner Machine Learning ... allows computers to find hidden insights without being explicitly programmed where to look. Machine Learning • Decision Trees • Naïve Bayes • Clustering • Neural Networks • etc. Deep Learning • CNN • RNN • Autoencoder • etc.
  7. 7. 7Apache Kafka and Machine Learning – Kai Waehner Real World Examples of Machine Learning Spam Detection Search Results + Product Recommendation Picture Detection (Friends, Locations, Products) Your Company The Next Disruption: Google Beats Go Champion
  8. 8. 8Apache Kafka and Machine Learning – Kai Waehner Leverage Machine Learning to Analyze and Act on Critical Business Moments Seconds Minutes Hours Price Optimization Predictive Maintenance Fraud Detection Cross Selling Transportation Rerouting Customer Service Inventory Management Windows of Opportunity
  9. 9. 9Apache Kafka and Machine Learning – Kai Waehner Live Demo – Building an Analytic Model Neural Networks in Action http://playground.tensorflow.org/
  10. 10. 10Apache Kafka and Machine Learning – Kai Waehner Languages, Frameworks and Tools for Machine Learning There is no Allrounder à ML-independent infrastructure needed! Portable Format for Analytics (PFA)
  11. 11. 11Apache Kafka and Machine Learning – Kai Waehner Machine Learning with H2O.ai H2O Engine R / Python / Scala / Flow UI Java Code
  12. 12. 12Apache Kafka and Machine Learning – Kai Waehner Live Demo – Building an Analytic Model Use Case: Airline Flight Delay Prediction Machine Learning Algorithm: Deep Learning using Neural Networks Technology: H2O.ai, TensorFlow
  13. 13. 13Apache Kafka and Machine Learning – Kai Waehner H2O Deep Water (TensorFlow, MXNet, …) https://h2o-release.s3.amazonaws.com/h2o/rel-vapnik/1/ docs-website/h2o-docs/booklets/DeepWaterBooklet.pdf Deep Water (H2O + TensorFlow) Pre-Defined Networks + User-Defined Networks
  14. 14. 14Apache Kafka and Machine Learning – Kai Waehner Hidden Technical Debt in Machine Learning Systems https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
  15. 15. 15Apache Kafka and Machine Learning – Kai Waehner Agenda 1) Added Business Value via Machine Learning 2) Apache Kafka Ecosystem as Infrastructure for Machine Learning 3) Data Ingestion and Preprocessing with Apache Kafka for Model Training 4) Model Inference in Real Time with Kafka Streams and KSQL 5) DevOps and Monitoring of a Machine Learning Infrastructure
  16. 16. 16Apache Kafka and Machine Learning – Kai Waehner The Log ConnectorsConnectors Producer Consumer Streaming Engine Apache Kafka – The Rise of a Streaming Platform
  17. 17. 17Apache Kafka and Machine Learning – Kai Waehner Apache Kafka at Scale https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/63921 https://qconlondon.com/london2018/presentation/cloud-native-and-scalable-kafka-architecture (2018) (2018)
  18. 18. 18Apache Kafka and Machine Learning – Kai Waehner Netflix’ Meson: Automation Engine for ML Pipelines https://www.infoq.com/presentations/netflix-ml-meson
  19. 19. 19Apache Kafka and Machine Learning – Kai Waehner Uber’s internal ML-as-a-Service Platform https://eng.uber.com/michelangelo • Cover the end-to-end ML workflow: manage data, train, evaluate, and deploy models, make predictions, and monitor predictions • Supports various AI technologies: Traditional ML models, time series forecasting, and deep learning
  20. 20. 20Apache Kafka and Machine Learning – Kai Waehner Paypal: Real Time Fraud Detection at Scale https://www.infoq.com/presentations/paypal-data-service-fraud “Scalable model / framework independent infrastructure for fraud detection”
  21. 21. 21Apache Kafka and Machine Learning – Kai Waehner Apache Kafka’s Open Source Ecosystem as Infrastructure for Machine Learning
  22. 22. 22Apache Kafka and Machine Learning – Kai Waehner Apache Kafka’s Open Source Ecosystem as Infrastructure for Machine Learning Kafka Streams Kafka Connect Rest Proxy Schema Registry Go / .NET / Python Kafka Producer KSQL Kafka Streams
  23. 23. 23Apache Kafka and Machine Learning – Kai Waehner Poll Which components of the Apache Kafka open source ecosystem do you use? 1) Kafka Connect 2) Kafka Streams 3) Confluent Open Source (Schema Registry, REST Proxy, KSQL, non-Java Clients, …) 4) Other Open Source Add-Ons (Kafka Manager, node-rdkafka Client, …) 5) Only Kafka for Messaging / Storage + Java Producers and Consumers
  24. 24. 24Apache Kafka and Machine Learning – Kai Waehner Replay-ability – A log never forgets! Time Model B Model XModel A Producer Distributed Commit Log Different models with same data Different ML Frameworks AutoML compatible A/B Testing
  25. 25. 25Apache Kafka and Machine Learning – Kai Waehner AutoML à No Data Scientist available for the ML Tasks? Hidden Technical Debt in Machine Learning Systems http://slideplayer.com/slide/10575150/ “One-Click Data-In Model-Out simplicity” This talk does not focus on building models, but scalable infrastructure for ML
  26. 26. 26Apache Kafka and Machine Learning – Kai Waehner Agenda 1) Added Business Value via Machine Learning 2) Apache Kafka Ecosystem as Infrastructure for Machine Learning 3) Data Ingestion and Preprocessing with Apache Kafka for Model Training 4) Model Inference in Real Time with Kafka Streams and KSQL 5) DevOps and Monitoring of a Machine Learning Infrastructure
  27. 27. 27Apache Kafka and Machine Learning – Kai Waehner Kafka Connect for Data Integration “Kafka benefits under the hood” Out-of-the-Box Connectivity Data Format Conversion Simple Message Transformation Connect Syslog (“you name the interface”)
  28. 28. 28Apache Kafka and Machine Learning – Kai Waehner Kafka Connect Configuration for MQTT Connector curl -s -X POST -H 'Content-Type: application/json' http://localhost:8083/connectors -d '{ "name" : "mqtt-source", "config" : { "connector.class" : "io.confluent.connect.mqtt.MqttSourceConnector", "tasks.max" : "1", "mqtt.server.uri" : "tcp://127.0.0.1:1883", "mqtt.topics" : "temperature", "kafka.topics" : "mqtt.", "transforms":"filter", "transforms.filter.type":"com.github.kaiwaehner.kafka.connect.smt.StringFilter", "transforms.filter.topic.format":"fraud" } }'
  29. 29. 29Apache Kafka and Machine Learning – Kai Waehner Kafka Streams / KSQL for Data Preprocessing Streams “Kafka benefits under the hood” Streaming ETL Same Pipeline for Training and Serving Input Data Output Data
  30. 30. 30Apache Kafka and Machine Learning – Kai Waehner KSQL – Continuous Query for Streaming ETL CREATE STREAM vip_actions AS SELECT userid, page, action FROM clickstream c LEFT JOIN users u ON c.userid = u.user_id WHERE u.level = 'Platinum';
  31. 31. 31Apache Kafka and Machine Learning – Kai Waehner Confluent Schema Registry for Message Validation Input Data Schema Registry Output Data“Kafka benefits under the hood” Schema Definition + Evolution Forward and Backward Compatibility
  32. 32. 32Apache Kafka and Machine Learning – Kai Waehner Kafka Connect for Data Ingestion Connect “Kafka benefits under the hood” Out-of-the-Box Connectivity Data Format Conversion Simple Message Transformation
  33. 33. 33Apache Kafka and Machine Learning – Kai Waehner Model Training Benefits of Public Cloud Extreme Scale Dynamic Instances Special Hardware
  34. 34. 34Apache Kafka and Machine Learning – Kai Waehner Agenda 1) Added Business Value via Machine Learning 2) Apache Kafka Ecosystem as Infrastructure for Machine Learning 3) Data Ingestion and Preprocessing with Apache Kafka for Model Training 4) Model Inference in Real Time with Kafka Streams and KSQL 5) DevOps and Monitoring of a Machine Learning Infrastructure
  35. 35. 35Apache Kafka and Machine Learning – Kai Waehner Model Serving / Inference / Deployment / Scoring Kafka Streams KSQL “Kafka benefits under the hood” Continuous Stream Processing Reuse Preprocessing Logic from Ingestion Pipeline Serving within the application (not via REST interface) Predictions in real time Streams Input Event Prediction
  36. 36. 36Apache Kafka and Machine Learning – Kai Waehner Stream Processing Data at Rest Data in Motion
  37. 37. 37Apache Kafka and Machine Learning – Kai Waehner Kafka Streams (shipped with Apache Kafka) / KSQL (Confluent Open Source)
  38. 38. 38Apache Kafka and Machine Learning – Kai Waehner Stream Processing Pipeline APIs Adapters / Channels Integration Messaging Stream Ingest Transformation Aggregation Enrichment Filtering Stream Preprocessing Process Management Analytics (Real Time) Applications & APIs Analytics / DW Reporting Stream Outcomes • Contextual Rules • Windowing • Patterns • Analytics • Machine Learning • … Stream Analytics Index / SearchNormalization Applying an Analytic Model is just a piece of the puzzle!
  39. 39. 39Apache Kafka and Machine Learning – Kai Waehner A complete streaming microservices, ready for production at large-scale Word Count App configuration Define processing (here: WordCount) Start processing
  40. 40. 40Apache Kafka and Machine Learning – Kai Waehner Why KSQL? Population CodingSophistication Realm of Stream Processing New, Expanded Realm BI Analysts Core Developers Data Engineers Core Developers who don’t like Java Kafka Streams KSQL
  41. 41. 41Apache Kafka and Machine Learning – Kai Waehner Trade-Offs • subscribe() • poll() • send() • flush() • mapValues() • filter() • punctuate() • Select…from… • Join…where… • Group by.. Flexibility Simplicity Kafka Streams KSQL Kafka / Consumer Producer
  42. 42. 42Apache Kafka and Machine Learning – Kai Waehner When to use Kafka Streams or KSQL for Stream Processing?
  43. 43. 43Apache Kafka and Machine Learning – Kai Waehner Kafka Streams (shipped with Apache Kafka) / KSQL (Confluent Open Source) Map, filter, aggregate, apply analytic model, „any business logic“ Input Stream (Kafka Topic) Kafka Cluster Output Stream (Kafka Topic) Kafka Cluster Stream Processing Microservice (Kafka Streams / KSQL) Deployed Anywhere Java App, Docker, Kubernetes, Mesos, “you-name-it”
  44. 44. 44Apache Kafka and Machine Learning – Kai Waehner Kafka Streams and KSQL Ok. Ok. Ok. are viable for S / M / L / XL / XXL use cases
  45. 45. 45Apache Kafka and Machine Learning – Kai Waehner Option 1: RPC communication to do model inference Streams Input Event Prediction Request Response Model Serving TensorFlow Serving RPC (e.g. gRPC, HTTP)
  46. 46. 46Apache Kafka and Machine Learning – Kai Waehner Option 2: Model interference natively integrated into the App Streams Input Event Prediction
  47. 47. 47Apache Kafka and Machine Learning – Kai Waehner Stream Processing vs. Request-Response for Model Serving Pros of a Model Server: • Simple integration with existing technologies and organizational processes • Easier to understand if you come from non- streaming world • Later migration to real streaming is also possible • Model management built-in for different models, versioning and A/B testing Cons (== Pros of Deployment in the Streaming App): • Worse latency as remote call instead of local inference • No offline inference (devices, edge processing, etc.) • Coupling the availability, scalability, and latency/throughput of your Kafka Streams application with the SLAs of the RPC interface • Side-effects (e.g., in case of failure) not covered by Kafka processing (e.g., exactly once)
  48. 48. 48Apache Kafka and Machine Learning – Kai Waehner Use Case: Airline Flight Delay Prediction Machine Learning Algorithm: Neural Network built with H2O and TensorFlow Streaming Platform: Apache Kafka and Kafka Streams Live Demo – Deployment of a Trained Model Streams
  49. 49. 49Apache Kafka and Machine Learning – Kai Waehner H2O.ai Model + Kafka Streams Filter Map 1) Create H2O DL model 2) Configure Kafka Streams Application 3) Apply H2O DL model to Streaming Data 4) Start Kafka Streams App
  50. 50. 50Apache Kafka and Machine Learning – Kai Waehner Github Examples: Kafka + Machine Learning https://www.confluent.io/blog/build-deploy-scalable-machine-learning-production-apache-kafka/ https://github.com/kaiwaehner/kafka-streams-machine-learning-examples 1) git clone à 2) mvn clean package à 3) look at implementations and unit tests
  51. 51. 51Apache Kafka and Machine Learning – Kai Waehner KSQL and Deep Learning (Autoencoder) for Fraud Detection “CREATE STREAM FraudDetection AS SELECT payment_id, applyFraudModel(payment_input) FROM payment_engine;“ User Defined Function (UDF) https://www.confluent.io/blog/build-udf-udaf-ksql-5-0 https://github.com/kaiwaehner/ksql-udf-deep-learning-mqtt-iot
  52. 52. 52Apache Kafka and Machine Learning – Kai Waehner Use Case: Anomaly Detection (Payment Fraud Detection) Machine Learning Algorithm: Autoencoder built with H2O Streaming Platform: Apache Kafka and KSQL Live Demo – Prebuilt Model Embedded in KSQL Function
  53. 53. 53Apache Kafka and Machine Learning – Kai Waehner Github Examples: KSQL + Deep Learning https://www.confluent.io/blog/build-deploy-scalable-machine-learning-production-apache-kafka/ https://github.com/kaiwaehner/ksql-udf-deep-learning-mqtt-iot https://github.com/kaiwaehner/ksql-fork-with-deep-learning-function + Kafka Connect + Elasticsearch
  54. 54. 54Apache Kafka and Machine Learning – Kai Waehner Agenda 1) Added Business Value via Machine Learning 2) Apache Kafka Ecosystem as Infrastructure for Machine Learning 3) Data Ingestion and Preprocessing with Apache Kafka for Model Training 4) Model Inference in Real Time with Kafka Streams and KSQL 5) DevOps and Monitoring of a Machine Learning Infrastructure
  55. 55. 55Apache Kafka and Machine Learning – Kai Waehner Automated Model Improvement with Apache Kafka and Kafka Streams How to deploy a model once, then monitor and improve it continuously?
  56. 56. 56Apache Kafka and Machine Learning – Kai Waehner Automated Model Improvement with Apache Kafka and Kafka Streams How to improve models? 1. Manual Update 2. Continuous Batch Updating 3. Real Time à Online Model Training Your choice… All possible with Kafka!
  57. 57. 57Apache Kafka and Machine Learning – Kai Waehner Caveats for Online Model Training • Processes and infrastructure not ready • Validation needed before production • Slows down the system • Only a few ML implementations à Build your own! • Only possible for unsupervised ML (e.g. clustering) • Many use cases do not need it à Do it only when feasible!
  58. 58. 58Apache Kafka and Machine Learning – Kai Waehner Continuous Batch Updating as “Best Feasible Option” DevOps Pipeline 1. Apply the model online to make predictions 2. Collect data and train a new model 3. Automated Re-Deployment (e.g. via a Kafka Topic) https://www.confluent.io/blog/predicting-flight-arrivals-with-the-apache-kafka-streams-api/ https://www.coveros.com/services/devops/
  59. 59. 59Apache Kafka and Machine Learning – Kai Waehner Kubernetes – The Winner of the Container and DevOps Wars! https://www.infoworld.com/article/3118345/cloud-computing/why-kubernetes-is-winning-the-container-war.html http://techgenix.com/year-of-kubernetes/ Docker, Inc
  60. 60. 60Apache Kafka and Machine Learning – Kai Waehner Kubernetes for Infrastructure Deployment https://kubernetes.io/blog/2016/10/kubernetes-and-openstack-at-yahoo-japan Stateful Backend • Zookeeper and Kafka Broker Pods • REST Proxy, Schema Registry • Persistent Volumes • Kubernetes Kafka / Confluent Operator Stateless and Stateful Clients • Java / .Net / Go / Python Kafka Clients • Kafka Streams / KSQL Apps • Scalability and Elasticity
  61. 61. 61Apache Kafka and Machine Learning – Kai Waehner Monitoring the Infrastructure for Machine Learning Kafka Streams Kafka Connect Rest Proxy Schema Registry Go / .NET / Python Kafka Producer KSQL Kafka Streams Control Center Build vs. Buy Hosted vs. Managed Basic vs. Advanced
  62. 62. 62Apache Kafka and Machine Learning – Kai Waehner Warning: Early Stage with focus on TensorFlow Training, TensorFlow Serving, Jupyter… Bigger ecosystem expected soon… Including Kafka components for ingestion, serving, monitoring… Kubernetes Deployment of ML Workflows https://github.com/kubeflow/kubeflow
  63. 63. 63Apache Kafka and Machine Learning – Kai Waehner Poll Which of the following use cases are you most likely to utilize Kafka for over the next year? 1. Data pipeline (not real time, e.g. batch in Spark) 2. Data pipeline (real time) 3. Stream processing (e.g. Kafka Streams, KSQL) 4. Stream processing with machine learning 5. Other (like microservices, event sourcing, storage)
  64. 64. 64Apache Kafka and Machine Learning – Kai Waehner Key Takeaways à Data Scientist and Developers have to work together continuously (org + tech!) à Mission critical, scalable production infrastructure is key for success of Machine Learning projects à Apache Kafka Ecosystem + Cloud = Machine Learning at Extreme Scale (Ingestion, Processing, Training, Inference, Monitoring)
  65. 65. 65Apache Kafka and Machine Learning – Kai Waehner Kai Waehner Technology Evangelist kontakt@kai-waehner.de @KaiWaehner www.kai-waehner.de www.confluent.io LinkedIn Questions? Feedback? Please contact me!

×