Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Advanced Spark and TensorFlow Meetup 08-04-2016 One Click Spark ML Pipeline Deploy to Production


Published on

Empowering the Data Scientist with "1-Click" Production Deployment and Canary Testing of High-Performance and Highly-Scalable Spark ML and TensorFlow Models directly from Jupyter/iPython Notebooks using Docker, Kubernetes, Netflix OSS, Microservices, and Spinnaker.

With proper tooling and metrics, Data Scientists can directly deploy, analyze, A/B test, rollback, and scale out their Spark ML and TensorFlow model into live production serving with zero friction.

We will show you the open source tools that we've built based on Docker, Kubernetes, Netflix Open Source, Microservices, Spinnaker - and even Chaos Monkey!

Speaker: Chris Fregly @ PipelineIO, formerly Databricks and Netflix

Published in: Software
  • Be the first to comment

Advanced Spark and TensorFlow Meetup 08-04-2016 One Click Spark ML Pipeline Deploy to Production

  1. 1. ADVANCED SPARK AND TENSORFLOW MEETUP Updates and Announcements August 4, 2016
  2. 2. MEETUP METRICS • 4,000 Members in 1Year!? • Github Repo: 600 Stars, 230 Forks • DockerHub Repo: 5,400 Pulls!!
  3. 3. MEETUP AGENDA • Deploying and Scaling ML Models with PipelineIO Open Source (Chris Fregly) • Fundamental Algorithms of Neural Networks (Sam Abrahams)
  4. 4. PIPELINE.IO ExtendingYour ML Pipelines into Production 100% Open Source!
  5. 5. BRAINSTORMING AND VALIDATING • Major Gaming Company • Large Ride Sharing Service • Popular Q & A Site • Online Clothing Retailer • DominantVideo Streaming
  6. 6. PIPELINE.IO FOCUS • Model Deploying andTesting • Model Scaling and Serving • Online ModelTraining • Dynamic Model Optimizing
  7. 7. MODEL DEPLOYING AND TESTING Continuously Test and Deploy Models in Production!
  8. 8. DEMO! Deploy Spark ML DecisionTree to Production Deploy to Cloud or On-Premise!
  10. 10. DEMO! Circuit Breakers and Request Batching
  11. 11. ONLINE MODEL TRAINING • Continuous, Incremental, and Partial Training • Kafka + Spark Streaming + Spark ML • Real-time, Dynamic Recommendations
  12. 12. DEMO! Real-time, Dynamic Recommendations
  13. 13. DYNAMIC MODEL OPTIMIZING Generate Optimized Code from Spark ML!
  14. 14. DEMO! Dynamic Code Generation of DecisionTree
  16. 16. WE’RE HIRING!! • Kafka, Spark ML, and TensorFlow Contributors • Systems Engineers • GPU/CUDA Engineers • C++, Java, Scala, Python ONLY NICE PEOPLE!!
  17. 17. SANTA CLARA WORKSHOP, AUG 6, 2016