Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Hopsworks - The Platform for Data-Intensive AI

129 views

Published on

Cloud Native Night July 2019, Munich: Talk by Steffen Srohsschmiedt (@grohsschmiedt, Head of Cloud at LogicalClocks)

=== Please download slides if blurred! ===

Abstract: Machine Learning (ML) pipelines are the fundamental building block for productionizing ML code. Building such pipelines with Big Data is a complex process. The different stages in ML pipelines also need to be orchestrated, from data ingestion and data transformation, to feature engineering, to model training, serving and monitoring.
Hopsworks is an open-source data platform that can be used to both develop and operate horizontally scalable machine learning (ML) pipelines. A key part of our pipelines is the world's first open-source Feature Store, that acts as a data warehouse for features, providing a natural API between data engineers - who write feature engineering code - and Data Scientists, who select features from the feature store to generate training/test data for models.

Join us next time: https://www.meetup.com/Cloud-Native-muc/events

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Hopsworks - The Platform for Data-Intensive AI

  1. 1. Hopsworks The Platform for Data-Intensive AI Steffen Grohsschmiedt Head of Cloud steffen@logicalclocks.com @grohsschmiedt
  2. 2. Hopsworks Timeline “If you’re working with big data and Hadoop, this one paper could repay your investment in the Morning Paper many times over.... HopsFS is a huge win.” - Adrian Colyer, The Morning Paper World’s fastest Hadoop Published at USENIX FAST with Oracle and Spotify World’s First #1 GPUs-as-a-Resource support in the Hopsworks platform World’s First #3 Open Source Feature Store for Machine Learning World’s First #2 Distributed File System to store small files in metadata on NVMe disks Winner of IEEE.. .. Scale Challenge 2017 with HopsFS - 1.2m ops/sec 2017 2018 2019 World’s most scalable Filesystem with Multi Data Center Availability
  3. 3. Example workflow in Hopsworks at Scale 1. Insert 1m images (<100kb) in seconds 2. Train a DNN classifier using 100s of GPUs 3. Run a Spark job to identify all objects in the 1m images and add the image annotations (JSON) as extended metadata to HopsFS 4. “show me the images with >3 bicycles” and get a sub-second response. Data scientists: Do it all in Jupyter notebooks and Python (if you want)! Ops folks: Remove the image directory, and elasticsearch is auto-cleaned up! Images Train DNN HopsFS Image Search App 1. 2. 3. 4. Elastic
  4. 4. Data validation Distributed Training Model Serving A/B Testing Monitoring Pipeline Management HyperParameter Tuning Feature Engineering Data Collection Hardware Management Data Model Prediction φ(x) Hopsworks hides the Complexity of Deep Learning *Figure from “Technical Debt in Machine Learning Systems”, Google research paper
  5. 5. Data validation Distributed Training Model Serving A/B Testing Monitoring Pipeline Management HyperParameter Tuning Feature Engineering Data Collection Hardware Management Data Model Prediction φ(x) Hopsworks hides the Complexity of Deep Learning Hopsworks Feature Store
  6. 6. Data validation Distributed Training Model Serving A/B Testing Monitoring Pipeline Management HyperParameter Tuning Feature Engineering Data Collection Hardware Management Data Model Prediction φ(x) Hopsworks hides the Complexity of Deep Learning Hopsworks Feature Store Hopsworks REST API
  7. 7. What is Hopsworks? Efficiency & Performance Security & GovernanceUsability & Process Secure Multi-Tenancy Project-based restricted access Encryption At-Rest, In-Motion TLS/SSL everywhere AI-Asset Governance Models, experiments, data, GPUs Data/Model/Feature Lineage Discover/track dependencies Jupyter/Python Development Notebooks in pipelines Version Everything Code, Infrastructure, Data Model Serving on Kubernetes TF Serving, MLeap, SkLearn End-to-End ML Pipelines Orchestrated by Airflow Feature Store Data warehouse for ML Distributed Deep Learning Faster with more GPUs HopsFS NVMe speed with Big Data Horizontally Scalable Ingestion, DataPrep, Training, Serving FS
  8. 8. Which services require Distributed Metadata (HopsFS)? Efficiency & Performance Security & GovernanceUsability & Process Secure Multi-Tenancy Project-based restricted access Encryption At-Rest, In-Motion TLS/SSL everywhere AI-Asset Governance Models, experiments, data, GPUs Data/Model/Feature Lineage Discover/track dependencies Jupyter/Python Development Notebooks in pipelines Version Everything Code, Infrastructure, Data Model Serving on Kubernetes TF Serving, MLeap, SkLearn End-to-End ML Pipelines Orchestrated by Airflow Feature Store Data warehouse for ML Distributed Deep Learning Faster with more GPUs HopsFS NVMe speed with Big Data Horizontally Scalable Ingestion, DataPrep, Training, Serving FS
  9. 9. End-to-End ML Pipelines in Hopsworks
  10. 10. End-to-End Pipelines can be factored into stages
  11. 11. Typical Feature Store Pipelines
  12. 12. Hopsworks’ Feature Store
  13. 13. Dev View: Pipelines of Jupyter Notebooks in Airflow
  14. 14. Hopsworks development environment
  15. 15. First Class Python: Conda in the Cluster Conda Repo Hopsworks Cluster No need to write Dockerfiles
  16. 16. Demo
  17. 17. How to get started with Hopsworks? @hopsworks Register for a free account at: www.hops.site Images available for AWS, GCE, Virtualbox. https://www.logicalclocks.com/ https://github.com/logicalclocks/hopsworks Reach us

×