Hopsworks MLOps World talk june 21
•
0 likes•157 views
- Jim Dowling, CEO of Logical Clocks, discusses breaking up monolithic ML pipelines into feature pipelines and training pipelines using a feature store. - A feature store allows teams to centrally manage features and training data, enabling more modular development and improved collaboration across roles like data scientists, data engineers, and ML engineers. - Feature pipelines are used to engineer, validate, and manage features over time, while training pipelines focus on model training, evaluation, and deployment.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Hopsworks MLOps World talk june 21
Similar to Hopsworks MLOps World talk june 21 (20)
More from Jim Dowling
More from Jim Dowling (19)
Recently uploaded
Recently uploaded (20)
Hopsworks MLOps World talk june 21
- 1. Jim Dowling CEO, Logical Clocks June 2021 Breaking the Monolithic ML Pipeline with a Feature Store MLOps World
- 2. The Jellyfish reminds me of Deep Learning…. Rich Sensory Input produces Complex “Intelligent” Behaviour.
- 3. The Jellyfish reminds me of Deep Learning…. You can make them more complex by stacking layers...
- 4. The Jellyfish reminds me of Deep Learning…. And it has no Brain!
- 5. CNN Model Stateless Model Serving with Rich Input Signals. Versioned libraries prevent train/serve skew. Stateless App Images crop resize crop resize Images Training Data Images TRAINING SERVING train Same image processing libraries used in train/serving.
- 6. NLP Model Stateless Model Serving with Rich Input Signals. Versioned libraries prevent train/serve skew. Stateless App Text token- ize token- ize Tokenized Text Text TRAINING SERVING train Same text processing libraries used in train/serving.
- 7. Useless Model Recommender Model with Only Application State - no user history or context available. Stateless App UserID, Action featurize featurize Not Enough Features in Training Data UserIDs, Actions TRAINING SERVING train
- 8. Stateful Model Serving using a Feature Store. Retrieve History and Context to build Feature Vectors. Model Stateless App UserID, ProdID Build Feature Vector JOIN Features Training Data Select Features TRAINING SERVING train [Feature Vectors] Feature Store [Dataframes of Features]
- 9. The Feature Store gives our Model a Brain. Our Jellyfish now looks more like an Octopus. In New Zealand, a captive octopus apparently took a dislike to one of the staff. Every time the person passed the tank, the octopus squirted a jet of water at her. That is more like intelligence!!
- 10. Feature Store Feature Engineering Model Training Model serving Model monitoring Validate & Test Input Data The End-to-End ML Pipeline with a Feature Store
- 11. Feature Store Feature Engineering Model Training Model serving Model monitoring ML Engineers Data Scientists Model Testing Data Engineers Architects (Governance) Roles and Responsibilities in a ML Pipeline
- 12. Refactor the End-to-End ML Pipeline into Feature and Training Pipelines Data Lake, Warehouse, Kafka Model Registry Feature Pipeline Model Serving Training Pipeline Feature Store Orchestrator: Airflow, Github Actions, Jenkins Actions: Code commit, New data, time trigger (e.g., daily)
- 13. What Feature Engineering do we typically perform where? Aggregations, Data Validation Training Data Serving Raw Data Feature Store Model Repo Transformations Input Data Need to ensure no skew between training and serving transformations
- 14. Data Science Data Engineering Compliance & Regulatory Feature Store Teams use the tools of their choice, integrated with the Hopsworks Feature Store Model Serving Hopsworks is an Open, Modular Feature Store that can Plug into ML Pipelines
- 15. Feature Group Feature 1 Feature M Primary Key 0 ... ... 1 ... ... 2 ... ... ... ... ... N ... ...
- 16. Training Dataset Feature 1 LABEL (CHURN_weekly) Feature N Primary Key 0 ... ... 1 1 ... ... 0 2 ... ... 0 ... ... ... ... N ... ... 1 Feature 1 Feature M Primary Key 0 ... ... 1 ... ... 2 ... ... ... ... ... N ... ... Feature 1 Feature J Primary Key 0 ... ... 1 ... ... 2 ... ... ... ... ... N ... ... Feature Group A Feature Group B Training Dataset
- 17. When Model Serving, we retrieve Feature Vectors Feature 1 CHURN_weekly Feature N Primary Key ID ... ... N/A From App From Feature Store No Label - Predict it Lookup Features from Feature Store using “ID”
- 18. transaction_type transaction_amount user_id user_nationality user_gender transactions_fg users_fg Feature Groups Training Datasets pk join fraud_td Descriptive Statistics, Feature Correlations, Histograms ... Use for Drift Detection fraud_classifier Models Training Data Features Models Raw Data From Raw Data to Production Models in Hopsworks
- 19. Provenance Graph of Dependencies Feature Groups Models Training Datasets Changes in upstream entities trigger actions that can cause downstream computations to run Upstream Downstream
- 20. Breaking the Monolithic Pipeline into Feature Pipelines and Training Pipelines transaction_type transaction_amount user_id user_nationality user_gender transactions_fg users_fg Feature Groups Training Datasets pk join fraud_td Descriptive Statistics, Feature Correlations, Histograms ... Use for Drift Detection fraud_classifier Models Feature Pipeline Training Pipeline
- 21. Orchestrate Feature and Training Pipelines with Airflow Feature Engineering Notebook/Job Validate on Data Slices & Deploy Model Run Experiment to Train Model Select Features, File Format and Create Training Data FEATURE STORE
- 22. Validate Data + Compute Features 2 Data Source 1 Feature Store 3 Click Data every 5 seconds Sensor Data every 10 seconds App logs every hour Customer Profile once/month User clicks Event Data Structured Data Data Lake Feature Store ... Commit-0002 Commit-0097 Commit-0001 Delta Lake Snowflake Amazon S3 Amazon Redshift External Table Cached Features Import Existing Features 2 Getting Features into the Hopsworks Feature Store
- 24. Feature Pipelines: Writing Features (Python or Spark) df = spark.read.json("abfs://dev@xyz.blob.core.windows.net/d/rain.json") # feature engineering on your dataframe df.withColumn('precipitation', (df.val-min)/(max-min)) fg = fs.create_feature_group("rain", version=1, description="Rain features", primary_key=['date', 'location_id'], online_enabled=True) fg.save(df) https://docs.hopsworks.ai/
- 25. Feature Pipelines: Schema and Data Versioning for Feature Groups upsert_df = # create from some source fg = fs.get_feature_group(“rain”, 1) fg.insert(upsert_df) # Read the state ‘as of’ the timestamp fg.read(“2020-12-15 09:00:01”).show() # Read changes for the time interval fg.read_changes(“2020-12-14 09:00:01”, “2020-12-15 09:00:01”).show() Commit1 Timestamp1 Commit2 Timestamp2 ... ... ... ... Commitn Timestampn rain (v1) rain (v2) latest commit of schema (v1)
- 26. Feature Pipelines: Data Validation for Feature Groups expectation_sales = fs.create_expectation(.., rules=[Rule(name="HAS_MIN", level="WARNING", min=0), Rule(name="HAS_MAX", level="ERROR", max=1000000)]) economy_fg = fs.create_feature_group(....,expectations= [expectation_sales] ) df = # get some dataframe to ingest into the feature store # Run Data Validation Rules when data is written economy_fg.insert(df)
- 27. Feature Pipelines: Real-Time Feature Engineering df_read = spark.readStream.format("kafka")...option("subscribe", KAFKA_TOPIC_NAME).load() # Deserialise data from Kafka and create streaming query df_deser = df_read.selectExpr(....).select(...) # 10 minute window windowed10mSignalDF = df_deser .selectExpr(...) .withWatermark(...) .groupBy(window("datetime", "10 minutes"), "cc_num").agg(avg("amount")) .select(...) card_transactions_10m_agg =fs.get_feature_group("card_transactions_10m_agg", version = 1) query_10m = card_transactions_10m_agg.insert_stream(windowed10mSignalDF)
- 29. Training Pipelines: Feature Selection and Training Dataset Creation feature_join = rain_fg.select_all() .join(temperature_fg.select_all()) .join(location_fg.select_all())) td = fs.create_training_dataset("training_dataset", version=1, data_format="tfrecord", description="Training dataset, TfRecords format") td.save(feature_join)
- 30. Training Pipelines: Feature Selection and Training Dataset Creation feature_join = rain_fg.select_all() .join(temperature_fg.select_all(), on=["date", "location_id"]) .join(location_fg.select_all())) sc = fs.get_storage_connector("adls_mystorage") td = fs.create_training_dataset("training_dataset", version=1, storage_connector=sc, data_format="tfrecord", description="Training dataset, TfRecords format", splits={'train': 0.7, 'test': 0.2, 'validate': 0.1}) # The train/test/validation files are now saved to the filesystem (S3, HDFS, etc) td.save(feature_join)
- 31. Training Pipelines: Training Dataset Creation with Online Transformation Functions feature_join = rain_fg.select_all() .join(temperature_fg.select_all(), on=["date", "location_id"]) .join(location_fg.select_all())) td = fs.create_training_dataset("precpitation", version=1, transformation_functions= {"date":one_hot_encode, "precipitation":rainfall_normalize}, data_format="tfrecord", description="Training dataset, TfRecords format", splits={'train': 0.7, 'test': 0.2, 'validate': 0.1}) # The train/test/validation files are now saved to the filesystem (S3, HDFS, etc) td.save(feature_join)
- 32. Training Pipelines: Model Training def train(): dataset = tf.Dataset(“s3://path/to/training_data/“) model = … model.compile(..) model.fit(..) tf.saved_model.save(model, export_path) hops.save_model(export_path, "model_name", metrics=metrics) train()
- 33. Training Pipelines: Model Analysis with the What-If Tool* * https://examples.hopsworks.ai/ml/plotting/what_if_tool_notebook/
- 34. Model Serving in Hopsworks - KFServing Source: https://github.com/kubeflow/kfserving/tree/master/docs Kafka Statistics Feature Store Evaluation Store (part of Feature Store) Inference Requests Drift, outliers
- 35. Model Serving: Building Feature Vectors before Making Inference Requests td = fs.get_training_dataset("precipitation", 1) # dict containing the primary key name/values for the FGs in the TD input_keys = { “location_id” : “59.314781 18.070232” } # retrieve a single feature vector and apply any online transformation functions ordered_feature_vector = td.get_serving_vector(input_keys) Alternatively, any JDBC client can retrieve feature vectors using SQL queries.
- 36. Complete MLOps Infrastructure with Hopsworks and its Feature Store Code and configuration Data Lake, Warehouse, Kafka Model Registry Feature Engineering Model Serving Model Training Model Deploy Model Monitoring Model Development Retrieve Online Features Log Predictions Training Data Statistics Sync Experiment Versioning Code/Environment Versioning Feature Versioning/Statistics A/B Test Model Versioning & Statistics Serving Statistics Free-text Search Feature Store Elasticsearch RonDB