The document discusses feature stores and their importance in machine learning pipelines. It introduces the concept of a feature store as a central repository for storing curated and documented features. A feature store provides standardized access to features for both model training and serving, and allows for feature reuse across models and teams. It also enables automatic feature documentation, versioning, and backfilling. The document argues that a feature store can help disentangle complex ML pipelines and reduce costs. Hopsworks is presented as an open-source feature store implementation.

1. Feature Store: the missing data layer in ML pipelines?1
HopsML Stockholm
Kim Hammar
kim@logicalclocks.com
January 29, 2019
1
Kim Hammar and Jim Dowling. Feature Store: the missing data layer in ML pipelines?
https://www.logicalclocks.com/feature-store/. 2018.
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 1 / 19

2. Model
ϕ(x)
Data Predictions
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 2 / 19

3. Model
ϕ(x)
Data Predictions
Distributed Training
Data Validation
Feature Engineering
Data Collection
Hardware
Management
HyperParameter
Tuning
Model
Serving
Pipeline Management
A/B
Testing
Monitoring
2
2
Image inspired from Sculley et al. (Google) Hidden Technical Debt in Machine Learning Systems
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 3 / 19

4. Outline
1 What is a Feature Store
2 Why You Need a Feature Store
3 How to Build a Feature Store (Hopsworks Feature Store)
4 Demo
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 4 / 19

5. ϕ(x)




y1
...
yn








x1,1 . . . x1,n
... . . .
...
xn,1 . . . xn,n




ˆy
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 5 / 19

6. ϕ(x)




y1
...
yn








x1,1 . . . x1,n
... . . .
...
xn,1 . . . xn,n




ˆy
?
_
_("))_/
_
3
Jeremy Hermann and Mike Del Balso. Scaling Machine Learning at Uber with Michelangelo.
https://eng.uber.com/scaling-michelangelo/. 2018.
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 6 / 19

7. ϕ(x)




y1
...
yn








x1,1 . . . x1,n
... . . .
...
xn,1 . . . xn,n




ˆy
?
_
_("))_/
_
“Data is the hardest part of ML and the most important piece to
get right.
Modelers spend most of their time selecting and transforming
features at training time and then building the pipelines to deliver
those features to production models.”
- Uber3
3
Jeremy Hermann and Mike Del Balso. Scaling Machine Learning at Uber with Michelangelo.
https://eng.uber.com/scaling-michelangelo/. 2018.
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 6 / 19

8. ϕ(x)




y1
...
yn








x1,1 . . . x1,n
... . . .
...
xn,1 . . . xn,n




ˆyFeature Store
“Data is the hardest part of ML and the most important piece to
get right.
Modelers spend most of their time selecting and transforming
features at training time and then building the pipelines to deliver
those features to production models.”
- Uber4
4
Jeremy Hermann and Mike Del Balso. Scaling Machine Learning at Uber with Michelangelo.
https://eng.uber.com/scaling-michelangelo/. 2018.
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 7 / 19

9. Solution: Disentangle ML Pipelines
with a Feature Store
Raw/Structured Data
Feature Store
Feature Engineering Training
Models
b0
x0,1
x0,2
x0,3
b1
x1,1
x1,2
x1,3
ˆy
A feature store is a central vault for storing documented, curated, and
access-controlled features.
The feature store is the interface between data engineering and data
model development
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 8 / 19

10. What is a Feature?
A feature is a measurable property of some data-sample
A feature could be..
An aggregate value (min, max, mean, sum)
A raw value (a pixel, a word from a piece of text)
A value from a database table (the age of a customer)
A derived representation: e.g an embedding or a cluster
Features are the fuel for AI systems:




x1
...
xn




Features
b0
x0,1
x0,2
x0,3
b1
x1,1
x1,2
x1,3
ˆy
Model θ
ˆy
Prediction
L(y, ˆy)
Loss
Gradient
θL(y, ˆy)
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 9 / 19

11. Motivation for the Feature Store
The feature store enables:
Reusability of features between models and teams
Automatic backfilling of features
Automatic feature documentation and analysis
Feature versioning
Standardized access of features between training and serving
Feature discovery
Num. Curated Features in Feature Store
Avg.CostofanewMLProject
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 10 / 19

12. Reusing Features
Without a Feature Store is Complex
Data Sources Dataset 1 Dataset 2 . . . Dataset n




w1,1 . . . w1,n
...
...
...
wn,1 . . . wn,n




Features




w1,1 . . . w1,n
...
...
...
wn,1 . . . wn,n




Features




w1,1 . . . w1,n
...
...
...
wn,1 . . . wn,n




Features
Siloed Feature Sets
Without a feature store it is typical to
have feature sets stored in
isolation from each other.
Models
Models are trained using sets of features.
Without a feature store each model typically
defines its own feature definitions,
without feature sharing across models.
b0
x0,1
x0,2
x0,3
b1
x1,1
x1,2
x1,3
ˆy≥ 0.9 < 0.9
≥ 0.2
< 0.2
≥ 11.2
< 11.2
B B
A
(−1, −1) (−8, −8)(−10, 0) (0, −10)
40 60 80 100
160
180
200
X
Y
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 11 / 19

13. Reusing Features
With a Feature Store is Simple
Data Sources Dataset 1 Dataset 2 . . . Dataset n
Feature Store
Feature Store
A data management platform for machine learning.
The interface between data engineering and data science.
Models
Models are trained using sets of features.
The features are fetched from the feature store
and can overlap between models.
b0
x0,1
x0,2
x0,3
b1
x1,1
x1,2
x1,3
ˆy≥ 0.9 < 0.9
≥ 0.2
< 0.2
≥ 11.2
< 11.2
B B
A
(−1, −1) (−8, −8)(−10, 0) (0, −10)
40 60 80 100
160
180
200
X
Y
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 12 / 19

14. The Components of a Feature Store
The Storage Layer: For storing feature data in the feature store
The Metadata Layer: For storing feature metadata (versioning,
feature analysis, documentation, jobs)
The Feature Engineering Jobs: For computing features
The Feature Registry: A user interface to share and discover
features
The Feature Store API: For writing/reading to/from the feature
store
Feature Storage
Feature Metadata Jobs
Feature Registry API
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 13 / 19

15. Feature Storage
Feature
Computation
Raw/Structured Data
Data Lake
Feature
Group 1
Feature
Group 2
Feature
Group 3
Feature
Group 4
project_featurestore.db
Hive
Metastore
Foreign keys
Feature Storage
Feature Metadata Jobs
Feature Registry API
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 14 / 19

16. Feature Metadata
Feature
Computation
Raw/Structured Data
Data Lake
Feature
Group 1
Feature
Group 2
Feature
Group 3
Feature
Group 4
project_featurestore.db
Hive
Metastore
Featurestore
Metadata
Foreign keys
Foreign keys
Feature Storage
Feature Metadata Jobs
Feature Registry API
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 15 / 19

17. Feature Registry and API
Feature
Computation
Raw/Structured Data
Data Lake
Feature
Group 1
Feature
Group 2
Feature
Group 3
Feature
Group 4
project_featurestore.db
Hive
Metastore
Featurestore
Metadata
Foreign keys
Foreign keys
Feature registry (UI)
REST API
Program APIs
Feature Storage
Feature Metadata Jobs
Feature Registry API
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 16 / 19

18. Feature
Computation
Raw/Structured Data
Data Lake
Feature Store
Curated Features
Model
b0
x0,1
x0,2
x0,3
b1
x1,1
x1,2
x1,3
ˆy
Demo-Setting
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 17 / 19

19. Summary
Machine learning comes with a high technical cost
Machine learning pipelines needs proper data management
A feature store is a place to store curated and documented features
The feature store serves as an interface between feature engineering
and model development, it can help disentangle complex ML pipelines
Hopsworks5 provides the world’s first open-source feature store
@hopshadoop
www.hops.io
@logicalclocks
www.logicalclocks.com
We are open source:
https://github.com/logicalclocks/hopsworks
https://github.com/hopshadoop/hops
6
5
Jim Dowling. Introducing Hopsworks. https://www.logicalclocks.com/introducing-hopsworks/. 2018.
6
Thanks to Logical Clocks Team: Jim Dowling, Seif Haridi, Theo Kakantousis, Fabio Buso, Gautier Berthou,
Ermias Gebremeskel, Mahmoud Ismail, Salman Niazi, Antonios Kouzoupis, Robin Andersson, and Alex Ormenisan
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 18 / 19

20. References
Hopsworks’ feature store7 (the only open-source one!)
Uber’s feature store8
Airbnb’s feature store9
Comcast’s feature store10
GO-JEK’s feature store11
HopsML12
Hopsworks13
7
Kim Hammar and Jim Dowling. Feature Store: the missing data layer in ML pipelines?
https://www.logicalclocks.com/feature-store/. 2018.
8
Li Erran Li et al. “Scaling Machine Learning as a Service”. In: Proceedings of The 3rd International
Conference on Predictive Applications and APIs. Ed. by Claire Hardgrove et al. Vol. 67. Proceedings of Machine
Learning Research. Microsoft NERD, Boston, USA: PMLR, 2017, pp. 14–29. URL:
http://proceedings.mlr.press/v67/li17a.html.
9
Nikhil Simha and Varant Zanoyan. Zipline: Airbnb’s Machine Learning Data Management Platform.
https://databricks.com/session/zipline-airbnbs-machine-learning-data-management-platform. 2018.
10
Nabeel Sarwar. Operationalizing Machine Learning—Managing Provenance from Raw Data to Predictions.
https://databricks.com/session/operationalizing-machine-learning-managing-provenance-from-raw-data-to-
predictions. 2018.
11
Willem Pienaar. Building a Feature Platform to Scale Machine Learning | DataEngConf BCN ’18.
https://www.youtube.com/watch?v=0iCXY6VnpCc. 2018.
12
Logical Clocks AB. HopsML: Python-First ML Pipelines.
https://hops.readthedocs.io/en/latest/hopsml/hopsML.html. 2018.
13
Jim Dowling. Introducing Hopsworks. https://www.logicalclocks.com/introducing-hopsworks/. 2018.
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 19 / 19

21. Backup Slides
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 20 / 19

22. Modeling Data in the Feature Store
A feature group is a logical grouping of features
Typically from the same input dataset and computed with the same job
A training dataset is a set of features suitable for a prediction task
Features in a training dataset are often from several feature groups
E.g features on customers, features on user activities, etc.
Training Datasets d
Feature groups g
Features f f1 f2 f3 f4 f5
g1
f6 f7 f8 f9 f10
g2
f11 f12
g3
d1 d2
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 21 / 19

23. Training Pipeline in HopsML
1 Create job/notebook to compute features and publish to the feature
store
2 Create job/notebook to read features/labels and save to a training
dataset
3 Read the training dataset into your model for training
HopsFS
Data Lake
Hive
Feature store
Raw data
Feature computation
Feature store features
hops-util
hops-util-py
Basis features Training dataset Model
b0
x0,1
x0,2
x0,3
b1
x1,1
x1,2
x1,3
ˆy
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 22 / 19

24. Hopsworks Feature Store API
Reading from the Feature Store:
from hops import featurestore
features_df = featurestore.get_features([
"average_attendance",
"average_player_age"
])
Writing to the Feature Store:
from hops import featurestore
raw_data = spark.read.parquet(filename)
pol_features = raw_data.map(lambda x: x^2)
featurestore.insert_into_featuregroup(pol_features , "pol_featuregroup")
Kim Hammar (Logical Clocks) Hopsworks Feature Store January 29, 2019 23 / 19