Downloaded 10 times
![D
A
T
A
S
E
T
Train
tokenize
pivot
impute
tf-idf
bucketize
combine
distribs
logreg
randfores
t
xgboost
auroc
fmeasure
aupr
calibrate loco
Score
tokenize
impute
tf-idf
bucketize
combine
xgboost
calibrate loco
bucketsmean
freqs
booster
langs
distribs
Train ( Dataset[T] ) => Score ( Dataset[T] ) =>
Dataset[S]](https://image.slidesharecdn.com/makingsparkmlmodelsportable-knowyouroptions-181116054505/85/Making-Spark-ML-Models-Portable-Know-Your-Options-8-320.jpg)
Uploaded byMatthew Tovbin
Making Spark ML Models Portable - Know Your Options
The document discusses various components of the data processing and machine learning pipeline, including feature extraction, engineering, and selection, as well as model evaluation and calibration. It mentions specific techniques like tokenization, imputation, and the use of algorithms such as logistic regression and XGBoost. Additionally, it highlights the importance of runtime environments like Spark and JVM for model implementation.
Making Spark ML Models Portable - Know Your Options
- 2.
- 3. Feature Extraction Feature Engineering Feature Selection D A T A ETL Scores & Insights ModelApplication Score Calibration & Insights Best Model INFERENCE
- 4. Feature Extraction Feature Engineering Feature Selection D A T A ETL Scores & Insights ModelApplication Score Calibration & Insights INFERENCE Best Model
- 5. Feature Extraction Feature Engineering Feature Selection D A T A ETL Scores & Insights ModelApplication Score Calibration & Insights INFERENCE Best Model Spark Application
- 6.
- 7. Model Application Feature Extraction Feature Engineering Feature Selection Score Calibration D A T A S E T / R D D ETL INFERENCE Scores& Insights tokenize impute tf-idf bucketize combine xgboost calibrate loco bucket s mean freqs booste r langs distrib s
- 8. D A T A S E T Train tokenize pivot impute tf-idf bucketize combine distribs logreg randfores t xgboost auroc fmeasure aupr calibrate loco Score tokenize impute tf-idf bucketize combine xgboost calibrate loco bucketsmean freqs booster langs distribs Train( Dataset[T] ) => Score ( Dataset[T] ) => Dataset[S]
- 10. Python/R/JavaScriptSpark Runtime JVM +?Spark Runtime (load) Spark Dependencies
- 16. Python/R/JavaScriptSpark Runtime JVM +?Spark Runtime (load) Spark Dependencies
- 24. Python/R/JavaScriptSpark Runtime JVM +?Spark Runtime (load) Spark Dependencies
- 30. Do you mind havingSpark Runtime?Yes No Do you mind having JVM runtime? Do you need sub-ms latency? Yes No Custom runtim e Use Spark. EOF MLeap NoYes PFA Deep Learning? ONNX Yes No