Downloaded 38 times
Uploaded bySri Ambati
Automatic Model Documentation with H2O
ML Autodoc is a tool designed to streamline the documentation process for machine learning models by automatically generating editable reports that detail model creation aspects such as algorithms and techniques. It supports various algorithms from H2O-3 and scikit-learn, enabling data scientists to save time and improve consistency in documentation. The service is integrated into the H2O environment and driverless AI, providing customizable reports that assist with compliance and model interpretability.
More Related Content
Similar to Automatic Model Documentation with H2O
![Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/apachesparkgettingstarted-260203175547-8361bcc3-thumbnail.jpg?width=640&height=640&fit=bounds)
Automatic Model Documentation with H2O
- 1. ML AutoDoc Auto Documentationfor ML Models Nikhil Shekhar (Machine Learning Engineer)
- 2. Confidential2 Machine Learning Documentation Challenges •Tedious for Data Scientist • Time Consuming – Iterations and reviews • Inconsistent • Incomplete • Error prone • Compliance Requirements – Banks – Healthcare
- 3. Confidential3 ML AutoDoc Overview •Automatically generates editable Word doc to document model creation (algos, techniques, data, etc) • Save Data Science Resources – automatically build required documentation • Customize to your business needs • Simple to use
- 4. Confidential4 AutoDoc Support Products Scikit-learn:Machine Learning in Python, Pedregosa et al., JMLR 12, pp. 2825-2830, 2011.
- 5.
- 6. Confidential6 H2O-3 ML AutoDoc SupportedAlgorithms • AutoDocs for Supervised Learning Models (H2O-3 and XGBoost) – XGBoost – Gradient Boosting Machine – Generalized Linear Model – Deep Learning – Distributed Random Forest (including Extremely Randomized Forest) – Stacked Ensembles Packaging • Integrated in H2O Steam • Python Package Docs https://s3.amazonaws.com/artifacts.h2o.ai/snapshots/ai/h2o/ml_autodoc/1.0.1-6/steamguide/autodoc_intro.html
- 7. Confidential7 Prediction StatsPartial Dependence FeatureImportanceActual vs. Predicted
- 8.
- 9. Confidential9 Confidential9 • Steamexposes a service to generate model report (ML AutoDoc) for OSS H2O-3 models (e.g., GBM, AutoML) – The same service is used in Driverless AI to generate documentation for DAI models Steam: ML AutoDoc AutoDoc for OSS H2O- 3 AutoML model
- 10.
- 11. Confidential11 Scikit-learn ML AutoDocs •ML AutoDoc has initial support of 3rd party models: Scikit Learn • AutoDocs for Supervised Learning Models • Scikit-Learn Linear Models: – LogisticRegression • Scikit Learn Ensemble Methods: – RandomForestClassifier – GradientBoostingClassifier – GradientBoostingRegressor AutoDoc for OSS Scikit Gradient Boosting Model
- 12. 12 Partial Dependence Response Rate AUC ShiftDetectionConfusion Matrix
- 13.
- 14. Confidential14 AutoDocs in DriverlessAI Algorithms Supported • XGBoost, LightGBM, Tensorflow, Additional Features • Included in Driverless AI • Explainability • Customized Reports
- 15. 15 Feature Importance ScoringPipeline Actual v. Predicted PDP/ICE
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23. Confidential23 Model Tuning All ModelsMore Details
- 24. Confidential24 Features and FeatureEngineering
- 25.
- 26.
- 27.
- 28.
- 29.
- 30. Confidential31 Model Tuning All ModelsMore Details
- 31.
- 32.
- 33. Confidential34 Customization Model Diagnostics ModelInterpretability • Additional Performance Metrics • Population Stability Index • Prediction Statistics per Quantile • Actual vs Predicted Plots • GINI Plot • Diagnostics on New Datasets • Perform Model Diagnostics on a list of new datasets • Partial Dependence Plots • Generate partial dependence plots on the n most important features • Includes histogram with frequency of each feature • Individual Conditional Expectation Plot • Add partial dependence plot for specific records only • Variable Importance • Calculate variable importance on original features using Permutation Importance • Filter variables to top relative importance or top n features
- 34. Confidential35 Population Stability Index #Calculate PSI config_overrides += "nautodoc_population_stability_index=true"
- 35. Confidential36 Prediction Statistics perQuantile # Enable the Prediction Statistics for each dataset split config_overrides += "nautodoc_prediction_stats=true"
- 36. Confidential37 Variable Importance # Enablethe permutation feature importance table and plot config_overrides += "nautodoc_include_permutation_feature_importance=true"
- 37.
- 38.
- 39. Confidential40 Response Rate Plots(Binary Use Cases Only) # Enable the response rate plot for each dataset split config_overrides += "nautodoc_response_rate=true"
- 40. Confidential41 GINI Plot (BinaryUse Cases Only) # Show the Gini Plot config_overrides += "nautodoc_gini_plot=true"