Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DutchMLSchool. ML Automation

263 views

Published on

Automating your own Machine Learning Projects - Workshop: Working with the Masters.
DutchMLSchool: 1st edition of the Machine Learning Summer School in The Netherlands.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

DutchMLSchool. ML Automation

  1. 1. Automating your ML projects Machine Learning as a system service jao@bigml.com July 2019
  2. 2. Outline 1 ML as a System Service 2 ML as a RESTful Cloudy Service 3 Machine Learning Workflows on an ML Service 4 Client–side Workflow Automation 5 Server–side Workflow Automation 6 ML Algorithms as Server–side Workflows 7 ML in health care: diagnosis via unsupervised learning 8 Automation outside of the box
  3. 3. Outline 1 ML as a System Service 2 ML as a RESTful Cloudy Service 3 Machine Learning Workflows on an ML Service 4 Client–side Workflow Automation 5 Server–side Workflow Automation 6 ML Algorithms as Server–side Workflows 7 ML in health care: diagnosis via unsupervised learning 8 Automation outside of the box
  4. 4. Machine Learning as a System Service The goal Machine Learning as a system level service • Accessibility • Integrability • Automation • Ease of use
  5. 5. Machine Learning as a System Service
  6. 6. Machine Learning as a System Service The goal Machine Learning as a system level service The means • APIs: ML building blocks • Abstraction layer over feature engineering • Abstraction layer over algorithms • Automation
  7. 7. Outline 1 ML as a System Service 2 ML as a RESTful Cloudy Service 3 Machine Learning Workflows on an ML Service 4 Client–side Workflow Automation 5 Server–side Workflow Automation 6 ML Algorithms as Server–side Workflows 7 ML in health care: diagnosis via unsupervised learning 8 Automation outside of the box
  8. 8. RESTful-ish ML Services
  9. 9. RESTful-ish ML Services
  10. 10. RESTful-ish ML Services
  11. 11. RESTful-ish ML Services • Excellent abstraction layer • Transparent data model • Immutable resources and UUIDs: traceability • Simple yet effective interaction model • Easy access from any language (API bindings) Algorithmic complexity and computing resources management problems mostly washed away
  12. 12. RESTful done right: Whitebox resources • Your data, your model • Model reverse engineering becomes moot • Maximizes reach (Web, CLI, desktop, IoT) Example: Alexa skills using ML resources
  13. 13. Outline 1 ML as a System Service 2 ML as a RESTful Cloudy Service 3 Machine Learning Workflows on an ML Service 4 Client–side Workflow Automation 5 Server–side Workflow Automation 6 ML Algorithms as Server–side Workflows 7 ML in health care: diagnosis via unsupervised learning 8 Automation outside of the box
  14. 14. Idealized Machine Learning Workflows Dr. Natalia Konstantinova (http://nkonst.com/machine-learning-explained-simple-words/)
  15. 15. Machine Learning Workflows for real
  16. 16. Machine Learning Workflows for real
  17. 17. Machine Learning Workflows for real Jeannine Takaki, Microsoft Azure Team
  18. 18. Outline 1 ML as a System Service 2 ML as a RESTful Cloudy Service 3 Machine Learning Workflows on an ML Service 4 Client–side Workflow Automation 5 Server–side Workflow Automation 6 ML Algorithms as Server–side Workflows 7 ML in health care: diagnosis via unsupervised learning 8 Automation outside of the box
  19. 19. Example workflow: Model and evaluate Objective: Model and evaluate input data in one click We need to... • Create dataset • Split it into train and test halves • Use the train dataset to create a model • Use the test dataset to evaluate it
  20. 20. Example workflow: Web UI
  21. 21. (Non) automation via Web UI Strengths of Web UI Simple Just clicking around Discoverable Exploration and experimenting Abstract Transparent error handling and scalability
  22. 22. (Non) automation via Web UI Strengths of Web UI Simple Just clicking around Discoverable Exploration and experimenting Abstract Transparent error handling and scalability Problems of Web UI Only simple Simple tasks are simple, hard tasks quickly get hard No automation or batch operations Clicking humans don’t scale well
  23. 23. RESTful Workflow curl -X POST "https://bigml.io/dataset" -D '{"source": "source/56fbbfea200d5a3403000db7"}' curl -X POST "https://bigml.io/model" -D '{"dataset": "dataset/43ffe231a34fff333000b65"}' curl -X POST "https://bigml.io/evaluation" -D '{"dataset": "dataset/43ffe231a34fff333000b65", "model": "cluster/33e2e231a34fff333000b65"}' curl -X GET "https://bigml.io/model/1234ff45eab8c0034334"
  24. 24. Automation via HTTP Strengths of HTTP automation Scriptable Automation is possible, but that’s about it
  25. 25. Automation via HTTP Strengths of HTTP automation Scriptable Automation is possible, but that’s about it Problems of direct HTTP automation Not Simple Explicit error handling and scalability Not Discoverable Or at least not easily Not Scalable Exponential complexity with workflow size
  26. 26. Abstracting over raw HTTP: bindings
  27. 27. Example workflow: Python bindings from bigml.api import BigML api = BigML() source = 'source/5643d345f43a234ff2310a3e' dataset = api.create_dataset(source) api.ok(dataset) train_dataset = api.create_dataset(dataset, {"rate": 0.8, "seed": "raboml"}) test_dataset = api.create_dataset(dataset, {"rate": 0.8, "seed": "raboml", "out_of_bag": True}) api.ok(train_dataset) model = api.create_model(dataset) api.ok(model) api.ok(train_dataset) evaluation = api.create_evaluation(model, test_dataset) api.ok(evaluation)
  28. 28. Example workflow: Python bindings # Now do it 100 times, serially for i in range(0, 100): train = api.create_dataset(dataset, {"rate": 0.8, "seed": i}) test = api.create_dataset(dataset, {"rate": 0.8, "seed": i, "out_of_bag": True} api.ok(train) model.append(api.create_model(train)) api.ok(model) api.ok(test) evaluation.append(api.create_evaluation(model, test)) api.ok(evaluation[i])
  29. 29. Example workflow: Python bindings # Now do it 100 times, serially, and with error handling... for i in range(0, 100): try: train = api.create_dataset(dataset, {"rate": 0.8, "seed": i}) test = api.create_dataset(dataset, {"rate": 0.8, "seed": i, "out_of_bag": True}) api.ok(train) model.append(api.create_model(train)) api.ok(model) api.ok(test) evaluation.append(api.create_evaluation(model, test)) api.ok(evaluation[i]) except: # Recover, retry? # What do we do with the previous i - 1 resources?
  30. 30. Example workflow: Python bindings # More efficient if we parallelize, but at what level? for i in range(0, 100): train.append(api.create_dataset(dataset, {"rate": 0.8, "seed": i})) test.append(api.create_dataset(dataset, {"rate": 0.8, "seed": i, "out_of_bag": True}) # Do we wait here? api.ok(train[i]) api.ok(test[i]) for i in range(0, 100): model.append(api.create_model(train[i])) api.ok(model[i]) for i in range(0, 100): evaluation.append(api.create_evaluation(model, test_dataset)) api.ok(evaluation[i])
  31. 31. Example workflow: Python bindings # More efficient if we parallelize, but at what level? for i in range(0, 100): train.append(api.create_dataset(dataset, {"rate": 0.8, "seed": i})) test.append(api.create_dataset(dataset, {"rate": 0.8, "seed": i, "out_of_bag": True}) for i in range(0, 100): # Or do we wait here? api.ok(train[i]) model.append(api.create_model(train[i])) for i in range(0, 100): # and here? api.ok(model[i]) api.ok(train[i]) evaluation.append(api.create_evaluation(model, test_dataset)) api.ok(evaluation[i])
  32. 32. Example workflow: Python bindings # More efficient if we parallelize, but how do we handle errors?? for i in range(0, 100): train.append(api.create_dataset(dataset, {"rate": 0.8, "seed": i})) test.append(api.create_dataset(dataset, {"rate": 0.8, "seed": i, "out_of_bag": True}) for i in range(0, 100): api.ok(train[i]) model.append(api.create_model(train[i])) for i in range(0, 100): try: api.ok(model[i]) api.ok(test[i]) evaluation.append(api.create_evaluation(model, test_dataset)) api.ok(evaluation[i]) except: # How to recover if test[i] is failed? New datasets? Abort?
  33. 33. Client-side Machine Learning Automation Problems of bindings-based, client solutions Complexity Lots of details outside the problem domain Reuse No inter-language compatibility Scalability Client-side workflows are hard to optimize Reproducibility Noisy, complex and hard to audit development environment Not enough abstraction
  34. 34. A partial solution: CLI declarative tools # "1-click" ensemble bigmler --train data/iris.csv --number-of-models 500 --sample-rate 0.85 --output-dir output/iris-ensemble --project "Rabobank Workshop" # "1-click" dataset with parameterized fields bigmler --train data/diabetes.csv --no-model --name "4-featured diabetes" --dataset-fields "plasma glucose,insulin,diabetes pedigree,diabetes" --output-dir output/diabetes --project "Rabobank Workshop"
  35. 35. Rich, parameterized workflows: cross-validation bigmler analyze --cross-validation # parameterized input --dataset $(cat output/diabetes/dataset) --k-folds 3 # number of folds during validation --output-dir output/diabetes-validation
  36. 36. Client-side Machine Learning Automation Problems of client-side solutions Hard to generalize Declarative client tools hide complexity at the cost of flexibility Hard to combine Black–box tools cannot be easily integrated as parts of bigger client–side workflows Hard to audit Client–side development environments are complex and very hard to sandbox Not enough automation
  37. 37. Client-side Machine Learning Automation Problems of client-side solutions Complex Too fine-grained, leaky abstractions Cumbersome Error handling, network issues Hard to reuse Tied to a single programming language Hard to scale Parallelization again a problem Hard to generalize Declarative client tools hide complexity at the cost of flexibility Hard to combine Black–box tools cannot be easily integrated as parts of bigger client–side workflows Hard to audit Client–side development environments are complex and very hard to sandbox Not enough abstraction
  38. 38. Client-side Machine Learning Automation Problems of client-side solutions Complex Too fine-grained, leaky abstractions Cumbersome Error handling, network issues Hard to reuse Tied to a single programming language Hard to scale Parallelization again a problem Hard to generalize Declarative client tools hide complexity at the cost of flexibility Hard to combine Black–box tools cannot be easily integrated as parts of bigger client–side workflows Hard to audit Client–side development environments are complex and very hard to sandbox Algorithmic complexity and computing resources management problems mostly washed away are back!
  39. 39. Outline 1 ML as a System Service 2 ML as a RESTful Cloudy Service 3 Machine Learning Workflows on an ML Service 4 Client–side Workflow Automation 5 Server–side Workflow Automation 6 ML Algorithms as Server–side Workflows 7 ML in health care: diagnosis via unsupervised learning 8 Automation outside of the box
  40. 40. Machine Learning Automation
  41. 41. Machine Learning Automation Solution (scalability, reuse): Back to the server
  42. 42. Machine Learning Automation Solution (complexity, reuse): Domain-specific languages
  43. 43. Machine Learning Automation Solution (complexity, reuse): Domain-specific languages venturebeat.com
  44. 44. Machine Learning Automation Solution (complexity, reuse): Domain-specific languages
  45. 45. In a Nutshell 1. Workflows reified as server–side, RESTful resources 2. Domain–specific language for ML workflow automation
  46. 46. Workflows as RESTful Resources Library Reusable building-block: a collection of WhizzML definitions that can be imported by other libraries or scripts. Script Executable code that describes an actual workflow. • Imports List of libraries with code used by the script. • Inputs List of input values that parameterize the workflow. • Outputs List of values computed by the script and returned to the user. Execution Given a script and a complete set of inputs, the workflow can be executed and its outputs generated.
  47. 47. Ways to create WhizzML Scripts and Libraries Github Script editor Gallery Other scripts Scriptify −→
  48. 48. Metaprogramming in reflective DSLs: Scriptify Resources that create resources that create resources that create resources that create resources that create resources that create . . .
  49. 49. Server-side Workflows: the bazaar
  50. 50. Syntactic Abstraction in WhizzML: Simple workflow ;; ML artifacts are first-class citizens, ;; we only need to talk about our domain (let ([train-id test-id] (create-dataset-split id 0.8) model-id (create-model train-id)) (create-evaluation test-id model-id {"name" "Evaluation 80/20" "missing_strategy" 0}))
  51. 51. Language Interoperability in WhizzML from bigml.api import BigML api = BigML() # choose workflow script = 'script/567b4b5be3f2a123a690ff56' # define parameters inputs = {'source': 'source/5643d345f43a234ff2310a3e'} # execute api.ok(api.create_execution(script, inputs))
  52. 52. Domain Specificity and Scalability: Trivial parallelization ;; Workflow for 1 resource (let ([train-id test-id] (create-dataset-split id 0.8) model-id (create-model train-id)) (create-evaluation test-id model-id))
  53. 53. Domain Specificity and Scalability: Trivial parallelization ;; Workflow for arbitrary number of resources (let (splits (for (id input-datasets) (create-dataset-split id 0.8))) (for (s splits) (create-evaluation (s 1) (create-model (s 0)))))
  54. 54. Domain Specificity and Scalability: Trivial parallelization from bigml.api import BigML api = BigML() # choose workflow script = 'script/567b4b5be3f2a123a690ff56' # define parameters inputs = {'input-dataset': 'dataset/5643d345f43a234ff2310a30'} # execute api.ok(api.create_execution(script, inputs))
  55. 55. Domain Specificity and Scalability: Trivial parallelization from bigml.api import BigML api = BigML() # choose workflow script = 'script/567b4b5be3f2a123a690de1228' # define parameters inputs = {'input-datasets': ['dataset/5643d345f43a234ff2310a30', 'dataset/5643d345f43a234ff2310a31', 'dataset/5643d345f43a234ff2310a32', ...]} # execute api.ok(api.create_execution(script, inputs))
  56. 56. Outline 1 ML as a System Service 2 ML as a RESTful Cloudy Service 3 Machine Learning Workflows on an ML Service 4 Client–side Workflow Automation 5 Server–side Workflow Automation 6 ML Algorithms as Server–side Workflows 7 ML in health care: diagnosis via unsupervised learning 8 Automation outside of the box
  57. 57. Advanced ML algorithms with WhizzML • Functional language, immutable data structures • Rich, very high-level libraries • Full tail call optimization
  58. 58. Advanced ML algorithms with WhizzML • Many ML algorithms can be thought of as workflows • In these algorithms, machine learning operations are the primitives Make a model Make a prediction Evaluate a model • Many such algorithms can be implemented in WhizzML Reap the advantages of BigML’s infrastructure Once implemented, it is language-agnostic
  59. 59. Simple example: Model or Ensemble? • Split a dataset in test and training parts • Create a model and an ensemble with the training dataset • Evaluate both with the test dataset • Choose the one with better evaluation (f-measure) https://github.com/whizzml/examples/tree/master/model-or-ensemble
  60. 60. Simple example: Model or Ensemble? ;; Function to extract a quality measure from the evaluation results (define (quality-measure ev-id) (let (ev (fetch (wait ev-id))) (or (ev ["result" "model" "average_f_measure"] false) (raise "The models couldn't be evaluated")))) ;; Function encapsulating the full workflow (define (model-or-ensemble src-id) (let (ds-id (create-dataset src-id) [train-id test-id] (create-random-dataset-split (wait ds-id) 0.8) m-id (create-model (wait train-id)) e-id (create-ensemble train-id {"number_of_models" size}) m-f (quality-measure (create-evaluation (wait m-id) test-id)) e-f (quality-measure (create-evaluation (wait e-id) test-id))) (if (> m-f e-f) m-id e-id)))
  61. 61. Creation waits for you: a better model or ensemble ;; Function to extract a quality measure from the evaluation results (define (quality-measure ev-id) (let (ev (fetch (wait ev-id))) (or (ev ["result" "model" "average_f_measure"] false) (raise "The models couldn't be evaluated")))) ;; Function encapsulating the full workflow (define (model-or-ensemble src-id) (let (ds-id (create-dataset src-id) ;; implicit wait for ds-id: [train-id test-id] (create-random-dataset-split ds-id 0.8) ;; implicit wait for train-id: m-id (create-model train-id) e-id (create-ensemble train-id {"number_of_models" size}) ;; implicit wait for test-id: m-f (quality-measure (create-evaluation m-id test-id)) e-f (quality-measure (create-evaluation e-id test-id))) (if (> m-f e-f) m-id e-id)))
  62. 62. Creation waits for you: a better model or ensemble ;; Function to extract a quality measure from the evaluation results (define (quality-measure ev-id) (let (ev (fetch (wait ev-id))) (or (ev ["result" "model" "average_f_measure"] false) (ev ["result" "model" "r_squared"] false) (raise "The models couldn't be evaluated")))) ;; Function encapsulating the full workflow (define (model-or-ensemble src-id) (let (ds-id (create-dataset src-id) [train-id test-id] (create-random-dataset-split ds-id 0.8) m-id (create-model train-id) e-id (create-ensemble train-id {"number_of_models" size}) m-f (quality-measure (create-evaluation m-id test-id)) e-f (quality-measure (create-evaluation e-id test-id))) (if (> m-f e-f) m-id e-id)))
  63. 63. Outline 1 ML as a System Service 2 ML as a RESTful Cloudy Service 3 Machine Learning Workflows on an ML Service 4 Client–side Workflow Automation 5 Server–side Workflow Automation 6 ML Algorithms as Server–side Workflows 7 ML in health care: diagnosis via unsupervised learning 8 Automation outside of the box
  64. 64. https://xkcd.com/1619/
  65. 65. Example: Tumor detection using anomalies Given data about a tumor: • Extract the relevant features that characterize it (unsupervised learning) • Classify the tumor as either benign or malignant, improving diagnosis and avoiding unnecessary surgery
  66. 66. Example: Tumor detection using anomalies Given data about a tumor: • Extract the relevant features that characterize it (unsupervised learning) • Classify the tumor as either benign or malignant, improving diagnosis and avoiding unnecessary surgery Example: University of Wisconsin Hospital’s Cancer dataset archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)
  67. 67. Tumor detection using anomalies: workflow
  68. 68. Tumor detection using anomalies: workflow
  69. 69. Tumor detection using anomalies: workflow
  70. 70. Tumor detection using anomalies: workflow
  71. 71. Tumor detection using anomalies: Evaluation Is the anomaly score a good predictor in real cases?
  72. 72. Tumor detection using anomalies: Automation
  73. 73. Tumor detection using anomalies: Automation
  74. 74. Example: Stacked Generalization Objective: Improve predictions by modeling the output scores of multiple trained models. • Create a training and a holdout set • Create n different models on the training set (with some difference among them; e.g., single-tree vs. ensemble vs. logistic regression) • Make predictions from those models on the holdout set • Train a model to predict the class based on the other models’ predictions
  75. 75. Example: Stacked Generalization
  76. 76. Example: Stacked Generalization
  77. 77. Example: Stacked Generalization
  78. 78. Example: Stacked Generalization
  79. 79. Example: Stacked Generalization
  80. 80. Example: Stacked Generalization
  81. 81. Example: Stacked Generalization
  82. 82. Example: Stacked Generalization (define [train-id hold-id] (create-random-dataset-split dataset-id 0.5)) (define models (create* ["model" "ensemble" "logisticregression"] {"dataset" train-id} {"dataset" train-id "number_of_models" 20} {"dataset" train-id}))
  83. 83. Example: Stacked Generalization
  84. 84. Example: Stacked Generalization (define (add-prediction-column dataset model) (let (bp (create-and-wait-batch-prediction did mid)) ((fetch bp) "output_dataset_resource"))) (define pred-dataset (reduce add-prediction-column hold-id models))
  85. 85. Example: Stacked Generalization
  86. 86. Example: Stacked Generalization (define meta-model (create-model pred-dataset {"excluded_fields" (input-fields dataset-id)}))
  87. 87. Example: Stacked Generalization (define [train-id hold-id] (create-random-dataset-split input-id 0.5)) (define models (create* ["model" "ensemble" "logisticregression"] {"dataset" train-id} {"dataset" train-id "number_of_models" 20} {"dataset" train-id})) (define (add-prediction-column dataset model) (let (bp (create-and-wait-batch-prediction did mid)) ((fetch bp) "output_dataset_resource"))) (define ds (reduce add-prediction-column hold-id models)) (define meta-model (create-model ds {"excluded_fields" (input-fields input-id)}))
  88. 88. Example: Stacked Generalization
  89. 89. Example: Stacked Generalization (define [models meta-model] (read-result execution-id)) (define predictions (for (model models) (create-prediction model input-data))) (define prediction-values (for (p predictions) (prediction-value p))) (create-prediction {"model" meta-model "input_data" prediction-values})
  90. 90. Example: Best-first Feature Selection
  91. 91. Example: Best-first Feature Selection Objective: Select the n best features for modeling your data • Initialize a set S of used features as the empty set • Split your dataset into training and test sets • For i in 1 . . . n For each feature f not in S, model and evaluate with feature set S + f Greedily select ˆf, the feature with the best performance and set S ← S + ˆf https://github.com/whizzml/examples/tree/master/best-first
  92. 92. Example: Hyperparameter Optimization Objective: Find the best set of model parameters Method: Bayesian parameter optimization • Do: Generate several random sets of parameters for an ML algorithm Do 10-fold cross-validation with those parameters Learn a predictive model to predict performance from parameter values Use the model to help you select the next set of parameters to evaluate • Until you get a set of parameters that performs “well” or you get bored Sequential Model-Based Optimization for Algorithm Configuration, Hutter et al. http://www.cs.ubc.ca/labs/beta/Projects/SMAC/
  93. 93. Outline 1 ML as a System Service 2 ML as a RESTful Cloudy Service 3 Machine Learning Workflows on an ML Service 4 Client–side Workflow Automation 5 Server–side Workflow Automation 6 ML Algorithms as Server–side Workflows 7 ML in health care: diagnosis via unsupervised learning 8 Automation outside of the box
  94. 94. On–line risk management: the problem • On–line bank operations need real–time monitoring for fraud prevention • Monitored events occur at high throughput (thousands per second) with narrow detection windows (a few seconds) • Human–based and traditional statistical analysis rules need to be integrated with new ML algorithms • Regulations compliance, explanaibility and traceability are a must for auditing
  95. 95. On–line risk management: integration Existing infrastructure • Real–time event analysis systems (e.g. RiskShield) • Hand–written rules • Risk models deployed in standard formats (e.g. PMML) Machine learning platform • White–box resources usable outside the platform, including PMML format • Cleanly decoupled training/scoring • Open API callable from any standard programming language
  96. 96. On–line risk management: modelling
  97. 97. On–line risk management: predicting
  98. 98. On–line risk management: predicting
  99. 99. On–line risk management: predicting

×