Ideas for furthering the development of agentMET4FOF. The main focus is on the first use case - Conducting Data Experiments. So far, the development has been successful in the second use case of deploying machine learning but a better job needs to be done for the first. https://github.com/bangxiangyong/agentMET4FOF https://www.ptb.de/empir2018/met4fof/home/

1. Ideas for agentMET4FOF
Nov 19'
BANG XIANG YONG

2. Aim: Bridging the gap between data streams owners
and data scientists in a metrological way

3. Use case 1: Conduct Data Experiments
• Description: Perform experiments to compare performance of
various ML pipelines, and develop new component for the pipeline
• Who: Data scientists (NPL , LNE)
Use case 2: Deploy ML pipeline
• Description: After figuring out the ‘best’ pipeline, deploy it with live
data streams where new data is pouring in
• Who: Sensors and Data Stream owners (BENE, OPENSENECA,
SHOESTRING, COMPANIES)
Use case 3 : Quick Demo
• Description: Get quick and easy data analytics (likely without
labels), for proof of concept and quick demo of basic data analytics.
Plug in the sensors and data analytic agents to show live updates of
data analytics such as FFT, Transfer function, smoothing, etc etc
• Who: Sensors owners (BENE, OPENSENECA, SHOESTRING)

4. USE CASE 1 (CONDUCT DATA EXPERIMENTS)
Existing Historical
data
•Database
•CSV
Specify controlled
experiments
•Components of ML
pipelines
•Parameters of
components
•Mode : train split ,
batch or prequential,
etc.
Run experiments
•Start experiments with
defined end
•Plots are generated
and performance of
pipelines are recorded
Diagnosis &
Comparisons
•Aggregate results from
multiple pipelines,
kfolds, parameters
Users: Data scientists

5. USE CASE 2 (DEPLOY ML PIPELINE)
Setup sensors
•Connect sensors
to agent
framework
Create & store
historical data
•Need mechanism
to store
persistently
Run
experiments
on historical
data to find
the "best"
pipeline
Deploy
•Connect live
data-stream to
ML pipeline
Display live
predictions
Users: Sensor and data owners
*Task of Use case 1

6. USE CASE 3 (QUICK DEMO)
Setup sensors
• Connect sensors to agent
framework
Apply Transformations
• FFT
• Transfer function
• Unsupervised clustering
• Filter
Display live updates
Users: Sensor owners

7. USE CASE 1: CONDUCTING
DATA EXPERIMENTS

8. Train, Test, Evaluate
Uncontrollable
Controllable
Input Output
Data noise, quality, total
number of input variables,
optimization randomness
Model choices,
Hyperparameters, perturbations,
Chosen Input Variables, etc
Historical data Aggregated
models/pipelines
performances
What does Design of Experiment in ML looks like?

9. Data Stream
Training splits:
Batch/Prequential
training
List of Pipelines Evaluation metrics
Aggregated metrics
over kfold
splits/prequential,
pipelines, grid-
searches
Experiment A
Encapsulation of an "Experiment"
Based on the experiments, one wishes to
1. find out the "best" pipeline(s) by comparing to other pipelines
2. understand them by plots and visualizations.
To be fully effective, the Experiment should support techniques of "Design of Experiments"
such as by full factorial, etc (pyDOE)

10. Sub-goals of ML experiments
Understanding Data Input
Determine importance of
sensors to the model
Determine importance of
features
Feature
dropping/selection/extraction
Hyperparameters
Parameters of components in
pipelines which are set
beforehand
ANN Model architecture
Model comparisons
Compare effectiveness of
pipelines (ANN vs LDA vs etc)
Robustness of models
Sensitivity to perturbations of
noise, bias, synchronization, etc

11. Viewing data analytics as pipelines
Data Stream FFT BFC
Pearson
Correlation
Linear
Discriminant
Analysis
Evaluation
Pipeline A
Pipeline B
Data
Stream
FFT BFC ANN Evaluation
Data
Stream
Deep
Neural Net
Evaluation Pipeline C
• Example 1: Comparing different pipelines

12. Viewing data analytics as pipelines
Data
Stream
Feature
extraction
Feature
selection
ML Base
Model
Evaluation
Pipeline A
Pipeline A*
Data
Stream
Add Noise
or Bias
Feature
extraction
Feature
selection
ML Base
Model
Evaluation
• Example 2: Investigating effects of noise and bias

13. PROPOSAL:
Need of
new
structures
MLFlow for recording Experiments
pyDOE for aiding DOE
Pipelines will be part of Experiments
Experiment
Analogous to sklearn: Made of up of classes with fit & transform functions
Handling of x (inputs) and y (targets)
Plug and play
Easy to include new plotting functions
Easy to create new component for pipeline
Pipeline
Compatible with sklearn, sklearn multiflow, generators, etc.Historical Data

14. ADDITIONAL
RESOURCES
(1)
1. https://www.cs.purdue.edu/homes/neville/courses/573/readi
ngs/08_design-and-analysis-expts.pdf
2. https://medium.com/@hadyelsahar/how-do-you-manage-
your-machine-learning-experiments-ab87508348ac
3. http://www.lithoguru.com/scientist/statistics/course.html
4. https://towardsdatascience.com/the-rise-of-the-term-mlops-
3b14d5bd1bdb

15. ADDITIONAL
RESOURCES
(2)